Eric J Ma's Website

Better conda environments on GitHub actions

written by Eric J. Ma on 2021-12-30 | tags: continuous integration github actions til

I recently figured out two tips to make GitHub actions play nicely with conda installations. Here they are.

Ensure bash is in login mode

The first is to use the following block:

    # https://github.com/marketplace/actions/setup-miniconda#use-a-default-shell
    defaults:
      run:
        shell: bash -l {0}

What this does is ensure that every shell command is run in login mode. As detailed in this StackOverflow answer:

  • -l to insure (sic) a login bash, where the environment is correctly set;
  • {0}, a template placeholder, replaced at pipeline execution time by the actual script command to execute.

Counterfactually, I would have had to use the deprecated source activate <env_name>, which always made me a bit nervous. Now, I can instead switch over to using conda activate <env_name> before executing environment-specific commands, thereby providing longevity for the build.

Use mambaforge

The other tip is to use the mambaforge installer to get a conda installation onto GitHub actions. The block I recently used for my causality repo is as follows:

      # See: https://github.com/marketplace/actions/setup-miniconda
      - name: Setup miniconda
        uses: conda-incubator/setup-miniconda@v2
        with:
          auto-update-conda: true
          miniforge-variant: Mambaforge
          channels: conda-forge
          python-version: 3.9
          activate-environment: causality
          environment-file: environment.yml
          use-mamba: true

This configuration guarantees the use of mamba to solve the environment, which means we will have blazingly fast builds. Previously, I used to use a different GitHub action (s-weigand/setup-conda@v1), original conda (rather than mamba), and a convoluted build that involved environment caching. You can take a look at an example that I copied over from my nxviz project repository by expanding the details below.


name: Build documentation

on:
  push:
    branches:
      - master

jobs:
  build-environment:
    runs-on: ubuntu-18.04
    name: Build conda environment
    steps:
      - uses: actions/checkout@v2
        name: Checkout repository

      # See: https://github.com/marketplace/actions/setup-conda
      - name: Setup anaconda
        uses: s-weigand/setup-conda@v1
        with:
          conda-channels: "conda-forge"

      # Build cache of environment
      - name: Cache conda environment
        id: cache-environment
        uses: actions/cache@v2
        with:
          path: nxviz.tar.gz
          # Note: Remember that whatever files the environment build depends on
          # should be hashed and added to the key.
          key: ${{ runner.os }}-env.${{ hashFiles('environment.yml') }}

      - name: Build environment
        if: steps.cache-environment.outputs.cache-hit != 'true'
        run: |
          conda env create -f environment.yml
          python -m pip install .

      - name: Install conda-pack
        if: steps.cache-environment.outputs.cache-hit != 'true'
        run: conda install -c conda-forge conda-pack

      - name: Run conda-pack
        if: steps.cache-environment.outputs.cache-hit != 'true'
        run: conda pack -n nxviz -o nxviz.tar.gz

      # See: https://github.com/actions/upload-artifact
      - name: Upload environment
        uses: actions/upload-artifact@v2
        with:
          name: nxviz-tarball
          path: nxviz.tar.gz

  docs:
    name: Build static site docs
    runs-on: ubuntu-latest
    needs: build-environment

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      # https://github.com/actions/download-artifact
      - name: Download environment tarball
        uses: actions/download-artifact@v2
        with:
          name: nxviz-tarball

      - name: Unpack environment and activate it
        run: |
          bash scripts/ci/unpack_environment.sh

      - name: Build docs
        run: |
          source /tmp/nxviz/bin/activate
          python -m ipykernel install --user --name nxviz
          make docs

      - name: Deploy website
        uses: peaceiris/actions-gh-pages@v3
        with:
          # https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-personal-access-token-personal_token
          personal_token: ${{ secrets.GHPAGES_TOKEN }}
          publish_dir: ./site
          publish_branch: gh-pages
          # destination_dir: manuscript
          allow_empty_commit: false
          keep_files: false
          force_orphan: true
          enable_jekyll: false
          disable_nojekyll: false

By contrast, the new build is much smaller and easier to maintain:


name: Build documentation

on:
  push:
    branches:
      - master

jobs:
  build-docs:
    runs-on: ubuntu-latest
    name: Build conda environment

    # https://github.com/marketplace/actions/setup-miniconda#use-a-default-shell
    defaults:
      run:
        shell: bash -l {0}

    steps:
      - uses: actions/checkout@v2
        name: Checkout repository

      # See: https://github.com/marketplace/actions/setup-miniconda
      - name: Setup miniconda
        uses: conda-incubator/setup-miniconda@v2
        with:
          auto-update-conda: true
          miniforge-variant: Mambaforge
          channels: conda-forge
          python-version: 3.9
          activate-environment: nxviz
          environment-file: environment.yml
          use-mamba: true

      - name: Build environment
        run: |
          conda activate nxviz
          python -m ipykernel install --user --name nxviz
          python -m pip install .
          make docs

      - name: Deploy website
        uses: peaceiris/actions-gh-pages@v3
        with:
          # https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-personal-access-token-personal_token
          personal_token: ${{ secrets.GHPAGES_TOKEN }}
          publish_dir: ./site
          publish_branch: gh-pages
          # destination_dir: manuscript
          allow_empty_commit: false
          keep_files: false
          force_orphan: true
          enable_jekyll: false
          disable_nojekyll: false

I send out a newsletter with tips and tools for data scientists. Come check it out at Substack.

If you would like to sponsor the coffee that goes into making my posts, please consider GitHub Sponsors!

Finally, I do free 30-minute GenAI strategy calls for organizations who are seeking guidance on how to best leverage this technology. Consider booking a call on Calendly if you're interested!