written by Eric J. Ma on 2021-12-30 | tags: continuous integration github actions til
I recently figured out two tips to make GitHub actions play nicely with conda installations. Here they are.
I recently figured out two tips to make GitHub actions play nicely with conda installations. Here they are.
The first is to use the following block:
# https://github.com/marketplace/actions/setup-miniconda#use-a-default-shell defaults: run: shell: bash -l {0}
What this does is ensure that every shell command is run in login mode. As detailed in this StackOverflow answer:
-lto insure (sic) a login bash, where the environment is correctly set;{0}, a template placeholder, replaced at pipeline execution time by the actual script command to execute.
Counterfactually, I would have had to use the deprecated source activate <env_name>,
which always made me a bit nervous.
Now, I can instead switch over to using conda activate <env_name>
before executing environment-specific commands,
thereby providing longevity for the build.
The other tip is to use the mambaforge installer to get a conda installation onto GitHub actions. The block I recently used for my causality repo is as follows:
# See: https://github.com/marketplace/actions/setup-miniconda - name: Setup miniconda uses: conda-incubator/setup-miniconda@v2 with: auto-update-conda: true miniforge-variant: Mambaforge channels: conda-forge python-version: 3.9 activate-environment: causality environment-file: environment.yml use-mamba: true
This configuration guarantees the use of mamba to solve the environment,
which means we will have blazingly fast builds.
Previously, I used to use a different GitHub action (s-weigand/setup-conda@v1),
original conda (rather than mamba),
and a convoluted build that involved environment caching.
You can take a look at an example that I copied over from my nxviz project repository
by expanding the details below.
name: Build documentation
on:
push:
branches:
- master
jobs:
build-environment:
runs-on: ubuntu-18.04
name: Build conda environment
steps:
- uses: actions/checkout@v2
name: Checkout repository
# See: https://github.com/marketplace/actions/setup-conda
- name: Setup anaconda
uses: s-weigand/setup-conda@v1
with:
conda-channels: "conda-forge"
# Build cache of environment
- name: Cache conda environment
id: cache-environment
uses: actions/cache@v2
with:
path: nxviz.tar.gz
# Note: Remember that whatever files the environment build depends on
# should be hashed and added to the key.
key: ${{ runner.os }}-env.${{ hashFiles('environment.yml') }}
- name: Build environment
if: steps.cache-environment.outputs.cache-hit != 'true'
run: |
conda env create -f environment.yml
python -m pip install .
- name: Install conda-pack
if: steps.cache-environment.outputs.cache-hit != 'true'
run: conda install -c conda-forge conda-pack
- name: Run conda-pack
if: steps.cache-environment.outputs.cache-hit != 'true'
run: conda pack -n nxviz -o nxviz.tar.gz
# See: https://github.com/actions/upload-artifact
- name: Upload environment
uses: actions/upload-artifact@v2
with:
name: nxviz-tarball
path: nxviz.tar.gz
docs:
name: Build static site docs
runs-on: ubuntu-latest
needs: build-environment
steps:
- name: Checkout repository
uses: actions/checkout@v2
# https://github.com/actions/download-artifact
- name: Download environment tarball
uses: actions/download-artifact@v2
with:
name: nxviz-tarball
- name: Unpack environment and activate it
run: |
bash scripts/ci/unpack_environment.sh
- name: Build docs
run: |
source /tmp/nxviz/bin/activate
python -m ipykernel install --user --name nxviz
make docs
- name: Deploy website
uses: peaceiris/actions-gh-pages@v3
with:
# https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-personal-access-token-personal_token
personal_token: ${{ secrets.GHPAGES_TOKEN }}
publish_dir: ./site
publish_branch: gh-pages
# destination_dir: manuscript
allow_empty_commit: false
keep_files: false
force_orphan: true
enable_jekyll: false
disable_nojekyll: false
By contrast, the new build is much smaller and easier to maintain:
name: Build documentation
on:
push:
branches:
- master
jobs:
build-docs:
runs-on: ubuntu-latest
name: Build conda environment
# https://github.com/marketplace/actions/setup-miniconda#use-a-default-shell
defaults:
run:
shell: bash -l {0}
steps:
- uses: actions/checkout@v2
name: Checkout repository
# See: https://github.com/marketplace/actions/setup-miniconda
- name: Setup miniconda
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
miniforge-variant: Mambaforge
channels: conda-forge
python-version: 3.9
activate-environment: nxviz
environment-file: environment.yml
use-mamba: true
- name: Build environment
run: |
conda activate nxviz
python -m ipykernel install --user --name nxviz
python -m pip install .
make docs
- name: Deploy website
uses: peaceiris/actions-gh-pages@v3
with:
# https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-personal-access-token-personal_token
personal_token: ${{ secrets.GHPAGES_TOKEN }}
publish_dir: ./site
publish_branch: gh-pages
# destination_dir: manuscript
allow_empty_commit: false
keep_files: false
force_orphan: true
enable_jekyll: false
disable_nojekyll: false
@article{
ericmjl-2021-better-conda-environments-on-github-actions,
author = {Eric J. Ma},
title = {Better conda environments on GitHub actions},
year = {2021},
month = {12},
day = {30},
howpublished = {\url{https://ericmjl.github.io}},
journal = {Eric J. Ma's Blog},
url = {https://ericmjl.github.io/blog/2021/12/30/better-conda-environments-on-github-actions},
}
I send out a newsletter with tips and tools for data scientists. Come check it out at Substack.
# TODO(eric): learn how to learn anything. feb 2027.
If you would like to sponsor the coffee that goes into making my posts, please consider GitHub Sponsors!