Where do we get our software from? Most commonly, they come from package repositories that we interact with using package managers. Package managers come in many flavours. At least in the Python data science world, there are 2-3 package managers that one needs to be aware of, as the ecosystem is big and wide. Oftentimes we have to compose them together.

The overview is that there are "conda" packages, which share a large overlap with "pip" packages, and both share very little overlap with "system" packages.

• Prioritize conda to install packages
• Use pip only when you cannot find packages on conda
• Use docker containers for system-level packages -- this is where we need Docker containers.

These are the general "levels" of abstraction at which packages can be installed:

• "Project-specific" - at which environment.yml comes into play
• "User-specific" - at which Homebrew comes into play
• "System-wide" - for which your system package manager comes into play (if applicable)

Be sure to know what is the best level of abstraction that you need in order to compose together the toolset that you need!

When you can use pip

If you can't find a package on conda (see Prioritize conda to install packages), then pip can serve as a viable alternative for adding packages to your environment.

How to use pip with conda environments

In your environment.yml file:

name: some_env_name
channels:
- conda-forge
dependencies:
- python=3.8
- pandas
- scipy
- numpy
- ...
- pip:
  - some_pip_package==2.1

Some things to note here.

Firstly the pip section uses the same syntax for setting versions as requirements.txt. It uses == rather than =, which conda uses. This is because its contents are dumped to a temporary text file that gets parsed by pip itself.

Secondly, keep monitoring for when the package shows up on conda-forge, as that will help you retain the advantages of installing packages by a single package manager.

Why you would want to configure your conda installation

Configuring some things with conda can help lubricate your interactions with the conda package manager. It will save you keystrokes at the terminal, primarily, thus saving you time. The place to do this configuration is in the .condarc file, which the conda package manager searches for by default in your user's home directory.

The condarc docs are your best bet for the full configuration, but I have some favourites that I'm more than happy to share below.

How to configure your condarc

Firstly, you create a file in your home directory called .condarc. Then edit it to have the following contents:

channels:
  - conda-forge
  - defaults

auto_update_conda: True

always_yes: True

The whys

• The auto_update_conda saves me from having to update conda all the time,
• always_yes lets me always answer y to the conda installation and update prompts.
• Setting conda-forge as the default channel above the defaults channel allows me to type conda install some_package rather than conda install -c conda-forge some_package each time I want to install a package, as conda will prioritize channels according to their order under the channels section.

About channel priorities

If you prefer, you can set the channel priorities in a different order and/or expand the list. For example, bioinformatics users may want to add in the bioconda channel, while R users may want to add in the r channel. Users who prefer stability may want to prioritize defaults ahead of conda-forge.

What this affects is how conda will look for packages when you execute the conda install command. However, it doesn't affect the channel priority in your per-project environment.yml file (see: Create one conda environment per project).

Other conda-related pages to look at

• Install Anaconda on your machine
• Prioritize conda to install packages
• Bootstrap a scratch conda environment