Install Anaconda on your machine
Anaconda is a way to get a Python installed on your system.
One of the neat but oftentimes confusing things about Python is that you can have multiple Python executables living around on your system. Anaconda makes it easy for you to:
Why is this a good thing? Primarily because you might have individual projects that need different version of Python and different versions of packages that are built for Python. Also, default Python installations, such as the ones shipped with older versions of macOS, tend to be versions behind the latest, which is to the detriment of your projects. Some built-in apps in an operating system may depend on that old version of Python (such as iPhoto), which means if you mess up the installation, you might break those built-in apps. Hence, you will want a tool that lets you easily create isolated Python environments.
The Anaconda Python distribution fulfills the following key needs:
Installing Anaconda on your local machine thus helps you get easy access to Python, Jupyter (see: Use Jupyter as an experimentation playground), and other tools for modelling and analysis.
If you're on macOS: I'm assuming you have installed
homebrew (see: Install homebrew on your machine) and
wget. Then, install Miniconda, which will be a lighter-weight installer, using the following command:
cd ~ wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O anaconda.sh
This will send you to your home directory, and then download the Miniconda bash script installer from Anaconda's download page.
If you're on Linux: Make sure you have
wget available on your system. Then:
cd ~ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O anaconda.sh
This will download the Miniconda installer for Linux operating sytems onto your home directory.
If you don't have
wget: You can head over to the Miniconda docs and download the bash installer to whatever location you want (the home directory is a convenient place). Rename it to
anaconda.sh to stay compatible with the instructions below.
Now, install Anaconda:
bash anaconda.sh -b -p $HOME/anaconda/
This will install the Anaconda distribution of Python onto your system inside your home directory. You can now install packages at will, without needing sudo privileges!
Bootstrap your base conda environment
In a pinch, you might want to muck around on your system with some quick-and-dirty experiment. Having a suite of packages inside your base environment can be handy. It's like having a scratch environment available.
I would recommend bootstrapping your base anaconda environment with some basic data science packages.
conda activate base conda install -c conda-forge \ scipy numpy pandas matplotlib \ numpy jupyter jupyterlab \ scikit-learn ipython ipykernel \ ipywidgets mamba
Doing so gives you an environment where you can quickly prototype new things without necessarily going through the overhead of creating an entirely new project (and with it, a full conda environment). Of course, the alternative is to set up a
scratch environment, in which you install packages on-the-fly.
mamba can be helpful if you want a faster drop-in replacement for
conda. (see: Use Mamba as a faster drop-in replacement for conda for more information.)
Use Mamba as a faster drop-in replacement for conda
Mamba is a project originally developed by the Quantstack team. They went in and solved some of the annoyances with the conda package manager - specifically the problem of how long it takes to solve an environment specification.
Mamba is available on conda-forge and PyPI. Follow the instructions on the mamba repo to install it.
If you have muscle memory and want to make the switch from
mamba as easy as possible, you can use a shell alias inside your sourced
See the page Create shell command aliases for your commonly used commands for more information on shell aliases.
Configure your conda installation
Configuring some things with conda can help lubricate your interactions with the conda package manager. It will save you keystrokes at the terminal, primarily, thus saving you time. The place to do this configuration is in the
.condarc file, which the
conda package manager searches for by default in your user's home directory.
The condarc docs are your best bet for the full configuration, but I have some favourites that I'm more than happy to share below.
Firstly, you create a file in your home directory called
.condarc. Then edit it to have the following contents:
channels: - conda-forge - defaults auto_update_conda: True always_yes: True
auto_update_condasaves me from having to update conda all the time,
always_yeslets me always answer
yto the conda installation and update prompts.
conda-forgeas the default channel above the
defaultschannel allows me to type
conda install some_packagerather than
conda install -c conda-forge some_packageeach time I want to install a package, as conda will prioritize channels according to their order under the
If you prefer, you can set the channel priorities in a different order and/or expand the list. For example, bioinformatics users may want to add in the
bioconda channel, while R users may want to add in the
r channel. Users who prefer stability may want to prioritize
defaults ahead of
What this affects is how
conda will look for packages when you execute the
conda install command. However, it doesn't affect the channel priority in your per-project
environment.yml file (see: Create one conda environment per project).
Use Jupyter as an experimentation playground
I use Jupyter notebooks in the following ways.
Firstly, I use them as a prototyping environment. They are wonderful, because I can hold the state of a program in memory and interactively modify it until I get what I need out of the program. (This especially saves on time spent re-computing things.)
Secondly, I use Jupyter as an authoring environment for interactive computational teaching material. For example, I structured Network Analysis Made Simple as a series of Jupyter notebooks.
Finally, on occasion, I use Jupyter with
ipywidgets and Voila to build out dashboards and interactive applications for my colleagues.
Get Jupyter installed in each of your environments, by including it in your
environment.yml file. (see: Create one conda environment per project)
Doing so is based on advice I received at SciPy 2016, in which one of the Jupyter developers strongly advised against "global" installations of Jupyter, to avoid package conflicts.
To get Jupyter to recognize the Python interpreter that defined by your conda environment (see: Create one conda environment per project), you need to make sure you have
ipykernel installed inside your environment. Then, use the following command:
export ENV_NAME="put_your_environment_name_here" conda activate $ENV_NAME python -m ipykernel install --user --name $ENV_NAME
Newcomers to Anaconda are usually spoonfed the GUI, but I am a proponent of launching Jupyter from the terminal because doing so makes us fully aware of our environment, including the environment variables. (see the related: Create runtime environment variable configuration files for each of your projects and Take full control of your shell environment variables)
To launch Jupyter:
In shell terms:
cd /path/to/project/directory conda activate $ENV_NAME jupyter lab
Install homebrew on your machine
Your Mac comes with a lot of neat apps, but it's a bit crippled when it comes to shell utilities. (Linux machines can use Homebrew too! Read on to see when you might need it.)
As claimed, Homebrew is the missing package manager for the Mac. From it, you can get shell utilities and apps that don't come pre-installed on your computer, such as
wget. Installing these shell utilities can give you a leg-up as you strive to gain mastery over your machine. (see: Install a suite of really cool utilities on your machine using homebrew)
Follow the instructions on the homebrew website, but essentially, it's a one bash command install. Usually, you would copy/paste it from the homebrew website, but I've copied it over so you don't have to context-switch:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
It can be executed anywhere, but if you're feeling superstitious, you can always move to your home directory first (
cd ~) before executing the command.
If you're planning to install Anaconda Install Anaconda on your machine, then make sure you install
wget, as my bootstrap step for installing Anaconda relies on using
wget to pull the installer from the internet.
brew install wget
You can also install some other cool utilities using brew! (see: Install a suite of really cool utilities on your machine using homebrew)
Linux machines usually come with their own package manager, such as
yum on CentOS and
apt on Ubuntu. If you have the necessary privileges to install packages, which usually means having
sudo privileges on your machine, then you probably don't need to install Homebrew on Linux.
However, if you do not have
sudo privileges on your machine, then you should consider installing Homebrew inside your home directory. This enables you to use
brew to install Linux utilities that might not be built-in to your system. It's a pretty neat hack to have when you're working on a managed system, such as a high performance computing system.
Configure your machine
After getting access to your development machine, you'll want to configure it and take full control over how it works. Backing the following steps are a core set of ideas:
Head over to the following pages to see how you can get things going.
Quickly remove a conda environment without using conda env remove
Removing conda environments is usually done with the following command at the terminal:
conda env remove -n some_environment_name
Sometimes, however, the command is slow.
Follow the rule of one-to-one in managing your projects
The one-to-one rule essentially means this. Each project that we work on gets:
In addition, when we name things, such as environment names, repository names, and more, we choose names that are consistent with one another (see: Sanely name things consistently for the reasons why).
Conventions help act as a lubricant - a shortcut for us to interact with others. Adopting the convention of one-to-one mappings helps us manage some of the complexity that may arise in a project.
Some teams have a habit of putting source code in one place (e.g. Bitbucket) and documentation in another (e.g. Confluence). I would discourage this; placing source code and documentation on how to use it next to each other is a much better way to work, because it gives you and your project stakeholders one single source of truth to find information related to a project.
A few guidelines can help you decide.
When a source repository matures enough such that you see a submodule that is generalizable beyond the project itself, then it's time to engage the help of a real software developer to refactor that chunk of code out of the source file into a separate package.
When the project matures enough such that there's a natural bifurcation in work that needs more independence from the original repository, then it's time to split the repository into two. At that point, apply the same principles to the new repository.
Create one conda environment per project
If you have multiple projects that you work on, but you install all project dependencies into a shared environment, then I guarantee you that at some point, you will run into dependency conflicts as you try to upgrade/update packages to try out new things.
"So what?" you might ask. Well, you'll end up breaking your code! Take this word of advice from someone who has had to deal with the consequences of having his code not working in one project even as code in another does. And finding out one day before an important presentation, and you have to put out figures. The horror!
You will want to ensure that you have an isolated conda environment for each project to keep your projects insulated from one another.
Here is a baseline that you can copy and modify at any time.
name: project ## CHANGE THIS TO YOUR ACTUAL PROJECT channels: ## Add any other channels below if necessary - conda-forge dependencies: ## Prioritize conda packages - python=3.8 - jupyter - conda - mamba - ipython - ipykernel - numpy - matplotlib - scipy - pandas - pip - pre-commit - black - nbstripout - mypy - flake8 - pycodestyle - pydocstyle - pytest - pytest-cov - pytest-xdist - pip: ## Add in pip packages if necessary - mkdocs - mkdocs-material - mkdocstrings - mknotebooks
If a package exists in both
pip and you rely primarily on
conda, then I recommend prioritizing the
conda package over the
pip package. The advantage here is that
conda's dependency solver can grab the latest compatible version without worrying about
pip clobbering over other dependencies. (h/t my reviewer Simon, who pointed out that newer versions of
pip have a dependency solver, though as far as possible, staying consistent is preferable, though mixing-and-matching is alright if you know what you're doing.)
This baseline helps me bootstrap conda environments. The packages that are in there each serve a purpose. You can read more about them on the page: Install code checking tools to help write better code.
Initially, I only specify the version of Python I want, and allow the conda package manager to solve the environment.
However, there may come a time when a new package version brings a new capability. That is when you may wish to pin the version of that particular package to be at the minimum that version. (See below for the syntax needed to pin a version.) At the same time, the new package version may break compatibility -- in this case, you will want to pin it to a maximum package version.
It's not always obvious, though, so be sure to use version control
If you wish, you can also pin versions to a minimum, maximum, or specific one, using version modifiers.
<. (You should be able to grok what is what!)
<. (Note: for pip, it is double equals
==and not single equals
So when do you use each of the modifiers?
==sparingly while in development: you will be stuck with a particular version and will find it difficult to update other packages together.
pipfrom upgrading a package beyond a certain version. This can be helpful if new versions of packages you rely on have breaking API changes.
pipfrom installing a package below a certain version. This is helpful if you've come to depend on breaking API changes from older versions.
Upgrading and/or installing packages should be done on an as-needed basis. There are two paths to do upgrade packages that I have found:
The principled way to do an upgrade is to first pin the version inside
environment.yml, and then use the following command to update the environment:
conda env update -f environment.yml
The hacky way to do the upgrade is to directly
pip install the package, and then add it (or modify its version) in the
environment.yml file. Do this only if you know what you're doing!
By practicing "one project gets one environment", then ensuring that those environments' Python interpreters are available to Jupyter is going to be crucial. If you find that your project's environment Python is unavailable, then you'll need to ensure that it's available. To do so, ensure that the Python environment has the package
ipykernel. (If not, install it by hand and add it to the
environment.yml file.) Then, run the following command:
# assuming you have already activated your environment, # replace $ENVIRONMENT_NAME with your environment's name. python -m ipykernel install --user --name $ENVIRONMENT_NAME
Now, it will show up as a "kernel" for executing Python code in your Jupyter notebooks. (see Configure Jupyter and Jupyter Lab for more information on how to configure it.)
Now, how should you name your conda environment? See the page: Sanely name things consistently!