Set up an awesome default gitignore for your projects
There will be some files you'll never want to commit to Git. Some include:
If you commit them, then:
Some believe that your
.gitignore should be curated. I believe that you should use a good default one that is widely applicable. To do so, go to gitignore.io, fill in the languages and operating systems involved in your project, and copy/paste the one that fits you. If you want an awesome default one for Python:
cd /path/to/project/root curl https://www.toptal.com/developers/gitignore/api/python
It will have
.env available in there too! (see: Create runtime environment variable configuration files for each of your projects)
.gitignore file is parsed according to the rules on its documentation page. It essentially follows the unix glob syntax while adding on logical modifiers. Here are a few examples to get you oriented:
These are files generated by macOS' Finder. You can ignore them by appending the following line to your
If you use MkDocs to build documentation, it will place the output into the directory
site/. You will want to ignore the entire directory appending the following line:
If you have Jupyter notebooks inside your repository, you can ignore any path containing
Adding this line will prevent your Jupyter notebook checkpoints from being committed into your Git repository.
Adhere to best git practices
Git is a unique piece of software. It does one and only one thing well: store versions of hand-curated files. Adhering to Git best practices will ensure that you use Git in its intended fashion.
The most significant point to keep in mind: only commit to Git files that you have had to create manually. That usually means version controlling:
There are also things you should actively avoid committing.
For specific files, you can set up a
See the page Set up an awesome default gitignore for your projects
for more information on preventing yourself from committing them automatically.
For Jupyter notebooks,
it is considered good practice to avoid committing notebooks that still have outputs.
It is best to clear them out using
That can be automated before committing them through the use of pre-commit hooks.
(See: Set up pre-commit hooks to automate checks before making Git commits)
One project should get one git repository
This helps a ton with organization. When you have one project targeted to one Git repository, you can easily house everything related to that project in that one Git repository. I mean everything. This includes:
In doing so, you have one mental location that you can point to for everything related to a project. This is a saner way of operating than over-engineering the separation of concerns at the beginning, with docs in one place and out-of-sync with the source code in another place... you get where we're going with this point.
Easy! Create your Git repo for the project, and then start putting stuff in there :).
Enough said here!
What should you name the Git repo? See the page: Sanely name things consistently
After you have set up your Git repo, make sure to Set up your project with a sane directory structure.
Get prepped per project
Treat your projects as if they were software projects for maximum organizational effectiveness. Why? The biggest reason is that it will nudge us towards getting organized. The "magic" behind well-constructed software projects is that someone sat down and thought clearly about how to organize things. The same principle can be applied to data analysis projects.
Firstly, some overall ideas to ground the specifics:
Some ideas pertaining to Git:
Notes that pertain to organizing files:
Notes that pertain to your compute environment:
And notes that pertain to good coding practices:
Treating projects as if they were software projects, but without software engineering's stricter practices, keeps us primed to think about the generalizability of what we do, but without the over-engineering that might constrain future flexibility.
Never commit data into version control repositories
Data should never be committed into your Git repositories. This is because
git was designed to version small files of source code; committing data, a different category of things from source code, into your repositories will first and foremost lead to repository size bloat. Also, committing data into repositories means the data get shipped alongside the source code to anybody who has access to the source code. This might not necessarily be in-line with organizational practices.
That said, in a pinch sometimes you need to work with data locally, so you might have a
data/ directory underneath the project root in which you temporarily store data. You might have chosen
data/ rather than
/tmp/ because it is easier to reference. To avoid accidentally committing any data to the repository, you might want to add the data directory to your
# Above is the rest of your .gitignore data/
The alternative is to ignore any file extensions that you know exclusively belong to the category of things called "data":
# Above is the rest of your .gitignore *.csv *.xlsx *.Rdata
Create runtime environment variable configuration files for each of your projects
When you work on your projects, one assumption you will usually have is that your development environment will look like your project's runtime environment with all of its environment variables. The runtime environment is usually your "production" setting: a web app or API, a model in a pipeline, or a software package that gets distributed. (For more on environment variables, see: Take full control of your shell environment variables)
Here, I'm assuming that you follow the practice of and that you Use pyprojroot to define relative paths to the project root.
To configure environment variables for your project,
a recommended practice is to create a
.env file in your project's root directory,
which stores your environment variables as such:
export ENV_VAR_1 = "some_value" export DATABASE_CONNECTION_STRING = "some_database_connection_string" export ENV_VAR_3 = "some_other_value"
We use the
export syntax here because we can, in our shells,
run the command
and have the environment variables defined in there applied to our environment.
Now, if you're using a Python project,
make sure you have the package
python-dotenv (Github repo here)
installed in the conda environment.
Then, in your Python
.py source files:
from dotenv import load_dotenv from pyprojroot import here import os dotenv_path = here() / ".env" load_dotenv(dotenv_path=dotenv_path) # this will load the .env file in your project directory root. # Now, get the environment variable. DATABASE_CONNECTION_STRING = os.getenv("DATABASE_CONNECTION_STRING")
In this way, your runtime environment variables get loaded into the runtime environment, and become available to all child processes started from within the shell (e.g. Jupyter Lab, or Python, etc.).
Your .env file might contain some sensitive secrets.
You should always ensure that your
.gitignore file contains
.env in it.