Skip to content

Use CI/CD to automate tasks

Continuous Integration and Continuous Deployment (CI/CD) are essential practices in modern software development, enabling automation of tasks at various stages of a project's lifecycle. This documentation aims to provide a comprehensive understanding of CI/CD, focusing on key concepts, components, and practical examples.

For the philosophical foundation behind automation, see the automation and standardization principle. This principle guides how to identify and implement automation opportunities.

For automating repetitive analysis tasks, see how to create one-off scripts. These scripts can be integrated into your CI/CD pipeline for automated execution.

For automating documentation updates, see how to write effective project documentation. CI/CD can automatically build and deploy documentation when code changes.

For ensuring code quality through automation, see how to write effective tests. Automated testing is a core component of any CI/CD pipeline.

Key Concepts of CI/CD

CI/CD is a methodology that automates the integration and deployment of code changes, ensuring that software can be reliably released at any time. Understanding the following concepts is crucial:

  • Runners: These are computers, often in the cloud, that execute your commands. They are the backbone of CI/CD, running workflows defined in YAML configuration files.
  • Workflows: A series of jobs that are executed by runners. Workflows are defined in YAML files and are triggered by specific events, such as code commits or pull requests.
  • Jobs: Each workflow consists of multiple jobs, which are collections of steps.
  • Steps: The individual tasks within a job, made up of commands or scripts that are executed sequentially.
  • Commands: These are often shell commands or scripts that you would run manually in your local environment.

Environment Considerations

One of the challenges in CI/CD is ensuring that tasks run consistently across different environments. Runners have no knowledge of your local environment, so it's crucial to make your code work reproducibly between your local setup and the CI/CD environment. This is achieved by defining the software environment of the CI/CD runner entirely in code, using configuration files in your repository. For more on project initialization, see project/repo.md.

Configuration and Environment Variables

Following the 12-factor app principles, use environment variables for project configuration. These variables are often defined in a .env file and can be auto-loaded in your shell using tools like direnv. For more details, refer to ../machine/shell.md. Common environment variables include:

  • DATABASE_URL or REDIS_URL for database connections
  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for AWS access
  • AWS_REGION for specifying the AWS region
  • AWS_BUCKET_NAME and AWS_BUCKET_REGION for S3 bucket configuration
  • OPENAI_API_KEY for OpenAI API access
  • HUGGINGFACE_API_KEY for Hugging Face API access
  • GITHUB_TOKEN for GitHub API access

Leveraging Pixi Environment

Utilize your pixi environment to define tasks, benefiting from caching and reproducibility. For more information, refer to the Pixi documentation. Tasks can include:

  • Building and publishing documentation to platforms like Confluence upon merging to the main branch
  • Running software tests on every commit
  • Deploying to production on a version bump
  • Running Python package installation tests

Practical Examples with GitHub Actions

To illustrate these concepts, here are minimal examples of tasks using GitHub Actions. Note the use of specific versions for actions, which ensures compatibility and stability:

Example 1: Running Tests

name: Run Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: prefix-dev/setup-pixi@v0.8.10
    - name: Run tests
      run: pixi run test # (1)!
  1. Assumes 'test' task is defined in pixi.toml or pyproject.toml

Example 2: Deploying to Production

name: Deploy to Production
on:
  push:
    tags:
    - 'v*.*.*'
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: prefix-dev/setup-pixi@v0.8.10
    - name: Deploy
      run: pixi run deploy # (1)
  1. Assumes 'deploy' task is defined in pixi.toml or pyproject.toml

Example 3: Building Documentation

name: Build Documentation
on:
  push:
    branches:
    - main
jobs:
  build-docs:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: prefix-dev/setup-pixi@v0.8.10
    - name: Build Docs
      run: pixi run build-docs # (1)!
    - name: Publish to Confluence
      run: uvx md2cf --space "TEAM" --parent "Project Docs" --update-existing docs/
      env:
        CONFLUENCE_URL: ${{ secrets.CONFLUENCE_URL }}
        CONFLUENCE_TOKEN: ${{ secrets.CONFLUENCE_TOKEN }}
  1. Assumes 'build-docs' task is defined in pixi.toml or pyproject.toml

Example 4: Running Python Package Installation Tests

name: Python Package Installation Tests
on: [push]
jobs:
  install-test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: prefix-dev/setup-pixi@v0.8.10
    - name: Install Package
      run: pixi run install-test # (1)!
    - name: Verify Installation
      run: pixi run verify-install # (2)!

These examples demonstrate how to automate tasks using GitHub Actions, focusing on simplicity and task-specific configurations. By understanding and implementing these CI/CD practices, you can enhance the efficiency and reliability of your software development process.

  1. Assumes 'install-test' task is defined in pixi.toml or pyproject.toml
  2. Assumes 'verify-install' task is defined in pixi.toml or pyproject.toml