All.devops docker.reference

Code Organization#

A repo or directory is "runnable" if it can build a Docker container and run its code inside the container
- E.g., //helpers, //amp, //orange, optimizer
Each directory that is runnable contains the files:
- changelog.txt: store the changelog
- devops: dir with all the Docker files needed to build and run a container
There are several types of devops dirs
- A devops where a container is built, run, and released
  - E.g., //amp, //helpers, //optimizer, //sports_analytics, //tutorials, //demo
- A devops where a container from a different dir is run
  - E.g., //orange reuses cmamp container

Devops to Build, Run, Release a Container#

An example of the devops structure is:

> cd //amp
> tree.sh -p devops
devops/
|-- compose/
|   `-- __init__.py
|-- debug/
|   `-- repo_compare.sh*
|-- docker_build/
|   |-- create_users.sh*
|   |-- dev.Dockerfile
|   |-- etc_sudoers
|   |-- fstab
|   |-- install_cprofile.sh
|   |-- install_dind.sh*
|   |-- install_jupyter_extensions.sh*
|   |-- install_os_packages.sh*
|   |-- install_python_packages.sh*
|   |-- poetry.lock
|   |-- poetry.toml
|   |-- prod.Dockerfile
|   `-- pyproject.toml
|-- docker_run/
|   |-- aws_credentials.sh
|   |-- bashrc
|   |-- entrypoint.sh*
|   |-- run_jupyter_server.sh*
|   |-- setenv.sh
|   `-- test_setup.sh*
|-- env/
|   `-- default.env
|-- notebooks/
...
|-- test/
...
`-- __init__.py

The organization of a devops dir is:
- compose: Docker compose files
- docker_build: everything related to building a Docker image
- docker_run: everything related to running a Docker image
- env: Docker env files
Optional dirs are:
- debug: scripts to debug and maintain the dir
- notebooks: notebooks used to analyze, monitor Docker container
  - E.g., Master_buildmeister_dashboard.ipynb
- test: everything needed to test a Docker image

Detailed Description of Files#

In devops/compose/
- devops/compose/docker-compose.yml: file generated by the invoke system based on templates, to run a Docker compose
In devops/docker_build/
- create_users.sh: create the needed users inside the container
- dev.Dockerfile: Dockerfile to build the dev version of the container
- etc_sudoers: give sudo permission to all users inside the container
- fstab: mount S3 and other dir in the container
  - TODO(gp): Not sure if it's needed anymore
- install_cprofile.sh: install the packages needed to profile Python code inside the repo
- install_dind.sh: install docker-in-docker
  - This workflow has been superseded by Docker sibling approach
- install_jupyter_extensions.sh: install Jupyter
- install_os_packages.sh: install the OS packages using the declarative approach
- os_packages/: directory containing individual OS package install scripts and configuration
- install_python_packages.sh: install the Python packages with poetry and pip
- poetry.lock: file generated by poetry after the list of packages has been generated
  - This tracks what packages were installed by poetry
- poetry.toml: configuration file for poetry
- prod.Dockerfile: Dockerfile to build the prod version of the container
- pyproject.toml: list of package specification for poetry
In devops/docker_run/
- aws_credentials.sh: loads the AWS credentials in the corresponding env vars
  - TODO(gp): This seems useless
- bashrc: basic configuration for bash shells inside a container
- entrypoint.sh: configures the container on startup
  - E.g., run setenv.sh, start dind, setup Git to run inside the container
- run_jupyter_server.sh: start Jupyter notebook
- setenv.sh: configure the virtual environment inside the container
  - E.g., update $PATH and $PYTHONPATH
  - This is equivalent to setenv.sh for the thin environment
- test_setup.sh: check that the major systems are configured properly
  - E.g., AWS, gspread, ...
  - TODO(gp): Probably obsolete
In env/
- default.env: set some variables and their defaults
  - TODO(gp): This seems useless

Declarative OS Package Installation#

The OS package installation system uses a declarative approach where packages are defined in configuration files rather than hardcoded in scripts
This approach provides better modularity, maintainability, and allows easy customization of which packages to install

Structure#

/devops/docker_build/install_os_packages.sh: main script that orchestrates the installation process
- Reads the list of packages to install from os_packages/os_packages.txt
- Executes individual install scripts located in the os_packages/ directory
- Each install script is responsible for installing a specific set of OS packages (e.g., AWS CLI, GitHub CLI, GraphViz)
devops/docker_build/os_packages/: directory containing individual OS package install scripts and configuration
- Install scripts follow the naming convention install_os_<package_name>.sh
- os_packages.txt: declarative list of install scripts to execute, one script per line
  - Lines starting with # are treated as comments
  - Empty lines are ignored
- Available install scripts:
  - install_os_aws_cli.sh: install AWS CLI V2 for multiple architectures
  - install_os_github_cli.sh: install GitHub CLI and configure repository
  - install_os_graphviz.sh: install GraphViz and related packages for diagram generation
  - install_os_publishing_tools.sh: install publishing tools like PlantUML

How to Disable or Enable OS Packages#

Edit /devops/docker_build/os_packages/os_packages.txt to control which packages are installed:
- Disable a package: add # at the beginning of the line
- Enable a package: remove the # from the beginning of the line
```
install_os_publishing_tools.sh
# install_os_aws_cli.sh          <- disabled
install_os_github_cli.sh
install_os_graphviz.sh
```
Important: While it is technically possible to build the Docker image without any OS packages listed in os_packages.txt (by commenting out all entries), be aware that certain helpers modules depend on specific OS packages to function correctly. Disabling required packages may cause runtime errors when those modules attempt to execute system-level operations. For example:
- hs3.py functions like copy_file_to_s3() and copy_data_from_s3_to_local_dir() use AWS CLI commands (aws s3 cp, aws s3 sync) which require the awscli package (installed via install_os_aws_cli.sh)
- lib_tasks_gh.py functions like gh_create_pr(), gh_workflow_list(), and gh_issue_title() use GitHub CLI commands (gh pr create, gh workflow list, gh issue view) which require the gh package (installed via install_os_github_cli.sh)
Always verify that the functionality you need will work with your selected package configuration.

How to Add a New OS Package Install Script#

Create a new install script following the naming convention:
- File name: devops/docker_build/os_packages/install_os_<package_name>.sh
- Make the script executable: chmod +x install_os_<package_name>.sh
- The script should handle package installation and any necessary configuration

Add the script to the package list:

Edit /devops/docker_build/os_packages/os_packages.txt

Add a new line with the script name:

install_os_publishing_tools.sh
install_os_aws_cli.sh
install_os_github_cli.sh
install_os_graphviz.sh
install_os_<package_name>.sh

Example script structure:

#!/usr/bin/env bash
#
# Install <package_name>.
#
apt-get install $APT_GET_OPTS <package_dependencies>
# Add installation commands here
echo "<PACKAGE_NAME> VERSION="$(<package_name> --version)

The APT_GET_OPTS environment variable is available and contains standard apt-get options (-y --no-install-recommends)

Devops to Run an Already Existing Container#

To reuse a Docker container from a different dir / repo, devops doesn't need the docker_build dir

> tree.sh -p devops
devops/
|-- debug/
|   `-- repo_compare.sh*
|-- docker_build/
|   `-- prod.Dockerfile -> ../../amp/devops/docker_build/prod.Dockerfile
|-- docker_run/
|   |-- aws_credentials.sh
...
`-- env/
        `-- default.env

Docker Invoke Flow#

`docker_bash`#

When invoke docker_bash is called, it generates a Docker compose command line like:

> IMAGE=623860924167.dkr.ecr.eu-north-1.amazonaws.com/sports_analytics:dev \
        docker compose \
            --file /Users/saggese/src/sports_analytics1/devops/compose/docker-compose.yml \
            --env-file devops/env/default.env \
            run \
            --rm \
            --name saggese.sports_analytics.app.sports_analytics1.20240817_113504 \
            --user $(id -u):$(id -g) \
            app \
            bash

The Docker compose file is generated programmatically based on the command line options

version: '3'

services:
    base_app:
        cap_add:
            - SYS_ADMIN
        environment:
            - AM_AWS_ACCESS_KEY_ID=$AM_AWS_ACCESS_KEY_ID
            ...
            - AM_ENABLE_DIND=0
            - AM_FORCE_TEST_FAIL=$AM_FORCE_TEST_FAIL
            - AM_HOST_NAME=gpmac.local
            - AM_HOST_OS_NAME=Darwin
            - AM_HOST_USER_NAME=saggese
            - AM_HOST_VERSION=22.3.0
            - AM_REPO_CONFIG_CHECK=True
            # Use inferred path for `repo_config.py`.
            - AM_REPO_CONFIG_PATH=
            - AM_TELEGRAM_TOKEN=$AM_TELEGRAM_TOKEN
            ...
            - CK_TELEGRAM_TOKEN=$CK_TELEGRAM_TOKEN
            # TODO(Vlad): consider removing, locally we use our personal tokens from files and
            # inside GitHub actions we use the `GH_TOKEN` environment variable.
            - GH_ACTION_ACCESS_TOKEN=$GH_ACTION_ACCESS_TOKEN
            # Inside GitHub Actions we use `GH_TOKEN` environment variable,
            # see https://cli.github.com/manual/gh_auth_login.
            - GH_TOKEN=$GH_ACTION_ACCESS_TOKEN
            # This env var is used by GH Action to signal that we are inside the CI.
            - CI=$CI
        image: ${IMAGE}

        restart: "no"
        volumes:
            # TODO(gp): We should pass the value of $HOME from dev.Dockerfile to here.
            # E.g., we might define $HOME in the env file.
            - ~/.aws:/home/.aws
            - ~/.config/gspread_pandas/:/home/.config/gspread_pandas/
            - ~/.config/gh:/home/.config/gh

            # Use sibling-container approach.
            - /var/run/docker.sock:/var/run/docker.sock

    # Mount `amp` when it is used as supermodule.
    app:
        extends:
            base_app
        volumes:
            - ../../:/app

    linter:
        extends:
            base_app
        volumes:
            - /Users/saggese/src/sports_analytics1:/src
            - ../../:/app
        working_dir: /src
        environment:
            - MYPYPATH

    jupyter_server:
        command: devops/docker_run/run_jupyter_server.sh
        environment:
            - PORT=${PORT}
        extends:
            app
        network_mode: ${NETWORK_MODE:-bridge}
        ports:
            # TODO(gp): Rename `AM_PORT`.
            - "${PORT}:${PORT}"

    jupyter_server_test:
        command: jupyter notebook -h 2>&1 >/dev/null
        environment:
            - PORT=${PORT}
        extends:
            app
        network_mode: ${NETWORK_MODE:-bridge}
        ports:
            - "${PORT}:${PORT}"

networks:
    default:
        name: main_network

`docker_jupyter`#

This invokes a different service
To disable vim go to Settings -> Enable Vim Mode
We use the percent format for Jupytext