Skip to content

All.create a runnable dir.how to guide

How to Create a Runnable Dir#

Definition#

  • A runnable dir is a directory containing code and a devops dir so that it can build its own container storing all the dependencies to run and be tested
  • A runnable dir can be

A Runnable Dir as Sub Directory Under a Super-Repo#

> export CSFY_RUNNABLE_DIR="ck.infra"
> export CSFY_RUNNABLE_DIR_SUFFIX="cmamp_infra"

1) Turn the Repo Into a Super-Repo with Helpers#

2) Copy and Customize Files in the Top Dir#

  • Some files need to be copied from helpers to the runnable dir to configure various tools (e.g., dev container workflow, pytest, invoke) 
    > (cd helpers_root; git pull)
> echo $CSFY_RUNNABLE_DIR
> mkdir $CSFY_RUNNABLE_DIR
> cp helpers_root/{changelog.txt,conftest.py,pytest.ini,invoke.yaml,repo_config.yaml,tasks.py} $CSFY_RUNNABLE_DIR
    • changelog.txt: this is copied from the repo that builds the used container or started from scratch for a new container 
      > vim $CSFY_RUNNABLE_DIR/changelog.txt
# cmamp-infra-1.0.0
- 2024-10-16
- First release
    • conftest.py: needed to configure pytest
    • pytest.ini: needed to configure pytest preferences
    • invoke.yaml: needed configure invoke
    • runnable_dir: needed to be created as it is used for automated discovery of runnable dirs for pytest runs 
      > touch $CSFY_RUNNABLE_DIR/runnable_dir
    • repo_config.yaml: store information about this specific repo (e.g., name, used container)
      • This needs to be modified 
        > vim $CSFY_RUNNABLE_DIR/repo_config.yaml
repo_info:
    repo_name: cmamp
...
docker_info:
    docker_image_name: cmamp-infra
...
runnable_dir_info:
    use_helpers_as_nested_module: True
    ...
    dir_suffix: cmamp_infra
    • tasks.py: the invoke tasks available in this container
      • This can be modified if needed

3) Copy and Customize Files in devops#

4) Copy and Customize Files in Thin_Client#

  • Create the dev_scripts_{runnable_dir_suffix} dir based off the template from helpers

    # Use a suffix based on the repo name and runnable dir name, e.g., `cmamp_infra`.
> SRC_DIR="./helpers_root/dev_scripts_helpers/thin_client"; echo $SRC_DIR
> DST_DIR="${CSFY_RUNNABLE_DIR}/dev_scripts_${CSFY_RUNNABLE_DIR_SUFFIX}/thin_client"; echo $DST_DIR
> mkdir -p $DST_DIR
> cp "$SRC_DIR/setenv.sh" $DST_DIR

  • The resulting dev_script should look like:

    > ls -1 $DST_DIR
setenv.sh

  • Replace file with symbolic links

    > echo $SRC_DIR
./helpers_root/dev_scripts_helpers/thin_client
> echo $DST_DIR
./ck.infra/dev_scripts_cmamp_infra/thin_client
> ./helpers_root/helpers/create_links.py --src_dir $SRC_DIR --dst_dir $DST_DIR --replace_links --use_relative_paths

  • Some common files can be replaced with symbolic links

    # Runnable dir is "ck.infra" in this case.
> ./helpers_root/helpers/create_links.py --src_dir ./helpers_root --dst_dir $CSFY_RUNNABLE_DIR --replace_links --use_relative_paths

  • Refer to Managing common files for explanation

  • Refer to Managing symbolic links between directories for how to use the commands

6) Commit Changes#

  • Commit changes 
    > git add $CSFY_RUNNABLE_DIR
> git commit -m "Add runnable dir"

7) Build a Container for a Runnable Dir#

  • Run the single-arch flow to test the flow

    > cd $CSFY_RUNNABLE_DIR
> source dev_scripts_${CSFY_RUNNABLE_DIR_SUFFIX}/thin_client/setenv.sh
> i docker_build_local_image --version 1.0.0 --container-dir-name $CSFY_RUNNABLE_DIR
> i docker_tag_local_image_as_dev --version 1.0.0
> i docker_bash --skip-pull --version 1.0.0

  • Run the multi-arch flow 

    > cd $CSFY_RUNNABLE_DIR
> source dev_scripts_${CSFY_RUNNABLE_DIR_SUFFIX}/thin_client/setenv.sh
> i docker_build_local_image --version 1.0.0 --container-dir-name $CSFY_RUNNABLE_DIR --multi-arch "linux/amd64,linux/arm64"
> i docker_tag_local_image_as_dev --version 1.0.0
> i docker_bash --skip-pull --version 1.0.0

8) Test the Code#

  • Run tests from the runnable dir (e.g. cmamp/ck.infra)

    # If the version of the locally built image is 1.0.0.
> cd $CSFY_RUNNABLE_DIR
> i run_fast_tests --version 1.0.0 --skip-pull
> i run_slow_tests --version 1.0.0 --skip-pull

  • Run tests from the root dir (e.g. cmamp) 

    > main_pytest.py run_fast_tests --dir ck.infra
> main_pytest.py run_slow_tests --dir ck.infra

9) Add the Dependency Lock Files to the Commit#

> cd $CSFY_RUNNABLE_DIR
> git add devops/docker_build/poetry.lock
> git add devops/docker_build/pip_list.txt
> git commit -m "Update dependencies"

10) Release the Docker Image#

  • Refer to the following docs for more info on image releases

  • To create new registory to store the image, please contact Infra team.

  • Release to ECR

    • This is required for running the container from the dev/prod servers

    > i docker_push_dev_image --version <version>
    - Or run the following command to build, test, and release the image

    > i docker_release_dev_image --version <version> --container-dir-name $CSFY_RUNNABLE_DIR

  • Release to GHCR

    • This is required for running the container from GH Actions CI/CD pipelines
    • TODO(heanh): Can we create an invoke target for this? 
      > docker login ghcr.io -u <username> -p <personal_access_token>
> docker tag 623860924167.dkr.ecr.eu-north-1.amazonaws.com/<image_name>:dev ghcr.io/causify-ai/<image_name>:dev
> docker push ghcr.io/causify-ai/<image_name>:dev

11) Update and Release a New Version of the Image#

We release a new version of the Docker image whenever we need to update its dependencies

  1. Modify changelog

  2. Specify what was changed

  3. Pick the release version according to semantic versioning convention
  4. For example for version 1.2.3:
    • 1 is major, 2 is minor, 3 is patch
> cd $CSFY_RUNNABLE_DIR
> source dev_scripts_${CSFY_RUNNABLE_DIR_SUFFIX}/thin_client/setenv.sh
> vim changelog.txt

# Cmamp-Infra-1.2.0
- 2025-05-18
- Add support for Kubernetes Kustomize
- Upgrade `kubectl` to v1.31.0
- Add `yq` command line tool

  1. Modify dependencies list

  2. Modify $CSFY_RUNNABLE_DIR/devops/docker_build/os_packages/os_packages.txt to add or remove OS packages using the declarative approach

  3. Modify $CSFY_RUNNABLE_DIR/devops/docker_build/pyproject.toml to add or remove Python packages

  4. Build the image locally

# Build the Image.
> i docker_build_local_image --version 1.2.0 --container-dir-name $CSFY_RUNNABLE_DIR

# Tag the Image as Dev.
> i docker_tag_local_image_as_dev --version 1.2.0
  1. Add the dependency lock files to the commit
> cd $CSFY_RUNNABLE_DIR
> git add devops/docker_build/poetry.lock
> git add devops/docker_build/pip_list.txt
> git commit -m "Update dependencies"
  1. Make sure we can run the container
> i docker_bash --skip-pull --stage local --version 1.2.0
> i docker_jupyter --skip-pull --stage local --version 1.2.0
  1. Make sure all tests pass
# Run Tests.
> i run_fast_tests --stage local --version 1.2.0
> i run_slow_tests --stage local --version 1.2.0
  1. Release the images to remote registries