Tips for Debugging

This page contains helpful commands and approaches for debugging.

Working with Docker and Docker Compose

The main deployment method for ASKCOS is using Docker Compose, which orchestrates a set of Docker containers on a single server. For many commands, there are similar versions available via both docker and docker compose command-line interfaces.

List Docker containers

1. This will list all running Docker containers on the machine. This command is the best way to obtain <container name> and <container id>, which are necessary for many other docker CLI commands. You can also include the -a option to also include stopped containers.

   docker ps [-a]

2. This will list running Docker containers which are part of the current Docker Compose deployment, based on docker-compose.yml. As such, you must be in the askcos-deploy directory when running this command. This is especially useful if there are other unrelated Docker containers running on the machine which you do not care about.

   docker-compose ps

Display logs

1. This method attaches to the container and redirects stdout and stderr of the application to the current tty. This method does not display the past logs but continuously prints future logs. This method also allows interactive debugging, see PDB section below.

   docker attach <container name>

2. This method displays all the past logs since the application starts. Adding -f option will continue streaming the logs. For more details, you can refer to the docs.

   docker logs [-f] <container name>

3. This method uses the docker-compose cli and is effectively the same as docker logs. Note that this command uses the service name instead of the container name, which for the django container is simply app instead of deploy_app_1.

   docker-compose logs [-f] <service name>

Inspect containers

1. The following command starts an interactive bash shell inside the docker container. You can examine the filesystem or execute programs using this command.

   docker exec -it <container name> /bin/bash

2. The docker-compose CLI provides a simpler equivalent to the above command. The -it arguments are unnecessary because they are enabled by default, and the service name is used instead of the container name.

   docker-compose exec <service name> /bin/bash

Note: The ASKCOS code directories are located in /usr/local inside the Docker image.

Working with Kubernetes

Basics

Fundamentally, Kubernetes (k8) relies on a very similar set of concepts as Docker Compose. Thus, it is generally possible to take a Docker Compose workflow and find the k8 equivalent.

Everything in k8 is a resource, defined by a yaml config file. You can get lists of various resource types using kubectl get, e.g.

kubectl get pods
kubectl get services

to get pods and services. Pods are wrappers for Docker containers, while services are a way to assign hostnames to certain pods to provide a static entry point.

For a full list of resource types and short names, see here.

Display logs

Logs can be displayed for a particular pod using its name, which is obtainable with kubectl get pods.

kubectl logs pod/<pod_name>

Obtain metadata

Metadata for any resource can be obtained with kubectl describe. This can be helpful to check on the status of pods or get details for pod failures.

kubectl describe pod/<pod_name>

Python built-in debugger: PDB

The built-in pdb module can be used to interactively debug a python program. Simply insert the following statement into where you want to break.

import pdb; pdb.set_trace()

# Or equivalently for python >= 3.7
breakpoint()

If you are debugging from inside a docker container, add the following to the docker-compose.yml in askcos-deploy repository. They are equivalent to -it command line argument.

    stdin_open: true
    tty: true

Here is an example.

  cr_network_v2_worker:
    stdin_open: true
    tty: true
    image: ${ASKCOS_IMAGE_REGISTRY}askcos-vue-nginx:${VERSION_NUMBER}
    restart: always
    env_file:
      - .env
    command: bash -c "celery -A askcos_site worker -c 1 -Q cr_network_v2_worker -n cr_network_v2_worker@%h --pool=gevent"
    volumes:
      - './custom_django_settings.py:/usr/local/askcos-site/askcos_site/custom_settings.py'

Then, rebuild and restart the ASKCOS deployment. Attach to the container as below, and trigger code execution.

docker attach <container name>

Invoke API endpoints

If a front-end UI is available, the API response can be obtained using developer tools in the browser.

Otherwise, curl can be used to make the API request. Below is an example.

# Make an API request
curl -vvvv -k https://localhost/api/v2/context-v2/ -X POST \
    -d 'reactants=CC=CC&products=CCCC&model=fp'

# Retrieve the results
curl -k https://localhost/api/v2/celery/task/{task_id}/

Meaning of the arguments:

• -k: Insecure SSL, no certificate verification for local deployment.
• -X POST: Post method. Get is the default.
• -d: Request body. By default, the format is application/x-www-form-urlencoded. If the body is sent using json, add -H 'Content-Type: application/json'.
• -vvvv: Show debug output.