2020.07 Release Notes

askcos-core

Developer notes:

• Renamed makeit module to askcos (MR askcos-core!2).
• Revert to default coordinate generator for RDKit drawing (MR askcos-core!5).
• Use askcos-base as base image for Docker build (MR askcos-core!7).

Bug fixes:

• Catch general selectivity error when target is not recovered (Issue ASKCOS#316; MR ASKCOS!346).
• Tree builder returns the incorrect supporting template (Issue ASKCOS#331; MR askcos-core!3).
• Fix template db lookup in tree builder to include template_set (MR askcos-core!3).
• Catch error when retro template cant be parsed by rdchiral when reversed (MR askcos-core!4).
• Do not use isomeric smiles for TFFP prediction (MR askcos-core!6).

askcos-site

User notes:

• Add api v2 drawing endpoint and new drawing page (MR ASKCOS!336).
• Ask for confirmation before clearing IPP results (Issue ASKCOS#308; MR ASKCOS!346).
• Refine user interface and upgrade to Bootstrap 4 (Issue ASKCOS#288; MR ASKCOS!341).
• Add option to completely disable third-party name resolution (Issue ASKCOS#305; MR ASKCOS!344).

Developer notes:

• Reduce initial website load time (MR ASKCOS!337).
• Optimize CI build time (Issue ASKCOS#304; MR ASKCOS!338).
• Add info about how to build image and use —local (MR ASKCOS!340).
• Use SUPPORT_EMAILS environment variable for support form (Issue ASKCOS#311; MR ASKCOS!348).
• Enable support for TensorFlow Serving model versions from TFServingAPIModel (Issue ASKCOS#323; MR askcos-site!3).
• Add modularity for new template relevance models to use different input sizes (Issue ASKCOS#310; MR askcos-site!7).
• Improve loading of cached IPP options to handle new options (MR askcos-site!4).
• Rename blacklist to banlist across askcos-site (MR askcos-site!10).
• Pass template prioritizer version to tree builder (MR askcos-site!11).
• Download new versions of browser cached static files on upgrades (Issue ASKCOS#306, MR askcos-site!12).

Bug fixes:

• Fix import issues with sites_worker (Issue ASKCOS#248; MR ASKCOS!337).
• Fix available worker calculation on Celery status page (Issue ASKCOS#237; MR askcos-site!13).
• Handle loss of connection error more gracefully during async polling in client (Issue ASKCOS#313; MR askcos-site!5).
• Fix issue preventing regioselectivity check from being disabled in IPP (Issue ASKCOS#317; MR ASKCOS!346).
• Remove hard-coded template set names in retro api (v2) endpoint (Issue ASKCOS#318; MR ASKCOS!346).
• Properly pass cluster settings in tb_c_worker.get_top_precursors (Issue ASKCOS#319; MR ASKCOS!346).
• Prevent logout/login loops (Issue ASKCOS#322; MR ASKCOS!349).
• Re-enable multi-select in IPP to remove multiple nodes (Issue ASKCOS#329; MR askcos-site!4).
• Fix backup issue by reverting to drf-jwt 1.14 (Issue askcos-deploy#11; MR ASKCOS!339).
• Remove custom css for browsable api due to Bootstrap version incompatibility (MR ASKCOS!345).

askcos-deploy

User notes:

• Add new docker volume based backup and restore commands to deploy.sh (MR askcos-deploy!27).
• Add option to completely disable third-party name resolution (Issue ASKCOS#305; MR askcos-deploy!29).
• Compare user customizable environment files when deploying (MR askcos-deploy!31).
• Expose SUPPORT_EMAILS as an environment variable on deployment (Issue ASKCOS#311; MR askcos-deploy!33).
• Add functionality in health_check.py to fix tree builder timeout (MR askcos-deploy!35).

Developer notes:

• Unpin sites_worker service from 0.4.0 (Issue ASKCOS#248; MR askcos-deploy!26).
• Updates to k8 configuration (Issue askcos-deploy#9; MR askcos-deploy!28).
• Change mongo db index types for better performance (Issue askcos-deploy#7; MR askcos-deploy!30, askcos-deploy!32).
• Update deployment for repository split (MR askcos-deploy!34).

Bug fixes:

• Add pull-images step to deploy-http command (MR askcos-deploy!25).
• Use COMPOSE_PROJECT_NAME from .env in backup.sh and restore.sh (Issue askcos-deploy#10; MR askcos-deploy!27).

Software Prerequisites

To deploy ASKCOS, you must have the following installed on your machine:

• git
• Docker (installation instructions)
• Docker Compose (installation instructions)

Quickstart

ASKCOS can be downloaded using deploy tokens, which provide read-only access to the source code and our container registry in GitLab. Below is a complete example showing how to deploy the ASKCOS application using deploy tokens (omitted in this example). The deploy tokens can be found on the MLPDS Member Resources ASKCOS Versions Page.

$ export DEPLOY_TOKEN_USERNAME=
$ export DEPLOY_TOKEN_PASSWORD=
$ docker login registry.gitlab.com -u $DEPLOY_TOKEN_USERNAME -p $DEPLOY_TOKEN_PASSWORD
$ git clone https://$DEPLOY_TOKEN_USERNAME:$DEPLOY_TOKEN_PASSWORD@gitlab.com/mlpds_mit/askcos/askcos-deploy.git
$ cd askcos-deploy
$ git checkout 2020.07
$ bash deploy.sh deploy

Upgrade Information

The easiest way to upgrade to a new version of ASKCOS is using Docker and Docker Compose. To get started, make sure both Docker and Docker Compose are installed on your machine. We have a pre-built docker image of ASKCOS hosted on GitLab. It is a private repository; if you do not have access to pull the image, please contact us. Start with the askcos-deploy repository. The process for cloning the repository and checking out the correct version tag is described above.

From v0.3.1 or above

$ git checkout 2020.07
$ bash deploy.sh update -v 2020.07

If you have not seeded the database before (if you're upgrading from v0.3.1), you will need to do so:

$ bash deploy.sh set-db-defaults seed-db

When upgrading to 2020.07, we highly recommend re-indexing the database because the index types have been updated to significantly improve lookup efficiency:

$ bash deploy.sh index-db --drop-indexes

From v0.2.x or v0.3.0

$ git checkout 2020.07
$ bash backup.sh
$ bash deploy.sh update -v 2020.07
$ bash deploy.sh set-db-defaults seed-db
$ bash restore.sh

NOTE: A large amount of data has been migrated to the mongo db starting in v0.4.1 (chemhistorian), and seeding may take some time to complete. We send this seeding task to the background so the rest of the application can start and become functional without having to wait. If using the default set of data (i.e. - using the exact commands above), you can monitor the progress of mongo db seeding using bash deploy.sh count-mongo-docs, which will tell you how many documents have been seeded out of the expected number. Complete seeding is not necessary for application functionality unless you use the chemical popularity logic in the tree builder.

First Time Deployment

Deploying the Web Application

Deployment is initiated by a bash script that runs a few docker-compose commands in a specific order. Several database services need to be started first, and more importantly seeded with data, before other services (which rely on the availability of data in the database) can start. The deploy.sh script is provided in the askcos-deploy repository and should be run as follows:

$ bash deploy.sh command [optional arguments]

There are a number of available commands, including the following for common deploy tasks:

• deploy: runs standard first-time deployment tasks, including seed-db
• update: pulls new docker image from GitLab repository and restarts all services
• seed-db: seed the database with default or custom data files
• start: start a deployment without performing first-time tasks
• stop: stop a running deployment
• clean: stop a running deployment and remove all docker containers

For a running deployment, new data can be seeded into the database using the seed-db command along with arguments indicating the types of data to be seeded. Note that this will replace the existing data in the database. The available arguments are as follows:

• -b, --buyables: specify buyables data to seed, either default or path to data file
• -c, --chemicals: specify chemicals data to seed, either default or path to data file
• -x, --reactions: specify reactions data to seed, either default or path to data file
• -r, --retro-templates: specify retrosynthetic templates to seed, either default or path to data file
• -f, --forward-templates: specify forward templates to seed, either default or path to data file

For example, to seed default buyables data and custom retrosynthetic pathways, run the following from the deploy folder:

$ bash deploy.sh seed-db --buyables default --retro-templates /path/to/my.retro.templates.json.gz

To update a deployment, run the following from the deploy folder:

$ bash deploy.sh update --version x.y.z

To stop a currently running application, run the following from the deploy folder:

$ bash deploy.sh stop

If you would like to clean up and remove everything from a previous deployment (NOTE: you will lose user data), run the following from the deploy folder:

$ bash deploy.sh clean

Backing Up User Data

From v0.3.1 or above

If you are upgrading from v0.3.1 or later, the backup/restore process is no longer needed unless you are moving deployments to a new machine.

New backup and restore functions were added in askcos-deploy 2020.07 to provide more robust backup/restore capabilities based on Docker volumes. The commands can be used whether the site is running or not; the only requirement is that the mongo_data and mysql_data Docker volumes exist.

To backup:

bash deploy.sh backup [-d /absolute/path/to/backup/dir]

To restore:

bash deploy.sh restore [-d /absolute/path/to/backup/dir]

From v0.2.x or v0.3.0

If you are upgrading the deployment from a previous version (prior to v0.3.1), or moving the application to a different server, you may want to retain user accounts and user-saved data/results. The provided backup.sh and restore.sh scripts in the askcos-deploy/utils/legacy directory are capable of handling the backup and restoring process. Please read the following carefully so as to not lose any user data:

1. Start by making sure the previous version you would like to backup is currently up and running with docker-compose ps.
2. Checkout the newest version of the askcos-deploy: git checkout 2020.07
3. Run $ bash utils/legacy/backup.sh
4. Make sure that the deploy/backup folder is present, and there is a folder with a long string of numbers (year+month+date+time) that corresponds to the time you just ran the backup command
5. If the backup was successful (db.json and user_saves (<v0.3.1) or results.mongo (>=0.3.1) should be present), you can safely tear down the old application with docker-compose down [-v]
6. Deploy the new application with bash deploy.sh deploy or update with bash deploy.sh update -v x.y.z
7. Restore user data with bash utils/legacy/restore.sh

Note: For versions >=0.3.1, user data persists in docker volumes and is not tied to the lifecycle of the container services. If the [-v] flag is not used with docker-compose down, volumes do not get removed, and user data is safe. In this case, the backup/restore procedure is not necessary as the containers that get created upon an install/upgrade will continue to use the docker volumes that contain all the important data. If the [-v] flag is used, all data will be removed and a restore will be required to recover user data.

Add Customization

There are a few parts of the application that you can customize:

• Header sub-title next to ASKCOS (to designate this as a local deployment at your organization)

This is handled as an environment variable that can change upon deployment (and are therefore not tied into the image directly). This can be found in deploy/customization. Please let us know what other degrees of customization you would like.

Managing Django

If you'd like to manage the Django app (i.e. - run python manage.py ...), for example, to create an admin superuser, you can run commands in the running app service (do this after docker-compose up) as follows:

$ docker-compose exec app bash -c "python /usr/local/ASKCOS/askcos/manage.py createsuperuser"

In this case you'll be presented an interactive prompt to create a superuser with your desired credentials.

Important Notes

Scaling Workers

Only 1 worker per queue is deployed by default with limited concurrency. This is not ideal for many-user demand. The scaling of each worker is defined at the top of the deploy.sh script. To scale a desired worker, change the appropriate value in deploy.sh, for example:

n_tb_c_worker=N          # Tree builder chiral worker

where N is the number of workers you want. Then run bash deploy.sh start [-v <version>].

(Optional) Building Docker Images

If you would like to build the askcos-site Docker image yourself, you will need to download the appropriate repositories depending on where you want to start.

To only build askcos-site using a pre-built askcos-core image:

$ git clone https://gitlab.com/mlpds_mit/askcos/askcos-site
$ cd askcos-site
$ make [TAG=my_tag]

A Makefile is provided to make it easier to build the image with a default image name. You can also use the docker build command directly:

$ docker build -t <image name>:<tag> .

NOTE: The image name should correspond with what exists in the docker-compose.yml file. By default, the image name is environment variable ASKCOS_IMAGE_REGISTRY + askcos-site. If you choose to use a custom image name, make sure to modify the ASKCOS_IMAGE_REGISTRY variable or the docker-compose.yml file accordingly.

Similarly, if you also want to build askcos-core:

$ git clone https://gitlab.com/mlpds_mit/askcos/askcos-core
$ cd askcos-core
$ make [TAG=my_tag]

Note that you will need to specify the appropriate askcos-core version when building askcos-site afterwards:

$ cd askcos-core
$ make TAG=my_tag
$ cd ../askcos-site
$ make CORE_VERSION=my_tag TAG=my_tag

How To Run Individual Modules

Many of the individual modules -- at least the ones that are the most interesting -- can be run "standalone". Examples of how to use them are often found in the if __name__ == '__main__' statement at the bottom of the script definitions. For example...

Using the learned synthetic complexity metric (SCScore):

makeit/prioritization/precursors/scscore.py

Obtaining a single-step retrosynthetic suggestion with consideration of chirality:

makeit/retrosynthetic/transformer.py

Finding recommended reaction conditions based on a trained neural network model:

makeit/synthetic/context/neuralnetwork.py

Using the template-free forward predictor:

makeit/synthetic/evaluation/template_free.py

Using the coarse "fast filter" (binary classifier) for evaluating reaction plausibility:

makeit/synthetic/evaluation/fast_filter.py

ASKCOS Development

Software package for the prediction of feasible synthetic routes towards a desired compound and associated tasks related to synthesis planning. Originally developed under the DARPA Make-It program and now being developed under the MLPDS Consortium.