ASKCOS

Appearance

2022.04 Release Notes ​

askcos-site ​

User notes:

  • Unify tree builder results view with IPP (MR askcos-site!122)
    • Tree builder results are presented in a new "Tree Explorer" tab in the IPP interface (Issue askcos-site#13)
    • More flexible options for adding tree builder results to the IPP network visualization
    • Expanded tree enumeration functionality for IPP networks (Issue askcos-site#15)
    • Ability to redo pathway search from tree builder graph using new terminal chemical criteria (Issue askcos-site#47)
    • Various user interface improvements and adjustments, including IPP selection details panel, IPP toolbar, and result info modal
  • Add new solvent screening tool (MR askcos-site!124)
    • Includes functionality to plot solubility vs. temperature (Issue askcos-site#77)
    • Includes conversion of logS to mg/mL (Issue askcos-site#78)
  • UI improvements and new features in IPP and tree explorer (MR askcos-site!125)
    • Sort trees by multiple fields in tree explorer
    • Create ChemHistorian API endpoint
    • Create SCScore batch API endpoint
    • Look up ChemHistorian data and SCScores for chemicals from IPP predictions
    • Update and standardize network visualization features across IPP and tree explorer including improvements for colorblind accessibility (Issue askcos-site#61)
    • Calculate molecular weight and overall atom economy for tree builder paths when loading results in backend
    • Add overlay displaying pathway statistics to tree explorer and update overlay styling in IPP
  • Additional IPP features and other fixes (MR askcos-site!126)
    • Add feature to refresh chemical metadata for IPP results
    • Add support for banned chemicals and reactions to /retro/ endpoint
    • Add ban button for adding chemicals and reactions to banlist (Issue askcos-site#79)
    • Add legend to explain border colors and annotation icons
    • Add note to forward predictor page about new model
    • Update interactive IPP tour and tutorial page
    • Set export filename based on upload filename in solubility prediction page (Issue askcos-site#83)

Developer notes:

  • Code cleanup and addition of lint and test CI jobs (MR askcos-site!120)
    • Automatic formatting using black
    • Linting using flake8
    • Remove cr_coordinator and cr_nn_worker
    • Remove single_solvent parameter for v1 context recommender API (did not have an effect)
    • Refactor globals module to delay instantiation/loading until first use
    • Change base classes for some API endpoints to APIView when GenericAPIView features aren't used
  • Add pre-commit config file (askcos-site!121)
  • Unify tree builder results view with IPP (MR askcos-site!122)
    • Consolidation of some tree builder result processing functionality in new askcos_site.utils.tree_results module
    • Tree builder results returned by the /results/<id>/ipp/ sub-endpoint no longer includes dispGraph, and paths no longer include metadata (This does not change the tree builder saved result format which can be retrieved using the main /results/<id>/ endpoint)
    • Graph optimization results returned by the /results/<id>/ipp/ sub-endpoint now include the dataGraph from the source tree builder job
    • Additional reaction metadata is now calculated for C++ tree builder results
    • New result info modal component created for unified view in IPP and Tree Explorer

Bug fixes:

  • Use global database connections in tb_coordinator_mcts (Issue askcos-site#72, MR askcos-site!119)
  • Fix issues with node/edge ID tracking in IPP trees (MR askcos-site!123)
  • Fix solubility prediction error if input fields are cleared (Issue askcos-site#80, MR askcos-site!124)
  • Reset atom filter selection after deselecting chemical node in IPP (Issue askcos-site#81, MR askcos-site!125)

askcos-core ​

Developer notes:

  • Code cleanup and addition of lint, test, pages CI jobs (MR askcos-core!81)
    • Automatic formatting using black
    • Linting using flake8
    • Removed of evaluation.rexgen_direct module
    • Removed of context.nearestneighbor module
    • Removed of utilities.model_loader module
    • Transition context.neuralnetwork to Tensorflow 2
    • Fix sphinx documentation build
    • Transition to pytest and additional fixes and improvements to tests
  • Add pre-commit config file (MR askcos-core!82)
  • Modularize ChemHistorian and other changes to support IPP (MR askcos-core!83)
    • Split ChemHistorian class into BaseChemHistorian and child classes FileChemHistorian and MongoChemHistorian
    • Create a molecular_weight function for getting the molecular weight of a single molecule
  • Add support for banned chemicals/reactions in RetroTransformer.get_outcomes (MR askcos-core!84)
    • Create canonicalization utility function
    • Add banned_chemicals and banned_reactions options to RetroTransformer.get_outcomes
    • Canonicalize banned_chemicals and banned_reactions in tree builders

Bug fixes:

  • Fix empty strings in output from utilities.contexts.clean_context (MR askcos-core!81)
  • Fix scscore calculation for tree builder reactions (MR askcos-core!83)
  • Fix a bug when converting the graph from C++ treebuilder to networkx (MR askcos-core!85)

askcos-data ​

  • Add new WLN forward model trained on Pistachio 2021Q3 (Issue askcos-core#17, Epic&3, MR askcos-data!36)
  • Re-canonicalize database files using RDKit 2020.09.05 (MR askcos-data!37)

askcos-deploy ​

  • Fix cpu_count determination in monitor.py (MR askcos-deploy!95)
  • Add new Pistachio forward model to model config (MR askcos-deploy!96)
  • Add support for exporting collections from MongoDB (MR askcos-deploy!97)
  • Add utility script for canonicalizing SMILES in ASKCOS data files (MR askcos-deploy!97)
  • Pull tfx-askcos image when updating deployment (MR askcos-deploy!97)

Docker Compose Deployment ​

We currently support two methods for deploying ASKCOS: Docker Compose and Kubernetes. Docker Compose is a simpler method for deploying on a single workstation, while Kubernetes is more complex but is suitable for scaling across multiple nodes.

Hardware Requirements ​

To deploy ASKCOS with the default number of workers, we recommend using a server with at least 16 CPU cores and 64 GB memory. The default configuration uses approximately 45 GB memory at deploy, but usage will increase while running some compute tasks. ASKCOS is not currently set up to use GPUs for machine learning predictions.

For deployment on AWS, this corresponds to an m5.4xlarge instance or similar. (Note that ASKCOS does not work on ARM-based instances.)

For deployment on Google Cloud, this corresponds to an e2-standard-16 instance or similar.

If you plan to increase worker scales, you should increase hardware resources accordingly.

Finally, you should provision at least 80 GB of drive space for a basic deployment. More disk space is recommended for long-term deployments to store user data and support updates and custom models and data.

Software Prerequisites ​

To deploy ASKCOS using Docker Compose, you must have the following installed on your machine:

Quickstart ​

ASKCOS can be downloaded using deploy tokens, which provide read-only access to the source code and our container registry in GitLab. Below is a complete example showing how to deploy the ASKCOS application using deploy tokens (omitted in this example). The deploy tokens can be found on the MLPDS Member Resources ASKCOS Versions Page.

bash
$ export DEPLOY_TOKEN_USERNAME=
$ export DEPLOY_TOKEN_PASSWORD=
$ docker login registry.gitlab.com -u $DEPLOY_TOKEN_USERNAME -p $DEPLOY_TOKEN_PASSWORD
$ git clone https://$DEPLOY_TOKEN_USERNAME:$DEPLOY_TOKEN_PASSWORD@gitlab.com/mlpds_mit/askcos/askcos-deploy.git
$ cd askcos-deploy
$ git checkout 2022.04
$ bash deploy.sh deploy

Upgrade Information ​

The askcos-deploy repository also provides scripts to upgrade an existing ASKCOS deployment in-place.

bash
$ cd askcos-deploy
$ git fetch origin
$ git checkout 2022.04
$ bash deploy.sh update -v 2022.04

Some releases include changes or additions which require further action. Depending on the version you are upgrading from, you may need to perform one or more of the following steps.

New in 2022.04 ​

The 2022.04 release includes a new WLN forward prediction model trained on Pistachio 2021Q3. This model should be deployed automatically if you go through the update process.

In addition, we identified some changes to SMILES canonicalization resulting from updates to the RDKit version used by ASKCOS in previous releases. The changes affect a very small fraction of documents in the ASKCOS buyables and chemical historian databases. More information on the effects and fixes can be found in the Note on SMILES canonicalization in the release highlights.

Notes from earlier releases ​

The 2022.01 release includes reaction precedent data for the CAS template relevance model. This data is necessary to use the new CAS SciFindern integration. The data can be imported into MongoDB using the following command:

bash
$ bash deploy.sh seed-db -x cas

!> Note: For on-premise deployments, you will need to register redirect URL's with CAS before the SciFinder integration will work properly. For example, the redirect URL registered for the demo site is https://askcos-demo.mit.edu/*. Please contact us or your CAS representative directly to begin a request.

Another major new feature in 2022.01 is the experimental C++ tree builder, which uses a reinforcement learning model and is highly optimized for computational speed. Due to high resource requirements, the C++ tree builder is not deployed by default. For more details and deployment info, please see the C++ tree builder deployment documentation.

The 2021.10 release includes two new template relevance models: one trained on enzymatic reactions from BKMS, and one trained on ring-breaking reactions from Pistachio. In order to use the new models, the associated data will need to be imported into MongoDB:

bash
$ bash deploy.sh seed-db -r bkms -c bkms -x bkms
$ bash deploy.sh seed-db -r ringbreaker

Note that the deploy.sh script has been updated to change the default behavior for seed-db to append new data. To drop all existing data (the previous behavior), you can pass the --drop argument.

The 2021.07 release includes a new template relevance model trained on a SciFinder/CAS template set. In order to use the new model, you will need to import the reaction templates into MongoDB:

bash
$ bash deploy.sh seed-db -r cas --append

!>Please note that chemical historian data is not included for the CAS model at this time, so chemical popularity information will not be available for tree builder jobs using the CAS model.

The 2021.07 also introduces a model serving configuration file, located at askcos-deploy/model_config.yaml. Models deployed using Tensorflow Serving or Torchserve must be added to the configuration file to provide connection parameters to ASKCOS. Any existing custom models should be added to ensure they work after updating.

First Time Deployment ​

Deploying the Web Application ​

Deployment is initiated by a bash script that runs a few docker-compose commands in a specific order. Several database services need to be started first, and more importantly seeded with data, before other services (which rely on the availability of data in the database) can start. The deploy.sh script is provided in the askcos-deploy repository and should be run as follows:

bash
$ bash deploy.sh command [optional arguments]

For a full list of available commands and options, use the help command.

There are a number of available commands for common deploy tasks:

  • deploy: runs standard first-time deployment tasks, including seed-db
  • update: pulls new docker image from GitLab repository and restarts all services
  • seed-db: seed the database with default or custom data files
  • start: start a deployment without performing first-time tasks
  • stop: stop a running deployment
  • clean: stop a running deployment and remove all docker containers and volumes

For a running deployment, new data can be seeded into the database using the seed-db command along with arguments indicating the types of data to be seeded. Note that this will replace the existing data in the database. The available arguments are as follows:

  • -b, --buyables: specify buyables data to seed, either default or path to data file
  • -c, --chemicals: specify chemicals data to seed, either default or path to data file
  • -x, --reactions: specify reactions data to seed, either default or path to data file
  • -r, --retro-templates: specify retrosynthetic templates to seed, either default or path to data file
  • -f, --forward-templates: specify forward templates to seed, either default or path to data file
  • -e, --references: specify model reference data to seed, only supports default currently

For example, to seed default buyables data and custom retrosynthetic pathways, run the following from the deploy folder:

bash
$ bash deploy.sh seed-db --buyables default --retro-templates /path/to/my.retro.templates.json.gz

To update a deployment, run the following from the deploy folder:

bash
$ bash deploy.sh update --version x.y.z

To stop a currently running application, run the following from the deploy folder:

bash
$ bash deploy.sh stop

If you would like to clean up and remove everything from a previous deployment (NOTE: you will lose user data), run the following from the deploy folder:

bash
$ bash deploy.sh clean

Backing Up User Data ​

If you are upgrading from v0.3.1 or later, the backup/restore process is no longer needed unless you are moving deployments to a new machine.

New backup and restore functions were added in askcos-deploy 2020.07 to provide more robust backup/restore capabilities based on Docker volumes. The commands can be used whether the site is running or not; the only requirement is that the mongo_data and mysql_data Docker volumes exist.

To backup:

bash
bash deploy.sh backup [-d /absolute/path/to/backup/dir]

To restore:

bash
bash deploy.sh restore [-d /absolute/path/to/backup/dir]

!>Note: These backup and restore processes are run in a bare alpine linux image which will be automatically pulled by Docker.

Add Customization ​

There are a few parts of the application that you can customize:

  • Header sub-title next to ASKCOS (to designate this as a local deployment at your organization)
  • Email addresses for the support form
  • Whether to enable the chemical name to SMILES resolver
  • Whether authorization is required to modify the buyables database
  • Add internal URL to a Pistachio web app deployment to enable direct links

These are handled as an environment variables that can change upon deployment (and are therefore not tied into the image directly). This can be found in the customization file, which is created automatically during deployment from the customization.example file.

In addition, the following methods enable more substantial customizations to the ASKCOS website without rebuilding the askcos-site image:

  • Customization of Django site settings
    • Include customizations in the askcos-deploy/custom_django_settings.py file which is mounted to /usr/local/askcos-site/askcos_site/custom_settings.py in the app container
  • Customization of web frontend
    • Include custom script or css tags in a custom_head.html Django template file which is mounted to /usr/local/askcos-site/askcos_site/templates/custom_head.html and included in the <head> section of every page

Please let us know what other degrees of customization you would like.

Managing Django ​

If you'd like to manage the Django app (i.e. - run python manage.py ...), for example, to create an admin superuser, you can run commands in the running app service as follows:

bash
$ docker-compose exec app bash -c "python /usr/local/askcos-site/manage.py createsuperuser"

In this case you'll be presented an interactive prompt to create a superuser with your desired credentials.

Scaling Workers ​

Only 1 worker per queue is deployed by default with limited concurrency. This is not ideal for many-user demand. The scaling of each worker is defined at the top of the deploy.sh script. To scale a desired worker, change the appropriate value in deploy.sh, for example:

n_tb_c_worker=N          # Tree builder chiral worker

where N is the number of workers you want. Then run bash deploy.sh start [-v <version>].

Kubernetes Deployment ​

ASKCOS 2022.04 includes a Helm chart to make it easier to deploy ASKCOS on Kubernetes. The previous Kubernetes configuration can still be used for 2020.07 or earlier but will no longer be updated.

Hardware Requirements ​

To deploy ASKCOS with the default number of workers, we recommend using a server with at least 16 CPU cores and 64 GB memory combined across nodes, and individual nodes with at least 16 GB memory. The default configuration uses approximately 45 GB memory total at deploy, with the most resource intensive worker needing about 14 GB, but usage will increase while running some compute tasks. ASKCOS is not currently set up to use GPUs for machine learning predictions.

For deployment on AWS, this corresponds to one m5.4xlarge instance or two m5.2xlarge instances. (Note that ASKCOS does not work on ARM-based instances.)

For deployment on Google Cloud, this corresponds to an e2-standard-16 instance or two e2-standard-8 instances.

If you plan to increase worker scales, you should increase hardware resources accordingly.

Software Prerequisites ​

In addition to git and Docker, we will assume that you are using a cluster which already has Kubernetes configured. You will also need to install Helm 3: https://helm.sh/docs/intro/install/.

Quickstart ​

Similar to the Docker Compose deployment, you will need to obtain the ASKCOS deploy tokens in order to clone the askcos-deploy repository and access the GitLab image registry. The deploy tokens can be found on the MLPDS Member Resources ASKCOS Versions Page.

bash
$ export DEPLOY_TOKEN_USERNAME=
$ export DEPLOY_TOKEN_PASSWORD=
$ git clone https://$DEPLOY_TOKEN_USERNAME:$DEPLOY_TOKEN_PASSWORD@gitlab.com/mlpds_mit/askcos/askcos-deploy.git
$ cd askcos-deploy
$ git checkout 2022.04
$ helm install --set imageCredentials.username=$DEPLOY_TOKEN_USERNAME --set imageCredentials.password=$$DEPLOY_TOKEN_PASSWORD mydeploy ./helm/askcos

For more configuration options, please check out the values file at askcos-deploy/helm/askcos/values.yaml.

Add Customization ​

For Kubernetes, the same customizations can be applied as for the Docker Compose deployment:

  • Header sub-title next to ASKCOS (to designate this as a local deployment at your organization)
  • Email addresses for the support form
  • Whether to enable the chemical name to SMILES resolver
  • Whether authorization is required to modify the buyables database.
  • Add internal URL to a Pistachio web app deployment to enable direct links

The environment variables for these customizations can be adjusted in the env block of the values.yaml file.

Managing Django ​

If you'd like to manage the Django app (i.e. - run python manage.py ...), for example, to create an admin superuser, you can run commands in the running app container as follows:

bash
$ kubectl exec [ASKCOS POD] -c app -i -t -- python /usr/local/askcos-site/manage.py createsuperuser

In this case you'll be presented an interactive prompt to create a superuser with your desired credentials.

Scaling Workers ​

For Kubernetes, worker replicas can also be set in the values.yaml file. Celery workers are defined in the celery block as a list, and each item has a replicaCount field for for setting the number of replicas.

(Optional) Building Docker Images ​

If you would like to build the askcos-site Docker image yourself, you will need to download the appropriate repositories depending on where you want to start.

To only build askcos-site using a pre-built askcos-core image:

bash
$ git clone https://gitlab.com/mlpds_mit/askcos/askcos-site
$ cd askcos-site
$ make [TAG=my_tag]

A Makefile is provided to make it easier to build the image with a default image name. You can also use the docker build command directly:

bash
$ docker build -t <image name>:<tag> .

!>Note: The image name should correspond with what exists in the docker-compose.yml file. By default, the image name is environment variable ASKCOS_IMAGE_REGISTRY + askcos-site. If you choose to use a custom image name, make sure to modify the ASKCOS_IMAGE_REGISTRY variable or the docker-compose.yml file accordingly. For Kubernetes deployment, the image registry and tag are defined in the values.yaml file.

Similarly, if you also want to build askcos-core:

bash
$ git clone https://gitlab.com/mlpds_mit/askcos/askcos-core
$ cd askcos-core
$ make [TAG=my_tag]

Note that you will need to specify the appropriate askcos-core version when building askcos-site afterwards:

bash
$ cd askcos-core
$ make TAG=my_tag
$ cd ../askcos-site
$ make CORE_VERSION=my_tag TAG=my_tag

ASKCOS Development ​

Software package for the prediction of feasible synthetic routes towards a desired compound and associated tasks related to synthesis planning. Originally developed under the DARPA Make-It program and now being developed under the MLPDS Consortium.

Released under the MIT License.

Copyright © 2024 MIT MLPDS