Running in the Cloud

Toil supports Amazon Web Services (AWS) and Google Compute Engine (GCE) in the cloud and has autoscaling capabilities that can adapt to the size of your workflow, whether your workflow requires 10 instances or 20,000.

Toil does this by creating a virtual cluster with Apache Mesos. Apache Mesos requires a leader node to coordinate the workflow, and worker nodes to execute the various tasks within the workflow. As the workflow runs, Toil will “autoscale”, creating and terminating workers as needed to meet the demands of the workflow.

Once a user is familiar with the basics of running toil locally (specifying a jobStore, and how to write a toil script), they can move on to the guides below to learn how to translate these workflows into cloud ready workflows.

Managing a Cluster of Virtual Machines (Provisioning)

Toil can launch and manage a cluster of virtual machines to run using the provisioner to run a workflow distributed over several nodes. The provisioner also has the ability to automatically scale up or down the size of the cluster to handle dynamic changes in computational demand (autoscaling). Currently we have working provisioners with AWS and GCE (Azure support has been deprecated).

Toil uses Apache Mesos as the Batch System.

See here for instructions for Running in AWS.

See here for instructions for Running in Google Compute Engine (GCE).

Storage (Toil jobStore)

Toil can make use of cloud storage such as AWS or Google buckets to take care of storage needs.

This is useful when running Toil in single machine mode on any cloud platform since it allows you to make use of their integrated storage systems.

For an overview of the job store see Job Store.

For instructions configuring a particular job store see:

• AWS Job Store
• Google Job Store

Cloud Platforms

• Running on Kubernetes
  • Preparing your Kubernetes environment
  • AWS Job Store for Kubernetes
  • Configuring Toil for your Kubernetes environment
  • Running workflows
    • Option 1: Running the Leader Inside Kubernetes
      • Monitoring and Debugging Kubernetes Jobs and Pods
      • When Things Go Wrong
    • Option 2: Running the Leader Outside Kubernetes
• Running in AWS
  • Preparing your AWS environment
  • AWS Job Store
  • Toil Provisioner
  • Details about Launching a Cluster in AWS
    • Static Provisioning
    • Uploading Workflows
    • Running a Workflow with Autoscaling
    • Preemptability
  • Dashboard
• Running in Google Compute Engine (GCE)
  • Preparing your Google environment
  • Google Job Store
  • Running a Workflow with Autoscaling
• Cluster Utilities
• Stats Command
• Status Command
• Clean Command
• Launch-Cluster Command
• Ssh-Cluster Command
• Rsync-Cluster Command
• Destroy-Cluster Command
• Kill Command