Amazon Web Services¶
Toil includes a native AWS provisioner that can be used to start autoscaling clusters. For more information on this provisioner, see the Autoscaling section. To provision static, non-autoscaling clusters we recommend using CGCloud.
Setting up clusters with CGCloud has the benefit of coming pre-packaged with Toil and Mesos, our preferred batch system for running on AWS.
Create and activate a virtualenv:
$ virtualenv ~/cgcloud $ source ~/cgcloud/bin/activate
Install CGCloud and the CGCloud Toil plugin:
$ pip install cgcloud-toil
Add the following to your
~/.profile, using the appropriate region for your account:
export CGCLOUD_ZONE=us-west-2a export CGCLOUD_PLUGINS="cgcloud.toil:$CGCLOUD_PLUGINS"
Setup credentials for your AWS account in
[default] aws_access_key_id=PASTE_YOUR_FOO_ACCESS_KEY_ID_HERE aws_secret_access_key=PASTE_YOUR_FOO_SECRET_KEY_ID_HERE region=us-west-2
Register your SSH key. If you don’t have one, create it with
$ cgcloud register-key ~/.ssh/id_rsa.pub
Create a template toil-box which will contain necessary prerequisites:
$ cgcloud create -IT toil-box
Create a small leader/worker cluster:
$ cgcloud create-cluster toil -s 2 -t m3.large
SSH into the leader:
$ cgcloud ssh toil-leader
At this point, any Toil script can be run on the distributed AWS cluster by following instructions in Running on AWS.
Finally, if you wish to tear down the cluster and remove all its data permanently, CGCloud allows you to do so without logging into the AWS web interface:
$ cgcloud terminate-cluster toil
Toil comes with a cluster template to facilitate easy deployment of clusters running Toil on Microsoft Azure. The template allows these clusters to be created and managed through the Azure portal. To use the template to set up a Toil Mesos cluster on Azure, use the deploy button above, or open the deploy link in your browser.
Our group is working to expand distributed cluster support to OpenStack by providing convenient Docker containers to launch Mesos from. Currently, OpenStack nodes can be set up to run Toil in single machine mode by following the Installation.
Google Compute Engine¶
Support for running on Google Cloud is currently experimental. Our group is working to expand distributed cluster support to Google Compute with a cluster provisioning tool based around a Dockerized Mesos setup. Currently, Google Compute Engine nodes can be configured to run Toil in single machine mode by following the Installation.