Installation

This document describes how to prepare for and install the Toil software. Note that we recommend running all the Toil commands inside a Python virtualenv. Instructions for installing and creating a Python virtual environment are provided below.

Preparing your Python runtime environment

Toil currently supports only Python 2.7. If you don’t satisfy this requirement, consider using anaconda to create an alternate Python 2.7 installation.

Install Python virtualenv using pip.

$ sudo pip install virtualenv

Create a virtual environment called venv in your home directory.

$ virtualenv ~/venv

Or, if using an Apache Mesos cluster (see mesos in the Extras section below).

$ virtualenv ~/venv --system-site-packages

Activate your virtual environment.

$ source ~/venv/bin/activate

Basic Installation

If you need only the basic version of Toil, it can be easily installed using pip:

$ pip install toil

Now you’re ready to run your first Toil workflow!

(If you need any of the extra features don’t do this yet and instead skip to the next section.)

Installing Toil with extra features

Some optional features, called extras, are not included in the basic installation of Toil. To install Toil with all its bells and whistles, first install any necessary headers and libraries (python-dev, libffi-dev). Then run

$ pip install toil[aws,mesos,azure,google,encryption,cwl]

Here’s what each extra provides:

Extra Description
aws Provides support for managing a cluster on Amazon Web Service (AWS) using Toil’s built in Cluster Utilities. Clusters can scale up and down automatically. It also supports storing workflow state. This extra has no native dependencies.
google Experimental. Stores workflow state in Google Cloud Storage. This extra has no native dependencies.
azure Stores workflow state in Microsoft Azure. This extra has no native dependencies.
mesos

Provides support for running Toil on an Apache Mesos cluster. Note that running Toil on other batch systems does not require an extra. The mesos extra requires the following native dependencies:

Important

If you want to install Toil with the mesos extra in a virtualenv, be sure to create that virtualenv with the --system-site-packages flag:

$ virtualenv ~/venv --system-site-packages

Otherwise, you’ll see something like this:

ImportError: No module named mesos.native
encryption

Provides client-side encryption for files stored in the Azure and AWS job stores. This extra requires the following native dependencies:

cwl Provides support for running workflows written using the Common Workflow Language. This extra has no native dependencies.

Python headers and static libraries

Only needed for the mesos and encryption extras.

On Ubuntu:

$ sudo apt-get install build-essential python-dev

On macOS:

$ xcode-select --install

Encryption specific headers and library

Only needed for the encryption extra.

On Ubuntu:

$ sudo apt-get install libssl-dev libffi-dev

On macOS:

$ brew install libssl libffi

Or see Cryptography for other systems.

Preparing your AWS environment

To use Amazon Web Services (AWS) to run Toil or to just use S3 to host the files during the computation of a workflow, first set up and configure an account with AWS.

  1. If necessary, create and activate an AWS account
  2. Create a key pair in the availability zone of your choice (our examples use us-west-2a).
  3. Follow Amazon’s instructions to create an SSH key and import it into EC2.
  4. Finally, you will need to install and configure the AWS Command Line Interface (CLI).

Preparing your Azure environment

Follow the steps below to prepare your Azure environment for running a Toil workflow.

  1. Create an Azure account.
  2. Make sure you have an SSH RSA public key, usually stored in ~/.ssh/id_rsa.pub. If not, you can use ssh-keygen -t rsa to create one.

Building from source

If developing with Toil, you will need to build from source. This allows changes you make to Toil to be reflected immediately in your runtime environment.

First, clone the source:

$ git clone https://github.com/BD2KGenomics/toil
$ cd toil

Then, create and activate a virtualenv:

$ virtualenv venv
$ . venv/bin/activate

From there, you can list all available Make targets by running make. First and foremost, we want to install Toil’s build requirements. (These are additional packages that Toil needs to be tested and built but not to be run.)

$ make prepare

Now, we can install Toil in development mode (such that changes to the source code will immediately affect the virtualenv):

$ make develop

Or, to install with support for all optional Installing Toil with extra features:

$ make develop extras=[aws,mesos,azure,google,encryption,cwl]

To build the docs, run make develop with all extras followed by

$ make docs