Installation¶
This document describes how to prepare for and install the Toil software. Note that we recommend running all the Toil commands inside a Python virtualenv. Instructions for installing and creating a Python virtual environment are provided below.
Preparing your Python runtime environment¶
Toil currently supports only Python 2.7. If you don’t satisfy this requirement, consider using anaconda to create an alternate Python 2.7 installation.
Install Python virtualenv
using pip.
$ sudo pip install virtualenv
Create a virtual environment called venv
in your home directory.
$ virtualenv ~/venv
Or, if using an Apache Mesos cluster (see mesos
in the Extras section below).
$ virtualenv ~/venv --system-site-packages
Activate your virtual environment.
$ source ~/venv/bin/activate
Basic Installation¶
Toil can be easily installed using pip:
$ pip install toil
Now you’re ready to run your first Toil workflow!
Installing extra features¶
Some optional features, called extras, are not included in the basic installation of Toil. To install Toil with all its bells and whistles, first install any necessary headers and libraries (python-dev, libffi-dev). Then run
$ pip install toil[aws,mesos,azure,google,encryption,cwl]
Here’s what each extra provides:
Extra | Description |
---|---|
aws |
Provides support for managing a cluster on Amazon Web Service (AWS) using Toil’s built in Cluster Utilities. Clusters can scale up and down automatically. It also supports storing workflow state. This extra has no native dependencies. |
google |
Experimental. Stores workflow state in Google Cloud Storage. This extra has no native dependencies. |
azure |
Stores workflow state in Microsoft Azure. This extra has no native dependencies. |
mesos |
Provides support for running Toil on an Apache Mesos
cluster. Note that running Toil on other batch systems
does not require an extra. The
Important If you want to install Toil with the $ virtualenv ~/venv --system-site-packages
Otherwise, you’ll see something like this: ImportError: No module named mesos.native
|
encryption |
Provides client-side encryption for files stored in the Azure and AWS job stores. This extra requires the following native dependencies: |
cwl |
Provides support for running workflows written using the Common Workflow Language. This extra has no native dependencies. |
Python headers and static libraries
Only needed for the mesos
and encryption
extras.
On Ubuntu:
$ sudo apt-get install build-essential python-dev
On macOS:
$ xcode-select --install
Encryption specific headers and library
Only needed for the encryption
extra.
On Ubuntu:
$ sudo apt-get install libssl-dev libffi-dev
On macOS:
$ brew install libssl libffi
Or see Cryptography for other systems.
Preparing your AWS environment¶
To use Amazon Web Services (AWS) to run Toil or to just use S3 to host the files during the computation of a workflow, first set up and configure an account with AWS.
- If necessary, create and activate an AWS account
- Create a key pair in the availability zone of your choice (our examples use
us-west-2a
). - Follow Amazon’s instructions to create an SSH key and import it into EC2.
- Finally, you will need to install and configure the AWS Command Line Interface (CLI).
Preparing your Azure environment¶
Follow the steps below to prepare your Azure environment for running a Toil workflow.
- Create an Azure account.
- Make sure you have an SSH RSA public key, usually stored in
~/.ssh/id_rsa.pub
. If not, you can usessh-keygen -t rsa
to create one.
Building from source¶
If developing with Toil, you will need to build from source. This allows changes you make to Toil to be reflected immediately in your runtime environment.
First, clone the source:
$ git clone https://github.com/BD2KGenomics/toil
$ cd toil
Then, create and activate a virtualenv:
$ virtualenv venv
$ . venv/bin/activate
From there, you can list all available Make targets by running make
.
First and foremost, we want to install Toil’s build requirements. (These are
additional packages that Toil needs to be tested and built but not to be run.)
$ make prepare
Now, we can install Toil in development mode (such that changes to the source code will immediately affect the virtualenv):
$ make develop
Or, to install with support for all optional Installing extra features:
$ make develop extras=[aws,mesos,azure,google,encryption,cwl]
To build the docs, run make develop
with all extras followed by
$ make docs