This document describes how to prepare for and install Toil. Note that Toil requires that the user run all commands inside of a Python virtualenv. Instructions for installing and creating a Python virtual environment are provided below.
Preparing Your Python Runtime Environment¶
Toil currently supports Python 3.7, 3.8, 3.9, and 3.10, and requires a virtualenv to be active to install.
If not already present, please install the latest Python
virtualenv using pip:
$ sudo pip install virtualenv
And create a virtual environment called
venv in your home directory:
$ virtualenv ~/venv
If the user does not have root privileges, there are a few more steps, but one can download a specific virtualenv package directly, untar the file, create, and source the virtualenv (version 15.1.0 as an example) using
$ curl -O https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz $ tar xvfz virtualenv-15.1.0.tar.gz $ cd virtualenv-15.1.0 $ python virtualenv.py ~/venv
Now that you’ve created your virtualenv, activate your virtual environment:
$ source ~/venv/bin/activate
If you need only the basic version of Toil, it can be easily installed using pip:
$ pip install toil
Now you’re ready to run your first Toil workflow!
(If you need any of the extra features don’t do this yet and instead skip to the next section.)
Installing Toil with Extra Features¶
Python headers and static libraries
Needed for the
$ sudo apt-get install build-essential python-dev
$ xcode-select --install
Encryption specific headers and library
Needed for the
$ sudo apt-get install libssl-dev libffi-dev
$ brew install libssl libffi
Or see Cryptography for other systems.
Some optional features, called extras, are not included in the basic installation of Toil. To install Toil with all its bells and whistles, first install any necessary headers and libraries (python-dev, libffi-dev). Then run
$ pip install toil[aws,google,mesos,encryption,cwl,wdl,kubernetes,server]
$ pip install toil[all]
Here’s what each extra provides:
Installs all extras (though htcondor is linux-only and will be skipped if not on a linux computer).
Provides support for managing a cluster on Amazon Web Service (AWS) using Toil’s built in Cluster Utilities. Clusters can scale up and down automatically. It also supports storing workflow state.
Experimental. Stores workflow state in Google Cloud Storage.
Provides support for running Toil on an Apache Mesos
cluster. Note that running Toil on other batch systems
does not require an extra. The
If launching toil remotely on a mesos instance,
to install Toil with the
$ virtualenv ~/venv --system-site-packages
Otherwise, you’ll see something like this:
ImportError: No module named mesos.native
Support for the htcondor batch system. This currently is a linux only extra.
Provides client-side encryption for files stored in the AWS job store. This extra requires the following native dependencies:
Provides support for running workflows written using the Common Workflow Language.
Provides support for running workflows written using the Workflow Description Language. This extra has no native dependencies.
Provides support for running workflows written using a Kubernetes cluster.
Provides support for Toil server mode, including support for the GA4GH Workflow Execution Service API.
Building from Source¶
If developing with Toil, you will need to build from source. This allows changes you make to Toil to be reflected immediately in your runtime environment.
First, clone the source:
$ git clone https://github.com/DataBiosphere/toil.git $ cd toil
Then, create and activate a virtualenv:
$ virtualenv venv $ . venv/bin/activate
From there, you can list all available Make targets by running
First and foremost, we want to install Toil’s build requirements (these are
additional packages that Toil needs to be tested and built but not to be run):
$ make prepare
Now, we can install Toil in development mode (such that changes to the source code will immediately affect the virtualenv):
$ make develop
Or, to install with support for all optional Installing Toil with Extra Features:
$ make develop extras=[aws,mesos,google,encryption,cwl]
$ make develop extras=[all]
To build the docs, run
make develop with all extras followed by
$ make docs
To run a quick batch of tests (this should take less than 30 minutes) run
$ export TOIL_TEST_QUICK=True; make test
For more information on testing see Running Tests.