Toil

Toil is a workflow engine entirely written in Python. It features:

  • Easy installation, e.g. pip install toil.

  • Common Workflow Language (CWL) support

    Complete support for the draft-3 CWL specification, allowing it to execute CWL workflows.

  • Cross platform support

    Develop and test on your laptop then deploy on any of the following:

  • A small API

    Easily mastered, the Python user API for defining and running workflows is built upon one core class.

  • Complete file and stream management:

    Temporary and persistent file management that abstracts the details of the underlying file system, providing a uniform interface regardless of environment. Supports both atomic file transfer and streaming interfaces, and provides encryption of user data.

  • Scalability:

    Toil can easily handle workflows concurrently using hundreds of nodes and thousands of cores.

  • Robustness:

    Toil workflows support arbitrary worker and leader failure, with strong check-pointing that always allows resumption.

  • Efficiency:

    Caching, fine grained, per task, resource requirement specifications, and support for the AWS spot market mean workflows can be executed with little waste.

  • Declarative and dynamic workflow creation:

    Workflows can be declared statically, but new jobs can be added dynamically during execution within any existing job, allowing arbitrarily complex workflow graphs with millions of jobs within them.

  • Support for databases and services:

    For example, Apache Spark clusters can be created quickly and easily integrated within a toil workflow as a service, with precisely defined time start and end times that fits with the flow of other jobs in the workflow.

  • Open Source: An Apache license allows unrestricted use, incorporation and modification.

Contents:

Indices and tables