Toil is a workflow engine entirely written in Python. It features:
Easy installation, e.g.
pip install toil.
Complete support for the draft-3 CWL specification, allowing it to execute CWL workflows.
Cross platform support
Develop and test on your laptop then deploy on any of the following:
A small API
Easily mastered, the Python user API for defining and running workflows is built upon one core class.
Complete file and stream management:
Temporary and persistent file management that abstracts the details of the underlying file system, providing a uniform interface regardless of environment. Supports both atomic file transfer and streaming interfaces, and provides encryption of user data.
Toil can easily handle workflows concurrently using hundreds of nodes and thousands of cores.
Toil workflows support arbitrary worker and leader failure, with strong check-pointing that always allows resumption.
Caching, fine grained, per task, resource requirement specifications, and support for the AWS spot market mean workflows can be executed with little waste.
Declarative and dynamic workflow creation:
Workflows can be declared statically, but new jobs can be added dynamically during execution within any existing job, allowing arbitrarily complex workflow graphs with millions of jobs within them.
Support for databases and services:
For example, Apache Spark clusters can be created quickly and easily integrated within a toil workflow as a service, with precisely defined time start and end times that fits with the flow of other jobs in the workflow.
Open Source: An Apache license allows unrestricted use, incorporation and modification.
- Cloud installation
- Running a workflow
- Running in the cloud
- Command line interface and arguments
- Developing a workflow
- Scripting quick start
- Job basics
- Invoking a workflow
- Specifying arguments via the command line
- Resuming a workflow
- Functions and job functions
- Workflows with multiple jobs
- Dynamic job creation
- Managing files within a workflow
- Toil API
- Toil architecture
- The batch system interface
- The job store interface