Toil Class API

The Toil class configures and starts a Toil run.

class toil.common.Toil(options, workflow_name=None, trs_spec=None)[source]

A context manager that represents a Toil workflow.

Specifically the batch system, job store, and its configuration.

Parameters:
__init__(options, workflow_name=None, trs_spec=None)[source]

Initialize a Toil object from the given options.

Note that this is very light-weight and that the bulk of the work is done when the context is entered.

Parameters:

  • options (Namespace) – command line options specified by the user

  • workflow_name (Optional[str]) – A human-readable name (probably a filename, URL, or TRS specifier) for the workflow being run. Used for Toil history storage.

  • trs_spec (Optional[str]) – A TRS id:version string for the workflow being run, if any. Used for Toil history storage and publishing workflow execution metrics to Dockstore.

Return type:

None

start(rootJob)[source]

Invoke a Toil workflow with the given job as the root for an initial run.

This method must be called in the body of a with Toil(...) as toil: statement. This method should not be called more than once for a workflow that has not finished.

Parameters:

rootJob (Job) – The root job of the workflow

Return type:

Any

Returns:

The root job’s return value

restart()[source]

Restarts a workflow that has been interrupted.

Return type:

Any

Returns:

The root job’s return value

classmethod getJobStore(locator)[source]

Create an instance of the concrete job store implementation that matches the given locator.

Parameters:

  • locator (str) – The location of the job store to be represent by the instance

  • locator

Return type:

AbstractJobStore

Returns:

an instance of a concrete subclass of AbstractJobStore

static parseLocator(locator)[source]

Parse a job store locator to a type string and the data needed for that implementation to connect to it.

Does not validate the set of possible job store types.

Raises:

RuntimeError – if the locator is not in the approproate syntax.

Parameters:

locator (str)

Return type:

tuple[str, str]

classmethod canonical_locator(locator)[source]

Turn a job store locator into one that will work from any directory and always includes the explicit type of job store.

Parameters:

locator (str)

Return type:

str

static createBatchSystem(config)[source]

Create an instance of the batch system specified in the given config.

Parameters:

config (Config) – the current configuration

Return type:

AbstractBatchSystem

Returns:

an instance of a concrete subclass of AbstractBatchSystem

import_file(src_uri, shared_file_name=None, symlink=True, check_existence=True)[source]

Import the file at the given URL into the job store.

By default, returns None if the file does not exist.

Parameters:

  • check_existence (bool) – If true, raise FileNotFoundError if the file does not exist. If false, return None when the file does not exist.

  • src_uri (str)

  • shared_file_name (str | None)

  • symlink (bool)

Return type:

Optional[FileID]

See toil.jobStores.abstractJobStore.AbstractJobStore.importFile() for a full description

Parameters:
Return type:

FileID | None

export_file(file_id, dst_uri)[source]

Export file to destination pointed at by the destination URL.

See toil.jobStores.abstractJobStore.AbstractJobStore.exportFile() for a full description

Parameters:
Return type:

None

static normalize_uri(uri, check_existence=False, dir_path=None)[source]

Given a URI, if it has no scheme, make it a properly quoted file: URI.

Parameters:

  • check_existence (bool) – If set, raise FileNotFoundError if a URI points to a local file that does not exist.

  • dir_path (Optional[str]) – If specified, interpret relative paths relative to the given directory path instead of the current one.

  • uri (str)

Return type:

str

static getToilWorkDir(configWorkDir=None)[source]

Return a path to a writable directory under which per-workflow directories exist.

This directory is always required to exist on a machine, even if the Toil worker has not run yet. If your workers and leader have different temp directories, you may need to set TOIL_WORKDIR.

Parameters:

configWorkDir (Optional[str]) – Value passed to the program using the –workDir flag

Return type:

str

Returns:

Path to the Toil work directory, constant across all machines

classmethod get_toil_coordination_dir(config_work_dir, config_coordination_dir)[source]

Return a path to a writable directory, which will be in memory if convenient. Ought to be used for file locking and coordination.

Parameters:

  • config_work_dir (Optional[str]) – Value passed to the program using the –workDir flag

  • config_coordination_dir (Optional[str]) – Value passed to the program using the –coordinationDir flag

  • workflow_id – Used if a tmpdir_prefix exists to create full directory paths unique per workflow

Return type:

str

Returns:

Path to the Toil coordination directory. Ought to be on a POSIX filesystem that allows directories containing open files to be deleted.

static get_workflow_path_component(workflow_id)[source]

Get a safe filesystem path component for a workflow.

Will be consistent for all processes on a given machine, and different for all processes on different machines.

Parameters:

workflow_id (str) – The ID of the current Toil workflow.

Return type:

str

classmethod getLocalWorkflowDir(workflowID, configWorkDir=None)[source]

Return the directory where worker directories and the cache will be located for this workflow on this machine.

Parameters:

  • configWorkDir (Optional[str]) – Value passed to the program using the –workDir flag

  • workflowID (str)

Return type:

str

Returns:

Path to the local workflow directory on this machine

classmethod get_local_workflow_coordination_dir(workflow_id, config_work_dir, config_coordination_dir)[source]

Return the directory where coordination files should be located for this workflow on this machine. These include internal Toil databases and lock files for the machine.

If an in-memory filesystem is available, it is used. Otherwise, the local workflow directory, which may be on a shared network filesystem, is used.

Parameters:

  • workflow_id (str) – Unique ID of the current workflow.

  • config_work_dir (Optional[str]) – Value used for the work directory in the current Toil Config.

  • config_coordination_dir (Optional[str]) – Value used for the coordination directory in the current Toil Config.

Return type:

str

Returns:

Path to the local workflow coordination directory on this machine.