toil.common

Attributes

JOBSTORE_HELP
dockerRegistry
dockerTag
version
UUID_LENGTH
logger
TOIL_HOME_DIR
DEFAULT_CONFIG_FILE

Exceptions

ToilRestartException	Common base class for all non-exit exceptions.
ToilContextManagerException	Common base class for all non-exit exceptions.

Classes

ClusterDesiredSizeMessage	Produced by the Toil-integrated autoscaler to describe the number of
ClusterSizeMessage	Produced by the Toil-integrated autoscaler describe the number of
JobCompletedMessage	Produced when a job is completed, whether successful or not.
JobFailedMessage	Produced when a job is completely failed, and will not be retried again.
JobIssuedMessage	Produced when a job is issued to run on the batch system.
JobMissingMessage	Produced when a job goes missing and should be in the batch system but isn't.
MessageBus	Holds messages that should cause jobs to change their scheduling states.
QueueSizeMessage	Produced to describe the size of the queue of jobs issued but not yet
FileID	A small wrapper around Python's builtin string class.
RealtimeLogger	Provide a logger that logs over UDP to the leader.
Config	Class to represent configuration operations for a toil workflow run.
Toil	A context manager that represents a Toil workflow.
ToilMetrics

Functions

add_base_toil_options(parser[, jobstore_as_flag, cwl])	Add base Toil command line options to the parser.
add_cwl_options(parser[, suppress])	Add CWL options to the parser. This only adds nonpositional CWL arguments.
add_wdl_options(parser[, suppress])	Add WDL options to a parser. This only adds nonpositional WDL arguments
logProcessContext(config)
lookupEnvVar(name, envName, defaultValue)	Look up environment variables that control Toil and log the result.
set_batchsystem_options(batch_system, set_option)	Call set_option for all the options for the given named batch system, or
deprecated(new_function_name)
try_path(path[, min_size])	Try to use the given path. Return it if it exists or can be made,
AtomicFileCreate(final_path[, keep])	Context manager to create a temporary file. Entering returns path to
retry([intervals, infinite_retries, errors, ...])	Retry a function if it fails with any Exception defined in "errors".
add_provisioner_options(parser)
cluster_factory(provisioner[, clusterName, ...])	Find and instantiate the appropriate provisioner instance to make clusters in the given cloud.
add_logging_options(parser[, default_level])	Add logging options to set the global log level.
set_logging_from_options(options)
check_and_create_toil_home_dir()	Ensure that TOIL_HOME_DIR exists.
check_and_create_default_config_file()	If the default config file does not exist, create it in the Toil home directory. Create the Toil home directory
check_and_create_config_file(filepath)	If the config file at the filepath does not exist, try creating it.
generate_config(filepath)	Write a Toil config file to the given path.
parser_with_common_options([provisioner_options, ...])
addOptions(parser[, jobstore_as_flag, cwl, wdl])	Add all Toil command line options to a parser.
getNodeID()	Return unique ID of the current node (host). The resulting string will be convertible to a uuid.UUID.
cacheDirName(workflowID)	return: Name of the cache directory.
getDirSizeRecursively(dirPath)	This method will return the cumulative number of bytes occupied by the files
getFileSystemSize(dirPath)	Return the free space, and total size of the file system hosting dirPath.
safeUnpickleFromStream(stream)

Module Contents

toil.common.add_base_toil_options(parser, jobstore_as_flag=False, cwl=False)

Add base Toil command line options to the parser. :param parser: Argument parser to add options to :param jobstore_as_flag: make the job store option a –jobStore flag instead of a required jobStore positional argument. :param cwl: whether CWL should be included or not

Parameters:

• parser (argparse.ArgumentParser)

• jobstore_as_flag (bool)

• cwl (bool)

Return type:

None

toil.common.JOBSTORE_HELP = Multiline-String

Show Value

"""The location of the job store for the workflow.  A job store holds persistent information about the jobs, stats, and files in a workflow. If the workflow is run with a distributed batch system, the job store must be accessible by all worker nodes. Depending on the desired job store implementation, the location should be formatted according to one of the following schemes:

file:<path> where <path> points to a directory on the file system

aws:<region>:<prefix> where <region> is the name of an AWS region like us-west-2 and <prefix> will be prepended to the names of any top-level AWS resources in use by job store, e.g. S3 buckets.

 google:<project_id>:<prefix> TODO: explain

For backwards compatibility, you may also specify ./foo (equivalent to file:./foo or just file:foo) or /bar (equivalent to file:/bar)."""

toil.common.add_cwl_options(parser, suppress=True)

Add CWL options to the parser. This only adds nonpositional CWL arguments.

Parameters:

• parser (argparse.ArgumentParser) – Parser to add options to

• suppress (bool) – Suppress help output

Returns:

None

Return type:

None

toil.common.add_wdl_options(parser, suppress=True)

Add WDL options to a parser. This only adds nonpositional WDL arguments :param parser: Parser to add options to :param suppress: Suppress help output :return: None

Parameters:

• parser (argparse.ArgumentParser)

• suppress (bool)

Return type:

None

toil.common.logProcessContext(config)

Parameters:

config (common.Config)

Return type:

None

toil.common.lookupEnvVar(name, envName, defaultValue)

Look up environment variables that control Toil and log the result.

Parameters:

• name (str) – the human readable name of the variable

• envName (str) – the name of the environment variable to lookup

• defaultValue (str) – the fall-back value

Returns:

the value of the environment variable or the default value the variable is not set

Return type:

str

toil.common.set_batchsystem_options(batch_system, set_option)

Call set_option for all the options for the given named batch system, or all batch systems if no name is provided.

Parameters:

• batch_system (Optional[str])

• set_option (OptionSetter)

Return type:

None

class toil.common.ClusterDesiredSizeMessage

Bases: NamedTuple

Produced by the Toil-integrated autoscaler to describe the number of instances of a certain type that it thinks will be needed.

instance_type: str

desired_size: int

class toil.common.ClusterSizeMessage

Bases: NamedTuple

Produced by the Toil-integrated autoscaler describe the number of instances of a certain type in a cluster.

instance_type: str

current_size: int

class toil.common.JobCompletedMessage

Bases: NamedTuple

Produced when a job is completed, whether successful or not.

job_type: str

job_id: str

exit_code: int

class toil.common.JobFailedMessage

Bases: NamedTuple

Produced when a job is completely failed, and will not be retried again.

job_type: str

job_id: str

class toil.common.JobIssuedMessage

Bases: NamedTuple

Produced when a job is issued to run on the batch system.

job_type: str

job_id: str

toil_batch_id: int

class toil.common.JobMissingMessage

Bases: NamedTuple

Produced when a job goes missing and should be in the batch system but isn’t.

job_id: str

class toil.common.MessageBus

Holds messages that should cause jobs to change their scheduling states. Messages are put in and buffered, and can be taken out and handled as batches when convenient.

All messages are NamedTuple objects of various subtypes.

Message order is guaranteed to be preserved within a type.

publish(message)

Put a message onto the bus. Can be called from any thread.

Parameters:

message (Any)

Return type:

None

check()

If we are in the owning thread, deliver any messages that are in the queue for us. Must be called every once in a while in the main thread, possibly through inbox objects.

Return type:

None

MessageType

subscribe(message_type, handler)

Register the given callable to be called when messages of the given type are sent. It will be called with messages sent after the subscription is created. Returns a subscription object; when the subscription object is GC’d the subscription will end.

Parameters:

• message_type (Type[MessageType])

• handler (Callable[[MessageType], Any])

Return type:

pubsub.core.listener.Listener

connect(wanted_types)

Get a connection object that serves as an inbox for messages of the given types. Messages of those types will accumulate in the inbox until it is destroyed. You can check for them at any time.

Parameters:

wanted_types (List[type])

Return type:

MessageBusConnection

outbox()

Get a connection object that only allows sending messages.

Return type:

MessageOutbox

connect_output_file(file_path)

Send copies of all messages to the given output file.

Returns connection data which must be kept alive for the connection to persist. That data is opaque: the user is not supposed to look at it or touch it or do anything with it other than store it somewhere or delete it.

Parameters:

file_path (str)

Return type:

Any

classmethod scan_bus_messages(stream, message_types)

Get an iterator over all messages in the given log stream of the given types, in order. Discard any trailing partial messages.

Parameters:

• stream (IO[bytes])

• message_types (List[Type[NamedTuple]])

Return type:

Iterator[Any]

class toil.common.QueueSizeMessage

Bases: NamedTuple

Produced to describe the size of the queue of jobs issued but not yet completed. Theoretically recoverable from other messages.

queue_size: int

class toil.common.FileID(fileStoreID, size, executable=False)

Bases: str

A small wrapper around Python’s builtin string class.

It is used to represent a file’s ID in the file store, and has a size attribute that is the file’s size in bytes. This object is returned by importFile and writeGlobalFile.

Calls into the file store can use bare strings; size will be queried from the job store if unavailable in the ID.

Parameters:

• fileStoreID (str)

• size (int)

• executable (bool)

pack()

Pack the FileID into a string so it can be passed through external code.

Return type:

str

classmethod forPath(fileStoreID, filePath)

Parameters:

• fileStoreID (str)

• filePath (str)

Return type:

FileID

classmethod unpack(packedFileStoreID)

Unpack the result of pack() into a FileID object.

Parameters:

packedFileStoreID (str)

Return type:

FileID

toil.common.deprecated(new_function_name)

Parameters:

new_function_name (str)

Return type:

Callable[Ellipsis, Any]

toil.common.try_path(path, min_size=100 * 1024 * 1024)

Try to use the given path. Return it if it exists or can be made, and we can make things within it, or None otherwise.

Parameters:

• min_size (int) – Reject paths on filesystems smaller than this many bytes.

• path (str)

Return type:

Optional[str]

toil.common.AtomicFileCreate(final_path, keep=False)

Context manager to create a temporary file. Entering returns path to the temporary file in the same directory as finalPath. If the code in context succeeds, the file renamed to its actually name. If an error occurs, the file is not installed and is removed unless keep is specified.

Parameters:

• final_path (str)

• keep (bool)

Return type:

Iterator[str]

toil.common.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)

Retry a function if it fails with any Exception defined in “errors”.

Does so every x seconds, where x is defined by a list of numbers (ints or floats) in “intervals”. Also accepts ErrorCondition events for more detailed retry attempts.

Parameters:

• intervals (Optional[List]) – A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s

• infinite_retries (bool) – If this is True, reset the intervals when they run out. Defaults to: False.

• errors (Optional[Sequence[Union[ErrorCondition, Type[Exception]]]]) –

  A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: - Exception (required) - Error codes that must match to be retried (optional; defaults to not checking) - A string that must be in the error message to be retried (optional; defaults to not checking) - A bool that can be set to False to always error on this condition.

  If not specified, this will default to a generic Exception.

• log_message (Optional[Tuple[Callable, str]]) – Optional tuple of (“log/print function()”, “message string”) that will precede each attempt.

• prepare (Optional[List[Callable]]) – Optional list of functions to call, with the function’s arguments, between retries, to reset state.

Returns:

The result of the wrapped function or raise.

Return type:

Callable[[Callable[Ellipsis, RT]], Callable[Ellipsis, RT]]

toil.common.add_provisioner_options(parser)

Parameters:

parser (argparse.ArgumentParser)

Return type:

None

toil.common.cluster_factory(provisioner, clusterName=None, clusterType='mesos', zone=None, nodeStorage=50, nodeStorageOverrides=None, sseKey=None, enable_fuse=False)

Find and instantiate the appropriate provisioner instance to make clusters in the given cloud.

Raises ClusterTypeNotSupportedException if the given provisioner does not implement clusters of the given type.

Parameters:

• provisioner (str) – The cloud type of the cluster.

• clusterName (Optional[str]) – The name of the cluster.

• clusterType (str) – The type of cluster: ‘mesos’ or ‘kubernetes’.

• zone (Optional[str]) – The cloud zone

• nodeStorage (int)

• nodeStorageOverrides (Optional[List[str]])

• sseKey (Optional[str])

• enable_fuse (bool)

Returns:

A cluster object for the the cloud type.

Return type:

Union[aws.awsProvisioner.AWSProvisioner, gceProvisioner.GCEProvisioner]

class toil.common.RealtimeLogger(batchSystem, level=defaultLevel)

Provide a logger that logs over UDP to the leader.

To use in a Toil job, do:

>>> from toil.realtimeLogger import RealtimeLogger
>>> RealtimeLogger.info("This logging message goes straight to the leader")

That’s all a user of Toil would need to do. On the leader, Job.Runner.startToil() automatically starts the UDP server by using an instance of this class as a context manager.

Parameters:

• batchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem)

• level (str)

envPrefix = 'TOIL_RT_LOGGING_'

defaultLevel = 'INFO'

lock

loggingServer = None

serverThread = None

initialized = 0

logger = None

classmethod getLogger()

Get the logger that logs real-time to the leader.

Note that if the returned logger is used on the leader, you will see the message twice, since it still goes to the normal log handlers, too.

Return type:

logging.Logger

__enter__()

Return type:

None

__exit__(exc_type, exc_val, exc_tb)

Parameters:

• exc_type (Optional[Type[BaseException]])

• exc_val (Optional[BaseException])

• exc_tb (Optional[types.TracebackType])

Return type:

None

toil.common.add_logging_options(parser, default_level=None)

Add logging options to set the global log level.

Parameters:

• default_level (Optional[int]) – A logging level, like logging.INFO, to use as the default.

• parser (argparse.ArgumentParser)

Return type:

None

toil.common.set_logging_from_options(options)

Parameters:

options (Union[toil.common.Config, argparse.Namespace])

Return type:

None

toil.common.dockerRegistry = 'quay.io/ucsc_cgl'

toil.common.dockerTag = '7.0.0-d569ea5711eb310ffd5703803f7250ebf7c19576-py3.9'

toil.common.version = '7.0.0-d569ea5711eb310ffd5703803f7250ebf7c19576'

toil.common.UUID_LENGTH = 32

toil.common.logger

toil.common.TOIL_HOME_DIR: str

toil.common.DEFAULT_CONFIG_FILE: str

class toil.common.Config

Class to represent configuration operations for a toil workflow run.

logFile: str | None

logRotating: bool

cleanWorkDir: str

max_jobs: int

max_local_jobs: int

manualMemArgs: bool

run_local_jobs_on_workers: bool

coalesceStatusCalls: bool

mesos_endpoint: str | None

mesos_framework_id: str | None

mesos_role: str | None

mesos_name: str

kubernetes_host_path: str | None

kubernetes_owner: str | None

kubernetes_service_account: str | None

kubernetes_pod_timeout: float

kubernetes_privileged: bool

tes_endpoint: str

tes_user: str

tes_password: str

tes_bearer_token: str

aws_batch_region: str | None

aws_batch_queue: str | None

aws_batch_job_role_arn: str | None

scale: float

batchSystem: str

batch_logs_dir: str | None

The backing scheduler will be instructed, if possible, to save logs to this directory, where the leader can read them.

statePollingWait: int

state_polling_timeout: int

disableAutoDeployment: bool

workflowID: str | None

This attribute uniquely identifies the job store and therefore the workflow. It is necessary in order to distinguish between two consecutive workflows for which self.jobStore is the same, e.g. when a job store name is reused after a previous run has finished successfully and its job store has been clean up.

workflowAttemptNumber: int

jobStore: str

logLevel: str

colored_logs: bool

workDir: str | None

coordination_dir: str | None

noStdOutErr: bool

stats: bool

clean: str | None

clusterStats: str

restart: bool

caching: bool | None

symlinkImports: bool

moveOutputs: bool

provisioner: str | None

nodeTypes: List[Tuple[Set[str], float | None]]

minNodes: List[int]

maxNodes: List[int]

targetTime: float

betaInertia: float

scaleInterval: int

preemptibleCompensation: float

nodeStorage: int

nodeStorageOverrides: List[str]

metrics: bool

assume_zero_overhead: bool

maxPreemptibleServiceJobs: int

maxServiceJobs: int

deadlockWait: float | int

deadlockCheckInterval: float | int

defaultMemory: int

defaultCores: float | int

defaultDisk: int

defaultPreemptible: bool

defaultAccelerators: List[toil.job.AcceleratorRequirement]

maxCores: int

maxMemory: int

maxDisk: int

retryCount: int

enableUnlimitedPreemptibleRetries: bool

doubleMem: bool

maxJobDuration: int

rescueJobsFrequency: int

job_store_timeout: float

maxLogFileSize: int

writeLogs: str

writeLogsGzip: str

writeLogsFromAllJobs: bool

write_messages: str | None

realTimeLogging: bool

environment: Dict[str, str]

disableChaining: bool

disableJobStoreChecksumVerification: bool

sseKey: str | None

servicePollingInterval: int

useAsync: bool

forceDockerAppliance: bool

statusWait: int

disableProgress: bool

readGlobalFileMutableByDefault: bool

debugWorker: bool

disableWorkerOutputCapture: bool

badWorker: float

badWorkerFailInterval: float

kill_polling_interval: int

cwl: bool

set_from_default_config()

Return type:

None

prepare_start()

After options are set, prepare for initial start of workflow.

Return type:

None

prepare_restart()

Before restart options are set, prepare for a restart of a workflow. Set up any execution-specific parameters and clear out any stale ones.

Return type:

None

setOptions(options)

Creates a config object from the options object.

Parameters:

options (argparse.Namespace)

Return type:

None

check_configuration_consistency()

Old checks that cannot be fit into an action class for argparse

Return type:

None

__eq__(other)

Return self==value.

Parameters:

other (object)

Return type:

bool

__hash__()

Return hash(self).

Return type:

int

toil.common.check_and_create_toil_home_dir()

Ensure that TOIL_HOME_DIR exists.

Raises an error if it does not exist and cannot be created. Safe to run simultaneously in multiple processes.

Return type:

None

toil.common.check_and_create_default_config_file()

If the default config file does not exist, create it in the Toil home directory. Create the Toil home directory if needed

Raises an error if the default config file cannot be created. Safe to run simultaneously in multiple processes. If this process runs this function, it will always see the default config file existing with parseable contents, even if other processes are racing to create it.

No process will see an empty or partially-written default config file.

Return type:

None

toil.common.check_and_create_config_file(filepath)

If the config file at the filepath does not exist, try creating it. The parent directory should be created prior to calling this :param filepath: path to config file :return: None

Parameters:

filepath (str)

Return type:

None

toil.common.generate_config(filepath)

Write a Toil config file to the given path.

Safe to run simultaneously in multiple processes. No process will see an empty or partially-written file at the given path.

Set include to “cwl” or “wdl” to include cwl options and wdl options respectfully

Parameters:

filepath (str)

Return type:

None

toil.common.parser_with_common_options(provisioner_options=False, jobstore_option=True, prog=None, default_log_level=None)

Parameters:

• provisioner_options (bool)

• jobstore_option (bool)

• prog (Optional[str])

• default_log_level (Optional[int])

Return type:

configargparse.ArgParser

toil.common.addOptions(parser, jobstore_as_flag=False, cwl=False, wdl=False)

Add all Toil command line options to a parser.

Support for config files if using configargparse. This will also check and set up the default config file.

Parameters:

• jobstore_as_flag (bool) – make the job store option a –jobStore flag instead of a required jobStore positional argument.

• cwl (bool) – Whether CWL options are expected. If so, CWL options won’t be suppressed.

• wdl (bool) – Whether WDL options are expected. If so, WDL options won’t be suppressed.

• parser (argparse.ArgumentParser)

Return type:

None

toil.common.getNodeID()

Return unique ID of the current node (host). The resulting string will be convertible to a uuid.UUID.

Tries several methods until success. The returned ID should be identical across calls from different processes on the same node at least until the next OS reboot.

The last resort method is uuid.getnode() that in some rare OS configurations may return a random ID each time it is called. However, this method should never be reached on a Linux system, because reading from /proc/sys/kernel/random/boot_id will be tried prior to that. If uuid.getnode() is reached, it will be called twice, and exception raised if the values are not identical.

Return type:

str

class toil.common.Toil(options)

Bases: ContextManager[Toil]

A context manager that represents a Toil workflow.

Specifically the batch system, job store, and its configuration.

Parameters:

options (argparse.Namespace)

config: Config

__enter__()

Derive configuration from the command line options.

Then load the job store and, on restart, consolidate the derived configuration with the one from the previous invocation of the workflow.

Return type:

Toil

__exit__(exc_type, exc_val, exc_tb)

Clean up after a workflow invocation.

Depending on the configuration, delete the job store.

Parameters:

• exc_type (Optional[Type[BaseException]])

• exc_val (Optional[BaseException])

• exc_tb (Optional[types.TracebackType])

Return type:

Literal[False]

start(rootJob)

Invoke a Toil workflow with the given job as the root for an initial run.

This method must be called in the body of a with Toil(...) as toil: statement. This method should not be called more than once for a workflow that has not finished.

Parameters:

rootJob (toil.job.Job) – The root job of the workflow

Returns:

The root job’s return value

Return type:

Any

restart()

Restarts a workflow that has been interrupted.

Returns:

The root job’s return value

Return type:

Any

classmethod getJobStore(locator)

Create an instance of the concrete job store implementation that matches the given locator.

Parameters:

locator (str) – The location of the job store to be represent by the instance

Returns:

an instance of a concrete subclass of AbstractJobStore

Return type:

toil.jobStores.abstractJobStore.AbstractJobStore

static parseLocator(locator)

Parameters:

locator (str)

Return type:

Tuple[str, str]

static buildLocator(name, rest)

Parameters:

• name (str)

• rest (str)

Return type:

str

classmethod resumeJobStore(locator)

Parameters:

locator (str)

Return type:

toil.jobStores.abstractJobStore.AbstractJobStore

static createBatchSystem(config)

Create an instance of the batch system specified in the given config.

Parameters:

config (Config) – the current configuration

Returns:

an instance of a concrete subclass of AbstractBatchSystem

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

importFile(srcUrl: str, sharedFileName: str, symlink: bool = True) → None

importFile(srcUrl: str, sharedFileName: None = None, symlink: bool = True) → toil.fileStores.FileID

import_file(src_uri: str, shared_file_name: str, symlink: bool = True, check_existence: bool = True) → None

import_file(src_uri: str, shared_file_name: None = None, symlink: bool = True, check_existence: bool = True) → toil.fileStores.FileID

Import the file at the given URL into the job store.

By default, returns None if the file does not exist.

Parameters:

check_existence – If true, raise FileNotFoundError if the file does not exist. If false, return None when the file does not exist.

See toil.jobStores.abstractJobStore.AbstractJobStore.importFile() for a full description

exportFile(jobStoreFileID, dstUrl)

Parameters:

• jobStoreFileID (toil.fileStores.FileID)

• dstUrl (str)

Return type:

None

export_file(file_id, dst_uri)

Export file to destination pointed at by the destination URL.

See toil.jobStores.abstractJobStore.AbstractJobStore.exportFile() for a full description

Parameters:

• file_id (toil.fileStores.FileID)

• dst_uri (str)

Return type:

None

static normalize_uri(uri, check_existence=False)

Given a URI, if it has no scheme, prepend “file:”.

Parameters:

• check_existence (bool) – If set, raise FileNotFoundError if a URI points to a local file that does not exist.

• uri (str)

Return type:

str

static getToilWorkDir(configWorkDir=None)

Return a path to a writable directory under which per-workflow directories exist.

This directory is always required to exist on a machine, even if the Toil worker has not run yet. If your workers and leader have different temp directories, you may need to set TOIL_WORKDIR.

Parameters:

configWorkDir (Optional[str]) – Value passed to the program using the –workDir flag

Returns:

Path to the Toil work directory, constant across all machines

Return type:

str

classmethod get_toil_coordination_dir(config_work_dir, config_coordination_dir)

Return a path to a writable directory, which will be in memory if convenient. Ought to be used for file locking and coordination.

Parameters:

• config_work_dir (Optional[str]) – Value passed to the program using the –workDir flag

• config_coordination_dir (Optional[str]) – Value passed to the program using the –coordinationDir flag

• workflow_id – Used if a tmpdir_prefix exists to create full directory paths unique per workflow

Returns:

Path to the Toil coordination directory. Ought to be on a POSIX filesystem that allows directories containing open files to be deleted.

Return type:

str

static get_workflow_path_component(workflow_id)

Get a safe filesystem path component for a workflow.

Will be consistent for all processes on a given machine, and different for all processes on different machines.

Parameters:

workflow_id (str) – The ID of the current Toil workflow.

Return type:

str

classmethod getLocalWorkflowDir(workflowID, configWorkDir=None)

Return the directory where worker directories and the cache will be located for this workflow on this machine.

Parameters:

• configWorkDir (Optional[str]) – Value passed to the program using the –workDir flag

• workflowID (str)

Returns:

Path to the local workflow directory on this machine

Return type:

str

classmethod get_local_workflow_coordination_dir(workflow_id, config_work_dir, config_coordination_dir)

Return the directory where coordination files should be located for this workflow on this machine. These include internal Toil databases and lock files for the machine.

If an in-memory filesystem is available, it is used. Otherwise, the local workflow directory, which may be on a shared network filesystem, is used.

Parameters:

• workflow_id (str) – Unique ID of the current workflow.

• config_work_dir (Optional[str]) – Value used for the work directory in the current Toil Config.

• config_coordination_dir (Optional[str]) – Value used for the coordination directory in the current Toil Config.

Returns:

Path to the local workflow coordination directory on this machine.

Return type:

str

exception toil.common.ToilRestartException(message)

Bases: Exception

Common base class for all non-exit exceptions.

Parameters:

message (str)

exception toil.common.ToilContextManagerException

Bases: Exception

Common base class for all non-exit exceptions.

class toil.common.ToilMetrics(bus, provisioner=None)

Parameters:

• bus (toil.bus.MessageBus)

• provisioner (Optional[toil.provisioners.abstractProvisioner.AbstractProvisioner])

startDashboard(clusterName, zone)

Parameters:

• clusterName (str)

• zone (str)

Return type:

None

add_prometheus_data_source()

Return type:

None

log(message)

Parameters:

message (str)

Return type:

None

logClusterSize(m)

Parameters:

m (toil.bus.ClusterSizeMessage)

Return type:

None

logClusterDesiredSize(m)

Parameters:

m (toil.bus.ClusterDesiredSizeMessage)

Return type:

None

logQueueSize(m)

Parameters:

m (toil.bus.QueueSizeMessage)

Return type:

None

logMissingJob(m)

Parameters:

m (toil.bus.JobMissingMessage)

Return type:

None

logIssuedJob(m)

Parameters:

m (toil.bus.JobIssuedMessage)

Return type:

None

logFailedJob(m)

Parameters:

m (toil.bus.JobFailedMessage)

Return type:

None

logCompletedJob(m)

Parameters:

m (toil.bus.JobCompletedMessage)

Return type:

None

shutdown()

Return type:

None

toil.common.cacheDirName(workflowID)

Returns:

Name of the cache directory.

Parameters:

workflowID (str)

Return type:

str

toil.common.getDirSizeRecursively(dirPath)

This method will return the cumulative number of bytes occupied by the files on disk in the directory and its subdirectories.

If the method is unable to access a file or directory (due to insufficient permissions, or due to the file or directory having been removed while this function was attempting to traverse it), the error will be handled internally, and a (possibly 0) lower bound on the size of the directory will be returned.

Parameters:

dirPath (str) – A valid path to a directory or file.

Returns:

Total size, in bytes, of the file or directory at dirPath.

Return type:

int

toil.common.getFileSystemSize(dirPath)

Return the free space, and total size of the file system hosting dirPath.

Parameters:

dirPath (str) – A valid path to a directory.

Returns:

free space and total size of file system

Return type:

Tuple[int, int]

toil.common.safeUnpickleFromStream(stream)

Parameters:

stream (IO[Any])

Return type:

Any