toil.test.utils.toilKillTest¶

Attributes¶

`logger`
`pkg_root`

Exceptions¶

`NoSuchFileException`	Indicates that the specified file does not exist.
`NoSuchJobStoreException`	Indicates that the specified job store does not exist.

Classes¶

`Toil`	A context manager that represents a Toil workflow.
`ToilTest`	A common base class for Toil tests.
`ToilKillTest`	A set of test cases for "toil kill".
`ToilKillTestWithAWSJobStore`	A set of test cases for "toil kill" using the AWS job store.

Functions¶

`generate_locator`(job_store_type[, local_suggestion, ...])	Generate a random locator for a job store of the given type. Raises an
`needs_aws_s3`(test_item)	Use as a decorator before test classes or methods to run only if AWS S3 is usable.
`needs_cwl`(test_item)	Use as a decorator before test classes or methods to only run them if CWLTool is installed

Module Contents¶

class toil.test.utils.toilKillTest.Toil(options)[source]¶

Bases: ContextManager[Toil]

A context manager that represents a Toil workflow.

Specifically the batch system, job store, and its configuration.

Parameters:: options (argparse.Namespace)

config: Config¶

__enter__()[source]¶

Derive configuration from the command line options.

Then load the job store and, on restart, consolidate the derived configuration with the one from the previous invocation of the workflow.

Return type:: Toil

__exit__(exc_type, exc_val, exc_tb)[source]¶

Clean up after a workflow invocation.

Depending on the configuration, delete the job store.

Parameters:

exc_type (Optional[Type[BaseException]])
exc_val (Optional[BaseException])
exc_tb (Optional[types.TracebackType])

Return type:

Literal[False]

start(rootJob)[source]¶

Invoke a Toil workflow with the given job as the root for an initial run.

This method must be called in the body of a with Toil(...) as toil: statement. This method should not be called more than once for a workflow that has not finished.

Parameters:: rootJob (toil.job.Job) – The root job of the workflow
Returns:: The root job’s return value
Return type:: Any

restart()[source]¶

Restarts a workflow that has been interrupted.

Returns:: The root job’s return value
Return type:: Any

classmethod getJobStore(locator)[source]¶

Create an instance of the concrete job store implementation that matches the given locator.

Parameters:: locator (str) – The location of the job store to be represent by the instance
Returns:: an instance of a concrete subclass of AbstractJobStore
Return type:: toil.jobStores.abstractJobStore.AbstractJobStore

static parseLocator(locator)[source]¶

Parameters:: locator (str)
Return type:: Tuple[str, str]

static buildLocator(name, rest)[source]¶

Parameters:

name (str)
rest (str)

Return type:

str

classmethod resumeJobStore(locator)[source]¶

Parameters:: locator (str)
Return type:: toil.jobStores.abstractJobStore.AbstractJobStore

static createBatchSystem(config)[source]¶

Create an instance of the batch system specified in the given config.

Parameters:: config (Config) – the current configuration
Returns:: an instance of a concrete subclass of AbstractBatchSystem
Return type:: toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

importFile(srcUrl: str, sharedFileName: str, symlink: bool = True) → None[source]¶
importFile(srcUrl: str, sharedFileName: None = None, symlink: bool = True) → toil.fileStores.FileID

import_file(src_uri: str, shared_file_name: str, symlink: bool = True, check_existence: bool = True) → None[source]¶

import_file(src_uri: str, shared_file_name: None = None, symlink: bool = True, check_existence: bool = True) → toil.fileStores.FileID

Import the file at the given URL into the job store.

By default, returns None if the file does not exist.

Parameters:: check_existence – If true, raise FileNotFoundError if the file does not exist. If false, return None when the file does not exist.

See toil.jobStores.abstractJobStore.AbstractJobStore.importFile() for a full description

exportFile(jobStoreFileID, dstUrl)[source]¶

Parameters:

jobStoreFileID (toil.fileStores.FileID)
dstUrl (str)

Return type:

None

export_file(file_id, dst_uri)[source]¶

Export file to destination pointed at by the destination URL.

See toil.jobStores.abstractJobStore.AbstractJobStore.exportFile() for a full description

Parameters:

file_id (toil.fileStores.FileID)
dst_uri (str)

Return type:

None

static normalize_uri(uri, check_existence=False)[source]¶

Given a URI, if it has no scheme, prepend “file:”.

Parameters:

check_existence (bool) – If set, raise FileNotFoundError if a URI points to a local file that does not exist.
uri (str)

Return type:

str

static getToilWorkDir(configWorkDir=None)[source]¶

Return a path to a writable directory under which per-workflow directories exist.

This directory is always required to exist on a machine, even if the Toil worker has not run yet. If your workers and leader have different temp directories, you may need to set TOIL_WORKDIR.

Parameters:: configWorkDir (Optional[str]) – Value passed to the program using the –workDir flag
Returns:: Path to the Toil work directory, constant across all machines
Return type:: str

classmethod get_toil_coordination_dir(config_work_dir, config_coordination_dir)[source]¶

Return a path to a writable directory, which will be in memory if convenient. Ought to be used for file locking and coordination.

Parameters:

config_work_dir (Optional[str]) – Value passed to the program using the –workDir flag
config_coordination_dir (Optional[str]) – Value passed to the program using the –coordinationDir flag
workflow_id – Used if a tmpdir_prefix exists to create full directory paths unique per workflow

Returns:

Path to the Toil coordination directory. Ought to be on a POSIX filesystem that allows directories containing open files to be deleted.

Return type:

str

static get_workflow_path_component(workflow_id)[source]¶

Get a safe filesystem path component for a workflow.

Will be consistent for all processes on a given machine, and different for all processes on different machines.

Parameters:: workflow_id (str) – The ID of the current Toil workflow.
Return type:: str

classmethod getLocalWorkflowDir(workflowID, configWorkDir=None)[source]¶

Return the directory where worker directories and the cache will be located for this workflow on this machine.

Parameters:

configWorkDir (Optional[str]) – Value passed to the program using the –workDir flag
workflowID (str)

Returns:

Path to the local workflow directory on this machine

Return type:

str

classmethod get_local_workflow_coordination_dir(workflow_id, config_work_dir, config_coordination_dir)[source]¶

Return the directory where coordination files should be located for this workflow on this machine. These include internal Toil databases and lock files for the machine.

If an in-memory filesystem is available, it is used. Otherwise, the local workflow directory, which may be on a shared network filesystem, is used.

Parameters:

workflow_id (str) – Unique ID of the current workflow.
config_work_dir (Optional[str]) – Value used for the work directory in the current Toil Config.
config_coordination_dir (Optional[str]) – Value used for the coordination directory in the current Toil Config.

Returns:

Path to the local workflow coordination directory on this machine.

Return type:

str

exception toil.test.utils.toilKillTest.NoSuchFileException(jobStoreFileID, customName=None, *extra)[source]¶

Bases: Exception

Indicates that the specified file does not exist.

Parameters:

jobStoreFileID (toil.fileStores.FileID)
customName (Optional[str])
extra (Any)

exception toil.test.utils.toilKillTest.NoSuchJobStoreException(locator, prefix)[source]¶

Bases: LocatorException

Indicates that the specified job store does not exist.

Parameters:

locator (str)
prefix (str)

toil.test.utils.toilKillTest.generate_locator(job_store_type, local_suggestion=None, decoration=None)[source]¶

Generate a random locator for a job store of the given type. Raises an JobStoreUnavailableException if that job store cannot be used.

Parameters:

job_store_type (str) – Registry name of the job store to use.
local_suggestion (Optional[str]) – Path to a nonexistent local directory suitable for use as a file job store.
decoration (Optional[str]) – Extra string to add to the job store locator, if convenient.

Return str:

Job store locator for a usable job store.

Return type:

str

class toil.test.utils.toilKillTest.ToilTest(methodName='runTest')[source]¶

Bases: unittest.TestCase

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

setup_method(method)[source]¶

Parameters:: method (Any)
Return type:: None

classmethod setUpClass()[source]¶

Hook method for setting up class fixture before running tests in the class.

Return type:: None

classmethod tearDownClass()[source]¶

Hook method for deconstructing the class fixture after running all tests in the class.

Return type:: None

setUp()[source]¶

Hook method for setting up the test fixture before exercising it.

Return type:: None

tearDown()[source]¶

Hook method for deconstructing the test fixture after testing it.

Return type:: None

classmethod awsRegion()[source]¶

Pick an appropriate AWS region.

Use us-west-2 unless running on EC2, in which case use the region in which the instance is located

Return type:: str

toil.test.utils.toilKillTest.needs_aws_s3(test_item)[source]¶

Use as a decorator before test classes or methods to run only if AWS S3 is usable.

Parameters:: test_item (MT)
Return type:: MT

toil.test.utils.toilKillTest.needs_cwl(test_item)[source]¶

Use as a decorator before test classes or methods to only run them if CWLTool is installed and configured.

Parameters:: test_item (MT)
Return type:: MT

toil.test.utils.toilKillTest.logger¶

toil.test.utils.toilKillTest.pkg_root¶

class toil.test.utils.toilKillTest.ToilKillTest(*args, **kwargs)[source]¶

Bases: toil.test.ToilTest

A set of test cases for “toil kill”.

setUp()[source]¶: Shared test variables.

tearDown()[source]¶: Default tearDown for unittest.

test_cwl_toil_kill()[source]¶: Test “toil kill” on a CWL workflow with a 100 second sleep.

class toil.test.utils.toilKillTest.ToilKillTestWithAWSJobStore(*args, **kwargs)[source]¶

Bases: ToilKillTest

A set of test cases for “toil kill” using the AWS job store.

toil.test.utils.toilKillTest¶

Attributes¶

Exceptions¶

Classes¶

Functions¶

Module Contents¶

Toil

Navigation

Related Topics