toil.test.src.autoDeploymentTest

Attributes

exactPython
logger

Exceptions

FailedJobsException

Common base class for all non-exit exceptions.

Classes

concat	A literal iterable to combine sequence literals (lists, set) with generators or list comprehensions.
ApplianceTestSupport	A Toil test that runs a user script on a minimal cluster of appliance containers.
AutoDeploymentTest	Tests various auto-deployment scenarios. Using the appliance, i.e. a docker container,

Functions

needs_local_appliance(test_item)	Use as a decorator before test classes or methods to only run them if
needs_mesos(test_item)	Use as a decorator before test classes or methods to run only if Mesos is installed.
slow(test_item)	Use this decorator to identify tests that are slow and not critical.

Module Contents

exception toil.test.src.autoDeploymentTest.FailedJobsException(job_store, failed_jobs, exit_code=1)

Bases: Exception

Common base class for all non-exit exceptions.

Parameters:

• job_store (toil.jobStores.abstractJobStore.AbstractJobStore)

• failed_jobs (List[toil.job.JobDescription])

• exit_code (int)

__str__()

Stringify the exception, including the message.

Return type:

str

class toil.test.src.autoDeploymentTest.concat(*args)

A literal iterable to combine sequence literals (lists, set) with generators or list comprehensions.

Instead of

>>> [ -1 ] + [ x * 2 for x in range( 3 ) ] + [ -1 ]
[-1, 0, 2, 4, -1]

you can write

>>> list( concat( -1, ( x * 2 for x in range( 3 ) ), -1 ) )
[-1, 0, 2, 4, -1]

This is slightly shorter (not counting the list constructor) and does not involve array construction or concatenation.

Note that concat() flattens (or chains) all iterable arguments into a single result iterable:

>>> list( concat( 1, range( 2, 4 ), 4 ) )
[1, 2, 3, 4]

It only does so one level deep. If you need to recursively flatten a data structure, check out crush().

If you want to prevent that flattening for an iterable argument, wrap it in concat():

>>> list( concat( 1, concat( range( 2, 4 ) ), 4 ) )
[1, range(2, 4), 4]

Some more example.

>>> list( concat() ) # empty concat
[]
>>> list( concat( 1 ) ) # non-iterable
[1]
>>> list( concat( concat() ) ) # empty iterable
[]
>>> list( concat( concat( 1 ) ) ) # singleton iterable
[1]
>>> list( concat( 1, concat( 2 ), 3 ) ) # flattened iterable
[1, 2, 3]
>>> list( concat( 1, [2], 3 ) ) # flattened iterable
[1, 2, 3]
>>> list( concat( 1, concat( [2] ), 3 ) ) # protecting an iterable from being flattened
[1, [2], 3]
>>> list( concat( 1, concat( [2], 3 ), 4 ) ) # protection only works with a single argument
[1, 2, 3, 4]
>>> list( concat( 1, 2, concat( 3, 4 ), 5, 6 ) )
[1, 2, 3, 4, 5, 6]
>>> list( concat( 1, 2, concat( [ 3, 4 ] ), 5, 6 ) )
[1, 2, [3, 4], 5, 6]

Note that while strings are technically iterable, concat() does not flatten them.

>>> list( concat( 'ab' ) )
['ab']
>>> list( concat( concat( 'ab' ) ) )
['ab']

Parameters:

args (Any)

__iter__()

Return type:

Iterator[Any]

class toil.test.src.autoDeploymentTest.ApplianceTestSupport(methodName='runTest')

Bases: ToilTest

A Toil test that runs a user script on a minimal cluster of appliance containers.

i.e. one leader container and one worker container.

class Appliance(outer, mounts, cleanMounts=False)

Bases: toil.lib.threading.ExceptionalThread

A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

When subclassing this thread, override tryRun() instead of run().

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Parameters:

• outer (ApplianceTestSupport)

• mounts (Dict[str, str])

• cleanMounts (bool)

lock

__enter__()

Return type:

Appliance

__exit__(exc_type, exc_val, exc_tb)

Parameters:

• exc_type (Type[BaseException])

• exc_val (Exception)

• exc_tb (Any)

Return type:

Literal[False]

tryRun()

Return type:

None

runOnAppliance(*args, **kwargs)

Parameters:

• args (str)

• kwargs (Any)

Return type:

None

writeToAppliance(path, contents)

Parameters:

• path (str)

• contents (Any)

Return type:

None

deployScript(path, packagePath, script)

Deploy a Python module on the appliance.

Parameters:

• path (str) – the path (absolute or relative to the WORDIR of the appliance container) to the root of the package hierarchy where the given module should be placed. The given directory should be on the Python path.

• packagePath (str) – the desired fully qualified module name (dotted form) of the module

• script (str|callable) – the contents of the Python module. If a callable is given, its source code will be extracted. This is a convenience that lets you embed user scripts into test code as nested function.

Return type:

None

class LeaderThread(outer, mounts, cleanMounts=False)

Bases: Appliance

A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

When subclassing this thread, override tryRun() instead of run().

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Parameters:

• outer (ApplianceTestSupport)

• mounts (Dict[str, str])

• cleanMounts (bool)

class WorkerThread(outer, mounts, numCores)

Bases: Appliance

A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

When subclassing this thread, override tryRun() instead of run().

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

Parameters:

• outer (ApplianceTestSupport)

• mounts (Dict[str, str])

• numCores (int)

toil.test.src.autoDeploymentTest.needs_local_appliance(test_item)

Use as a decorator before test classes or methods to only run them if the Toil appliance Docker image is downloaded.

Parameters:

test_item (MT)

Return type:

MT

toil.test.src.autoDeploymentTest.needs_mesos(test_item)

Use as a decorator before test classes or methods to run only if Mesos is installed.

Parameters:

test_item (MT)

Return type:

MT

toil.test.src.autoDeploymentTest.slow(test_item)

Use this decorator to identify tests that are slow and not critical. Skip if TOIL_TEST_QUICK is true.

Parameters:

test_item (MT)

Return type:

MT

toil.test.src.autoDeploymentTest.exactPython = 'python3.9'

toil.test.src.autoDeploymentTest.logger

class toil.test.src.autoDeploymentTest.AutoDeploymentTest(methodName='runTest')

Bases: toil.test.ApplianceTestSupport

Tests various auto-deployment scenarios. Using the appliance, i.e. a docker container, for these tests allows for running worker processes on the same node as the leader process while keeping their file systems separate from each other and the leader process. Separate file systems are crucial to prove that auto-deployment does its job.

setUp()

Hook method for setting up the test fixture before exercising it.

sitePackages

testRestart()

Test whether auto-deployment works on restart.

testSplitRootPackages()

Test whether auto-deployment works with a virtualenv in which jobs are defined in completely separate branches of the package hierarchy. Initially, auto-deployment did deploy the entire virtualenv but jobs could only be defined in one branch of the package hierarchy. We define a branch as the maximum set of fully qualified package paths that share the same first component. IOW, a.b and a.c are in the same branch, while a.b and d.c are not.

testUserTypesInJobFunctionArgs()

Test encapsulated, function-wrapping jobs where the function arguments reference user-defined types.

Mainly written to cover https://github.com/BD2KGenomics/toil/issues/1259 but then also revealed https://github.com/BD2KGenomics/toil/issues/1278.

testDeferralWithConcurrentEncapsulation()

Ensure that the following DAG succeeds:

              ┌───────────┐
              │ Root (W1) │
              └───────────┘
                    │
         ┌──────────┴─────────┐
         ▼                    ▼
┌────────────────┐ ┌────────────────────┐
│ Deferring (W2) │ │ Encapsulating (W3) │═══════════════╗
└────────────────┘ └────────────────────┘               ║
                              │                         ║
                              ▼                         ▼
                    ┌───────────────────┐      ┌────────────────┐
                    │ Encapsulated (W3) │      │ Follow-on (W6) │
                    └───────────────────┘      └────────────────┘
                              │                         │
                      ┌───────┴────────┐                │
                      ▼                ▼                ▼
              ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
              │ Dummy 1 (W4) │ │ Dummy 2 (W5) │ │  Last (W6)   │
              └──────────────┘ └──────────────┘ └──────────────┘

The Wn numbers denote the worker processes that a particular job is run in. Deferring adds a deferred function and then runs for a long time. The deferred function will be present in the cache state for the duration of Deferred. Follow-on is the generic Job instance that’s added by encapsulating a job. It runs on the same worker node but in a separate worker process, as the first job in that worker. Because …

1) it is the first job in its worker process (the user script has not been made available on the sys.path by a previous job in that worker) and

2. it shares the cache state with the Deferring job and

3. it is an instance of Job (and so does not introduce the user script to sys.path itself),

… it might cause problems with deserializing a deferred function defined in the user script.

Encapsulated has two children to ensure that Follow-on is run in a separate worker.

testDeferralWithFailureAndEncapsulation()

Ensure that the following DAG succeeds:

              ┌───────────┐
              │ Root (W1) │
              └───────────┘
                    │
         ┌──────────┴─────────┐
         ▼                    ▼
┌────────────────┐ ┌────────────────────┐
│ Deferring (W2) │ │ Encapsulating (W3) │═══════════════════════╗
└────────────────┘ └────────────────────┘                       ║
                              │                                 ║
                              ▼                                 ▼
                    ┌───────────────────┐              ┌────────────────┐
                    │ Encapsulated (W3) │════════════╗ │ Follow-on (W7) │
                    └───────────────────┘            ║ └────────────────┘
                              │                      ║
                       ┌──────┴──────┐               ║
                       ▼             ▼               ▼
                ┌────────────┐┌────────────┐ ┌──────────────┐
                │ Dummy (W4) ││ Dummy (W5) │ │ Trigger (W6) │
                └────────────┘└────────────┘ └──────────────┘

Trigger causes Deferring to crash. Follow-on runs next, detects Deferring’s left-overs and runs the deferred function. Follow-on is an instance of Job and the first job in its worker process. This test ensures that despite these circumstances, the user script is loaded before the deferred functions defined in it are being run.

Encapsulated has two children to ensure that Follow-on is run in a new worker. That’s the only way to guarantee that the user script has not been loaded yet, which would cause the test to succeed coincidentally. We want to test that auto-deploying and loading of the user script are done properly before deferred functions are being run and before any jobs have been executed by that worker.