toil.test.src.autoDeploymentTest

Module Contents

Classes

AutoDeploymentTest

Tests various auto-deployment scenarios. Using the appliance, i.e. a docker container,

Attributes

logger

toil.test.src.autoDeploymentTest.logger

class toil.test.src.autoDeploymentTest.AutoDeploymentTest(methodName='runTest')

Bases: toil.test.ApplianceTestSupport

Tests various auto-deployment scenarios. Using the appliance, i.e. a docker container, for these tests allows for running worker processes on the same node as the leader process while keeping their file systems separate from each other and the leader process. Separate file systems are crucial to prove that auto-deployment does its job.

sitePackages

setUp()

Hook method for setting up the test fixture before exercising it.

testRestart()

Test whether auto-deployment works on restart.

testSplitRootPackages()

Test whether auto-deployment works with a virtualenv in which jobs are defined in completely separate branches of the package hierarchy. Initially, auto-deployment did deploy the entire virtualenv but jobs could only be defined in one branch of the package hierarchy. We define a branch as the maximum set of fully qualified package paths that share the same first component. IOW, a.b and a.c are in the same branch, while a.b and d.c are not.

testUserTypesInJobFunctionArgs()

Test encapsulated, function-wrapping jobs where the function arguments reference user-defined types.

Mainly written to cover https://github.com/BD2KGenomics/toil/issues/1259 but then also revealed https://github.com/BD2KGenomics/toil/issues/1278.

testDeferralWithConcurrentEncapsulation()

Ensure that the following DAG succeeds:

              ┌───────────┐
              │ Root (W1) │
              └───────────┘
                    │
         ┌──────────┴─────────┐
         ▼                    ▼
┌────────────────┐ ┌────────────────────┐
│ Deferring (W2) │ │ Encapsulating (W3) │═══════════════╗
└────────────────┘ └────────────────────┘               ║
                              │                         ║
                              ▼                         ▼
                    ┌───────────────────┐      ┌────────────────┐
                    │ Encapsulated (W3) │      │ Follow-on (W6) │
                    └───────────────────┘      └────────────────┘
                              │                         │
                      ┌───────┴────────┐                │
                      ▼                ▼                ▼
              ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
              │ Dummy 1 (W4) │ │ Dummy 2 (W5) │ │  Last (W6)   │
              └──────────────┘ └──────────────┘ └──────────────┘

The Wn numbers denote the worker processes that a particular job is run in. Deferring adds a deferred function and then runs for a long time. The deferred function will be present in the cache state for the duration of Deferred. Follow-on is the generic Job instance that’s added by encapsulating a job. It runs on the same worker node but in a separate worker process, as the first job in that worker. Because …

1) it is the first job in its worker process (the user script has not been made available on the sys.path by a previous job in that worker) and

2. it shares the cache state with the Deferring job and

3. it is an instance of Job (and so does not introduce the user script to sys.path itself),

… it might cause problems with deserializing a defered function defined in the user script.

Encapsulated has two children to ensure that Follow-on is run in a separate worker.

testDeferralWithFailureAndEncapsulation()

Ensure that the following DAG succeeds:

              ┌───────────┐
              │ Root (W1) │
              └───────────┘
                    │
         ┌──────────┴─────────┐
         ▼                    ▼
┌────────────────┐ ┌────────────────────┐
│ Deferring (W2) │ │ Encapsulating (W3) │═══════════════════════╗
└────────────────┘ └────────────────────┘                       ║
                              │                                 ║
                              ▼                                 ▼
                    ┌───────────────────┐              ┌────────────────┐
                    │ Encapsulated (W3) │════════════╗ │ Follow-on (W7) │
                    └───────────────────┘            ║ └────────────────┘
                              │                      ║
                       ┌──────┴──────┐               ║
                       ▼             ▼               ▼
                ┌────────────┐┌────────────┐ ┌──────────────┐
                │ Dummy (W4) ││ Dummy (W5) │ │ Trigger (W6) │
                └────────────┘└────────────┘ └──────────────┘

Trigger causes Deferring to crash. Follow-on runs next, detects Deferring’s left-overs and runs the deferred function. Follow-on is an instance of Job and the first job in its worker process. This test ensures that despite these circumstances, the user script is loaded before the deferred functions defined in it are being run.

Encapsulated has two children to ensure that Follow-on is run in a new worker. That’s the only way to guarantee that the user script has not been loaded yet, which would cause the test to succeed coincidentally. We want to test that auto-deploying and loading of the user script are done properly before deferred functions are being run and before any jobs have been executed by that worker.