toil.test.batchSystems.batchSystemTest

Module Contents

Classes

BatchSystemPluginTest	Class for testing batch system plugin functionality.
hidden	Hide abstract base class from unittest's test case loader
KubernetesBatchSystemTest	Tests against the Kubernetes batch system
KubernetesBatchSystemBenchTest	Kubernetes batch system unit tests that don't need to actually talk to a cluster.
AWSBatchBatchSystemTest	Tests against the AWS Batch batch system
MesosBatchSystemTest	Tests against the Mesos batch system
SingleMachineBatchSystemTest	Tests against the single-machine batch system
MaxCoresSingleMachineBatchSystemTest	This test ensures that single machine batch system doesn't exceed the configured number
Service	Abstract class used to define the interface to a service.
GridEngineBatchSystemTest	Tests against the GridEngine batch system
SlurmBatchSystemTest	Tests against the Slurm batch system
LSFBatchSystemTest	Tests against the LSF batch system
TorqueBatchSystemTest	Tests against the Torque batch system
HTCondorBatchSystemTest	Tests against the HTCondor batch system
SingleMachineBatchSystemJobTest	Tests Toil workflow against the SingleMachine batch system
MesosBatchSystemJobTest	Tests Toil workflow against the Mesos batch system

Functions

write_temp_file(s, temp_dir)	Dump a string into a temp file and return its path.
parentJob(job, cmd)
childJob(job, cmd)
grandChildJob(job, cmd)
greatGrandChild(cmd)
measureConcurrency(filepath[, sleep_time])	Run in parallel to determine the number of concurrent tasks.
count(delta, file_path)	Increments counter file and returns the max number of times the file
getCounters(path)
resetCounters(path)
get_omp_threads()

Attributes

logger
numCores
preemptible
defaultRequirements

toil.test.batchSystems.batchSystemTest.logger

toil.test.batchSystems.batchSystemTest.numCores = 2

toil.test.batchSystems.batchSystemTest.preemptible = False

toil.test.batchSystems.batchSystemTest.defaultRequirements

class toil.test.batchSystems.batchSystemTest.BatchSystemPluginTest(methodName='runTest')

Bases: toil.test.ToilTest

digraph inheritance2382fc7f17 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "BatchSystemPluginTest" [URL="#toil.test.batchSystems.batchSystemTest.BatchSystemPluginTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class for testing batch system plugin functionality."]; "ToilTest" -> "BatchSystemPluginTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Class for testing batch system plugin functionality.

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

test_add_batch_system_factory()

class toil.test.batchSystems.batchSystemTest.hidden

Hide abstract base class from unittest’s test case loader

http://stackoverflow.com/questions/1323455/python-unit-test-with-base-and-sub-class#answer-25695512

class AbstractBatchSystemTest(methodName='runTest')

Bases: toil.test.ToilTest

digraph inheritance5d75f67c20 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

A base test case with generic tests that every batch system should pass.

Cannot assume that the batch system actually executes commands on the local machine/filesystem.

abstract createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

supportsWallTime()

classmethod createConfig()

Returns a dummy config for the batch system tests. We need a workflowID to be set up since we are running tests without setting up a jobstore. This is the class version to be used when an instance is not available.

Return type:

toil.common.Config

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

get_max_startup_seconds()

Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up.

Return type:

int

test_available_cores()

test_run_jobs()

test_set_env()

test_set_job_env()

Test the mechanism for setting per-job environment variables to batch system jobs.

testCheckResourceRequest()

testScalableBatchSystem()

class AbstractBatchSystemJobTest(methodName='runTest')

Bases: toil.test.ToilTest

digraph inheritancef3a6ad94ec { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemJobTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemJobTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract base class for batch system tests that use a full Toil workflow rather"]; "ToilTest" -> "AbstractBatchSystemJobTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

An abstract base class for batch system tests that use a full Toil workflow rather than using the batch system directly.

cpuCount

allocatedCores

sleepTime = 5

abstract getBatchSystemName()

Return type:

(str, AbstractBatchSystem)

getOptions(tempDir)

Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

testJobConcurrency()

Tests that the batch system is allocating core resources properly for concurrent tasks.

test_omp_threads()

Test if the OMP_NUM_THREADS env var is set correctly based on jobs.cores.

class AbstractGridEngineBatchSystemTest(methodName='runTest')

Bases: hidden.AbstractBatchSystemTest

digraph inheritance0534adb5d2 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractGridEngineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch"]; "AbstractBatchSystemTest" -> "AbstractGridEngineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch systems

class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemTest(methodName='runTest')

Bases: hidden

digraph inheritancebfde620e6c { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "KubernetesBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the Kubernetes batch system"]; "AbstractBatchSystemTest" -> "KubernetesBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the Kubernetes batch system

supportsWallTime()

createBatchSystem()

class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemBenchTest(methodName='runTest')

Bases: toil.test.ToilTest

digraph inheritance0e807b3024 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "KubernetesBatchSystemBenchTest" [URL="#toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemBenchTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Kubernetes batch system unit tests that don't need to actually talk to a cluster."]; "ToilTest" -> "KubernetesBatchSystemBenchTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Kubernetes batch system unit tests that don’t need to actually talk to a cluster.

test_preemptability_constraints()

Make sure we generate the right preemptability constraints.

test_label_constraints()

Make sure we generate the right preemptability constraints.

class toil.test.batchSystems.batchSystemTest.AWSBatchBatchSystemTest(methodName='runTest')

Bases: hidden

digraph inheritancec30457f9e4 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AWSBatchBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.AWSBatchBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the AWS Batch batch system"]; "AbstractBatchSystemTest" -> "AWSBatchBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the AWS Batch batch system

supportsWallTime()

createBatchSystem()

get_max_startup_seconds()

Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up.

Return type:

int

class toil.test.batchSystems.batchSystemTest.MesosBatchSystemTest(methodName='runTest')

Bases: hidden, toil.batchSystems.mesos.test.MesosTestSupport

digraph inheritanced52a3594ce { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.MesosBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the Mesos batch system"]; "AbstractBatchSystemTest" -> "MesosBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosTestSupport" -> "MesosBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosTestSupport" [URL="../../../batchSystems/mesos/test/index.html#toil.batchSystems.mesos.test.MesosTestSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Mixin for test cases that need a running Mesos master and agent on the local host."]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the Mesos batch system

classmethod createConfig()

needs to set mesos_endpoint to localhost for testing since the default is now the private IP address

supportsWallTime()

createBatchSystem()

tearDown()

Hook method for deconstructing the test fixture after testing it.

testIgnoreNode()

toil.test.batchSystems.batchSystemTest.write_temp_file(s, temp_dir)

Dump a string into a temp file and return its path.

Parameters:

• s (str) –

• temp_dir (str) –

Return type:

str

class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemTest(methodName='runTest')

Bases: hidden

digraph inheritance1a759bd197 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SingleMachineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the single-machine batch system"]; "AbstractBatchSystemTest" -> "SingleMachineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the single-machine batch system

supportsWallTime()

Return type:

bool

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

testProcessEscape(hide=False)

Test to make sure that child processes and their descendants go away when the Toil workflow stops.

If hide is true, will try and hide the child processes to make them hard to stop.

Parameters:

hide (bool) –

Return type:

None

testHidingProcessEscape()

Test to make sure that child processes and their descendants go away when the Toil workflow stops, even if the job process stops and leaves children.

class toil.test.batchSystems.batchSystemTest.MaxCoresSingleMachineBatchSystemTest(methodName='runTest')

Bases: toil.test.ToilTest

digraph inheritance2f11684c09 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "MaxCoresSingleMachineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.MaxCoresSingleMachineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="This test ensures that single machine batch system doesn't exceed the configured number"]; "ToilTest" -> "MaxCoresSingleMachineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

This test ensures that single machine batch system doesn’t exceed the configured number cores

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

Return type:

None

setUp()

Hook method for setting up the test fixture before exercising it.

Return type:

None

tearDown()

Hook method for deconstructing the test fixture after testing it.

Return type:

None

scriptCommand()

Return type:

str

test()

testServices()

toil.test.batchSystems.batchSystemTest.parentJob(job, cmd)

toil.test.batchSystems.batchSystemTest.childJob(job, cmd)

toil.test.batchSystems.batchSystemTest.grandChildJob(job, cmd)

toil.test.batchSystems.batchSystemTest.greatGrandChild(cmd)

class toil.test.batchSystems.batchSystemTest.Service(cmd)

Bases: toil.job.Job.Service

digraph inheritanceaeb2cce49b { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Requirer" [URL="../../../job/index.html#toil.job.Requirer",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class implementing the storage and presentation of requirements."]; "Service" [URL="../../../job/index.html#toil.job.Job.Service",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Abstract class used to define the interface to a service."]; "Requirer" -> "Service" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Service" [URL="#toil.test.batchSystems.batchSystemTest.Service",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "Service" -> "Service" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Abstract class used to define the interface to a service.

Should be subclassed by the user to define services.

Is not executed as a job; runs within a ServiceHostJob.

start(fileStore)

Start the service.

Parameters:

job – The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

Returns:

An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService()).

check()

Checks the service is still running.

Raises:

exceptions.RuntimeError – If the service failed, this will cause the service job to be labeled failed.

Returns:

True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

stop(fileStore)

Stops the service. Function can block until complete.

Parameters:

job – The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

class toil.test.batchSystems.batchSystemTest.GridEngineBatchSystemTest(methodName='runTest')

Bases: hidden

digraph inheritanceae6728af28 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractGridEngineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch"]; "AbstractBatchSystemTest" -> "AbstractGridEngineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "GridEngineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.GridEngineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the GridEngine batch system"]; "AbstractGridEngineBatchSystemTest" -> "GridEngineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the GridEngine batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown()

Hook method for deconstructing the test fixture after testing it.

class toil.test.batchSystems.batchSystemTest.SlurmBatchSystemTest(methodName='runTest')

Bases: hidden

digraph inheritance78834bde98 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractGridEngineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch"]; "AbstractBatchSystemTest" -> "AbstractGridEngineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SlurmBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.SlurmBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the Slurm batch system"]; "AbstractGridEngineBatchSystemTest" -> "SlurmBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the Slurm batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown()

Hook method for deconstructing the test fixture after testing it.

class toil.test.batchSystems.batchSystemTest.LSFBatchSystemTest(methodName='runTest')

Bases: hidden

digraph inheritance2419e34180 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractGridEngineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch"]; "AbstractBatchSystemTest" -> "AbstractGridEngineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "LSFBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.LSFBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the LSF batch system"]; "AbstractGridEngineBatchSystemTest" -> "LSFBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the LSF batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

class toil.test.batchSystems.batchSystemTest.TorqueBatchSystemTest(methodName='runTest')

Bases: hidden

digraph inheritance787418e1f9 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractGridEngineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch"]; "AbstractBatchSystemTest" -> "AbstractGridEngineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TorqueBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.TorqueBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the Torque batch system"]; "AbstractGridEngineBatchSystemTest" -> "TorqueBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the Torque batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown()

Hook method for deconstructing the test fixture after testing it.

class toil.test.batchSystems.batchSystemTest.HTCondorBatchSystemTest(methodName='runTest')

Bases: hidden

digraph inheritance365d7cf8b8 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A base test case with generic tests that every batch system should pass."]; "ToilTest" -> "AbstractBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractGridEngineBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch"]; "AbstractBatchSystemTest" -> "AbstractGridEngineBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "HTCondorBatchSystemTest" [URL="#toil.test.batchSystems.batchSystemTest.HTCondorBatchSystemTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests against the HTCondor batch system"]; "AbstractGridEngineBatchSystemTest" -> "HTCondorBatchSystemTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests against the HTCondor batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown()

Hook method for deconstructing the test fixture after testing it.

class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemJobTest(methodName='runTest')

Bases: hidden

digraph inheritanceeb1b8c864f { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemJobTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemJobTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract base class for batch system tests that use a full Toil workflow rather"]; "ToilTest" -> "AbstractBatchSystemJobTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SingleMachineBatchSystemJobTest" [URL="#toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemJobTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests Toil workflow against the SingleMachine batch system"]; "AbstractBatchSystemJobTest" -> "SingleMachineBatchSystemJobTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests Toil workflow against the SingleMachine batch system

getBatchSystemName()

Return type:

(str, AbstractBatchSystem)

testConcurrencyWithDisk()

Tests that the batch system is allocating disk resources properly

testNestedResourcesDoNotBlock()

Resources are requested in the order Memory > Cpu > Disk. Test that unavailability of cpus for one job that is scheduled does not block another job that can run.

class toil.test.batchSystems.batchSystemTest.MesosBatchSystemJobTest(methodName='runTest')

Bases: hidden, toil.batchSystems.mesos.test.MesosTestSupport

digraph inheritance886ceed052 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AbstractBatchSystemJobTest" [URL="#toil.test.batchSystems.batchSystemTest.hidden.AbstractBatchSystemJobTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract base class for batch system tests that use a full Toil workflow rather"]; "ToilTest" -> "AbstractBatchSystemJobTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosBatchSystemJobTest" [URL="#toil.test.batchSystems.batchSystemTest.MesosBatchSystemJobTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Tests Toil workflow against the Mesos batch system"]; "AbstractBatchSystemJobTest" -> "MesosBatchSystemJobTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosTestSupport" -> "MesosBatchSystemJobTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosTestSupport" [URL="../../../batchSystems/mesos/test/index.html#toil.batchSystems.mesos.test.MesosTestSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Mixin for test cases that need a running Mesos master and agent on the local host."]; "TestCase" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class whose instances are single test cases."]; "ToilTest" [URL="../../index.html#toil.test.ToilTest",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A common base class for Toil tests."]; "TestCase" -> "ToilTest" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Tests Toil workflow against the Mesos batch system

getOptions(tempDir)

Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object

getBatchSystemName()

Return type:

(str, AbstractBatchSystem)

tearDown()

Hook method for deconstructing the test fixture after testing it.

toil.test.batchSystems.batchSystemTest.measureConcurrency(filepath, sleep_time=10)

Run in parallel to determine the number of concurrent tasks. This code was copied from toil.batchSystemTestMaxCoresSingleMachineBatchSystemTest :param str filepath: path to counter file :param int sleep_time: number of seconds to sleep before counting down :return int max concurrency value:

toil.test.batchSystems.batchSystemTest.count(delta, file_path)

Increments counter file and returns the max number of times the file has been modified. Counter data must be in the form: concurrent tasks, max concurrent tasks (counter should be initialized to 0,0)

Parameters:

• delta (int) – increment value

• file_path (str) – path to shared counter file

Return int max concurrent tasks:

toil.test.batchSystems.batchSystemTest.getCounters(path)

toil.test.batchSystems.batchSystemTest.resetCounters(path)

toil.test.batchSystems.batchSystemTest.get_omp_threads()

Return type:

str