toil.test.batchSystems.batchSystemTest

Attributes

logger
numCores
preemptible
defaultRequirements

Classes

BatchSystemPluginTest	Class for testing batch system plugin functionality.
hidden	Hide abstract base class from unittest's test case loader
KubernetesBatchSystemTest	Tests against the Kubernetes batch system
KubernetesBatchSystemBenchTest	Kubernetes batch system unit tests that don't need to actually talk to a cluster.
AWSBatchBatchSystemTest	Tests against the AWS Batch batch system
MesosBatchSystemTest	Tests against the Mesos batch system
SingleMachineBatchSystemTest	Tests against the single-machine batch system
MaxCoresSingleMachineBatchSystemTest	This test ensures that single machine batch system doesn't exceed the configured number
Service	Abstract class used to define the interface to a service.
GridEngineBatchSystemTest	Tests against the GridEngine batch system
SlurmBatchSystemTest	Tests against the Slurm batch system
LSFBatchSystemTest	Tests against the LSF batch system
TorqueBatchSystemTest	Tests against the Torque batch system
HTCondorBatchSystemTest	Tests against the HTCondor batch system
SingleMachineBatchSystemJobTest	Tests Toil workflow against the SingleMachine batch system
MesosBatchSystemJobTest	Tests Toil workflow against the Mesos batch system

Functions

write_temp_file(s, temp_dir)	Dump a string into a temp file and return its path.
parentJob(job, cmd)
childJob(job, cmd)
grandChildJob(job, cmd)
greatGrandChild(cmd)
measureConcurrency(filepath[, sleep_time])	Run in parallel to determine the number of concurrent tasks.
count(delta, file_path)	Increments counter file and returns the max number of times the file
getCounters(path)
resetCounters(path)
get_omp_threads()

Module Contents

toil.test.batchSystems.batchSystemTest.logger

toil.test.batchSystems.batchSystemTest.numCores = 2

toil.test.batchSystems.batchSystemTest.preemptible = False

toil.test.batchSystems.batchSystemTest.defaultRequirements

class toil.test.batchSystems.batchSystemTest.BatchSystemPluginTest(methodName='runTest')

Bases: toil.test.ToilTest

Class for testing batch system plugin functionality.

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

test_add_batch_system_factory()

class toil.test.batchSystems.batchSystemTest.hidden

Hide abstract base class from unittest’s test case loader

http://stackoverflow.com/questions/1323455/python-unit-test-with-base-and-sub-class#answer-25695512

class AbstractBatchSystemTest(methodName='runTest')

Bases: toil.test.ToilTest

A base test case with generic tests that every batch system should pass.

Cannot assume that the batch system actually executes commands on the local machine/filesystem.

abstract createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

supportsWallTime()

classmethod createConfig()

Returns a dummy config for the batch system tests. We need a workflowID to be set up since we are running tests without setting up a jobstore. This is the class version to be used when an instance is not available.

Return type:

toil.common.Config

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

get_max_startup_seconds()

Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up.

Return type:

int

test_available_cores()

test_run_jobs()

test_set_env()

test_set_job_env()

Test the mechanism for setting per-job environment variables to batch system jobs.

testCheckResourceRequest()

testScalableBatchSystem()

class AbstractBatchSystemJobTest(methodName='runTest')

Bases: toil.test.ToilTest

An abstract base class for batch system tests that use a full Toil workflow rather than using the batch system directly.

cpuCount

allocatedCores

sleepTime = 30

abstract getBatchSystemName()

Return type:

(str, AbstractBatchSystem)

getOptions(tempDir)

Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

testJobConcurrency()

Tests that the batch system is allocating core resources properly for concurrent tasks.

test_omp_threads()

Test if the OMP_NUM_THREADS env var is set correctly based on jobs.cores.

class AbstractGridEngineBatchSystemTest(methodName='runTest')

Bases: AbstractBatchSystemTest

An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch systems

class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemTest(methodName='runTest')

Bases: hidden

Tests against the Kubernetes batch system

supportsWallTime()

createBatchSystem()

class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemBenchTest(methodName='runTest')

Bases: toil.test.ToilTest

Kubernetes batch system unit tests that don’t need to actually talk to a cluster.

test_preemptability_constraints()

Make sure we generate the right preemptability constraints.

test_label_constraints()

Make sure we generate the right preemptability constraints.

class toil.test.batchSystems.batchSystemTest.AWSBatchBatchSystemTest(methodName='runTest')

Bases: hidden

Tests against the AWS Batch batch system

supportsWallTime()

createBatchSystem()

get_max_startup_seconds()

Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up.

Return type:

int

class toil.test.batchSystems.batchSystemTest.MesosBatchSystemTest(methodName='runTest')

Bases: hidden, toil.batchSystems.mesos.test.MesosTestSupport

Tests against the Mesos batch system

classmethod createConfig()

needs to set mesos_endpoint to localhost for testing since the default is now the private IP address

supportsWallTime()

createBatchSystem()

tearDown()

Hook method for deconstructing the test fixture after testing it.

testIgnoreNode()

toil.test.batchSystems.batchSystemTest.write_temp_file(s, temp_dir)

Dump a string into a temp file and return its path.

Parameters:

• s (str)

• temp_dir (str)

Return type:

str

class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemTest(methodName='runTest')

Bases: hidden

Tests against the single-machine batch system

supportsWallTime()

Return type:

bool

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

testProcessEscape(hide=False)

Test to make sure that child processes and their descendants go away when the Toil workflow stops.

If hide is true, will try and hide the child processes to make them hard to stop.

Parameters:

hide (bool)

Return type:

None

testHidingProcessEscape()

Test to make sure that child processes and their descendants go away when the Toil workflow stops, even if the job process stops and leaves children.

class toil.test.batchSystems.batchSystemTest.MaxCoresSingleMachineBatchSystemTest(methodName='runTest')

Bases: toil.test.ToilTest

This test ensures that single machine batch system doesn’t exceed the configured number cores

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

Return type:

None

setUp()

Hook method for setting up the test fixture before exercising it.

Return type:

None

tearDown()

Hook method for deconstructing the test fixture after testing it.

Return type:

None

scriptCommand()

Return type:

str

test()

testServices()

toil.test.batchSystems.batchSystemTest.parentJob(job, cmd)

toil.test.batchSystems.batchSystemTest.childJob(job, cmd)

toil.test.batchSystems.batchSystemTest.grandChildJob(job, cmd)

toil.test.batchSystems.batchSystemTest.greatGrandChild(cmd)

class toil.test.batchSystems.batchSystemTest.Service(cmd)

Bases: toil.job.Job.Service

Abstract class used to define the interface to a service.

Should be subclassed by the user to define services.

Is not executed as a job; runs within a ServiceHostJob.

cmd

start(fileStore)

Start the service.

Parameters:

job – The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

Returns:

An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService()).

check()

Checks the service is still running.

Raises:

exceptions.RuntimeError – If the service failed, this will cause the service job to be labeled failed.

Returns:

True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

stop(fileStore)

Stops the service. Function can block until complete.

Parameters:

job – The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

class toil.test.batchSystems.batchSystemTest.GridEngineBatchSystemTest(methodName='runTest')

Bases: hidden

Tests against the GridEngine batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown()

Hook method for deconstructing the test fixture after testing it.

class toil.test.batchSystems.batchSystemTest.SlurmBatchSystemTest(methodName='runTest')

Bases: hidden

Tests against the Slurm batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown()

Hook method for deconstructing the test fixture after testing it.

class toil.test.batchSystems.batchSystemTest.LSFBatchSystemTest(methodName='runTest')

Bases: hidden

Tests against the LSF batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

class toil.test.batchSystems.batchSystemTest.TorqueBatchSystemTest(methodName='runTest')

Bases: hidden

Tests against the Torque batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown()

Hook method for deconstructing the test fixture after testing it.

class toil.test.batchSystems.batchSystemTest.HTCondorBatchSystemTest(methodName='runTest')

Bases: hidden

Tests against the HTCondor batch system

createBatchSystem()

Return type:

toil.batchSystems.abstractBatchSystem.AbstractBatchSystem

tearDown()

Hook method for deconstructing the test fixture after testing it.

class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemJobTest(methodName='runTest')

Bases: hidden

Tests Toil workflow against the SingleMachine batch system

getBatchSystemName()

Return type:

(str, AbstractBatchSystem)

testConcurrencyWithDisk()

Tests that the batch system is allocating disk resources properly

testNestedResourcesDoNotBlock()

Resources are requested in the order Memory > Cpu > Disk. Test that unavailability of cpus for one job that is scheduled does not block another job that can run.

class toil.test.batchSystems.batchSystemTest.MesosBatchSystemJobTest(methodName='runTest')

Bases: hidden, toil.batchSystems.mesos.test.MesosTestSupport

Tests Toil workflow against the Mesos batch system

getOptions(tempDir)

Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object

getBatchSystemName()

Return type:

(str, AbstractBatchSystem)

tearDown()

Hook method for deconstructing the test fixture after testing it.

toil.test.batchSystems.batchSystemTest.measureConcurrency(filepath, sleep_time=10)

Run in parallel to determine the number of concurrent tasks. This code was copied from toil.batchSystemTestMaxCoresSingleMachineBatchSystemTest :param str filepath: path to counter file :param int sleep_time: number of seconds to sleep before counting down :return int max concurrency value:

toil.test.batchSystems.batchSystemTest.count(delta, file_path)

Increments counter file and returns the max number of times the file has been modified. Counter data must be in the form: concurrent tasks, max concurrent tasks (counter should be initialized to 0,0)

Parameters:

• delta (int) – increment value

• file_path (str) – path to shared counter file

Return int max concurrent tasks:

toil.test.batchSystems.batchSystemTest.getCounters(path)

toil.test.batchSystems.batchSystemTest.resetCounters(path)

toil.test.batchSystems.batchSystemTest.get_omp_threads()

Return type:

str