toil.test.batchSystems.batchSystemTest¶
Attributes¶
Classes¶
Class for testing batch system plugin functionality. |
|
Hide abstract base class from unittest's test case loader |
|
Tests against the Kubernetes batch system |
|
Kubernetes batch system unit tests that don't need to actually talk to a cluster. |
|
Tests against the AWS Batch batch system |
|
Tests against the Mesos batch system |
|
Tests against the single-machine batch system |
|
This test ensures that single machine batch system doesn't exceed the configured number |
|
Abstract class used to define the interface to a service. |
|
Tests against the GridEngine batch system |
|
Tests against the Slurm batch system |
|
Tests against the LSF batch system |
|
Tests against the Torque batch system |
|
Tests against the HTCondor batch system |
|
Tests Toil workflow against the SingleMachine batch system |
|
Tests Toil workflow against the Mesos batch system |
Functions¶
|
Dump a string into a temp file and return its path. |
|
|
|
|
|
|
|
|
|
Run in parallel to determine the number of concurrent tasks. |
|
Increments counter file and returns the max number of times the file |
|
|
|
|
Module Contents¶
- toil.test.batchSystems.batchSystemTest.logger¶
- toil.test.batchSystems.batchSystemTest.numCores = 2¶
- toil.test.batchSystems.batchSystemTest.preemptible = False¶
- toil.test.batchSystems.batchSystemTest.defaultRequirements¶
- class toil.test.batchSystems.batchSystemTest.BatchSystemPluginTest(methodName='runTest')[source]¶
Bases:
toil.test.ToilTest
Class for testing batch system plugin functionality.
Hide abstract base class from unittest’s test case loader
http://stackoverflow.com/questions/1323455/python-unit-test-with-base-and-sub-class#answer-25695512
Bases:
toil.test.ToilTest
A base test case with generic tests that every batch system should pass.
Cannot assume that the batch system actually executes commands on the local machine/filesystem.
Returns a dummy config for the batch system tests. We need a workflowID to be set up since we are running tests without setting up a jobstore. This is the class version to be used when an instance is not available.
- Return type:
Hook method for setting up class fixture before running tests in the class.
Hook method for setting up the test fixture before exercising it.
Hook method for deconstructing the test fixture after testing it.
Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up.
- Return type:
Test the mechanism for setting per-job environment variables to batch system jobs.
Bases:
toil.test.ToilTest
An abstract base class for batch system tests that use a full Toil workflow rather than using the batch system directly.
- Return type:
Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object
Hook method for setting up the test fixture before exercising it.
Hook method for deconstructing the test fixture after testing it.
Tests that the batch system is allocating core resources properly for concurrent tasks.
Test if the OMP_NUM_THREADS env var is set correctly based on jobs.cores.
Bases:
AbstractBatchSystemTest
An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch systems
- class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
Tests against the Kubernetes batch system
- class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemBenchTest(methodName='runTest')[source]¶
Bases:
toil.test.ToilTest
Kubernetes batch system unit tests that don’t need to actually talk to a cluster.
- class toil.test.batchSystems.batchSystemTest.AWSBatchBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
Tests against the AWS Batch batch system
- class toil.test.batchSystems.batchSystemTest.MesosBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
,toil.batchSystems.mesos.test.MesosTestSupport
Tests against the Mesos batch system
- toil.test.batchSystems.batchSystemTest.write_temp_file(s, temp_dir)[source]¶
Dump a string into a temp file and return its path.
- class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
Tests against the single-machine batch system
- class toil.test.batchSystems.batchSystemTest.MaxCoresSingleMachineBatchSystemTest(methodName='runTest')[source]¶
Bases:
toil.test.ToilTest
This test ensures that single machine batch system doesn’t exceed the configured number cores
- classmethod setUpClass()[source]¶
Hook method for setting up class fixture before running tests in the class.
- Return type:
None
- setUp()[source]¶
Hook method for setting up the test fixture before exercising it.
- Return type:
None
- class toil.test.batchSystems.batchSystemTest.Service(cmd)[source]¶
Bases:
toil.job.Job.Service
Abstract class used to define the interface to a service.
Should be subclassed by the user to define services.
Is not executed as a job; runs within a ServiceHostJob.
- cmd¶
- start(fileStore)[source]¶
Start the service.
- Parameters:
job – The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.
- Returns:
An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see
toil.job.Job.addService()
).
- check()[source]¶
Checks the service is still running.
- Raises:
exceptions.RuntimeError – If the service failed, this will cause the service job to be labeled failed.
- Returns:
True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!
- class toil.test.batchSystems.batchSystemTest.GridEngineBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
Tests against the GridEngine batch system
- class toil.test.batchSystems.batchSystemTest.SlurmBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
Tests against the Slurm batch system
- class toil.test.batchSystems.batchSystemTest.LSFBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
Tests against the LSF batch system
- class toil.test.batchSystems.batchSystemTest.TorqueBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
Tests against the Torque batch system
- class toil.test.batchSystems.batchSystemTest.HTCondorBatchSystemTest(methodName='runTest')[source]¶
Bases:
hidden
Tests against the HTCondor batch system
- class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemJobTest(methodName='runTest')[source]¶
Bases:
hidden
Tests Toil workflow against the SingleMachine batch system
- class toil.test.batchSystems.batchSystemTest.MesosBatchSystemJobTest(methodName='runTest')[source]¶
Bases:
hidden
,toil.batchSystems.mesos.test.MesosTestSupport
Tests Toil workflow against the Mesos batch system
- getOptions(tempDir)[source]¶
Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object
- toil.test.batchSystems.batchSystemTest.measureConcurrency(filepath, sleep_time=10)[source]¶
Run in parallel to determine the number of concurrent tasks. This code was copied from toil.batchSystemTestMaxCoresSingleMachineBatchSystemTest :param str filepath: path to counter file :param int sleep_time: number of seconds to sleep before counting down :return int max concurrency value: