toil.test.src.jobTest

Classes

TestJob

Tests the job class.

TrivialService

Abstract class used to define the interface to a service.

Functions

simpleJobFn(job, value)

fn1Test(string, outputFile)

Function appends the next character after the last character in the given

fn2Test(pStrings, s, outputFile)

Function concatenates the strings in pStrings and s, in that order, and writes the result to

trivialParent(job)

parent(job)

diamond(job)

child(job)

errorChild(job)

Module Contents

class toil.test.src.jobTest.TestJob

Tests the job class.

testStatic(tmp_path)

Create a DAG of jobs non-dynamically and run it. DAG is:

A -> F
\-------
B -> D  \
 \       \
  ------- C -> E

Follow on is marked by ->

Parameters:

tmp_path (pathlib.Path)

Return type:

None

testStatic2(tmp_path)

Create a DAG of jobs non-dynamically and run it. DAG is:

A -> F
\-------
B -> D  \
 \       \
  ------- C -> E

Follow on is marked by ->

Parameters:

tmp_path (pathlib.Path)

Return type:

None

testTrivialDAGConsistency(tmp_path)
Parameters:

tmp_path (pathlib.Path)

Return type:

None

testDAGConsistency(tmp_path)
Parameters:

tmp_path (pathlib.Path)

Return type:

None

testSiblingDAGConsistency(tmp_path)

Slightly more complex case. The stranded job’s predecessors are siblings instead of parent/child.

Parameters:

tmp_path (pathlib.Path)

Return type:

None

testDeadlockDetection()

Randomly generate job graphs with various types of cycle in them and check they cause an exception properly. Also check that multiple roots causes a deadlock exception.

Return type:

None

testNewCheckpointIsLeafVertexNonRootCase(tmp_path, subtests)

Test for issue #1465: Detection of checkpoint jobs that are not leaf vertices identifies leaf vertices incorrectly

Test verification of new checkpoint jobs being leaf vertices, starting with the following baseline workflow:

Parent
  |
Child # Checkpoint=True
Parameters:

  • tmp_path (pathlib.Path)

  • subtests (pytest_subtests.SubTests)

Return type:

None

testNewCheckpointIsLeafVertexRootCase(tmp_path, subtests)
Test for issue #1466: Detection of checkpoint jobs that are not leaf vertices

omits the workflow root job

Test verification of a new checkpoint job being leaf vertex, starting with a baseline workflow of a single, root job:

Root # Checkpoint=True
Parameters:

  • tmp_path (pathlib.Path)

  • subtests (pytest_subtests.SubTests)

Return type:

None

runNewCheckpointIsLeafVertexTest(tmp_path, subtests, createWorkflowFn)

Test verification that a checkpoint job is a leaf vertex using both valid and invalid cases.

Parameters:
Return type:

None

runCheckpointVertexTest(workflowRootJob, checkpointJob, tmp_path, checkpointJobService=None, checkpointJobChild=None, checkpointJobFollowOn=None, expectedException=None)

Modifies the checkpoint job according to the given parameters then runs the workflow, checking for the expected exception, if any.

Parameters:
Return type:

None

testEvaluatingRandomDAG(tmp_path)

Randomly generate test input then check that the job graph can be run successfully, using the existence of promises to validate the run.

Parameters:

tmp_path (pathlib.Path)

Return type:

None

static getRandomEdge(nodeNumber)
Parameters:

nodeNumber (int)

Return type:

tuple[int, int]

static makeRandomDAG(nodeNumber)

Makes a random dag with “nodeNumber” nodes in which all nodes are connected. Return value is list of edges, each of form (a, b), where a and b are integers >= 0 < nodeNumber referring to nodes and the edge is from a to b.

Parameters:

nodeNumber (int)

Return type:

set[tuple[int, int]]

static getAdjacencyList(nodeNumber, edges)

Make adjacency list representation of edges

Parameters:
Return type:

list[set[int]]

reachable(node, adjacencyList)

Find the set of nodes reachable from this node (including the node). Return is a set of integers.

Parameters:
Return type:

set[int]

addRandomFollowOnEdges(childAdjacencyList)

Adds random follow on edges to the graph, represented as an adjacency list. The follow on edges are returned as a set and their augmented edges are added to the adjacency list.

Parameters:

childAdjacencyList (list[set[int]])

Return type:

set[tuple[int, int]]

makeJobGraph(nodeNumber, childEdges, followOnEdges, outPath, addServices=True)

Converts a DAG into a job graph. childEdges and followOnEdges are the lists of child and followOn edges.

Parameters:
Return type:

toil.job.Job

isAcyclic(adjacencyList)

Returns true if there are any cycles in the graph, which is represented as an adjacency list.

Parameters:

adjacencyList (list[set[int]])

Return type:

bool

toil.test.src.jobTest.simpleJobFn(job, value)
Parameters:
Return type:

None

toil.test.src.jobTest.fn1Test(string, outputFile)

Function appends the next character after the last character in the given string to the string, writes the string to a file, and returns it. For example, if string is “AB”, we will write and return “ABC”.

Parameters:

  • string (str)

  • outputFile (toil.lib.misc.FileDescriptorOrPath)

Return type:

str

toil.test.src.jobTest.fn2Test(pStrings, s, outputFile)

Function concatenates the strings in pStrings and s, in that order, and writes the result to the output file. Returns s.

Parameters:
Return type:

str

toil.test.src.jobTest.trivialParent(job)
Parameters:

job (toil.job.Job)

Return type:

None

toil.test.src.jobTest.parent(job)
Parameters:

job (toil.job.Job)

Return type:

None

toil.test.src.jobTest.diamond(job)
Parameters:

job (toil.job.Job)

Return type:

None

toil.test.src.jobTest.child(job)
Parameters:

job (toil.job.Job)

Return type:

None

toil.test.src.jobTest.errorChild(job)
Parameters:

job (toil.job.Job)

Return type:

NoReturn

class toil.test.src.jobTest.TrivialService(message, *args, **kwargs)

Bases: toil.job.Job.Service

Abstract class used to define the interface to a service.

Should be subclassed by the user to define services.

Is not executed as a job; runs within a ServiceHostJob.

Parameters:

  • message (str)

  • args (Any)

  • kwargs (Any)

message
start(job)

Start the service.

Parameters:

job (toil.job.ServiceHostJob) – The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

Returns:

An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService()).

Return type:

str

stop(job)

Stops the service. Function can block until complete.

Parameters:

job (toil.job.ServiceHostJob) – The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

Return type:

None

check()

Checks the service is still running.

Raises:

exceptions.RuntimeError – If the service failed, this will cause the service job to be labeled failed.

Returns:

True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

Return type:

bool