toil.batchSystems.mesos.test

Package Contents

Classes

ExceptionalThread	A thread whose join() method re-raises exceptions raised during run(). While join() is
MesosTestSupport	Mixin for test cases that need a running Mesos master and agent on the local host.

Functions

retry([intervals, infinite_retries, errors, ...])	Retry a function if it fails with any Exception defined in "errors".
cpu_count()	Get the rounded-up integer number of whole CPUs available.

Attributes

log

toil.batchSystems.mesos.test.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)

Retry a function if it fails with any Exception defined in “errors”.

Does so every x seconds, where x is defined by a list of numbers (ints or floats) in “intervals”. Also accepts ErrorCondition events for more detailed retry attempts.

Parameters

• intervals (Optional[List]) – A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s

• infinite_retries (bool) – If this is True, reset the intervals when they run out. Defaults to: False.

• errors (Optional[Sequence[Union[ErrorCondition, Type[Exception]]]]) –

  A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: - Exception (required) - Error codes that must match to be retried (optional; defaults to not checking) - A string that must be in the error message to be retried (optional; defaults to not checking) - A bool that can be set to False to always error on this condition.

  If not specified, this will default to a generic Exception.

• log_message (Optional[Tuple[Callable, str]]) – Optional tuple of (“log/print function()”, “message string”) that will precede each attempt.

• prepare (Optional[List[Callable]]) – Optional list of functions to call, with the function’s arguments, between retries, to reset state.

Returns

The result of the wrapped function or raise.

Return type

Callable[[Any], Any]

class toil.batchSystems.mesos.test.ExceptionalThread(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None)

Bases: threading.Thread

digraph inheritance46209316ca { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ExceptionalThread" [URL="../../../lib/threading/index.html#toil.lib.threading.ExceptionalThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A thread whose join() method re-raises exceptions raised during run(). While join() is"]; "Thread" -> "ExceptionalThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Thread" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that represents a thread of control."]; }

A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run().

When subclassing this thread, override tryRun() instead of run().

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

exc_info

run()

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

Return type

None

tryRun()

Return type

None

join(*args, **kwargs)

Wait until the thread terminates.

This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception or until the optional timeout occurs.

When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened – if the thread is still alive, the join() call timed out.

When the timeout argument is not present or None, the operation will block until the thread terminates.

A thread can be join()ed many times.

join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception.

Parameters

• args (Optional[float]) –

• kwargs (Optional[float]) –

Return type

None

toil.batchSystems.mesos.test.cpu_count()

Get the rounded-up integer number of whole CPUs available.

Counts hyperthreads as CPUs.

Uses the system’s actual CPU count, or the current v1 cgroup’s quota per period, if the quota is set.

Ignores the cgroup’s cpu shares value, because it’s extremely difficult to interpret. See https://github.com/kubernetes/kubernetes/issues/81021.

Caches result for efficiency.

Returns

Integer count of available CPUs, minimum 1.

Return type

int

toil.batchSystems.mesos.test.log

class toil.batchSystems.mesos.test.MesosTestSupport

Mixin for test cases that need a running Mesos master and agent on the local host.

class MesosThread(numCores)

Bases: toil.lib.threading.ExceptionalThread

digraph inheritancee4cc322d0b { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ExceptionalThread" [URL="../../../lib/threading/index.html#toil.lib.threading.ExceptionalThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A thread whose join() method re-raises exceptions raised during run(). While join() is"]; "Thread" -> "ExceptionalThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosThread" [URL="#toil.batchSystems.mesos.test.MesosTestSupport.MesosThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "ExceptionalThread" -> "MesosThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Thread" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that represents a thread of control."]; }

A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run().

When subclassing this thread, override tryRun() instead of run().

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

lock

abstract mesosCommand()

tryRun()

findMesosBinary(names)

class MesosMasterThread(numCores)

Bases: MesosTestSupport.MesosThread

digraph inheritanceefe7f9ad07 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ExceptionalThread" [URL="../../../lib/threading/index.html#toil.lib.threading.ExceptionalThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A thread whose join() method re-raises exceptions raised during run(). While join() is"]; "Thread" -> "ExceptionalThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosMasterThread" [URL="#toil.batchSystems.mesos.test.MesosTestSupport.MesosMasterThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "MesosThread" -> "MesosMasterThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosThread" [URL="#toil.batchSystems.mesos.test.MesosTestSupport.MesosThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "ExceptionalThread" -> "MesosThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Thread" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that represents a thread of control."]; }

A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run().

When subclassing this thread, override tryRun() instead of run().

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

mesosCommand()

class MesosAgentThread(numCores)

Bases: MesosTestSupport.MesosThread

digraph inheritancecdf82395b7 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ExceptionalThread" [URL="../../../lib/threading/index.html#toil.lib.threading.ExceptionalThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A thread whose join() method re-raises exceptions raised during run(). While join() is"]; "Thread" -> "ExceptionalThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosAgentThread" [URL="#toil.batchSystems.mesos.test.MesosTestSupport.MesosAgentThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "MesosThread" -> "MesosAgentThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "MesosThread" [URL="#toil.batchSystems.mesos.test.MesosTestSupport.MesosThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "ExceptionalThread" -> "MesosThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Thread" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that represents a thread of control."]; }

A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occured in run().

When subclassing this thread, override tryRun() instead of run().

>>> def f():
...     assert 0
>>> t = ExceptionalThread(target=f)
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

>>> class MyThread(ExceptionalThread):
...     def tryRun( self ):
...         assert 0
>>> t = MyThread()
>>> t.start()
>>> t.join()
Traceback (most recent call last):
...
AssertionError

mesosCommand()

wait_for_master()