toil.statsAndLogging

Attributes

logger
root_logger
toil_logger
DEFAULT_LOGLEVEL

Classes

Expando	Pass initial attributes to the constructor:
ResourceMonitor	Global resource monitoring widget.
StatsAndLogging	A thread to aggregate statistics and logging.

Functions

strtobool(val)	Make a human-readable string into a bool.
set_log_level(level[, set_logger])	Sets the root logger level to a given string level (like "INFO").
install_log_color([set_logger])	Make logs colored.
add_logging_options(parser[, default_level])	Add logging options to set the global log level.
configure_root_logger()	Set up the root logger with handlers and formatting.
log_to_file(log_file, log_rotation)
set_logging_from_options(options)
suppress_exotic_logging(local_logger)	Attempts to suppress the loggers of all non-Toil packages by setting them to CRITICAL.

Module Contents

toil.statsAndLogging.strtobool(val)

Make a human-readable string into a bool.

Convert a string along the lines of “y”, “1”, “ON”, “TrUe”, or “Yes” to True, and the corresponding false-ish values to False.

Parameters:

val (str)

Return type:

bool

class toil.statsAndLogging.Expando(*args, **kwargs)

Bases: dict

Pass initial attributes to the constructor:

>>> o = Expando(foo=42)
>>> o.foo
42

Dynamically create new attributes:

>>> o.bar = 'hi'
>>> o.bar
'hi'

Expando is a dictionary:

>>> isinstance(o,dict)
True
>>> o['foo']
42

Works great with JSON:

>>> import json
>>> s='{"foo":42}'
>>> o = json.loads(s,object_hook=Expando)
>>> o.foo
42
>>> o.bar = 'hi'
>>> o.bar
'hi'

And since Expando is a dict, it serializes back to JSON just fine:

>>> json.dumps(o, sort_keys=True)
'{"bar": "hi", "foo": 42}'

Attributes can be deleted, too:

>>> o = Expando(foo=42)
>>> o.foo
42
>>> del o.foo
>>> o.foo
Traceback (most recent call last):
...
AttributeError: 'Expando' object has no attribute 'foo'
>>> o['foo']
Traceback (most recent call last):
...
KeyError: 'foo'

>>> del o.foo
Traceback (most recent call last):
...
AttributeError: foo

And copied:

>>> o = Expando(foo=42)
>>> p = o.copy()
>>> isinstance(p,Expando)
True
>>> o == p
True
>>> o is p
False

Same with MagicExpando …

>>> o = MagicExpando()
>>> o.foo.bar = 42
>>> p = o.copy()
>>> isinstance(p,MagicExpando)
True
>>> o == p
True
>>> o is p
False

… but the copy is shallow:

>>> o.foo is p.foo
True

copy()

D.copy() -> a shallow copy of D

class toil.statsAndLogging.ResourceMonitor

Global resource monitoring widget.

Presents class methods to get the resource usage of this process and child processes, and other class methods to adjust the statistics so they can account for e.g. resources used inside containers, or other resource usage that should be billable to the current process.

classmethod record_extra_memory(peak_ki)

Become responsible for the given peak memory usage, in kibibytes.

The memory will be treated as if it was used by a child process at the time our real child processes were also using their peak memory.

Parameters:

peak_ki (int)

Return type:

None

classmethod record_extra_cpu(seconds)

Become responsible for the given CPU time.

The CPU time will be treated as if it had been used by a child process.

Parameters:

seconds (float)

Return type:

None

classmethod get_total_cpu_time_and_memory_usage()

Gives the total cpu time of itself and all its children, and the maximum RSS memory usage of itself and its single largest child (in kibibytes).

Return type:

Tuple[float, int]

classmethod get_total_cpu_time()

Gives the total cpu time, including the children.

Return type:

float

toil.statsAndLogging.logger

toil.statsAndLogging.root_logger

toil.statsAndLogging.toil_logger

toil.statsAndLogging.DEFAULT_LOGLEVEL

class toil.statsAndLogging.StatsAndLogging(jobStore, config)

A thread to aggregate statistics and logging.

Parameters:

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

• config (toil.common.Config)

start()

Start the stats and logging thread.

Return type:

None

classmethod formatLogStream(stream, stream_name)

Given a stream of text or bytes, and the job name, job itself, or some other optional stringifyable identity info for the job, return a big text string with the formatted job log, suitable for printing for the user.

We don’t want to prefix every line of the job’s log with our own logging info, or we get prefixes wider than any reasonable terminal and longer than the messages.

Parameters:

• stream (Union[IO[str], IO[bytes]]) – The stream of text or bytes to print for the user.

• stream_name (str)

Return type:

str

classmethod logWithFormatting(stream_name, jobLogs, method=logger.debug, message=None)

Parameters:

• stream_name (str)

• jobLogs (Union[IO[str], IO[bytes]])

• method (Callable[[str], None])

• message (Optional[str])

Return type:

None

classmethod writeLogFiles(jobNames, jobLogList, config, failed=False)

Parameters:

• jobNames (List[str])

• jobLogList (List[str])

• config (toil.common.Config)

• failed (bool)

Return type:

None

classmethod statsAndLoggingAggregator(jobStore, stop, config)

The following function is used for collating stats/reporting log messages from the workers. Works inside of a thread, collates as long as the stop flag is not True.

Parameters:

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

• stop (threading.Event)

• config (toil.common.Config)

Return type:

None

check()

Check on the stats and logging aggregator. :raise RuntimeError: If the underlying thread has quit.

Return type:

None

shutdown()

Finish up the stats/logging aggregation thread.

Return type:

None

toil.statsAndLogging.set_log_level(level, set_logger=None)

Sets the root logger level to a given string level (like “INFO”).

Parameters:

• level (str)

• set_logger (Optional[logging.Logger])

Return type:

None

toil.statsAndLogging.install_log_color(set_logger=None)

Make logs colored.

Parameters:

set_logger (Optional[logging.Logger])

Return type:

None

toil.statsAndLogging.add_logging_options(parser, default_level=None)

Add logging options to set the global log level.

Parameters:

• default_level (Optional[int]) – A logging level, like logging.INFO, to use as the default.

• parser (argparse.ArgumentParser)

Return type:

None

toil.statsAndLogging.configure_root_logger()

Set up the root logger with handlers and formatting.

Should be called before any entry point tries to log anything, to ensure consistent formatting.

Return type:

None

toil.statsAndLogging.log_to_file(log_file, log_rotation)

Parameters:

• log_file (Optional[str])

• log_rotation (bool)

Return type:

None

toil.statsAndLogging.set_logging_from_options(options)

Parameters:

options (Union[toil.common.Config, argparse.Namespace])

Return type:

None

toil.statsAndLogging.suppress_exotic_logging(local_logger)

Attempts to suppress the loggers of all non-Toil packages by setting them to CRITICAL.

For example: ‘requests_oauthlib’, ‘google’, ‘boto’, ‘websocket’, ‘oauthlib’, etc.

This will only suppress loggers that have already been instantiated and can be seen in the environment, except for the list declared in “always_suppress”.

This is important because some packages, particularly boto3, are not always instantiated yet in the environment when this is run, and so we create the logger and set the level preemptively.

Parameters:

local_logger (str)

Return type:

None