toil.statsAndLogging

Module Contents

Classes

StatsAndLogging

A thread to aggregate statistics and logging.

Functions

set_log_level(level[, set_logger])	Sets the root logger level to a given string level (like "INFO").
install_log_color([set_logger])	Make logs colored.
add_logging_options(parser[, default_level])	Add logging options to set the global log level.
configure_root_logger()	Set up the root logger with handlers and formatting.
log_to_file(log_file, log_rotation)
set_logging_from_options(options)
suppress_exotic_logging(local_logger)	Attempts to suppress the loggers of all non-Toil packages by setting them to CRITICAL.

Attributes

logger
root_logger
toil_logger
DEFAULT_LOGLEVEL

toil.statsAndLogging.logger

toil.statsAndLogging.root_logger

toil.statsAndLogging.toil_logger

toil.statsAndLogging.DEFAULT_LOGLEVEL

class toil.statsAndLogging.StatsAndLogging(jobStore, config)

A thread to aggregate statistics and logging.

Parameters:

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

• config (toil.common.Config)

start()

Start the stats and logging thread.

Return type:

None

classmethod formatLogStream(stream, stream_name)

Given a stream of text or bytes, and the job name, job itself, or some other optional stringifyable identity info for the job, return a big text string with the formatted job log, suitable for printing for the user.

We don’t want to prefix every line of the job’s log with our own logging info, or we get prefixes wider than any reasonable terminal and longer than the messages.

Parameters:

• stream (Union[IO[str], IO[bytes]]) – The stream of text or bytes to print for the user.

• stream_name (str)

Return type:

str

classmethod logWithFormatting(stream_name, jobLogs, method=logger.debug, message=None)

Parameters:

• stream_name (str)

• jobLogs (Union[IO[str], IO[bytes]])

• method (Callable[[str], None])

• message (Optional[str])

Return type:

None

classmethod writeLogFiles(jobNames, jobLogList, config, failed=False)

Parameters:

• jobNames (List[str])

• jobLogList (List[str])

• config (toil.common.Config)

• failed (bool)

Return type:

None

classmethod statsAndLoggingAggregator(jobStore, stop, config)

The following function is used for collating stats/reporting log messages from the workers. Works inside of a thread, collates as long as the stop flag is not True.

Parameters:

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

• stop (threading.Event)

• config (toil.common.Config)

Return type:

None

check()

Check on the stats and logging aggregator. :raise RuntimeError: If the underlying thread has quit.

Return type:

None

shutdown()

Finish up the stats/logging aggregation thread.

Return type:

None

toil.statsAndLogging.set_log_level(level, set_logger=None)

Sets the root logger level to a given string level (like “INFO”).

Parameters:

• level (str)

• set_logger (Optional[logging.Logger])

Return type:

None

toil.statsAndLogging.install_log_color(set_logger=None)

Make logs colored.

Parameters:

set_logger (Optional[logging.Logger])

Return type:

None

toil.statsAndLogging.add_logging_options(parser, default_level=None)

Add logging options to set the global log level.

Parameters:

• default_level (Optional[int]) – A logging level, like logging.INFO, to use as the default.

• parser (argparse.ArgumentParser)

Return type:

None

toil.statsAndLogging.configure_root_logger()

Set up the root logger with handlers and formatting.

Should be called before any entry point tries to log anything, to ensure consistent formatting.

Return type:

None

toil.statsAndLogging.log_to_file(log_file, log_rotation)

Parameters:

• log_file (Optional[str])

• log_rotation (bool)

Return type:

None

toil.statsAndLogging.set_logging_from_options(options)

Parameters:

options (Union[toil.common.Config, argparse.Namespace])

Return type:

None

toil.statsAndLogging.suppress_exotic_logging(local_logger)

Attempts to suppress the loggers of all non-Toil packages by setting them to CRITICAL.

For example: ‘requests_oauthlib’, ‘google’, ‘boto’, ‘websocket’, ‘oauthlib’, etc.

This will only suppress loggers that have already been instantiated and can be seen in the environment, except for the list declared in “always_suppress”.

This is important because some packages, particularly boto3, are not always instantiated yet in the environment when this is run, and so we create the logger and set the level preemptively.

Parameters:

local_logger (str)

Return type:

None