toil.realtimeLogger

Implements a real-time UDP-based logging system that user scripts can use for debugging.

Module Contents

Classes

LoggingDatagramHandler	Receive logging messages from the jobs and display them on the leader.
JSONDatagramHandler	Send logging records over UDP serialized as JSON.
RealtimeLoggerMetaclass	Metaclass for RealtimeLogger that lets add logging methods.
RealtimeLogger	Provide a logger that logs over UDP to the leader.

Attributes

logger

toil.realtimeLogger.logger

class toil.realtimeLogger.LoggingDatagramHandler(request, client_address, server)

Bases: socketserver.BaseRequestHandler

digraph inheritance8da91132b4 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "BaseRequestHandler" [URL="https://docs.python.org/3/library/socketserver.html#socketserver.BaseRequestHandler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for request handler classes."]; "LoggingDatagramHandler" [URL="#toil.realtimeLogger.LoggingDatagramHandler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Receive logging messages from the jobs and display them on the leader."]; "BaseRequestHandler" -> "LoggingDatagramHandler" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Receive logging messages from the jobs and display them on the leader.

Uses bare JSON message encoding.

handle()

Handle a single message. SocketServer takes care of splitting out the messages.

Messages are JSON-encoded logging module records.

Return type:

None

class toil.realtimeLogger.JSONDatagramHandler(host, port)

Bases: logging.handlers.DatagramHandler

digraph inheritanceec941f9482 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "DatagramHandler" [URL="https://docs.python.org/3/library/logging.handlers.html#logging.handlers.DatagramHandler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A handler class which writes logging records, in pickle format, to"]; "SocketHandler" -> "DatagramHandler" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Filterer" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A base class for loggers and handlers which allows them to share"]; "Handler" [URL="https://docs.python.org/3/library/logging.html#logging.Handler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Handler instances dispatch logging events to specific destinations."]; "Filterer" -> "Handler" [arrowsize=0.5,style="setlinewidth(0.5)"]; "JSONDatagramHandler" [URL="#toil.realtimeLogger.JSONDatagramHandler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Send logging records over UDP serialized as JSON."]; "DatagramHandler" -> "JSONDatagramHandler" [arrowsize=0.5,style="setlinewidth(0.5)"]; "SocketHandler" [URL="https://docs.python.org/3/library/logging.handlers.html#logging.handlers.SocketHandler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A handler class which writes logging records, in pickle format, to"]; "Handler" -> "SocketHandler" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Send logging records over UDP serialized as JSON.

They have to fit in a single UDP datagram, so don’t try to log more than 64kb at once.

makePickle(record)

Actually, encode the record as bare JSON instead.

Parameters:

record (logging.LogRecord) –

Return type:

bytes

class toil.realtimeLogger.RealtimeLoggerMetaclass

Bases: type

digraph inheritancea8da0bf919 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "RealtimeLoggerMetaclass" [URL="#toil.realtimeLogger.RealtimeLoggerMetaclass",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Metaclass for RealtimeLogger that lets add logging methods."]; }

Metaclass for RealtimeLogger that lets add logging methods.

Like RealtimeLogger.warning(), RealtimeLogger.info(), etc.

__getattr__(name)

Fallback to attributes on the logger.

Parameters:

name (str) –

Return type:

Any

class toil.realtimeLogger.RealtimeLogger(batchSystem, level=defaultLevel)

Provide a logger that logs over UDP to the leader.

To use in a Toil job, do:

>>> from toil.realtimeLogger import RealtimeLogger
>>> RealtimeLogger.info("This logging message goes straight to the leader")

That’s all a user of Toil would need to do. On the leader, Job.Runner.startToil() automatically starts the UDP server by using an instance of this class as a context manager.

Parameters:

• batchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem) –

• level (str) –

envPrefix = 'TOIL_RT_LOGGING_'

defaultLevel = 'INFO'

lock

loggingServer

serverThread

initialized = 0

logger

classmethod getLogger()

Get the logger that logs real-time to the leader.

Note that if the returned logger is used on the leader, you will see the message twice, since it still goes to the normal log handlers, too.

Return type:

logging.Logger

__enter__()

Return type:

None

__exit__(exc_type, exc_val, exc_tb)

Parameters:

• exc_type (Optional[Type[BaseException]]) –

• exc_val (Optional[BaseException]) –

• exc_tb (Optional[types.TracebackType]) –

Return type:

None