toil.test.src.resourceTest

Attributes

exactPython

Exceptions

ResourceException

Common base class for all non-exit exceptions.

Classes

ModuleDescriptor	A path to a Python module decomposed into a namedtuple of three elements
Resource	Represents a file or directory that will be deployed to each node before any jobs in the user script are invoked.
ToilTest	A common base class for Toil tests.
ResourceTest	Test module descriptors and resources derived from them.

Functions

inVirtualEnv()	Test if we are inside a virtualenv or Conda virtual environment.
tempFileContaining(content[, suffix])	Write a file with the given contents, and keep it on disk as long as the context is active.

Module Contents

toil.test.src.resourceTest.inVirtualEnv()

Test if we are inside a virtualenv or Conda virtual environment.

Return type:

bool

class toil.test.src.resourceTest.ModuleDescriptor

Bases: namedtuple('ModuleDescriptor', ('dirPath', 'name', 'fromVirtualEnv'))

A path to a Python module decomposed into a namedtuple of three elements

• dirPath, the path to the directory that should be added to sys.path before importing the module,

• moduleName, the fully qualified name of the module with leading package names separated by dot and

>>> import toil.resource
>>> ModuleDescriptor.forModule('toil.resource') 
ModuleDescriptor(dirPath='/.../src', name='toil.resource', fromVirtualEnv=False)

>>> import subprocess, tempfile, os
>>> dirPath = tempfile.mkdtemp()
>>> path = os.path.join( dirPath, 'foo.py' )
>>> with open(path,'w') as f:
...     _ = f.write('from toil.resource import ModuleDescriptor\n'
...                 'print(ModuleDescriptor.forModule(__name__))')
>>> subprocess.check_output([ sys.executable, path ]) 
b"ModuleDescriptor(dirPath='...', name='foo', fromVirtualEnv=False)\n"

>>> from shutil import rmtree
>>> rmtree( dirPath )

Now test a collision. ‘collections’ is part of the standard library in Python 2 and 3. >>> dirPath = tempfile.mkdtemp() >>> path = os.path.join( dirPath, ‘collections.py’ ) >>> with open(path,’w’) as f: … _ = f.write(‘from toil.resource import ModuleDescriptorn’ … ‘ModuleDescriptor.forModule(__name__)’)

This should fail and return exit status 1 due to the collision with the built-in module: >>> subprocess.call([ sys.executable, path ]) 1

Clean up >>> rmtree( dirPath )

dirPath: str

name: str

classmethod forModule(name)

Return an instance of this class representing the module of the given name.

If the given module name is “__main__”, it will be translated to the actual file name of the top-level script without the .py or .pyc extension. This method assumes that the module with the specified name has already been loaded.

Parameters:

name (str)

Return type:

ModuleDescriptor

property belongsToToil: bool

True if this module is part of the Toil distribution

Return type:

bool

saveAsResourceTo(jobStore)

Store the file containing this module–or even the Python package directory hierarchy containing that file–as a resource to the given job store and return the corresponding resource object. Should only be called on a leader node.

Parameters:

jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type:

Resource

localize()

Check if this module was saved as a resource.

If it was, return a new module descriptor that points to a local copy of that resource. Should only be called on a worker node. On the leader, this method returns this resource, i.e. self.

Return type:

ModuleDescriptor

globalize()

Reverse the effect of localize().

Return type:

ModuleDescriptor

toCommand()

Return type:

Sequence[str]

classmethod fromCommand(command)

Parameters:

command (Sequence[str])

Return type:

ModuleDescriptor

makeLoadable()

Return type:

ModuleDescriptor

load()

Return type:

Optional[types.ModuleType]

class toil.test.src.resourceTest.Resource

Bases: namedtuple('Resource', ('name', 'pathHash', 'url', 'contentHash'))

Represents a file or directory that will be deployed to each node before any jobs in the user script are invoked.

Each instance is a namedtuple with the following elements:

The pathHash element contains the MD5 (in hexdigest form) of the path to the resource on the leader node. The path, and therefore its hash is unique within a job store.

The url element is a “file:” or “http:” URL at which the resource can be obtained.

The contentHash element is an MD5 checksum of the resource, allowing for validation and caching of resources.

If the resource is a regular file, the type attribute will be ‘file’.

If the resource is a directory, the type attribute will be ‘dir’ and the URL will point at a ZIP archive of that directory.

resourceEnvNamePrefix = 'JTRES_'

rootDirPathEnvName

classmethod create(jobStore, leaderPath)

Saves the content of the file or directory at the given path to the given job store and returns a resource object representing that content for the purpose of obtaining it again at a generic, public URL. This method should be invoked on the leader node.

Parameters:

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

• leaderPath (str)

Return type:

Resource

refresh(jobStore)

Parameters:

jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type:

Resource

classmethod prepareSystem()

Prepares this system for the downloading and lookup of resources. This method should only be invoked on a worker node. It is idempotent but not thread-safe.

Return type:

None

classmethod cleanSystem()

Remove all downloaded, localized resources.

Return type:

None

register()

Register this resource for later retrieval via lookup(), possibly in a child process.

Return type:

None

classmethod lookup(leaderPath)

Return a resource object representing a resource created from a file or directory at the given path on the leader.

This method should be invoked on the worker. The given path does not need to refer to an existing file or directory on the worker, it only identifies the resource within an instance of toil. This method returns None if no resource for the given path exists.

Parameters:

leaderPath (str)

Return type:

Optional[Resource]

download(callback=None)

Download this resource from its URL to a file on the local system.

This method should only be invoked on a worker node after the node was setup for accessing resources via prepareSystem().

Parameters:

callback (Optional[Callable[[str], None]])

Return type:

None

property localPath: str

Abstractmethod:

Return type:

str

Get the path to resource on the worker.

The file or directory at the returned path may or may not yet exist. Invoking download() will ensure that it does.

property localDirPath: str

The path to the directory containing the resource on the worker.

Return type:

str

pickle()

Return type:

str

classmethod unpickle(s)

Parameters:

s (str)

Return type:

Resource

exception toil.test.src.resourceTest.ResourceException

Bases: Exception

Common base class for all non-exit exceptions.

class toil.test.src.resourceTest.ToilTest(methodName='runTest')

Bases: unittest.TestCase

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

setup_method(method)

Parameters:

method (Any)

Return type:

None

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

Return type:

None

classmethod tearDownClass()

Hook method for deconstructing the class fixture after running all tests in the class.

Return type:

None

setUp()

Hook method for setting up the test fixture before exercising it.

Return type:

None

tearDown()

Hook method for deconstructing the test fixture after testing it.

Return type:

None

classmethod awsRegion()

Pick an appropriate AWS region.

Use us-west-2 unless running on EC2, in which case use the region in which the instance is located

Return type:

str

toil.test.src.resourceTest.exactPython = 'python3.9'

toil.test.src.resourceTest.tempFileContaining(content, suffix='')

Write a file with the given contents, and keep it on disk as long as the context is active. :param str content: The contents of the file. :param str suffix: The extension to use for the temporary file.

class toil.test.src.resourceTest.ResourceTest(methodName='runTest')

Bases: toil.test.ToilTest

Test module descriptors and resources derived from them.

testStandAlone()

testPackage()

testVirtualEnv()

testStandAloneInPackage()

testBuiltIn()

testNonPyStandAlone()

Asserts that Toil enforces the user script to have a .py or .pyc extension because that’s the only way auto-deployment can re-import the module on a worker. See

https://github.com/BD2KGenomics/toil/issues/631 and https://github.com/BD2KGenomics/toil/issues/858