toil.test.src.miscTests

Attributes

log

Exceptions

CalledProcessErrorStderr

Version of CalledProcessError that include stderr in the error message if it is set

Classes

panic	The Python idiom for reraising a primary exception fails when the except block raises a
ToilTest	A common base class for Toil tests.
MiscTests	This class contains miscellaneous tests that don't have enough content to be their own test
TestPanic	A common base class for Toil tests.

Functions

getNodeID()	Return unique ID of the current node (host). The resulting string will be convertible to a uuid.UUID.
raise_(exc_type, exc_value, traceback)
AtomicFileCreate(final_path[, keep])	Context manager to create a temporary file. Entering returns path to
atomic_install(tmp_path, final_path)	atomic install of tmp_path as final_path
atomic_tmp_file(final_path)	Return a tmp file name to use with atomic_install. This will be in the
mkdtemp([suffix, prefix, dir])	Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions.
call_command(cmd, *args[, input, timeout, useCLocale, ...])	Simplified calling of external commands.
slow(test_item)	Use this decorator to identify tests that are slow and not critical.

Module Contents

toil.test.src.miscTests.getNodeID()

Return unique ID of the current node (host). The resulting string will be convertible to a uuid.UUID.

Tries several methods until success. The returned ID should be identical across calls from different processes on the same node at least until the next OS reboot.

The last resort method is uuid.getnode() that in some rare OS configurations may return a random ID each time it is called. However, this method should never be reached on a Linux system, because reading from /proc/sys/kernel/random/boot_id will be tried prior to that. If uuid.getnode() is reached, it will be called twice, and exception raised if the values are not identical.

Return type:

str

class toil.test.src.miscTests.panic(log=None)

The Python idiom for reraising a primary exception fails when the except block raises a secondary exception, e.g. while trying to cleanup. In that case the original exception is lost and the secondary exception is reraised. The solution seems to be to save the primary exception info as returned from sys.exc_info() and then reraise that.

This is a contextmanager that should be used like this

try:

# do something that can fail

except:

with panic( log ):

# do cleanup that can also fail

If a logging logger is passed to panic(), any secondary Exception raised within the with block will be logged. Otherwise those exceptions are swallowed. At the end of the with block the primary exception will be reraised.

__enter__()

__exit__(*exc_info)

toil.test.src.miscTests.raise_(exc_type, exc_value, traceback)

Return type:

None

toil.test.src.miscTests.AtomicFileCreate(final_path, keep=False)

Context manager to create a temporary file. Entering returns path to the temporary file in the same directory as finalPath. If the code in context succeeds, the file renamed to its actually name. If an error occurs, the file is not installed and is removed unless keep is specified.

Parameters:

• final_path (str)

• keep (bool)

Return type:

Iterator[str]

toil.test.src.miscTests.atomic_install(tmp_path, final_path)

atomic install of tmp_path as final_path

Return type:

None

toil.test.src.miscTests.atomic_tmp_file(final_path)

Return a tmp file name to use with atomic_install. This will be in the same directory as final_path. The temporary file will have the same extension as finalPath. It the final path is in /dev (/dev/null, /dev/stdout), it is returned unchanged and atomic_tmp_install will do nothing.

Parameters:

final_path (str)

Return type:

str

toil.test.src.miscTests.mkdtemp(suffix=None, prefix=None, dir=None)

Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions.

The permissions on the directory will be 711 instead of 700, allowing the group and all other users to traverse the directory. This is necessary if the directory is on NFS and the Docker daemon would like to mount it or a file inside it into a container, because on NFS even the Docker daemon appears bound by the file permissions.

See <https://github.com/DataBiosphere/toil/issues/4644>, and <https://stackoverflow.com/a/67928880> which talks about a similar problem but in the context of user namespaces.

Parameters:

• suffix (Optional[str])

• prefix (Optional[str])

• dir (Optional[str])

Return type:

str

exception toil.test.src.miscTests.CalledProcessErrorStderr(returncode, cmd, output=None, stderr=None)

Bases: subprocess.CalledProcessError

Version of CalledProcessError that include stderr in the error message if it is set

__str__()

Return str(self).

Return type:

str

toil.test.src.miscTests.call_command(cmd, *args, input=None, timeout=None, useCLocale=True, env=None, quiet=False)

Simplified calling of external commands.

If the process fails, CalledProcessErrorStderr is raised.

The captured stderr is always printed, regardless of if an exception occurs, so it can be logged.

Always logs the command at debug log level.

Parameters:

• quiet (Optional[bool]) – If True, do not log the command output. If False (the default), do log the command output at debug log level.

• useCLocale (bool) – If True, C locale is forced, to prevent failures that can occur in some batch systems when using UTF-8 locale.

• cmd (List[str])

• args (str)

• input (Optional[str])

• timeout (Optional[float])

• env (Optional[Dict[str, str]])

Returns:

Command standard output, decoded as utf-8.

Return type:

str

class toil.test.src.miscTests.ToilTest(methodName='runTest')

Bases: unittest.TestCase

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

setup_method(method)

Parameters:

method (Any)

Return type:

None

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

Return type:

None

classmethod tearDownClass()

Hook method for deconstructing the class fixture after running all tests in the class.

Return type:

None

setUp()

Hook method for setting up the test fixture before exercising it.

Return type:

None

tearDown()

Hook method for deconstructing the test fixture after testing it.

Return type:

None

classmethod awsRegion()

Pick an appropriate AWS region.

Use us-west-2 unless running on EC2, in which case use the region in which the instance is located

Return type:

str

toil.test.src.miscTests.slow(test_item)

Use this decorator to identify tests that are slow and not critical. Skip if TOIL_TEST_QUICK is true.

Parameters:

test_item (MT)

Return type:

MT

toil.test.src.miscTests.log

class toil.test.src.miscTests.MiscTests(methodName='runTest')

Bases: toil.test.ToilTest

This class contains miscellaneous tests that don’t have enough content to be their own test file, and that don’t logically fit in with any of the other test suites.

setUp()

Hook method for setting up the test fixture before exercising it.

testIDStability()

testGetSizeOfDirectoryWorks()

A test to make sure toil.common.getDirSizeRecursively does not underestimate the amount of disk space needed.

Disk space allocation varies from system to system. The computed value should always be equal to or slightly greater than the creation value. This test generates a number of random directories and randomly sized files to test this using getDirSizeRecursively.

test_atomic_install()

test_atomic_install_dev()

test_atomic_context_ok()

test_atomic_context_error()

test_call_command_ok()

test_call_command_err()

class toil.test.src.miscTests.TestPanic(methodName='runTest')

Bases: toil.test.ToilTest

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

test_panic_by_hand()

test_panic()

test_panic_with_secondary()

test_nested_panic()

try_and_panic_by_hand()

try_and_panic()

try_and_panic_with_secondary()

try_and_nested_panic_with_secondary()