toil.lib.io

Module Contents

Classes

WriteWatchingStream

A stream wrapping class that calls any functions passed to onWrite() with the number of bytes written for every write.

Functions

mkdtemp([suffix, prefix, dir])

Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions.

robust_rmtree(path)

Robustly tries to delete paths.

atomic_tmp_file(final_path)

Return a tmp file name to use with atomic_install. This will be in the

atomic_install(tmp_path, final_path)

atomic install of tmp_path as final_path

AtomicFileCreate(final_path[, keep])

Context manager to create a temporary file. Entering returns path to

atomic_copy(src_path, dest_path[, executable])

Copy a file using posix atomic creations semantics.

atomic_copyobj(src_fh, dest_path[, length, executable])

Copy an open file using posix atomic creations semantics.

make_public_dir(in_directory[, suggested_name])

Make a publicly-accessible directory in the given directory.

try_path(path[, min_size])

Try to use the given path. Return it if it exists or can be made,

Attributes

logger

toil.lib.io.logger
toil.lib.io.mkdtemp(suffix=None, prefix=None, dir=None)[source]

Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions.

The permissions on the directory will be 711 instead of 700, allowing the group and all other users to traverse the directory. This is necessary if the direcotry is on NFS and the Docker daemon would like to mount it or a file inside it into a container, because on NFS even the Docker daemon appears bound by the file permissions.

See <https://github.com/DataBiosphere/toil/issues/4644>, and <https://stackoverflow.com/a/67928880> which talks about a similar problem but in the context of user namespaces.

Parameters:

  • suffix (Optional[str])

  • prefix (Optional[str])

  • dir (Optional[str])

Return type:

str

toil.lib.io.robust_rmtree(path)[source]

Robustly tries to delete paths.

Continues silently if the path to be removed is already gone, or if it goes away while this function is executing.

May raise an error if a path changes between file and directory while the function is executing, or if a permission error is encountered.

Parameters:

path (Union[str, bytes])

Return type:

None

toil.lib.io.atomic_tmp_file(final_path)[source]

Return a tmp file name to use with atomic_install. This will be in the same directory as final_path. The temporary file will have the same extension as finalPath. It the final path is in /dev (/dev/null, /dev/stdout), it is returned unchanged and atomic_tmp_install will do nothing.

Parameters:

final_path (str)

Return type:

str

toil.lib.io.atomic_install(tmp_path, final_path)[source]

atomic install of tmp_path as final_path

Return type:

None

toil.lib.io.AtomicFileCreate(final_path, keep=False)[source]

Context manager to create a temporary file. Entering returns path to the temporary file in the same directory as finalPath. If the code in context succeeds, the file renamed to its actually name. If an error occurs, the file is not installed and is removed unless keep is specified.

Parameters:
Return type:

Iterator[str]

toil.lib.io.atomic_copy(src_path, dest_path, executable=None)[source]

Copy a file using posix atomic creations semantics.

Parameters:

  • src_path (str)

  • dest_path (str)

  • executable (Optional[bool])

Return type:

None

toil.lib.io.atomic_copyobj(src_fh, dest_path, length=16384, executable=False)[source]

Copy an open file using posix atomic creations semantics.

Parameters:
Return type:

None

toil.lib.io.make_public_dir(in_directory, suggested_name=None)[source]

Make a publicly-accessible directory in the given directory.

Parameters:

  • suggested_name (Optional[str]) – Use this directory name first if possible.

  • in_directory (str)

Return type:

str

Try to make a random directory name with length 4 that doesn’t exist, with the given prefix. Otherwise, try length 5, length 6, etc, up to a max of 32 (len of uuid4 with dashes replaced). This function’s purpose is mostly to avoid having long file names when generating directories. If somehow this fails, which should be incredibly unlikely, default to a normal uuid4, which was our old default.

toil.lib.io.try_path(path, min_size=100 * 1024 * 1024)[source]

Try to use the given path. Return it if it exists or can be made, and we can make things within it, or None otherwise.

Parameters:

  • min_size (int) – Reject paths on filesystems smaller than this many bytes.

  • path (str)

Return type:

Optional[str]

class toil.lib.io.WriteWatchingStream(backingStream)[source]

A stream wrapping class that calls any functions passed to onWrite() with the number of bytes written for every write.

Not seekable.

Parameters:

backingStream (IO[Any])

onWrite(listener)[source]

Call the given listener with the number of bytes written on every write.

Parameters:

listener (Callable[[int], None])

Return type:

None

write(data)[source]

Write the given data to the file.

writelines(datas)[source]

Write each string from the given iterable, without newlines.

flush()[source]

Flush the backing stream.

close()[source]

Close the backing stream.