toil.lib.url

Attributes

logger

Exceptions

ProxyConnectionError

Dummy class.

Classes

URLAccess

Widget for accessing external storage (URLs).

Functions

file_job_store_factory()
google_job_store_factory()
aws_job_store_factory()
job_store_support_factory()

Module Contents

exception toil.lib.url.ProxyConnectionError

Bases: BaseException

Dummy class.

toil.lib.url.logger

class toil.lib.url.URLAccess

Widget for accessing external storage (URLs).

classmethod url_exists(src_uri)

Return True if the file at the given URI exists, and False otherwise.

May raise an error if file existence cannot be determined.

Parameters:

src_uri (str) – URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket.

Return type:

bool

classmethod get_size(src_uri)

Get the size in bytes of the file at the given URL, or None if it cannot be obtained.

Parameters:

src_uri (str) – URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket.

Return type:

Optional[int]

classmethod get_is_directory(src_uri)

Return True if the thing at the given URL is a directory, and False if it is a file. The URL may or may not end in ‘/’.

Parameters:

src_uri (str)

Return type:

bool

classmethod list_url(src_uri)

List the directory at the given URL. Returned path components can be joined with ‘/’ onto the passed URL to form new URLs. Those that end in ‘/’ correspond to directories. The provided URL may or may not end with ‘/’.

Currently supported schemes are:

• ‘s3’ for objects in Amazon S3

  e.g. s3://bucket/prefix/

• ‘file’ for local files

  e.g. file:///local/dir/path/

Parameters:

src_uri (str) – URL that points to a directory or prefix in the storage mechanism of a supported URL scheme e.g. a prefix in an AWS s3 bucket.

Returns:

A list of URL components in the given directory, already URL-encoded.

Return type:

list[str]

classmethod read_from_url(src_uri, writable)

Read the given URL and write its content into the given writable stream.

Raises FileNotFoundError if the URL doesn’t exist.

Returns:

The size of the file in bytes and whether the executable permission bit is set

Parameters:

• src_uri (str)

• writable (IO[bytes])

Return type:

tuple[int, bool]

classmethod open_url(src_uri)

Read from the given URI.

Raises FileNotFoundError if the URL doesn’t exist.

Has a readable stream interface, unlike read_from_url() which takes a writable stream.

Parameters:

src_uri (str)

Return type:

IO[bytes]

toil.lib.url.file_job_store_factory()

Return type:

type[URLAccess]

toil.lib.url.google_job_store_factory()

Return type:

type[URLAccess]

toil.lib.url.aws_job_store_factory()

Return type:

type[URLAccess]

toil.lib.url.job_store_support_factory()

Return type:

type[URLAccess]