toil.lib.aws.utils¶

Attributes¶

`AWSRegionName`
`AWSServerErrors`
`DEFAULT_DELAYS`
`DEFAULT_TIMEOUT`
`ClientError`
`logger`
`THROTTLED_ERROR_CODES`

Classes¶

ErrorCondition

A wrapper describing an error condition.

Functions¶

`printq`(msg, quiet)
`get_error_code`(e)	Get the error code name from a Boto 2 or 3 error, or compatible types.
`get_error_status`(e)	Get the HTTP status code from a compatible source.
`old_retry`([delays, timeout, predicate])	Deprecated.
`retry`([intervals, infinite_retries, errors, ...])	Retry a function if it fails with any Exception defined in "errors".
`delete_iam_role`(role_name[, region, quiet])
`delete_iam_instance_profile`(instance_profile_name[, ...])
`delete_sdb_domain`(sdb_domain_name[, region, quiet])
`connection_reset`(e)	Return true if an error is a connection reset error.
`connection_error`(e)	Return True if an error represents a failure to make a network connection.
`retryable_s3_errors`(e)	Return true if this is an error from S3 that looks like we ought to retry our request.
`retry_s3`([delays, timeout, predicate])	Retry iterator of context managers specifically for S3 operations.
`delete_s3_bucket`(s3_resource, bucket[, quiet])	Delete the given S3 bucket.
`create_s3_bucket`(s3_resource, bucket_name, region)	Create an AWS S3 bucket, using the given Boto3 S3 session, with the
`enable_public_objects`(bucket_name)	Enable a bucket to contain objects which are public.
`get_bucket_region`(bucket_name[, endpoint_url, ...])	Get the AWS region name associated with the given S3 bucket.
`region_to_bucket_location`(region)
`bucket_location_to_region`(location)
`get_object_for_url`(url[, existing])	Extracts a key (object) from a given parsed s3:// URL.
`list_objects_for_url`(url)	Extracts a key (object) from a given parsed s3:// URL. The URL will be
`flatten_tags`(tags)	Convert tags from a key to value dict into a list of 'Key': xxx, 'Value': xxx dicts.
`boto3_pager`(requestor_callable, result_attribute_name, ...)	Yield all the results from calling the given Boto 3 method with the
`get_item_from_attributes`(attributes, name)	Given a list of attributes, find the attribute associated with the name and return its corresponding value.

Module Contents¶

toil.lib.aws.utils.AWSRegionName¶

toil.lib.aws.utils.AWSServerErrors¶

toil.lib.aws.utils.printq(msg, quiet)¶

Parameters:

msg (str)
quiet (bool)

Return type:

None

toil.lib.aws.utils.DEFAULT_DELAYS = (0, 1, 1, 4, 16, 64)¶

toil.lib.aws.utils.DEFAULT_TIMEOUT = 300¶

toil.lib.aws.utils.get_error_code(e)¶

Get the error code name from a Boto 2 or 3 error, or compatible types.

Returns empty string for other errors.

Parameters:: e (Exception)
Return type:: str

toil.lib.aws.utils.get_error_status(e)¶

Get the HTTP status code from a compatible source.

Such as a Boto 2 or 3 error, kubernetes.client.rest.ApiException, http.client.HTTPException, urllib3.exceptions.HTTPError, requests.exceptions.HTTPError, urllib.error.HTTPError, or compatible type

Returns 0 from other errors.

Parameters:: e (Exception)
Return type:: int

toil.lib.aws.utils.old_retry(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=lambda e: ...)¶

Deprecated.

Retry an operation while the failure matches a given predicate and until a given timeout expires, waiting a given amount of time in between attempts. This function is a generator that yields contextmanagers. See doctests below for example usage.

Parameters:

delays (Iterable[float]) – an interable yielding the time in seconds to wait before each retried attempt, the last element of the iterable will be repeated.
timeout (float) – a overall timeout that should not be exceeded for all attempts together. This is a best-effort mechanism only and it won’t abort an ongoing attempt, even if the timeout expires during that attempt.
predicate (Callable[[Exception],bool]) – a unary callable returning True if another attempt should be made to recover from the given exception. The default value for this parameter will prevent any retries!

Returns:

a generator yielding context managers, one per attempt

Return type:

Iterator

Retry for a limited amount of time:

>>> true = lambda _:True
>>> false = lambda _:False
>>> i = 0
>>> for attempt in old_retry( delays=[0], timeout=.1, predicate=true ):
...     with attempt:
...         i += 1
...         raise RuntimeError('foo')
Traceback (most recent call last):
...
RuntimeError: foo
>>> i > 1
True

If timeout is 0, do exactly one attempt:

>>> i = 0
>>> for attempt in old_retry( timeout=0 ):
...     with attempt:
...         i += 1
...         raise RuntimeError( 'foo' )
Traceback (most recent call last):
...
RuntimeError: foo
>>> i
1

Don’t retry on success:

>>> i = 0
>>> for attempt in old_retry( delays=[0], timeout=.1, predicate=true ):
...     with attempt:
...         i += 1
>>> i
1

Don’t retry on unless predicate returns True:

>>> i = 0
>>> for attempt in old_retry( delays=[0], timeout=.1, predicate=false):
...     with attempt:
...         i += 1
...         raise RuntimeError( 'foo' )
Traceback (most recent call last):
...
RuntimeError: foo
>>> i
1

toil.lib.aws.utils.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)¶

Retry a function if it fails with any Exception defined in “errors”.

Does so every x seconds, where x is defined by a list of numbers (ints or floats) in “intervals”. Also accepts ErrorCondition events for more detailed retry attempts.

Parameters:

intervals (Optional[List]) – A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s
infinite_retries (bool) – If this is True, reset the intervals when they run out. Defaults to: False.
errors (Optional[Sequence[Union[ErrorCondition, Type[Exception]]]]) –
A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: - Exception (required) - Error codes that must match to be retried (optional; defaults to not checking) - A string that must be in the error message to be retried (optional; defaults to not checking) - A bool that can be set to False to always error on this condition.

If not specified, this will default to a generic Exception.
log_message (Optional[Tuple[Callable, str]]) – Optional tuple of (“log/print function()”, “message string”) that will precede each attempt.
prepare (Optional[List[Callable]]) – Optional list of functions to call, with the function’s arguments, between retries, to reset state.

Returns:

The result of the wrapped function or raise.

Return type:

Callable[[Callable[Ellipsis, RT]], Callable[Ellipsis, RT]]

class toil.lib.aws.utils.ErrorCondition(error=None, error_codes=None, boto_error_codes=None, error_message_must_include=None, retry_on_this_condition=True)¶

A wrapper describing an error condition.

ErrorCondition events may be used to define errors in more detail to determine whether to retry.

Parameters:

error (Optional[Any])
error_codes (List[int])
boto_error_codes (List[str])
error_message_must_include (str)
retry_on_this_condition (bool)

toil.lib.aws.utils.ClientError = None¶

toil.lib.aws.utils.logger¶

toil.lib.aws.utils.THROTTLED_ERROR_CODES = ['Throttling', 'ThrottlingException', 'ThrottledException', 'RequestThrottledException',...¶

toil.lib.aws.utils.delete_iam_role(role_name, region=None, quiet=True)¶

Parameters:

role_name (str)
region (Optional[str])
quiet (bool)

Return type:

None

toil.lib.aws.utils.delete_iam_instance_profile(instance_profile_name, region=None, quiet=True)¶

Parameters:

instance_profile_name (str)
region (Optional[str])
quiet (bool)

Return type:

None

toil.lib.aws.utils.delete_sdb_domain(sdb_domain_name, region=None, quiet=True)¶

Parameters:

sdb_domain_name (str)
region (Optional[str])
quiet (bool)

Return type:

None

toil.lib.aws.utils.connection_reset(e)¶

Return true if an error is a connection reset error.

Parameters:: e (Exception)
Return type:: bool

toil.lib.aws.utils.connection_error(e)¶

Return True if an error represents a failure to make a network connection.

Parameters:: e (Exception)
Return type:: bool

toil.lib.aws.utils.retryable_s3_errors(e)¶

Return true if this is an error from S3 that looks like we ought to retry our request.

Parameters:: e (Exception)
Return type:: bool

toil.lib.aws.utils.retry_s3(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=retryable_s3_errors)¶

Retry iterator of context managers specifically for S3 operations.

Parameters:

delays (Iterable[float])
timeout (float)
predicate (Callable[[Exception], bool])

Return type:

Iterator[ContextManager[None]]

toil.lib.aws.utils.delete_s3_bucket(s3_resource, bucket, quiet=True)¶

Delete the given S3 bucket.

Parameters:

s3_resource (mypy_boto3_s3.S3ServiceResource)
bucket (str)
quiet (bool)

Return type:

None

toil.lib.aws.utils.create_s3_bucket(s3_resource, bucket_name, region)¶

Create an AWS S3 bucket, using the given Boto3 S3 session, with the given name, in the given region.

Supports the us-east-1 region, where bucket creation is special.

ALL S3 bucket creation should use this function.

Parameters:

s3_resource (mypy_boto3_s3.S3ServiceResource)
bucket_name (str)
region (toil.lib.aws.AWSRegionName)

Return type:

mypy_boto3_s3.service_resource.Bucket

toil.lib.aws.utils.enable_public_objects(bucket_name)¶

Enable a bucket to contain objects which are public.

This adjusts the bucket’s Public Access Block setting to not block all public access, and also adjusts the bucket’s Object Ownership setting to a setting which enables object ACLs.

Does not touch the account’s Public Access Block setting, which can also interfere here. That is probably best left to the account administrator.

This configuration used to be the default, and is what most of Toil’s code is written to expect, but it was changed so that new buckets default to the more restrictive setting <https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/>, with the expectation that people would write IAM policies for the buckets to allow public access if needed. Toil expects to be able to make arbitrary objects in arbitrary places public, and naming them all in an IAM policy would be a very awkward way to do it. So we restore the old behavior.

Parameters:: bucket_name (str)
Return type:: None

toil.lib.aws.utils.get_bucket_region(bucket_name, endpoint_url=None, only_strategies=None)¶

Get the AWS region name associated with the given S3 bucket.

Takes an optional S3 API URL override.

Parameters:

only_strategies (Optional[Set[int]]) – For testing, use only strategies with 1-based numbers in this set.
bucket_name (str)
endpoint_url (Optional[str])

Return type:

str

toil.lib.aws.utils.region_to_bucket_location(region)¶

Parameters:: region (str)
Return type:: str

toil.lib.aws.utils.bucket_location_to_region(location)¶

Parameters:: location (Optional[str])
Return type:: str

toil.lib.aws.utils.get_object_for_url(url, existing=None)¶

Extracts a key (object) from a given parsed s3:// URL.

If existing is true and the object does not exist, raises FileNotFoundError.

Parameters:

existing (bool) – If True, key is expected to exist. If False, key is expected not to exists and it will be created. If None, the key will be created if it doesn’t exist.
url (urllib.parse.ParseResult)

Return type:

mypy_boto3_s3.service_resource.Object

toil.lib.aws.utils.list_objects_for_url(url)¶

Extracts a key (object) from a given parsed s3:// URL. The URL will be supplemented with a trailing slash if it is missing.

Parameters:: url (urllib.parse.ParseResult)
Return type:: List[str]

toil.lib.aws.utils.flatten_tags(tags)¶

Convert tags from a key to value dict into a list of ‘Key’: xxx, ‘Value’: xxx dicts.

Parameters:: tags (Dict[str, str])
Return type:: List[Dict[str, str]]

toil.lib.aws.utils.boto3_pager(requestor_callable, result_attribute_name, **kwargs)¶

Yield all the results from calling the given Boto 3 method with the given keyword arguments, paging through the results using the Marker or NextToken, and fetching out and looping over the list in the response with the given attribute name.

Parameters:

requestor_callable (Callable[Ellipsis, Any])
result_attribute_name (str)
kwargs (Any)

Return type:

Iterable[Any]

toil.lib.aws.utils.get_item_from_attributes(attributes, name)¶

Given a list of attributes, find the attribute associated with the name and return its corresponding value.

The attribute_list will be a list of TypedDict’s (which boto3 SDB functions commonly return), where each TypedDict has a “Name” and “Value” key value pair. This function grabs the value out of the associated TypedDict.

If the attribute with the name does not exist, the function will return None.

Parameters:

attributes (List[mypy_boto3_sdb.type_defs.AttributeTypeDef]) – list of attributes as List[AttributeTypeDef]
name (str) – name of the attribute

Returns:

value of the attribute

Return type:

Any

toil.lib.aws.utils¶

Attributes¶

Classes¶

Functions¶

Module Contents¶

Toil

Navigation

Related Topics