toil.lib.aws.utils

Module Contents

Functions

delete_iam_role(role_name[, region, quiet])
delete_iam_instance_profile(instance_profile_name[, ...])
delete_sdb_domain(sdb_domain_name[, region, quiet])
connection_reset(e)	Return true if an error is a connection reset error.
retryable_s3_errors(e)	Return true if this is an error from S3 that looks like we ought to retry our request.
retry_s3([delays, timeout, predicate])	Retry iterator of context managers specifically for S3 operations.
delete_s3_bucket(s3_resource, bucket[, quiet])	Delete the given S3 bucket.
create_s3_bucket(s3_resource, bucket_name, region)	Create an AWS S3 bucket, using the given Boto3 S3 session, with the
enable_public_objects(bucket_name)	Enable a bucket to contain objects which are public.
get_bucket_region(bucket_name[, endpoint_url, ...])	Get the AWS region name associated with the given S3 bucket.
region_to_bucket_location(region)
bucket_location_to_region(location)
get_object_for_url(url[, existing])	Extracts a key (object) from a given parsed s3:// URL.
list_objects_for_url(url)	Extracts a key (object) from a given parsed s3:// URL. The URL will be
flatten_tags(tags)	Convert tags from a key to value dict into a list of 'Key': xxx, 'Value': xxx dicts.

Attributes

BotoServerError
logger
THROTTLED_ERROR_CODES

toil.lib.aws.utils.BotoServerError

toil.lib.aws.utils.logger

toil.lib.aws.utils.THROTTLED_ERROR_CODES = ['Throttling', 'ThrottlingException', 'ThrottledException', 'RequestThrottledException',...

toil.lib.aws.utils.delete_iam_role(role_name, region=None, quiet=True)

Parameters:

• role_name (str)

• region (Optional[str])

• quiet (bool)

Return type:

None

toil.lib.aws.utils.delete_iam_instance_profile(instance_profile_name, region=None, quiet=True)

Parameters:

• instance_profile_name (str)

• region (Optional[str])

• quiet (bool)

Return type:

None

toil.lib.aws.utils.delete_sdb_domain(sdb_domain_name, region=None, quiet=True)

Parameters:

• sdb_domain_name (str)

• region (Optional[str])

• quiet (bool)

Return type:

None

toil.lib.aws.utils.connection_reset(e)

Return true if an error is a connection reset error.

Parameters:

e (Exception)

Return type:

bool

toil.lib.aws.utils.retryable_s3_errors(e)

Return true if this is an error from S3 that looks like we ought to retry our request.

Parameters:

e (Exception)

Return type:

bool

toil.lib.aws.utils.retry_s3(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=retryable_s3_errors)

Retry iterator of context managers specifically for S3 operations.

Parameters:

• delays (Iterable[float])

• timeout (float)

• predicate (Callable[[Exception], bool])

Return type:

Iterator[ContextManager[None]]

toil.lib.aws.utils.delete_s3_bucket(s3_resource, bucket, quiet=True)

Delete the given S3 bucket.

Parameters:

• s3_resource (mypy_boto3_s3.S3ServiceResource)

• bucket (str)

• quiet (bool)

Return type:

None

toil.lib.aws.utils.create_s3_bucket(s3_resource, bucket_name, region)

Create an AWS S3 bucket, using the given Boto3 S3 session, with the given name, in the given region.

Supports the us-east-1 region, where bucket creation is special.

ALL S3 bucket creation should use this function.

Parameters:

• s3_resource (mypy_boto3_s3.S3ServiceResource)

• bucket_name (str)

• region (toil.lib.aws.AWSRegionName)

Return type:

mypy_boto3_s3.service_resource.Bucket

toil.lib.aws.utils.enable_public_objects(bucket_name)

Enable a bucket to contain objects which are public.

This adjusts the bucket’s Public Access Block setting to not block all public access, and also adjusts the bucket’s Object Ownership setting to a setting which enables object ACLs.

Does not touch the account’s Public Access Block setting, which can also interfere here. That is probably best left to the account administrator.

This configuration used to be the default, and is what most of Toil’s code is written to expect, but it was changed so that new buckets default to the more restrictive setting <https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/>, with the expectation that people would write IAM policies for the buckets to allow public access if needed. Toil expects to be able to make arbitrary objects in arbitrary places public, and naming them all in an IAM policy would be a very awkward way to do it. So we restore the old behavior.

Parameters:

bucket_name (str)

Return type:

None

toil.lib.aws.utils.get_bucket_region(bucket_name, endpoint_url=None, only_strategies=None)

Get the AWS region name associated with the given S3 bucket.

Takes an optional S3 API URL override.

Parameters:

• only_strategies (Optional[Set[int]]) – For testing, use only strategies with 1-based numbers in this set.

• bucket_name (str)

• endpoint_url (Optional[str])

Return type:

str

toil.lib.aws.utils.region_to_bucket_location(region)

Parameters:

region (str)

Return type:

str

toil.lib.aws.utils.bucket_location_to_region(location)

Parameters:

location (Optional[str])

Return type:

str

toil.lib.aws.utils.get_object_for_url(url, existing=None)

Extracts a key (object) from a given parsed s3:// URL.

If existing is true and the object does not exist, raises FileNotFoundError.

Parameters:

• existing (bool) – If True, key is expected to exist. If False, key is expected not to exists and it will be created. If None, the key will be created if it doesn’t exist.

• url (urllib.parse.ParseResult)

Return type:

mypy_boto3_s3.service_resource.Object

toil.lib.aws.utils.list_objects_for_url(url)

Extracts a key (object) from a given parsed s3:// URL. The URL will be supplemented with a trailing slash if it is missing.

Parameters:

url (urllib.parse.ParseResult)

Return type:

List[str]

toil.lib.aws.utils.flatten_tags(tags)

Convert tags from a key to value dict into a list of ‘Key’: xxx, ‘Value’: xxx dicts.

Parameters:

tags (Dict[str, str])

Return type:

List[Dict[str, str]]