toil.lib.aws.utils

Module Contents

Functions

delete_iam_role(role_name[, region, quiet])

delete_iam_instance_profile(instance_profile_name[, ...])

delete_sdb_domain(sdb_domain_name[, region, quiet])

connection_reset(e)

Return true if an error is a connection reset error.

retryable_s3_errors(e)

Return true if this is an error from S3 that looks like we ought to retry our request.

retry_s3([delays, timeout, predicate])

Retry iterator of context managers specifically for S3 operations.

delete_s3_bucket(s3_resource, bucket[, quiet])

Delete the given S3 bucket.

create_s3_bucket(s3_resource, bucket_name, region)

Create an AWS S3 bucket, using the given Boto3 S3 session, with the

enable_public_objects(bucket_name)

Enable a bucket to contain objects which are public.

get_bucket_region(bucket_name[, endpoint_url, ...])

Get the AWS region name associated with the given S3 bucket.

region_to_bucket_location(region)

bucket_location_to_region(location)

get_object_for_url(url[, existing])

Extracts a key (object) from a given parsed s3:// URL.

list_objects_for_url(url)

Extracts a key (object) from a given parsed s3:// URL. The URL will be

flatten_tags(tags)

Convert tags from a key to value dict into a list of 'Key': xxx, 'Value': xxx dicts.

Attributes

BotoServerError

logger

THROTTLED_ERROR_CODES

toil.lib.aws.utils.BotoServerError
toil.lib.aws.utils.logger
toil.lib.aws.utils.THROTTLED_ERROR_CODES = ['Throttling', 'ThrottlingException', 'ThrottledException', 'RequestThrottledException',...
toil.lib.aws.utils.delete_iam_role(role_name, region=None, quiet=True)
Parameters

  • role_name (str) –

  • region (Optional[str]) –

  • quiet (bool) –

Return type

None

toil.lib.aws.utils.delete_iam_instance_profile(instance_profile_name, region=None, quiet=True)
Parameters

  • instance_profile_name (str) –

  • region (Optional[str]) –

  • quiet (bool) –

Return type

None

toil.lib.aws.utils.delete_sdb_domain(sdb_domain_name, region=None, quiet=True)
Parameters

  • sdb_domain_name (str) –

  • region (Optional[str]) –

  • quiet (bool) –

Return type

None

toil.lib.aws.utils.connection_reset(e)

Return true if an error is a connection reset error.

Parameters

e (Exception) –

Return type

bool

toil.lib.aws.utils.retryable_s3_errors(e)

Return true if this is an error from S3 that looks like we ought to retry our request.

Parameters

e (Exception) –

Return type

bool

toil.lib.aws.utils.retry_s3(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=retryable_s3_errors)

Retry iterator of context managers specifically for S3 operations.

Parameters
Return type

Iterator[ContextManager[None]]

toil.lib.aws.utils.delete_s3_bucket(s3_resource, bucket, quiet=True)

Delete the given S3 bucket.

Parameters

  • s3_resource (mypy_boto3_s3.S3ServiceResource) –

  • bucket (str) –

  • quiet (bool) –

Return type

None

toil.lib.aws.utils.create_s3_bucket(s3_resource, bucket_name, region)

Create an AWS S3 bucket, using the given Boto3 S3 session, with the given name, in the given region.

Supports the us-east-1 region, where bucket creation is special.

ALL S3 bucket creation should use this function.

Parameters

  • s3_resource (mypy_boto3_s3.S3ServiceResource) –

  • bucket_name (str) –

  • region (Union[mypy_boto3_s3.literals.BucketLocationConstraintType, Literal[us-east-1]]) –

Return type

mypy_boto3_s3.service_resource.Bucket

toil.lib.aws.utils.enable_public_objects(bucket_name)

Enable a bucket to contain objects which are public.

This adjusts the bucket’s Public Access Block setting to not block all public access, and also adjusts the bucket’s Object Ownership setting to a setting which enables object ACLs.

Does not touch the account’s Public Access Block setting, which can also interfere here. That is probably best left to the account administrator.

This configuration used to be the default, and is what most of Toil’s code is written to expect, but it was changed so that new buckets default to the more restrictive setting <https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/>, with the expectation that people would write IAM policies for the buckets to allow public access if needed. Toil expects to be able to make arbitrary objects in arbitrary places public, and naming them all in an IAM policy would be a very awkward way to do it. So we restore the old behavior.

Parameters

bucket_name (str) –

Return type

None

toil.lib.aws.utils.get_bucket_region(bucket_name, endpoint_url=None, only_strategies=None)

Get the AWS region name associated with the given S3 bucket.

Takes an optional S3 API URL override.

Parameters

  • only_strategies (Optional[Set[int]]) – For testing, use only strategies with 1-based numbers in this set.

  • bucket_name (str) –

  • endpoint_url (Optional[str]) –

Return type

str

toil.lib.aws.utils.region_to_bucket_location(region)
Parameters

region (str) –

Return type

str

toil.lib.aws.utils.bucket_location_to_region(location)
Parameters

location (Optional[str]) –

Return type

str

toil.lib.aws.utils.get_object_for_url(url, existing=None)

Extracts a key (object) from a given parsed s3:// URL.

Parameters

  • existing (bool) – If True, key is expected to exist. If False, key is expected not to exists and it will be created. If None, the key will be created if it doesn’t exist.

  • url (urllib.parse.ParseResult) –

Return type

mypy_boto3_s3.service_resource.Object

toil.lib.aws.utils.list_objects_for_url(url)

Extracts a key (object) from a given parsed s3:// URL. The URL will be supplemented with a trailing slash if it is missing.

Parameters

url (urllib.parse.ParseResult) –

Return type

List[str]

toil.lib.aws.utils.flatten_tags(tags)

Convert tags from a key to value dict into a list of ‘Key’: xxx, ‘Value’: xxx dicts.

Parameters

tags (Dict[str, str]) –

Return type

List[Dict[str, str]]