toil.lib.aws.utils
¶
Module Contents¶
Functions¶
|
|
|
|
|
|
Return true if an error is a connection reset error. |
|
Return true if this is an error from S3 that looks like we ought to retry our request. |
|
|
Retry iterator of context managers specifically for S3 operations. |
|
Delete the given S3 bucket. |
|
Create an AWS S3 bucket, using the given Boto3 S3 session, with the |
|
Enable a bucket to contain objects which are public. |
|
Get the AWS region name associated with the given S3 bucket. |
|
|
|
|
|
Extracts a key (object) from a given parsed s3:// URL. |
|
Extracts a key (object) from a given parsed s3:// URL. The URL will be |
|
Convert tags from a key to value dict into a list of 'Key': xxx, 'Value': xxx dicts. |
Attributes¶
- toil.lib.aws.utils.BotoServerError¶
- toil.lib.aws.utils.logger¶
- toil.lib.aws.utils.THROTTLED_ERROR_CODES = ['Throttling', 'ThrottlingException', 'ThrottledException', 'RequestThrottledException',...¶
- toil.lib.aws.utils.delete_iam_role(role_name, region=None, quiet=True)¶
- toil.lib.aws.utils.delete_iam_instance_profile(instance_profile_name, region=None, quiet=True)¶
- toil.lib.aws.utils.delete_sdb_domain(sdb_domain_name, region=None, quiet=True)¶
- toil.lib.aws.utils.connection_reset(e)¶
Return true if an error is a connection reset error.
- toil.lib.aws.utils.retryable_s3_errors(e)¶
Return true if this is an error from S3 that looks like we ought to retry our request.
- toil.lib.aws.utils.retry_s3(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=retryable_s3_errors)¶
Retry iterator of context managers specifically for S3 operations.
- toil.lib.aws.utils.delete_s3_bucket(s3_resource, bucket, quiet=True)¶
Delete the given S3 bucket.
- toil.lib.aws.utils.create_s3_bucket(s3_resource, bucket_name, region)¶
Create an AWS S3 bucket, using the given Boto3 S3 session, with the given name, in the given region.
Supports the us-east-1 region, where bucket creation is special.
ALL S3 bucket creation should use this function.
- Parameters:
s3_resource (mypy_boto3_s3.S3ServiceResource)
bucket_name (str)
region (toil.lib.aws.AWSRegionName)
- Return type:
mypy_boto3_s3.service_resource.Bucket
- toil.lib.aws.utils.enable_public_objects(bucket_name)¶
Enable a bucket to contain objects which are public.
This adjusts the bucket’s Public Access Block setting to not block all public access, and also adjusts the bucket’s Object Ownership setting to a setting which enables object ACLs.
Does not touch the account’s Public Access Block setting, which can also interfere here. That is probably best left to the account administrator.
This configuration used to be the default, and is what most of Toil’s code is written to expect, but it was changed so that new buckets default to the more restrictive setting <https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/>, with the expectation that people would write IAM policies for the buckets to allow public access if needed. Toil expects to be able to make arbitrary objects in arbitrary places public, and naming them all in an IAM policy would be a very awkward way to do it. So we restore the old behavior.
- Parameters:
bucket_name (str)
- Return type:
None
- toil.lib.aws.utils.get_bucket_region(bucket_name, endpoint_url=None, only_strategies=None)¶
Get the AWS region name associated with the given S3 bucket.
Takes an optional S3 API URL override.
- toil.lib.aws.utils.bucket_location_to_region(location)¶
- toil.lib.aws.utils.get_object_for_url(url, existing=None)¶
Extracts a key (object) from a given parsed s3:// URL.
If existing is true and the object does not exist, raises FileNotFoundError.
- Parameters:
existing (bool) – If True, key is expected to exist. If False, key is expected not to exists and it will be created. If None, the key will be created if it doesn’t exist.
url (urllib.parse.ParseResult)
- Return type:
mypy_boto3_s3.service_resource.Object
- toil.lib.aws.utils.list_objects_for_url(url)¶
Extracts a key (object) from a given parsed s3:// URL. The URL will be supplemented with a trailing slash if it is missing.
- Parameters:
url (urllib.parse.ParseResult)
- Return type:
List[str]