toil.jobStores.aws.utils

Attributes

logger

DIAL_SPECIFIC_REGION_CONFIG

Exceptions

ServerSideCopyProhibitedError

Raised when AWS refuses to perform a server-side copy between S3 keys, and

Functions

fileSizeAndTime(localFilePath)

uploadFromPath(localFilePath, resource, bucketName, fileID)

Uploads a file to s3, using multipart uploading if applicable

uploadFile(readable, resource, bucketName, fileID[, ...])

Upload a readable object to s3, using multipart uploading if applicable.

copyKeyMultipart(resource, srcBucketName, srcKeyName, ...)

Copies a key from a source key to a destination key in multiple parts. Note that if the

retryable_ssl_error(e)

Module Contents

toil.jobStores.aws.utils.logger
toil.jobStores.aws.utils.DIAL_SPECIFIC_REGION_CONFIG
toil.jobStores.aws.utils.fileSizeAndTime(localFilePath)
Parameters:

localFilePath (str)

Return type:

tuple[int, float]

toil.jobStores.aws.utils.uploadFromPath(localFilePath, resource, bucketName, fileID, headerArgs=None, partSize=50 << 20)

Uploads a file to s3, using multipart uploading if applicable

Parameters:

  • localFilePath (str) – Path of the file to upload to s3

  • resource (S3.Resource) – boto3 resource

  • bucketName (str) – name of the bucket to upload to

  • fileID (str) – the name of the file to upload to

  • headerArgs (dict) – http headers to use when uploading - generally used for encryption purposes

  • partSize (int) – max size of each part in the multipart upload, in bytes

Returns:

version of the newly uploaded file

Return type:

str | None

toil.jobStores.aws.utils.uploadFile(readable, resource, bucketName, fileID, headerArgs=None, partSize=50 << 20)

Upload a readable object to s3, using multipart uploading if applicable. :param readable: a readable stream or a file path to upload to s3 :param S3.Resource resource: boto3 resource :param str bucketName: name of the bucket to upload to :param str fileID: the name of the file to upload to :param dict headerArgs: http headers to use when uploading - generally used for encryption purposes :param int partSize: max size of each part in the multipart upload, in bytes :return: version of the newly uploaded file

Parameters:

  • readable (IO[bytes])

  • resource (mypy_boto3_s3.S3ServiceResource)

  • bucketName (str)

  • fileID (str)

  • headerArgs (dict[str, Any] | None)

  • partSize (int)

Return type:

str | None

exception toil.jobStores.aws.utils.ServerSideCopyProhibitedError

Bases: RuntimeError

Raised when AWS refuses to perform a server-side copy between S3 keys, and insists that you pay to download and upload the data yourself instead.

toil.jobStores.aws.utils.copyKeyMultipart(resource, srcBucketName, srcKeyName, srcKeyVersion, dstBucketName, dstKeyName, sseAlgorithm=None, sseKey=None, copySourceSseAlgorithm=None, copySourceSseKey=None)

Copies a key from a source key to a destination key in multiple parts. Note that if the destination key exists it will be overwritten implicitly, and if it does not exist a new key will be created. If the destination bucket does not exist an error will be raised.

This function will always do a fast, server-side copy, at least until/unless <https://github.com/boto/boto3/issues/3270> is fixed. In some situations, a fast, server-side copy is not actually possible. For example, when residing in an AWS VPC with an S3 VPC Endpoint configured, copying from a bucket in another region to a bucket in your own region cannot be performed server-side. This is because the VPC Endpoint S3 API servers refuse to perform server-side copies between regions, the source region’s API servers refuse to initiate the copy and refer you to the destination bucket’s region’s API servers, and the VPC routing tables are configured to redirect all access to the current region’s S3 API servers to the S3 Endpoint API servers instead.

If a fast server-side copy is not actually possible, a ServerSideCopyProhibitedError will be raised.

Parameters:

  • resource (mypy_boto3_s3.S3ServiceResource) – boto3 resource

  • srcBucketName (str) – The name of the bucket to be copied from.

  • srcKeyName (str) – The name of the key to be copied from.

  • srcKeyVersion (str) – The version of the key to be copied from.

  • dstBucketName (str) – The name of the destination bucket for the copy.

  • dstKeyName (str) – The name of the destination key that will be created or overwritten.

  • sseAlgorithm (str) – Server-side encryption algorithm for the destination.

  • sseKey (str) – Server-side encryption key for the destination.

  • copySourceSseAlgorithm (str) – Server-side encryption algorithm for the source.

  • copySourceSseKey (str) – Server-side encryption key for the source.

Returns:

The version of the copied file (or None if versioning is not enabled for dstBucket).

Return type:

str | None

toil.jobStores.aws.utils.retryable_ssl_error(e)
Parameters:

e (BaseException)

Return type:

bool