toil.lib.aws.ami

Attributes

logger

Functions

retry([intervals, infinite_retries, errors, ...])

Retry a function if it fails with any Exception defined in "errors".

get_flatcar_ami(ec2_client[, architecture])

Retrieve the flatcar AMI image to use as the base for all Toil autoscaling instances.

flatcar_release_feed_amis(region[, architecture, source])

Yield AMI IDs for the given architecture from the Flatcar release feed.

feed_flatcar_ami_release(ec2_client[, architecture, ...])

Check a Flatcar release feed for the latest flatcar AMI.

aws_marketplace_flatcar_ami_search(ec2_client[, ...])

Query AWS for all AMI names matching Flatcar-stable-* and return the most recent one.

Module Contents

toil.lib.aws.ami.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)

Retry a function if it fails with any Exception defined in “errors”.

Does so every x seconds, where x is defined by a list of numbers (ints or floats) in “intervals”. Also accepts ErrorCondition events for more detailed retry attempts.

Parameters:
  • intervals (Optional[List]) – A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s

  • infinite_retries (bool) – If this is True, reset the intervals when they run out. Defaults to: False.

  • errors (Optional[Sequence[Union[ErrorCondition, Type[Exception]]]]) –

    A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: - Exception (required) - Error codes that must match to be retried (optional; defaults to not checking) - A string that must be in the error message to be retried (optional; defaults to not checking) - A bool that can be set to False to always error on this condition.

    If not specified, this will default to a generic Exception.

  • log_message (Optional[Tuple[Callable, str]]) – Optional tuple of (“log/print function()”, “message string”) that will precede each attempt.

  • prepare (Optional[List[Callable]]) – Optional list of functions to call, with the function’s arguments, between retries, to reset state.

Returns:

The result of the wrapped function or raise.

Return type:

Callable[[Callable[Ellipsis, RT]], Callable[Ellipsis, RT]]

toil.lib.aws.ami.logger
toil.lib.aws.ami.get_flatcar_ami(ec2_client, architecture='amd64')

Retrieve the flatcar AMI image to use as the base for all Toil autoscaling instances.

AMI must be available to the user on AWS (attempting to launch will return a 403 otherwise).

Priority is:
  1. User specified AMI via TOIL_AWS_AMI

  2. Official AMI from stable.release.flatcar-linux.net

  3. Search the AWS Marketplace

If all of these sources fail, we raise an error to complain.

Parameters:
  • ec2_client (botocore.client.BaseClient) – Boto3 EC2 Client

  • architecture (str) – The architecture type for the new AWS machine. Can be either amd64 or arm64

Return type:

str

toil.lib.aws.ami.flatcar_release_feed_amis(region, architecture='amd64', source='stable')

Yield AMI IDs for the given architecture from the Flatcar release feed.

Parameters:
  • source (str) – can be set to a Flatcar release channel (‘stable’, ‘beta’, or ‘alpha’), ‘archive’ to check the Internet Archive for a feed, and ‘toil’ to check if the Toil project has put up a feed.

  • region (str)

  • architecture (str)

Return type:

Iterator[str]

Retries if the release feed cannot be fetched. If the release feed has a permanent error, yields nothing. If some entries in the release feed are unparseable, yields the others.

toil.lib.aws.ami.feed_flatcar_ami_release(ec2_client, architecture='amd64', source='stable')

Check a Flatcar release feed for the latest flatcar AMI.

Verify it’s on AWS.

Parameters:
  • ec2_client (botocore.client.BaseClient) – Boto3 EC2 Client

  • architecture (str) – The architecture type for the new AWS machine. Can be either amd64 or arm64

  • source (str) – can be set to a Flatcar release channel (‘stable’, ‘beta’, or ‘alpha’), ‘archive’ to check the Internet Archive for a feed, and ‘toil’ to check if the Toil project has put up a feed.

Return type:

Optional[str]

Query AWS for all AMI names matching Flatcar-stable-* and return the most recent one.

Parameters:
  • ec2_client (botocore.client.BaseClient)

  • architecture (str)

Return type:

Optional[str]