toil.lib.ec2

Module Contents

Functions

not_found(e)

inconsistencies_detected(e)

retry_ec2([t, retry_for, retry_while])

wait_transition(resource, from_states, to_state[, ...])

Wait until the specified EC2 resource (instance, image, volume, ...) transitions from any

wait_instances_running(ec2, instances)

Wait until no instance in the given iterable is 'pending'. Yield every instance that

wait_spot_requests_active(ec2, requests[, timeout, ...])

Wait until no spot request in the given iterator is in the 'open' state or, optionally,

create_spot_instances(ec2, price, image_id, spec[, ...])

Create instances on the spot market.

create_ondemand_instances(ec2, image_id, spec[, ...])

Requests the RunInstances EC2 API call but accounts for the race between recently created

prune(bushy)

Prune entries in the given dict with false-y values.

wait_until_instance_profile_arn_exists(...)

create_instances(ec2_resource, image_id, key_name, ...)

Replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts.

create_launch_template(ec2_client, template_name, ...)

Creates a launch template with the given name for launching instances with the given parameters.

create_auto_scaling_group(autoscaling_client, ...[, ...])

Create a new Auto Scaling Group with the given name (which is also its

Attributes

a_short_time

a_long_time

logger

INCONSISTENCY_ERRORS

iam_client

toil.lib.ec2.a_short_time = 5
toil.lib.ec2.a_long_time
toil.lib.ec2.logger
exception toil.lib.ec2.UserError(message=None, cause=None)

Bases: RuntimeError

digraph inheritance753713cd36 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "UserError" [URL="#toil.lib.ec2.UserError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; }

Unspecified run-time error.

toil.lib.ec2.not_found(e)
toil.lib.ec2.inconsistencies_detected(e)
toil.lib.ec2.INCONSISTENCY_ERRORS
toil.lib.ec2.retry_ec2(t=a_short_time, retry_for=10 * a_short_time, retry_while=not_found)
exception toil.lib.ec2.UnexpectedResourceState(resource, to_state, state)

Bases: Exception

digraph inheritancee12ede5ae8 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "UnexpectedResourceState" [URL="#toil.lib.ec2.UnexpectedResourceState",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; }

Common base class for all non-exit exceptions.

toil.lib.ec2.wait_transition(resource, from_states, to_state, state_getter=attrgetter('state'))

Wait until the specified EC2 resource (instance, image, volume, …) transitions from any of the given ‘from’ states to the specified ‘to’ state. If the instance is found in a state other that the to state or any of the from states, an exception will be thrown.

Parameters

  • resource – the resource to monitor

  • from_states – a set of states that the resource is expected to be in before the transition occurs

  • to_state – the state of the resource when this method returns

toil.lib.ec2.wait_instances_running(ec2, instances)

Wait until no instance in the given iterable is ‘pending’. Yield every instance that entered the running state as soon as it does.

Parameters

  • ec2 (boto.ec2.connection.EC2Connection) – the EC2 connection to use for making requests

  • instances (Iterable[Boto2Instance]) – the instances to wait on

Return type

Iterable[Boto2Instance]

toil.lib.ec2.wait_spot_requests_active(ec2, requests, timeout=None, tentative=False)

Wait until no spot request in the given iterator is in the ‘open’ state or, optionally, a timeout occurs. Yield spot requests as soon as they leave the ‘open’ state.

Parameters

  • requests (Iterable[boto.ec2.spotinstancerequest.SpotInstanceRequest]) – The requests to wait on.

  • timeout (float) – Maximum time in seconds to spend waiting or None to wait forever. If a

  • tentative (bool) –

Return type

Iterable[List[boto.ec2.spotinstancerequest.SpotInstanceRequest]]

timeout occurs, the remaining open requests will be cancelled.

Parameters

  • tentative (bool) – if True, give up on a spot request at the earliest indication of it

  • requests (Iterable[boto.ec2.spotinstancerequest.SpotInstanceRequest]) –

  • timeout (float) –

Return type

Iterable[List[boto.ec2.spotinstancerequest.SpotInstanceRequest]]

not being fulfilled immediately

toil.lib.ec2.create_spot_instances(ec2, price, image_id, spec, num_instances=1, timeout=None, tentative=False, tags=None)

Create instances on the spot market.

Return type

Iterable[List[boto.ec2.instance.Instance]]

toil.lib.ec2.create_ondemand_instances(ec2, image_id, spec, num_instances=1)

Requests the RunInstances EC2 API call but accounts for the race between recently created instance profiles, IAM roles and an instance creation that refers to them.

Return type

List[Boto2Instance]

toil.lib.ec2.prune(bushy)

Prune entries in the given dict with false-y values. Boto3 may not like None and instead wants no key.

Parameters

bushy (dict) –

Return type

dict

toil.lib.ec2.iam_client
toil.lib.ec2.wait_until_instance_profile_arn_exists(instance_profile_arn)
Parameters

instance_profile_arn (str) –

toil.lib.ec2.create_instances(ec2_resource, image_id, key_name, instance_type, num_instances=1, security_group_ids=None, user_data=None, block_device_map=None, instance_profile_arn=None, placement_az=None, subnet_id=None, tags=None)

Replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts.

See “create_instances” (returns a list of ec2.Instance objects):

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances

Not to be confused with “run_instances” (same input args; returns a dictionary):

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.run_instances

Tags, if given, are applied to the instances, and all volumes.

Parameters

  • ec2_resource (boto3.resources.base.ServiceResource) –

  • image_id (str) –

  • key_name (str) –

  • instance_type (str) –

  • num_instances (int) –

  • security_group_ids (Optional[List]) –

  • user_data (Optional[Union[str, bytes]]) –

  • block_device_map (Optional[List[Dict]]) –

  • instance_profile_arn (Optional[str]) –

  • placement_az (Optional[str]) –

  • subnet_id (str) –

  • tags (Optional[Dict[str, str]]) –

Return type

List[dict]

toil.lib.ec2.create_launch_template(ec2_client, template_name, image_id, key_name, instance_type, security_group_ids=None, user_data=None, block_device_map=None, instance_profile_arn=None, placement_az=None, subnet_id=None, tags=None)

Creates a launch template with the given name for launching instances with the given parameters.

We only ever use the default version of any launch template.

Internally calls https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html?highlight=create_launch_template#EC2.Client.create_launch_template

Parameters

  • tags (Optional[Dict[str, str]]) – Tags, if given, are applied to the template itself, all instances, and all volumes.

  • user_data (Optional[Union[str, bytes]]) – non-base64-encoded user data to pass to the instances.

  • ec2_client (botocore.client.BaseClient) –

  • template_name (str) –

  • image_id (str) –

  • key_name (str) –

  • instance_type (str) –

  • security_group_ids (Optional[List]) –

  • block_device_map (Optional[List[Dict]]) –

  • instance_profile_arn (Optional[str]) –

  • placement_az (Optional[str]) –

  • subnet_id (Optional[str]) –

Returns

the ID of the launch template.

Return type

str

toil.lib.ec2.create_auto_scaling_group(autoscaling_client, asg_name, launch_template_ids, vpc_subnets, min_size, max_size, instance_types=None, spot_bid=None, spot_cheapest=False, tags=None)

Create a new Auto Scaling Group with the given name (which is also its unique identifier).

Parameters

  • autoscaling_client (botocore.client.BaseClient) – Boto3 client for autoscaling.

  • asg_name (str) – Unique name for the autoscaling group.

  • launch_template_ids (Dict[str, str]) – ID of the launch template to make instances from, for each instance type.

  • vpc_subnets (List[str]) – One or more subnet IDs to place instances in the group into. Determine the availability zone(s) instances will launch into.

  • min_size (int) – Minimum number of instances to have in the group at all times.

  • max_size (int) – Maximum number of instances to allow in the group at any time.

  • instance_types (Optional[List[str]]) – Use a pool over the given instance types, instead of the type given in the launch template. For on-demand groups, this is a prioritized list. For spot groups, we let AWS balance according to spot_strategy. Must be 20 types or shorter.

  • spot_bid (Optional[float]) – If set, the ASG will be a spot market ASG. Bid is in dollars per instance hour. All instance types in the group are bid on equivalently.

  • spot_cheapest (bool) – If true, use the cheapest spot instances available out of instance_types, instead of the spot instances that minimize eviction probability.

  • tags (Optional[Dict[str, str]]) – Tags to apply to the ASG only. Tags for the instances should be added to the launch template instead.

Return type

None

The default version of the launch template is used.