toil.lib.ec2

Module Contents

Functions

not_found(e)

inconsistencies_detected(e)

retry_ec2([t, retry_for, retry_while])

wait_transition(resource, from_states, to_state[, ...])

Wait until the specified EC2 resource (instance, image, volume, ...) transitions from any

wait_instances_running(ec2, instances)

Wait until no instance in the given iterable is 'pending'. Yield every instance that

wait_spot_requests_active(ec2, requests[, timeout, ...])

Wait until no spot request in the given iterator is in the 'open' state or, optionally,

create_spot_instances(ec2, price, image_id, spec[, ...])

Create instances on the spot market.

create_ondemand_instances(ec2, image_id, spec[, ...])

Requests the RunInstances EC2 API call but accounts for the race between recently created

prune(bushy)

Prune entries in the given dict with false-y values.

wait_until_instance_profile_arn_exists(...)

create_instances(ec2_resource, image_id, key_name, ...)

Replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts.

create_launch_template(ec2_client, template_name, ...)

Creates a launch template with the given name for launching instances with the given parameters.

create_auto_scaling_group(autoscaling_client, ...[, ...])

Create a new Auto Scaling Group with the given name (which is also its

Attributes

a_short_time

a_long_time

logger

INCONSISTENCY_ERRORS

iam_client

toil.lib.ec2.a_short_time = 5
toil.lib.ec2.a_long_time
toil.lib.ec2.logger
exception toil.lib.ec2.UserError(message=None, cause=None)

Bases: RuntimeError

Unspecified run-time error.

toil.lib.ec2.not_found(e)
toil.lib.ec2.inconsistencies_detected(e)
toil.lib.ec2.INCONSISTENCY_ERRORS
toil.lib.ec2.retry_ec2(t=a_short_time, retry_for=10 * a_short_time, retry_while=not_found)
exception toil.lib.ec2.UnexpectedResourceState(resource, to_state, state)

Bases: Exception

Common base class for all non-exit exceptions.

toil.lib.ec2.wait_transition(resource, from_states, to_state, state_getter=attrgetter('state'))

Wait until the specified EC2 resource (instance, image, volume, …) transitions from any of the given ‘from’ states to the specified ‘to’ state. If the instance is found in a state other that the to state or any of the from states, an exception will be thrown.

Parameters:

  • resource – the resource to monitor

  • from_states – a set of states that the resource is expected to be in before the transition occurs

  • to_state – the state of the resource when this method returns

toil.lib.ec2.wait_instances_running(ec2, instances)

Wait until no instance in the given iterable is ‘pending’. Yield every instance that entered the running state as soon as it does.

Parameters:

  • ec2 (boto.ec2.connection.EC2Connection) – the EC2 connection to use for making requests

  • instances (Iterable[Boto2Instance]) – the instances to wait on

Return type:

Iterable[Boto2Instance]

toil.lib.ec2.wait_spot_requests_active(ec2, requests, timeout=None, tentative=False)

Wait until no spot request in the given iterator is in the ‘open’ state or, optionally, a timeout occurs. Yield spot requests as soon as they leave the ‘open’ state.

Parameters:

  • requests (Iterable[boto.ec2.spotinstancerequest.SpotInstanceRequest]) – The requests to wait on.

  • timeout (float) – Maximum time in seconds to spend waiting or None to wait forever. If a timeout occurs, the remaining open requests will be cancelled.

  • tentative (bool) – if True, give up on a spot request at the earliest indication of it not being fulfilled immediately

Return type:

Iterable[List[boto.ec2.spotinstancerequest.SpotInstanceRequest]]

toil.lib.ec2.create_spot_instances(ec2, price, image_id, spec, num_instances=1, timeout=None, tentative=False, tags=None)

Create instances on the spot market.

Return type:

Iterable[List[boto.ec2.instance.Instance]]

toil.lib.ec2.create_ondemand_instances(ec2, image_id, spec, num_instances=1)

Requests the RunInstances EC2 API call but accounts for the race between recently created instance profiles, IAM roles and an instance creation that refers to them.

Return type:

List[Boto2Instance]

toil.lib.ec2.prune(bushy)

Prune entries in the given dict with false-y values. Boto3 may not like None and instead wants no key.

Parameters:

bushy (dict) –

Return type:

dict

toil.lib.ec2.iam_client
toil.lib.ec2.wait_until_instance_profile_arn_exists(instance_profile_arn)
Parameters:

instance_profile_arn (str) –

toil.lib.ec2.create_instances(ec2_resource, image_id, key_name, instance_type, num_instances=1, security_group_ids=None, user_data=None, block_device_map=None, instance_profile_arn=None, placement_az=None, subnet_id=None, tags=None)

Replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts.

See “create_instances” (returns a list of ec2.Instance objects):

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances

Not to be confused with “run_instances” (same input args; returns a dictionary):

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.run_instances

Tags, if given, are applied to the instances, and all volumes.

Parameters:

  • ec2_resource (boto3.resources.base.ServiceResource) –

  • image_id (str) –

  • key_name (str) –

  • instance_type (str) –

  • num_instances (int) –

  • security_group_ids (Optional[List]) –

  • user_data (Optional[Union[str, bytes]]) –

  • block_device_map (Optional[List[Dict]]) –

  • instance_profile_arn (Optional[str]) –

  • placement_az (Optional[str]) –

  • subnet_id (str) –

  • tags (Optional[Dict[str, str]]) –

Return type:

List[dict]

toil.lib.ec2.create_launch_template(ec2_client, template_name, image_id, key_name, instance_type, security_group_ids=None, user_data=None, block_device_map=None, instance_profile_arn=None, placement_az=None, subnet_id=None, tags=None)

Creates a launch template with the given name for launching instances with the given parameters.

We only ever use the default version of any launch template.

Internally calls https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html?highlight=create_launch_template#EC2.Client.create_launch_template

Parameters:

  • tags (Optional[Dict[str, str]]) – Tags, if given, are applied to the template itself, all instances, and all volumes.

  • user_data (Optional[Union[str, bytes]]) – non-base64-encoded user data to pass to the instances.

  • ec2_client (botocore.client.BaseClient) –

  • template_name (str) –

  • image_id (str) –

  • key_name (str) –

  • instance_type (str) –

  • security_group_ids (Optional[List]) –

  • block_device_map (Optional[List[Dict]]) –

  • instance_profile_arn (Optional[str]) –

  • placement_az (Optional[str]) –

  • subnet_id (Optional[str]) –

Returns:

the ID of the launch template.

Return type:

str

toil.lib.ec2.create_auto_scaling_group(autoscaling_client, asg_name, launch_template_ids, vpc_subnets, min_size, max_size, instance_types=None, spot_bid=None, spot_cheapest=False, tags=None)

Create a new Auto Scaling Group with the given name (which is also its unique identifier).

Parameters:

  • autoscaling_client (botocore.client.BaseClient) – Boto3 client for autoscaling.

  • asg_name (str) – Unique name for the autoscaling group.

  • launch_template_ids (Dict[str, str]) – ID of the launch template to make instances from, for each instance type.

  • vpc_subnets (List[str]) – One or more subnet IDs to place instances in the group into. Determine the availability zone(s) instances will launch into.

  • min_size (int) – Minimum number of instances to have in the group at all times.

  • max_size (int) – Maximum number of instances to allow in the group at any time.

  • instance_types (Optional[List[str]]) – Use a pool over the given instance types, instead of the type given in the launch template. For on-demand groups, this is a prioritized list. For spot groups, we let AWS balance according to spot_strategy. Must be 20 types or shorter.

  • spot_bid (Optional[float]) – If set, the ASG will be a spot market ASG. Bid is in dollars per instance hour. All instance types in the group are bid on equivalently.

  • spot_cheapest (bool) – If true, use the cheapest spot instances available out of instance_types, instead of the spot instances that minimize eviction probability.

  • tags (Optional[Dict[str, str]]) – Tags to apply to the ASG only. Tags for the instances should be added to the launch template instead.

Return type:

None

The default version of the launch template is used.