toil.provisioners.aws

Submodules

Package Contents

Functions

get_aws_zone_from_boto()

Get the AWS zone from the Boto config file, if it is configured and the

get_aws_zone_from_environment()

Get the AWS zone from TOIL_AWS_ZONE if set.

get_aws_zone_from_environment_region()

Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set.

get_aws_zone_from_metadata()

Get the AWS zone from instance metadata, if on EC2 and the boto module is

get_aws_zone_from_spot_market(spotBid, nodeType, ...)

If a spot bid, node type, and Boto2 EC2 connection are specified, picks a

get_best_aws_zone([spotBid, nodeType, boto2_ec2, ...])

Get the right AWS zone to use.

choose_spot_zone(zones, bid, spot_history)

Returns the zone to put the spot request based on, in order of priority:

optimize_spot_bid(boto2_ec2, instance_type, spot_bid, ...)

Check whether the bid is in line with history and makes an effort to place

Attributes

logger

ZoneTuple

toil.provisioners.aws.get_aws_zone_from_boto()[source]

Get the AWS zone from the Boto config file, if it is configured and the boto module is available.

Return type:

Optional[str]

toil.provisioners.aws.get_aws_zone_from_environment()[source]

Get the AWS zone from TOIL_AWS_ZONE if set.

Return type:

Optional[str]

toil.provisioners.aws.get_aws_zone_from_environment_region()[source]

Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set.

Return type:

Optional[str]

toil.provisioners.aws.get_aws_zone_from_metadata()[source]

Get the AWS zone from instance metadata, if on EC2 and the boto module is available. Otherwise, gets the AWS zone from ECS task metadata, if on ECS.

Return type:

Optional[str]

toil.provisioners.aws.logger
toil.provisioners.aws.ZoneTuple
toil.provisioners.aws.get_aws_zone_from_spot_market(spotBid, nodeType, boto2_ec2, zone_options)[source]

If a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all.

In this case, zone_options can be used to restrict to a subset of the zones in the region.

Parameters:

  • spotBid (Optional[float])

  • nodeType (Optional[str])

  • boto2_ec2 (Optional[boto.connection.AWSAuthConnection])

  • zone_options (Optional[List[str]])

Return type:

Optional[str]

toil.provisioners.aws.get_best_aws_zone(spotBid=None, nodeType=None, boto2_ec2=None, zone_options=None)[source]

Get the right AWS zone to use.

Reports the TOIL_AWS_ZONE environment variable if set.

Otherwise, if we are running on EC2 or ECS, reports the zone we are running in.

Otherwise, if a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all.

In this case, zone_options can be used to restrict to a subset of the zones in the region.

Otherwise, if we have the TOIL_AWS_REGION variable set, chooses a zone in that region.

Finally, if a default region is configured in Boto 2, chooses a zone in that region.

Returns None if no method can produce a zone to use.

Parameters:

  • spotBid (Optional[float])

  • nodeType (Optional[str])

  • boto2_ec2 (Optional[boto.connection.AWSAuthConnection])

  • zone_options (Optional[List[str]])

Return type:

Optional[str]

toil.provisioners.aws.choose_spot_zone(zones, bid, spot_history)[source]

Returns the zone to put the spot request based on, in order of priority:

  1. zones with prices currently under the bid

  2. zones with the most stable price

Returns:

the name of the selected zone

Parameters:

  • zones (List[str])

  • bid (float)

  • spot_history (List[boto.ec2.spotpricehistory.SpotPriceHistory])

Return type:

str

>>> from collections import namedtuple
>>> FauxHistory = namedtuple('FauxHistory', ['price', 'availability_zone'])
>>> zones = ['us-west-2a', 'us-west-2b']
>>> spot_history = [FauxHistory(0.1, 'us-west-2a'),                         FauxHistory(0.2, 'us-west-2a'),                         FauxHistory(0.3, 'us-west-2b'),                         FauxHistory(0.6, 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2a'

>>> spot_history=[FauxHistory(0.3, 'us-west-2a'),                       FauxHistory(0.2, 'us-west-2a'),                       FauxHistory(0.1, 'us-west-2b'),                       FauxHistory(0.6, 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2b'

>>> spot_history=[FauxHistory(0.1, 'us-west-2a'),                       FauxHistory(0.7, 'us-west-2a'),                       FauxHistory(0.1, 'us-west-2b'),                       FauxHistory(0.6, 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2b'
toil.provisioners.aws.optimize_spot_bid(boto2_ec2, instance_type, spot_bid, zone_options)[source]

Check whether the bid is in line with history and makes an effort to place the instance in a sensible zone.

Parameters:

zone_options (List[str]) – The collection of allowed zones to consider, within the region associated with the Boto2 connection.