toil.provisioners.aws

Submodules

Attributes

logger

ZoneTuple

Functions

get_aws_zone_from_spot_market(spotBid, nodeType, ...)

If a spot bid, node type, and Boto2 EC2 connection are specified, picks a

get_best_aws_zone([spotBid, nodeType, boto3_ec2, ...])

Get the right AWS zone to use.

choose_spot_zone(zones, bid, spot_history)

Returns the zone to put the spot request based on, in order of priority:

optimize_spot_bid(boto3_ec2, instance_type, spot_bid, ...)

Check whether the bid is in line with history and makes an effort to place

Package Contents

toil.provisioners.aws.logger
toil.provisioners.aws.ZoneTuple
toil.provisioners.aws.get_aws_zone_from_spot_market(spotBid, nodeType, boto3_ec2, zone_options)

If a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all.

In this case, zone_options can be used to restrict to a subset of the zones in the region.

Parameters:

  • spotBid (Optional[float])

  • nodeType (Optional[str])

  • boto3_ec2 (Optional[botocore.client.BaseClient])

  • zone_options (Optional[List[str]])

Return type:

Optional[str]

toil.provisioners.aws.get_best_aws_zone(spotBid=None, nodeType=None, boto3_ec2=None, zone_options=None)

Get the right AWS zone to use.

Reports the TOIL_AWS_ZONE environment variable if set.

Otherwise, if we are running on EC2 or ECS, reports the zone we are running in.

Otherwise, if a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all.

In this case, zone_options can be used to restrict to a subset of the zones in the region.

Otherwise, if we have the TOIL_AWS_REGION variable set, chooses a zone in that region.

Finally, if a default region is configured in Boto 2, chooses a zone in that region.

Returns None if no method can produce a zone to use.

Parameters:

  • spotBid (Optional[float])

  • nodeType (Optional[str])

  • boto3_ec2 (Optional[botocore.client.BaseClient])

  • zone_options (Optional[List[str]])

Return type:

Optional[str]

toil.provisioners.aws.choose_spot_zone(zones, bid, spot_history)

Returns the zone to put the spot request based on, in order of priority:

  1. zones with prices currently under the bid

  2. zones with the most stable price

Returns:

the name of the selected zone

Parameters:

  • zones (List[str])

  • bid (float)

  • spot_history (List[boto.ec2.spotpricehistory.SpotPriceHistory])

Return type:

str

>>> from collections import namedtuple
>>> FauxHistory = namedtuple('FauxHistory', ['price', 'availability_zone'])
>>> zones = ['us-west-2a', 'us-west-2b']
>>> spot_history = [FauxHistory(0.1, 'us-west-2a'),                         FauxHistory(0.2, 'us-west-2a'),                         FauxHistory(0.3, 'us-west-2b'),                         FauxHistory(0.6, 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2a'

>>> spot_history=[FauxHistory(0.3, 'us-west-2a'),                       FauxHistory(0.2, 'us-west-2a'),                       FauxHistory(0.1, 'us-west-2b'),                       FauxHistory(0.6, 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2b'

>>> spot_history=[FauxHistory(0.1, 'us-west-2a'),                       FauxHistory(0.7, 'us-west-2a'),                       FauxHistory(0.1, 'us-west-2b'),                       FauxHistory(0.6, 'us-west-2b')]
>>> choose_spot_zone(zones, 0.15, spot_history)
'us-west-2b'
toil.provisioners.aws.optimize_spot_bid(boto3_ec2, instance_type, spot_bid, zone_options)

Check whether the bid is in line with history and makes an effort to place the instance in a sensible zone.

Parameters:

  • zone_options (List[str]) – The collection of allowed zones to consider, within the region associated with the Boto2 connection.

  • boto3_ec2 (botocore.client.BaseClient)

  • instance_type (str)

  • spot_bid (float)