toil.lib.ec2nodes

Module Contents

Classes

InstanceType

Functions

is_number(s)	Determines if a unicode string (that may include commas) is a number.
parse_storage(storage_info)	Parses EC2 JSON storage param string into a number.
parse_memory(mem_info)	Returns EC2 'memory' string as a float.
download_region_json(filename[, region])	Downloads and writes the AWS Billing JSON to a file using the AWS pricing API.
reduce_region_json_size(filename)	Deletes information in the json file that we don't need, and rewrites it. This makes the file smaller.
updateStaticEC2Instances()	Generates a new python file of fetchable EC2 Instances by region with current prices and specs.

Attributes

logger
manager
dirname
region_json_dirname
EC2Regions

toil.lib.ec2nodes.logger

toil.lib.ec2nodes.manager

toil.lib.ec2nodes.dirname

toil.lib.ec2nodes.region_json_dirname

toil.lib.ec2nodes.EC2Regions

class toil.lib.ec2nodes.InstanceType(name, cores, memory, disks, disk_capacity, architecture)

Parameters:

• name (str)

• cores (int)

• memory (float)

• disks (float)

• disk_capacity (float)

• architecture (str)

__slots__ = ('name', 'cores', 'memory', 'disks', 'disk_capacity', 'architecture')

__str__()

Return str(self).

Return type:

str

__eq__(other)

Return self==value.

Parameters:

other (object)

Return type:

bool

toil.lib.ec2nodes.is_number(s)

Determines if a unicode string (that may include commas) is a number.

Parameters:

s (str) – Any unicode string.

Returns:

True if s represents a number, False otherwise.

Return type:

bool

toil.lib.ec2nodes.parse_storage(storage_info)

Parses EC2 JSON storage param string into a number.

Examples:

“2 x 160 SSD” “3 x 2000 HDD” “EBS only” “1 x 410” “8 x 1.9 NVMe SSD” “900 GB NVMe SSD”

Parameters:

storage_info (str) – EC2 JSON storage param string.

Returns:

Two floats representing: (# of disks), and (disk_capacity in GiB of each disk).

Return type:

Union[List[int], Tuple[Union[int, float], float]]

toil.lib.ec2nodes.parse_memory(mem_info)

Returns EC2 ‘memory’ string as a float.

Format should always be ‘#’ GiB (example: ‘244 GiB’ or ‘1,952 GiB’). Amazon loves to put commas in their numbers, so we have to accommodate that. If the syntax ever changes, this will raise.

Parameters:

mem_info (str) – EC2 JSON memory param string.

Returns:

A float representing memory in GiB.

Return type:

float

toil.lib.ec2nodes.download_region_json(filename, region='us-east-1')

Downloads and writes the AWS Billing JSON to a file using the AWS pricing API.

See: https://aws.amazon.com/blogs/aws/new-aws-price-list-api/

Returns:

A dict of InstanceType objects, where the key is the string: aws instance name (example: ‘t2.micro’), and the value is an InstanceType object representing that aws instance name.

Parameters:

• filename (str)

• region (str)

Return type:

None

toil.lib.ec2nodes.reduce_region_json_size(filename)

Deletes information in the json file that we don’t need, and rewrites it. This makes the file smaller.

The reason being: we used to download the unified AWS Bulk API JSON, which eventually crept up to 5.6Gb, the loading of which could not be done on a 32Gb RAM machine. Now we download each region JSON individually (with AWS’s new Query API), but even those may eventually one day grow ridiculously large, so we do what we can to keep the file sizes down (and thus also the amount loaded into memory) to keep this script working for longer.

Parameters:

filename (str)

Return type:

List[Dict[str, Any]]

toil.lib.ec2nodes.updateStaticEC2Instances()

Generates a new python file of fetchable EC2 Instances by region with current prices and specs.

Takes a few (~3+) minutes to run (you’ll need decent internet).

Returns:

Nothing. Writes a new ‘generatedEC2Lists.py’ file.

Return type:

None