toil.provisioners.aws.awsProvisioner

Module Contents

Classes

AWSProvisioner

Interface for provisioning worker nodes to use in a Toil cluster.

Functions

awsRetryPredicate(e)

expectedShutdownErrors(e)

Matches errors that we expect to occur during shutdown, and which indicate

awsRetry(f)

This decorator retries the wrapped function if aws throws unexpected errors.

awsFilterImpairedNodes(nodes, boto3_ec2)

collapse_tags(instance_tags)

Collapse tags from boto3 format to node format

Attributes

logger

F

toil.provisioners.aws.awsProvisioner.logger
toil.provisioners.aws.awsProvisioner.awsRetryPredicate(e)
Parameters:

e (Exception)

Return type:

bool

toil.provisioners.aws.awsProvisioner.expectedShutdownErrors(e)

Matches errors that we expect to occur during shutdown, and which indicate that we need to wait or try again.

Should not match any errors which indicate that an operation is impossible or unnecessary (such as errors resulting from a thing not existing to be deleted).

Parameters:

e (Exception)

Return type:

bool

toil.provisioners.aws.awsProvisioner.F
toil.provisioners.aws.awsProvisioner.awsRetry(f)

This decorator retries the wrapped function if aws throws unexpected errors.

It should wrap any function that makes use of boto

Parameters:

f (Callable[Ellipsis, F])

Return type:

Callable[Ellipsis, F]

toil.provisioners.aws.awsProvisioner.awsFilterImpairedNodes(nodes, boto3_ec2)
Parameters:

  • nodes (List[mypy_boto3_ec2.type_defs.InstanceTypeDef])

  • boto3_ec2 (mypy_boto3_ec2.client.EC2Client)

Return type:

List[mypy_boto3_ec2.type_defs.InstanceTypeDef]

exception toil.provisioners.aws.awsProvisioner.InvalidClusterStateException

Bases: Exception

Common base class for all non-exit exceptions.

toil.provisioners.aws.awsProvisioner.collapse_tags(instance_tags)

Collapse tags from boto3 format to node format :param instance_tags: tags as list of TagTypeDef :return: Dict of tags

Parameters:

instance_tags (List[mypy_boto3_ec2.type_defs.TagTypeDef])

Return type:

Dict[str, str]

class toil.provisioners.aws.awsProvisioner.AWSProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey, enable_fuse)

Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner

Interface for provisioning worker nodes to use in a Toil cluster.

Parameters:

  • clusterName (Optional[str])

  • clusterType (Optional[str])

  • zone (Optional[str])

  • nodeStorage (int)

  • nodeStorageOverrides (Optional[List[str]])

  • sseKey (Optional[str])

  • enable_fuse (bool)

supportedClusterTypes()

Get all the cluster types that this provisioner implementation supports.

Return type:

Set[str]

createClusterSettings()

Create a new set of cluster settings for a cluster to be deployed into AWS.

Return type:

None

readClusterSettings()

Reads the cluster settings from the instance metadata, which assumes the instance is the leader.

Return type:

None

launchCluster(leaderNodeType, leaderStorage, owner, keyName, botoPath, userTags, vpcSubnet, awsEc2ProfileArn, awsEc2ExtraSecurityGroupIds, **kwargs)

Starts a single leader node and populates this class with the leader’s metadata.

Parameters:

  • leaderNodeType (str) – An AWS instance type, like “t2.medium”, for example.

  • leaderStorage (int) – An integer number of gigabytes to provide the leader instance with.

  • owner (str) – Resources will be tagged with this owner string.

  • keyName (str) – The ssh key to use to access the leader node.

  • botoPath (str) – The path to the boto credentials directory.

  • userTags (Optional[Dict[str, str]]) – Optionally provided user tags to put on the cluster.

  • vpcSubnet (Optional[str]) – Optionally specify the VPC subnet for the leader.

  • awsEc2ProfileArn (Optional[str]) – Optionally provide the profile ARN.

  • awsEc2ExtraSecurityGroupIds (Optional[List[str]]) – Optionally provide additional security group IDs.

  • kwargs (Dict[str, Any])

Returns:

None

Return type:

None

toil_service_env_options()

Set AWS tags in user docker container

Return type:

str

getKubernetesAutoscalerSetupCommands(values)

Get the Bash commands necessary to configure the Kubernetes Cluster Autoscaler for AWS.

Parameters:

values (Dict[str, str])

Return type:

str

getKubernetesCloudProvider()

Use the “aws” Kubernetes cloud provider when setting up Kubernetes.

Return type:

Optional[str]

getNodeShape(instance_type, preemptible=False)

Get the Shape for the given instance type (e.g. ‘t2.medium’).

Parameters:

  • instance_type (str)

  • preemptible (bool)

Return type:

toil.provisioners.abstractProvisioner.Shape

static retryPredicate(e)

Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff.

Parameters:

e (Exception) – exception raised during execution of setNodeCount

Returns:

boolean indicating whether the exception e should be retried

Return type:

bool

destroyCluster()

Terminate instances and delete the profile and security group.

Return type:

None

terminateNodes(nodes)

Terminate the nodes represented by given Node objects

Parameters:

nodes (List[toil.provisioners.node.Node]) – list of Node objects

Return type:

None

addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

Used to add worker nodes to the cluster

Parameters:

  • numNodes (int) – The number of nodes to add

  • preemptible (bool) – whether or not the nodes will be preemptible

  • spotBid (Optional[float]) – The bid for preemptible nodes if applicable (this can be set in config, also).

  • nodeTypes (Set[str])

Returns:

number of nodes successfully added

Return type:

int

addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)

Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load.

Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can’t have managed nodes.

Parameters:

  • minNodes (int) – The minimum number of nodes to scale to

  • maxNodes (int) – The maximum number of nodes to scale to

  • preemptible (bool) – whether or not the nodes will be preemptible

  • spotBid (Optional[float]) – The bid for preemptible nodes if applicable (this can be set in config, also).

  • nodeTypes (Set[str])

Return type:

None

getProvisionedWorkers(instance_type=None, preemptible=None)

Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

Parameters:

  • preemptible (Optional[bool]) – Boolean value to restrict to preemptible nodes or non-preemptible nodes

  • instance_type (Optional[str])

Returns:

list of Node objects

Return type:

List[toil.provisioners.node.Node]

getLeader(wait=False)

Get the leader for the cluster as a Toil Node object.

Parameters:

wait (bool)

Return type:

toil.provisioners.node.Node

full_policy(resource)

Produce a dict describing the JSON form of a full-access-granting AWS IAM policy for the service with the given name (e.g. ‘s3’).

Parameters:

resource (str)

Return type:

Dict[str, Any]

kubernetes_policy()

Get the Kubernetes policy grants not provided by the full grants on EC2 and IAM. See <https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#leader-policy> and <https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#worker-policy>.

These are mostly needed to support Kubernetes’ AWS CloudProvider, and some are for the Kubernetes Cluster Autoscaler’s AWS integration.

Some of these are really only needed on the leader.

Return type:

Dict[str, Any]