toil.provisioners.aws.awsProvisioner

Module Contents

Classes

AWSProvisioner

Interface for provisioning worker nodes to use in a Toil cluster.

Functions

awsRetryPredicate(e)

expectedShutdownErrors(e)

Matches errors that we expect to occur during shutdown, and which indicate

awsRetry(f)

This decorator retries the wrapped function if aws throws unexpected errors

awsFilterImpairedNodes(nodes, ec2)

Attributes

logger

toil.provisioners.aws.awsProvisioner.logger
toil.provisioners.aws.awsProvisioner.awsRetryPredicate(e)
toil.provisioners.aws.awsProvisioner.expectedShutdownErrors(e)

Matches errors that we expect to occur during shutdown, and which indicate that we need to wait or try again.

Should not match any errors which indicate that an operation is impossible or unnecessary (such as errors resulting from a thing not existing to be deleted).

Parameters

e (Exception) –

Return type

bool

toil.provisioners.aws.awsProvisioner.awsRetry(f)

This decorator retries the wrapped function if aws throws unexpected errors errors. It should wrap any function that makes use of boto

toil.provisioners.aws.awsProvisioner.awsFilterImpairedNodes(nodes, ec2)
exception toil.provisioners.aws.awsProvisioner.InvalidClusterStateException

Bases: Exception

digraph inheritance1ec339b5f2 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "InvalidClusterStateException" [URL="#toil.provisioners.aws.awsProvisioner.InvalidClusterStateException",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; }

Common base class for all non-exit exceptions.

class toil.provisioners.aws.awsProvisioner.AWSProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey)

Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner

digraph inheritance532fb64cb7 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "AWSProvisioner" [URL="#toil.provisioners.aws.awsProvisioner.AWSProvisioner",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "AbstractProvisioner" -> "AWSProvisioner" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractProvisioner" [URL="../../abstractProvisioner/index.html#toil.provisioners.abstractProvisioner.AbstractProvisioner",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Interface for provisioning worker nodes to use in a Toil cluster."]; "ABC" -> "AbstractProvisioner" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Interface for provisioning worker nodes to use in a Toil cluster.

supportedClusterTypes()

Get all the cluster types that this provisioner implementation supports.

createClusterSettings()

Create a new set of cluster settings for a cluster to be deployed into AWS.

readClusterSettings()

Reads the cluster settings from the instance metadata, which assumes the instance is the leader.

launchCluster(leaderNodeType, leaderStorage, owner, keyName, botoPath, userTags, vpcSubnet, awsEc2ProfileArn, awsEc2ExtraSecurityGroupIds, **kwargs)

Starts a single leader node and populates this class with the leader’s metadata.

Parameters

  • leaderNodeType (str) – An AWS instance type, like “t2.medium”, for example.

  • leaderStorage (int) – An integer number of gigabytes to provide the leader instance with.

  • owner (str) – Resources will be tagged with this owner string.

  • keyName (str) – The ssh key to use to access the leader node.

  • botoPath (str) – The path to the boto credentials directory.

  • userTags (Optional[dict]) – Optionally provided user tags to put on the cluster.

  • vpcSubnet (Optional[str]) – Optionally specify the VPC subnet for the leader.

  • awsEc2ProfileArn (Optional[str]) – Optionally provide the profile ARN.

  • awsEc2ExtraSecurityGroupIds (Optional[list]) – Optionally provide additional security group IDs.

Returns

None

toil_service_env_options()

Set AWS tags in user docker container

Return type

str

getKubernetesAutoscalerSetupCommands(values)

Get the Bash commands necessary to configure the Kubernetes Cluster Autoscaler for AWS.

Parameters

values (Dict[str, str]) –

Return type

str

getKubernetesCloudProvider()

Use the “aws” Kubernetes cloud provider when setting up Kubernetes.

Return type

Optional[str]

getNodeShape(instance_type, preemptible=False)

Get the Shape for the given instance type (e.g. ‘t2.medium’).

Parameters

instance_type (str) –

Return type

toil.provisioners.abstractProvisioner.Shape

static retryPredicate(e)

Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff.

Parameters

e – exception raised during execution of setNodeCount

Returns

boolean indicating whether the exception e should be retried

destroyCluster()

Terminate instances and delete the profile and security group.

Return type

None

terminateNodes(nodes)

Terminate the nodes represented by given Node objects

Parameters

nodes (List[toil.provisioners.node.Node]) – list of Node objects

Return type

None

addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

Used to add worker nodes to the cluster

Parameters

  • numNodes – The number of nodes to add

  • preemptible – whether or not the nodes will be preemptible

  • spotBid – The bid for preemptible nodes if applicable (this can be set in config, also).

  • nodeTypes (Set[str]) –

Returns

number of nodes successfully added

Return type

int

addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)

Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load.

Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can’t have managed nodes.

Parameters

  • minNodes – The minimum number of nodes to scale to

  • maxNodes – The maximum number of nodes to scale to

  • preemptible – whether or not the nodes will be preemptible

  • spotBid – The bid for preemptible nodes if applicable (this can be set in config, also).

  • nodeTypes (Set[str]) –

Return type

None

getProvisionedWorkers(instance_type=None, preemptible=None)

Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

Parameters

  • preemptible (Optional[bool]) – Boolean value to restrict to preemptible nodes or non-preemptible nodes

  • instance_type (Optional[str]) –

Returns

list of Node objects

Return type

List[toil.provisioners.node.Node]

getLeader(wait=False)

Get the leader for the cluster as a Toil Node object.

Return type

toil.provisioners.node.Node

full_policy(resource)

Produce a dict describing the JSON form of a full-access-granting AWS IAM policy for the service with the given name (e.g. ‘s3’).

Parameters

resource (str) –

Return type

dict

kubernetes_policy()

Get the Kubernetes policy grants not provided by the full grants on EC2 and IAM. See <https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#leader-policy> and <https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#worker-policy>.

These are mostly needed to support Kubernetes’ AWS CloudProvider, and some are for the Kubernetes Cluster Autoscaler’s AWS integration.

Some of these are really only needed on the leader.

Return type

dict