toil.provisioners.aws.awsProvisioner

Module Contents

Classes

AWSProvisioner

Interface for provisioning worker nodes to use in a Toil cluster.

Functions

awsRetryPredicate(e)

expectedShutdownErrors(e)

Matches errors that we expect to occur during shutdown, and which indicate

awsRetry(f)

This decorator retries the wrapped function if aws throws unexpected errors

awsFilterImpairedNodes(nodes, ec2)

Attributes

logger

toil.provisioners.aws.awsProvisioner.logger
toil.provisioners.aws.awsProvisioner.awsRetryPredicate(e)
toil.provisioners.aws.awsProvisioner.expectedShutdownErrors(e)

Matches errors that we expect to occur during shutdown, and which indicate that we need to wait or try again.

Should not match any errors which indicate that an operation is impossible or unnecessary (such as errors resulting from a thing not existing to be deleted).

Parameters:

e (Exception) –

Return type:

bool

toil.provisioners.aws.awsProvisioner.awsRetry(f)

This decorator retries the wrapped function if aws throws unexpected errors errors. It should wrap any function that makes use of boto

toil.provisioners.aws.awsProvisioner.awsFilterImpairedNodes(nodes, ec2)
exception toil.provisioners.aws.awsProvisioner.InvalidClusterStateException

Bases: Exception

Common base class for all non-exit exceptions.

class toil.provisioners.aws.awsProvisioner.AWSProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey)

Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner

Interface for provisioning worker nodes to use in a Toil cluster.

supportedClusterTypes()

Get all the cluster types that this provisioner implementation supports.

createClusterSettings()

Create a new set of cluster settings for a cluster to be deployed into AWS.

readClusterSettings()

Reads the cluster settings from the instance metadata, which assumes the instance is the leader.

launchCluster(leaderNodeType, leaderStorage, owner, keyName, botoPath, userTags, vpcSubnet, awsEc2ProfileArn, awsEc2ExtraSecurityGroupIds, **kwargs)

Starts a single leader node and populates this class with the leader’s metadata.

Parameters:

  • leaderNodeType (str) – An AWS instance type, like “t2.medium”, for example.

  • leaderStorage (int) – An integer number of gigabytes to provide the leader instance with.

  • owner (str) – Resources will be tagged with this owner string.

  • keyName (str) – The ssh key to use to access the leader node.

  • botoPath (str) – The path to the boto credentials directory.

  • userTags (Optional[dict]) – Optionally provided user tags to put on the cluster.

  • vpcSubnet (Optional[str]) – Optionally specify the VPC subnet for the leader.

  • awsEc2ProfileArn (Optional[str]) – Optionally provide the profile ARN.

  • awsEc2ExtraSecurityGroupIds (Optional[list]) – Optionally provide additional security group IDs.

Returns:

None

toil_service_env_options()

Set AWS tags in user docker container

Return type:

str

getKubernetesAutoscalerSetupCommands(values)

Get the Bash commands necessary to configure the Kubernetes Cluster Autoscaler for AWS.

Parameters:

values (Dict[str, str]) –

Return type:

str

getKubernetesCloudProvider()

Use the “aws” Kubernetes cloud provider when setting up Kubernetes.

Return type:

Optional[str]

getNodeShape(instance_type, preemptible=False)

Get the Shape for the given instance type (e.g. ‘t2.medium’).

Parameters:

instance_type (str) –

Return type:

toil.provisioners.abstractProvisioner.Shape

static retryPredicate(e)

Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff.

Parameters:

e – exception raised during execution of setNodeCount

Returns:

boolean indicating whether the exception e should be retried

destroyCluster()

Terminate instances and delete the profile and security group.

Return type:

None

terminateNodes(nodes)

Terminate the nodes represented by given Node objects

Parameters:

nodes (List[toil.provisioners.node.Node]) – list of Node objects

Return type:

None

addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

Used to add worker nodes to the cluster

Parameters:

  • numNodes – The number of nodes to add

  • preemptible – whether or not the nodes will be preemptible

  • spotBid – The bid for preemptible nodes if applicable (this can be set in config, also).

  • nodeTypes (Set[str]) –

Returns:

number of nodes successfully added

Return type:

int

addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)

Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load.

Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can’t have managed nodes.

Parameters:

  • minNodes – The minimum number of nodes to scale to

  • maxNodes – The maximum number of nodes to scale to

  • preemptible – whether or not the nodes will be preemptible

  • spotBid – The bid for preemptible nodes if applicable (this can be set in config, also).

  • nodeTypes (Set[str]) –

Return type:

None

getProvisionedWorkers(instance_type=None, preemptible=None)

Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

Parameters:

  • preemptible (Optional[bool]) – Boolean value to restrict to preemptible nodes or non-preemptible nodes

  • instance_type (Optional[str]) –

Returns:

list of Node objects

Return type:

List[toil.provisioners.node.Node]

getLeader(wait=False)

Get the leader for the cluster as a Toil Node object.

Return type:

toil.provisioners.node.Node

full_policy(resource)

Produce a dict describing the JSON form of a full-access-granting AWS IAM policy for the service with the given name (e.g. ‘s3’).

Parameters:

resource (str) –

Return type:

dict

kubernetes_policy()

Get the Kubernetes policy grants not provided by the full grants on EC2 and IAM. See <https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#leader-policy> and <https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#worker-policy>.

These are mostly needed to support Kubernetes’ AWS CloudProvider, and some are for the Kubernetes Cluster Autoscaler’s AWS integration.

Some of these are really only needed on the leader.

Return type:

dict