toil.provisioners.abstractProvisioner

Module Contents

Classes

Shape	Represents a job or a node's "shape", in terms of the dimensions of memory, cores, disk and
AbstractProvisioner	Interface for provisioning worker nodes to use in a Toil cluster.

Attributes

a_short_time
logger

toil.provisioners.abstractProvisioner.a_short_time = 5

toil.provisioners.abstractProvisioner.logger

exception toil.provisioners.abstractProvisioner.ManagedNodesNotSupportedException

Bases: RuntimeError

Raised when attempting to add managed nodes (which autoscale up and down by themselves, without the provisioner doing the work) to a provisioner that does not support them.

Polling with this and try/except is the Right Way to check if managed nodes are available from a provisioner.

class toil.provisioners.abstractProvisioner.Shape(wallTime, memory, cores, disk, preemptible)

Represents a job or a node’s “shape”, in terms of the dimensions of memory, cores, disk and wall-time allocation.

The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs?

The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively.

Parameters

• wallTime (Union[int, float]) –

• memory (int) –

• cores (Union[int, float]) –

• disk (int) –

• preemptible (bool) –

__eq__(other)

Return self==value.

Parameters

other (Any) –

Return type

bool

greater_than(other)

Parameters

other (Any) –

Return type

bool

__gt__(other)

Return self>value.

Parameters

other (Any) –

Return type

bool

__repr__()

Return repr(self).

Return type

str

__str__()

Return str(self).

Return type

str

__hash__()

Return hash(self).

Return type

int

class toil.provisioners.abstractProvisioner.AbstractProvisioner(clusterName=None, clusterType='mesos', zone=None, nodeStorage=50, nodeStorageOverrides=None)

Bases: abc.ABC

Interface for provisioning worker nodes to use in a Toil cluster.

Parameters

• clusterName (Optional[str]) –

• clusterType (Optional[str]) –

• zone (Optional[str]) –

• nodeStorage (int) –

• nodeStorageOverrides (Optional[List[str]]) –

class InstanceConfiguration

Allows defining the initial setup for an instance and then turning it into an Ignition configuration for instance user data.

addFile(path, filesystem='root', mode='0755', contents='', append=False)

Make a file on the instance with the given filesystem, mode, and contents.

See the storage.files section: https://github.com/kinvolk/ignition/blob/flatcar-master/doc/configuration-v2_2.md

Parameters

• path (str) –

• filesystem (str) –

• mode (Union[str, int]) –

• contents (str) –

• append (bool) –

addUnit(name, enabled=True, contents='')

Make a systemd unit on the instance with the given name (including .service), and content. Units will be enabled by default.

Unit logs can be investigated with:

systemctl status whatever.service

or:

journalctl -xe

Parameters

• name (str) –

• enabled (bool) –

• contents (str) –

addSSHRSAKey(keyData)

Authorize the given bare, encoded RSA key (without “ssh-rsa”).

Parameters

keyData (str) –

toIgnitionConfig()

Return an Ignition configuration describing the desired config.

Return type

str

LEADER_HOME_DIR = '/root/'

cloud: str

abstract supportedClusterTypes()

Get all the cluster types that this provisioner implementation supports.

Return type

Set[str]

abstract createClusterSettings()

Initialize class for a new cluster, to be deployed, when running outside the cloud.

abstract readClusterSettings()

Initialize class from an existing cluster. This method assumes that the instance we are running on is the leader.

Implementations must call _setLeaderWorkerAuthentication().

setAutoscaledNodeTypes(nodeTypes)

Set node types, shapes and spot bids for Toil-managed autoscaling. :param nodeTypes: A list of node types, as parsed with parse_node_types.

Parameters

nodeTypes (List[Tuple[Set[str], Optional[float]]]) –

hasAutoscaledNodeTypes()

Check if node types have been configured on the provisioner (via setAutoscaledNodeTypes).

Returns

True if node types are configured for autoscaling, and false otherwise.

Return type

bool

getAutoscaledInstanceShapes()

Get all the node shapes and their named instance types that the Toil autoscaler should manage.

Return type

Dict[Shape, str]

static retryPredicate(e)

Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff.

Parameters

e – exception raised during execution of setNodeCount

Returns

boolean indicating whether the exception e should be retried

abstract launchCluster(*args, **kwargs)

Initialize a cluster and create a leader node.

Implementations must call _setLeaderWorkerAuthentication() with the leader so that workers can be launched.

Parameters

• leaderNodeType – The leader instance.

• leaderStorage – The amount of disk to allocate to the leader in gigabytes.

• owner – Tag identifying the owner of the instances.

abstract addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

Used to add worker nodes to the cluster

Parameters

• numNodes (int) – The number of nodes to add

• preemptible (bool) – whether or not the nodes will be preemptible

• spotBid (Optional[float]) – The bid for preemptible nodes if applicable (this can be set in config, also).

• nodeTypes (Set[str]) –

Returns

number of nodes successfully added

Return type

int

addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)

Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load.

Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can’t have managed nodes.

Parameters

• minNodes – The minimum number of nodes to scale to

• maxNodes – The maximum number of nodes to scale to

• preemptible – whether or not the nodes will be preemptible

• spotBid – The bid for preemptible nodes if applicable (this can be set in config, also).

• nodeTypes (Set[str]) –

Return type

None

abstract terminateNodes(nodes)

Terminate the nodes represented by given Node objects

Parameters

nodes (List[toil.provisioners.node.Node]) – list of Node objects

Return type

None

abstract getLeader()

Returns

The leader node.

abstract getProvisionedWorkers(instance_type=None, preemptible=None)

Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

Parameters

• preemptible (Optional[bool]) – Boolean value to restrict to preemptible nodes or non-preemptible nodes

• instance_type (Optional[str]) –

Returns

list of Node objects

Return type

List[toil.provisioners.node.Node]

abstract getNodeShape(instance_type, preemptible=False)

The shape of a preemptible or non-preemptible node managed by this provisioner. The node shape defines key properties of a machine, such as its number of cores or the time between billing intervals.

Parameters

instance_type (str) – Instance type name to return the shape of.

Return type

Shape

abstract destroyCluster()

Terminates all nodes in the specified cluster and cleans up all resources associated with the cluster. :param clusterName: identifier of the cluster to terminate.

Return type

None

getBaseInstanceConfiguration()

Get the base configuration for both leader and worker instances for all cluster types.

Return type

InstanceConfiguration

addVolumesService(config)

Add a service to prepare and mount local scratch volumes.

Parameters

config (InstanceConfiguration) –

addNodeExporterService(config)

Add the node exporter service for Prometheus to an instance configuration.

Parameters

config (InstanceConfiguration) –

toil_service_env_options()

Return type

str

add_toil_service(config, role, keyPath=None, preemptible=False)

Add the Toil leader or worker service to an instance configuration.

Will run Mesos master or agent as appropriate in Mesos clusters. For Kubernetes clusters, will just sleep to provide a place to shell into on the leader, and shouldn’t run on the worker.

Parameters

• role (str) – Should be ‘leader’ or ‘worker’. Will not work for ‘worker’ until leader credentials have been collected.

• keyPath (str) – path on the node to a server-side encryption key that will be added to the node after it starts. The service will wait until the key is present before starting.

• preemptible (bool) – Whether a worker should identify itself as preemptible or not to the scheduler.

• config (InstanceConfiguration) –

getKubernetesValues(architecture='amd64')

Returns a dict of Kubernetes component versions and paths for formatting into Kubernetes-related templates.

Parameters

architecture (str) –

addKubernetesServices(config, architecture='amd64')

Add installing Kubernetes and Kubeadm and setting up the Kubelet to run when configured to an instance configuration. The same process applies to leaders and workers.

Parameters

• config (InstanceConfiguration) –

• architecture (str) –

abstract getKubernetesAutoscalerSetupCommands(values)

Return Bash commands that set up the Kubernetes cluster autoscaler for provisioning from the environment supported by this provisioner.

Should only be implemented if Kubernetes clusters are supported.

Parameters

values (Dict[str, str]) – Contains definitions of cluster variables, like AUTOSCALER_VERSION and CLUSTER_NAME.

Returns

Bash snippet

Return type

str

getKubernetesCloudProvider()

Return the Kubernetes cloud provider (for example, ‘aws’), to pass to the kubelets in a Kubernetes cluster provisioned using this provisioner.

Defaults to None if not overridden, in which case no cloud provider integration will be used.

Returns

Cloud provider name, or None

Return type

Optional[str]

addKubernetesLeader(config)

Add services to configure as a Kubernetes leader, if Kubernetes is already set to be installed.

Parameters

config (InstanceConfiguration) –

addKubernetesWorker(config, authVars, preemptible=False)

Add services to configure as a Kubernetes worker, if Kubernetes is already set to be installed.

Authenticate back to the leader using the JOIN_TOKEN, JOIN_CERT_HASH, and JOIN_ENDPOINT set in the given authentication data dict.

Parameters

• config (InstanceConfiguration) – The configuration to add services to

• authVars (Dict[str, str]) – Dict with authentication info

• preemptible (bool) – Whether the worker should be labeled as preemptible or not