toil.provisioners.gceProvisioner

Attributes

logger

Classes

GCEProvisioner

Implements a Google Compute Engine Provisioner using libcloud.

Module Contents

toil.provisioners.gceProvisioner.logger
class toil.provisioners.gceProvisioner.GCEProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey, enable_fuse)

Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner

Implements a Google Compute Engine Provisioner using libcloud.

NODE_BOTO_PATH = '/root/.boto'
SOURCE_IMAGE = b'projects/kinvolk-public/global/images/family/flatcar-stable'
cloud = 'gce'
supportedClusterTypes()

Get all the cluster types that this provisioner implementation supports.

createClusterSettings()

Initialize class for a new cluster, to be deployed, when running outside the cloud.

readClusterSettings()

Read the cluster settings from the instance, which should be the leader. See https://cloud.google.com/compute/docs/storing-retrieving-metadata for details about reading the metadata.

launchCluster(leaderNodeType, leaderStorage, owner, **kwargs)

In addition to the parameters inherited from the abstractProvisioner, the Google launchCluster takes the following parameters: keyName: The key used to communicate with instances botoPath: Boto credentials for reading an AWS jobStore (optional). network: a network (optional) vpcSubnet: A subnet (optional). use_private_ip: even though a public ip exists, ignore it (optional)

getNodeShape(instance_type, preemptible=False)

The shape of a preemptible or non-preemptible node managed by this provisioner. The node shape defines key properties of a machine, such as its number of cores or the time between billing intervals.

Parameters:

instance_type (str) – Instance type name to return the shape of.

Return type:

toil.provisioners.abstractProvisioner.Shape

static retryPredicate(e)

Not used by GCE

destroyCluster()

Try a few times to terminate all of the instances in the group.

Return type:

None

terminateNodes(nodes)

Terminate the nodes represented by given Node objects

Parameters:

nodes – list of Node objects

addNodes(nodeTypes, numNodes, preemptible, spotBid=None)

Used to add worker nodes to the cluster

Parameters:

  • numNodes – The number of nodes to add

  • preemptible – whether or not the nodes will be preemptible

  • spotBid – The bid for preemptible nodes if applicable (this can be set in config, also).

  • nodeTypes (Set[str])

Returns:

number of nodes successfully added

Return type:

int

getProvisionedWorkers(instance_type=None, preemptible=None)

Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

Parameters:

  • preemptible (Optional[bool]) – Boolean value to restrict to preemptible nodes or non-preemptible nodes

  • instance_type (Optional[str])

Returns:

list of Node objects

getLeader()
Returns:

The leader node.

DEFAULT_TASK_COMPLETION_TIMEOUT = 180
ex_create_multiple_nodes(base_name, size, image, number, location=None, ex_network='default', ex_subnetwork=None, ex_tags=None, ex_metadata=None, ignore_errors=True, use_existing_disk=True, poll_interval=2, external_ip='ephemeral', ex_disk_type='pd-standard', ex_disk_auto_delete=True, ex_service_accounts=None, timeout=DEFAULT_TASK_COMPLETION_TIMEOUT, description=None, ex_can_ip_forward=None, ex_disks_gce_struct=None, ex_nic_gce_struct=None, ex_on_host_maintenance=None, ex_automatic_restart=None, ex_image_family=None, ex_preemptible=None)

Monkey patch to gce.py in libcloud to allow disk and images to be specified. Also changed name to a uuid below. The prefix ‘wp’ identifies preemptible nodes and ‘wn’ non-preemptible nodes.