toil.test.provisioners.aws.awsProvisionerTest¶

Attributes¶

`exactPython`
`log`

Classes¶

`AWSProvisioner`	Interface for provisioning worker nodes to use in a Toil cluster.
`ToilTest`	A common base class for Toil tests.
`AbstractClusterTest`	A common base class for Toil tests.
`AWSProvisionerBenchTest`	Tests for the AWS provisioner that don't actually provision anything.
`AbstractAWSAutoscaleTest`	A common base class for Toil tests.
`AWSAutoscaleTest`	A common base class for Toil tests.
`AWSStaticAutoscaleTest`	Runs the tests on a statically provisioned cluster with autoscaling enabled.
`AWSManagedAutoscaleTest`	Runs the tests on a self-scaling Kubernetes cluster.
`AWSAutoscaleTestMultipleNodeTypes`	A common base class for Toil tests.
`AWSRestartTest`	This test insures autoscaling works on a restarted Toil run.
`PreemptibleDeficitCompensationTest`	A common base class for Toil tests.

Functions¶

`cluster_factory`(provisioner[, clusterName, ...])	Find and instantiate the appropriate provisioner instance to make clusters in the given cloud.
`integrative`(test_item)	Use this to decorate integration tests so as to skip them during regular builds.
`needs_aws_ec2`(test_item)	Use as a decorator before test classes or methods to run only if AWS EC2 is usable.
`needs_fetchable_appliance`(test_item)	Use as a decorator before test classes or methods to only run them if
`needs_mesos`(test_item)	Use as a decorator before test classes or methods to run only if Mesos is installed.
`slow`(test_item)	Use this decorator to identify tests that are slow and not critical.
`timeLimit`(seconds)	Use to limit the execution time of a function.

Module Contents¶

toil.test.provisioners.aws.awsProvisionerTest.cluster_factory(provisioner, clusterName=None, clusterType='mesos', zone=None, nodeStorage=50, nodeStorageOverrides=None, sseKey=None, enable_fuse=False)¶

Find and instantiate the appropriate provisioner instance to make clusters in the given cloud.

Raises ClusterTypeNotSupportedException if the given provisioner does not implement clusters of the given type.

Parameters:

provisioner (str) – The cloud type of the cluster.
clusterName (Optional[str]) – The name of the cluster.
clusterType (str) – The type of cluster: ‘mesos’ or ‘kubernetes’.
zone (Optional[str]) – The cloud zone
nodeStorage (int)
nodeStorageOverrides (Optional[List[str]])
sseKey (Optional[str])
enable_fuse (bool)

Returns:

A cluster object for the the cloud type.

Return type:

Union[aws.awsProvisioner.AWSProvisioner, gceProvisioner.GCEProvisioner]

class toil.test.provisioners.aws.awsProvisionerTest.AWSProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey, enable_fuse)¶

Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner

Interface for provisioning worker nodes to use in a Toil cluster.

Parameters:

clusterName (Optional[str])
clusterType (Optional[str])
zone (Optional[str])
nodeStorage (int)
nodeStorageOverrides (Optional[List[str]])
sseKey (Optional[str])
enable_fuse (bool)

supportedClusterTypes()¶

Get all the cluster types that this provisioner implementation supports.

Return type:: Set[str]

createClusterSettings()¶

Create a new set of cluster settings for a cluster to be deployed into AWS.

Return type:: None

readClusterSettings()¶

Reads the cluster settings from the instance metadata, which assumes the instance is the leader.

Return type:: None

launchCluster(leaderNodeType, leaderStorage, owner, keyName, botoPath, userTags, vpcSubnet, awsEc2ProfileArn, awsEc2ExtraSecurityGroupIds, **kwargs)¶

Starts a single leader node and populates this class with the leader’s metadata.

Parameters:

leaderNodeType (str) – An AWS instance type, like “t2.medium”, for example.
leaderStorage (int) – An integer number of gigabytes to provide the leader instance with.
owner (str) – Resources will be tagged with this owner string.
keyName (str) – The ssh key to use to access the leader node.
botoPath (str) – The path to the boto credentials directory.
userTags (Optional[Dict[str, str]]) – Optionally provided user tags to put on the cluster.
vpcSubnet (Optional[str]) – Optionally specify the VPC subnet for the leader.
awsEc2ProfileArn (Optional[str]) – Optionally provide the profile ARN.
awsEc2ExtraSecurityGroupIds (Optional[List[str]]) – Optionally provide additional security group IDs.
kwargs (Dict[str, Any])

Returns:

None

Return type:

None

toil_service_env_options()¶

Set AWS tags in user docker container

Return type:: str

getKubernetesAutoscalerSetupCommands(values)¶

Get the Bash commands necessary to configure the Kubernetes Cluster Autoscaler for AWS.

Parameters:: values (Dict[str, str])
Return type:: str

getKubernetesCloudProvider()¶

Use the “aws” Kubernetes cloud provider when setting up Kubernetes.

Return type:: Optional[str]

getNodeShape(instance_type, preemptible=False)¶

Get the Shape for the given instance type (e.g. ‘t2.medium’).

Parameters:

instance_type (str)
preemptible (bool)

Return type:

toil.provisioners.abstractProvisioner.Shape

static retryPredicate(e)¶

Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff.

Parameters:: e (Exception) – exception raised during execution of setNodeCount
Returns:: boolean indicating whether the exception e should be retried
Return type:: bool

destroyCluster()¶

Terminate instances and delete the profile and security group.

Return type:: None

terminateNodes(nodes)¶

Terminate the nodes represented by given Node objects

Parameters:: nodes (List[toil.provisioners.node.Node]) – list of Node objects
Return type:: None

addNodes(nodeTypes, numNodes, preemptible, spotBid=None)¶

Used to add worker nodes to the cluster

Parameters:

numNodes (int) – The number of nodes to add
preemptible (bool) – whether or not the nodes will be preemptible
spotBid (Optional[float]) – The bid for preemptible nodes if applicable (this can be set in config, also).
nodeTypes (Set[str])

Returns:

number of nodes successfully added

Return type:

int

addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)¶

Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load.

Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can’t have managed nodes.

Parameters:

minNodes (int) – The minimum number of nodes to scale to
maxNodes (int) – The maximum number of nodes to scale to
preemptible (bool) – whether or not the nodes will be preemptible
spotBid (Optional[float]) – The bid for preemptible nodes if applicable (this can be set in config, also).
nodeTypes (Set[str])

Return type:

None

getProvisionedWorkers(instance_type=None, preemptible=None)¶

Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

Parameters:

preemptible (Optional[bool]) – Boolean value to restrict to preemptible nodes or non-preemptible nodes
instance_type (Optional[str])

Returns:

list of Node objects

Return type:

List[toil.provisioners.node.Node]

getLeader(wait=False)¶

Get the leader for the cluster as a Toil Node object.

Parameters:: wait (bool)
Return type:: toil.provisioners.node.Node

full_policy(resource)¶

Produce a dict describing the JSON form of a full-access-granting AWS IAM policy for the service with the given name (e.g. ‘s3’).

Parameters:: resource (str)
Return type:: Dict[str, Any]

kubernetes_policy()¶

Get the Kubernetes policy grants not provided by the full grants on EC2 and IAM. See <https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#leader-policy> and <https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#worker-policy>.

These are mostly needed to support Kubernetes’ AWS CloudProvider, and some are for the Kubernetes Cluster Autoscaler’s AWS integration.

Some of these are really only needed on the leader.

Return type:: Dict[str, Any]

class toil.test.provisioners.aws.awsProvisionerTest.ToilTest(methodName='runTest')¶

Bases: unittest.TestCase

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

setup_method(method)¶

Parameters:: method (Any)
Return type:: None

classmethod setUpClass()¶

Hook method for setting up class fixture before running tests in the class.

Return type:: None

classmethod tearDownClass()¶

Hook method for deconstructing the class fixture after running all tests in the class.

Return type:: None

setUp()¶

Hook method for setting up the test fixture before exercising it.

Return type:: None

tearDown()¶

Hook method for deconstructing the test fixture after testing it.

Return type:: None

classmethod awsRegion()¶

Pick an appropriate AWS region.

Use us-west-2 unless running on EC2, in which case use the region in which the instance is located

Return type:: str

toil.test.provisioners.aws.awsProvisionerTest.integrative(test_item)¶

Use this to decorate integration tests so as to skip them during regular builds.

We define integration tests as A) involving other, non-Toil software components that we develop and/or B) having a higher cost (time or money).

Parameters:: test_item (MT)
Return type:: MT

toil.test.provisioners.aws.awsProvisionerTest.needs_aws_ec2(test_item)¶

Use as a decorator before test classes or methods to run only if AWS EC2 is usable.

Parameters:: test_item (MT)
Return type:: MT

toil.test.provisioners.aws.awsProvisionerTest.needs_fetchable_appliance(test_item)¶

Use as a decorator before test classes or methods to only run them if the Toil appliance Docker image is able to be downloaded from the Internet.

Parameters:: test_item (MT)
Return type:: MT

toil.test.provisioners.aws.awsProvisionerTest.needs_mesos(test_item)¶

Use as a decorator before test classes or methods to run only if Mesos is installed.

Parameters:: test_item (MT)
Return type:: MT

toil.test.provisioners.aws.awsProvisionerTest.slow(test_item)¶

Use this decorator to identify tests that are slow and not critical. Skip if TOIL_TEST_QUICK is true.

Parameters:: test_item (MT)
Return type:: MT

toil.test.provisioners.aws.awsProvisionerTest.timeLimit(seconds)¶

Use to limit the execution time of a function.

Raises an exception if the execution of the function takes more than the specified amount of time. See <http://stackoverflow.com/a/601168>.

Parameters:: seconds (int) – maximum allowable time, in seconds
Return type:: Generator[None, None, None]

>>> import time
>>> with timeLimit(2):
...    time.sleep(1)
>>> import time
>>> with timeLimit(1):
...    time.sleep(2)
Traceback (most recent call last):
    ...
RuntimeError: Timed out

class toil.test.provisioners.aws.awsProvisionerTest.AbstractClusterTest(methodName)¶

Bases: toil.test.ToilTest

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

Parameters:: methodName (str)

python()¶

Return the full path to the venv Python on the leader.

Return type:: str

pip()¶

Return the full path to the venv pip on the leader.

Return type:: str

destroyCluster()¶

Destroy the cluster we built, if it exists.

Succeeds if the cluster does not currently exist.

Return type:: None

setUp()¶

Set up for the test. Must be overridden to call this method and set self.jobStore.

Return type:: None

tearDown()¶

Hook method for deconstructing the test fixture after testing it.

Return type:: None

sshUtil(command)¶

Run the given command on the cluster. Raise subprocess.CalledProcessError if it fails.

Parameters:: command (List[str])
Return type:: None

rsync_util(from_file, to_file)¶

Transfer a file to/from the cluster.

The cluster-side path should have a ‘:’ in front of it.

Parameters:

from_file (str)
to_file (str)

Return type:

None

createClusterUtil(args=None)¶

Parameters:: args (Optional[List[str]])
Return type:: None

launchCluster()¶

Return type:: None

toil.test.provisioners.aws.awsProvisionerTest.exactPython = 'python3.9'¶

toil.test.provisioners.aws.awsProvisionerTest.log¶

class toil.test.provisioners.aws.awsProvisionerTest.AWSProvisionerBenchTest(methodName='runTest')¶

Bases: toil.test.ToilTest

Tests for the AWS provisioner that don’t actually provision anything.

test_AMI_finding()¶

test_read_write_global_files()¶: Make sure the _write_file_to_cloud() and _read_file_from_cloud() functions of the AWS provisioner work as intended.

class toil.test.provisioners.aws.awsProvisionerTest.AbstractAWSAutoscaleTest(methodName)¶

Bases: toil.test.provisioners.clusterTest.AbstractClusterTest

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

script()¶: Return the full path to the user script on the leader.

data(filename)¶: Return the full path to the data file with the given name on the leader.

rsyncUtil(src, dest)¶

getRootVolID()¶

Return type:: str

putScript(content)¶

Helper method for _getScript to inject a script file at the configured script path, from text.

Parameters:: content (str)

class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTest(name)¶

Bases: AbstractAWSAutoscaleTest

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

setUp()¶: Set up for the test. Must be overridden to call this method and set self.jobStore.

launchCluster()¶

getRootVolID()¶

Adds in test to check that EBS volume is build with adequate size. Otherwise is functionally equivalent to parent. :return: volumeID

Return type:: str

testAutoScale()¶

testSpotAutoScale()¶

testSpotAutoScaleBalancingTypes()¶

class toil.test.provisioners.aws.awsProvisionerTest.AWSStaticAutoscaleTest(name)¶

Bases: AWSAutoscaleTest

Runs the tests on a statically provisioned cluster with autoscaling enabled.

launchCluster()¶

class toil.test.provisioners.aws.awsProvisionerTest.AWSManagedAutoscaleTest(name)¶

Bases: AWSAutoscaleTest

Runs the tests on a self-scaling Kubernetes cluster.

launchCluster()¶

class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTestMultipleNodeTypes(name)¶

Bases: AbstractAWSAutoscaleTest

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

setUp()¶: Set up for the test. Must be overridden to call this method and set self.jobStore.

testAutoScale()¶

class toil.test.provisioners.aws.awsProvisionerTest.AWSRestartTest(name)¶

Bases: AbstractAWSAutoscaleTest

This test insures autoscaling works on a restarted Toil run.

setUp()¶: Set up for the test. Must be overridden to call this method and set self.jobStore.

testAutoScaledCluster()¶

class toil.test.provisioners.aws.awsProvisionerTest.PreemptibleDeficitCompensationTest(name)¶

Bases: AbstractAWSAutoscaleTest

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

setUp()¶: Set up for the test. Must be overridden to call this method and set self.jobStore.

test()¶

toil.test.provisioners.aws.awsProvisionerTest¶

Attributes¶

Classes¶

Functions¶

Module Contents¶

Toil

Navigation

Related Topics