toil.test.provisioners.aws.awsProvisionerTest

Attributes

log

Classes

TestAWSProvisionerBenchTest	Tests for the AWS provisioner that don't actually provision instances.
AbstractAWSAutoscaleTest	Legacy common base class for Toil tests.
AWSAutoscaleTest	Legacy common base class for Toil tests.
AWSStaticAutoscaleTest	Runs the tests on a statically provisioned cluster with autoscaling enabled.
AWSManagedAutoscaleTest	Runs the tests on a self-scaling Kubernetes cluster.
AWSAutoscaleTestMultipleNodeTypes	Legacy common base class for Toil tests.
AWSRestartTest	This test insures autoscaling works on a restarted Toil run.
PreemptibleDeficitCompensationTest	Legacy common base class for Toil tests.

Functions

aws_zone()	Supply an appropriate AWS zone to work in to tests that need one.
aws_region(aws_zone)	Supply an appropriate AWS region to work in to tests that need one.
ec2_client(aws_region)	Supply an AWS EC2 client tests that need one.

Module Contents

toil.test.provisioners.aws.awsProvisionerTest.log

toil.test.provisioners.aws.awsProvisionerTest.aws_zone()

Supply an appropriate AWS zone to work in to tests that need one.

toil.test.provisioners.aws.awsProvisionerTest.aws_region(aws_zone)

Supply an appropriate AWS region to work in to tests that need one.

toil.test.provisioners.aws.awsProvisionerTest.ec2_client(aws_region)

Supply an AWS EC2 client tests that need one.

class toil.test.provisioners.aws.awsProvisionerTest.TestAWSProvisionerBenchTest

Tests for the AWS provisioner that don’t actually provision instances.

test_AMI_finding()

test_read_write_global_files(aws_zone)

Make sure the _write_file_to_cloud() and _read_file_from_cloud() functions of the AWS provisioner work as intended.

test_get_spot_history(ec2_client)

Make sure that we can download spot price history from AWS.

Return type:

None

test_get_aws_zone_from_spot_market(ec2_client)

Make sure that we can process spot price history to pick a zone.

Return type:

None

class toil.test.provisioners.aws.awsProvisionerTest.AbstractAWSAutoscaleTest(methodName)

Bases: toil.test.provisioners.clusterTest.AbstractClusterTest

Legacy common base class for Toil tests.

New tests should be made in the ‘pytest’ style and not use this class.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

instanceTypes = ['m5a.large']

clusterName = 'aws-provisioner-test-'

numWorkers = ['2']

numSamples = 2

spotBid = 0.15

scriptDir = '/tmp/t'

venvDir = '/tmp/venv'

dataDir = '/tmp'

scriptName = 'test_script.py'

script()

Return the full path to the user script on the leader.

data(filename)

Return the full path to the data file with the given name on the leader.

rsyncUtil(src, dest)

Parameters:

• src (str)

• dest (str)

Return type:

None

getRootVolID()

Return type:

str

putScript(content)

Helper method for _getScript to inject a script file at the configured script path, from text.

Parameters:

content (str)

class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTest(name)

Bases: AbstractAWSAutoscaleTest

Legacy common base class for Toil tests.

New tests should be made in the ‘pytest’ style and not use this class.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

clusterName = 'provisioner-test-'

requestedLeaderStorage = 80

scriptName = 'sort.py'

setUp()

Set up for the test. Must be overridden to call this method and set self.jobStore.

launchCluster()

getRootVolID()

Adds in test to check that EBS volume is build with adequate size. Otherwise is functionally equivalent to parent. :return: volumeID

Return type:

str

testAutoScale()

testSpotAutoScale()

testSpotAutoScaleBalancingTypes()

class toil.test.provisioners.aws.awsProvisionerTest.AWSStaticAutoscaleTest(name)

Bases: AWSAutoscaleTest

Runs the tests on a statically provisioned cluster with autoscaling enabled.

requestedNodeStorage = 20

launchCluster()

class toil.test.provisioners.aws.awsProvisionerTest.AWSManagedAutoscaleTest(name)

Bases: AWSAutoscaleTest

Runs the tests on a self-scaling Kubernetes cluster.

requestedNodeStorage = 20

launchCluster()

class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTestMultipleNodeTypes(name)

Bases: AbstractAWSAutoscaleTest

Legacy common base class for Toil tests.

New tests should be made in the ‘pytest’ style and not use this class.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

clusterName = 'provisioner-test-'

setUp()

Set up for the test. Must be overridden to call this method and set self.jobStore.

testAutoScale()

class toil.test.provisioners.aws.awsProvisionerTest.AWSRestartTest(name)

Bases: AbstractAWSAutoscaleTest

This test insures autoscaling works on a restarted Toil run.

clusterName = 'restart-test-'

scriptName = 'restartScript.py'

setUp()

Set up for the test. Must be overridden to call this method and set self.jobStore.

testAutoScaledCluster()

class toil.test.provisioners.aws.awsProvisionerTest.PreemptibleDeficitCompensationTest(name)

Bases: AbstractAWSAutoscaleTest

Legacy common base class for Toil tests.

New tests should be made in the ‘pytest’ style and not use this class.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

clusterName = 'deficit-test-'

scriptName = 'userScript.py'

setUp()

Set up for the test. Must be overridden to call this method and set self.jobStore.

test()