toil.test.provisioners.clusterTest¶

Attributes¶

log

Classes¶

`ToilTest`	A common base class for Toil tests.
`AWSConnectionManager`	Class that represents a connection to AWS. Caches Boto 3 and Boto 2 objects
`AbstractClusterTest`	A common base class for Toil tests.
`CWLOnARMTest`	Run the CWL 1.2 conformance tests on ARM specifically.

Functions¶

`retry`([intervals, infinite_retries, errors, ...])	Retry a function if it fails with any Exception defined in "errors".
`needs_aws_ec2`(test_item)	Use as a decorator before test classes or methods to run only if AWS EC2 is usable.
`needs_fetchable_appliance`(test_item)	Use as a decorator before test classes or methods to only run them if
`slow`(test_item)	Use this decorator to identify tests that are slow and not critical.
`needs_env_var`(var_name[, comment])	Use as a decorator before test classes or methods to run only if the given
`zone_to_region`(zone)	Get a region (e.g. us-west-2) from a zone (e.g. us-west-1c).
`cluster_factory`(provisioner[, clusterName, ...])	Find and instantiate the appropriate provisioner instance to make clusters in the given cloud.
`get_best_aws_zone`([spotBid, nodeType, boto3_ec2, ...])	Get the right AWS zone to use.

Module Contents¶

toil.test.provisioners.clusterTest.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)¶

Retry a function if it fails with any Exception defined in “errors”.

Does so every x seconds, where x is defined by a list of numbers (ints or floats) in “intervals”. Also accepts ErrorCondition events for more detailed retry attempts.

Parameters:

intervals (Optional[List]) – A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s
infinite_retries (bool) – If this is True, reset the intervals when they run out. Defaults to: False.
errors (Optional[Sequence[Union[ErrorCondition, Type[Exception]]]]) –
A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: - Exception (required) - Error codes that must match to be retried (optional; defaults to not checking) - A string that must be in the error message to be retried (optional; defaults to not checking) - A bool that can be set to False to always error on this condition.

If not specified, this will default to a generic Exception.
log_message (Optional[Tuple[Callable, str]]) – Optional tuple of (“log/print function()”, “message string”) that will precede each attempt.
prepare (Optional[List[Callable]]) – Optional list of functions to call, with the function’s arguments, between retries, to reset state.

Returns:

The result of the wrapped function or raise.

Return type:

Callable[[Callable[Ellipsis, RT]], Callable[Ellipsis, RT]]

class toil.test.provisioners.clusterTest.ToilTest(methodName='runTest')¶

Bases: unittest.TestCase

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

setup_method(method)¶

Parameters:: method (Any)
Return type:: None

classmethod setUpClass()¶

Hook method for setting up class fixture before running tests in the class.

Return type:: None

classmethod tearDownClass()¶

Hook method for deconstructing the class fixture after running all tests in the class.

Return type:: None

setUp()¶

Hook method for setting up the test fixture before exercising it.

Return type:: None

tearDown()¶

Hook method for deconstructing the test fixture after testing it.

Return type:: None

classmethod awsRegion()¶

Pick an appropriate AWS region.

Use us-west-2 unless running on EC2, in which case use the region in which the instance is located

Return type:: str

toil.test.provisioners.clusterTest.needs_aws_ec2(test_item)¶

Use as a decorator before test classes or methods to run only if AWS EC2 is usable.

Parameters:: test_item (MT)
Return type:: MT

toil.test.provisioners.clusterTest.needs_fetchable_appliance(test_item)¶

Use as a decorator before test classes or methods to only run them if the Toil appliance Docker image is able to be downloaded from the Internet.

Parameters:: test_item (MT)
Return type:: MT

toil.test.provisioners.clusterTest.slow(test_item)¶

Use this decorator to identify tests that are slow and not critical. Skip if TOIL_TEST_QUICK is true.

Parameters:: test_item (MT)
Return type:: MT

toil.test.provisioners.clusterTest.needs_env_var(var_name, comment=None)¶

Use as a decorator before test classes or methods to run only if the given environment variable is set. Can include a comment saying what the variable should be set to.

Parameters:

var_name (str)
comment (Optional[str])

Return type:

Callable[[MT], MT]

toil.test.provisioners.clusterTest.zone_to_region(zone)¶

Get a region (e.g. us-west-2) from a zone (e.g. us-west-1c).

Parameters:: zone (str)
Return type:: AWSRegionName

class toil.test.provisioners.clusterTest.AWSConnectionManager¶

Class that represents a connection to AWS. Caches Boto 3 and Boto 2 objects by region.

Access to any kind of item goes through the particular method for the thing you want (session, resource, service, Boto2 Context), and then you pass the region you want to work in, and possibly the type of thing you want, as arguments.

This class is intended to eventually enable multi-region clusters, where connections to multiple regions may need to be managed in the same provisioner.

We also support None for a region, in which case no region will be passed to Boto/Boto3. The caller is responsible for implementing e.g. TOIL_AWS_REGION support.

Since connection objects may not be thread safe (see <https://boto3.amazonaws.com/v1/documentation/api/1.14.31/guide/session.html#multithreading-or-multiprocessing-with-sessions>), one is created for each thread that calls the relevant lookup method.

session(region)¶

Get the Boto3 Session to use for the given region.

Parameters:: region (Optional[str])
Return type:: boto3.session.Session

resource(region: str | None, service_name: Literal['s3'], endpoint_url: str | None = None) → mypy_boto3_s3.S3ServiceResource¶

resource(region: str | None, service_name: Literal['iam'], endpoint_url: str | None = None) → mypy_boto3_iam.IAMServiceResource

resource(region: str | None, service_name: Literal['ec2'], endpoint_url: str | None = None) → mypy_boto3_ec2.EC2ServiceResource

Get the Boto3 Resource to use with the given service (like ‘ec2’) in the given region.

Parameters:: endpoint_url – AWS endpoint URL to use for the client. If not specified, a default is used.

client(region: str | None, service_name: Literal['ec2'], endpoint_url: str | None = None, config: botocore.client.Config | None = None) → mypy_boto3_ec2.EC2Client¶

client(region: str | None, service_name: Literal['iam'], endpoint_url: str | None = None, config: botocore.client.Config | None = None) → mypy_boto3_iam.IAMClient

client(region: str | None, service_name: Literal['s3'], endpoint_url: str | None = None, config: botocore.client.Config | None = None) → mypy_boto3_s3.S3Client

client(region: str | None, service_name: Literal['sts'], endpoint_url: str | None = None, config: botocore.client.Config | None = None) → mypy_boto3_sts.STSClient

client(region: str | None, service_name: Literal['sdb'], endpoint_url: str | None = None, config: botocore.client.Config | None = None) → mypy_boto3_sdb.SimpleDBClient

client(region: str | None, service_name: Literal['autoscaling'], endpoint_url: str | None = None, config: botocore.client.Config | None = None) → mypy_boto3_autoscaling.AutoScalingClient

Get the Boto3 Client to use with the given service (like ‘ec2’) in the given region.

Parameters:

endpoint_url – AWS endpoint URL to use for the client. If not specified, a default is used.
config – Custom configuration to use for the client.

toil.test.provisioners.clusterTest.cluster_factory(provisioner, clusterName=None, clusterType='mesos', zone=None, nodeStorage=50, nodeStorageOverrides=None, sseKey=None, enable_fuse=False)¶

Find and instantiate the appropriate provisioner instance to make clusters in the given cloud.

Raises ClusterTypeNotSupportedException if the given provisioner does not implement clusters of the given type.

Parameters:

provisioner (str) – The cloud type of the cluster.
clusterName (Optional[str]) – The name of the cluster.
clusterType (str) – The type of cluster: ‘mesos’ or ‘kubernetes’.
zone (Optional[str]) – The cloud zone
nodeStorage (int)
nodeStorageOverrides (Optional[List[str]])
sseKey (Optional[str])
enable_fuse (bool)

Returns:

A cluster object for the the cloud type.

Return type:

Union[aws.awsProvisioner.AWSProvisioner, gceProvisioner.GCEProvisioner]

toil.test.provisioners.clusterTest.get_best_aws_zone(spotBid=None, nodeType=None, boto3_ec2=None, zone_options=None)¶

Get the right AWS zone to use.

Reports the TOIL_AWS_ZONE environment variable if set.

Otherwise, if we are running on EC2 or ECS, reports the zone we are running in.

Otherwise, if a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all.

In this case, zone_options can be used to restrict to a subset of the zones in the region.

Otherwise, if we have the TOIL_AWS_REGION variable set, chooses a zone in that region.

Finally, if a default region is configured in Boto 2, chooses a zone in that region.

Returns None if no method can produce a zone to use.

Parameters:

spotBid (Optional[float])
nodeType (Optional[str])
boto3_ec2 (Optional[botocore.client.BaseClient])
zone_options (Optional[List[str]])

Return type:

Optional[str]

toil.test.provisioners.clusterTest.log¶

class toil.test.provisioners.clusterTest.AbstractClusterTest(methodName)¶

Bases: toil.test.ToilTest

A common base class for Toil tests.

Please have every test case directly or indirectly inherit this one.

When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn’t exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system’s default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

Parameters:: methodName (str)

python()¶

Return the full path to the venv Python on the leader.

Return type:: str

pip()¶

Return the full path to the venv pip on the leader.

Return type:: str

destroyCluster()¶

Destroy the cluster we built, if it exists.

Succeeds if the cluster does not currently exist.

Return type:: None

setUp()¶

Set up for the test. Must be overridden to call this method and set self.jobStore.

Return type:: None

tearDown()¶

Hook method for deconstructing the test fixture after testing it.

Return type:: None

sshUtil(command)¶

Run the given command on the cluster. Raise subprocess.CalledProcessError if it fails.

Parameters:: command (List[str])
Return type:: None

rsync_util(from_file, to_file)¶

Transfer a file to/from the cluster.

The cluster-side path should have a ‘:’ in front of it.

Parameters:

from_file (str)
to_file (str)

Return type:

None

createClusterUtil(args=None)¶

Parameters:: args (Optional[List[str]])
Return type:: None

launchCluster()¶

Return type:: None

class toil.test.provisioners.clusterTest.CWLOnARMTest(methodName)¶

Bases: AbstractClusterTest

Run the CWL 1.2 conformance tests on ARM specifically.

Parameters:: methodName (str)

setUp()¶

Set up for the test. Must be overridden to call this method and set self.jobStore.

Return type:: None

test_cwl_on_arm()¶

Return type:: None

toil.test.provisioners.clusterTest¶

Attributes¶

Classes¶

Functions¶

Module Contents¶

Toil

Navigation

Related Topics