toil.test.src.fileStoreTest

Module Contents

Classes

hidden

Hiding the abstract test classes from the Unittest loader so it can be inherited in different

NonCachingFileStoreTestWithFileJobStore

Abstract tests for the the various functions in

CachingFileStoreTestWithFileJobStore

Abstract tests for the the various cache-related functions in

NonCachingFileStoreTestWithAwsJobStore

Abstract tests for the the various functions in

CachingFileStoreTestWithAwsJobStore

Abstract tests for the the various cache-related functions in

NonCachingFileStoreTestWithGoogleJobStore

Abstract tests for the the various functions in

CachingFileStoreTestWithGoogleJobStore

Abstract tests for the the various cache-related functions in

Attributes

testingIsAutomatic

logger

toil.test.src.fileStoreTest.testingIsAutomatic = True
toil.test.src.fileStoreTest.logger
class toil.test.src.fileStoreTest.hidden[source]

Hiding the abstract test classes from the Unittest loader so it can be inherited in different test suites for the different job stores.

class AbstractFileStoreTest(methodName='runTest')[source]

Bases: toil.test.ToilTest

An abstract base class for testing the various general functions described in :class:toil.fileStores.abstractFileStore.AbstractFileStore

jobStoreType
setUp()[source]

Hook method for setting up the test fixture before exercising it.

create_file(content, executable=False)[source]
testToilIsNotBroken()[source]

Runs a simple DAG to test if if any features other that caching were broken.

testFileStoreLogging()[source]

Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files.

testFileStoreOperations()[source]

Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files.

testWriteReadGlobalFilePermissions()[source]

Ensures that uploaded files preserve their file permissions when they are downloaded again. This function checks that a written executable file maintains its executability after being read.

testWriteExportFileCompatibility()[source]

Ensures that files created in a job preserve their executable permissions when they are exported from the leader.

testImportReadFileCompatibility()[source]

Ensures that files imported to the leader preserve their executable permissions when they are read by the fileStore.

testReadWriteFileStreamTextMode()[source]

Checks if text mode is compatible with file streams.

class AbstractNonCachingFileStoreTest(methodName='runTest')[source]

Bases: AbstractFileStoreTest

Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore.

setUp()[source]

Hook method for setting up the test fixture before exercising it.

class AbstractCachingFileStoreTest(methodName='runTest')[source]

Bases: AbstractFileStoreTest

Abstract tests for the the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore.

setUp()[source]

Hook method for setting up the test fixture before exercising it.

testExtremeCacheSetup()[source]

Try to create the cache with bad worker active and then have 10 child jobs try to run in the chain. This tests whether the cache is created properly even when the job crashes randomly.

testCacheEvictionPartialEvict()[source]

Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected.

testCacheEvictionTotalEvict()[source]

Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected.

testCacheEvictionFailCase()[source]

Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected.

testAsyncWriteWithCaching()[source]

Ensure the Async Writing of files happens as expected. The first Job forcefully modifies the cache size to 1GB. The second asks for 1GB of disk and writes a 900MB file into cache then rewrites it to the job store triggering an async write since the two unique jobstore IDs point to the same local file. Also, the second write is not cached since the first was written to cache, and there “isn’t enough space” to cache the second. Immediately assert that the second write isn’t cached, and is being asynchronously written to the job store.

Attempting to get the file from the jobstore should not fail.

testWriteNonLocalFileToJobStore()[source]

Write a file not in localTempDir to the job store. Such a file should not be cached. Ensure the file is not cached.

testWriteLocalFileToJobStore()[source]

Write a file from the localTempDir to the job store. Such a file will be cached by default. Ensure the file is cached.

testReadCacheMissFileFromJobStoreWithoutCachingReadFile()[source]

Read a file from the file store that does not have a corresponding cached copy. Do not cache the read file. Ensure the number of links on the file are appropriate.

testReadCacheMissFileFromJobStoreWithCachingReadFile()[source]

Read a file from the file store that does not have a corresponding cached copy. Cache the read file. Ensure the number of links on the file are appropriate.

testReadCachHitFileFromJobStore()[source]

Read a file from the file store that has a corresponding cached copy. Ensure the number of links on the file are appropriate.

testMultipleJobsReadSameCacheHitGlobalFile()[source]

Write a local file to the job store (hence adding a copy to cache), then have 10 jobs read it. Assert cached file size never goes up, assert unused job required disk space is always (a multiple of job reqs) - (number of current file readers * filesize). At the end, assert the cache shows unused job-required space = 0.

testMultipleJobsReadSameCacheMissGlobalFile()[source]

Write a non-local file to the job store(hence no cached copy), then have 10 jobs read it. Assert cached file size never goes up, assert unused job required disk space is always (a multiple of job reqs) - (number of current file readers * filesize). At the end, assert the cache shows unused job-required space = 0.

testFileStoreExportFile()[source]
testReturnFileSizes()[source]

Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files. Ensure that after every step that the cache is in a valid state.

testReturnFileSizesWithBadWorker()[source]

Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files. Ensure that after every step that the cache is in a valid state.

testControlledFailedWorkerRetry()[source]

Conduct a couple of job store operations. Then die. Ensure that the restarted job is tracking values in the cache state file appropriately.

testRemoveLocalMutablyReadFile()[source]

If a mutably read file is deleted by the user, it is ok.

testRemoveLocalImmutablyReadFile()[source]

If an immutably read file is deleted by the user, it is not ok.

testDeleteLocalFile()[source]

Test the deletion capabilities of deleteLocalFile

testSimultaneousReadsUncachedStream()[source]

Test many simultaneous read attempts on a file created via a stream directly to the job store.

class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore(methodName='runTest')[source]

Bases: hidden

Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore.

jobStoreType = 'file'
class toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore(methodName='runTest')[source]

Bases: hidden

Abstract tests for the the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore.

jobStoreType = 'file'
class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithAwsJobStore(methodName='runTest')[source]

Bases: hidden

Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore.

jobStoreType = 'aws'
class toil.test.src.fileStoreTest.CachingFileStoreTestWithAwsJobStore(methodName='runTest')[source]

Bases: hidden

Abstract tests for the the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore.

jobStoreType = 'aws'
class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithGoogleJobStore(methodName='runTest')[source]

Bases: hidden

Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore.

jobStoreType = 'google'
class toil.test.src.fileStoreTest.CachingFileStoreTestWithGoogleJobStore(methodName='runTest')[source]

Bases: hidden

Abstract tests for the the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore.

jobStoreType = 'google'