Toil Documentation¶
Toil is an open-source pure-Python workflow engine that lets people write better pipelines.
Check out our website for a comprehensive list of Toil’s features and read our paper to learn what Toil can do in the real world. Please subscribe to our low-volume announce mailing list and feel free to also join us on GitHub and Gitter.
If using Toil for your research, please cite
Vivian, J., Rao, A. A., Nothaft, F. A., Ketchum, C., Armstrong, J., Novak, A., … Paten, B. (2017). Toil enables reproducible, open source, big biomedical data analyses. Nature Biotechnology, 35(4), 314–316. http://doi.org/10.1038/nbt.3772
- Introduction
- Commandline Options
- Toil Utilities
- Toil Debugging
- Running in the Cloud
- Cloud Platforms
- HPC Environments
- Workflow Execution Service (WES)
- Developing a Python Workflow
- Scripting Quick Start
- Job Basics
- Invoking a Workflow
- Specifying Commandline Arguments
- Resuming a Workflow
- Functions and Job Functions
- Workflows with Multiple Jobs
- Dynamic Job Creation
- Promises
- Promised Requirements
- FileID
- Managing files within a workflow
- Using Docker Containers in Toil
- Services
- Checkpoints
- Encapsulation
- Depending on Toil
- Best Practices for Dockerizing Toil Workflows
- Toil Class API
- Job Store API
- Toil Job API
- Job Methods API
- Job.Runner API
- job.fileStore API
- Batch System API
- Job.Service API
- Exceptions API
- Toil Architecture
- Minimum AWS IAM permissions
- Auto-Deployment
- Environment Variables
- API Reference
- toil
- Submodules
- toil.batchSystems
- Submodules
- toil.batchSystems.abstractBatchSystem
- Attributes
- Exceptions
- Classes
- Module Contents
logger
EXIT_STATUS_UNAVAILABLE_VALUE
BatchJobExitReason
UpdatedBatchJobInfo
WorkerCleanupInfo
AbstractBatchSystem
AbstractBatchSystem.supportsAutoDeployment()
AbstractBatchSystem.supportsWorkerCleanup()
AbstractBatchSystem.setUserScript()
AbstractBatchSystem.set_message_bus()
AbstractBatchSystem.issueBatchJob()
AbstractBatchSystem.killBatchJobs()
AbstractBatchSystem.getIssuedBatchJobIDs()
AbstractBatchSystem.getRunningBatchJobIDs()
AbstractBatchSystem.getUpdatedBatchJob()
AbstractBatchSystem.getSchedulingStatusMessage()
AbstractBatchSystem.shutdown()
AbstractBatchSystem.setEnv()
AbstractBatchSystem.add_options()
AbstractBatchSystem.setOptions()
AbstractBatchSystem.getWorkerContexts()
BatchSystemSupport
BatchSystemSupport.config
BatchSystemSupport.maxCores
BatchSystemSupport.maxMemory
BatchSystemSupport.maxDisk
BatchSystemSupport.environment
BatchSystemSupport.check_resource_request()
BatchSystemSupport.setEnv()
BatchSystemSupport.set_message_bus()
BatchSystemSupport.get_batch_logs_dir()
BatchSystemSupport.format_std_out_err_path()
BatchSystemSupport.format_std_out_err_glob()
BatchSystemSupport.workerCleanup()
NodeInfo
AbstractScalableBatchSystem
InsufficientSystemResources
AcquisitionTimeoutException
ResourcePool
ResourceSet
- toil.batchSystems.abstractGridEngineBatchSystem
- Attributes
- Exceptions
- Classes
- Module Contents
logger
JobTuple
ExceededRetryAttempts
AbstractGridEngineBatchSystem
AbstractGridEngineBatchSystem.GridEngineThreadException
AbstractGridEngineBatchSystem.GridEngineThread
AbstractGridEngineBatchSystem.GridEngineThread.boss
AbstractGridEngineBatchSystem.GridEngineThread.newJobsQueue
AbstractGridEngineBatchSystem.GridEngineThread.updatedJobsQueue
AbstractGridEngineBatchSystem.GridEngineThread.killQueue
AbstractGridEngineBatchSystem.GridEngineThread.killedJobsQueue
AbstractGridEngineBatchSystem.GridEngineThread.waitingJobs
AbstractGridEngineBatchSystem.GridEngineThread.runningJobs
AbstractGridEngineBatchSystem.GridEngineThread.runningJobsLock
AbstractGridEngineBatchSystem.GridEngineThread.batchJobIDs
AbstractGridEngineBatchSystem.GridEngineThread.exception
AbstractGridEngineBatchSystem.GridEngineThread.getBatchSystemID()
AbstractGridEngineBatchSystem.GridEngineThread.forgetJob()
AbstractGridEngineBatchSystem.GridEngineThread.createJobs()
AbstractGridEngineBatchSystem.GridEngineThread.killJobs()
AbstractGridEngineBatchSystem.GridEngineThread.checkOnJobs()
AbstractGridEngineBatchSystem.GridEngineThread.run()
AbstractGridEngineBatchSystem.GridEngineThread.coalesce_job_exit_codes()
AbstractGridEngineBatchSystem.GridEngineThread.prepareSubmission()
AbstractGridEngineBatchSystem.GridEngineThread.submitJob()
AbstractGridEngineBatchSystem.GridEngineThread.getRunningJobIDs()
AbstractGridEngineBatchSystem.GridEngineThread.killJob()
AbstractGridEngineBatchSystem.GridEngineThread.getJobExitCode()
AbstractGridEngineBatchSystem.config
AbstractGridEngineBatchSystem.currentJobs
AbstractGridEngineBatchSystem.newJobsQueue
AbstractGridEngineBatchSystem.updatedJobsQueue
AbstractGridEngineBatchSystem.killQueue
AbstractGridEngineBatchSystem.killedJobsQueue
AbstractGridEngineBatchSystem.background_thread
AbstractGridEngineBatchSystem.supportsAutoDeployment()
AbstractGridEngineBatchSystem.count_needed_gpus()
AbstractGridEngineBatchSystem.issueBatchJob()
AbstractGridEngineBatchSystem.killBatchJobs()
AbstractGridEngineBatchSystem.getIssuedBatchJobIDs()
AbstractGridEngineBatchSystem.getRunningBatchJobIDs()
AbstractGridEngineBatchSystem.getUpdatedBatchJob()
AbstractGridEngineBatchSystem.shutdown()
AbstractGridEngineBatchSystem.setEnv()
AbstractGridEngineBatchSystem.getWaitDuration()
AbstractGridEngineBatchSystem.sleepSeconds()
AbstractGridEngineBatchSystem.with_retries()
- toil.batchSystems.awsBatch
- Attributes
- Classes
- Module Contents
logger
STATE_TO_EXIT_REASON
MAX_POLL_COUNT
MIN_REQUESTABLE_MIB
MIN_REQUESTABLE_CORES
AWSBatchBatchSystem
AWSBatchBatchSystem.supportsAutoDeployment()
AWSBatchBatchSystem.region
AWSBatchBatchSystem.client
AWSBatchBatchSystem.queue
AWSBatchBatchSystem.job_role_arn
AWSBatchBatchSystem.owner_tag
AWSBatchBatchSystem.worker_work_dir
AWSBatchBatchSystem.user_script
AWSBatchBatchSystem.docker_image
AWSBatchBatchSystem.job_definition
AWSBatchBatchSystem.bs_id_to_aws_id
AWSBatchBatchSystem.aws_id_to_bs_id
AWSBatchBatchSystem.killed_job_aws_ids
AWSBatchBatchSystem.setUserScript()
AWSBatchBatchSystem.issueBatchJob()
AWSBatchBatchSystem.getUpdatedBatchJob()
AWSBatchBatchSystem.shutdown()
AWSBatchBatchSystem.getIssuedBatchJobIDs()
AWSBatchBatchSystem.getRunningBatchJobIDs()
AWSBatchBatchSystem.killBatchJobs()
AWSBatchBatchSystem.add_options()
AWSBatchBatchSystem.setOptions()
- toil.batchSystems.cleanup_support
- toil.batchSystems.contained_executor
- toil.batchSystems.gridengine
- Attributes
- Classes
- Module Contents
logger
GridEngineBatchSystem
GridEngineBatchSystem.GridEngineThread
GridEngineBatchSystem.GridEngineThread.getRunningJobIDs()
GridEngineBatchSystem.GridEngineThread.killJob()
GridEngineBatchSystem.GridEngineThread.prepareSubmission()
GridEngineBatchSystem.GridEngineThread.submitJob()
GridEngineBatchSystem.GridEngineThread.getJobExitCode()
GridEngineBatchSystem.GridEngineThread.prepareQsub()
GridEngineBatchSystem.getWaitDuration()
- toil.batchSystems.htcondor
- Attributes
- Classes
- Module Contents
logger
JobTuple
schedd_lock
HTCondorBatchSystem
HTCondorBatchSystem.GridEngineThread
HTCondorBatchSystem.GridEngineThread.createJobs()
HTCondorBatchSystem.GridEngineThread.prepareSubmission()
HTCondorBatchSystem.GridEngineThread.submitJob()
HTCondorBatchSystem.GridEngineThread.getRunningJobIDs()
HTCondorBatchSystem.GridEngineThread.killJob()
HTCondorBatchSystem.GridEngineThread.getJobExitCode()
HTCondorBatchSystem.GridEngineThread.connectSchedd()
HTCondorBatchSystem.GridEngineThread.duplicate_quotes()
HTCondorBatchSystem.GridEngineThread.getEnvString()
HTCondorBatchSystem.issueBatchJob()
- toil.batchSystems.kubernetes
- Attributes
- Classes
- Functions
- Module Contents
logger
retryable_kubernetes_errors
is_retryable_kubernetes_error()
KeyValuesList
KubernetesBatchSystem
KubernetesBatchSystem.supportsAutoDeployment()
KubernetesBatchSystem.credential_time
KubernetesBatchSystem.namespace
KubernetesBatchSystem.host_path
KubernetesBatchSystem.service_account
KubernetesBatchSystem.pod_timeout
KubernetesBatchSystem.unique_id
KubernetesBatchSystem.job_prefix
KubernetesBatchSystem.finished_job_ttl
KubernetesBatchSystem.user_script
KubernetesBatchSystem.docker_image
KubernetesBatchSystem.worker_work_dir
KubernetesBatchSystem.aws_secret_name
KubernetesBatchSystem.enable_watching
KubernetesBatchSystem.run_id
KubernetesBatchSystem.resource_sources
KubernetesBatchSystem.schedulingThread
KubernetesBatchSystem.DecoratorWrapper
KubernetesBatchSystem.ItemT
KubernetesBatchSystem.CovItemT
KubernetesBatchSystem.P
KubernetesBatchSystem.R
KubernetesBatchSystem.setUserScript()
KubernetesBatchSystem.Placement
KubernetesBatchSystem.issueBatchJob()
KubernetesBatchSystem.getUpdatedBatchJob()
KubernetesBatchSystem.shutdown()
KubernetesBatchSystem.getIssuedBatchJobIDs()
KubernetesBatchSystem.getRunningBatchJobIDs()
KubernetesBatchSystem.killBatchJobs()
KubernetesBatchSystem.get_default_kubernetes_owner()
KubernetesBatchSystem.KubernetesConfig
KubernetesBatchSystem.add_options()
KubernetesBatchSystem.OptionType
KubernetesBatchSystem.setOptions()
- toil.batchSystems.local_support
- Attributes
- Classes
- Module Contents
logger
BatchSystemLocalSupport
BatchSystemLocalSupport.localBatch
BatchSystemLocalSupport.handleLocalJob()
BatchSystemLocalSupport.killLocalJobs()
BatchSystemLocalSupport.getIssuedLocalJobIDs()
BatchSystemLocalSupport.getRunningLocalJobIDs()
BatchSystemLocalSupport.getUpdatedLocalJob()
BatchSystemLocalSupport.getNextJobID()
BatchSystemLocalSupport.shutdownLocal()
- toil.batchSystems.lsf
- Attributes
- Classes
- Module Contents
logger
LSFBatchSystem
LSFBatchSystem.GridEngineThread
LSFBatchSystem.GridEngineThread.getRunningJobIDs()
LSFBatchSystem.GridEngineThread.fallbackRunningJobIDs()
LSFBatchSystem.GridEngineThread.killJob()
LSFBatchSystem.GridEngineThread.prepareSubmission()
LSFBatchSystem.GridEngineThread.submitJob()
LSFBatchSystem.GridEngineThread.coalesce_job_exit_codes()
LSFBatchSystem.GridEngineThread.getJobExitCode()
LSFBatchSystem.GridEngineThread.parse_bjobs_record()
LSFBatchSystem.GridEngineThread.getJobExitCodeBACCT()
LSFBatchSystem.GridEngineThread.fallbackGetJobExitCode()
LSFBatchSystem.GridEngineThread.prepareBsub()
LSFBatchSystem.GridEngineThread.parseBjobs()
LSFBatchSystem.GridEngineThread.parseMaxMem()
LSFBatchSystem.getWaitDuration()
- toil.batchSystems.lsfHelper
- Attributes
- Functions
- Module Contents
LSB_PARAMS_FILENAME
LSF_CONF_FILENAME
LSF_CONF_ENV
DEFAULT_LSF_UNITS
DEFAULT_RESOURCE_UNITS
LSF_JSON_OUTPUT_MIN_VERSION
logger
find()
find_first_match()
get_conf_file()
apply_conf_file()
per_core_reserve_from_stream()
get_lsf_units_from_stream()
tokenize_conf_stream()
apply_bparams()
apply_lsadmin()
get_lsf_units()
parse_mem_and_cmd_from_output()
get_lsf_version()
check_lsf_json_output_supported()
parse_memory()
per_core_reservation()
- toil.batchSystems.mesos
- Submodules
- toil.batchSystems.mesos.batchSystem
- Attributes
- Classes
- Module Contents
log
MesosBatchSystem
MesosBatchSystem.supportsAutoDeployment()
MesosBatchSystem.supportsWorkerCleanup()
MesosBatchSystem.ExecutorInfo
MesosBatchSystem.userScript
MesosBatchSystem.jobQueues
MesosBatchSystem.mesos_endpoint
MesosBatchSystem.mesos_name
MesosBatchSystem.killedJobIds
MesosBatchSystem.killJobIds
MesosBatchSystem.intendedKill
MesosBatchSystem.hostToJobIDs
MesosBatchSystem.nodeFilter
MesosBatchSystem.runningJobMap
MesosBatchSystem.taskResources
MesosBatchSystem.updatedJobsQueue
MesosBatchSystem.driver
MesosBatchSystem.frameworkId
MesosBatchSystem.executors
MesosBatchSystem.agentsByID
MesosBatchSystem.nonPreemptibleNodes
MesosBatchSystem.executor
MesosBatchSystem.lastTimeOfferLogged
MesosBatchSystem.logPeriod
MesosBatchSystem.ignoredNodes
MesosBatchSystem.setUserScript()
MesosBatchSystem.ignoreNode()
MesosBatchSystem.unignoreNode()
MesosBatchSystem.issueBatchJob()
MesosBatchSystem.killBatchJobs()
MesosBatchSystem.getIssuedBatchJobIDs()
MesosBatchSystem.getRunningBatchJobIDs()
MesosBatchSystem.getUpdatedBatchJob()
MesosBatchSystem.nodeInUse()
MesosBatchSystem.getWaitDuration()
MesosBatchSystem.shutdown()
MesosBatchSystem.registered()
MesosBatchSystem.resourceOffers()
MesosBatchSystem.statusUpdate()
MesosBatchSystem.frameworkMessage()
MesosBatchSystem.getNodes()
MesosBatchSystem.reregistered()
MesosBatchSystem.executorLost()
MesosBatchSystem.get_default_mesos_endpoint()
MesosBatchSystem.add_options()
MesosBatchSystem.setOptions()
- toil.batchSystems.mesos.conftest
- toil.batchSystems.mesos.executor
- Attributes
- Classes
- Functions
- Module Contents
log
MesosExecutor
MesosExecutor.popenLock
MesosExecutor.runningTasks
MesosExecutor.workerCleanupInfo
MesosExecutor.address
MesosExecutor.id
MesosExecutor.registered()
MesosExecutor.reregistered()
MesosExecutor.disconnected()
MesosExecutor.killTask()
MesosExecutor.shutdown()
MesosExecutor.error()
MesosExecutor.launchTask()
MesosExecutor.frameworkMessage()
main()
- toil.batchSystems.mesos.test
- toil.batchSystems.mesos.batchSystem
- Classes
- Package Contents
- Submodules
- toil.batchSystems.options
- toil.batchSystems.registry
- Attributes
- Functions
- Module Contents
logger
add_batch_system_factory()
get_batch_systems()
get_batch_system()
DEFAULT_BATCH_SYSTEM
aws_batch_batch_system_factory()
gridengine_batch_system_factory()
lsf_batch_system_factory()
single_machine_batch_system_factory()
mesos_batch_system_factory()
slurm_batch_system_factory()
torque_batch_system_factory()
htcondor_batch_system_factory()
kubernetes_batch_system_factory()
__getattr__()
addBatchSystemFactory()
save_batch_system_plugin_state()
restore_batch_system_plugin_state()
- toil.batchSystems.singleMachine
- Attributes
- Classes
- Module Contents
logger
SingleMachineBatchSystem
SingleMachineBatchSystem.supportsAutoDeployment()
SingleMachineBatchSystem.supportsWorkerCleanup()
SingleMachineBatchSystem.numCores
SingleMachineBatchSystem.minCores
SingleMachineBatchSystem.physicalMemory
SingleMachineBatchSystem.config
SingleMachineBatchSystem.physicalDisk
SingleMachineBatchSystem.scale
SingleMachineBatchSystem.debugWorker
SingleMachineBatchSystem.jobIndex
SingleMachineBatchSystem.jobIndexLock
SingleMachineBatchSystem.jobs
SingleMachineBatchSystem.inputQueue
SingleMachineBatchSystem.outputQueue
SingleMachineBatchSystem.runningJobs
SingleMachineBatchSystem.children
SingleMachineBatchSystem.childToJob
SingleMachineBatchSystem.accelerator_identities
SingleMachineBatchSystem.resource_sources
SingleMachineBatchSystem.schedulingStatusMessage
SingleMachineBatchSystem.shuttingDown
SingleMachineBatchSystem.daddyThread
SingleMachineBatchSystem.daddyException
SingleMachineBatchSystem.daddy()
SingleMachineBatchSystem.getSchedulingStatusMessage()
SingleMachineBatchSystem.check_resource_request()
SingleMachineBatchSystem.issueBatchJob()
SingleMachineBatchSystem.killBatchJobs()
SingleMachineBatchSystem.getIssuedBatchJobIDs()
SingleMachineBatchSystem.getRunningBatchJobIDs()
SingleMachineBatchSystem.shutdown()
SingleMachineBatchSystem.getUpdatedBatchJob()
SingleMachineBatchSystem.add_options()
SingleMachineBatchSystem.setOptions()
Info
- toil.batchSystems.slurm
- Attributes
- Classes
- Functions
- Module Contents
logger
TERMINAL_STATES
NONTERMINAL_STATES
parse_slurm_time()
SlurmBatchSystem
SlurmBatchSystem.PartitionInfo
SlurmBatchSystem.PartitionSet
SlurmBatchSystem.GridEngineThread
SlurmBatchSystem.GridEngineThread.boss
SlurmBatchSystem.GridEngineThread.getRunningJobIDs()
SlurmBatchSystem.GridEngineThread.killJob()
SlurmBatchSystem.GridEngineThread.prepareSubmission()
SlurmBatchSystem.GridEngineThread.submitJob()
SlurmBatchSystem.GridEngineThread.coalesce_job_exit_codes()
SlurmBatchSystem.GridEngineThread.getJobExitCode()
SlurmBatchSystem.GridEngineThread.prepareSbatch()
SlurmBatchSystem.partitions
SlurmBatchSystem.issueBatchJob()
SlurmBatchSystem.add_options()
SlurmBatchSystem.OptionType
SlurmBatchSystem.setOptions()
- toil.batchSystems.torque
- Attributes
- Classes
- Module Contents
logger
TorqueBatchSystem
TorqueBatchSystem.GridEngineThread
TorqueBatchSystem.GridEngineThread.getRunningJobIDs()
TorqueBatchSystem.GridEngineThread.getUpdatedBatchJob()
TorqueBatchSystem.GridEngineThread.killJob()
TorqueBatchSystem.GridEngineThread.prepareSubmission()
TorqueBatchSystem.GridEngineThread.submitJob()
TorqueBatchSystem.GridEngineThread.getJobExitCode()
TorqueBatchSystem.GridEngineThread.prepareQsub()
TorqueBatchSystem.GridEngineThread.generateTorqueWrapper()
- toil.batchSystems.abstractBatchSystem
- Exceptions
- Package Contents
- Submodules
- toil.bus
- Attributes
- Classes
- Functions
- Module Contents
logger
Names
get_job_kind()
JobIssuedMessage
JobUpdatedMessage
JobCompletedMessage
JobFailedMessage
JobMissingMessage
JobAnnotationMessage
ExternalBatchIdMessage
QueueSizeMessage
ClusterSizeMessage
ClusterDesiredSizeMessage
message_to_bytes()
MessageType
bytes_to_message()
MessageBus
MessageBusClient
MessageInbox
MessageOutbox
MessageBusConnection
JobStatus
replay_message_bus()
gen_message_bus_path()
- toil.common
- Attributes
- Exceptions
- Classes
- Functions
- Module Contents
UUID_LENGTH
logger
TOIL_HOME_DIR
DEFAULT_CONFIG_FILE
Config
Config.logFile
Config.logRotating
Config.cleanWorkDir
Config.max_jobs
Config.max_local_jobs
Config.manualMemArgs
Config.run_local_jobs_on_workers
Config.coalesceStatusCalls
Config.mesos_endpoint
Config.mesos_framework_id
Config.mesos_role
Config.mesos_name
Config.kubernetes_host_path
Config.kubernetes_owner
Config.kubernetes_service_account
Config.kubernetes_pod_timeout
Config.kubernetes_privileged
Config.tes_endpoint
Config.tes_user
Config.tes_password
Config.tes_bearer_token
Config.aws_batch_region
Config.aws_batch_queue
Config.aws_batch_job_role_arn
Config.scale
Config.batchSystem
Config.batch_logs_dir
Config.statePollingWait
Config.state_polling_timeout
Config.disableAutoDeployment
Config.workflowID
Config.workflowAttemptNumber
Config.jobStore
Config.logLevel
Config.colored_logs
Config.workDir
Config.coordination_dir
Config.noStdOutErr
Config.stats
Config.clean
Config.clusterStats
Config.restart
Config.caching
Config.symlinkImports
Config.moveOutputs
Config.symlink_job_store_reads
Config.provisioner
Config.nodeTypes
Config.minNodes
Config.maxNodes
Config.targetTime
Config.betaInertia
Config.scaleInterval
Config.preemptibleCompensation
Config.nodeStorage
Config.nodeStorageOverrides
Config.metrics
Config.assume_zero_overhead
Config.maxPreemptibleServiceJobs
Config.maxServiceJobs
Config.deadlockWait
Config.deadlockCheckInterval
Config.defaultMemory
Config.defaultCores
Config.defaultDisk
Config.defaultPreemptible
Config.defaultAccelerators
Config.maxCores
Config.maxMemory
Config.maxDisk
Config.retryCount
Config.enableUnlimitedPreemptibleRetries
Config.doubleMem
Config.maxJobDuration
Config.rescueJobsFrequency
Config.job_store_timeout
Config.maxLogFileSize
Config.writeLogs
Config.writeLogsGzip
Config.writeLogsFromAllJobs
Config.write_messages
Config.realTimeLogging
Config.environment
Config.disableChaining
Config.disableJobStoreChecksumVerification
Config.sseKey
Config.servicePollingInterval
Config.useAsync
Config.forceDockerAppliance
Config.statusWait
Config.disableProgress
Config.readGlobalFileMutableByDefault
Config.debugWorker
Config.disableWorkerOutputCapture
Config.badWorker
Config.badWorkerFailInterval
Config.kill_polling_interval
Config.cwl
Config.memory_is_product
Config.set_from_default_config()
Config.prepare_start()
Config.prepare_restart()
Config.setOptions()
Config.check_configuration_consistency()
Config.__eq__()
Config.__hash__()
check_and_create_toil_home_dir()
check_and_create_default_config_file()
check_and_create_config_file()
generate_config()
parser_with_common_options()
addOptions()
getNodeID()
Toil
Toil.config
Toil.options
Toil.__enter__()
Toil.__exit__()
Toil.start()
Toil.restart()
Toil.getJobStore()
Toil.parseLocator()
Toil.buildLocator()
Toil.resumeJobStore()
Toil.createBatchSystem()
Toil.url_exists()
Toil.importFile()
Toil.import_file()
Toil.exportFile()
Toil.export_file()
Toil.normalize_uri()
Toil.getToilWorkDir()
Toil.get_toil_coordination_dir()
Toil.get_workflow_path_component()
Toil.getLocalWorkflowDir()
Toil.get_local_workflow_coordination_dir()
ToilRestartException
ToilContextManagerException
ToilMetrics
ToilMetrics.mtailImage
ToilMetrics.grafanaImage
ToilMetrics.prometheusImage
ToilMetrics.nodeExporterProc
ToilMetrics.startDashboard()
ToilMetrics.add_prometheus_data_source()
ToilMetrics.log()
ToilMetrics.logClusterSize()
ToilMetrics.logClusterDesiredSize()
ToilMetrics.logQueueSize()
ToilMetrics.logMissingJob()
ToilMetrics.logIssuedJob()
ToilMetrics.logFailedJob()
ToilMetrics.logCompletedJob()
ToilMetrics.shutdown()
cacheDirName()
getDirSizeRecursively()
getFileSystemSize()
safeUnpickleFromStream()
- toil.cwl
- Submodules
- toil.cwl.conftest
- toil.cwl.cwltoil
- Attributes
- Exceptions
- Classes
- Functions
- Module Contents
logger
DEFAULT_TMPDIR
DEFAULT_TMPDIR_PREFIX
cwltoil_was_removed()
UnresolvedDict
SkipNull
filter_skip_null()
ensure_no_collisions()
try_prepull()
Conditional
ResolveSource
StepValueFrom
DefaultWithSource
JustAValue
resolve_dict_w_promises()
simplify_list()
ToilPathMapper
ToilSingleJobExecutor
ToilTool
ToilCommandLineTool
ToilExpressionTool
toil_make_tool()
MISSING_FILE
DirectoryContents
check_directory_dict_invariants()
decode_directory()
encode_directory()
ToilFsAccess
toil_get_file()
convert_file_uri_to_toil_uri()
path_to_loc()
extract_file_uri_once()
V
VisitFunc
visit_files()
upload_directory()
extract_and_convert_file_to_toil_uri()
writeGlobalFileWrapper()
remove_empty_listings()
CWLNamedJob
ResolveIndirect
toilStageFiles()
CWLJobWrapper
CWLJob
get_container_engine()
makeRootJob()
makeJob()
CWLScatter
CWLGather
SelfJob
ProcessType
remove_pickle_problems()
CWLWorkflow
CWLInstallImportsJob
CWLImportWrapper
CWLStartJob
extract_workflow_inputs()
import_workflow_inputs()
T
visitSteps()
rm_unprocessed_secondary_files()
filtered_secondary_files()
scan_for_unsupported_requirements()
determine_load_listing()
NoAvailableJobStoreException
generate_default_job_store()
usage_message
get_options()
main()
find_default_container()
- toil.cwl.utils
- Functions
- Package Contents
- Submodules
- toil.deferred
- Attributes
- Classes
- Module Contents
logger
DeferredFunction
DeferredFunctionManager
DeferredFunctionManager.STATE_DIR_STEM
DeferredFunctionManager.PREFIX
DeferredFunctionManager.WIP_SUFFIX
DeferredFunctionManager.stateDir
DeferredFunctionManager.stateFileName
DeferredFunctionManager.stateFileOut
DeferredFunctionManager.stateFileIn
DeferredFunctionManager.__del__()
DeferredFunctionManager.open()
DeferredFunctionManager.cleanupWorker()
- toil.exceptions
- toil.fileStores
- Submodules
- toil.fileStores.abstractFileStore
- Attributes
- Classes
- Module Contents
logger
AbstractFileStore
AbstractFileStore.jobStore
AbstractFileStore.jobDesc
AbstractFileStore.localTempDir
AbstractFileStore.workflow_dir
AbstractFileStore.coordination_dir
AbstractFileStore.jobName
AbstractFileStore.waitForPreviousCommit
AbstractFileStore.logging_messages
AbstractFileStore.logging_user_streams
AbstractFileStore.filesToDelete
AbstractFileStore.createFileStore()
AbstractFileStore.shutdownFileStore()
AbstractFileStore.open()
AbstractFileStore.get_disk_usage()
AbstractFileStore.getLocalTempDir()
AbstractFileStore.getLocalTempFile()
AbstractFileStore.getLocalTempFileName()
AbstractFileStore.writeGlobalFile()
AbstractFileStore.writeGlobalFileStream()
AbstractFileStore.logAccess()
AbstractFileStore.readGlobalFile()
AbstractFileStore.readGlobalFileStream()
AbstractFileStore.getGlobalFileSize()
AbstractFileStore.deleteLocalFile()
AbstractFileStore.deleteGlobalFile()
AbstractFileStore.importFile()
AbstractFileStore.import_file()
AbstractFileStore.exportFile()
AbstractFileStore.export_file()
AbstractFileStore.log_to_leader()
AbstractFileStore.logToMaster()
AbstractFileStore.log_user_stream()
AbstractFileStore.startCommit()
AbstractFileStore.waitForCommit()
AbstractFileStore.shutdown()
- toil.fileStores.cachingFileStore
- Attributes
- Exceptions
- Classes
- Module Contents
logger
SQLITE_TIMEOUT_SECS
CacheError
CacheUnbalancedError
IllegalDeletionCacheError
InvalidSourceCacheError
CachingFileStore
CachingFileStore.forceNonFreeCaching
CachingFileStore.forceDownloadDelay
CachingFileStore.contentionBackoff
CachingFileStore.localCacheDir
CachingFileStore.jobName
CachingFileStore.jobID
CachingFileStore.jobDiskBytes
CachingFileStore.workflowAttemptNumber
CachingFileStore.dbPath
CachingFileStore.process_identity_lock
CachingFileStore.commitThread
CachingFileStore.as_process()
CachingFileStore.con
CachingFileStore.cur
CachingFileStore.getCacheLimit()
CachingFileStore.getCacheUsed()
CachingFileStore.getCacheExtraJobSpace()
CachingFileStore.getCacheAvailable()
CachingFileStore.getSpaceUsableForJobs()
CachingFileStore.getCacheUnusedJobRequirement()
CachingFileStore.adjustCacheLimit()
CachingFileStore.fileIsCached()
CachingFileStore.getFileReaderCount()
CachingFileStore.cachingIsFree()
CachingFileStore.open()
CachingFileStore.writeGlobalFile()
CachingFileStore.readGlobalFile()
CachingFileStore.readGlobalFileStream()
CachingFileStore.deleteLocalFile()
CachingFileStore.deleteGlobalFile()
CachingFileStore.exportFile()
CachingFileStore.export_file()
CachingFileStore.waitForCommit()
CachingFileStore.startCommit()
CachingFileStore.startCommitThread()
CachingFileStore.shutdown()
CachingFileStore.__del__()
- toil.fileStores.nonCachingFileStore
- Attributes
- Classes
- Module Contents
logger
NonCachingFileStore
NonCachingFileStore.jobStateFile
NonCachingFileStore.localFileMap
NonCachingFileStore.check_for_coordination_corruption()
NonCachingFileStore.check_for_state_corruption()
NonCachingFileStore.open()
NonCachingFileStore.writeGlobalFile()
NonCachingFileStore.readGlobalFile()
NonCachingFileStore.readGlobalFileStream()
NonCachingFileStore.exportFile()
NonCachingFileStore.export_file()
NonCachingFileStore.deleteLocalFile()
NonCachingFileStore.deleteGlobalFile()
NonCachingFileStore.waitForCommit()
NonCachingFileStore.startCommit()
NonCachingFileStore.__del__()
NonCachingFileStore.shutdown()
- toil.fileStores.abstractFileStore
- Classes
- Package Contents
- Submodules
- toil.job
- Attributes
- Exceptions
- Classes
- Functions
- Module Contents
logger
JobPromiseConstraintError
ConflictingPredecessorError
DebugStoppingPointReached
FilesDownloadedStoppingPointReached
TemporaryID
AcceleratorRequirement
parse_accelerator()
accelerator_satisfies()
accelerators_fully_satisfy()
RequirementsDict
REQUIREMENT_NAMES
ParsedRequirement
ParseableIndivisibleResource
ParseableDivisibleResource
ParseableFlag
ParseableAcceleratorRequirement
ParseableRequirement
Requirer
JobBodyReference
JobDescription
JobDescription.local
JobDescription.jobName
JobDescription.unitName
JobDescription.displayName
JobDescription.jobStoreID
JobDescription.filesToDelete
JobDescription.predecessorNumber
JobDescription.predecessorsFinished
JobDescription.childIDs
JobDescription.followOnIDs
JobDescription.successor_phases
JobDescription.serviceTree
JobDescription.logJobStoreFileID
JobDescription.files_to_use
JobDescription.get_names()
JobDescription.get_chain()
JobDescription.serviceHostIDsInBatches()
JobDescription.successorsAndServiceHosts()
JobDescription.allSuccessors()
JobDescription.successors_by_phase()
JobDescription.services
JobDescription.has_body()
JobDescription.attach_body()
JobDescription.detach_body()
JobDescription.get_body()
JobDescription.nextSuccessors()
JobDescription.filterSuccessors()
JobDescription.filterServiceHosts()
JobDescription.clear_nonexistent_dependents()
JobDescription.clear_dependents()
JobDescription.is_subtree_done()
JobDescription.replace()
JobDescription.assert_is_not_newer_than()
JobDescription.is_updated_by()
JobDescription.addChild()
JobDescription.addFollowOn()
JobDescription.addServiceHostJob()
JobDescription.hasChild()
JobDescription.hasFollowOn()
JobDescription.hasServiceHostJob()
JobDescription.renameReferences()
JobDescription.addPredecessor()
JobDescription.onRegistration()
JobDescription.setupJobAfterFailure()
JobDescription.getLogFileHandle()
JobDescription.remainingTryCount
JobDescription.clearRemainingTryCount()
JobDescription.__str__()
JobDescription.__repr__()
JobDescription.reserve_versions()
JobDescription.pre_update_hook()
ServiceJobDescription
CheckpointJobDescription
Job
Job.userModule
Job.__str__()
Job.check_initialized()
Job.jobStoreID
Job.description
Job.disk
Job.memory
Job.cores
Job.accelerators
Job.preemptible
Job.preemptable()
Job.checkpoint
Job.files_to_use
Job.add_to_files_to_use()
Job.remove_from_files_to_use()
Job.assignConfig()
Job.run()
Job.addChild()
Job.hasChild()
Job.addFollowOn()
Job.hasPredecessor()
Job.hasFollowOn()
Job.addService()
Job.hasService()
Job.addChildFn()
Job.addFollowOnFn()
Job.addChildJobFn()
Job.addFollowOnJobFn()
Job.tempDir
Job.log()
Job.wrapFn()
Job.wrapJobFn()
Job.encapsulate()
Job.rv()
Job.registerPromise()
Job.prepareForPromiseRegistration()
Job.checkJobGraphForDeadlocks()
Job.getRootJobs()
Job.checkJobGraphConnected()
Job.checkJobGraphAcylic()
Job.checkNewCheckpointsAreLeafVertices()
Job.defer()
Job.Runner
Job.Service
Job.getUserScript()
Job.getTopologicalOrderingOfJobs()
Job.saveBody()
Job.saveAsRootJob()
Job.loadJob()
Job.set_debug_flag()
Job.has_debug_flag()
Job.files_downloaded_hook()
JobException
JobGraphDeadlockException
FunctionWrappingJob
JobFunctionWrappingJob
PromisedRequirementFunctionWrappingJob
PromisedRequirementJobFunctionWrappingJob
EncapsulatedJob
ServiceHostJob
FileMetadata
potential_absolute_uris()
get_file_sizes()
CombineImportsJob
WorkerImportJob
ImportsJob
Promise
T
Promised
unwrap()
unwrap_all()
PromisedRequirement
UnfulfilledPromiseSentinel
- toil.jobStores
- Submodules
- toil.jobStores.abstractJobStore
- Attributes
- Exceptions
- Classes
- Module Contents
logger
ProxyConnectionError
LocatorException
InvalidImportExportUrlException
NoSuchJobException
ConcurrentFileModificationException
NoSuchFileException
NoSuchJobStoreException
JobStoreExistsException
AbstractJobStore
AbstractJobStore.initialize()
AbstractJobStore.writeConfig()
AbstractJobStore.write_config()
AbstractJobStore.resume()
AbstractJobStore.config
AbstractJobStore.locator
AbstractJobStore.rootJobStoreIDFileName
AbstractJobStore.setRootJob()
AbstractJobStore.set_root_job()
AbstractJobStore.loadRootJob()
AbstractJobStore.load_root_job()
AbstractJobStore.createRootJob()
AbstractJobStore.create_root_job()
AbstractJobStore.getRootJobReturnValue()
AbstractJobStore.get_root_job_return_value()
AbstractJobStore.importFile()
AbstractJobStore.import_file()
AbstractJobStore.exportFile()
AbstractJobStore.export_file()
AbstractJobStore.url_exists()
AbstractJobStore.get_size()
AbstractJobStore.get_is_directory()
AbstractJobStore.list_url()
AbstractJobStore.read_from_url()
AbstractJobStore.open_url()
AbstractJobStore.destroy()
AbstractJobStore.getEnv()
AbstractJobStore.get_env()
AbstractJobStore.clean()
AbstractJobStore.assignID()
AbstractJobStore.assign_job_id()
AbstractJobStore.batch()
AbstractJobStore.create()
AbstractJobStore.create_job()
AbstractJobStore.exists()
AbstractJobStore.job_exists()
AbstractJobStore.publicUrlExpiration
AbstractJobStore.getPublicUrl()
AbstractJobStore.get_public_url()
AbstractJobStore.getSharedPublicUrl()
AbstractJobStore.get_shared_public_url()
AbstractJobStore.load()
AbstractJobStore.load_job()
AbstractJobStore.update()
AbstractJobStore.update_job()
AbstractJobStore.delete()
AbstractJobStore.delete_job()
AbstractJobStore.jobs()
AbstractJobStore.writeFile()
AbstractJobStore.write_file()
AbstractJobStore.writeFileStream()
AbstractJobStore.write_file_stream()
AbstractJobStore.getEmptyFileStoreID()
AbstractJobStore.get_empty_file_store_id()
AbstractJobStore.readFile()
AbstractJobStore.read_file()
AbstractJobStore.readFileStream()
AbstractJobStore.read_file_stream()
AbstractJobStore.deleteFile()
AbstractJobStore.delete_file()
AbstractJobStore.fileExists()
AbstractJobStore.file_exists()
AbstractJobStore.getFileSize()
AbstractJobStore.get_file_size()
AbstractJobStore.updateFile()
AbstractJobStore.update_file()
AbstractJobStore.updateFileStream()
AbstractJobStore.update_file_stream()
AbstractJobStore.sharedFileNameRegex
AbstractJobStore.writeSharedFileStream()
AbstractJobStore.write_shared_file_stream()
AbstractJobStore.readSharedFileStream()
AbstractJobStore.read_shared_file_stream()
AbstractJobStore.writeStatsAndLogging()
AbstractJobStore.write_logs()
AbstractJobStore.readStatsAndLogging()
AbstractJobStore.read_logs()
AbstractJobStore.write_leader_pid()
AbstractJobStore.read_leader_pid()
AbstractJobStore.write_leader_node_id()
AbstractJobStore.read_leader_node_id()
AbstractJobStore.write_kill_flag()
AbstractJobStore.read_kill_flag()
AbstractJobStore.default_caching()
JobStoreSupport
- toil.jobStores.aws
- Submodules
- toil.jobStores.aws.jobStore
- Attributes
- Exceptions
- Classes
- Module Contents
boto3_session
s3_boto3_resource
s3_boto3_client
logger
CONSISTENCY_TICKS
CONSISTENCY_TIME
ChecksumError
DomainDoesNotExist
AWSJobStore
AWSJobStore.bucketNameRe
AWSJobStore.minBucketNameLen
AWSJobStore.maxBucketNameLen
AWSJobStore.maxNameLen
AWSJobStore.nameSeparator
AWSJobStore.region
AWSJobStore.name_prefix
AWSJobStore.part_size
AWSJobStore.jobs_domain_name
AWSJobStore.files_domain_name
AWSJobStore.files_bucket
AWSJobStore.db
AWSJobStore.s3_resource
AWSJobStore.s3_client
AWSJobStore.initialize()
AWSJobStore.sseKeyPath
AWSJobStore.resume()
AWSJobStore.jobsPerBatchInsert
AWSJobStore.batch()
AWSJobStore.assign_job_id()
AWSJobStore.create_job()
AWSJobStore.job_exists()
AWSJobStore.jobs()
AWSJobStore.load_job()
AWSJobStore.update_job()
AWSJobStore.itemsPerBatchDelete
AWSJobStore.delete_job()
AWSJobStore.get_empty_file_store_id()
AWSJobStore.write_file()
AWSJobStore.write_file_stream()
AWSJobStore.write_shared_file_stream()
AWSJobStore.update_file()
AWSJobStore.update_file_stream()
AWSJobStore.file_exists()
AWSJobStore.get_file_size()
AWSJobStore.read_file()
AWSJobStore.read_file_stream()
AWSJobStore.read_shared_file_stream()
AWSJobStore.delete_file()
AWSJobStore.write_logs()
AWSJobStore.read_logs()
AWSJobStore.get_public_url()
AWSJobStore.get_shared_public_url()
AWSJobStore.sharedFileOwnerID
AWSJobStore.statsFileOwnerID
AWSJobStore.readStatsFileOwnerID
AWSJobStore.FileInfo
AWSJobStore.FileInfo.outer
AWSJobStore.FileInfo.encrypted
AWSJobStore.FileInfo.fileID
AWSJobStore.FileInfo.ownerID
AWSJobStore.FileInfo.version
AWSJobStore.FileInfo.previousVersion
AWSJobStore.FileInfo.content
AWSJobStore.FileInfo.checksum
AWSJobStore.FileInfo.create()
AWSJobStore.FileInfo.presenceIndicator()
AWSJobStore.FileInfo.exists()
AWSJobStore.FileInfo.load()
AWSJobStore.FileInfo.loadOrCreate()
AWSJobStore.FileInfo.loadOrFail()
AWSJobStore.FileInfo.fromItem()
AWSJobStore.FileInfo.toItem()
AWSJobStore.FileInfo.maxInlinedSize()
AWSJobStore.FileInfo.save()
AWSJobStore.FileInfo.upload()
AWSJobStore.FileInfo.uploadStream()
AWSJobStore.FileInfo.copyFrom()
AWSJobStore.FileInfo.copyTo()
AWSJobStore.FileInfo.download()
AWSJobStore.FileInfo.downloadStream()
AWSJobStore.FileInfo.delete()
AWSJobStore.FileInfo.getSize()
AWSJobStore.FileInfo.__repr__()
AWSJobStore.versionings
AWSJobStore.destroy()
aRepr
custom_repr
BucketLocationConflictException
- toil.jobStores.aws.utils
- Attributes
- Exceptions
- Classes
- Functions
- Module Contents
- toil.jobStores.aws.jobStore
- Submodules
- toil.jobStores.conftest
- toil.jobStores.fileJobStore
- Attributes
- Classes
- Module Contents
logger
FileJobStore
FileJobStore.validDirs
FileJobStore.validDirsSet
FileJobStore.JOB_DIR_PREFIX
FileJobStore.JOB_NAME_DIR_PREFIX
FileJobStore.BUFFER_SIZE
FileJobStore.LOG_TEMP_SUFFIX
FileJobStore.LOG_PREFIX
FileJobStore.default_caching()
FileJobStore.jobStoreDir
FileJobStore.jobsDir
FileJobStore.statsDir
FileJobStore.stats_inbox
FileJobStore.stats_archive
FileJobStore.filesDir
FileJobStore.jobFilesDir
FileJobStore.sharedFilesDir
FileJobStore.fanOut
FileJobStore.linkImports
FileJobStore.moveExports
FileJobStore.symlink_job_store_reads
FileJobStore.__repr__()
FileJobStore.initialize()
FileJobStore.resume()
FileJobStore.destroy()
FileJobStore.assign_job_id()
FileJobStore.create_job()
FileJobStore.batch()
FileJobStore.job_exists()
FileJobStore.get_public_url()
FileJobStore.get_shared_public_url()
FileJobStore.load_job()
FileJobStore.update_job()
FileJobStore.delete_job()
FileJobStore.jobs()
FileJobStore.write_file()
FileJobStore.write_file_stream()
FileJobStore.get_empty_file_store_id()
FileJobStore.update_file()
FileJobStore.read_file()
FileJobStore.delete_file()
FileJobStore.file_exists()
FileJobStore.get_file_size()
FileJobStore.update_file_stream()
FileJobStore.read_file_stream()
FileJobStore.write_shared_file_stream()
FileJobStore.read_shared_file_stream()
FileJobStore.list_all_file_names()
FileJobStore.write_logs()
FileJobStore.read_logs()
- toil.jobStores.googleJobStore
- Attributes
- Classes
- Functions
- Module Contents
log
GOOGLE_STORAGE
MAX_BATCH_SIZE
google_retry_predicate()
google_retry()
GoogleJobStore
GoogleJobStore.nodeServiceAccountJson
GoogleJobStore.projectID
GoogleJobStore.bucketName
GoogleJobStore.bucket
GoogleJobStore.statsBaseID
GoogleJobStore.statsReadPrefix
GoogleJobStore.readStatsBaseID
GoogleJobStore.sseKey
GoogleJobStore.storageClient
GoogleJobStore.create_client()
GoogleJobStore.initialize()
GoogleJobStore.resume()
GoogleJobStore.destroy()
GoogleJobStore.assign_job_id()
GoogleJobStore.batch()
GoogleJobStore.create_job()
GoogleJobStore.job_exists()
GoogleJobStore.get_public_url()
GoogleJobStore.get_shared_public_url()
GoogleJobStore.load_job()
GoogleJobStore.update_job()
GoogleJobStore.delete_job()
GoogleJobStore.get_env()
GoogleJobStore.jobs()
GoogleJobStore.write_file()
GoogleJobStore.write_file_stream()
GoogleJobStore.get_empty_file_store_id()
GoogleJobStore.read_file()
GoogleJobStore.read_file_stream()
GoogleJobStore.delete_file()
GoogleJobStore.file_exists()
GoogleJobStore.get_file_size()
GoogleJobStore.update_file()
GoogleJobStore.update_file_stream()
GoogleJobStore.write_shared_file_stream()
GoogleJobStore.read_shared_file_stream()
GoogleJobStore.write_logs()
GoogleJobStore.read_logs()
- toil.jobStores.utils
- toil.jobStores.abstractJobStore
- Submodules
- toil.leader
- Attributes
- Classes
- Module Contents
logger
Leader
Leader.config
Leader.jobStore
Leader.jobStoreLocator
Leader.toilState
Leader.batchSystem
Leader.issued_jobs_by_batch_system_id
Leader.preemptibleJobsIssued
Leader.serviceJobsIssued
Leader.serviceJobsToBeIssued
Leader.preemptibleServiceJobsIssued
Leader.preemptibleServiceJobsToBeIssued
Leader.timeSinceJobsLastRescued
Leader.reissueMissingJobs_missingHash
Leader.provisioner
Leader.clusterScaler
Leader.serviceManager
Leader.statsAndLogging
Leader.potentialDeadlockedJobs
Leader.potentialDeadlockTime
Leader.toilMetrics
Leader.debugJobNames
Leader.deadlockThrottler
Leader.statusThrottler
Leader.kill_throttler
Leader.progress_overall
Leader.progress_failed
Leader.GOOD_COLOR
Leader.BAD_COLOR
Leader.PROGRESS_BAR_FORMAT
Leader.recommended_fail_exit_code
Leader.run()
Leader.create_status_sentinel_file()
Leader.innerLoop()
Leader.checkForDeadlocks()
Leader.feed_deadlock_watchdog()
Leader.issueJob()
Leader.issueJobs()
Leader.issueServiceJob()
Leader.issueQueingServiceJobs()
Leader.getNumberOfJobsIssued()
Leader.removeJob()
Leader.getJobs()
Leader.killJobs()
Leader.reissueOverLongJobs()
Leader.reissueMissingJobs()
Leader.processRemovedJob()
Leader.process_finished_job()
Leader.process_finished_job_description()
Leader.getSuccessors()
Leader.processTotallyFailedJob()
- toil.lib
- Submodules
- toil.lib.accelerators
- toil.lib.aws
- Submodules
- toil.lib.aws.ami
- toil.lib.aws.iam
- Attributes
- Functions
- Module Contents
logger
CLUSTER_LAUNCHING_PERMISSIONS
AllowedActionCollection
delete_iam_instance_profile()
delete_iam_role()
create_iam_role()
init_action_collection()
add_to_action_collection()
policy_permissions_allow()
permission_matches_any()
get_actions_from_policy_document()
allowed_actions_attached()
allowed_actions_roles()
collect_policy_actions()
allowed_actions_user()
allowed_actions_group()
get_policy_permissions()
get_aws_account_num()
- toil.lib.aws.s3
- toil.lib.aws.session
- toil.lib.aws.utils
- Attributes
- Exceptions
- Functions
- Module Contents
ClientError
logger
THROTTLED_ERROR_CODES
delete_sdb_domain()
connection_reset()
connection_error()
retryable_s3_errors()
retry_s3()
delete_s3_bucket()
create_s3_bucket()
enable_public_objects()
NoBucketLocationError
get_bucket_region()
region_to_bucket_location()
bucket_location_to_region()
get_object_for_url()
list_objects_for_url()
flatten_tags()
boto3_pager()
get_item_from_attributes()
- Attributes
- Functions
- Package Contents
- Submodules
- toil.lib.bioio
- toil.lib.compatibility
- toil.lib.conversions
- toil.lib.docker
- toil.lib.ec2
- Attributes
- Exceptions
- Functions
- Module Contents
a_short_time
a_long_time
logger
UserError
not_found()
inconsistencies_detected()
INCONSISTENCY_ERRORS
retry_ec2()
UnexpectedResourceState
wait_transition()
wait_instances_running()
wait_spot_requests_active()
create_spot_instances()
create_ondemand_instances()
increase_instance_hop_limit()
prune()
iam_client
wait_until_instance_profile_arn_exists()
create_instances()
create_launch_template()
create_auto_scaling_group()
- toil.lib.ec2nodes
- toil.lib.encryption
- toil.lib.exceptions
- toil.lib.expando
- toil.lib.ftp_utils
- toil.lib.generatedEC2Lists
- toil.lib.humanize
- toil.lib.integration
- toil.lib.io
- Attributes
- Classes
- Functions
- Module Contents
logger
TOIL_URI_SCHEME
STANDARD_SCHEMES
REMOTE_SCHEMES
ALL_SCHEMES
is_standard_url()
is_remote_url()
is_any_url()
is_url_with_scheme()
is_toil_url()
is_file_url()
mkdtemp()
robust_rmtree()
atomic_tmp_file()
atomic_install()
AtomicFileCreate()
atomic_copy()
atomic_copyobj()
make_public_dir()
try_path()
WriteWatchingStream
ReadableFileObj
file_digest()
- toil.lib.iterables
- toil.lib.memoize
- toil.lib.misc
- toil.lib.objects
- toil.lib.resources
- toil.lib.retry
- Attributes
- Classes
- Functions
- Module Contents
SUPPORTED_HTTP_ERRORS
kubernetes
botocore
logger
ErrorCondition
RT
retry()
return_status_code()
get_error_code()
get_error_message()
get_error_status()
get_error_body()
meets_error_message_condition()
meets_error_code_condition()
meets_boto_error_code_condition()
error_meets_conditions()
DEFAULT_DELAYS
DEFAULT_TIMEOUT
E
old_retry()
retry_flaky_test
- toil.lib.threading
- Attributes
- Classes
- Functions
- Module Contents
- toil.lib.throttle
- Submodules
- toil.options
- toil.provisioners
- Submodules
- toil.provisioners.abstractProvisioner
- Attributes
- Exceptions
- Classes
- Module Contents
a_short_time
logger
ManagedNodesNotSupportedException
Shape
AbstractProvisioner
AbstractProvisioner.LEADER_HOME_DIR
AbstractProvisioner.cloud
AbstractProvisioner.clusterName
AbstractProvisioner.clusterType
AbstractProvisioner.enable_fuse
AbstractProvisioner.supportedClusterTypes()
AbstractProvisioner.createClusterSettings()
AbstractProvisioner.readClusterSettings()
AbstractProvisioner.setAutoscaledNodeTypes()
AbstractProvisioner.hasAutoscaledNodeTypes()
AbstractProvisioner.getAutoscaledInstanceShapes()
AbstractProvisioner.retryPredicate()
AbstractProvisioner.launchCluster()
AbstractProvisioner.addNodes()
AbstractProvisioner.addManagedNodes()
AbstractProvisioner.terminateNodes()
AbstractProvisioner.getLeader()
AbstractProvisioner.getProvisionedWorkers()
AbstractProvisioner.getNodeShape()
AbstractProvisioner.destroyCluster()
AbstractProvisioner.InstanceConfiguration
AbstractProvisioner.InstanceConfiguration.files
AbstractProvisioner.InstanceConfiguration.units
AbstractProvisioner.InstanceConfiguration.sshPublicKeys
AbstractProvisioner.InstanceConfiguration.addFile()
AbstractProvisioner.InstanceConfiguration.addUnit()
AbstractProvisioner.InstanceConfiguration.addSSHRSAKey()
AbstractProvisioner.InstanceConfiguration.toIgnitionConfig()
AbstractProvisioner.getBaseInstanceConfiguration()
AbstractProvisioner.addVolumesService()
AbstractProvisioner.addNodeExporterService()
AbstractProvisioner.toil_service_env_options()
AbstractProvisioner.add_toil_service()
AbstractProvisioner.getKubernetesValues()
AbstractProvisioner.addKubernetesServices()
AbstractProvisioner.getKubernetesAutoscalerSetupCommands()
AbstractProvisioner.getKubernetesCloudProvider()
AbstractProvisioner.addKubernetesLeader()
AbstractProvisioner.addKubernetesWorker()
- toil.provisioners.aws
- Submodules
- toil.provisioners.aws.awsProvisioner
- Attributes
- Exceptions
- Classes
- Functions
- Module Contents
logger
awsRetryPredicate()
expectedShutdownErrors()
F
awsRetry()
awsFilterImpairedNodes()
InvalidClusterStateException
collapse_tags()
AWSProvisioner
AWSProvisioner.cloud
AWSProvisioner.aws
AWSProvisioner.s3_bucket_name
AWSProvisioner.supportedClusterTypes()
AWSProvisioner.createClusterSettings()
AWSProvisioner.readClusterSettings()
AWSProvisioner.launchCluster()
AWSProvisioner.toil_service_env_options()
AWSProvisioner.getKubernetesAutoscalerSetupCommands()
AWSProvisioner.getKubernetesCloudProvider()
AWSProvisioner.getNodeShape()
AWSProvisioner.retryPredicate()
AWSProvisioner.destroyCluster()
AWSProvisioner.terminateNodes()
AWSProvisioner.addNodes()
AWSProvisioner.addManagedNodes()
AWSProvisioner.getProvisionedWorkers()
AWSProvisioner.getLeader()
AWSProvisioner.full_policy()
AWSProvisioner.kubernetes_policy()
- toil.provisioners.aws.awsProvisioner
- Attributes
- Classes
- Functions
- Package Contents
- Submodules
- toil.provisioners.clusterScaler
- Attributes
- Exceptions
- Classes
- Functions
- Module Contents
logger
EVICTION_THRESHOLD
RESERVE_SMALL_LIMIT
RESERVE_SMALL_AMOUNT
RESERVE_BREAKPOINTS
RESERVE_FRACTIONS
OS_SIZE
FailedConstraint
BinPackedFit
NodeReservation
adjustEndingReservationForJob()
split()
binPacking()
ClusterScaler
ClusterScaler.provisioner
ClusterScaler.leader
ClusterScaler.config
ClusterScaler.static
ClusterScaler.on_too_big
ClusterScaler.jobNameToAvgRuntime
ClusterScaler.jobNameToNumCompleted
ClusterScaler.totalAvgRuntime
ClusterScaler.totalJobsCompleted
ClusterScaler.targetTime
ClusterScaler.betaInertia
ClusterScaler.nodeShapeToType
ClusterScaler.instance_types
ClusterScaler.nodeShapes
ClusterScaler.ignoredNodes
ClusterScaler.preemptibleNodeDeficit
ClusterScaler.previousWeightedEstimate
ClusterScaler.minNodes
ClusterScaler.maxNodes
ClusterScaler.node_shapes_after_overhead
ClusterScaler.without_overhead
ClusterScaler.getAverageRuntime()
ClusterScaler.addCompletedJob()
ClusterScaler.setStaticNodes()
ClusterScaler.getStaticNodes()
ClusterScaler.smoothEstimate()
ClusterScaler.getEstimatedNodeCounts()
ClusterScaler.updateClusterSize()
ClusterScaler.setNodeCount()
ClusterScaler.filter_out_static_nodes()
ClusterScaler.getNodes()
ClusterScaler.shutDown()
JobTooBigError
ScalerThread
ClusterStats
- toil.provisioners.gceProvisioner
- Attributes
- Classes
- Module Contents
logger
GCEProvisioner
GCEProvisioner.NODE_BOTO_PATH
GCEProvisioner.SOURCE_IMAGE
GCEProvisioner.cloud
GCEProvisioner.supportedClusterTypes()
GCEProvisioner.createClusterSettings()
GCEProvisioner.readClusterSettings()
GCEProvisioner.launchCluster()
GCEProvisioner.getNodeShape()
GCEProvisioner.retryPredicate()
GCEProvisioner.destroyCluster()
GCEProvisioner.terminateNodes()
GCEProvisioner.addNodes()
GCEProvisioner.getProvisionedWorkers()
GCEProvisioner.getLeader()
GCEProvisioner.DEFAULT_TASK_COMPLETION_TIMEOUT
GCEProvisioner.ex_create_multiple_nodes()
- toil.provisioners.node
- Attributes
- Classes
- Module Contents
a_short_time
logger
Node
Node.maxWaitTime
Node.publicIP
Node.privateIP
Node.name
Node.nodeType
Node.preemptible
Node.tags
Node.__str__()
Node.__repr__()
Node.__hash__()
Node.remainingBillingInterval()
Node.waitForNode()
Node.copySshKeys()
Node.injectFile()
Node.extractFile()
Node.sshAppliance()
Node.sshInstance()
Node.coreSSH()
Node.coreRsync()
- toil.provisioners.abstractProvisioner
- Attributes
- Exceptions
- Functions
- Package Contents
- Submodules
- toil.realtimeLogger
- toil.resource
- Attributes
- Exceptions
- Classes
- Module Contents
logger
Resource
FileResource
DirectoryResource
VirtualEnvResource
ModuleDescriptor
ModuleDescriptor.dirPath
ModuleDescriptor.name
ModuleDescriptor.forModule()
ModuleDescriptor.belongsToToil
ModuleDescriptor.saveAsResourceTo()
ModuleDescriptor.localize()
ModuleDescriptor.globalize()
ModuleDescriptor.toCommand()
ModuleDescriptor.fromCommand()
ModuleDescriptor.makeLoadable()
ModuleDescriptor.load()
ResourceException
- toil.server
- Submodules
- toil.server.api_spec
- toil.server.app
- toil.server.celery_app
- toil.server.cli
- toil.server.utils
- Attributes
- Classes
- Functions
- Module Contents
HAVE_S3
logger
get_iso_time()
link_file()
download_file_from_internet()
download_file_from_s3()
get_file_class()
safe_read_file()
safe_write_file()
MemoryStateCache
AbstractStateStore
MemoryStateStore
FileStateStore
S3StateStore
state_store_cache
connect_to_state_store()
WorkflowStateStore
connect_to_workflow_state_store()
TERMINAL_STATES
MAX_CANCELING_SECONDS
WorkflowStateMachine
WorkflowStateMachine.send_enqueue()
WorkflowStateMachine.send_initialize()
WorkflowStateMachine.send_run()
WorkflowStateMachine.send_cancel()
WorkflowStateMachine.send_canceled()
WorkflowStateMachine.send_complete()
WorkflowStateMachine.send_executor_error()
WorkflowStateMachine.send_system_error()
WorkflowStateMachine.get_current_state()
- toil.server.wes
- Submodules
- toil.server.wes.abstract_backend
- toil.server.wes.amazon_wes_utils
- toil.server.wes.tasks
- Attributes
- Classes
- Functions
- Module Contents
logger
WAIT_FOR_DEATH_TIMEOUT
ToilWorkflowRunner
ToilWorkflowRunner.scratch_dir
ToilWorkflowRunner.store
ToilWorkflowRunner.state_machine
ToilWorkflowRunner.request
ToilWorkflowRunner.engine_options
ToilWorkflowRunner.wf_type
ToilWorkflowRunner.version
ToilWorkflowRunner.exec_dir
ToilWorkflowRunner.out_dir
ToilWorkflowRunner.default_job_store
ToilWorkflowRunner.job_store
ToilWorkflowRunner.write_scratch_file()
ToilWorkflowRunner.get_state()
ToilWorkflowRunner.write_workflow()
ToilWorkflowRunner.sort_options()
ToilWorkflowRunner.initialize_run()
ToilWorkflowRunner.call_cmd()
ToilWorkflowRunner.run()
ToilWorkflowRunner.write_output_files()
run_wes_task()
run_wes
cancel_run()
TaskRunner
MultiprocessingTaskRunner
- toil.server.wes.toil_backend
- Attributes
- Classes
- Module Contents
logger
ToilWorkflow
ToilWorkflow.base_scratch_dir
ToilWorkflow.state_store_url
ToilWorkflow.run_id
ToilWorkflow.scratch_dir
ToilWorkflow.exec_dir
ToilWorkflow.store
ToilWorkflow.state_machine
ToilWorkflow.fetch_state()
ToilWorkflow.fetch_scratch()
ToilWorkflow.exists()
ToilWorkflow.get_state()
ToilWorkflow.check_on_run()
ToilWorkflow.set_up_run()
ToilWorkflow.clean_up()
ToilWorkflow.queue_run()
ToilWorkflow.get_output_files()
ToilWorkflow.get_stdout_path()
ToilWorkflow.get_stderr_path()
ToilWorkflow.get_messages_path()
ToilWorkflow.get_task_logs()
ToilBackend
ToilBackend.run_id_prefix
ToilBackend.task_runner
ToilBackend.wes_dialect
ToilBackend.dest_bucket_base
ToilBackend.work_dir
ToilBackend.supported_versions
ToilBackend.get_runs()
ToilBackend.get_state()
ToilBackend.get_service_info()
ToilBackend.list_runs()
ToilBackend.run_workflow()
ToilBackend.get_run_log()
ToilBackend.cancel_run()
ToilBackend.get_run_status()
ToilBackend.get_stdout()
ToilBackend.get_stderr()
ToilBackend.get_health()
ToilBackend.get_homepage()
- Submodules
- toil.server.wsgi_app
- Submodules
- toil.serviceManager
- Attributes
- Classes
- Module Contents
logger
ServiceManager
ServiceManager.services_are_starting()
ServiceManager.get_job_count()
ServiceManager.start()
ServiceManager.put_client()
ServiceManager.get_ready_client()
ServiceManager.get_unservable_client()
ServiceManager.get_startable_service()
ServiceManager.kill_services()
ServiceManager.is_active()
ServiceManager.is_running()
ServiceManager.check()
ServiceManager.shutdown()
- toil.statsAndLogging
- toil.test
- Submodules
- toil.test.batchSystems
- Submodules
- toil.test.batchSystems.batchSystemTest
- Attributes
- Classes
- Functions
- Module Contents
logger
numCores
preemptible
defaultRequirements
BatchSystemPluginTest
hidden
hidden.AbstractBatchSystemTest
hidden.AbstractBatchSystemTest.createBatchSystem()
hidden.AbstractBatchSystemTest.supportsWallTime()
hidden.AbstractBatchSystemTest.createConfig()
hidden.AbstractBatchSystemTest.setUpClass()
hidden.AbstractBatchSystemTest.setUp()
hidden.AbstractBatchSystemTest.tearDown()
hidden.AbstractBatchSystemTest.get_max_startup_seconds()
hidden.AbstractBatchSystemTest.test_available_cores()
hidden.AbstractBatchSystemTest.test_run_jobs()
hidden.AbstractBatchSystemTest.test_set_env()
hidden.AbstractBatchSystemTest.test_set_job_env()
hidden.AbstractBatchSystemTest.testCheckResourceRequest()
hidden.AbstractBatchSystemTest.testScalableBatchSystem()
hidden.AbstractBatchSystemJobTest
hidden.AbstractBatchSystemJobTest.cpuCount
hidden.AbstractBatchSystemJobTest.allocatedCores
hidden.AbstractBatchSystemJobTest.sleepTime
hidden.AbstractBatchSystemJobTest.getBatchSystemName()
hidden.AbstractBatchSystemJobTest.getOptions()
hidden.AbstractBatchSystemJobTest.setUp()
hidden.AbstractBatchSystemJobTest.tearDown()
hidden.AbstractBatchSystemJobTest.testJobConcurrency()
hidden.AbstractBatchSystemJobTest.test_omp_threads()
hidden.AbstractGridEngineBatchSystemTest
KubernetesBatchSystemTest
KubernetesBatchSystemBenchTest
AWSBatchBatchSystemTest
MesosBatchSystemTest
write_temp_file()
SingleMachineBatchSystemTest
MaxCoresSingleMachineBatchSystemTest
parentJob()
childJob()
grandChildJob()
greatGrandChild()
Service
GridEngineBatchSystemTest
SlurmBatchSystemTest
LSFBatchSystemTest
TorqueBatchSystemTest
HTCondorBatchSystemTest
SingleMachineBatchSystemJobTest
MesosBatchSystemJobTest
measureConcurrency()
count()
getCounters()
resetCounters()
get_omp_threads()
- toil.test.batchSystems.batch_system_plugin_test
- Attributes
- Classes
- Module Contents
logger
FakeBatchSystem
FakeBatchSystem.supportsAutoDeployment()
FakeBatchSystem.issueBatchJob()
FakeBatchSystem.killBatchJobs()
FakeBatchSystem.getIssuedBatchJobIDs()
FakeBatchSystem.getRunningBatchJobIDs()
FakeBatchSystem.getUpdatedBatchJob()
FakeBatchSystem.shutdown()
FakeBatchSystem.add_options()
FakeBatchSystem.setOptions()
BatchSystemPluginTest
- toil.test.batchSystems.test_gridengine
- toil.test.batchSystems.test_lsf_helper
- toil.test.batchSystems.test_slurm
- Classes
- Functions
- Module Contents
call_sacct()
call_scontrol()
call_sacct_raises()
FakeBatchSystem
SlurmTest
SlurmTest.setUp()
SlurmTest.test_getJobDetailsFromSacct_one_exists()
SlurmTest.test_getJobDetailsFromSacct_one_not_exists()
SlurmTest.test_getJobDetailsFromSacct_many_all_exist()
SlurmTest.test_getJobDetailsFromSacct_many_some_exist()
SlurmTest.test_getJobDetailsFromSacct_many_none_exist()
SlurmTest.test_getJobDetailsFromScontrol_one_exists()
SlurmTest.test_getJobDetailsFromScontrol_one_not_exists()
SlurmTest.test_getJobDetailsFromScontrol_many_all_exist()
SlurmTest.test_getJobDetailsFromScontrol_many_some_exist()
SlurmTest.test_getJobDetailsFromScontrol_many_none_exist()
SlurmTest.test_getJobExitCode_job_exists()
SlurmTest.test_getJobExitCode_job_not_exists()
SlurmTest.test_getJobExitCode_sacct_raises_job_exists()
SlurmTest.test_getJobExitCode_sacct_raises_job_not_exists()
SlurmTest.test_coalesce_job_exit_codes_one_exists()
SlurmTest.test_coalesce_job_exit_codes_one_not_exists()
SlurmTest.test_coalesce_job_exit_codes_many_all_exist()
SlurmTest.test_coalesce_job_exit_codes_some_exists()
SlurmTest.test_coalesce_job_exit_codes_sacct_raises_job_exists()
SlurmTest.test_coalesce_job_exit_codes_sacct_raises_job_not_exists()
- toil.test.batchSystems.batchSystemTest
- Submodules
- toil.test.cactus
- toil.test.cwl
- Submodules
- toil.test.cwl.conftest
- toil.test.cwl.cwlTest
- Attributes
- Classes
- Functions
- Module Contents
pkg_root
log
CONFORMANCE_TEST_TIMEOUT
run_conformance_tests()
TesterFuncType
CWLWorkflowTest
CWLWorkflowTest.setUp()
CWLWorkflowTest.tearDown()
CWLWorkflowTest.test_cwl_cmdline_input()
CWLWorkflowTest.revsort()
CWLWorkflowTest.revsort_no_checksum()
CWLWorkflowTest.download()
CWLWorkflowTest.load_contents()
CWLWorkflowTest.download_directory()
CWLWorkflowTest.download_subdirectory()
CWLWorkflowTest.test_mpi()
CWLWorkflowTest.test_s3_as_secondary_file()
CWLWorkflowTest.test_run_revsort()
CWLWorkflowTest.test_run_revsort_nochecksum()
CWLWorkflowTest.test_run_revsort_no_container()
CWLWorkflowTest.test_run_revsort2()
CWLWorkflowTest.test_run_revsort_debug_worker()
CWLWorkflowTest.test_run_colon_output()
CWLWorkflowTest.test_run_dockstore_trs()
CWLWorkflowTest.test_glob_dir_bypass_file_store()
CWLWorkflowTest.test_required_input_condition_protection()
CWLWorkflowTest.test_slurm_node_memory()
CWLWorkflowTest.test_download_s3()
CWLWorkflowTest.test_download_http()
CWLWorkflowTest.test_download_https()
CWLWorkflowTest.test_download_https_reference()
CWLWorkflowTest.test_download_file()
CWLWorkflowTest.test_download_directory_s3()
CWLWorkflowTest.test_download_directory_s3_reference()
CWLWorkflowTest.test_download_directory_file()
CWLWorkflowTest.test_download_subdirectory_s3()
CWLWorkflowTest.test_download_subdirectory_file()
CWLWorkflowTest.test_load_contents_s3()
CWLWorkflowTest.test_load_contents_http()
CWLWorkflowTest.test_load_contents_https()
CWLWorkflowTest.test_load_contents_file()
CWLWorkflowTest.test_bioconda()
CWLWorkflowTest.test_default_args()
CWLWorkflowTest.test_biocontainers()
CWLWorkflowTest.test_cuda()
CWLWorkflowTest.test_restart()
CWLWorkflowTest.test_streamable()
CWLWorkflowTest.test_streamable_reference()
CWLWorkflowTest.test_preemptible()
CWLWorkflowTest.test_preemptible_expression()
CWLv10Test
CWLv10Test.setUp()
CWLv10Test.tearDown()
CWLv10Test.test_run_conformance_with_caching()
CWLv10Test.test_run_conformance()
CWLv10Test.test_lsf_cwl_conformance()
CWLv10Test.test_slurm_cwl_conformance()
CWLv10Test.test_torque_cwl_conformance()
CWLv10Test.test_gridengine_cwl_conformance()
CWLv10Test.test_mesos_cwl_conformance()
CWLv10Test.test_kubernetes_cwl_conformance()
CWLv10Test.test_lsf_cwl_conformance_with_caching()
CWLv10Test.test_slurm_cwl_conformance_with_caching()
CWLv10Test.test_torque_cwl_conformance_with_caching()
CWLv10Test.test_gridengine_cwl_conformance_with_caching()
CWLv10Test.test_mesos_cwl_conformance_with_caching()
CWLv10Test.test_kubernetes_cwl_conformance_with_caching()
CWLv11Test
CWLv12Test
CWLv12Test.rootDir
CWLv12Test.cwlSpec
CWLv12Test.test_yaml
CWLv12Test.setUpClass()
CWLv12Test.tearDown()
CWLv12Test.test_run_conformance()
CWLv12Test.test_run_conformance_with_caching()
CWLv12Test.test_run_conformance_with_in_place_update()
CWLv12Test.test_kubernetes_cwl_conformance()
CWLv12Test.test_kubernetes_cwl_conformance_with_caching()
CWLv12Test.test_wes_server_cwl_conformance()
test_workflow_echo_string_scatter_stderr_log_dir()
test_log_dir_echo_no_output()
test_log_dir_echo_stderr()
test_filename_conflict_resolution()
test_filename_conflict_resolution_3_or_more()
test_filename_conflict_detection()
test_filename_conflict_detection_at_root()
test_pick_value_with_one_null_value()
test_workflow_echo_string()
test_workflow_echo_string_scatter_capture_stdout()
test_visit_top_cwl_class()
test_visit_cwl_class_and_reduce()
test_download_structure()
test_import_on_workers()
ImportWorkersMessageHandler
- Submodules
- toil.test.docs
- Submodules
- toil.test.docs.scriptsTest
- Attributes
- Classes
- Module Contents
pkg_root
ToilDocumentationTest
ToilDocumentationTest.setUpClass()
ToilDocumentationTest.tearDown()
ToilDocumentationTest.checkExitCode()
ToilDocumentationTest.checkExpectedOut()
ToilDocumentationTest.checkExpectedPattern()
ToilDocumentationTest.testStats()
ToilDocumentationTest.testDynamic()
ToilDocumentationTest.testEncapsulation()
ToilDocumentationTest.testEncapsulation2()
ToilDocumentationTest.testHelloworld()
ToilDocumentationTest.testInvokeworkflow()
ToilDocumentationTest.testInvokeworkflow2()
ToilDocumentationTest.testJobFunctions()
ToilDocumentationTest.testManaging()
ToilDocumentationTest.testManaging2()
ToilDocumentationTest.testMultiplejobs()
ToilDocumentationTest.testMultiplejobs2()
ToilDocumentationTest.testMultiplejobs3()
ToilDocumentationTest.testPromises2()
ToilDocumentationTest.testQuickstart()
ToilDocumentationTest.testRequirements()
ToilDocumentationTest.testArguments()
ToilDocumentationTest.testDocker()
ToilDocumentationTest.testPromises()
ToilDocumentationTest.testServices()
ToilDocumentationTest.testStaging()
- toil.test.docs.scriptsTest
- Submodules
- toil.test.jobStores
- Submodules
- toil.test.jobStores.jobStoreTest
- Attributes
- Classes
- Functions
- Module Contents
google_retry()
logger
tearDownModule()
AbstractJobStoreTest
AbstractJobStoreTest.Test
AbstractJobStoreTest.Test.setUpClass()
AbstractJobStoreTest.Test.setUp()
AbstractJobStoreTest.Test.tearDown()
AbstractJobStoreTest.Test.testInitialState()
AbstractJobStoreTest.Test.testJobCreation()
AbstractJobStoreTest.Test.testConfigEquality()
AbstractJobStoreTest.Test.testJobLoadEquality()
AbstractJobStoreTest.Test.testChildLoadingEquality()
AbstractJobStoreTest.Test.testPersistantFilesToDelete()
AbstractJobStoreTest.Test.testUpdateBehavior()
AbstractJobStoreTest.Test.testJobDeletions()
AbstractJobStoreTest.Test.testSharedFiles()
AbstractJobStoreTest.Test.testReadWriteSharedFilesTextMode()
AbstractJobStoreTest.Test.testReadWriteFileStreamTextMode()
AbstractJobStoreTest.Test.testPerJobFiles()
AbstractJobStoreTest.Test.testStatsAndLogging()
AbstractJobStoreTest.Test.testWriteLogFiles()
AbstractJobStoreTest.Test.testBatchCreate()
AbstractJobStoreTest.Test.testGrowingAndShrinkingJob()
AbstractJobStoreTest.Test.externalStoreCache
AbstractJobStoreTest.Test.cleanUpExternalStores()
AbstractJobStoreTest.Test.mpTestPartSize
AbstractJobStoreTest.Test.makeImportExportTests()
AbstractJobStoreTest.Test.testImportHttpFile()
AbstractJobStoreTest.Test.testImportFtpFile()
AbstractJobStoreTest.Test.testFileDeletion()
AbstractJobStoreTest.Test.testMultipartUploads()
AbstractJobStoreTest.Test.testZeroLengthFiles()
AbstractJobStoreTest.Test.testLargeFile()
AbstractJobStoreTest.Test.fetch_url()
AbstractJobStoreTest.Test.assertUrl()
AbstractJobStoreTest.Test.testCleanCache()
AbstractJobStoreTest.Test.testPartialReadFromStream()
AbstractJobStoreTest.Test.testDestructionOfCorruptedJobStore()
AbstractJobStoreTest.Test.testDestructionIdempotence()
AbstractJobStoreTest.Test.testEmptyFileStoreIDIsReadable()
AbstractEncryptedJobStoreTest
FileJobStoreTest
GoogleJobStoreTest
AWSJobStoreTest
InvalidAWSJobStoreTest
EncryptedAWSJobStoreTest
StubHttpRequestHandler
- toil.test.jobStores.jobStoreTest
- Submodules
- toil.test.lib
- Submodules
- toil.test.lib.aws
- toil.test.lib.dockerTest
- Attributes
- Classes
- Module Contents
logger
DockerTest
DockerTest.setUp()
DockerTest.testDockerClean()
DockerTest.testDockerClean_CRx_FORGO()
DockerTest.testDockerClean_CRx_STOP()
DockerTest.testDockerClean_CRx_RM()
DockerTest.testDockerClean_CRx_None()
DockerTest.testDockerClean_CxD_FORGO()
DockerTest.testDockerClean_CxD_STOP()
DockerTest.testDockerClean_CxD_RM()
DockerTest.testDockerClean_CxD_None()
DockerTest.testDockerClean_Cxx_FORGO()
DockerTest.testDockerClean_Cxx_STOP()
DockerTest.testDockerClean_Cxx_RM()
DockerTest.testDockerClean_Cxx_None()
DockerTest.testDockerClean_xRx_FORGO()
DockerTest.testDockerClean_xRx_STOP()
DockerTest.testDockerClean_xRx_RM()
DockerTest.testDockerClean_xRx_None()
DockerTest.testDockerClean_xxD_FORGO()
DockerTest.testDockerClean_xxD_STOP()
DockerTest.testDockerClean_xxD_RM()
DockerTest.testDockerClean_xxD_None()
DockerTest.testDockerClean_xxx_FORGO()
DockerTest.testDockerClean_xxx_STOP()
DockerTest.testDockerClean_xxx_RM()
DockerTest.testDockerClean_xxx_None()
DockerTest.testDockerPipeChain()
DockerTest.testDockerPipeChainErrorDetection()
DockerTest.testNonCachingDockerChain()
DockerTest.testNonCachingDockerChainErrorDetection()
DockerTest.testDockerLogs()
DockerTest.testDockerLogs_Stream()
DockerTest.testDockerLogs_Demux()
DockerTest.testDockerLogs_Demux_Stream()
- toil.test.lib.test_conversions
- toil.test.lib.test_ec2
- toil.test.lib.test_integration
- toil.test.lib.test_misc
- Submodules
- toil.test.mesos
- toil.test.options
- toil.test.provisioners
- Submodules
- toil.test.provisioners.aws
- Submodules
- toil.test.provisioners.aws.awsProvisionerTest
- Attributes
- Classes
- Module Contents
log
AWSProvisionerBenchTest
AbstractAWSAutoscaleTest
AbstractAWSAutoscaleTest.instanceTypes
AbstractAWSAutoscaleTest.clusterName
AbstractAWSAutoscaleTest.numWorkers
AbstractAWSAutoscaleTest.numSamples
AbstractAWSAutoscaleTest.spotBid
AbstractAWSAutoscaleTest.scriptDir
AbstractAWSAutoscaleTest.venvDir
AbstractAWSAutoscaleTest.dataDir
AbstractAWSAutoscaleTest.scriptName
AbstractAWSAutoscaleTest.script()
AbstractAWSAutoscaleTest.data()
AbstractAWSAutoscaleTest.rsyncUtil()
AbstractAWSAutoscaleTest.getRootVolID()
AbstractAWSAutoscaleTest.putScript()
AWSAutoscaleTest
AWSAutoscaleTest.clusterName
AWSAutoscaleTest.requestedLeaderStorage
AWSAutoscaleTest.scriptName
AWSAutoscaleTest.setUp()
AWSAutoscaleTest.launchCluster()
AWSAutoscaleTest.getRootVolID()
AWSAutoscaleTest.testAutoScale()
AWSAutoscaleTest.testSpotAutoScale()
AWSAutoscaleTest.testSpotAutoScaleBalancingTypes()
AWSStaticAutoscaleTest
AWSManagedAutoscaleTest
AWSAutoscaleTestMultipleNodeTypes
AWSRestartTest
PreemptibleDeficitCompensationTest
- toil.test.provisioners.aws.awsProvisionerTest
- Submodules
- toil.test.provisioners.clusterScalerTest
- Attributes
- Classes
- Module Contents
logger
c4_8xlarge_preemptible
c4_8xlarge
r3_8xlarge
r5_2xlarge
r5_4xlarge
t2_micro
BinPackingTest
BinPackingTest.setUp()
BinPackingTest.testPackingOneShape()
BinPackingTest.testSorting()
BinPackingTest.testAddingInitialNode()
BinPackingTest.testLowTargetTime()
BinPackingTest.testHighTargetTime()
BinPackingTest.testZeroResourceJobs()
BinPackingTest.testLongRunningJobs()
BinPackingTest.run1000JobsOnMicros()
BinPackingTest.testPathologicalCase()
BinPackingTest.testJobTooLargeForAllNodes()
ClusterScalerTest
ClusterScalerTest.setUp()
ClusterScalerTest.testRounding()
ClusterScalerTest.testMaxNodes()
ClusterScalerTest.testMinNodes()
ClusterScalerTest.testPreemptibleDeficitResponse()
ClusterScalerTest.testPreemptibleDeficitIsSet()
ClusterScalerTest.testNoLaunchingIfDeltaAlreadyMet()
ClusterScalerTest.testBetaInertia()
ClusterScalerTest.test_overhead_accounting_large()
ClusterScalerTest.test_overhead_accounting_small()
ClusterScalerTest.test_overhead_accounting_observed()
ScalerThreadTest
MockBatchSystemAndProvisioner
MockBatchSystemAndProvisioner.config
MockBatchSystemAndProvisioner.secondsPerJob
MockBatchSystemAndProvisioner.provisioner
MockBatchSystemAndProvisioner.batchSystem
MockBatchSystemAndProvisioner.jobQueue
MockBatchSystemAndProvisioner.updatedJobsQueue
MockBatchSystemAndProvisioner.jobBatchSystemIDToIssuedJob
MockBatchSystemAndProvisioner.totalJobs
MockBatchSystemAndProvisioner.totalWorkerTime
MockBatchSystemAndProvisioner.toilMetrics
MockBatchSystemAndProvisioner.nodesToWorker
MockBatchSystemAndProvisioner.workers
MockBatchSystemAndProvisioner.maxWorkers
MockBatchSystemAndProvisioner.running
MockBatchSystemAndProvisioner.leaderThread
MockBatchSystemAndProvisioner.toilState
MockBatchSystemAndProvisioner.start()
MockBatchSystemAndProvisioner.shutDown()
MockBatchSystemAndProvisioner.nodeInUse()
MockBatchSystemAndProvisioner.ignoreNode()
MockBatchSystemAndProvisioner.unignoreNode()
MockBatchSystemAndProvisioner.supportedClusterTypes()
MockBatchSystemAndProvisioner.createClusterSettings()
MockBatchSystemAndProvisioner.readClusterSettings()
MockBatchSystemAndProvisioner.setAutoscaledNodeTypes()
MockBatchSystemAndProvisioner.getProvisionedWorkers()
MockBatchSystemAndProvisioner.terminateNodes()
MockBatchSystemAndProvisioner.remainingBillingInterval()
MockBatchSystemAndProvisioner.addJob()
MockBatchSystemAndProvisioner.getNumberOfJobsIssued()
MockBatchSystemAndProvisioner.getJobs()
MockBatchSystemAndProvisioner.getNodes()
MockBatchSystemAndProvisioner.addNodes()
MockBatchSystemAndProvisioner.getNodeShape()
MockBatchSystemAndProvisioner.getWorkersInCluster()
MockBatchSystemAndProvisioner.launchCluster()
MockBatchSystemAndProvisioner.destroyCluster()
MockBatchSystemAndProvisioner.getLeader()
MockBatchSystemAndProvisioner.getNumberOfNodes()
- toil.test.provisioners.clusterTest
- Attributes
- Classes
- Module Contents
log
AbstractClusterTest
AbstractClusterTest.keyName
AbstractClusterTest.clusterName
AbstractClusterTest.leaderNodeType
AbstractClusterTest.clusterType
AbstractClusterTest.zone
AbstractClusterTest.region
AbstractClusterTest.aws
AbstractClusterTest.venvDir
AbstractClusterTest.python()
AbstractClusterTest.pip()
AbstractClusterTest.destroyCluster()
AbstractClusterTest.setUp()
AbstractClusterTest.tearDown()
AbstractClusterTest.sshUtil()
AbstractClusterTest.rsync_util()
AbstractClusterTest.createClusterUtil()
AbstractClusterTest.launchCluster()
CWLOnARMTest
- toil.test.provisioners.gceProvisionerTest
- Attributes
- Classes
- Module Contents
log
AbstractGCEAutoscaleTest
AbstractGCEAutoscaleTest.projectID
AbstractGCEAutoscaleTest.sshUtil()
AbstractGCEAutoscaleTest.rsyncUtil()
AbstractGCEAutoscaleTest.destroyClusterUtil()
AbstractGCEAutoscaleTest.createClusterUtil()
AbstractGCEAutoscaleTest.cleanJobStoreUtil()
AbstractGCEAutoscaleTest.keyName
AbstractGCEAutoscaleTest.botoDir
AbstractGCEAutoscaleTest.googleZone
AbstractGCEAutoscaleTest.leaderInstanceType
AbstractGCEAutoscaleTest.instanceTypes
AbstractGCEAutoscaleTest.numWorkers
AbstractGCEAutoscaleTest.numSamples
AbstractGCEAutoscaleTest.spotBid
AbstractGCEAutoscaleTest.setUp()
AbstractGCEAutoscaleTest.tearDown()
AbstractGCEAutoscaleTest.launchCluster()
GCEAutoscaleTest
GCEStaticAutoscaleTest
GCEAutoscaleTestMultipleNodeTypes
GCERestartTest
- toil.test.provisioners.provisionerTest
- toil.test.provisioners.restartScript
- toil.test.provisioners.aws
- Submodules
- toil.test.server
- Submodules
- toil.test.server.serverTest
- Attributes
- Classes
- Module Contents
logger
ToilServerUtilsTest
hidden
FileStateStoreTest
FileStateStoreURLTest
BucketUsingTest
AWSStateStoreTest
AbstractToilWESServerTest
ToilWESServerBenchTest
ToilWESServerWorkflowTest
ToilWESServerWorkflowTest.run_zip_workflow()
ToilWESServerWorkflowTest.test_run_workflow_relative_url_no_attachments_fails()
ToilWESServerWorkflowTest.test_run_workflow_relative_url()
ToilWESServerWorkflowTest.test_run_workflow_https_url()
ToilWESServerWorkflowTest.test_run_workflow_single_file_zip()
ToilWESServerWorkflowTest.test_run_workflow_multi_file_zip()
ToilWESServerWorkflowTest.test_run_workflow_manifest_zip()
ToilWESServerWorkflowTest.test_run_workflow_inputs_zip()
ToilWESServerWorkflowTest.test_run_workflow_manifest_and_inputs_zip()
ToilWESServerWorkflowTest.test_run_workflow_no_params_zip()
ToilWESServerWorkflowTest.test_run_and_cancel_workflows()
ToilWESServerCeleryWorkflowTest
ToilWESServerCeleryS3StateWorkflowTest
- toil.test.server.serverTest
- Submodules
- toil.test.sort
- Submodules
- toil.test.sort.restart_sort
- toil.test.sort.sort
- toil.test.sort.sortTest
- Attributes
- Classes
- Functions
- Module Contents
logger
defaultLineLen
defaultLines
defaultN
runMain()
SortTest
SortTest.setUp()
SortTest.tearDown()
SortTest.testAwsSingle()
SortTest.testAwsMesos()
SortTest.testFileMesos()
SortTest.testGoogleSingle()
SortTest.testGoogleMesos()
SortTest.testFileSingle()
SortTest.testFileSingleNonCaching()
SortTest.testFileSingleCheckpoints()
SortTest.testFileSingle10000()
SortTest.testFileGridEngine()
SortTest.testFileTorqueEngine()
SortTest.testNo
SortTest.testSort()
SortTest.testMerge()
SortTest.testCopySubRangeOfFile()
SortTest.testGetMidPoint()
- Submodules
- toil.test.src
- Submodules
- toil.test.src.autoDeploymentTest
- Attributes
- Classes
- Module Contents
logger
AutoDeploymentTest
AutoDeploymentTest.setUp()
AutoDeploymentTest.sitePackages
AutoDeploymentTest.testRestart()
AutoDeploymentTest.testSplitRootPackages()
AutoDeploymentTest.testUserTypesInJobFunctionArgs()
AutoDeploymentTest.testDeferralWithConcurrentEncapsulation()
AutoDeploymentTest.testDeferralWithFailureAndEncapsulation()
- toil.test.src.busTest
- toil.test.src.checkpointTest
- toil.test.src.deferredFunctionTest
- Attributes
- Classes
- Module Contents
logger
DeferredFunctionTest
DeferredFunctionTest.jobStoreType
DeferredFunctionTest.setUp()
DeferredFunctionTest.testDeferredFunctionRunsWithMethod()
DeferredFunctionTest.testDeferredFunctionRunsWithClassMethod()
DeferredFunctionTest.testDeferredFunctionRunsWithLambda()
DeferredFunctionTest.testDeferredFunctionRunsWithFailures()
DeferredFunctionTest.testNewJobsCanHandleOtherJobDeaths()
DeferredFunctionTest.testBatchSystemCleanupCanHandleWorkerDeaths()
- toil.test.src.dockerCheckTest
- Classes
- Module Contents
DockerCheckTest
DockerCheckTest.testOfficialUbuntuRepo()
DockerCheckTest.testBroadDockerRepo()
DockerCheckTest.testBroadDockerRepoBadTag()
DockerCheckTest.testNonexistentRepo()
DockerCheckTest.testToilQuayRepo()
DockerCheckTest.testBadQuayRepoNTag()
DockerCheckTest.testBadQuayRepo()
DockerCheckTest.testBadQuayTag()
DockerCheckTest.testGoogleRepo()
DockerCheckTest.testBadGoogleRepo()
DockerCheckTest.testApplianceParser()
- toil.test.src.environmentTest
- toil.test.src.fileStoreTest
- Attributes
- Classes
- Module Contents
testingIsAutomatic
logger
hidden
hidden.AbstractFileStoreTest
hidden.AbstractFileStoreTest.jobStoreType
hidden.AbstractFileStoreTest.setUp()
hidden.AbstractFileStoreTest.create_file()
hidden.AbstractFileStoreTest.testToilIsNotBroken()
hidden.AbstractFileStoreTest.testFileStoreLogging()
hidden.AbstractFileStoreTest.testFileStoreOperations()
hidden.AbstractFileStoreTest.testWriteReadGlobalFilePermissions()
hidden.AbstractFileStoreTest.testWriteExportFileCompatibility()
hidden.AbstractFileStoreTest.testImportReadFileCompatibility()
hidden.AbstractFileStoreTest.testReadWriteFileStreamTextMode()
hidden.AbstractNonCachingFileStoreTest
hidden.AbstractCachingFileStoreTest
hidden.AbstractCachingFileStoreTest.setUp()
hidden.AbstractCachingFileStoreTest.testExtremeCacheSetup()
hidden.AbstractCachingFileStoreTest.testCacheEvictionPartialEvict()
hidden.AbstractCachingFileStoreTest.testCacheEvictionTotalEvict()
hidden.AbstractCachingFileStoreTest.testCacheEvictionFailCase()
hidden.AbstractCachingFileStoreTest.testAsyncWriteWithCaching()
hidden.AbstractCachingFileStoreTest.testWriteNonLocalFileToJobStore()
hidden.AbstractCachingFileStoreTest.testWriteLocalFileToJobStore()
hidden.AbstractCachingFileStoreTest.testReadCacheMissFileFromJobStoreWithoutCachingReadFile()
hidden.AbstractCachingFileStoreTest.testReadCacheMissFileFromJobStoreWithCachingReadFile()
hidden.AbstractCachingFileStoreTest.testReadCachHitFileFromJobStore()
hidden.AbstractCachingFileStoreTest.testMultipleJobsReadSameCacheHitGlobalFile()
hidden.AbstractCachingFileStoreTest.testMultipleJobsReadSameCacheMissGlobalFile()
hidden.AbstractCachingFileStoreTest.testFileStoreExportFile()
hidden.AbstractCachingFileStoreTest.testReturnFileSizes()
hidden.AbstractCachingFileStoreTest.testReturnFileSizesWithBadWorker()
hidden.AbstractCachingFileStoreTest.testControlledFailedWorkerRetry()
hidden.AbstractCachingFileStoreTest.testRemoveLocalMutablyReadFile()
hidden.AbstractCachingFileStoreTest.testRemoveLocalImmutablyReadFile()
hidden.AbstractCachingFileStoreTest.testDeleteLocalFile()
hidden.AbstractCachingFileStoreTest.testSimultaneousReadsUncachedStream()
NonCachingFileStoreTestWithFileJobStore
CachingFileStoreTestWithFileJobStore
NonCachingFileStoreTestWithAwsJobStore
CachingFileStoreTestWithAwsJobStore
NonCachingFileStoreTestWithGoogleJobStore
CachingFileStoreTestWithGoogleJobStore
- toil.test.src.helloWorldTest
- toil.test.src.importExportFileTest
- toil.test.src.autoDeploymentTest
- Submodules
- toil.test.batchSystems
- Submodules
- toil.batchSystems
- Submodules
- toil