toil.test.server.safeFileTest

Tests for safe_read_file and safe_write_file that verify correct locking behavior by testing specific interleavings.

Approach: Mock fcntl.flock and file I/O operations to inject synchronization checkpoints, allowing deterministic control over thread execution order. This tests that the locking protocol is correct - the code acquires the right locks at the right times.

The tests use no sleeps - all synchronization is done via events/conditions.

Attributes

SYNC_TIMEOUT

Classes

Checkpoint

A synchronization point that allows a test to pause a thread and know

SimulatedLock

Simulates flock semantics using threading primitives.

Checkpointer

Base class for hooking checkpoints into operations.

LockCheckpointer

Checkpointer that pauses after flock acquisition.

ReadCheckpointer

Checkpointer that pauses during file read.

WriteCheckpointer

Checkpointer that pauses during file write.

TestSafeFileInterleaving

Tests that verify locking correctness through deterministic interleavings.

Module Contents

toil.test.server.safeFileTest.SYNC_TIMEOUT = 10.0
class toil.test.server.safeFileTest.Checkpoint

A synchronization point that allows a test to pause a thread and know when the thread has arrived.

Usage:

checkpoint = Checkpoint()

# In worker thread: checkpoint.arrive_and_wait() # signals arrival, then blocks

# In test: checkpoint.wait_for_arrival() # blocks until thread arrives # … check state … checkpoint.release() # allows thread to proceed

arrive_and_wait(timeout=SYNC_TIMEOUT)

Signal that we’ve arrived at the checkpoint, then wait for release. Returns True if released, False if timed out.

Parameters:

timeout (float)

Return type:

bool

wait_for_arrival(timeout=SYNC_TIMEOUT)

Wait for a thread to arrive at this checkpoint. Returns True if arrived, False if timed out.

Parameters:

timeout (float)

Return type:

bool

release()

Release the thread waiting at this checkpoint.

Return type:

None

has_arrived()

Check if a thread has arrived (non-blocking).

Return type:

bool

class toil.test.server.safeFileTest.SimulatedLock

Simulates flock semantics using threading primitives.

Supports shared (LOCK_SH) and exclusive (LOCK_EX) locks with proper blocking behavior: - Multiple shared locks can be held simultaneously - Exclusive lock blocks until all shared locks are released - Shared locks block while exclusive lock is held

acquire_shared()

Acquire a shared lock (blocks if exclusive lock held).

Return type:

None

acquire_exclusive()

Acquire an exclusive lock (blocks if any lock held).

Return type:

None

release()

Release whatever lock this thread holds.

Return type:

None

property has_exclusive: bool
Return type:

bool

property shared_count: int
Return type:

int

class toil.test.server.safeFileTest.Checkpointer

Bases: abc.ABC

Base class for hooking checkpoints into operations.

add(thread_name, checkpoint)

Register a checkpoint for the given thread.

Parameters:
Return type:

None

get(thread_name)

Get checkpoint for thread, if any.

Parameters:

thread_name (str)

Return type:

Checkpoint | None

abstractmethod install()

Install patches for this checkpointer.

Each checkpointer provides its own context manager that patches the necessary functions. Multiple checkpointers compose by each capturing the current (possibly already-patched) functions.

Return type:

collections.abc.Generator[None, None, None]

class toil.test.server.safeFileTest.LockCheckpointer

Bases: Checkpointer

Checkpointer that pauses after flock acquisition.

get_lock_for_path(path)

Get the lock for a path (for test assertions).

Parameters:

path (str)

Return type:

SimulatedLock | None

install()

Patch open to register fds, and patch flock with lock simulation.

Return type:

collections.abc.Generator[None, None, None]

class toil.test.server.safeFileTest.ReadCheckpointer

Bases: Checkpointer

Checkpointer that pauses during file read.

install()

Patch open to wrap read operations with checkpoint hooks.

Return type:

collections.abc.Generator[None, None, None]

class toil.test.server.safeFileTest.WriteCheckpointer

Bases: Checkpointer

Checkpointer that pauses during file write.

install()

Patch open to wrap write operations with checkpoint hooks.

Return type:

collections.abc.Generator[None, None, None]

class toil.test.server.safeFileTest.TestSafeFileInterleaving

Tests that verify locking correctness through deterministic interleavings.

Each test explicitly controls thread execution order using checkpoints to verify that locks are held and respected at the right times. No sleeps are used - all synchronization is event-based.

setup_test_file(tmp_path)

Set up test file path for each test.

Parameters:

tmp_path (pathlib.Path)

Return type:

collections.abc.Generator[None]

patched_io(*checkpointers)

Context manager that installs all checkpointer patches.

Uses ExitStack to compose the context managers from each checkpointer. Patches compose naturally since each captures the current open.

Parameters:

checkpointers (Checkpointer)

Return type:

collections.abc.Generator[None, None, None]

test_reader_blocked_while_writer_holds_lock()

Verify that a reader cannot proceed while a writer holds the exclusive lock.

Sequence: 1. Writer acquires exclusive lock, arrives at checkpoint 2. Test verifies writer has lock 3. Reader tries to acquire shared lock (will block on simulated lock) 4. Test verifies reader is blocked 5. Test releases writer checkpoint 6. Both complete, reader sees written content

Return type:

None

test_writer_blocked_while_reader_holds_lock()

Verify that a writer cannot proceed while a reader holds a shared lock.

Return type:

None

test_multiple_readers_not_blocked()

Verify that multiple readers can hold shared locks simultaneously.

Return type:

None

test_writers_serialize()

Verify that two writers cannot hold exclusive locks simultaneously.

Return type:

None

test_reader_paused_mid_read_blocks_writer()

Verify that a writer is blocked even when reader is paused during the actual read operation (not just after lock acquisition).

Return type:

None

test_writer_paused_mid_write_blocks_reader()

Verify that a reader is blocked even when writer is paused during the actual write operation (not just after lock acquisition).

Return type:

None