Circuit Breaker

techrevati.runtime.circuit_breaker

Circuit Breaker — Fault-tolerant execution with state machine.

Implements the Circuit Breaker pattern to prevent cascading failures when calling unreliable services or operations. Transitions between CLOSED (normal), OPEN (failing), and HALF_OPEN (testing) states.

Thread-safe with configurable failure threshold, recovery timeout, and number of in-flight probes permitted in HALF_OPEN. Uses time.monotonic for duration checks so clock jumps don't affect behavior; the clock function is injectable for deterministic tests.

Probe bookkeeping in HALF_OPEN:

  • At most half_open_max_probes probes are issued per cycle, not merely concurrently, so a cycle always finishes under sustained load: the circuit closes once every admitted probe has succeeded.
  • A cycle whose probes have not reported after recovery_timeout_seconds is abandoned and a new probe is admitted, so a hung probe (for example a worker thread orphaned by run_turn(timeout=...)) cannot block recovery forever. With recovery_timeout_seconds=0 there is no such window and no reclaim.
  • Every call is stamped with the generation current at admission, and the generation moves on each trip. A result from an older generation is ignored: a failure has been seen since that call started, and a slow call that fails late must not delay recovery.
  • Abandoning a cycle does not move the generation, because no failure was seen. A success from an abandoned probe closes the circuit, so a downstream slower than recovery_timeout_seconds can still recover. Its failure is ignored, because the current probes decide. A downstream that never answers gets at most one new probe per recovery_timeout_seconds.

CircuitState

Bases: str, Enum

Circuit breaker lifecycle states.

CircuitOpenError

CircuitOpenError(name)

Bases: Exception

Raised when circuit breaker is open and request is blocked.

CircuitBreaker dataclass

CircuitBreaker(
    name,
    failure_threshold=5,
    recovery_timeout_seconds=60.0,
    half_open_max_probes=1,
    clock=time.monotonic,
)

Bases: _BreakerStateMachine

Stateful circuit breaker with thread-safe transitions.

Parameters

name: Human-readable identifier (included in CircuitOpenError). failure_threshold: Consecutive failures before the circuit opens. recovery_timeout_seconds: Duration the circuit stays open before allowing probes. half_open_max_probes: Probe calls issued per HALF_OPEN cycle. Default 1 (Polly convention); raising to N spreads recovery risk over multiple calls (Resilience4j defaults to 10). The circuit closes once every admitted probe has succeeded. clock: Monotonic time source. Defaults to time.monotonic. Override in tests to make timing-dependent behavior deterministic.

call

call(fn, *args, **kwargs)

Execute fn with breaker protection. Raises CircuitOpenError if open.

In HALF_OPEN state, at most half_open_max_probes calls are admitted per cycle; excess callers receive CircuitOpenError until the cycle completes or expires.

record_success

record_success()

Record a successful execution. Closes the circuit if HALF_OPEN.

record_failure

record_failure()

Record a failed execution. Opens the circuit at threshold.

state

state()

Get current circuit state.

is_open

is_open()

Return True if circuit is open (blocking requests).

reset

reset()

Manually reset the circuit to CLOSED state.

AsyncCircuitBreaker dataclass

AsyncCircuitBreaker(
    name,
    failure_threshold=5,
    recovery_timeout_seconds=60.0,
    half_open_max_probes=1,
    clock=time.monotonic,
)

Bases: _BreakerStateMachine

Async sibling of CircuitBreaker — same state semantics, asyncio.Lock.

Independent from the sync variant: state is not shared. Choose one per downstream. The probe-serialization, monotonic clock, and clock-injection contracts match the sync class exactly so behavior is portable between sync and async code paths.

call async

call(coro_factory, *args, **kwargs)

Execute coro with breaker protection. Raises CircuitOpenError if open.

record_success async

record_success()

Record a successful execution. Closes the circuit if HALF_OPEN.

record_failure async

record_failure()

Record a failed execution. Opens the circuit at threshold.

state async

state()

Get current circuit state.

is_open async

is_open()

Return True if circuit is open (blocking requests).

reset async

reset()

Manually reset the circuit to CLOSED state.