Circuit Breaker¶
techrevati.runtime.circuit_breaker ¶
Circuit Breaker — Fault-tolerant execution with state machine.
Implements the Circuit Breaker pattern to prevent cascading failures when calling unreliable services or operations. Transitions between CLOSED (normal), OPEN (failing), and HALF_OPEN (testing) states.
Thread-safe with configurable failure threshold, recovery timeout,
and number of in-flight probes permitted in HALF_OPEN. Uses
time.monotonic for duration checks so clock jumps don't affect
behavior; the clock function is injectable for deterministic tests.
Probe bookkeeping in HALF_OPEN:
- At most
half_open_max_probesprobes are issued per cycle, not merely concurrently, so a cycle always finishes under sustained load: the circuit closes once every admitted probe has succeeded. - A cycle whose probes have not reported after
recovery_timeout_secondsis abandoned and a new probe is admitted, so a hung probe (for example a worker thread orphaned byrun_turn(timeout=...)) cannot block recovery forever. Withrecovery_timeout_seconds=0there is no such window and no reclaim. - Every call is stamped with the generation current at admission, and the generation moves on each trip. A result from an older generation is ignored: a failure has been seen since that call started, and a slow call that fails late must not delay recovery.
- Abandoning a cycle does not move the generation, because no failure was seen. A
success from an abandoned probe closes the circuit, so a downstream slower than
recovery_timeout_secondscan still recover. Its failure is ignored, because the current probes decide. A downstream that never answers gets at most one new probe perrecovery_timeout_seconds.
CircuitState ¶
Bases: str, Enum
Circuit breaker lifecycle states.
CircuitOpenError ¶
CircuitOpenError(name)
Bases: Exception
Raised when circuit breaker is open and request is blocked.
CircuitBreaker
dataclass
¶
CircuitBreaker(
name,
failure_threshold=5,
recovery_timeout_seconds=60.0,
half_open_max_probes=1,
clock=time.monotonic,
)
Bases: _BreakerStateMachine
Stateful circuit breaker with thread-safe transitions.
Parameters¶
name:
Human-readable identifier (included in CircuitOpenError).
failure_threshold:
Consecutive failures before the circuit opens.
recovery_timeout_seconds:
Duration the circuit stays open before allowing probes.
half_open_max_probes:
Probe calls issued per HALF_OPEN cycle. Default 1 (Polly
convention); raising to N spreads recovery risk over multiple
calls (Resilience4j defaults to 10). The circuit closes once every
admitted probe has succeeded.
clock:
Monotonic time source. Defaults to time.monotonic. Override
in tests to make timing-dependent behavior deterministic.
call ¶
call(fn, *args, **kwargs)
Execute fn with breaker protection. Raises CircuitOpenError if open.
In HALF_OPEN state, at most half_open_max_probes calls are admitted
per cycle; excess callers receive CircuitOpenError until the cycle
completes or expires.
AsyncCircuitBreaker
dataclass
¶
AsyncCircuitBreaker(
name,
failure_threshold=5,
recovery_timeout_seconds=60.0,
half_open_max_probes=1,
clock=time.monotonic,
)
Bases: _BreakerStateMachine
Async sibling of CircuitBreaker — same state semantics, asyncio.Lock.
Independent from the sync variant: state is not shared. Choose one per downstream. The probe-serialization, monotonic clock, and clock-injection contracts match the sync class exactly so behavior is portable between sync and async code paths.