Skip to content

Routing

techrevati.runtime.routing

Provider routing — Pluggable strategy for picking the next LLM provider.

The runtime already knows that a provider failed (classify_exception maps the exception to FailureScenario.PROVIDER_FAILURE or LLM_ERROR with a SWITCH_PROVIDER step in the recipe). What it does not know on its own is which provider to try next when the caller has a stable list of options. ProviderRouter is the extension point.

Three reference implementations cover the common patterns:

  • StaticProviderRouter — wraps the existing next_provider function: cycle through available_providers skipping ones the caller has already excluded. Zero-config and stateless.
  • RoundRobinProviderRouter — strictly rotates through the list, regardless of which provider failed. Good for cost balancing.
  • WeightedProviderRouter — picks the first provider whose weight is non-zero, in declaration order. Deterministic; pair with a cron / config reload to express "prefer A, fall back to B at 9pm".

All three are sync. Async sessions call them synchronously (selecting a provider name is cheap); if a future implementation needs to talk to a control plane, wrap it in a thread executor in the caller, not here.

ProviderRouter

Bases: Protocol

Strategy that picks the next provider to try on a failure step.

Implementations may be stateful (RoundRobinProviderRouter keeps a cursor) or stateless. They must never raise — return None when there is no acceptable next provider so the caller can escalate.

select 

select(*, scenario, attempt, current, exclude=())

Return the next provider name, or None if none qualifies.

Parameters

scenario: The failure that triggered the switch. Implementations may ignore it; useful for "only fail over on PROVIDER_FAILURE" policies. attempt: Which switch attempt this is within the current recovery run (1-indexed). Lets weighted routers skip already-tried options. current: The provider the failing call used; routers typically avoid returning the same one. exclude: Additional providers the caller wants skipped (e.g. ones that already 429'd this minute).

StaticProviderRouter dataclass 

StaticProviderRouter(available_providers)

First-acceptable router. Wraps :func:next_provider for parity.

available_providers is the ordered fallback list; the router returns the first entry that is neither current nor in exclude. Stateless; safe to share across sessions.

RoundRobinProviderRouter dataclass 

RoundRobinProviderRouter(available_providers)

Strict-rotation router. Stateful — keeps a cursor across calls.

WeightedProviderRouter dataclass 

WeightedProviderRouter(providers)

Pick the highest-weight non-excluded provider.

providers is an ordered tuple of (name, weight) pairs. Zero or negative weights are treated as disabled. Ties favor earlier entries to keep selection deterministic.