Rate limit

techrevati.runtime.rate_limit

Rate limiting — Token-bucket primitives for sync and async call paths.

Token-aware throttling is the modern shape for LLM-provider rate limits because providers themselves are token-based (TPM, RPM, daily caps). A single TokenBucket admits or delays one resource (e.g. input tokens-per-minute); a RateLimiter composes three named buckets so typical provider limits (input TPM, output TPM, request RPM) can be expressed as one object.

Both TokenBucket (sync, threading.Lock) and AsyncTokenBucket (asyncio.Lock + asyncio.sleep) implement the same conceptual algorithm; async wins by yielding the event loop while waiting for refill instead of blocking it.

Clock is injectable on both variants (Callable[[], float] returning monotonic seconds). Tests pass a ManualClock to make timing-dependent behavior deterministic; production code uses time.monotonic by default.

Zero new runtime dependencies — stdlib only.

TokenBucket dataclass

TokenBucket(
    name, capacity, refill_per_second, clock=time.monotonic
)

Classic token-bucket limiter — sync variant.

try_acquire is non-blocking and returns True only when the bucket has enough tokens. acquire sleeps until the bucket refills, capped by an optional wait timeout; on timeout it raises RateLimitExceededError rather than silently exceeding the bound.

Parameters

name: Human-readable identifier used in error messages. capacity: Maximum tokens the bucket holds. Bursts up to this many requests can pass immediately. refill_per_second: Steady-state admission rate. clock: Monotonic time source. Defaults to time.monotonic.

available property

available

Current token balance (mostly useful for diagnostics + tests).

try_acquire

try_acquire(tokens=1.0)

Spend tokens if available; return whether the spend succeeded.

acquire

acquire(tokens=1.0, *, timeout=None)

Block until tokens are available, or raise on timeout.

timeout is the maximum wall-clock time we will sleep waiting for refill; None waits indefinitely.

debit

debit(tokens)

Record tokens that were already spent, without waiting.

Unlike acquire, this cannot refuse: the spend already happened, so the balance may go negative (even past -capacity). Later acquires wait until refill has paid the debt. The debt is capped at 24 hours of refill so it always stays repayable; anything beyond that is forgotten. A single debit above 1e15 tokens raises ValueError.

AsyncTokenBucket dataclass

AsyncTokenBucket(
    name, capacity, refill_per_second, clock=time.monotonic
)

Async sibling of TokenBucket.

Uses asyncio.Lock so refill bookkeeping is coroutine-safe, and asyncio.sleep so waiting yields control to the event loop instead of pinning the thread. State is independent from the sync variant — choose one per downstream.

debit async

debit(tokens)

Async sibling of TokenBucket.debit, with the same debt cap.

RateLimiter dataclass

RateLimiter(buckets=dict())

Composite of named token buckets, one per dimension.

Typical LLM-provider shape: rpm for requests-per-minute, input_tpm for input tokens-per-minute, output_tpm for output tokens-per-minute. Each bucket is independent; an empty buckets mapping is a valid no-op limiter.

acquire_pre_call spends RPM up front. After the call returns and UsageSnapshot is known, acquire_usage spends the input + output token budgets. This split mirrors how providers actually enforce limits.

acquire_pre_call

acquire_pre_call(*, request_cost=1.0, timeout=None)

Block on RPM bucket (buckets["rpm"]) if configured.

acquire_usage

acquire_usage(*, input_tokens, output_tokens, timeout=None)

Charge a completed turn's tokens, then wait out any resulting debt.

The tokens are already spent, so every configured bucket is debited first and the amount may exceed capacity (the balance goes negative). timeout is one budget shared across the buckets. On timeout RateLimitExceededError is raised and the debits stand, because the provider has counted those tokens either way.

AsyncRateLimiter dataclass

AsyncRateLimiter(buckets=dict())

Async sibling of RateLimiter — same semantics, async buckets.

RateLimitExceededError

RateLimitExceededError(bucket_name, tokens)

Bases: Exception

Raised when an acquire call exceeds the bucket's wait budget.

Carries the bucket name and the cost the caller tried to spend so the error message tells the caller which dimension blocked (input TPM vs RPM) and how big the request was. When the error escapes an AgentSession, the terminal event uses the public rate_limit failure class.