Rate limiting

TokenBucket and its async sibling AsyncTokenBucket are classic token-bucket limiters wired to be reusable across the runtime — sessions consume them per-turn so RPM, input-TPM and output-TPM caps match the way LLM providers actually enforce limits.

Quick example

from techrevati.runtime import (
    AgentSession, RateLimiter, TokenBucket, UsageSnapshot,
)

limiter = RateLimiter({
    "rpm":        TokenBucket("rpm",        capacity=60,     refill_per_second=1.0),
    "input_tpm":  TokenBucket("input_tpm",  capacity=200_000, refill_per_second=3_333.0),
    "output_tpm": TokenBucket("output_tpm", capacity=60_000,  refill_per_second=1_000.0),
})

agent = AgentSession(role="writer", phase="draft", rate_limiter=limiter)

with agent.session() as session:
    text, usage = session.run_turn(
        lambda: call_model(prompt),
        usage=UsageSnapshot(input_tokens=4_000, output_tokens=900),
    )

The session spends 1 token from "rpm" before calling the model and 4 000 / 900 from "input_tpm" / "output_tpm" after the snapshot is known. Empty buckets block until refill (or raise RateLimitExceededError when an explicit timeout is set). If that exception escapes an AgentSession, the terminal agent.failed event uses failure_class="rate_limit".

When to use

When NOT to use

Async vs sync

Choose one per code path. AsyncTokenBucket uses asyncio.Lock + asyncio.sleep, so waiting yields the event loop instead of pinning it; TokenBucket uses threading.Lock + time.sleep. State is independent (no shared counters), so a sync and an async bucket pointing at the same provider must be kept in sync by you — or just use one shape.

Usage is charged after the fact

acquire_usage records tokens the provider has already counted, so it cannot refuse them. Every configured bucket is debited first, and the balance may go negative when a turn exceeds capacity. The call then waits until the debt is repaid. timeout is one budget shared across the buckets; on timeout RateLimitExceededError is raised and the debits stand. TokenBucket.debit exposes the same charge directly.

Debt is capped at 24 hours of refill, so a bucket always recovers; tokens beyond the cap are forgotten. A single charge above 1e15 tokens raises ValueError.

acquire still refuses an amount above capacity, because it could never be served in advance.

Waiters are not served in order: a large acquire can wait a long time while many small ones keep spending the refill.

Tuning

Knob Default When to touch
capacity required Max burst you want to admit instantly. Set near the provider's 1-minute cap.
refill_per_second required Steady-state admission rate; divide the provider's per-minute cap by 60.
acquire(..., timeout=...) None (wait forever) Set when you'd rather fail fast than queue indefinitely.

Anti-patterns

See also