Concurrency Limit Policy • AI Agent SDK

ConcurrencyLimitPolicy: Config-driven inbound admission policy for a bounded gateway.

ConcurrencyLimitPolicy

Defined in the protocols module.

Config-driven inbound admission policy for a bounded gateway.

The default referenced by `gateway.max_concurrent_runs / gateway.queue_depth / gateway.overflow_policy in gateway.yaml and the BotOS(..., max_concurrent_runs=...)` Python surface. It is intentionally minimal and dependency-free so the decision lives in core and is provable in isolation; the wrapper owns the side effects (acquire a semaphore slot, enqueue/dequeue, return a busy ack).

The decision is:

• `ADMIT while in_flight < max_concurrent_runs`.
• At the ceiling, `QUEUE while queued < queue_depth` and the
• `overflow_policy` permits waiting.
• Otherwise the `overflow_policy` decides the shed behaviour:
• `"reject" → :attr:AdmissionDecision.REJECT` (busy ack).
• `"queue" → :attr:AdmissionDecision.QUEUE` (block beyond the
• declared depth — for callers that prefer unbounded waiting to
• shedding; the wrapper still bounds the actual queue object).
• `"shed_oldest" → :attr:AdmissionDecision.QUEUE`; the wrapper
• drops the oldest waiter to make room rather than rejecting the new
• arrival.

A `max_concurrent_runs of 0` disables admission control entirely (today’s behaviour: every inbound turn is admitted immediately).

On a shared/multi-tenant gateway an optional `max_concurrent_runs_per_scope caps concurrency per tenant/scope within the global ceiling (Issue #5168): a tenant that already holds its sub-limit is queued/shed against its own slice even when a global slot is notionally free, so a noisy neighbour cannot starve quiet tenants. A value of 0` (the default) disables the per-scope sub-limit, preserving the byte-for-byte global-only behaviour.

Example::

ConcurrencyLimitPolicy(max_concurrent_runs=32, queue_depth=128,
                       overflow_policy="reject",
                       max_concurrent_runs_per_scope=4)

Constructor

Methods

Source