Gateway Admission Control

The gateway now ships in the praisonai-bot package. praisonai serve gateway still works exactly as documented here; for a standalone install see praisonai-bot Migration.

For the composed one-switch experience, see Reliability Preset. This page documents the admission-control knob in isolation.

Admission control caps the number of concurrent inbound agent runs across all users, queues the overflow fairly, and explicitly sheds load when the queue is full.

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=32,
)
bot.start()

The user sends a chat message on a channel; admission control admits, queues, or rejects the run before the agent replies.

Admission control bounds concurrent inbound runs. For a bound on request rate per identity, see Gateway Rate Limit.

Quick Start

For a one-switch preset that turns on admission with sensible defaults alongside graceful drain, see Gateway Reliability Presets.

Simple Usage

Cap aggregate concurrent runs with a single parameter:

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=32,
)
bot.start()

With Configuration

Add a wait queue and choose what happens when the queue is full:

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

bot = BotOS(
    agent=agent,
    platforms=["telegram", "discord"],
    max_concurrent_runs=32,
    queue_depth=128,
    overflow_policy="reject",   # reject | queue | shed_oldest
)
bot.start()

How It Works

Decision	When	What the user sees
`ADMIT`	`in_flight < max_concurrent_runs`	Immediate response
`QUEUE`	At ceiling, queue has room	Brief wait, then response
`REJECT`	Queue full and policy is `reject` (or `shed_oldest` can’t evict)	Friendly busy acknowledgement

Configuration Options

These fields live on GatewayConfig (read from praisonaiagents/gateway/config.py):

Option	Type	Default	Description
`max_concurrent_runs`	`int`	`0`	Max concurrent inbound agent runs across all users. `0` disables the gate (legacy behaviour).
`max_concurrent_runs_per_scope`	`int`	`0`	Optional per-tenant/per-scope sub-limit within the global ceiling. A tenant already at its sub-limit is queued/shed against its own slice even when a global slot is free, so a noisy tenant cannot starve quiet ones. `0` disables it (concurrency-only global behaviour). See Per-tenant fairness.
`queue_depth`	`int`	`128`	Max number of inbound turns that can wait when at the ceiling.
`overflow_policy`	`str`	`"reject"`	Behaviour when at the ceiling and the queue is full: `reject`, `queue`, or `shed_oldest`.
`max_rss_mb`	`float`	`0.0`	Hard RSS ceiling in MiB for memory-aware admission. `0` disables it (concurrency-only, legacy behaviour).

Precedence: CLI flags → YAML → Python defaults.

Python

from praisonai.bots import BotOS
from praisonaiagents import Agent

agent = Agent(name="Support", instructions="Help users")

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=32,
    max_concurrent_runs_per_scope=4,   # per-tenant cap within the ceiling
    queue_depth=128,
    overflow_policy="reject",
)

YAML (`gateway.yaml`)

gateway:
  host: "127.0.0.1"
  port: 8765
  max_concurrent_runs: 32
  max_concurrent_runs_per_scope: 4   # per-tenant cap within the ceiling
  queue_depth: 128
  overflow_policy: reject   # reject | queue | shed_oldest

CLI

praisonai gateway start \
  --max-concurrent-runs 32 \
  --queue-depth 128 \
  --overflow-policy reject

CLI flags override YAML, which overrides Python defaults. Set max_concurrent_runs_per_scope via YAML or the Python surface.

Common Patterns

Production multi-tenant bot — explicit busy ack under load; no OOM risk; no provider 429 storm:

bot = BotOS(
    agent=agent,
    platforms=["telegram", "discord", "slack"],
    max_concurrent_runs=32,
    queue_depth=128,
    overflow_policy="reject",
)

Burst-tolerant single-channel — lower aggregate concurrency, deeper queue, no rejections under modest bursts:

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=8,
    queue_depth=64,
    overflow_policy="queue",
)

Newest-message-wins — useful when conversation freshness matters more than fairness. When shed_oldest can’t evict a live waiter, the newcomer is rejected rather than overfilling the queue:

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=16,
    queue_depth=32,
    overflow_policy="shed_oldest",
)

Multi-tenant fair-share — a global ceiling of 32 with a per-tenant cap of 4, so no single tenant monopolises the gateway even under a burst:

bot = BotOS(
    agent=agent,
    platforms=["telegram", "discord", "slack"],
    max_concurrent_runs=32,
    queue_depth=128,
    overflow_policy="reject",
    max_concurrent_runs_per_scope=4,
)

Observability

BotOS.admission_stats exposes live counters without any extra setup:

print(bot.admission_stats)
# {
#   "max_concurrent_runs": 32, "queue_depth": 128,
#   "max_concurrent_runs_per_scope": 4,   # ← per-tenant sub-limit (0 when disabled)
#   "in_flight": 12, "queued": 3,
#   "admitted": 1247, "rejected": 0, "shed": 0,
#   "max_rss_mb": 0.0,   # ← memory-aware ceiling (0.0 when disabled)
# }

Watch rejected alongside your LLM provider’s 429 rate. When both rise together, raise max_concurrent_runs. When only rejected rises, deepen queue_depth or switch to overflow_policy="queue".

Memory-aware admission

Set one number — max_rss_mb — and the gateway queues turns under soft RSS pressure and sheds them under hard pressure, before the OOM killer fires. Zero deps, no new subsystem — the same admission gate that enforces the concurrency ceiling folds a memory decision into every admit().

MemoryPressurePolicy sheds new turns under pressure; see Gateway Memory-Pressure Eviction for reclaiming memory from idle warm caches before the OOM killer fires.

Quick start

Python (single knob)
YAML
Explicit ladder

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

# 1 GiB hard ceiling; soft (queue) threshold auto-derived at 90% (~921 MiB).
bot = BotOS(agent=agent, platforms=["telegram"], max_rss_mb=1024)

print(bot.admission_stats["max_rss_mb"])
# 1024.0

# Top-level (canonical praisonai.yaml or gateway.yaml)
max_rss_mb: 1024

# Or nested under gateway:
gateway:
  max_rss_mb: 1024

from praisonaiagents.gateway import MemoryPressurePolicy
from praisonai_bot.bots._admission import build_admission_gate

# Explicit two-threshold ladder instead of the single-knob 90% default.
policy = MemoryPressurePolicy(soft_rss_mb=800, hard_rss_mb=1024)
gate = build_admission_gate(
    max_concurrent_runs=32,   # concurrency ceiling (optional)
    resource_policy=policy,   # memory-aware layer
)

The pressure ladder

Given a sample’s resident set size (rss_mb), the policy decides:

RSS	Decision	What the gateway does
`rss < soft_rss_mb`	ADMIT	Run the turn immediately.
`soft_rss_mb ≤ rss < hard_rss_mb`	QUEUE	Apply backpressure via the existing bounded wait queue.^*
`rss ≥ hard_rss_mb`	REJECT	Shed with a polite busy ack (honours the existing `reject` / `shed_oldest` overflow policy).
`rss is None` (platform can’t report)	ADMIT	Never block on a missing signal — sampler self-disables with a single warning.

^* Memory-only mode caveat. With no concurrency ceiling (max_concurrent_runs=0), there is no slot to wait on, so a soft-pressure QUEUE degrades to ADMIT — real wait-queue backpressure needs a concurrency ceiling alongside max_rss_mb. The hard threshold always REJECTs regardless, which is what prevents the OOM kill.

Memory-aware configuration

Option	Type	Default	Description
`max_rss_mb`	`float`	`0.0`	Hard RSS ceiling in MiB. `0` disables memory-aware admission (concurrency-only, byte-for-byte legacy behaviour).
`MemoryPressurePolicy.soft_rss_mb`	`float`	`0.0`	Soft (QUEUE) threshold in MiB. When you set only `max_rss_mb`, this is auto-derived as `0.9 × max_rss_mb`.
`MemoryPressurePolicy.hard_rss_mb`	`float`	`0.0`	Hard (REJECT) threshold in MiB.
`soft_ratio` (in `build_memory_pressure_policy`)	`float`	`0.9`	Fraction of the hard ceiling that becomes the soft threshold when a single-knob `max_rss_mb` is used. Clamped to `[0.0, 1.0]`.

MemoryPressurePolicy fails fast at construction — a ValueError is raised if soft_rss_mb or hard_rss_mb is negative or non-numeric, or if soft_rss_mb > hard_rss_mb (which would queue turns that should be shed). No silent pressure-ladder inversion.

Choosing values

Small always-on host (a $5 VPS)

Set max_rss_mb at ~70–80% of the box’s RAM ceiling minus other resident daemons. Example: on a 1 GiB VPS running only the gateway, max_rss_mb: 700 gives a ~630 MiB soft threshold — bursts queue, sustained pressure sheds, and the OOM killer stays quiet.

Containerised deployment with a memory limit

Set max_rss_mb to ~80% of the container’s memory limit. The pressure ladder then queues before the kernel starts reclaiming pages aggressively, preserving observability and giving the graceful-drain window a chance to complete in-flight turns.

Large host combining concurrency + memory

Combine max_concurrent_runs (concurrency ceiling) with max_rss_mb (memory ceiling). Concurrency handles CPU-scaled bursts; memory handles per-turn resident growth (large tool outputs, long transcripts). Either dimension can shed independently.

Development / test (off)

Default max_rss_mb=0 disables memory-aware admission entirely. admit() is bit-for-bit as before — the exact behaviour of every release prior to 2026-07-27.

Sampler behaviour

The wrapper samples process RSS on a lightweight cadence:

Preferred: psutil.Process().memory_info().rss — a live, monotonic reading. Enabled automatically if psutil is installed.
Fallback: stdlib resource.getrusage(RUSAGE_SELF).ru_maxrss — a peak (not live) reading, good enough to catch a climbing leak. Kilobytes on Linux, bytes on macOS/BSD; the sampler normalises to MiB.
Self-disable: if the platform can report neither (Windows without psutil, exotic runtimes), the sampler emits a single AdmissionGate: resource sampling unavailable on this platform; memory-pressure admission disabled. warning and thereafter returns a rss_mb=None sample. The policy admits on None — the monitor never crashes the gateway it protects.

The resource stdlib module is Unix-only. On Windows without psutil, memory-aware admission is silently disabled — a single warning is logged at first sample. Install psutil in production Windows deployments if you rely on max_rss_mb.

Memory observability

admission_stats gains a max_rss_mb field so operators can confirm the ceiling is wired end-to-end:

bot.admission_stats
# {
#   "max_concurrent_runs": 32, "queue_depth": 128,
#   "in_flight": 4, "queued": 0,
#   "admitted": 1247, "rejected": 3, "shed": 0,
#   "max_rss_mb": 1024.0,     # ← the configured hard ceiling (0.0 when disabled)
# }

stats() surfaces only the configured ceiling, not a rolling RSS window. For historical RSS for capacity planning, scrape psutil.Process(<pid>).memory_info().rss or your existing container / host metrics pipeline.

Per-tenant fairness

Cap concurrency per tenant so a single noisy tenant can’t monopolise every run slot. The per-scope gate runs first and is the tighter, per-tenant ceiling; the global ceiling still bounds aggregate concurrency second. A tenant already at max_concurrent_runs_per_scope is queued (or shed by the declared overflow_policy) against its own slice — even when the global ceiling has free slots — so its burst can never consume another tenant’s capacity.

scope is the caller’s session_id at the wrapper call site — the per-tenant profile scope, mirroring the scope axis of rate-limit and spend-budget admission. Unscoped callers (scope="") fall through to the global decision, so the option is safe to enable incrementally.

Python
YAML
CLI

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

bot = BotOS(
    agent=agent,
    platforms=["telegram", "discord"],
    max_concurrent_runs=32,             # global ceiling
    max_concurrent_runs_per_scope=4,    # per-tenant cap within the ceiling
    queue_depth=128,
    overflow_policy="reject",
)
bot.start()

gateway:
  host: "127.0.0.1"
  port: 8765
  max_concurrent_runs: 32
  max_concurrent_runs_per_scope: 4
  queue_depth: 128
  overflow_policy: reject

# Set the per-scope sub-limit via gateway.yaml, then start:
praisonai gateway start --config gateway.yaml

How per-scope admission decides

With max_concurrent_runs=32 and max_concurrent_runs_per_scope=4:

Quiet tenant B sends a turn — its scope has 0 in-flight, well under 4, and the global count is under 32, so the turn is admitted immediately.
Noisy tenant A bursts 10 concurrent turns: the first 4 are admitted (they fill A’s per-scope slice); turns 5–10 are queued against A’s slice — even though the global ceiling still has 28 free slots.
Tenant B sends a second turn while A’s burst is queued: it is admitted immediately, because B’s per-scope count is 1 and the global ceiling has room. No starvation.
As A’s in-flight turns complete, its queued turns dispatch one at a time, keeping A at most 4 concurrent.

max_concurrent_runs_per_scope = 0 (the default) or an empty scope makes the decision byte-for-byte the current global-only behaviour — no existing configuration changes. The option is fail-open.

Best Practices

Start with max_concurrent_runs ≈ 2× expected steady-state

Set max_concurrent_runs to roughly twice your expected concurrent-user baseline. Watch admission_stats.rejected — if rejections are non-zero under normal load, raise the ceiling.

Size max_concurrent_runs_per_scope for your tenant count

On a multi-tenant hosted gateway, set max_concurrent_runs_per_scope to roughly max_concurrent_runs / expected_active_tenants. Rate-limit and spend-budget admission are already per-tenant; this closes the last global-only axis of the admission triad.

Default to overflow_policy='reject' for multi-user deployments

Silent unbounded queueing under load is harder to debug than an explicit busy ack. reject surfaces pressure immediately and lets users retry on their own schedule.

Pair with flow control for full gateway protection

Admission control bounds inbound concurrent runs; flow control bounds outbound send throughput and per-session inbox depth. Production gateways usually want both.

Leave the gate off only for single-user or local dev

max_concurrent_runs=0 (the default) disables the gate entirely — every inbound turn runs immediately. Suitable for local development or single-operator deployments where there is no shared provider quota to protect.

Per-tenant fairness

Cap concurrency per tenant so one noisy tenant can’t monopolise the gateway

Memory-aware admission

Queue under soft RSS pressure, shed under hard pressure — before the OOM killer fires

Memory-Pressure Eviction

The eviction-side sibling — reclaim memory from idle warm caches before the OOM killer fires

Gateway Reliability Presets

One switch that turns on admission + graceful drain with sensible defaults

Gateway Flow Control

Outbound counterpart — bounded inboxes and slow-consumer disconnect

Gateway Rate Limit

Bound inbound turns per identity/scope with a sliding window or custom limiter

Gateway Overview

Full gateway architecture and feature index

Graceful Drain

Bound in-flight turns on shutdown — Python-first via GatewayConfig(drain_timeout=…)

Event-Loop Watchdog

Detect a wedged asyncio loop — Python-first via GatewayConfig(watchdog=True)

​Quick Start

​How It Works

​Configuration Options

​Python

​YAML (gateway.yaml)

​CLI

​Common Patterns

​Observability

​Memory-aware admission

​Quick start

​The pressure ladder

​Memory-aware configuration

​Choosing values

​Sampler behaviour

​Memory observability

​Per-tenant fairness

​How per-scope admission decides

​Best Practices

​Related

Per-tenant fairness

Memory-aware admission

Memory-Pressure Eviction

Gateway Reliability Presets

Gateway Flow Control

Gateway Rate Limit

Gateway Overview

Graceful Drain

Event-Loop Watchdog

Quick Start

How It Works

Configuration Options

Python

YAML (`gateway.yaml`)

CLI

Common Patterns

Observability

Memory-aware admission

Quick start

The pressure ladder

Memory-aware configuration

Choosing values

Sampler behaviour

Memory observability

Per-tenant fairness

How per-scope admission decides

Best Practices

Related