Gateway Admission Control

Bound the number of concurrent inbound agent runs with a fair queue and explicit overflow policy

Admission control caps the number of concurrent inbound agent runs across all users, queues the overflow fairly, and explicitly sheds load when the queue is full.

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=32,
)
bot.start()

The user sends a chat message on a channel; admission control admits, queues, or rejects the run before the agent replies.

Quick Start

1. Simple Usage

   Cap aggregate concurrent runs with a single parameter:

   from praisonaiagents import Agent
   from praisonai.bots import BotOS
   
   agent = Agent(name="Support", instructions="Help users")
   
   bot = BotOS(
       agent=agent,
       platforms=["telegram"],
       max_concurrent_runs=32,
   )
   bot.start()

2. With Configuration

   Add a wait queue and choose what happens when the queue is full:

   from praisonaiagents import Agent
   from praisonai.bots import BotOS
   
   agent = Agent(name="Support", instructions="Help users")
   
   bot = BotOS(
       agent=agent,
       platforms=["telegram", "discord"],
       max_concurrent_runs=32,
       queue_depth=128,
       overflow_policy="reject",   # reject | queue | shed_oldest
   )
   bot.start()

How It Works

Configuration Options

These fields live on GatewayConfig (read from praisonaiagents/gateway/config.py):

Precedence: CLI flags → YAML → Python defaults.

Python

from praisonai.bots import BotOS
from praisonaiagents import Agent

agent = Agent(name="Support", instructions="Help users")

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=32,
    max_concurrent_runs_per_scope=4,   # per-tenant cap within the ceiling
    queue_depth=128,
    overflow_policy="reject",
    per_user_per_minute=20,            # optional per-user turn budget
)

YAML (gateway.yaml)

gateway:
  host: "127.0.0.1"
  port: 8765
  max_concurrent_runs: 32
  max_concurrent_runs_per_scope: 4   # per-tenant cap within the ceiling
  queue_depth: 128
  overflow_policy: reject   # reject | queue | shed_oldest
  per_user_per_minute: 20   # optional; 0 = unlimited

CLI

praisonai gateway start \
  --max-concurrent-runs 32 \
  --queue-depth 128 \
  --overflow-policy reject

CLI flags override YAML, which overrides Python defaults. Set max_concurrent_runs_per_scope via YAML or the Python surface.

Common Patterns

Production multi-tenant bot — explicit busy ack under load; no OOM risk; no provider 429 storm:

bot = BotOS(
    agent=agent,
    platforms=["telegram", "discord", "slack"],
    max_concurrent_runs=32,
    queue_depth=128,
    overflow_policy="reject",
)

Burst-tolerant single-channel — lower aggregate concurrency, deeper queue, no rejections under modest bursts:

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=8,
    queue_depth=64,
    overflow_policy="queue",
)

Newest-message-wins — useful when conversation freshness matters more than fairness. When shed_oldest can’t evict a live waiter, the newcomer is rejected rather than overfilling the queue:

bot = BotOS(
    agent=agent,
    platforms=["telegram"],
    max_concurrent_runs=16,
    queue_depth=32,
    overflow_policy="shed_oldest",
)

Multi-tenant fair-share — a global ceiling of 32 with a per-tenant cap of 4, so no single tenant monopolises the gateway even under a burst:

bot = BotOS(
    agent=agent,
    platforms=["telegram", "discord", "slack"],
    max_concurrent_runs=32,
    queue_depth=128,
    overflow_policy="reject",
    max_concurrent_runs_per_scope=4,
)

Observability

BotOS.admission_stats exposes live counters without any extra setup:

print(bot.admission_stats)
# {
#   "max_concurrent_runs": 32, "queue_depth": 128,
#   "max_concurrent_runs_per_scope": 4,   # ← per-tenant sub-limit (0 when disabled)
#   "in_flight": 12, "queued": 3,
#   "awaiting_human": 2,   # ← admitted turns parked on a human approval, not holding a slot
#   "admitted": 1247, "rejected": 0, "shed": 0,
#   "max_rss_mb": 0.0,   # ← memory-aware ceiling (0.0 when disabled)
#   "per_user_per_minute": 20,   # ← configured budget (0 when disabled)
#   "rate_limited": 0,           # ← refusals due to the per-user budget
# }

Watch rejected alongside your LLM provider’s 429 rate. When both rise together, raise max_concurrent_runs. When only rejected rises, deepen queue_depth or switch to overflow_policy="queue".

• Watch rate_limited to spot a chatty client; a steady rise with rejected flat means raising max_concurrent_runs won’t help that user — raise per_user_per_minute instead.

Memory-aware admission

Set one number — max_rss_mb — and the gateway queues turns under soft RSS pressure and sheds them under hard pressure, before the OOM killer fires. Zero deps, no new subsystem — the same admission gate that enforces the concurrency ceiling folds a memory decision into every admit().

Quick start

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

# 1 GiB hard ceiling; soft (queue) threshold auto-derived at 90% (~921 MiB).
bot = BotOS(agent=agent, platforms=["telegram"], max_rss_mb=1024)

print(bot.admission_stats["max_rss_mb"])
# 1024.0

# Top-level (canonical praisonai.yaml or gateway.yaml)
max_rss_mb: 1024

# Or nested under gateway:
gateway:
  max_rss_mb: 1024

from praisonaiagents.gateway import MemoryPressurePolicy
from praisonai_bot.bots._admission import build_admission_gate

# Explicit two-threshold ladder instead of the single-knob 90% default.
policy = MemoryPressurePolicy(soft_rss_mb=800, hard_rss_mb=1024)
gate = build_admission_gate(
    max_concurrent_runs=32,   # concurrency ceiling (optional)
    resource_policy=policy,   # memory-aware layer
)

The pressure ladder

Given a sample’s resident set size (rss_mb), the policy decides:

Memory-aware configuration

Choosing values

Sampler behaviour

The wrapper samples process RSS on a lightweight cadence:

• Preferred: psutil.Process().memory_info().rss — a live, monotonic reading. Enabled automatically if psutil is installed.
• Fallback: stdlib resource.getrusage(RUSAGE_SELF).ru_maxrss — a peak (not live) reading, good enough to catch a climbing leak. Kilobytes on Linux, bytes on macOS/BSD; the sampler normalises to MiB.
• Self-disable: if the platform can report neither (Windows without psutil, exotic runtimes), the sampler emits a single AdmissionGate: resource sampling unavailable on this platform; memory-pressure admission disabled. warning and thereafter returns a rss_mb=None sample. The policy admits on None — the monitor never crashes the gateway it protects.

Memory observability

admission_stats gains a max_rss_mb field so operators can confirm the ceiling is wired end-to-end:

bot.admission_stats
# {
#   "max_concurrent_runs": 32, "queue_depth": 128,
#   "in_flight": 4, "queued": 0,
#   "awaiting_human": 0,
#   "admitted": 1247, "rejected": 3, "shed": 0,
#   "max_rss_mb": 1024.0,     # ← the configured hard ceiling (0.0 when disabled)
# }

Per-tenant fairness

Cap concurrency per tenant so a single noisy tenant can’t monopolise every run slot.

The per-scope gate runs first and is the tighter, per-tenant ceiling; the global ceiling still bounds aggregate concurrency second. A tenant already at max_concurrent_runs_per_scope is queued (or shed by the declared overflow_policy) against its own slice — even when the global ceiling has free slots — so its burst can never consume another tenant’s capacity.

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

bot = BotOS(
    agent=agent,
    platforms=["telegram", "discord"],
    max_concurrent_runs=32,             # global ceiling
    max_concurrent_runs_per_scope=4,    # per-tenant cap within the ceiling
    queue_depth=128,
    overflow_policy="reject",
)
bot.start()

gateway:
  host: "127.0.0.1"
  port: 8765
  max_concurrent_runs: 32
  max_concurrent_runs_per_scope: 4
  queue_depth: 128
  overflow_policy: reject

# Set the per-scope sub-limit via gateway.yaml, then start:
praisonai gateway start --config gateway.yaml

How per-scope admission decides

With max_concurrent_runs=32 and max_concurrent_runs_per_scope=4:

1. Quiet tenant B sends a turn — its scope has 0 in-flight, well under 4, and the global count is under 32, so the turn is admitted immediately.
2. Noisy tenant A bursts 10 concurrent turns: the first 4 are admitted (they fill A’s per-scope slice); turns 5–10 are queued against A’s slice — even though the global ceiling still has 28 free slots.
3. Tenant B sends a second turn while A’s burst is queued: it is admitted immediately, because B’s per-scope count is 1 and the global ceiling has room. No starvation.
4. As A’s in-flight turns complete, its queued turns dispatch one at a time, keeping A at most 4 concurrent.

Admission during human approval

A turn parked on a human approval does not hold its slot — other turns can run while the approver decides.

No configuration is needed. With max_concurrent_runs=1, a single pending approval no longer blocks every other turn:

• Every approval backend — the slot is released while ApprovalRegistry.approve_sync / approve_async waits on the backend, so presentation (gateway), Slack, Telegram, Discord, HTTP and webhook approvals all behave the same.
• Sync and async approvals — async approvals flip the slot on the gate’s event loop directly; sync approvals running in a worker thread hop onto that loop without blocking it.
• Non-blocking resume — when the human decides (or the approval times out), the turn takes a free slot if one exists; otherwise it borrows one that the next released slot repays. A sync approval parked in a thread-pool worker therefore cannot deadlock against a turn waiting for that worker.
• Parallel approvals — every branch of a turn that requests several approvals at once (e.g. asyncio.gather) resumes already counted in in_flight.
• Cancellation-safe — cancelling a parked turn (/stop, turn timeout) never double-releases or leaks a slot. See Approval Turn Liveness.

Reading the counters

Best Practices

Related