Gateway Memory-Pressure Cache Eviction

One pure planner names the coldest idle memory to reclaim under pressure across two layers — warm per-session agent caches and live gateway sessions — so a busy gateway on a small host survives instead of being OOM-killed.

from praisonaiagents import Agent
from praisonai_bot.gateway import WebSocketGateway

# On a cgroup-aware Linux host with a session store (SQLite is the default),
# the gateway automatically soft-evicts the coldest idle sessions under memory
# pressure — both the agent's warm cache AND its live gateway session, before
# the OOM killer fires. No config needed.
gateway = WebSocketGateway()
gateway.register_agent(Agent(name="Support", instructions="Help users"))

The same guard runs behind a multi-channel BotOS deployment:

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

# A persistent session store (SQLite, Postgres) is what makes soft
# eviction rebuildable, so it must be configured for eviction to be active.
bot = BotOS(agent=agent, platforms=["telegram"])
bot.start()

Each evicted cache or session is transparently rebuilt from the persisted session store on its next turn or join — cheap and lossless, with no user-visible impact.

MemoryPressurePolicy sheds new turns under pressure; this planner reclaims memory from idle warm caches. See Admission Control · Memory-aware admission for the sibling admission-side control.

Two eviction layers, one planner

Two enactors share one decision core — the same pure planner and the same cgroup probe drive both. Layer 1 (existing) sheds warm per-session agent caches — the chat history plus agent object held by BotSessionManager. Layer 2 (new in PR #5237) sheds live WebSocketGateway session objects — the GatewaySession with its pending inbox, protocol cursors, executing-turn flag, and event log. Both call the same pure planner (plan_pressure_evictions) with WarmSession facts, and both read the same CgroupMemoryPressure probe. The decision is central; the enactors are layer-local. Which layer sheds first? On a running gateway both sweeps run on the housekeeping tick — the warm-cache one from BotSessionManager (inside BotOS) and the gateway-session one from WebSocketGateway._liveness_loop. Together they cover both memory sources.

Quick Start

On by default on cgroup-aware Linux hosts

No knob to set. On a Linux host that can read a cgroup v1/v2 memory limit, the gateway samples RSS and soft-evicts the coldest rebuildable caches under pressure on its periodic tick. On a host that can’t report a cgroup limit (macOS/Windows, unconstrained host), the sweep is a no-op — legacy behaviour is preserved exactly.

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")
bot = BotOS(agent=agent, platforms=["telegram"])
bot.start()

Add a persistent store to make eviction rebuildable

A cache is only evicted when its transcript is durably persisted, so it can be losslessly rebuilt on the next turn. A BotSessionManager reads/writes that store via its store= argument. Without a store, nothing is flushed, nothing is evicted, and the sweep is a no-op.

from praisonaiagents.session.store import DefaultSessionStore
from praisonai_bot.bots._session import BotSessionManager

# A persistent store makes each warm cache rebuildable — the precondition
# for soft eviction. See Gateway Session Persistence for store options.
session_manager = BotSessionManager(store=DefaultSessionStore("./sessions"))

For a full BotOS deployment, configure the store through gateway.yaml (session.persist: true) — see Gateway Session Persistence.

Trigger the sweep yourself

Running the session manager standalone (no BotOS)? Call the enactor directly with the concrete cgroup probe:

from praisonai_bot.gateway import CgroupMemoryPressure

# session_manager is your BotSessionManager instance
evicted = session_manager.sweep_under_pressure(CgroupMemoryPressure(), headroom=0.9)

Which primitive?

Two gateway primitives protect memory from opposite sides — this diagram picks the right one. Use both together for full memory protection: admission gates new work; the planner reclaims from idle warm caches.

API Reference

Three symbols export from praisonaiagents.gateway:

from praisonaiagents.gateway import (
    WarmSession,
    MemoryPressureProtocol,
    plan_pressure_evictions,
)

`WarmSession`

A frozen dataclass carrying a pure fact from the running gateway to the planner.

Field	Type	Default	Description
`session_id`	`str`	(required)	The cache key to soft-evict.
`last_activity`	`float`	`0.0`	Monotonic/epoch seconds of the session’s last turn; the planner evicts the coldest (smallest `last_activity`) first.
`in_flight`	`bool`	`False`	`True` while a turn is executing — never evicted (would abort live work).
`flushed`	`bool`	`True`	`True` when the transcript is durably persisted — only a flushed cache is rebuildable, so an unflushed one is never evicted (would lose data).

`MemoryPressureProtocol`

A @runtime_checkable Protocol the gateway implements over its own process to report the container’s memory ceiling and current RSS.

Method	Returns	Description
`cgroup_limit_mb()`	`Optional[float]`	The container memory limit in MiB (from the cgroup v1/v2 memory limit), or `None` if unknown.
`anon_rss_mb()`	`float`	Current anonymous (non-reclaimable) RSS in MiB.

`plan_pressure_evictions`

plan_pressure_evictions(
    budget_mb: Optional[float],
    rss_mb: float,
    warm_sessions: Sequence[WarmSession],
    *,
    headroom_ratio: float = 0.9,
) -> List[str]

Returns the session_ids to soft-evict, coldest (LRU) first, tie-broken by session_id (stable). It never touches the caches — the gateway is the enactor.

Parameter	Type	Default	Description
`budget_mb`	`Optional[float]`	(required)	The container memory budget in MiB (typically the cgroup limit).
`rss_mb`	`float`	(required)	The current anonymous RSS in MiB.
`warm_sessions`	`Sequence[WarmSession]`	(required)	The warm-cache registry to consider.
`headroom_ratio`	`float`	`0.9`	Fraction of the budget to shed back down to. Clamped to `(0.0, 1.0]`; non-finite / out-of-range values fall back to `0.9`.

Returns [] when RSS is within budget, the budget is unknown (None / <= 0), the budget or RSS is non-finite (NaN/inf), or nothing evictable remains.

Using `CgroupMemoryPressure`

CgroupMemoryPressure is the concrete, stdlib-only MemoryPressureProtocol the gateway uses to read the real cgroup budget and anonymous RSS.

from praisonai_bot.gateway import CgroupMemoryPressure

probe = CgroupMemoryPressure()
probe.cgroup_limit_mb()  # -> float | None (MiB)
probe.anon_rss_mb()      # -> float (MiB)

Method	Reads	Unavailable degrades to
`cgroup_limit_mb()`	cgroup v2 `memory.high` / `memory.max`, then cgroup v1 `memory.limit_in_bytes`	`None`
`anon_rss_mb()`	`RssAnon` from `/proc/self/status`	`0.0`

A nested cgroup’s lower limit is preferred: cgroup_limit_mb() resolves the process’s own cgroup path from /proc/self/cgroup first, then falls back to the filesystem root. Every source degrades safely, so on a host with no cgroup limit the sweep is a no-op.

Wrapper-side wiring

BotSessionManager is the runtime enactor — it snapshots warm caches, calls the pure planner, and soft-evicts the losers.

`warm_sessions`

BotSessionManager.warm_sessions() -> List[WarmSession]

Snapshots the warm per-session caches as core WarmSession facts (LRU last_activity, in_flight, flushed). A cache is flushed only when a persistent store is configured and its last store write succeeded.

`sweep_under_pressure`

BotSessionManager.sweep_under_pressure(
    memory_pressure: MemoryPressureProtocol,
    headroom: float = 0.9,
) -> int

Reads the real cgroup budget/RSS from memory_pressure, calls plan_pressure_evictions, and soft-evicts the coldest rebuildable caches LRU-first — re-sampling RSS to shed the minimum. Returns the number evicted. A no-op (returns 0) when no cgroup limit is reported.

Parameter	Type	Default	Description
`memory_pressure`	`MemoryPressureProtocol`	(required)	The probe reporting `cgroup_limit_mb()` and `anon_rss_mb()` — typically `CgroupMemoryPressure()`.
`headroom`	`float`	`0.9`	Shed until RSS is back within `headroom × budget`. Passed through to the planner’s `headroom_ratio`.

Live gateway sessions (`WebSocketGateway._sweep_sessions_under_pressure`)

The gateway’s own enactor sheds cold live GatewaySession objects before the kernel does — the second layer, new in PR #5237. What it evicts. A GatewaySession in WebSocketGateway._sessions that is (a) not currently executing a turn, (b) has no live WebSocket bound to it, and (c) whose transcript is durably persisted. When shed it is closed with persist=True, dropped from the live set, and its resume_window TTL is stamped so the next join transparently rehydrates it from the store. Guards — a session is skipped (never evicted) when:

Guard	Why	Test
No session store configured	The transcript isn’t rebuildable, so evicting would lose data. Sweep returns `0` — a no-op.	`test_gateway_noop_without_session_store`
No cgroup limit reported (`cgroup_limit_mb() is None`)	The host isn’t running under a memory ceiling; today’s behaviour is preserved exactly.	`test_gateway_noop_when_no_cgroup_budget`
`_is_executing=True` on the session	An executing turn must not be aborted. Re-checked at enact time.	`test_gateway_never_evicts_executing_session`
Live WebSocket bound (`session_id ∈ _client_sessions.values()`)	Ordinary `message` frames look the session up in `_sessions` directly and would be silently dropped (only `hello`/`join` rehydrate) — evicting a bound session would strand its client. Re-checked at enact time.	`test_gateway_never_evicts_session_with_live_client`, `test_gateway_sheds_disconnected_but_keeps_bound_session`
Failed final persist (`store.add_message` raises or returns `False`)	Under pressure the live session may hold the only copy of recent messages/events. If persisting the final snapshot fails, the session is retained in memory and the sweep counts it as un-shed.	`test_gateway_keeps_session_when_persist_fails`, `test_gateway_keeps_session_when_store_returns_false`

Interaction with close_session(persist=True). The sweep does not call close_session directly — it uses a stricter helper (_evict_persisted_session) that only drops the in-memory session after the final store.add_message write returns truthy (or None, for legacy stores that don’t report).

_evict_persisted_session is stricter than the ordinary close_session(persist=True) path. close_session best-effort persists but still drops the in-memory object even when the store write fails; under memory pressure that would be data loss, because the live session may hold the only copy of recent messages. The pressure enactor therefore retains the session in memory on any failed persist — a raised exception or a falsy add_message return — and counts it as un-shed.

Headroom. The gateway sweeper uses _pressure_headroom = 0.9 (matching the warm-cache default). It sheds until RSS ≤ headroom × budget, re-sampling anonymous RSS after each eviction and stopping as soon as it fits — never more (test_gateway_sheds_coldest_session_first_until_within_budget, test_gateway_noop_when_within_budget, test_gateway_noop_when_no_sessions).

How it works — live sessions

The gateway builds a candidate list of disconnected sessions, asks the planner for the LRU order, then persists-then-drops each victim while re-sampling RSS. The resume_window TTL stamped on each eviction is the persistence contract that keeps the session resumable — see test_gateway_evict_persisted_session_sets_resume_ttl.

Runs even when liveness is disabled

Since PR #5237, the housekeeping loop that drives this sweep no longer exits when LivenessConfig(enabled=False) / interval_ms=0 — only the PING/reap logic is gated. When liveness is disabled the loop falls back to the config’s default interval so the pressure sweep still fires. This preserves the OOM guard as a hard invariant that does not depend on any other feature being enabled. See Gateway Liveness for how the always-on housekeeping tick composes with the ping/reap logic.

How It Works

The gateway samples its own memory, asks the planner for the order to evict, then enacts it — re-sampling RSS as it goes. The planner returns the full ordered list, not a prefix — the gateway re-samples RSS as it evicts and stops as soon as RSS is back within headroom_ratio × budget (default 0.9). It is byte-agnostic by design: it never guesses per-cache sizes, so over-shedding is avoided by the enactor and under-shedding is caught on the next pass. Guards — a victim is skipped entirely (never evicted) when:

Guard	Why
`in_flight=True`	An executing turn must not be aborted. Re-checked at enact time — a turn may have started against a planned victim between the plan and the enact.
`flushed=False`	An unflushed transcript is not yet rebuildable from the store — evicting it would lose data. `flushed=True` requires a configured store and a successful last write; a caught store-write failure clears the marker, so the sweep skips the cache.

Degraded modes — the planner never crashes the gateway it protects, returning [] when:

Input	Result
`budget_mb=None`	`[]` (host can’t report a cgroup limit — never soft-evicts)
`budget_mb <= 0`	`[]`
non-finite `budget`/`rss` (NaN/inf)	`[]` (a NaN would otherwise slip past the threshold and evict every cache)
non-numeric `budget`/`rss`	`[]`
non-finite / non-numeric `headroom_ratio`	falls back to `0.9`

Configuration

The single tunable is headroom_ratio — the fraction of the budget the gateway sheds back down to.

Option	Type	Default	Description
`headroom_ratio`	`float`	`0.9`	Shed until RSS is back within `headroom_ratio × budget`. Clamped to `(0.0, 1.0]`; non-finite or out-of-range values fall back to `0.9`.

from praisonaiagents.gateway import WarmSession, plan_pressure_evictions

warm = [WarmSession(session_id="a", last_activity=1.0)]

# 800 MiB is within 90% of 1000, but breaches 70% — drop the ratio to reclaim earlier.
assert plan_pressure_evictions(1000.0, 800.0, warm) == []
assert plan_pressure_evictions(1000.0, 800.0, warm, headroom_ratio=0.7) == ["a"]

User Interaction Flow

Warm agent caches (Layer 1)

A Fly machine with a 512 MiB cgroup limit hosts a Telegram bot that has accumulated 60 warm session caches. A burst pushes RSS to 490 MiB — over 0.9 × 512 ≈ 460 MiB.

CgroupMemoryPressure reads cgroup_limit_mb() = 512, anon_rss_mb() = 490.
plan_pressure_evictions names the 40 evictable sessions LRU-first (in-flight and unflushed ones are skipped).
sweep_under_pressure evicts down the list, re-sampling RSS, and stops at ~460 MiB.
Those users get their cache rebuilt transparently on their next turn.

No user-visible impact. No OOM kill.

Live WebSocket sessions (Layer 2)

A gateway with a 512 MiB cgroup limit is serving 80 concurrent WebSocket clients. 25 of them idle out (no live socket bound; transcripts already persisted to the SQLite store from ordinary turn completions). A tool-heavy burst on the remaining 55 pushes RSS to 490 MiB — over 0.9 × 512 ≈ 460 MiB.

On the next housekeeping tick, _sweep_sessions_under_pressure() reads cgroup_limit_mb() = 512, anon_rss_mb() = 490.
It builds a WarmSession list of the disconnected sessions only — bound sessions and executing turns are excluded.
plan_pressure_evictions returns the 25 evictable session IDs LRU-first.
_evict_persisted_session closes them one by one — each final snapshot written, then dropped from _sessions, with a resume_window TTL stamped so they remain resumable.
It re-samples RSS after each eviction and stops when RSS is back inside the target, well before it exhausts the 25 candidates.
When one of those users re-joins, the gateway rehydrates the session from the store transparently — no user-visible impact, no OOM kill, no dropped socket.

On each periodic tick the gateway already runs reap_stale; the new sweep sits alongside it and is a no-op when the host has no cgroup limit:

from praisonai_bot.gateway import CgroupMemoryPressure

probe = CgroupMemoryPressure()

# session_manager is your BotSessionManager instance
session_manager.reap_stale(max_age_seconds=3600)
session_manager.sweep_under_pressure(probe, headroom=0.9)

Common Patterns

Combine with MemoryPressurePolicy for full memory protection — admission gates new work, the planner reclaims from idle warm caches:

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

# Admission side: shed new turns under RSS pressure. Eviction side is on by
# default on cgroup-aware hosts — together they cover both memory sources.
bot = BotOS(agent=agent, platforms=["telegram"], max_rss_mb=512)
bot.start()

Custom MemoryPressureProtocol for non-Linux hosts — supply a macOS/Windows probe:

from praisonaiagents.gateway import MemoryPressureProtocol

class MacProbe:
    def cgroup_limit_mb(self):
        return 2048.0  # from your platform's memory limit
    def anon_rss_mb(self):
        return 512.0   # from psutil or a platform API

assert isinstance(MacProbe(), MemoryPressureProtocol)

Tune headroom_ratio for hosts with spiky per-turn allocations — drop to 0.8 to leave more headroom:

from praisonaiagents.gateway import WarmSession, plan_pressure_evictions

warm = [WarmSession(session_id="s", last_activity=1.0)]
victims = plan_pressure_evictions(1000.0, 850.0, warm, headroom_ratio=0.8)
assert victims == ["s"]

Best Practices

Never re-implement the planner in an adapter

Call plan_pressure_evictions and enact the returned list. The decision is pure and unit-tested in isolation — a re-implementation drifts from the guards and degraded-mode returns.

The planner is byte-agnostic — re-sample RSS as you evict

It returns the full LRU-ordered list, not a prefix, and never guesses per-cache sizes. Evict down the list and stop as soon as RSS is back within headroom_ratio × budget. Never guess how much each cache will free.

Never evict in_flight=True or flushed=False

The planner already skips them — an executing turn would be aborted, and an unflushed transcript isn’t rebuildable yet. Do not override the guards in your enactor.

Trust the degraded-mode returns

An empty list means “no work to do” — RSS within budget, unknown/<= 0 budget, non-finite inputs, or nothing evictable. It never means the planner is broken; it never crashes the gateway it protects.

Admission Control · Memory-aware

The admission-side sibling — sheds new turns under RSS pressure with MemoryPressurePolicy

Gateway Reliability Presets

One switch that composes the gateway’s protective primitives

Gateway Session Persistence

Why evicted caches can be losslessly rebuilt — an unflushed transcript is never evicted

Gateway Graceful Drain

The other “avoid dropping live sessions” primitive

Gateway Liveness

Housekeeping tick that drives the sweep — runs even when disabled

Gateway Session Continuity

What survives an eviction — the resume path

Core planner introduced in PraisonAI PR #3805 (fixes #3804). Wrapper-side wiring (CgroupMemoryPressure, BotSessionManager.warm_sessions / sweep_under_pressure, per-turn in-flight and flushed guards) added in PR #4296 (fixes #4294). Live-session shedding (WebSocketGateway._sweep_sessions_under_pressure, bound-client guard, failed-persist retention, always-on housekeeping tick) added in PR #5237 (fixes #5153).

​Two eviction layers, one planner

​Quick Start

​Which primitive?

​API Reference

​WarmSession

​MemoryPressureProtocol

​plan_pressure_evictions

​Using CgroupMemoryPressure

​Wrapper-side wiring

​warm_sessions

​sweep_under_pressure

​Live gateway sessions (WebSocketGateway._sweep_sessions_under_pressure)

​How it works — live sessions

​Runs even when liveness is disabled

​How It Works

​Configuration

​User Interaction Flow

​Warm agent caches (Layer 1)

​Live WebSocket sessions (Layer 2)

​Common Patterns

​Best Practices

​Related

Admission Control · Memory-aware

Gateway Reliability Presets

Gateway Session Persistence

Gateway Graceful Drain

Gateway Liveness

Gateway Session Continuity

Two eviction layers, one planner

Quick Start

Which primitive?

API Reference

`WarmSession`

`MemoryPressureProtocol`

`plan_pressure_evictions`

Using `CgroupMemoryPressure`

Wrapper-side wiring

`warm_sessions`

`sweep_under_pressure`

Live gateway sessions (`WebSocketGateway._sweep_sessions_under_pressure`)

How it works — live sessions

Runs even when liveness is disabled

How It Works

Configuration

User Interaction Flow

Warm agent caches (Layer 1)

Live WebSocket sessions (Layer 2)

Common Patterns

Best Practices

Related