Gateway Memory-Pressure Cache Eviction

Soft-evict the coldest warm agent caches and cold live gateway sessions before the kernel OOM-kills the gateway.

One pure planner names the coldest idle memory to reclaim under pressure across two layers — warm per-session agent caches and live gateway sessions — so a busy gateway on a small host survives instead of being OOM-killed.

from praisonaiagents import Agent
from praisonai_bot.gateway import WebSocketGateway

# On a cgroup-aware Linux host with a session store (SQLite is the default),
# the gateway automatically soft-evicts the coldest idle sessions under memory
# pressure — both the agent's warm cache AND its live gateway session, before
# the OOM killer fires. No config needed.
gateway = WebSocketGateway()
gateway.register_agent(Agent(name="Support", instructions="Help users"))

The same guard runs behind a multi-channel BotOS deployment:

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

# A persistent session store (SQLite, Postgres) is what makes soft
# eviction rebuildable, so it must be configured for eviction to be active.
bot = BotOS(agent=agent, platforms=["telegram"])
bot.start()

Each evicted cache or session is transparently rebuilt from the persisted session store on its next turn or join — cheap and lossless, with no user-visible impact.

Two eviction layers, one planner

Two enactors share one decision core — the same pure planner and the same cgroup probe drive both.

Layer 1 (existing) sheds warm per-session agent caches — the chat history plus agent object held by BotSessionManager. Layer 2 (new in PR #5237) sheds live WebSocketGateway session objects — the GatewaySession with its pending inbox, protocol cursors, executing-turn flag, and event log. Both call the same pure planner (plan_pressure_evictions) with WarmSession facts, and both read the same CgroupMemoryPressure probe. The decision is central; the enactors are layer-local.

Which layer sheds first? On a running gateway both sweeps run on the housekeeping tick — the warm-cache one from BotSessionManager (inside BotOS) and the gateway-session one from WebSocketGateway._liveness_loop. Together they cover both memory sources.

Quick Start

1. On by default on cgroup-aware Linux hosts

   No knob to set. On a Linux host that can read a cgroup v1/v2 memory limit, the gateway samples RSS and soft-evicts the coldest rebuildable caches under pressure on its periodic tick. On a host that can’t report a cgroup limit (macOS/Windows, unconstrained host), the sweep is a no-op — legacy behaviour is preserved exactly.

   from praisonaiagents import Agent
   from praisonai.bots import BotOS
   
   agent = Agent(name="Support", instructions="Help users")
   bot = BotOS(agent=agent, platforms=["telegram"])
   bot.start()

2. Add a persistent store to make eviction rebuildable

   A cache is only evicted when its transcript is durably persisted, so it can be losslessly rebuilt on the next turn. A BotSessionManager reads/writes that store via its store= argument. Without a store, nothing is flushed, nothing is evicted, and the sweep is a no-op.

   from praisonaiagents.session.store import DefaultSessionStore
   from praisonai_bot.bots._session import BotSessionManager
   
   # A persistent store makes each warm cache rebuildable — the precondition
   # for soft eviction. See Gateway Session Persistence for store options.
   session_manager = BotSessionManager(store=DefaultSessionStore("./sessions"))

   For a full BotOS deployment, configure the store through gateway.yaml (session.persist: true) — see Gateway Session Persistence.

3. Trigger the sweep yourself

   Running the session manager standalone (no BotOS)? Call the enactor directly with the concrete cgroup probe:

   from praisonai_bot.gateway import CgroupMemoryPressure
   
   # session_manager is your BotSessionManager instance
   evicted = session_manager.sweep_under_pressure(CgroupMemoryPressure(), headroom=0.9)

Which primitive?

Two gateway primitives protect memory from opposite sides — this diagram picks the right one.

Use both together for full memory protection: admission gates new work; the planner reclaims from idle warm caches.

API Reference

Three symbols export from praisonaiagents.gateway:

from praisonaiagents.gateway import (
    WarmSession,
    MemoryPressureProtocol,
    plan_pressure_evictions,
)

WarmSession

A frozen dataclass carrying a pure fact from the running gateway to the planner.

MemoryPressureProtocol

A @runtime_checkable Protocol the gateway implements over its own process to report the container’s memory ceiling and current RSS.

plan_pressure_evictions

plan_pressure_evictions(
    budget_mb: Optional[float],
    rss_mb: float,
    warm_sessions: Sequence[WarmSession],
    *,
    headroom_ratio: float = 0.9,
) -> List[str]

Returns the session_ids to soft-evict, coldest (LRU) first, tie-broken by session_id (stable). It never touches the caches — the gateway is the enactor.

Returns [] when RSS is within budget, the budget is unknown (None / <= 0), the budget or RSS is non-finite (NaN/inf), or nothing evictable remains.

Using CgroupMemoryPressure

CgroupMemoryPressure is the concrete, stdlib-only MemoryPressureProtocol the gateway uses to read the real cgroup budget and anonymous RSS.

from praisonai_bot.gateway import CgroupMemoryPressure

probe = CgroupMemoryPressure()
probe.cgroup_limit_mb()  # -> float | None (MiB)
probe.anon_rss_mb()      # -> float (MiB)

A nested cgroup’s lower limit is preferred: cgroup_limit_mb() resolves the process’s own cgroup path from /proc/self/cgroup first, then falls back to the filesystem root. Every source degrades safely, so on a host with no cgroup limit the sweep is a no-op.

Wrapper-side wiring

BotSessionManager is the runtime enactor — it snapshots warm caches, calls the pure planner, and soft-evicts the losers.

warm_sessions

BotSessionManager.warm_sessions() -> List[WarmSession]

Snapshots the warm per-session caches as core WarmSession facts (LRU last_activity, in_flight, flushed). A cache is flushed only when a persistent store is configured and its last store write succeeded.

sweep_under_pressure

BotSessionManager.sweep_under_pressure(
    memory_pressure: MemoryPressureProtocol,
    headroom: float = 0.9,
) -> int

Reads the real cgroup budget/RSS from memory_pressure, calls plan_pressure_evictions, and soft-evicts the coldest rebuildable caches LRU-first — re-sampling RSS to shed the minimum. Returns the number evicted. A no-op (returns 0) when no cgroup limit is reported.

Live gateway sessions (WebSocketGateway._sweep_sessions_under_pressure)

The gateway’s own enactor sheds cold live GatewaySession objects before the kernel does — the second layer, new in PR #5237.

What it evicts. A GatewaySession in WebSocketGateway._sessions that is (a) not currently executing a turn, (b) has no live WebSocket bound to it, and (c) whose transcript is durably persisted. When shed it is closed with persist=True, dropped from the live set, and its resume_window TTL is stamped so the next join transparently rehydrates it from the store.

Guards — a session is skipped (never evicted) when:

Interaction with close_session(persist=True). The sweep does not call close_session directly — it uses a stricter helper (_evict_persisted_session) that only drops the in-memory session after the final store.add_message write returns truthy (or None, for legacy stores that don’t report).

Headroom. The gateway sweeper uses _pressure_headroom = 0.9 (matching the warm-cache default). It sheds until RSS ≤ headroom × budget, re-sampling anonymous RSS after each eviction and stopping as soon as it fits — never more (test_gateway_sheds_coldest_session_first_until_within_budget, test_gateway_noop_when_within_budget, test_gateway_noop_when_no_sessions).

How it works — live sessions

The gateway builds a candidate list of disconnected sessions, asks the planner for the LRU order, then persists-then-drops each victim while re-sampling RSS.

sequenceDiagram
    participant Loop as _liveness_loop tick
    participant Probe as CgroupMemoryPressure
    participant Planner as plan_pressure_evictions
    participant GW as WebSocketGateway
    participant Store as SessionStore

    Loop->>GW: _sweep_sessions_under_pressure()
    GW->>Probe: cgroup_limit_mb() + anon_rss_mb()
    Probe-->>GW: budget, rss
    GW->>GW: build WarmSession list<br/>(exclude bound sessions)
    GW->>Planner: budget, rss, warm
    Planner-->>GW: LRU-ordered session_ids
    loop for each victim
        GW->>GW: skip if executing / now bound
        GW->>Store: add_message(session_data)
        alt persisted OK
            GW->>GW: session.close() + stamp resume TTL
            GW->>Probe: re-sample anon_rss_mb()
            alt back within headroom × budget
                GW-->>Loop: stop
            end
        else raised / returned False
            GW-->>GW: retain in memory (no data loss)
        end
    end

The resume_window TTL stamped on each eviction is the persistence contract that keeps the session resumable — see test_gateway_evict_persisted_session_sets_resume_ttl.

Runs even when liveness is disabled

Since PR #5237, the housekeeping loop that drives this sweep no longer exits when LivenessConfig(enabled=False) / interval_ms=0 — only the PING/reap logic is gated. When liveness is disabled the loop falls back to the config’s default interval so the pressure sweep still fires. This preserves the OOM guard as a hard invariant that does not depend on any other feature being enabled.

See Gateway Liveness for how the always-on housekeeping tick composes with the ping/reap logic.

How It Works

The gateway samples its own memory, asks the planner for the order to evict, then enacts it — re-sampling RSS as it goes.

The planner returns the full ordered list, not a prefix — the gateway re-samples RSS as it evicts and stops as soon as RSS is back within headroom_ratio × budget (default 0.9). It is byte-agnostic by design: it never guesses per-cache sizes, so over-shedding is avoided by the enactor and under-shedding is caught on the next pass.

Guards — a victim is skipped entirely (never evicted) when:

Degraded modes — the planner never crashes the gateway it protects, returning [] when:

Configuration

The single tunable is headroom_ratio — the fraction of the budget the gateway sheds back down to.

from praisonaiagents.gateway import WarmSession, plan_pressure_evictions

warm = [WarmSession(session_id="a", last_activity=1.0)]

# 800 MiB is within 90% of 1000, but breaches 70% — drop the ratio to reclaim earlier.
assert plan_pressure_evictions(1000.0, 800.0, warm) == []
assert plan_pressure_evictions(1000.0, 800.0, warm, headroom_ratio=0.7) == ["a"]

User Interaction Flow

Warm agent caches (Layer 1)

A Fly machine with a 512 MiB cgroup limit hosts a Telegram bot that has accumulated 60 warm session caches. A burst pushes RSS to 490 MiB — over 0.9 × 512 ≈ 460 MiB.

1. CgroupMemoryPressure reads cgroup_limit_mb() = 512, anon_rss_mb() = 490.
2. plan_pressure_evictions names the 40 evictable sessions LRU-first (in-flight and unflushed ones are skipped).
3. sweep_under_pressure evicts down the list, re-sampling RSS, and stops at ~460 MiB.
4. Those users get their cache rebuilt transparently on their next turn.

No user-visible impact. No OOM kill.

Live WebSocket sessions (Layer 2)

A gateway with a 512 MiB cgroup limit is serving 80 concurrent WebSocket clients. 25 of them idle out (no live socket bound; transcripts already persisted to the SQLite store from ordinary turn completions). A tool-heavy burst on the remaining 55 pushes RSS to 490 MiB — over 0.9 × 512 ≈ 460 MiB.

1. On the next housekeeping tick, _sweep_sessions_under_pressure() reads cgroup_limit_mb() = 512, anon_rss_mb() = 490.
2. It builds a WarmSession list of the disconnected sessions only — bound sessions and executing turns are excluded.
3. plan_pressure_evictions returns the 25 evictable session IDs LRU-first.
4. _evict_persisted_session closes them one by one — each final snapshot written, then dropped from _sessions, with a resume_window TTL stamped so they remain resumable.
5. It re-samples RSS after each eviction and stops when RSS is back inside the target, well before it exhausts the 25 candidates.
6. When one of those users re-joins, the gateway rehydrates the session from the store transparently — no user-visible impact, no OOM kill, no dropped socket.

On each periodic tick the gateway already runs reap_stale; the new sweep sits alongside it and is a no-op when the host has no cgroup limit:

from praisonai_bot.gateway import CgroupMemoryPressure

probe = CgroupMemoryPressure()

# session_manager is your BotSessionManager instance
session_manager.reap_stale(max_age_seconds=3600)
session_manager.sweep_under_pressure(probe, headroom=0.9)

Common Patterns

Combine with MemoryPressurePolicy for full memory protection — admission gates new work, the planner reclaims from idle warm caches:

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users")

# Admission side: shed new turns under RSS pressure. Eviction side is on by
# default on cgroup-aware hosts — together they cover both memory sources.
bot = BotOS(agent=agent, platforms=["telegram"], max_rss_mb=512)
bot.start()

Custom MemoryPressureProtocol for non-Linux hosts — supply a macOS/Windows probe:

from praisonaiagents.gateway import MemoryPressureProtocol

class MacProbe:
    def cgroup_limit_mb(self):
        return 2048.0  # from your platform's memory limit
    def anon_rss_mb(self):
        return 512.0   # from psutil or a platform API

assert isinstance(MacProbe(), MemoryPressureProtocol)

Tune headroom_ratio for hosts with spiky per-turn allocations — drop to 0.8 to leave more headroom:

from praisonaiagents.gateway import WarmSession, plan_pressure_evictions

warm = [WarmSession(session_id="s", last_activity=1.0)]
victims = plan_pressure_evictions(1000.0, 850.0, warm, headroom_ratio=0.8)
assert victims == ["s"]

Best Practices

Related