Degraded Channel Isolation

One channel's missing credential no longer takes the whole gateway down — that channel is isolated as degraded, the others keep serving

One channel’s unavailable credential isolates just that channel as degraded — every healthy channel keeps serving. The same surface now covers two cases: a token that was unavailable at boot (this page), and a token the platform rejected at runtime (see Runtime credential rejection).

python
from praisonaiagents import Agent

agent = Agent(name="assistant", instructions="Be helpful.")
# Rotate Slack's token overnight — Telegram + Discord keep serving; Slack is
# marked degraded until the new token is available.
agent.start("Anyone there?")
Diagram

Channel Validation

token available

configured-but-unavailable

📨 Telegram

🔐 Resolve
secret

📨 Slack

📨 Discord

✅ Serves traffic

⚠️ Empty token
no raise

⚙️ Runtime skips
no token → skip

🛡️ health: degraded

Before this change one channel’s ValueError aborted the whole GatewayConfigSchema, taking every channel offline. Now the schema returns an empty token for the unavailable credential, the runtime’s pre-existing “no token → skip” path isolates that one channel, and health() reports it as degraded.

Token-free-by-design channels

These channels legitimately run with no token: and are never marked degraded for the missing credential:

ChannelWhy token-free
emailSMTP/IMAP credentials live in their own config fields, not a token:
agentmailAI-native inbox with its own auth surface
signalTalks to a locally linked device, not a bearer token
localOwner-trusted terminal; nothing to authenticate against

What Changed

A single channel’s unavailable credential is isolated; structural errors and the gateway’s own ingress secret still fail closed.

ScenarioBeforeAfter
Single channel token env var unset at bootGatewayConfigSchema raised; all channels deadOnly that channel is skipped and marked degraded; siblings keep serving
{source: file, id: /run/secrets/x} file empty at bootSame — whole gateway downSame isolation as above
Structurally invalid config (missing platform:)Fails closedStill fails closed (unchanged)
Gateway’s own ingress / auth secret unavailableFails closedStill fails closed (unchanged)
Preflight probe running (--preflight, default)Hard-abort on failureIsolate the failing channel and serve every healthy channel — changed by PR #4863. The CLI start-gate now defers to the same degradation contract: a bad token no longer takes healthy channels offline. Fails closed only when --strict-preflight / gateway.preflight.strict: true is set, or when no channel is left serviceable
Token accepted at boot, rejected mid-flight (revoked/rotated/expired → 401/403)Whole channel went to FAILED; required a process restart to try a new tokenIsolated as CREDENTIAL_UNAVAILABLE (degraded); reconnect() / hot-reload picks up the fresh token without a restart. See Channel Supervision.

The single most important distinction: a channel credential fails open (isolated, degraded); the gateway’s own config and ingress secret fail closed (abort).

Diagram

A channel’s
token / app_token / verify_token

Structural config
missing platform, bad type

Gateway ingress /
auth secret

What is
unavailable?

🛡️ Fail open
isolate that channel
siblings keep serving

🛑 Fail closed
exit 78


How to Observe the Degraded State

The degraded channel is visible in the startup log, gateway doctor, channels --probe, and GET /health — no value is ever printed.

Startup log

One redacted WARNING is logged per degraded channel. An unset ${ENV} token logs:

Channel token env var 'SLACK_BOT_TOKEN' not set — channel will be skipped (degraded). Set it with: export SLACK_BOT_TOKEN=your_token

A {source, id} reference that resolves to configured-but-unavailable / missing logs:

Channel secret configured-unavailable — channel will be skipped (degraded): env var 'SLACK_BOT_TOKEN' empty

The token value is never printed — resolved secrets are registered for redaction.

praisonai gateway doctor

The secrets block reports configured-but-unavailable or missing for the affected field.

bash
$ praisonai gateway doctor
Credential availability (values never shown):
telegram     token         ✓  available
slack        token         ✓  available
slack        app_token     ✗  configured-but-unavailable

See Credential availability (without revealing values) for the full table.

praisonai gateway channels --probe

The channel listing surfaces the same per-channel verdict — a one-line probe you can run against a live gateway.

GET /health on the running gateway

The degraded channel appears under channels, distinguished from a running channel by status: "degraded" and a reason.

json
{
  "status": "healthy",
  "channels": {
    "telegram": { "platform": "telegram", "running": true },
    "discord":  { "platform": "discord",  "running": true },
    "slack":    {
      "platform": "slack",
      "running": false,
      "status": "degraded",
      "reason": "credential unavailable"
    }
  }
}

The same degraded channel also appears in the unified degraded_owners list, alongside any degraded provider, capability, or route — see Unified Degraded-Capability Registry.

json
{
  "degraded_owners": [
    {
      "owner_kind": "channel",
      "owner_id": "slack",
      "state": "cold",
      "reason": "credential unavailable",
      "retry_hint": "praisonai gateway doctor --fix"
    }
  ]
}

A degraded channel stays queryable in health() — it does not silently vanish, so a monitor can tell “configured-but-unavailable” apart from “never configured”.


How to Recover

  1. Fix the credential

    Rotate the env var, refill the mounted secret file, or restore the secret manager so the token resolves to available again.

  2. Reload the config

    An empty credential is resolved once at config load, so praisonai gateway reconnect <channel> alone does not re-read it — reconnect drives the channel supervisor, not the secret resolver. Trigger a config reload (or restart) so the schema re-resolves the now-available token. See Channel Supervision for reconnect/pause/resume.

  3. Confirm it recovered

    On the next config load, a channel that previously degraded but now has a token is removed from the degraded set automatically. Verify with praisonai gateway doctor or GET /health.


When You Do NOT Want This Behaviour

Some operators prefer strict fail-closed — enforce it at the operator layer, not the schema.

Preflight (--preflight, the default) now applies the same isolation contract at start-time: a channel with a bad credential is reported, skipped, and parked in CREDENTIAL_UNAVAILABLE while every healthy channel keeps serving. Preflight fails closed only when the operator opts into strict mode (--strict-preflight or gateway.preflight.strict: true) or when no channel is left serviceable (nothing to serve). Operators who want the pre-PR-#4863 behaviour — one bad token aborts everything — should opt into strict mode explicitly.

There is intentionally no knob to make the schema raise again — that behaviour was removed precisely because it took healthy channels down. To gate a deploy on strict availability, fail the release if praisonai gateway doctor --json reports any configured-but-unavailable or missing:

bash
praisonai gateway doctor --config gateway.yaml --json > doctor.json
python -c "import json,sys; d=json.load(open('doctor.json')); \
  bad=[(c,f,v) for c,fs in d.get('secrets',{}).items() for f,v in fs.items() if v not in ('available','configured')]; \
  sys.exit(1) if bad else None"

Strict opt-in

Opt back into fail-closed preflight when one bad token should abort the whole gateway. Set it in YAML for every start, or pass the flag for a one-off.

yaml
# gateway.yaml — opt back into fail-closed preflight
gateway:
  preflight:
    strict: true
bash
# One-off override without touching YAML
praisonai gateway start --config gateway.yaml --strict-preflight

--strict-preflight / --no-strict-preflight overrides the YAML strict key for that run. Even without strict mode, preflight still fails closed when no channel is left serviceable.


Best Practices

Gate deploys on praisonai gateway doctor –json

Read the secrets block and fail the deploy if any field is not available (or configured for exec sources). This gives you strict fail-closed at release time while keeping the resilient fail-open behaviour at runtime.

Use {source: file} for K8s / Docker secrets

A mounted secret file survives a container restart; a process env var may not. Cross-link: Secret References. File-sourced credentials keep resolving after a restart, so a channel recovers on the next config load without manual intervention.

Alert on the degraded WARNING log line

The channel will be skipped (degraded) WARNING is your operator canary. Wire it into your alerting so a rotated-but-not-yet-refilled credential pages you before users notice the channel is quiet.

Never rely on preflight alone for continuous safety

Preflight only checks credentials at start. A token can go stale after preflight passes — expiry, rotation, a secret-store blip. Degraded-channel isolation is what keeps the other channels serving when that happens.


Related