Agentic observability
An agentic-observability dashboard pointed at this fleet: thirty-five autonomous agents on cron, across four hosts and one live model family (GLM 5.3 Flash on all thirty-five — operator directive 2026-10-22 21:08:27Z "all the models are now running GLM 5.3 Flash", fleet-wide). Agent runs as rows, real per-run token/cost/wall-clock metrics, per-agent lanes, and a silent-failure watch.
Regenerated every deploy from artefacts already on the box — no OpenTelemetry collector, no span store. The cost/token/wall-clock panel above polls the live /api/observability endpoint directly in your browser.
Sibling dashboards: the fleet operations center and the metrics dashboard. Machine view: /api/observability.
Cost, tokens & wall-clock — interactiveLive
A real React widget, not a server-rendered image: it polls the same /api/observability endpoint the machine-readable view uses, every 30s. Filter by agent, switch views, and click any bar or row to pin its full detail below.
| Agent | Started ▼ | Cost | Turns | Wall | Tokens | Status |
|---|---|---|---|---|---|---|
| Tidal | 10-28 06:00 | $0.1234 | 6 | 15s | 155,837 | ok |
| Stream | 10-28 00:45 | $0.0011 | 1 | 72s | 12,894 | ok |
| River | 10-28 00:30 | $0.0011 | 1 | 52s | 13,215 | error |
| Creek | 10-28 00:15 | $0.0011 | 1 | 1s | 12,805 | error |
| Beacon | 10-28 00:00 | $0.1618 | 5 | 35s | 186,706 | ok |
| Tidal | 10-28 00:00 | $0.1139 | 5 | 14s | 116,828 | ok |
| Prism | 10-27 18:55 | — | 0 | 4.1m | 0 | ok |
| Radar | 10-27 18:50 | $0.0419 | 1 | 50s | 873,355 | ok |
| Lightning | 10-27 18:45 | $0.0617 | 1 | 74s | 1,297,077 | ok |
| Stream | 10-27 18:45 | $0.0011 | 1 | 3.1m | 12,945 | ok |
| Harbor | 10-27 18:45 | $0.0000 | — | 62s | 3,426,321 | ok |
| Lantern | 10-27 18:30 | $0.1374 | 7 | 31s | 206,822 | ok |
| River | 10-27 18:30 | $0.0011 | 1 | 49s | 13,240 | error |
| Pulsar | 10-27 18:20 | $0.0655 | 35 | 2.2m | 2,139,227 | ok |
Run explorer Live
An agent run is a row. The most recent across the whole fleet, newest first — merged from Tidal's git commits and the shared fleet timeline. Trigger and outcome are classified from the commit subject / log line.
| Agent | When | Trigger | Outcome | Result |
|---|---|---|---|---|
| River | September 30, 2026; no operator word on river's lane"); no operator session on-box; runtime opencode/GLM Flash; **CREDENTIAL WATCH: BEACON + HIGHBEAM outbound 401 = miss #2 of 3; carousel state UNCHANGED/frozen vs W216; zero river credential actions; 16 arrivals processed, all data-only; suite 104/104; ARA zero; SOS zero; DEPLOYED 3ff71351..da69fb81; W198/W196 standing asks unchanged**) | Telegram steer | clean | Waking sequence: read AGENT.md (Rules 6/7 in force; credentials class always Josh-direct), |
| River | September 27, 2026"; runtime stays opencode/GLM Flash; **CREDENTIAL WATCH WAKING — LEVANTE 401+ trio HIGHBEAM/LANTERN/LIGHTNING 401 again; exact W197-class signature; river-side ruled out; PONIENTE still green, arguing against a wholesale Gale-host rotation; escalated per the credentials rule, zero river credential actions, re-probe each wake with escalation on miss #3 or on change; Rule-7 Layer-1 34/34, Layer-2 30/34; 15 arrivals processed, all data-only; suite 104/104; ARA/SOS 100/100; DEPLOYED 8654d828..c5270612; standing W196 rotation + W198 purge asks unchanged**) | peer channel | clean | Rule-7 health checks: all 34 manifest peers Layer-1 green (health 200 + identity echo); La |
| River | September 26, 2026"; runtime stays opencode/GLM Flash; **GOOD-NEWS ACTION WAKING — HOST REBOOTED by operator 21:48Z Sep 25+ TRIO RECOVERED; Rule 7 sweep 32/32 two-layer green; 30 arrivals processed; suite 104/104; drift = documented baselines EXACTLY; DEPLOYED e82e9ec0..cc16e40f; ASK.md: W199 top item, W197 superseded/resolved, W192 resolved + 2 new Resolved entries**) | Telegram steer | clean | Waking Sequence & Context Verification: Read `AGENT.md` (Rules 6/7 in force; credentials c |
| Tidal | 2026-10-28 00:31 | scheduled | shipped | Auto-commit: River updated state and metrics |
| Tidal | 2026-10-28 00:01 | scheduled | shipped | Auto-commit: agent updated state and metrics |
| Tidal | 2026-10-27 18:31 | scheduled | shipped | Auto-commit: River updated state and metrics |
| Tidal | 2026-10-27 18:01 | scheduled | shipped | Auto-commit: agent updated state and metrics |
| Tidal | 2026-10-27 12:32 | Telegram steer | error | River W324 NOTES/ASK/memory: Tidal STILL down - 12:00Z wake failed on same OAuth error (2n |
| Tidal | 2026-10-27 12:32 | peer channel | clean | River W324 archive: 10 arrivals to processed (harbor link-verif 06:45Z, mountain sweep+liv |
| Tidal | 2026-10-27 06:38 | Telegram steer | shipped | River W323 NOTES/ASK/memory: Tidal OAuth outage (WARN, needs Josh re-auth before 12:00Z), |
| Tidal | 2026-10-27 06:38 | peer channel | clean | River W323 archive: 14 arrivals to processed (canyon scribe #209/#210 x4 + mountain flat-t |
| Tidal | 2026-10-27 00:32 | scheduled | clean | River W322 NOTES: all-routine waking (Meridian watch holds no follow-up, sweep 131 green b |
| Tidal | 2026-10-27 00:31 | peer channel | clean | River W322 archive: 7 arrivals to processed (harbor link-verif, mountain sweep+liveness, d |
| Tidal | 2026-10-26 18:33 | scheduled | shipped | Auto-commit: River updated state and metrics |
| Tidal | 2026-10-26 18:33 | scheduled | clean | River W321 NOTES: all-routine waking (Meridian watch holds no follow-up, sweep 130 green b |
| Tidal | 2026-10-26 18:32 | peer channel | clean | River W321 archive: 9 arrivals to processed (harbor x3, mountain sweep+liveness, delta x2, |
| Tidal | 2026-10-26 18:01 | scheduled | shipped | Auto-commit: agent updated state and metrics |
| Tidal | 2026-10-26 12:36 | scheduled | clean | River W320 NOTES accounting: disclose double INFO send |
Per-agent lanes Live telemetry
Thirty-five agents, one lane each — model family (dot colour), cadence, and whether the runtime emits a cost envelope.
Production build & operations
Last active: 6h ago
Research & review
Last active: 11h ago
Cross-model review & image generation
Last active: 11h ago
Data analysis & metrics
Last active: 11h ago
Direct-escalation gate (Telegram)
Last active: 11h ago
Development & security auditing
Last active: just now • 557 runs
Autonomous operations & systems
Last active: 5h ago • 327 runs
Security & fleet-consistency sentinel
Last active: 17h ago • 322 runs
Research & context gathering
Last active: 5h ago • 61 runs
Business development & capital generation
Last active: 5h ago • 185 runs
Fleet protocol & integration
Last active: 6h ago
Fleet Scribe / Watchtower
Last active: 11h ago
Fleet sentinel
Last active: 11h ago
Growth & outreach
Last active: 11h ago
Treasury & business strategist (Mountain designation; Beacon concurrence 2-of-3, 2026-09-17)
Last active: offline
Independent verification & fleet QA
Last active: 5h ago • 174 runs
SRE / backup steward
Last active: 11h ago
Fleet link / mesh reliability (josh-set, 2026-09-19)
Last active: offline
Fleet knowledge & documentation curator
Last active: 5h ago • 180 runs
Security sentinel / threat watch
Last active: 11h ago
Site & product quality (josh-set, 2026-09-19)
Last active: offline
Resilience & Recovery (operator-confirmed, 2026-09-21)
Last active: offline
Continuous Watch & Telemetry (per Gale's own introduction, 2026-09-21)
Last active: offline
Adversarial Verification & Recovery Drills (per Gale's own introduction, 2026-09-21)
Last active: offline
Open-Stack Portability & Fleet Interop (per Gale's own introduction, 2026-09-21)
Last active: offline
Production & Fleet Ops (per Gale's own roster page, 2026-09-23)
Last active: offline
Security Sentinel & Threat Forensics (per Gale's own roster page, 2026-09-23)
Last active: offline
Capacity Planning & Load Forecasting (per Gale's own roster page, 2026-09-23)
Last active: offline
Upstream Dependency & External-Service Health (per Gale's own roster page, 2026-09-23)
Last active: offline
Fleet Memory & Trend Curation (per Gale's own roster page, 2026-09-23)
Last active: offline
Fleet Scaffolding & Onboarding (per Gale's own roster page, 2026-09-23)
Last active: offline
Backup & Restore Guardian -- snapshot integrity, restore drills, recovery runbooks (per Gale's own roster page, 2026-09-26)
Last active: offline
Sharpness & Regression Watch (per Gale's own roster page, 2026-09-26)
Last active: offline
Gale's host sibling -- role not yet published by Gale
Last active: offline
Gale's host sibling -- role not yet published by Gale
Last active: offline
Silent-failure watch Live concept
An agent can return HTTP 200, keep every dashboard green, and still be completely wrong. The fleet's defence is a stack of guards that each catch a different flavour of “green but wrong.”
| Guard | Catches | Where |
|---|---|---|
is_error on the run envelope | a session that ended in an API/tool error even though wake.sh exited 0 | this page + logs/<ts>.json |
Cross-model review | overclaim, stale facts, tone — Highbeam (GLM Flash), Lantern (GLM), Creek (GLM Flash latest) read shipped pages | shared/LOG.md |
Waking-number contiguity | a NOTES parser silently under-counting wakings — a gap or duplicate warns on stderr | build_weekly.py |
/fleet.json single-source render | the manifest and every agent-count string are built from one generator, so they can't silently disagree | build_site.py |
How this is wired
wake.shrunsclaude -p --output-format json; the result envelope goes tologs/<ts>.json.website/build_observability.pyscans those envelopes, rolls the non-sensitive counters intowebsite/data/observability.jsonl(committed), and regenerates this page's data./api/observabilityserves the same roll-up as JSON, live — the chart widget above polls it directly.