TwoDot Engineering

Agent Observability: see what your coding agents are doing

Every Claude Code / Codex session on the team, streamed into shared Grafana dashboards — live sessions, per-machine activity, and day/week/month trends. One command to opt in.

hooks → collector.internal.twodot.ai → grafana.internal.twodot.ai

The pipeline · Six dashboards · Access & secrets · Installing the hook

Why

Agents do a lot of work we never see

🕶️

Invisible sessions

Dozens of Claude/Codex sessions a day across the team — no shared record of what ran, where, or for how long.

📈

No trend line

Is agent usage growing? Which projects get the most agent time? Which harness do we actually use? Nobody could answer.

🧯

No live view

When an agent is stuck waiting for a human, or rate-limited, that's invisible until you go look at the terminal it runs in.

The fix: the agents already emit lifecycle hooks for everything they do. We collect them centrally and dashboard them.

Architecture

The pipeline, end to end

Your machine

Claude Code + Codex hooks fire on every event (30 + 10 event types) and POST via collector.py

HTTPS + secret→

Collector

collector.internal.twodot.ai — sink on the shared GKE cluster, stores every event in Postgres

scrape + SQL→

Dashboards

Prometheus (live counts) + Grafana (six dashboards) + the control-plane web UI

Events carry host identity — X-Collector-Host / a stable machine id cached at ~/.collector/host-id — so every dashboard can break down per person/machine.
Hooks are best-effort and async: a dead network or collector never blocks or slows your agent.

Access

Four URLs, all locked

URL	What it is	Auth
grafana.internal.twodot.ai	The six dashboards	Google OAuth (@twodot.ai)
cacp.internal.twodot.ai	Control-plane web UI — live event log, sessions, transcripts, analytics	Google OAuth (@twodot.ai)
features.internal.twodot.ai	Feature Tracer — sessions → user stories → crewbase tickets	Google OAuth (@twodot.ai)
collector.internal.twodot.ai	Ingest endpoint your hooks POST to	shared secret header (401 without)

Nothing is anonymous: every UI — Grafana included — signs you in with your @twodot.ai Google account, and the collector rejects unsigned events. Manifests + full runbook: tools/experiments/stack/deploy.md in mono.

Dashboard 1 / 6

Agent Sessions — the lifecycle view

Stat tiles: Live (working right now), Open, Waiting for human, Rate limited, Ended.
Timeseries of every state over time — spot the daily rhythm, or a rate-limit storm.
In-flight tool spans: how many tools are mid-execution across all sessions.
Filter by $host — just your machine, or the whole team.

🫀

"Live" is honest

A session counts as live only with an open working turn AND recent tool activity (~90 s freshness window) — long-idle sessions age out instead of lying.

Dashboards 2–3 / 6

Live Agents & Harness Usage

Live Agents

The war-room view: only live and waiting-for-human sessions. Big numbers, glanceable on a shared screen — "is anything stuck on a permission prompt right now?"

Harness Usage

Claude Code vs Codex share: sessions per harness, live/active split per harness, in-flight tools per harness. The agent label sums back to the exact totals — one calculation, partitioned.

Both respect the $host filter — compare your own harness mix against the team's.

Dashboard 4 / 6

Hosts — who's running what, where

Hosts reporting — how many machines have ever sent events.
Live / open sessions by host — per-machine session activity over time.
Stored events per hour by host — ingest rate per machine.
Last event per host — a freshness table: seconds since each machine last reported. A hook install that broke shows up here first.

Host identity is deterministic: hostname + a hashed machine id, computed client-side and cached — the same machine always reports as the same host, even behind tunnels/NAT. Events from before host identity existed group under unknown.

Dashboard 5 / 6

Agent Trends — days, weeks, months

📅

Calendar bucketing, full history

SQL straight against Postgres (not Prometheus), so it covers everything ever ingested — pick $bucket = day / week / month, UTC calendar boundaries.

🔬

What it answers

Sessions per day by host & by harness · hook events by type · sessions per project (from the working directory) · this-week-vs-last-week · active machines per week.

The team-growth question — "is agent usage actually going up?" — is one glance at Sessions per week by host.

Dashboard 6 / 6

Feature Tracer trends — from sessions to shipped work

Sessions tracked per day/week by harness and host — the sessions that became tracked work.
Tickets created vs done per week — real close events (done_at), not last-modified noise.
Tickets by project — where the staged work concentrates.

Feature Tracer (features.internal.twodot.ai) turns a coding session into a user story and a crewbase ticket; when the session ends, linked tickets auto-close. These panels are its pulse.

Beyond Grafana

The control-plane UI has its own views

📡

Live event log

Every hook event streaming in over a WebSocket — sessions as lanes, states, prompts, git branch chips, transcript links.

📊

/dashboard

Built-in live tiles + history chart (reads Prometheus) — no Grafana login needed for a quick check.

🏆

/analytics

Leaderboards, streaks and achievement badges — longest sessions, busiest days, most-used projects. Gamified, but the numbers are real.

All at cacp.internal.twodot.ai, behind the Google login.

Opt in

Install the hook — one merge command

# from your mono checkout (template: PR #3461)
cd tools/experiments/hooks-mgr
python3 hooks_mgr.py merge --template templates/coding-agent-remote-collector.json \
    --var collector="$(cd ../coding-agent-control-plane && pwd)/collector.py" \
    --var collector_secret="$(kubectl get secret collector-auth -n coding-agent -o jsonpath='{.data.COLLECTOR_SECRET}' | base64 -d)"

Installs collection hooks for both Claude Code (~/.claude/settings.json) and Codex — 40 event types total.
Surgical merge: only our hook entries are touched; your other settings and hooks stay intact, and a .bak is written first.
No cluster access? Ask for the collector secret — it's the one non-guessable input.

Opt in

Designed to fail loudly, not silently

1

collector and collector_secret are required — hooks-mgr aborts on unresolved variables, so you cannot install hooks that would silently 401.

2

Preview first with diff instead of merge — exact changes, no writes.

3

Hooks run with --best-effort --timeout 5, async — an unreachable collector can never block a tool call or fail your session.

4

Opt out anytime: hooks_mgr.py remove with the same template strips only our entries.

Verify

Prove it's flowing in 60 seconds

1

Open a new Claude Code session and run any command.

2

cacp.internal.twodot.ai — your session appears in the live log within seconds.

3

grafana.internal.twodot.ai → Hosts (sign in with Google) — your machine shows in "last event per host" with a fresh timestamp.

⚠ Session doesn't appear?

401s mean a wrong/missing secret — re-run merge with the right collector_secret. Hooks only attach to new sessions, so start a fresh one. Your host reading unknown only affects events from before the install.

TwoDot Engineering

Questions?

hooks_mgr.py merge → grafana.internal.twodot.ai

Six dashboards. Every session. Per machine, per harness, per week. Docs: tools/experiments/stack/deploy.md · template PR #3461.