Observability — superadmin notifications page

The page at /admin/notifications is the single place to diagnose notification-delivery failures without SSH-ing into the EC2 box. Visible only to users with isSuperAdmin: true.

It has five tabs, each backed by a RequireSuperAdmin-guarded GraphQL query that polls every 5 seconds.

Where the data comes from

Connection-level signals come from an in-memory SseConnectionRegistry living inside the backend process:

• connections: Map<sessionId, ConnectionInfo> — currently-open SSE sessions
• Ring buffer of the last 200 connect/disconnect events
• Ring buffer of the last 500 deliveries (notifications successfully written to a session response stream)

Notification history comes from a paginated DB read against the Notification table.

info

The registry is in-memory and per-instance. A backend restart zeros it — that's correct, because every browser also reconnects after a deploy. If we ever scale to >1 backend instance per environment, this will need to move to Redis. Documented as out-of-scope in the v1 plan.

Tab cookbook

Health (default tab)

Headline KPIs plus auto-detected anomalies. Open this tab first when triaging an incident.

KPI	Meaning
Active sessions	Currently-open SSE streams
Active users	Distinct users behind those sessions
Delivered last min / last hour	Count of notifications written to a stream
Created last hour	Count of Notification rows with createdAt > now-1h
Disconnects (5 min)	Total SSE closures in the last 5 minutes
Error disconnects	Subset where reason = SERVER_ERROR (iterator threw)
Stale unread (>1h)	Notification rows with status = UNREAD AND createdAt < now-1h

Anomaly banners appear automatically when:

Banner	Condition	Diagnosis
🔴 N server-error disconnects in 5 min	errorDisconnectsLast5Min > 0	The iterator is throwing — typically a Redis or pub/sub issue. Check backend logs.
🔴 N created but 0 delivered	notificationsCreatedLastHour > 0 && deliveriesLastHour == 0 && activeSessions > 0	Pub/sub channel is broken — notifications hit the DB but never get to subscribers. Check NotificationPubSubService connection and Redis health.
⚠️ N reconnecting in a storm pattern	Any user with ≥3 CONNECTs in the last minute	Client-side reconnect loop — token churn, auth flap, or a buggy client. Find the user in Live Connections and ask.
⚠️ N notifications still UNREAD with createdAt > 1h	staleUnreadCount > 100	Users may be missing notifications. Could be DND, broken email, or just a backlog. Cross-check with History + email send queue.
✅ No anomalies detected	none of the above	Healthy.

Live Connections

• "N sessions across M users" counters
• Per-user breakdown table (sessions count, oldest connection age)
• Per-session detail (IP, User-Agent, age)

Use this to answer: "Is user X actually connected right now?"

A session disappears within ~30 seconds of the tab closing — the heartbeat-driven error then req.on('close') propagates through the controller's teardown.

Connection Trail

Timeline of the last 200 CONNECT / DISCONNECT events with relative time, session-ID prefix, and (on DISCONNECT) duration + reason chip.

Marker	Meaning
🟢 CONNECT	Stream opened
⚫ DISCONNECT (no chip)	Normal close — reason = CLIENT_CLOSE, the user closed the tab or navigated away
🔴 DISCONNECT with red SERVER_ERROR chip	Iterator threw — investigate the backend

Reconnect storms show up here as alternating 🟢/⚫ rows for the same user every few seconds.

History

Paginated cross-user notification list (50 per page) with filters: status, category, priority, free-text search.

The SSE column cross-references the in-memory delivery ring buffer:

Cell	Meaning
✅	Delivered via SSE since the backend last restarted
—	No SSE delivery on record. Either the backend restarted (the buffer is gone), the user wasn't connected when it fired, or it failed to deliver

Rows where status = UNREAD and createdAt > 1h ago are highlighted in amber so you can spot users who've missed a notification.

Stats

The classic notification-statistics view (totals, today/this-week, read/click rates, status & delivery-channel breakdowns). Preserved unchanged from the previous single-view page.

GraphQL surface

All queries require JwtAuthGuard + SuperAdminGuard + @RequireSuperAdmin:

Query	Purpose
adminSseActiveConnections	Currently-open sessions
adminSseConnectionEvents(limit)	Connect/disconnect ring buffer
adminSseRecentDeliveries(limit)	Delivery ring buffer
adminNotificationsHealth	KPIs + anomaly inputs
adminNotificationsList(input)	Paginated DB history with filters

DTOs are in aiqlick-backend/src/admin-dashboard/dto/admin-sse-observability.output.ts and admin-notifications-list.input.ts.

Triage runbook

When a user reports "I didn't get a notification":

1. Health tab — any red banner? If yes, follow the diagnosis column above.
2. Live Connections — is the user there? If no, they weren't connected when the notification fired; SSE couldn't deliver. Email/push should have backstopped it. Check the user's notification preferences.
3. History tab — find their notification by user filter. Did the row exist? Is its SSE column ✅ or —?
4. Connection Trail — search for their email. Are there reconnect-storm rows? Or a recent SERVER_ERROR disconnect?

If all four come up clean and the user still claims no notification arrived, check the channel-specific failure paths:

• Email: Email.status in DB — FAILED/BOUNCED rows surface in /admin/system log explorer
• Browser-level OS notification: requires browser permission grant
• DND: see Notification preferences in the user's profile