Incident Response — Agentic MTTR Demo

INC178077835481225

P2pending_human_reviewai_assisted

users-db read replica lag exceeding 90s after migration

Verify agent flagged this as unresolved — responsible group: database-reliability
Pipeline Status
Context Assembly
Diagnose Agent
Severity / Routing Agent
Scenario Match
Remediate Agent
Confidence Gate
Gate Decisionauto executed
Execute Action
Verify Agent
Resolution
Assembled Context — bounded to last 30 minutes
Rolling logs
Recent commits
27fb08f chore(db): add index on users.email for lookup perf — a.chen
Similar past incidents
INC177635354140634 · database/P2 · 51d ago — users-db read replica lag exceeding 90s after migration
INC178026679020738 · database/P2 · 6d ago — users-db read replica lag exceeding 90s after migration
INC177780994415853 · database/P2 · 34d ago — users-db read replica lag exceeding 90s after migration
Diagnosis Agent
ran in parallel
The migration that added an index on users.email triggered a heavy backfill on the primary, overwhelming replication and causing the read replica to fall behind, resulting in >90 s lag and stale reads.
  • [users-db-migrations] applied migration 0047_add_index_users_email.sql
  • [users-db-replica-2] WARN replication lag 94s (threshold 30s)
  • [users-service] WARN stale read detected for user profile lookups
  • Similar past incidents (INC177635354140634, INC178026679020738, INC177780994415853) also reported replica lag after adding the same index
Assessed: P2 / database
Self-reported confidence: 92% (audit-only — not used by the gate)
openai/gpt-oss-120b · 1565ms
Severity / Routing Agent
ran in parallel
P2 / database
Routed to: database-reliability
The incident involves significant replication lag on the users-db following a schema migration, leading to observable stale reads in the users-service. This mirrors recurring issues tracked in historical incidents (INC177635354140634, INC178026679020738, INC177780994415853), which are consistently classified as P2 database issues. While critical, it is not a P1 because it is a partial degradation (read replicas) rather than total system failure.
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-lite-latest · 1340ms
Scenario Matcher
deterministic, pre-LLM
Replica lag after schema migration
Replica lag following a schema migration requires a DBA to assess migration impact before any automated action; do not act automatically.
Match score: 85% · Recommended action: escalate_to_human
Matched tags: replica lag, migration, replication
Remediate Agent
grounded by runbook match, pre-gate
Proposed action: escalate_to_human
team: database-reliability · reason: Replication lag exceeds threshold due to backfill from schema migration. · message: Replica lag >90 s after adding users.email index; requires DBA assessment. · serviceName: users-db · justification: Follow runbook recommendation to involve human DBA for safe remediation.
The root‑cause analysis shows a heavy backfill from the new email index causing the replica to fall behind. The matched runbook explicitly advises escalation to a human DBA for this scenario, and automated actions could worsen data consistency. Therefore we comply with the runbook and escalate to the database‑reliability team.
openai/gpt-oss-120b · 1692ms
Confidence Gate
auto_executed

Computed from 4 measurable signals — not the model's self-reported confidence.

Severity match rate100%
x
Runbook coverage85%
x
Data completeness100%
x
Recency93%
x
Computed confidence (v1)94%
Auto-execute threshold: 75%
Proposed action: escalate_to_human
Verify Agent
runs after execution
Not resolved · residual risk: medium
Follow-up: escalate_to_human
The action of escalating to a human DBA was successfully executed as per the runbook. The incident is not resolved because the read replica lag persists due to the ongoing heavy backfill process, which requires human intervention to throttle or optimize without compromising data integrity.
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-lite-latest · 854ms