Incident Response — Agentic MTTR Demo

INC17838786812406

P2resolvedai_assisted

search-api request queue depth climbing under traffic spike

Pipeline Status
Context Assembly
Diagnose Agent
Severity / Routing Agent
Scenario Match
Remediate Agent
Confidence Gate
Gate Decisionauto executed
Execute Action
Verify Agent
Resolution
Assembled Context — bounded to last 30 minutes
Rolling logs
Recent commits
(none in lookback window)
Similar past incidents
INC177882654715913 · application/P2 · 58d ago — search-api request queue depth climbing under traffic spike
INC17821165852576 · application/P2 · 20d ago — search-api request queue depth climbing under traffic spike
INC178038577873729 · application/P2 · 40d ago — search-api request queue depth climbing under traffic spike
Diagnosis Agent
ran in parallel
Autoscaler hit its configured maximum (6 instances) during a traffic spike, preventing additional search‑api instances from being launched and causing the request queue to grow, leading to high latency.
  • [search-api] INFO requests/sec 4200 (baseline 1100)
  • [search-api] WARN queue depth 3800, p99 latency 2100ms
  • [autoscaler] current instances 6/6 (max reached)
Assessed: P2 / application
Self-reported confidence: 90% (audit-only — not used by the gate)
openai/gpt-oss-120b · 2093ms
Severity / Routing Agent
ran in parallel
P2 / application
Routed to: search-api-team
The search-api is experiencing significant latency degradation and queue buildup due to a traffic spike that has hit the maximum instance count. This is a partial outage affecting performance for users, consistent with previous incidents of this nature. It is classified as an application-level scaling issue.
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-lite-latest · 961ms
Scenario Matcher
deterministic, pre-LLM
Traffic spike exceeding capacity
Sustained traffic above provisioned capacity with healthy error rates is resolved by scaling out instances rather than restarting or rolling back.
Match score: 79% · Recommended action: scale_up
Matched tags: traffic spike, queue depth
Remediate Agent
grounded by runbook match, pre-gate
Proposed action: scale_up
team: search-api-team · reason: Autoscaler hit configured maximum (6) causing queue buildup and high latency · message: Scaling up search-api instances to handle traffic spike and reduce queue depth · serviceName: search-api · justification: Increase instance count to 10 to provide capacity for sustained traffic above current limit · targetInstances: 10
The diagnosis shows the autoscaler is capped at 6 instances, leading to a growing request queue and high latency. The matched runbook also recommends scaling out. Scaling up to a higher instance count directly addresses the capacity shortfall without altering code or restarting services.
openai/gpt-oss-120b · 1211ms
Confidence Gate
auto_executed

Computed from 4 measurable signals — not the model's self-reported confidence.

Severity match rate100%
x
Runbook coverage79%
x
Data completeness67%
x
Recency78%
x
Computed confidence (v1)84%
Auto-execute threshold: 75%
Proposed action: scale_up
#search-api-team — Slack-style FYI notification
in-app simulated

Auto-executed "scale_up" for "search-api request queue depth climbing under traffic spike" — confidence 84% cleared the auto-execute threshold, no human review required. Execution succeeded.

Already executed — nothing to approve or cancel. Posted here purely so the trail isn't audit-log-only.

Verify Agent
runs after execution
Resolved · residual risk: low
Follow-up: notify_oncall
Scaling the search-api to 10 instances successfully reduced the request queue depth and latency. While the immediate incident is resolved, the underlying autoscaling configuration limit needs to be permanently adjusted to prevent recurrence during future traffic spikes.
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-lite-latest · 862ms