Incident Response — Agentic MTTR Demo

INC17837788205276

P2newai_assisted

Elevated NXDOMAIN rate for internal service discovery after config change

Assembled Context — bounded to last 30 minutes
Rolling logs
Recent commits
f9c966f infra: update internal DNS zone for service mesh migration — s.kim
Similar past incidents
(none found)
Diagnosis Agent
ran in parallel
An update to the internal DNS zone records (commit f9c966f for service mesh migration) misconfigured or deleted the record for 'orders-service.internal', leading to resolution failures.
  • commit f9c966f infra: update internal DNS zone for service mesh migration (s.kim)
  • [dns-config] applied change to internal zone records at 14:02 UTC
  • [gateway] ERROR upstream DNS resolution failed: NXDOMAIN for orders-service.internal
Assessed: P2 / network
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-latest · 39269ms
Severity / Routing Agent
ran in parallel
P2 / infrastructure
Routed to: infrastructure-networking
The incident involves a core service discovery mechanism failure triggered by a configuration change, impacting internal service communication (orders-service). While it is critical, the system is failing over to cached records, preventing a complete outage, which warrants a P2 severity.
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-lite-latest · 969ms
Scenario Matcher
deterministic, pre-LLM
DNS resolution failures after config change
DNS failures correlated with a recent config/deploy change are best resolved by rolling back the change rather than restarting downstream services.
Match score: 100% · Recommended action: rollback_deploy
Matched tags: dns, resolution failure, nxdomain, config change
Confidence Gate

Computed from 4 measurable signals — not the model's self-reported confidence.

Severity match rate50%
x
Runbook coverage100%
x
Data completeness67%
x
Recency50%
x
Computed confidence (v1)70%
Auto-execute threshold: 75%