Incident Response — Agentic MTTR Demo

INC17838763831591

P2resolvedai_assisted

Elevated NXDOMAIN rate for internal service discovery after config change

Pipeline Status
Context Assembly
Diagnose Agent
Severity / Routing Agent
Scenario Match
Remediate Agent
Confidence Gate
Gate Decisionauto executed
Execute Action
Verify Agent
Resolution
Assembled Context — bounded to last 30 minutes
Rolling logs
Recent commits
f9c966f infra: update internal DNS zone for service mesh migration — s.kim
Similar past incidents
INC17776528750659 · network/P2 · 72d ago — Elevated NXDOMAIN rate for internal service discovery after config change
INC178131738072211 · network/P2 · 29d ago — Elevated NXDOMAIN rate for internal service discovery after config change
INC177982807075926 · network/P2 · 46d ago — Elevated NXDOMAIN rate for internal service discovery after config change
Diagnosis Agent
ran in parallel
A configuration change to the internal DNS zone records for service mesh migration caused upstream DNS resolution failures (NXDOMAIN) for orders-service.internal, forcing the gateway to fall back to stale cached resolver entries.
  • [dns-config] applied change to internal zone records at 14:02 UTC
  • [gateway] ERROR upstream DNS resolution failed: NXDOMAIN for orders-service.internal
  • f9c966f infra: update internal DNS zone for service mesh migration (s.kim)
Assessed: P2 / network
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-latest · 7744ms
Severity / Routing Agent
ran in parallel
P2 / network
Routed to: dns-infrastructure-team
The incident involves a failure in internal DNS resolution (NXDOMAIN) following a configuration change, which is a network-layer issue. While the gateway is falling back to stale cache, the inability to resolve the orders-service indicates a significant degradation in service discovery, consistent with previous P2 incidents of this nature.
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-lite-latest · 1015ms
Scenario Matcher
deterministic, pre-LLM
DNS resolution failures after config change
DNS failures correlated with a recent config/deploy change are best resolved by rolling back the change rather than restarting downstream services.
Match score: 93% · Recommended action: rollback_deploy
Matched tags: dns, resolution failure, nxdomain, config change
Remediate Agent
grounded by runbook match, pre-gate
Proposed action: rollback_deploy
team: dns-infrastructure-team · reason: NXDOMAIN errors observed after recent DNS zone config change · message: Rollback DNS zone configuration to previous stable version to stop NXDOMAIN errors · toVersion: f9c966f^ · serviceName: internal-dns · justification: Runbook for DNS resolution failures after config change recommends rollback_deploy; the root cause is the config change, so rolling back is the appropriate remediation
The diagnosis identifies the recent DNS zone configuration change as the cause of NXDOMAIN failures. The matched runbook explicitly advises a rollback_deploy for such scenarios, and there is no conflicting evidence suggesting another action. Therefore, rolling back the DNS configuration to the prior version is the correct bounded action.
openai/gpt-oss-120b · 19139ms
Confidence Gate
auto_executed

Computed from 4 measurable signals — not the model's self-reported confidence.

Severity match rate100%
x
Runbook coverage93%
x
Data completeness100%
x
Recency68%
x
Computed confidence (v1)93%
Auto-execute threshold: 75%
Proposed action: rollback_deploy
#dns-infrastructure-team — Slack-style FYI notification
in-app simulated

Auto-executed "rollback_deploy" for "Elevated NXDOMAIN rate for internal service discovery after config change" — confidence 93% cleared the auto-execute threshold, no human review required. Execution succeeded.

Already executed — nothing to approve or cancel. Posted here purely so the trail isn't audit-log-only.

Verify Agent
runs after execution
Resolved · residual risk: none
Follow-up: none
The rollback successfully reverted the configuration change that was identified as the root cause of the NXDOMAIN errors. Service discovery metrics have returned to baseline levels following the deployment of the previous stable version.
Self-reported confidence: 100% (audit-only — not used by the gate)
gemini-flash-lite-latest · 856ms