Incident Response — Agentic MTTR Demo

INC17837979749026

P2newai_assistedLive — updating every 2s

Elevated NXDOMAIN rate for internal service discovery after config change

Pipeline Status
Context Assembly
Diagnose Agent
Severity / Routing Agent
Scenario Match
Remediate Agent
Confidence Gate
Gate Decision
Execute Action
Verify Agent
Resolution
Assembled Context — bounded to last 30 minutes
Rolling logs
Recent commits
f9c966f infra: update internal DNS zone for service mesh migration — s.kim
Similar past incidents
(none found)
Diagnosis Agent
ran in parallel
A recent internal DNS zone update introduced incorrect or missing records for orders-service.internal, causing NXDOMAIN responses and forcing gateways to fall back to stale cache entries.
  • [dns-config] applied change to internal zone records at 14:02 UTC
  • [gateway] ERROR upstream DNS resolution failed: NXDOMAIN for orders-service.internal
  • [gateway] WARN failover to cached resolver entries (stale, ttl expired)
  • Recent commits: infra: update internal DNS zone for service mesh migration
Assessed: P2 / network
Self-reported confidence: 92% (audit-only — not used by the gate)
openai/gpt-oss-120b · 2187ms
Severity / Routing Agent
ran in parallel
P1 / network
Routed to: DNS/Network Infrastructure Team
The incident is caused by a recent configuration change to internal DNS zones, leading to NXDOMAIN errors for a critical service (orders-service). This impacts service-to-service communication, effectively breaking dependent business functions. The immediate correlation between the config commit and the errors confirms a network/DNS service failure.
Self-reported confidence: 95% (audit-only — not used by the gate)
gemini-flash-lite-latest · 1037ms
Scenario Matcher
deterministic, pre-LLM
DNS resolution failures after config change
DNS failures correlated with a recent config/deploy change are best resolved by rolling back the change rather than restarting downstream services.
Match score: 87% · Recommended action: rollback_deploy
Matched tags: dns, nxdomain, config change
Remediate Agent
grounded by runbook match, pre-gate
Proposed action: rollback_deploy
team: DNS/Network Infrastructure Team · reason: Incorrect DNS records causing NXDOMAIN responses. · message: Rollback DNS zone change to restore orders-service.internal records. · serviceName: internal-dns · justification: Rollback is recommended by runbook to revert misconfiguration quickly.
The diagnosis identifies the recent DNS zone update as the root cause of NXDOMAIN errors. The matched runbook explicitly recommends a rollback_deploy for DNS failures after a config change, and no contradictory evidence exists. Therefore, the appropriate bounded action is to rollback the DNS configuration change.
openai/gpt-oss-120b · 1480ms
Confidence Gate

Computed from 4 measurable signals — not the model's self-reported confidence.

Severity match rate50%
x
Runbook coverage87%
x
Data completeness67%
x
Recency50%
x
Computed confidence (v1)65%
Auto-execute threshold: 75%
Verify Agent
runs after execution
Pending…