Anirudh
Senior DevOps Engineer
gets pinged every time a pipeline fails, including errors he's already fixed three times before. He wants a system that recognizes a failure it's seen and skips straight to the fix.
Owned the product lifecycle for an AI-powered CI/CD failure triage tool at Amdocs — from discovery with SRE/DevOps engineers to a self improving knowledge base that turns repeat build failures into instant fixes
The results
15–30 minseconds
Manual Triage Time Cut
100%
KB Coverage of Indexed Failures
40%
Fix Reuse Rate (zero LLM calls)
0
Pipeline Changes Required to Integrate
The Problem
When a CI pipeline goes red, engineers scroll through thousands of lines of raw console output looking for one relevant clue — a process that takes 15–30 minutes per incident. Worse, the same failures recur week after week, because the fix lives only in one engineer's head and disappears the moment they move teams or roll off the project. With hundreds of MB of logs generated per build, manual triage simply doesn't scale.
Discovery
I ran discovery interviews with SRE and DevOps engineers to get past "logs are big" and into the specific failure modes: manual effort scrolling raw output, siloed tribal knowledge of past fixes, and a volume problem that made human parsing impossible at scale. Two personas anchored the rest of the work:
Priya
Backend Engineer
blocked behind three other developers in the queue, waiting for someone to look at her failed build. She doesn't need a deep explanation, she needs an accurate next step, fast.
From Problem to PRD
I framed the opportunity around one question: how do we let repeat failures resolve themselves, and reserve the LLM for genuinely new ones? I wrote the PRD defining the ingestion path — a Jenkins RSS listener plus Pipeline APIs, with Postgres-backed idempotency so no build gets processed twice — and scoped the processing pipeline: filter noisy logs, generate a stable error fingerprint, embed it, then run a kNN similarity search against a knowledge base of previously approved fixes.
Scoping the LangGraph/Groq Workflow
Working with engineering, I set the threshold logic: any failure scoring ≥0.90 similarity against the knowledge base returns the past fix instantly, with zero LLM calls — sub-second, no token cost. Below that threshold, the failure is treated as new and routed through a LangGraph-orchestrated workflow on Groq that performs full root-cause analysis and proposes a step-by-step fix from the filtered logs.
The Accept/Reject Loop
The hardest product call was whether to auto apply LLM-suggested fixes. I decided against it. Every LLM-generated fix goes to the engineer who owns the failing service for a one click Accept or Reject before it's written into the shared knowledge base.
I traded a small amount of automation speed for a knowledge base I could actually trust. A fully autonomous fixer risks quietly poisoning the KB with a wrong fix that then gets reused with false confidence on the next similar failure. Accept/reject keeps a human in the loop once per new error type
not once per occurrence.
What Shipped
Partnered with engineering to ship a FastAPI + Next.js product with Elasticsearch backed similarity search across [X]+ indexed failure sessions [confirm current number], a Root Cause Analysis view (error signature, detected class, suggested fix with copyable code snippets, and a compressed log excerpt), and a Knowledge Map showing which fixes are being reused and where coverage gaps remain.
Outcome
On indexed failures, triage drops from 15–30 minutes of manual log-scrolling to a KB-matched fix in under a second — or a few minutes for genuinely new errors that still need full LLM analysis. Fix reuse rate is already at 40%, with 100% KB coverage across indexed failures in an early deployment.
Trade off I'd Revisit
the 0.90 similarity threshold is a blunt instrument , a near-match at 0.89 still triggers a full LLM re analysis it may not need. Phase 2, already scoped, shifts the LLM's role from generating full solutions to generating context for a human or a lighter weight matcher, so even novel failures lean more on retrieval and less on generation , cutting cost, latency, and hallucination risk further.
Product StrategyGenerative AIDeveloper ToolsPRDsUser Research