We built an AI agent that fixes bugs reliably — six workflow controls made the difference

By lucdev · · 0 replies

Our team built Sherlock, a dedicated bug-fixing agent that takes bugs from Slack and processes them as structured tasks in Sharkly.ai.

Over a recent 15-day window, Sherlock completed 13% of all company tasks. Of its last 100 assigned tasks, 90 finished successfully.

The reliability came from six controls:

A 10-stage state machine: Sherlock cannot skip stages. Diagnosis cannot touch product code.

Two approval tokens: [START DEVELOPMENT] allows code changes; [SUBMIT FOR TESTING] allows push/merge. We reduced the original 7 approval checkpoints. "Looks good" and "continue" are not authorization.

Required artifacts: diagnosis.md, idea.md, plan.md, and review-report.md. A chat message saying "diagnosis complete" has no authority.

Disk-based recovery: all state is saved in state.json. Interrupted sessions resume from disk because reconstructing progress from chat history caused hallucinated progress.

A six-iteration review cap: the agent reviews its own diff as a clean-room reviewer, patches blocking issues, and repeats up to 6 times. Then it stops and escalates to a human.

Screenshots at the decision point: before/after screenshots for UI changes appear directly in the conversation, not merely as file paths.

A real "list not scrollable" bug went from Slack → structured task → 14 minutes of autonomous diagnosis → 3 decisions for a human → approval → 37 minutes of implementation → PR + preview URL + screenshots.

Two human decisions total.

The 90/100 result is conditional on task routing. It is not a universal bug-fixing benchmark, and reopen-rate data has not been published.

Full article: https://dev.to/yukioikeda/how-to-build-an-ai-agent-that-fixes-bugs-reliably-practice-in-sharkly-o63

What controls do you use to make coding agents predictable and reviewable?

Editing your thread.
Reply
Add images Up to 2 images, 3MB each.
0/2
You must be signed in to reply.
Replies

No replies yet.

Related discussions

Popular in this category