Our team built Sherlock, a dedicated bug-fixing agent that takes bugs from Slack and processes them as structured tasks in Sharkly.ai.
Over a recent 15-day window, Sherlock completed 13% of all company tasks. Of its last 100 assigned tasks, 90 finished successfully.
The reliability came from six controls:
A 10-stage state machine: Sherlock cannot skip stages. Diagnosis cannot touch product code.
Two approval tokens: [START DEVELOPMENT] allows code changes; [SUBMIT FOR TESTING] allows push/merge. We reduced the original 7 approval checkpoints. "Looks good" and "continue" are not authorization.
Required artifacts: diagnosis.md, idea.md, plan.md, and review-report.md. A chat message saying "diagnosis complete" has no authority.
Disk-based recovery: all state is saved in state.json. Interrupted sessions resume from disk because reconstructing progress from chat history caused hallucinated progress.
A six-iteration review cap: the agent reviews its own diff as a clean-room reviewer, patches blocking issues, and repeats up to 6 times. Then it stops and escalates to a human.
Screenshots at the decision point: before/after screenshots for UI changes appear directly in the conversation, not merely as file paths.
A real "list not scrollable" bug went from Slack → structured task → 14 minutes of autonomous diagnosis → 3 decisions for a human → approval → 37 minutes of implementation → PR + preview URL + screenshots.
Two human decisions total.
The 90/100 result is conditional on task routing. It is not a universal bug-fixing benchmark, and reopen-rate data has not been published.
Full article: https://dev.to/yukioikeda/how-to-build-an-ai-agent-that-fixes-bugs-reliably-practice-in-sharkly-o63
What controls do you use to make coding agents predictable and reviewable?
No replies yet.