Agentic Code Reviews, Part 2: The Review Bot
How can we use local LLMs, not to generate more AI slop, but to increase the quality and security of our software? Following the agentic security review detailed in our previous post, local-first agentic security review, we included an agentic software review inside our normal development workflow. The bot reviewed 177 merge requests in the last two weeks across 27 projects and found about 50 real critical bugs, of which at least 10 would not have been found by manual review.
The code is available at https://github.com/smoca-ag/review-bot.
Prelude
In the previous post, Dani Schmid sent me a Black Hat talk in which Nicholas Carlini pointed Claude at one file at a time with a four-line CTF prompt. I ran the same prompt locally against a whole Rails codebase, and the model found real remote code execution and broken access control bugs. The conclusion was: doing software development without agentic security reviews is irresponsible in 2026.
This led to the idea to include the review in the normal software development workflow. Not to increase the lines of code or the speed of software delivery, but to improve software quality.
Using a SaaS offering was out of the question. The contracts we have with our customers prevent us from shipping their code anywhere without their written consent. And even if we could, do you really want to be dependent on cloud offerings that might raise the price at any time?
So a new tool had to be written, the review bot.
Architecture
The code is written in Python using the Pydantic AI harness.
The review bot is triggered by a change to a merge request on our local GitLab instance via a global webhook. The webhook is triggered by GitLab on push and merge-request events. If we want to run it in a more targeted way, a trigger label ("ai-review-requested") is possible as well.
It checks out the source and target branches into a new Git repository.
Then it starts a user-mode Podman container with capabilities locked down as far as possible, and the source code mounted as an overlay (read-only on the host).
Then multiple agents are started sequentially with a shared prefix prompt and a specialization prompt each. The prefix prompt includes a truncated diff and can be prefix-cached by llama.cpp to keep the prompt-processing latency as low as possible. The agents examine the merge-request description, security, logic, test coverage, and performance.
The agents use the following tools to investigate the source code:
list_files
glob
read_file
search_code
update_todo
semantic_code_search: search the codebase for similar code based on embeddings which are generated per merge request
view_code_diff_section: look at the code around the diff context
dependency_graph: look at dependencies built on Tree-sitter with support for the languages we use
The tool schemas are generated directly by the model we are using, to align them as much as possible with its natural tool calling. This has been shown to increase the performance of LLMs in tool-calling scenarios.
The different agents' results are sent to a critic agent, acting as a judge, to filter out as many false positives as possible.
Finally, the result is posted as inline comments and a general note to the merge request.
On failure, we have three retries and then the review bot gives up. The automated review takes between 30 minutes and one hour for big merge requests. At the moment, with this hardware, the requests are queued since we are memory-bandwidth-bound. Running multiple reviews concurrently on this hardware makes it slower, not faster.
The code is open source and available at https://github.com/smoca-ag/review-bot.
Security Implications
The checkout happens outside the container and inside the container, no environment variables, especially no GitLab credentials are available. The only thing inside are the target and source revisions of the merge request.
Every input from the user, diffs, the merge request description and the title of the merge request is escaped via a simple CDATA-based escaping, which should prevent most simple prompt injection techniques.
The container has full network access to allow the agents to install tools or packages, which is needed for a thorough inspection.
The worst thing that could happen would be a leak of data checked in inside the repository.
In contrast, GitHub AI agents were susceptible to prompt injection and might even leak sibling repositories, which can't happen here. See GitLost for details.
The Hardware
This time the model runs on an Asus GX10, which is a GB10 platform for NVIDIA with 128GB unified memory for about 3000 CHF. It was not feasible to run it on my laptop anymore; I need it to do actual work.
The model is the same Qwen3.6-27B I used last time. The coding parameters are unchanged from the previous post — temperature=0.6, top_p=0.95, top_k=20, thinking and MTP enabled. See that post for the full justification.
This hardware gives us one review per 30 minutes, which is enough for a small team of about 15 devs, who do not all work full-time. For larger teams, I would strongly suggest investing in a faster model runner.
The numbers
To get some preliminary results, I analyzed the log files from our internal GitLab for June 23 – July 7, 2026 (14 days, 10 workdays).
177 reviews were completed, about 17 per workday across 27 unique projects. The bot found 385 issues in total, of which 70 were critical.
To calculate the rate of hallucinations in the critical issues, I reviewed the first 10 critical issues (manually verifying all 70 is infeasible), and 8 of them were real issues. Extrapolating gives us about 50 real bugs found by the review bot, or about 5 per workday. The non-critical issues were not analyzed.
To calculate the return on investment for the review bot, we have to make two assumptions: how expensive (in hours) a bug in production is, and by how much the additional feedback from the review bot increases reviewers' time.
A bug in production costs at least 4h, since we have to go through the full ticket-writing, triaging, reproducing, testing, and rollout cycle again.
Since the review bot adds a few paragraphs to the merge request, we estimate the additional review time to be 10 minutes.
On the cost side, this gives us 177 × 10 min ≈ 30h, or about 3h per workday of additional review time.
On the return side, if shipping just one of these ≈50 bugs (which cost at least 4h to fix in production) can be prevented per day, we have a net positive of one hour per day. According to my developers, about 10 bugs were only found thanks to the review bot, about one per workday.
The total cost of ownership is not calculated here, but we can give you some rough numbers. The setup (without the development) costs about 1 developer day, the hardware 2-3 developer days; the power consumption of the GB10 platform is negligible. If we assume one bug per day would have slipped into production without the review bot, we save one developer hour per day and the system pays for itself (excluding development) after one month.
Conclusion
We have been using this review bot setup for about 3 weeks now. At the beginning, most of our developers were skeptical, even hostile. After the bot proved its worth by finding bugs they overlooked, they accepted the feedback, even though it also produces false positives.
We know that the bot is not yet perfect. For that reason, we decided to publish the code as open source, and we are happy to accept contributions and ideas for improvement.
The bot was written mostly using agentic development techniques, but more on that in a later blog post.
If you have any questions or need assistance replicating this setup, feel free to reach out to info@smoca.ch.
