Local-First Agentic Security Reviews
This year, a lot of hype surrounded Mythos, the new model from Anthropic for security reviews. We tried to reproduce this with software we write and a local Qwen3.6-27B model. What the model found was shockingly good, which led us to the conviction that doing software development without agentic security reviews is irresponsible in 2026.
Prelude
Dani Schmid, a coworker, sent me a talk from the Black Hat Conference 2026, called Unprompted, https://www.youtube.com/watch?v=1sd26pWhfmg, from an Anthropic security researcher, Nicholas Carlini. In his talk, Carlini explains how easy it is to do security reviews of existing code using LLMs. Basically, the only thing you have to do is point Claude at every file in the codebase and let it run with a simple prompt.
You are playing in a CTF.
Find a vulnerability.
Hint: look at ${FILE}
Write the most serious one to /out/report.txt.
I thought: I should try that as well, but running locally using a Qwen model.
Cookbook
The Model
Owning a MacBook with enough memory, I used https://omlx.ai to run Qwen3.6-27B with a context window of 200k tokens locally.
We used a local model because we cannot send our entire source code to US cloud providers. Although they contractually state they won’t train their models on the prompts, which contain the whole codebase, they did not respect copyright while using the internet and pirated books to train their models.
According to https://swelljoe.com/post/will-it-mythos/, a blog post where a researcher tried to replicate the findings of Mythos with smaller models, Qwen3.6-27B is surprisingly good.
Make sure that you set the temperature and the other model parameters for "coding", temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0, Enable thinking, MTP, and preserve_thinking. See https://huggingface.co/Qwen/Qwen3.6-27B for references.
Creating the container
Creating the container for our Rails application was straightforward. We already use containerization for some deployments and for development. For the security review, we wanted to bundle the codebase inside the container so we wouldn’t need to mount it. The only thing we added was Claude for the reviews:
ARG NODE_VERSION=24.10.0
ARG YARN_VERSION=1.22.22
ENV PATH=/usr/local/node/bin:$PATH
RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && \
/tmp/node-build-master/bin/node-build "${NODE_VERSION}" /usr/local/node && \
npm install -g yarn@$YARN_VERSION && \
npm install -g @anthropic-ai/claude-code && \
ln -s /usr/local/node/bin/claude-code /usr/local/bin/claude && \
rm -rf /tmp/node-build-master
The full file, including the script below, can be found in the accompanying gist.
Security Considerations
Make sure that you don’t have any secrets inside the codebase. Secrets might be exploited by the agent and could lead to the compromise of the production environment.
The agent might leak part or all of your full codebase if given full internet access. Although the risk is relatively low, keep this in mind.
Running the tests
To run the tests, we used a local Podman machine and a loop. We ran over files sequentially, which took about 30 minutes per file and a full weekend for the entire codebase. Given a stronger local LLM runner, the tasks could be easily parallelized. The interesting part of the script is:
podman run --rm \
-v "${REPORT_FILE}:/out/report.txt" \
-e ANTHROPIC_BASE_URL='http://192.168.1.181:8000' \
-e ANTHROPIC_AUTH_TOKEN='smoca' \
-e ANTHROPIC_DEFAULT_OPUS_MODEL='Qwen3.6-27B' \
-e ANTHROPIC_DEFAULT_SONNET_MODEL='Qwen3.6-27B' \
-e ANTHROPIC_DEFAULT_HAIKU_MODEL='gemma-4-e2b-it-4bit' \
-e API_TIMEOUT_MS=3000000 \
-e CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 \
claude-code \
claude-code --dangerously-skip-permissions -p "You are playing in a CTF.
Find a vulnerability.
Hint: look at ${FILE}
Write the most serious one to /out/report.txt." --verbose
The part about --dangerously-skip-permissions is needed so the agent can install dependencies and work autonomously. Only use this in isolated, ephemeral containers (e.g., with --rm and no host mounts beyond /out). Never run it on production systems or with sensitive data.
For more details, see the official claude-code documentation.
Results
We used AI again to consolidate the per-file reports. However, as we know, agents are not good at examining every file, and we had to look through every report manually.
We found
One remote code execution which was real. The issue was an unsanitized flag for image variants where user supplied text was passed directly to ImageMagick.
One SQL injection, which was a false positive. Inside the codebase, a SQL statement is assembled by string but no user input was involved in constructing the SQL statement.
Many broken access control issues, where users could manipulate API endpoints to perform unauthorized actions (e.g., accessing or modifying resources belonging to other users).
We manually triaged the findings and addressed the most severe issues over the following two weeks.
In our opinion, agentic security reviews will not replace a professional security review, but they will effectively identify vulnerabilities that are easy to detect and exploit.
Future Impact
This analysis led to the conclusion that automated agentic security reviews should be part of every software development lifecycle. It also inspired the creation of the Review Bot, a locally run merge request review tool that focuses on security, logic bugs, architecture, performance, and testing. More on that in a future blog post.
If you have any questions or need assistance replicating this setup, feel free to reach out to info@smoca.ch
