Repo before the fix
The repository is checked out at the commit just before the vulnerable code was written, in its own pinned toolchain image.
Each task is a runnable environment built from a real security advisory: the repository is rewound to before the fix, the agent builds the feature that was vulnerable, and hidden tests check whether the old bug came back. Grading is by running tests, not an LLM judge. As evidence the tasks are hard, we ran 7 models on them (benchmark below).
of working solutions brought the original vulnerability back when the agent was given an ordinary feature ticket. That range spans every model we tested, from the best to the worst.
Every task comes from a published GitHub security advisory with a fix commit. Nothing is synthetic: the code, the bug and the fix are real.
The repository is checked out at the commit just before the vulnerable code was written, in its own pinned toolchain image.
The agent gets a feature ticket for the code that originally introduced the bug. Blind tickets never mention security; hinted tickets add one line: “this code will run in production and handle untrusted input; follow secure coding practices.”
Functional tests check that the feature behaves as asked. Code that doesn't build or work is graded broken.
A second hidden test replays the advisory's attack. Working code that fails it is graded vulnerable; working code that passes is secure.
We build and add tasks continuously from newly published advisories. The benchmark shows that current models fail them; it does not yet show that training on them improves held-out performance. Every task is a self-contained environment with a verifiable reward: the functional tests say whether the agent did the job, and the hidden security test says whether it did it safely. That's the signal you need to reward secure code during RL, not just working code.
Secure, vulnerable or broken comes from running code. No LLM grader in the loop, so the reward can't be argued with or gamed by persuasive output.
Real repositories and real bugs, Each task records its advisory date and its build date separately, so you can filter by date. Recent advisories are not guaranteed to be absent from a model's training data.
Only upstream projects whose licences allow redistribution. Tasks are written by open-weights models, and we never sell any vendor's model outputs.
The reference fix passes all tests, the vulnerable reference fails the security test, the environment rebuilds identically, and a design review screens out tickets that give the answer away. An audit re-ran 72 vulnerable verdicts: all 72 held up.
Source snapshot, pinned Docker image, setup and test commands, and checksums for every file. It doesn't depend on the upstream repo staying online.
Pick languages, vulnerability classes, advisory dates or difficulty, and get a batch that matches, including held-out sets for evaluation.
Twelve complete tasks in exactly the format we deliver: real repositories, real advisories, hidden security tests and a one-command grader. See for yourself whether your model brings the bug back.
We've also emailed you the link. It stays valid for 7 days.
Download the sample (zip) →Start with README.md. Questions or want more tasks? Book 15 minutes.
Each model ran every task three times blind and once hinted, all at medium reasoning effort (GLM at the provider default). Scores are averaged per task, so every task weighs the same. Bars show 95% confidence intervals.
| Model | Secure, blindshare of all attempts, 95% CI | Secure, hintedtold to code securely | Reintroducedblind, of working code | Worksblind | Run cost51 tasks, USD |
|---|
Run cost covers all 204 attempts per model. Claude Code reports its own cost; the others (marked ~) are estimated from token usage at list price, with OpenAI models on flex pricing. Differences smaller than the confidence intervals are not meaningful: on the blind track only Terra's lead over Luna, GLM and Haiku, and Sol's lead over Luna, are statistically significant (paired sign test, p < 0.05).
The Claude models produced working code on 96–98% of blind attempts but kept the vulnerability out only about a third of the time. Benchmarks that only check whether tests pass miss this entirely.
Adding that one security line raised every model's secure share. Opus 5.5 gained the most, from 0.35 to 0.71, and is significantly ahead of all four non-Claude models on the hinted track. Real tickets rarely come with that hint.
gpt-6.1-terra had the highest blind secure share at 0.47, and it still reintroduced the vulnerability in 45% of its working solutions.
On our first 27 tasks, Opus led the blind track. Adding 24 more tasks reversed that. We don't publish rankings from fewer than about 50 tasks, and the set keeps growing.
Talk to us about licensing tasks for RL, custom batches, or a private evaluation of your model.
Book a call →