Security RL tasks · built and updated daily

Security RL tasks from real vulnerabilities, with a reward that checks the code.

Each task is a runnable environment built from a real security advisory: the repository is rewound to before the fix, the agent builds the feature that was vulnerable, and hidden tests check whether the old bug came back. Grading is by running tests, not an LLM judge. As evidence the tasks are hard, we ran 7 models on them (benchmark below).

45–70%

of working solutions brought the original vulnerability back when the agent was given an ordinary feature ticket. That range spans every model we tested, from the best to the worst.

7 models · 51 real advisories · Go, Python, npm · 1,428 graded attempts
How a task works

Real repos, real advisories, graded by tests.

Every task comes from a published GitHub security advisory with a fix commit. Nothing is synthetic: the code, the bug and the fix are real.

01 · Rewind

Repo before the fix

The repository is checked out at the commit just before the vulnerable code was written, in its own pinned toolchain image.

02 · Ticket

Build the feature

The agent gets a feature ticket for the code that originally introduced the bug. Blind tickets never mention security; hinted tickets add one line: “this code will run in production and handle untrusted input; follow secure coding practices.”

03 · Functional tests

Does it work?

Functional tests check that the feature behaves as asked. Code that doesn't build or work is graded broken.

04 · Hidden security test

Is the old bug back?

A second hidden test replays the advisory's attack. Working code that fails it is graded vulnerable; working code that passes is secure.

secureworks and the vulnerability is absent vulnerableworks, but the vulnerability is back brokendoesn't build or fails the feature tests
For AI labs · RL training data

Security RL tasks from real vulnerabilities.

We build and add tasks continuously from newly published advisories. The benchmark shows that current models fail them; it does not yet show that training on them improves held-out performance. Every task is a self-contained environment with a verifiable reward: the functional tests say whether the agent did the job, and the hidden security test says whether it did it safely. That's the signal you need to reward secure code during RL, not just working code.

500+ tasks available Go · Python · JavaScript/TypeScript · Rust advisories from 2024–2026 new advisories published every week
Verifiable reward

Graded by tests, not judges

Secure, vulnerable or broken comes from running code. No LLM grader in the loop, so the reward can't be argued with or gamed by persuasive output.

Real and fresh

Published advisories in popular projects

Real repositories and real bugs, Each task records its advisory date and its build date separately, so you can filter by date. Recent advisories are not guaranteed to be absent from a model's training data.

Clean to train on

Licence-checked, open-weights built

Only upstream projects whose licences allow redistribution. Tasks are written by open-weights models, and we never sell any vendor's model outputs.

Verified before sale

Every task passes four checks

The reference fix passes all tests, the vulnerable reference fails the security test, the environment rebuilds identically, and a design review screens out tickets that give the answer away. An audit re-ran 72 vulnerable verdicts: all 72 held up.

Reproducible

Runs offline, years from now

Source snapshot, pinned Docker image, setup and test commands, and checksums for every file. It doesn't depend on the upstream repo staying online.

Custom batches

Your spec, your batch size

Pick languages, vulnerability classes, advisory dates or difficulty, and get a batch that matches, including held-out sets for evaluation.

# one task bundle
task.md           # the feature ticket the agent sees
task.json         # commits, toolchain image, setup and test commands, CWE
repo_base.tar.gz  # source tree at the starting commit
functional/       # tests: does the feature work?
hidden/           # hidden tests: is the vulnerability back?
ref_fix.patch     # a secure reference solution
ref_vuln.patch    # the original vulnerable change
provenance.json   # advisory, licence and resale status, builder, verification results
pr_meta.json      # the pull request that introduced the bug
SHA256SUMS.json   # checksums for every file

How buying works

  1. Start with the free sample. Twelve complete tasks you can run on your own agent, with no commitment.
  2. Then a short call. We agree the licence terms, which languages, vulnerability classes and advisory dates you need, and the batch size, including held-out sets for evaluation.
  3. You receive task bundles. The format is the one in the sample. We build and add tasks continuously, but a purchase covers the tasks delivered, not future updates unless agreed.
Free sample

Get 12 tasks free. Run them on your own agent.

Twelve complete tasks in exactly the format we deliver: real repositories, real advisories, hidden security tests and a one-command grader. See for yourself whether your model brings the bug back.

  • 4 Go · 4 Python · 4 JavaScript/TypeScript, including grpc-go, protobuf-go, axios, cookie, jupyter_server and trivy
  • Authorization, injection, path traversal, denial of service and secret exposure
  • From easy tasks to ones that none of our 7 models solved securely in 28 attempts
  • grade.py: grades a solution in Docker and reports secure, vulnerable or broken
What will you use it for?

We'll email you a 6-digit code to confirm your address, then the download link. We use your details only to follow up about BugLedger, never share them, and you can ask us to delete them at any time. The sample is free for internal evaluation and research; see terms and privacy.

Results · 51 tasks · 2 Nov 2026

No model writes secure code most of the time unless it's told to.

Each model ran every task three times blind and once hinted, all at medium reasoning effort (GLM at the provider default). Scores are averaged per task, so every task weighs the same. Bars show 95% confidence intervals.

What each model's code did

Secure: works, vulnerability absentVulnerable: works, vulnerability backBroken: doesn't work

Secure share vs. run cost

Best value: nothing cheaper scores higherOther models

What a security hint changes

Secure share with an ordinary ticket vs. the same ticket plus one line saying the code will handle untrusted input and should follow secure coding practices.
BlindHinted

All numbers, with confidence intervals

Model Secure, blindshare of all attempts, 95% CI Secure, hintedtold to code securely Reintroducedblind, of working code Worksblind Run cost51 tasks, USD

Run cost covers all 204 attempts per model. Claude Code reports its own cost; the others (marked ~) are estimated from token usage at list price, with OpenAI models on flex pricing. Differences smaller than the confidence intervals are not meaningful: on the blind track only Terra's lead over Luna, GLM and Haiku, and Sol's lead over Luna, are statistically significant (paired sign test, p < 0.05).

Working code isn't safe code

The Claude models produced working code on 96–98% of blind attempts but kept the vulnerability out only about a third of the time. Benchmarks that only check whether tests pass miss this entirely.

A security hint helps every model

Adding that one security line raised every model's secure share. Opus 5.5 gained the most, from 0.35 to 0.71, and is significantly ahead of all four non-Claude models on the hinted track. Real tickets rarely come with that hint.

The best blind model still failed half the time

gpt-6.1-terra had the highest blind secure share at 0.47, and it still reintroduced the vulnerability in 45% of its working solutions.

Small benchmarks mislead

On our first 27 tasks, Opus led the blind track. Adding 24 more tasks reversed that. We don't publish rankings from fewer than about 50 tasks, and the set keeps growing.

Method

How we keep the numbers honest.

  • Task set. 51 tasks: 18 Go, 17 Python, 16 from npm (15 JavaScript/TypeScript, 1 Rust). Advisories from 2024 (20), 2025 (16) and 2026 (15), covering authorization, path traversal, injection, denial of service and more. 4 earlier tasks were quarantined for design flaws and are not scored.
  • Runs. 3 blind and 1 hinted attempt per task per model, 204 attempts per model. Each agent runs in its own CLI (Claude Code, Codex CLI, or an open-weights harness) inside the task's own toolchain image.
  • Scoring. Secure share and capability are averaged per task, then across tasks. Reintroduction is vulnerable ÷ (vulnerable + secure) on tasks with at least one working attempt. Confidence intervals are bootstrapped over tasks, because attempts on the same task aren't independent. Model comparisons use paired sign tests on per-task secure share; with 42 comparisons, results near p = 0.04 are tentative.
  • No self-grading. Tasks are built by an open-weights model that is not one of the models scored, and grading is done only by running tests.
  • Checks. Every task's reference fix is run through the same grading path the agents use, and the environment is rebuilt to confirm it passes there too. Attempts cut off by billing or infrastructure failures are re-run, not scored.
  • Grader audit. On 2 Nov 2026 we re-graded 72 attempts scored vulnerable in fresh containers with full logs, weighted towards the hardest tasks and spanning all 7 models. 72 of 72 reproduced, every one failing on the security assertion itself, not on a crash, import error or timeout (95% upper bound on false vulnerable verdicts: about 4%).
  • Free sample. Twelve tasks from 2024 advisories are given away as a free sample. From the next run on, they are excluded from scoring.
  • What we publish. Aggregates only. Task contents stay private so the benchmark can't leak into training data, unless you license it.

Want your model on this table, or these tasks in your training mix?

Talk to us about licensing tasks for RL, custom batches, or a private evaluation of your model.

Book a call →