- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors argue that the main reason agents can't do long-horizon work is because no one has made environments that check their intermediate steps. So they propose VERA: convert benchmark data into a sandbox of five stages—plan, setup, verify, execute, submit—and have an agent write rubrics for each stage, then a judge decides which sandboxes get added to training. Then VERA alternates rounds where it updates the model or harness while keeping the other fixed. VERA is able to beat other systems across domains like SWE, math, medicine, and coworking.
Critical strengths
- Full VERA beats every single-axis version on all four 9B param benchmarks. This is especially impressive because SWE-Bench Verified uses their official scoring rather than verifiable rubric scores, and MedXpertQA uses a completely different domain, proving VERA has real gain.
- The authors release 9000 new sandboxes which can restart at any point during a task. These will be invaluable to the research community if they are clean.
Critical weaknesses
- I could not find AutoCoWorkBench anywhere. Is this a real benchmark?
- Several claims seem to contradict each other within the paper.
- In the abstract it says we keep a change if it improves the model. But the method section says we retain a checkpoint if its development score decreases by less than 20%.
- It says that freezing either side loses half the gain. But on AutoMedBench, keeping the harness alone keeps 36.2 out of 49.2 points, suggesting it's more like losing ~26%.
- It's claimed that on AutoCoWorkbench, VERA surpasses current state-of-the-art models, but this only seems true against Claude Opus 4.8; GPT 5.6 Sol actually does better (72.6 vs 71.6).
- It's claimed that general skills are kept intact, but AIME falls by up to 59.7 at both 4B and 9B
- Adding VERA to the base model decreases AutoMedBench at 27B which seems contradictory to the claim that both sides help each other
Questions for authors
- How did you ensure that the training set and test sets didn't overlap for AutoMedBench and SWE-Bench?
- How much of VERA's improvement comes from distillation? GPT 5.6 Sol judges which sandboxes go into training, determines what stage each round trains on, and writes the skills edits. It's also used as a baseline against which VERA is compared. Some of the results will be comparing GPT 5.6 Sol vs. VERA-GPT 5.6 Sol.
- Did you use the same amount of compute for full VERA as for single-axis VERA?
- Why does coding-focused training raise AIME in the 27B version but lower it in the 9B?