- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1(−1)
- Soundness
- 2
Read review
Paper summary
Vestrum is a loop for improving an agent harness from failure traces without changing the underlying model. An LLM reads development traces and describes failures as recognizable "failure classes" within "task kinds". It then proposes changes along four dimensions: verification, retrieval, decomposition and knowledge synthesis. The proposals are screened for task-specific leakage, and a bundle is kept only if it improves development results, and in some settings validation results too. A lessons file carries experience across epochs.
The learned harnesses improve over their starting points:
- starting from GAM, on UltraHorizon, LoCoMo and AMA-Bench;
- starting from Claude Code, on 8 held-out Terminal-Bench 4 Hard tasks and on a spatial-transcriptomics cell-type annotation task.
Based on its development record, the paper also argues that verification works best when it is grounded in evidence and placed where intermediate results are committed, while critics that rewrite final answers do more harm than good.
Critical strengths
The results are on held-out data and the bookkeeping is careful. Development, validation and test are disjoint, and harnesses are frozen before testing. The paper also keeps a record of what was tried and rejected, including negative results (Table 19), which I appreciated and rarely see.
The editable surface is broader than prompt optimization: it includes memory construction, deterministic views and hooks. Gains go in the same direction across seeds and replicates in all five settings.
Critical weaknesses
-
All evaluation stays inside the benchmark the harness was tuned on. As a result, I can't tell whether Vestrum produces better harnesses in general or harnesses fitted to these benchmarks. Development, validation and test always come from the same benchmark, and for PathBench even from the same slide.
The admitted mechanisms clearly encode benchmark structure:
- per-rule-class notebooks, and a commit gate that is disabled in UltraHorizon's biology environment;
- event and object views that match AMA-Bench's question axes;
- a "never refuse" rule on a LoCoMo subset that excludes unanswerable questions;
- reference tables built from labelled sections of the same slide.
Improving an untuned GAM or Claude Code using in-distribution development data is expected. The question for this paper is whether the procedure produces something that transfers. That is what the overhypothesis framing claims, and the conclusion calls Vestrum "a common procedure … across different workloads". A frozen harness evaluated on a different benchmark of the same kind would address this, for example the LoCoMo harness on LongMemEval, which you cite.
Relatedly, the only baseline is GEPA. GEPA edits prompts only, and on UltraHorizon it ended up keeping GAM's original prompt. HarnessFix and Meta-Harness, which you discuss as the nearest neighbours, both report Terminal-Bench 2 results with code available. HarnessFix already compares against both GEPA and Meta-Harness.
-
The verification lesson is not supported by test-time evidence, and the paper's own data partly contradict it. This lesson appears in the title, the abstract and Section 4.
- No verifier was ablated at test.
- On AMA-Bench, the verify-on-build check was designed from GPT-4o-mini failures and repaired 17 of 18 notes. With the test model, it checked 107 notes and repaired none (App. N). So the AMA-Bench gain cannot come from verification.
- This also suggests the diagnosed failures are specific to the model, which the task-kind / failure-class framing does not account for.
- On Terminal-Bench, the edit gate "had nothing to refuse" during development, and its effect was not measured.
- The remaining evidence is too thin and mixed to support a general claim about where and how verification should be placed. It consists of counts such as 9 vs 5 and 6 vs 2, from development sets of a few tasks or a single conversation, obtained with different models and harness versions.
Questions for authors
- Can you evaluate frozen harnesses on different benchmarks of the same kind, without any further search?
- How variable is the outcome across independent searches? Each setting reports a single searched harness. Even 2–3 searches in one memory setting would help.
- Which steps in each search involved human decisions? Were the per-setting admission criteria in Table 8 fixed before the searches were run?
3 comments
Thank you for the detailed response and the revision. The revised framing matches the evidence better, and the new test-time ablation of self-verification partially addresses my point that no verifier was ablated at test.
However, my central concern W1&Q1 is still not resolved; task kinds are defined inside each benchmark, so held-out tasks of the same kinds are the in-distribution test split. I don't think it's resolvable by simply rephrasing your claim, which, in other words, will downgrade the paper's informativeness as well. I believe that most of my proposed experiments are easy to run and easy to testify. So I will keep my score.
New results for W1, W5 and Q1
Thank you again for the detailed review. We ran the experiments behind three of your points, and the revised PDF now includes them.
W1 / Q1: transfer to another benchmark (Appendix O.2, Table 23). We ran the frozen LoCoMo harness and GAM on a stratified sample of 150 LongMemEval_S questions (Gemini 3.7 Flash, three seeds, the same blinded judge). Nothing was tuned on LongMemEval.
| Questions | GAM | Vestrum (LoCoMo harness) |
|---|---|---|
| Answerable (140) | 87.4 ± 2.2% | 89.3 ± 1.4% |
| Abstention (10) | 53.3% | 10.0% |
| All (150) | 85.1 ± 2.1% | 84.0 ± 1.3% |
The learned harness carries over to answerable questions, where it leads GAM (21 question wins, 13 losses). On the ten abstention questions, its LoCoMo rule to commit to a best-supported answer makes it answer. Section 6 names this rule as the limit of the LoCoMo gains. It costs about 2.9 points here, so the harness trails GAM by 1.1 points over all 150.
W1: UltraHorizon outside development (Appendix O.1, Table 22). Development used only grid episodes. Split by environment, the test runs give:
| Environment | GAM | Vestrum | Difference | Seeds positive |
|---|---|---|---|---|
| grid (development) | 56.5 | 70.8 | +14.3 | 3/3 |
| sequence (never seen) | 45.2 | 63.3 | +18.1 | 3/3 |
| biology (never seen) | 25.5 | 19.5 | −6.0 | 0/3 |
| all | 47.6 | 59.8 | +12.2 | 3/3 |
The gain carries over to the sequence environment, which the search never saw, and is larger there than on grid. In biology no rule can be checked, so the commit gate is off (Appendix M), and the harness scores below GAM.
W5: self-verification at test. Switching the kept checks off gives up about half of the gain (Table 20, Appendix M.1):
| Vestrum | without self-verification | GAM | |
|---|---|---|---|
| UltraHorizon, grid test episodes | 70.8 ± 3.1 | 64.1 ± 3.6 | 56.5 ± 1.5 |
| LoCoMo, judge accuracy | 75.1 ± 0.8% | 73.2 ± 0.5% | 71.5 ± 0.8% |
Your doubt about the small auditor counts was well placed. In development the auditors broke more correct answers than they fixed, and a test run changed that picture (Appendix M.2, Table 21). We ran the LoCoMo every-answer auditor on 3,824 test answers (three GAM seeds, one Vestrum seed). With Gemini 3.7 Flash it proposed no hedges, which was its failure in development with Haiku 4.5. It fixed 44 answers and broke 5. Most fixes (32) completed answers cut off at the 256-token answer limit that both systems share; on complete answers it fixed 12 and broke 4. The revision now states the auditor result as a development finding (Sections 1 and 5.1) and reports these test counts, and the abstract reports the test ablation instead.
Q2 and W4. We would have liked to run both. The runs above evaluate frozen harnesses and fit within our remaining budget. A full search costs much more, and after pilot runs our budget covered one search per setting. Repeated searches with several seeds, and other harness searches such as Meta-Harness, which learn from tens to hundreds of tasks, were therefore out of reach; Section 6 states both. Our aim was to show that an existing harness can be improved from one to ten long episodes, and the results above add held-out evidence for it.
Thank you for your review, and for noting the record of what was tried and rejected. Appendix M and Table 19 keep it.
W1 (evaluation inside one benchmark). The claim is scoped to task kinds. The fixed harnesses are tested on held-out tasks of the same kinds, and on the memory benchmarks under a different test model (Table 2). UltraHorizon is the exception: development used only the grid environment by accident, so the grid-learned mechanisms also run on the sequence and biology test episodes, an out-of-development test (Appendix G). No other benchmark was tested (Section 6).
W2 (mechanisms encode benchmark structure). Scoping to task kinds is the design. A task kind is an environment, question category or probe axis (Section 3.1). The generalization filter refuses development facts, not fit to a benchmark's task kinds (Section 6). On the cases you list:
W3 (gains are expected; transfer claim). The gains are not automatic. On the same data, GEPA kept GAM's original prompt on UltraHorizon (Appendix K). The revision removes the overhypothesis framing and the "common procedure" sentence.
W4 (baselines). GEPA is the only learned baseline (Section 6). Meta-Harness and HarnessFix rewrite harness code they own and learn from tens to hundreds of tasks. Vestrum targets closed harnesses and one to ten episodes (Section 2; Table 7).
W5 (verification lesson).
Q1. Not run. Your example, the frozen LoCoMo harness on LongMemEval, is the transfer test Section 6 lists as the next experiment.
Q2. Unmeasured: each setting has one search, because the budget did not allow repeated searches (Section 6).
Q3. The admission criteria were fixed before each search (Section 3.5; Table 9), and task kinds were assigned before any proposal (Section 3.1). Before each search, the authors fixed the splits, the editable surface, the task-kind definitions, the admission criteria and the budgets. Diagnosis, proposals, filtering, evaluation and the lessons were made by the loop's model calls.