acceptodds
Under review as a conference paper at ICLR 2027

Vestrum: Improving Agent Harnesses by Adapting Their Verification, Structure and Memory

Abstract

An agent harness controls how a language model accesses information, uses tools, preserves memory, and checks its work. Improving this software is costly when each evaluation requires a long interaction with an environment. We introduce Vestrum, a framework that turns failures in execution traces into scoped harness changes without training the task model. Its organizing overhypothesis is that tasks of a shared kind may exhibit recurring failures whose remedies transfer within that kind. Vestrum expresses failures as recognizable classes, proposes changes across verification, retrieval, decomposition, and knowledge synthesis, and screens their scope before evaluating them as a bundle. A persistent lessons file informs subsequent proposals. Across five settings and two baseline harnesses, the frozen harnesses improve held-out performance: UltraHorizon rises from 47.6 to 59.8 over GAM, Terminal-Bench 4 Hard from 63.7% to 70.3% of checks passed over Claude Code on eight held-out tasks at 1.03× test cost, and cell-type annotation agreement from 67.5% to 77.8% on held-out sections of one slide, alongside gains on LoCoMo and AMA-Bench. Across our searches, verification grounded in evidence helped both intermediate steps and final answers, at lower cost at intermediate steps, while critics asked to rebuild finished answers broke more than they repaired. On the three memory benchmarks, Vestrum also scores above the evaluated GEPA configurations in every paired evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic AI, Just-In-Time harnesses, agentic memory, self-verification, adaptive agent architectures, long-horizon agents, retrieval, decomposition, knowledge synthesis, continual improvement, autonomous agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zAefOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1(−1)
Soundness
2
Read review
Paper summary

Vestrum is a loop for improving an agent harness from failure traces without changing the underlying model. An LLM reads development traces and describes failures as recognizable "failure classes" within "task kinds". It then proposes changes along four dimensions: verification, retrieval, decomposition and knowledge synthesis. The proposals are screened for task-specific leakage, and a bundle is kept only if it improves development results, and in some settings validation results too. A lessons file carries experience across epochs.

The learned harnesses improve over their starting points:

  • starting from GAM, on UltraHorizon, LoCoMo and AMA-Bench;
  • starting from Claude Code, on 8 held-out Terminal-Bench 4 Hard tasks and on a spatial-transcriptomics cell-type annotation task.

Based on its development record, the paper also argues that verification works best when it is grounded in evidence and placed where intermediate results are committed, while critics that rewrite final answers do more harm than good.

Critical strengths

The results are on held-out data and the bookkeeping is careful. Development, validation and test are disjoint, and harnesses are frozen before testing. The paper also keeps a record of what was tried and rejected, including negative results (Table 19), which I appreciated and rarely see.

The editable surface is broader than prompt optimization: it includes memory construction, deterministic views and hooks. Gains go in the same direction across seeds and replicates in all five settings.

Critical weaknesses
  1. All evaluation stays inside the benchmark the harness was tuned on. As a result, I can't tell whether Vestrum produces better harnesses in general or harnesses fitted to these benchmarks. Development, validation and test always come from the same benchmark, and for PathBench even from the same slide.

    The admitted mechanisms clearly encode benchmark structure:

    • per-rule-class notebooks, and a commit gate that is disabled in UltraHorizon's biology environment;
    • event and object views that match AMA-Bench's question axes;
    • a "never refuse" rule on a LoCoMo subset that excludes unanswerable questions;
    • reference tables built from labelled sections of the same slide.

    Improving an untuned GAM or Claude Code using in-distribution development data is expected. The question for this paper is whether the procedure produces something that transfers. That is what the overhypothesis framing claims, and the conclusion calls Vestrum "a common procedure … across different workloads". A frozen harness evaluated on a different benchmark of the same kind would address this, for example the LoCoMo harness on LongMemEval, which you cite.

    Relatedly, the only baseline is GEPA. GEPA edits prompts only, and on UltraHorizon it ended up keeping GAM's original prompt. HarnessFix and Meta-Harness, which you discuss as the nearest neighbours, both report Terminal-Bench 2 results with code available. HarnessFix already compares against both GEPA and Meta-Harness.

  2. The verification lesson is not supported by test-time evidence, and the paper's own data partly contradict it. This lesson appears in the title, the abstract and Section 4.

    • No verifier was ablated at test.
    • On AMA-Bench, the verify-on-build check was designed from GPT-4o-mini failures and repaired 17 of 18 notes. With the test model, it checked 107 notes and repaired none (App. N). So the AMA-Bench gain cannot come from verification.
    • This also suggests the diagnosed failures are specific to the model, which the task-kind / failure-class framing does not account for.
    • On Terminal-Bench, the edit gate "had nothing to refuse" during development, and its effect was not measured.
    • The remaining evidence is too thin and mixed to support a general claim about where and how verification should be placed. It consists of counts such as 9 vs 5 and 6 vs 2, from development sets of a few tasks or a single conversation, obtained with different models and harness versions.
Questions for authors
  1. Can you evaluate frozen harnesses on different benchmarks of the same kind, without any further search?
  2. How variable is the outcome across independent searches? Each setting reports a single searched harness. Even 2–3 searches in one memory setting would help.
  3. Which steps in each search involved human decisions? Were the per-setting admission criteria in Table 8 fixed before the searches were run?
3 comments

Thank you for your review, and for noting the record of what was tried and rejected. Appendix M and Table 19 keep it.

W1 (evaluation inside one benchmark). The claim is scoped to task kinds. The fixed harnesses are tested on held-out tasks of the same kinds, and on the memory benchmarks under a different test model (Table 2). UltraHorizon is the exception: development used only the grid environment by accident, so the grid-learned mechanisms also run on the sequence and biology test episodes, an out-of-development test (Appendix G). No other benchmark was tested (Section 6).

W2 (mechanisms encode benchmark structure). Scoping to task kinds is the design. A task kind is an environment, question category or probe axis (Section 3.1). The generalization filter refuses development facts, not fit to a benchmark's task kinds (Section 6). On the cases you list:

  • The commit gate is off in the biology environment because that environment has no machine-checkable rules to replay (Appendix M).
  • The LoCoMo gains concern the answerable categories (Section 6).
  • The PathBench tables come from the development sections only (Appendix E.1). The test sections come from the same slide, so reference tables built from the development sections are allowed there.

W3 (gains are expected; transfer claim). The gains are not automatic. On the same data, GEPA kept GAM's original prompt on UltraHorizon (Appendix K). The revision removes the overhypothesis framing and the "common procedure" sentence.

W4 (baselines). GEPA is the only learned baseline (Section 6). Meta-Harness and HarnessFix rewrite harness code they own and learn from tens to hundreds of tasks. Vestrum targets closed harnesses and one to ten episodes (Section 2; Table 7).

W5 (verification lesson).

  • No self-verification was ablated at test (Section 6). The revision presents the lesson as what the loop kept and dropped in development (Section 5.1).
  • The revision removes the "107 notes" sentence; the paper does not attribute the AMA-Bench gain to verify-on-build.
  • The mechanisms are not bound to one model: harnesses learned with Haiku 4.5 and GPT-4o-mini improve Gemini 3.7 Flash (Table 2).
  • The LoCoMo auditor counts are small, but each comparison is paired within a question and graded blind (Appendix M).

Q1. Not run. Your example, the frozen LoCoMo harness on LongMemEval, is the transfer test Section 6 lists as the next experiment.

Q2. Unmeasured: each setting has one search, because the budget did not allow repeated searches (Section 6).

Q3. The admission criteria were fixed before each search (Section 3.5; Table 9), and task kinds were assigned before any proposal (Section 3.1). Before each search, the authors fixed the splits, the editable surface, the task-kind definitions, the admission criteria and the budgets. Diagnosis, proposals, filtering, evaluation and the lessons were made by the loop's model calls.

Thank you for the detailed response and the revision. The revised framing matches the evidence better, and the new test-time ablation of self-verification partially addresses my point that no verifier was ablated at test.

However, my central concern W1&Q1 is still not resolved; task kinds are defined inside each benchmark, so held-out tasks of the same kinds are the in-distribution test split. I don't think it's resolvable by simply rephrasing your claim, which, in other words, will downgrade the paper's informativeness as well. I believe that most of my proposed experiments are easy to run and easy to testify. So I will keep my score.

AuthorsOpenReview ↗

New results for W1, W5 and Q1

Thank you again for the detailed review. We ran the experiments behind three of your points, and the revised PDF now includes them.

W1 / Q1: transfer to another benchmark (Appendix O.2, Table 23). We ran the frozen LoCoMo harness and GAM on a stratified sample of 150 LongMemEval_S questions (Gemini 3.7 Flash, three seeds, the same blinded judge). Nothing was tuned on LongMemEval.

QuestionsGAMVestrum (LoCoMo harness)
Answerable (140)87.4 ± 2.2%89.3 ± 1.4%
Abstention (10)53.3%10.0%
All (150)85.1 ± 2.1%84.0 ± 1.3%

The learned harness carries over to answerable questions, where it leads GAM (21 question wins, 13 losses). On the ten abstention questions, its LoCoMo rule to commit to a best-supported answer makes it answer. Section 6 names this rule as the limit of the LoCoMo gains. It costs about 2.9 points here, so the harness trails GAM by 1.1 points over all 150.

W1: UltraHorizon outside development (Appendix O.1, Table 22). Development used only grid episodes. Split by environment, the test runs give:

EnvironmentGAMVestrumDifferenceSeeds positive
grid (development)56.570.8+14.33/3
sequence (never seen)45.263.3+18.13/3
biology (never seen)25.519.5−6.00/3
all47.659.8+12.23/3

The gain carries over to the sequence environment, which the search never saw, and is larger there than on grid. In biology no rule can be checked, so the commit gate is off (Appendix M), and the harness scores below GAM.

W5: self-verification at test. Switching the kept checks off gives up about half of the gain (Table 20, Appendix M.1):

Vestrumwithout self-verificationGAM
UltraHorizon, grid test episodes70.8 ± 3.164.1 ± 3.656.5 ± 1.5
LoCoMo, judge accuracy75.1 ± 0.8%73.2 ± 0.5%71.5 ± 0.8%

Your doubt about the small auditor counts was well placed. In development the auditors broke more correct answers than they fixed, and a test run changed that picture (Appendix M.2, Table 21). We ran the LoCoMo every-answer auditor on 3,824 test answers (three GAM seeds, one Vestrum seed). With Gemini 3.7 Flash it proposed no hedges, which was its failure in development with Haiku 4.5. It fixed 44 answers and broke 5. Most fixes (32) completed answers cut off at the 256-token answer limit that both systems share; on complete answers it fixed 12 and broke 4. The revision now states the auditor result as a development finding (Sections 1 and 5.1) and reports these test counts, and the abstract reports the test ablation instead.

Q2 and W4. We would have liked to run both. The runs above evaluate frozen harnesses and fit within our remaining budget. A full search costs much more, and after pilot runs our budget covered one search per setting. Repeated searches with several seeds, and other harness searches such as Meta-Harness, which learn from tens to hundreds of tasks, were therefore out of reach; Section 6 states both. Our aim was to show that an existing harness can be improved from one to ten long episodes, and the results above add held-out evidence for it.

Reviewer_mU1tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Vestrum identifies failure classes from agent traces, scopes changes by task kind, and searches mechanisms for verification, retrieval, decomposition, and memory. Candidate changes undergo screening and bundle evaluation before admission, while accumulated lessons guide later search. The paper evaluates the resulting harnesses on memory tasks, terminal tasks, and cell annotation, and discusses when verification mechanisms help or fail to help.

Critical strengths

1.Improvements extend across different settings, and the memory evaluations also consider changes in the task model between development and testing. This supports the practical potential of revising harnesses from failures beyond an isolated task demonstration. 2. The procedure is concrete. Failure predicates, applicability scopes, candidate bundles, and admission rules explain how the authors turn failure traces into usable changes. 3. The paper retains ineffective and harmful verifiers and discusses why they did not help. This is more informative than reporting successful harnesses alone and shows that verification does not automatically improve performance.

Critical weaknesses
  • GEPA primarily modify the prompts, whereas Vestrum can change tools, memory operations, and control flow; search models and costs also differ. The advantage could reflect the broader set of allowed changes rather than the proposed failure classification and screening procedure.
  • The contribution of verification is not isolated. Some verifiers make no repairs in the final setting, some prerequisites are already met by other procedures, and placement comparisons also change routing. Gains from the complete harness therefore do not establish that verification or its placement is responsible.
  • Evidence for search stability is limited. Repeated evaluation of the selected harness measures its performance variability, not whether an independent search would discover useful mechanisms again. The cumulative ablations do not resolve this distinction.
Questions for authors

-How does simple trace-reflection search compare with Vestrum when the editable surface, model, and budget are matched? Does an existing experiment already provide this comparison?

  • With routing, memory, and other procedures fixed, what changes in task performance and actual intervention when verification alone is disabled?
  • Do the reported repetitions include independent searches from the beginning, or only evaluations of the selected harness? If independent searches are available, are their gains consistent?
6 comments

Thank you for your review.

W1 (broader editable surface than GEPA). The broader surface is the method: Vestrum changes memory operations, tools and control flow, which a prompt cannot reach (Section 4.2). We agree that the GEPA comparison does not separate the surface from the procedure (Section 6). The procedure's parts are tested on one fixed surface in Table 4. GEPA's task model during search was the test model itself, and Vestrum used an estimated quarter of GEPA's task executions on LoCoMo and AMA-Bench (Appendix K).

W2 (verification not isolated). Correct. No self-verification was ablated at test, so the held-out gains belong to the whole harness (Section 6). The revision presents placement as a development finding, and it says that the LoCoMo placements also differ in routing (Section 5.1).

W3 (search stability). Correct. Each setting has one search, so the repetitions measure execution and judging variability only (Section 6).

Q1. No experiment matches surface, model and budget against an outside trace-reflection search. The closest is Table 4, on one surface and one task model. A loop that reads traces but lacks the lessons, ceiling probe, failure classes, generalization filter and scope gained 2 development checks at 1.7× cost, against 16 at 1.32× for the full loop.

Q2. Not run at test. In development, with the question and evidence fixed, the LoCoMo auditor (an answer-level critic) changed 22 answers: it improved 2 and broke 6 (Appendix M). At test, the kept claim judge edited claims on 4–5% of questions (Table 6).

Q3. The repetitions are evaluations of the selected harness; we ran no independent searches.

Thanks for the authors' efforts during the rebuttal. I have read the rebuttal content and I still have some concerns on them. If the authors would further clarify the following concerns, I would consider to raise the score.

the authors have acknowledged that there are no test-time verification ablation and the improvement cannot be contributed to the veriy-on-build. the authors' solution is to narrow the conclusion but not to present more experimental results to demonstrate their initial conclusion. I think revising the claims is reasonable, but it does not address my original concern about the actual contribution of verification. it is recommended to try to do more exps and demonstrate the points.

i appreciate the authors' clarifications. if the authors provide some new evidence to try to tackle the concern, i would reconsider the final judgement to this manuscript.

Thank you. We ran the experiment from your Q2 on two benchmarks: each frozen harness was run on its test set with self-verification switched off and everything else unchanged (Gemini 3.7 Flash, three seeds). On UltraHorizon this removes the commit gate's rule replay; on LoCoMo, the claim judge and span checks.

Vestrumwithout self-verificationGAM
UltraHorizon, grid test episodes70.8 ± 3.164.1 ± 3.656.5 ± 1.5
LoCoMo, strict accuracy75.1 ± 0.8%73.2 ± 0.5%71.5 ± 0.8%

Removing self-verification lowers the score at every seed in both settings, by about half of Vestrum's gain over GAM. The revision adds this as Table 20 (Appendix M.1), with one sentence in Section 5.1.

Reviewer_eCPVOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

An agent harness is software that determines how a language model reads information, uses tools, maintains memory, and checks results, and because a single evaluation requires long interactions, the cost of improving it is high. This paper presents Vestrum, which improves harnesses based on the overhypothesis that, in tasks of the same task kind, similar failures recur and their solutions also work for tasks of the same task kind. Vestrum organizes failures found in execution traces into failure classes and proposes harness changes along four design dimensions: verification, retrieval, decomposition, and knowledge synthesis. The proposed changes are evaluated together on development tasks and are incorporated only when they pass the admission criteria, and they are applied only to the task kind in which the corresponding failure was observed. It also records the results in a lessons file and uses them for subsequent proposals. Through this, the paper claims that performance on test tasks not used for development improves in five experimental environments starting from GAM and Claude Code, and that it obtains higher scores than GEPA on three memory benchmarks. It also reports the observation that verification that checks results against recorded evidence helps, but verification that has the model re-examine and rewrite an answer it has already given makes more correct answers wrong.

Critical strengths
  1. Vestrum shows that, when applied in diverse environments such as memory systems, coding agents, and scientific analysis tasks, it can improve performance.
  2. On the three memory benchmarks, the improvement is maintained even when testing with a model different from the one used during search, which shows that the effect of the learned harness is not limited to the model used for search.
Critical weaknesses
  1. It is not apparent what has been newly added compared to prior work, and what has improved as a result. Among Vestrum's components, trace-driven harness revision has already been presented in Meta-Harness and others, and failure classification in MAST and others. To claim novelty specific to Vestrum, the paper should present what problem existing methods have not solved and how this paper solves it. In addition, all learned harness changes are evaluated only within the same benchmark as the development tasks, so one cannot tell whether they also work for tasks of the same task kind in other benchmarks, as the overhypothesis states, or whether they are fitted only to that benchmark.
  2. Baselines are insufficient, and it is not isolated whether the gains come from the paper's design. Comparisons are mostly made against the starting harness, and the comparison with GEPA, a method of the same family, is also not a comparison under the same conditions, because the editable surface and the search model differ. Also, the loop ablation on Terminal-Bench is the result of searching only once per variant, so it is difficult to distinguish whether the differences are due to the components or due to chance in the search. A comparison with prior methods under the same conditions, or at least repeated searches in one environment, seems necessary.
Questions for authors
  1. According to Appendix D, the search on Terminal-Bench is the first epoch of the loop. If so, is the lessons file removed in Table 3 the initial lessons? Also, how were the initial lessons and the PRINCIPLES.md in the Supplementary Material written?
  2. Looking at the Supplementary Material, it seems that in PathBench Vestrum is additionally provided with knowledge.md and others compared to the baseline. Please confirm whether the information received by the two systems differs.
  3. The PathBench scoring method described in Appendix K and the scoring method in the Supplementary Material seem not to match each other. Please check.
8 comments

Thank you for your review.

W1 (novelty; within-benchmark evaluation). We build on two earlier ideas and credit both: raw traces (Meta-Harness) and failure classes (MAST). What is new is the setting and how Vestrum handles it. Most earlier searches rewrite code they own and learn from tens to hundreds of tasks. Vestrum changes single components of harnesses it does not own, from one to ten episodes, and filters every mechanism before it runs (Section 2; Table 7). The claim is scoped to task kinds, and held-out tasks of those kinds test it. On the memory benchmarks, the test model also differs from the one the search used (Table 2). Transfer to another benchmark was not tested (Section 6).

W2 (baselines; single-search ablation). GEPA is the only learned baseline, and its editable surface differs (Section 6). That difference is the point of changing components, because a prompt cannot add a memory operation (Section 4.2). GEPA's task model during search was the test model itself, Gemini 3.7 Flash, while Vestrum's was Haiku 4.5 or GPT-4o-mini. Each ablation variant is one search (Section 4.4).

Q1 (lessons file, PRINCIPLES.md). The submitted Appendix D wrongly called the reported run the first epoch. The reported Terminal-Bench harness comes from the loop's third epoch. The first two epochs did not improve the development set. After each one, the loop's meta-review wrote task-free lessons to the lessons file, and the third epoch's proposer read them. The supplement's loop files call these "starting lessons"; they are the lessons as they stood before the third epoch. The "− lessons file" row of Table 4 (Table 3 in the submission) runs the third epoch without these lessons: it gains 10 development checks instead of 16. PRINCIPLES.md is a fixed design document that the proposer reads and the loop never edits. It holds the rules for each design dimension, scoping by task kind and the cost cap (Appendix D). The authors wrote it before the reported search.

Q2 (PathBench inputs). Both systems receive the same input files, and run Claude Opus 5 on the same brief. Vestrum additionally learns from its failures on the two development sections and writes a knowledge file (rules, reference tables built from the development sections, and a helper) that the agent reads as memory. Nothing comes from the test sections (Section 4.3).

Q3 (PathBench scoring). The submitted Appendix K described an earlier grader. The reported numbers come from score_run.R (vocabulary mode) and partition.py in the supplement:

  • labels are joined on cell ID;
  • scoring uses the brief's 16 labels, and Squam3 cells (no label in the brief) are dropped;
  • accuracy is over the joined cells, and macro-F1 is over the 15 classes other than Unknown;
  • ARI and NMI are computed on the raw partitions.

All numbers are agreement with the bioinformatician's labels, and none change. Appendix J now describes this script.

Thank you for the response and the revised version. I have confirmed that, through the answers to my questions, the descriptions of the PathBench scoring method and of the epoch in Terminal-Bench have been corrected. However, the revised version contains no additional experiments, and the changes were limited to removing existing claims and adding new claims.

Removing these claims (the overhypothesis, the claim that it is a procedure that applies commonly across multiple workloads, and the conclusion that verification helps) has made the paper's descriptions more accurate. However, this is a reduction in the scope of what the paper demonstrates, not a strengthening of the contribution.

There is a gap between the two claims newly put forward as contributions and what the experiments show.

First, for component-wise harness learning, the paper cites as its difference from existing methods that it modifies only the one component where the error occurred, and only in the corresponding task kind. However, the experiments consist only of the improvement over the base harness and the comparison with GEPA, which modifies only the prompt. These only show that the loop as a whole is effective, and that a broader editable surface is better than modifying only the prompt. They do not show that restricting modifications to the component level is better than modifying the same harness without restriction. Although such a comparison is possible because GAM's code is publicly available, it was not performed.

Second, there are two pieces of evidence presented for the claim that the generalization filter enables learning from a few episodes: the number of proposals refused by the filter, and the result of a single search in Table 4 in which the filter was cumulatively removed together with failure classes and task-kind scope. The number of refusals is only a record that the filter operated and does not show its effect on performance, and a test-set run with the filter switched off remains future work in the revised version as well. Also, in PathBench, reference tables built from the reference labels of the development sections are allowed, so the boundary between what the filter blocks and what it allows is also unclear.

An academic paper should show what problem existing methods have not solved, how the proposed method solves it, and, through experiments, that this solution is actually the cause of the improvement. The revised version reorganized the description of the problem and the contributions, but there are still no experiments that demonstrate those contributions.

This is not a request for large-scale new experiments during the rebuttal period. What I requested in the initial review was a comparison under the same conditions, or at least repeated searches in one environment. The test-set run with the filter switched off, which the revised version itself lists among the next experiments, is also an experiment of limited scope. This is the minimum evidence needed to verify the contributions the paper claims, and something that should have been included in the paper from the beginning. In a situation where only the definition of the contributions has changed and the evidence supporting them remains the same, I think there is no reason to change my assessment, and I will keep my score.

Thank you for confirming the corrections and for the clear statement of what remains.

We agree that the revision adds no experiments, and that the evidence for the filter is the Table 4 search and the refusal records, not a test-set comparison.

Two clarifications of what the revised paper claims:

  • Component-wise learning is the means of improving a harness the user does not own. Claude Code's code cannot be modified, so on Terminal-Bench and PathBench an unrestricted rewrite of the same harness is not available. On GAM such a comparison is possible, and we did not run it. The wording "only in the corresponding task kind" overstated the method: each change targets a failure that recurs across tasks of one kind, and the whole harness is tested on held-out tasks. On Terminal-Bench, the learned harness also classifies each request by task kind (Section 4.1). The current revision states it this way.
  • What the filter allows is set per setting (Appendix E.1). On PathBench, whose test sections come from the same slide, rules and reference tables built from the development sections' labels are permitted; the filter's reviewer refused 5 of 19 proposals there as specific to the development sections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.