- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes harness learning, which trains a proposer model to revise the executable workflow around a frozen solver using execution feedback. The method optionally initializes the proposer through supervised fine-tuning on teacher-generated revisions, followed by RL based on revised-harness performance. At test time, both models remain frozen, while the harness is iteratively modified using target-task development feedback. Experiments on Reasoning Gym and multi-hop QA investigate transfer to unseen tasks, successive revision, and single-step versus multistep training.
Critical strengths
The paper addresses a practically relevant problem: improving agent behavior through executable workflow changes without updating the solver’s parameters.
Critical weaknesses
-
Limited novelty and insufficient comparison with closely related work The framework substantially overlaps with existing methods that learning agent-harness construction and revision. As described in the paper’s related work section, Harness-R1 already combines supervised initialization and reinforcement learning for an editor with a frozen solver, while JIT-Agent studies task-conditioned harness generation and revision. The paper does not clearly identify the contribution that distinguishes its approach from these methods. The meta-learning formulation primarily reframes the revision process, while optimization still uses immediate, per-revision rewards. Without direct empirical comparisons against closely related approaches, the incremental contribution remains unclear.
-
The adaptation setting requires clearer qualification Conventional unsupervised TTA assumes access to unlabeled target-domain data. Here, adaptation uses target-task development examples and explicit supervision. QA feedback includes correctness, gold answers, and supporting-document information. Candidate harnesses are also selected using target-task scores. Although final evaluation examples are held out, the harness has already been optimized using supervision from the same target task. The results therefore establish supervised target-task adaptation more directly than conventional unsupervised TTA. This distinction should be prominent.
-
Performance-dependent task selection weakens the generalization evidence The evaluation subsets are partly determined by observed performance. Appendix B.1 excludes five difficult OOD families from the main comparison. More concerningly, Appendix C.2 and Table 12 select additional multi-step evaluation families using earlier results, including criteria favoring the ordering RL > SFT > Base. Although these choices are disclosed, they introduce selection bias into the aggregate evaluation. The generalization claims should be supported by comprehensive task coverage or a subset defined independently of the compared methods’ results.
-
The presentation is unnecessarily difficult to follow Table 1 compresses heterogeneous information into a crowded layout, making the experimental settings difficult to compare. Figures 3 and 4 contain overlapping information without clearly separating the purpose of each comparison. Across Figures 3–5, numerous colors, legends, markers, and line styles are used without a sufficiently consistent visual encoding. Readers must repeatedly distinguish training configurations, revision protocols, averages, and oracle statistics. The presentation would benefit from fewer redundant panels, standardized symbols, and a clearer separation between development-selected results and held-out oracle statistics.
Questions for authors
See above.
6 comments
W2. The adaptation setting is supervised target-task adaptation
We agree that this distinction is important. Supervised adaptation from a small labeled development set is the setting our method is designed for. Section 4 defines adaptation with a feedback set and a scoring set of labeled development questions and keeps the evaluation questions disjoint from both, and the paper does not describe the setting as unsupervised.
Supervised adaptation from a small labeled set matches how adaptation is done in practice. A coding task, for example, usually comes with a few tests with known expected outputs, which are used to check a solution before it is deployed on inputs it has not seen. Our development set plays the role of those tests, and the held-out set plays the role of the unseen inputs. The same structure is the standard few-shot meta-learning setting: in MAML, the inner loop adapts from a labeled support set, and evaluation uses a disjoint query set. Search-based harness optimizers (AFlow, Meta-Harness, AutoHarness [1,2,3]) also optimize against labeled task scores, and they must run their search anew on every task.
Labels enter our results at three points:
- The single-step means involve no selection: every proposal counts, failed ones at the seed's score. Labels enter only through the round-1 execution report.
- Multistep adaptation on Reasoning Gym keeps a revision only if its development score improves, and QA picks the best of eight by development exact match.
- Oracle best-of-N numbers select on held-out scores, and the paper labels them as oracle statistics.
Selection without labels. In our method, the development gate uses labeled questions to decide which proposed harness to keep. We tested a version of this decision that uses no labels, so that adaptation needs only unlabeled questions from the target task. Our proposer was never trained for this step, so some loss against the labeled gate is expected. The harness is selected in three steps:
- The frozen solver answers 25 unlabeled questions of the task with each harness: the seed and each proposal.
- Where the harnesses give different answers, the frozen proposer checks each answer on its own. It sees the task family, the question, and the proposed answer, with no reference answer.
- The harness with the most answers judged correct is kept. If the seed has the most, no proposal is kept.
The harness is selected on one half of the held-out questions and scored on the other half, and then the halves swap. Every rule in the table, including the labeled gate, is scored the same way, averaged over 17 unseen families with wins/ties/losses in parentheses.
| proposer | seed | random candidate | selection without labels | development gate (labels) |
|---|---|---|---|---|
| Base | 0.621 | 0.339 | 0.723 | 0.760 |
| Single-step RL | 0.621 | 0.596 | 0.796 | 0.834 |
Without any labels, the Single-step RL proposer improves on the seed by +0.176 (14/2/1), leading over the Base proposer by +0.074 (11/0/6). We will update the numbers when all results are out. We will add this label-free variant to the appendix.
Updates on Nov 18th: we have finished all 21 tasks and the table is as below:
| proposer | seed | random candidate | selection without labels | development gate (labels) |
|---|---|---|---|---|
| Base | 0.555 | 0.332 | 0.691 | 0.718 |
| Single-step RL | 0.555 | 0.623 | 0.792 | 0.821 |
Without any labels, the Single-step RL proposer improves on the seed by +0.237 ± 0.070 (18/2/1) and keeps 89% of the labeled gate's gain. Its lead over the Base proposer is +0.100 ± 0.043 (13/1/7), almost the same as under the labeled gate (+0.103 ± 0.041). The Single-step RL proposer without labels scores above the Base proposer with labels (0.792 vs 0.718).
W3. Performance-dependent task selection
Single-step (Appendix B.1). The paper already reports all 26 unseen families. With all 26, the scores are Base 0.282, SFT 0.415, Single-step RL 0.528, and teacher 0.501, in the same order as on the 21 families of the main comparison (0.32, 0.49, 0.62, and 0.56), and the 5 extra families are hard for every method, including the teacher and the fixed tool loop (Table 9). The main comparison excludes 5 unseen families: knight_swap, letter_jumble, modulo_grid, sokoban, and ab. The stated criterion is that both the 4B proposers and the teacher find them hard to address with an algorithmic harness.
Multistep (Appendix C.2, Table 12). The paper's 12 additional multistep families were chosen from a pool of 22 unseen families using earlier results, and we thank the reviewer for pointing this out. To remove this selection, we ran the other 10 families of the pool with the same protocol. In summary, the order of models does not change with single-step RL scoring much higher on the format-varied data. The changes in scores are within the confidence intervals of already reported numbers.
Canonical questions (22 families; seed 0.552):
| method | round 1 | round 5 | round 5 − seed | round 5 − Base | selected 12: round 5 − Base |
|---|---|---|---|---|---|
| Base | 0.573 | 0.657 | +0.104 ± 0.053 | ref. | ref. |
| Single-step RL | 0.634 | 0.662 | +0.110 ± 0.048 | +0.005 ± 0.037 (10/5/7) | +0.047 ± 0.061 |
| Multistep RL | 0.674 | 0.705 | +0.153 ± 0.056 | +0.048 ± 0.035 (11/4/7) | +0.104 ± 0.058 |
Format-varied questions (21 families*; seed 0.519):
| method | round 1 | round 5 | round 5 − seed | round 5 − Base | selected 12: round 5 − Base |
|---|---|---|---|---|---|
| Base | 0.612 | 0.617 | +0.098 ± 0.055 | ref. | ref. |
| Single-step RL | 0.672 | 0.714 | +0.195 ± 0.067 | +0.097 ± 0.035 (9/8/4) | +0.152 ± 0.050 |
| Multistep RL | 0.652 | 0.652 | +0.133 ± 0.058 | +0.035 ± 0.033 (7/7/7) | +0.050 ± 0.051 |
-
One family was excluded from the format-varied comparison since the Base proposer declined to edit in all five rounds.
-
Every trained proposer ends at or above Base on both question formats.
-
The methods rank in the same order as on the 12 selected families.
In the revision, the main text will report the unselected family numbers next to the selected ones, and the Fig. 3 caption will state how its families were chosen. The QA multistep results (Fig. 4) involve no task selection: each benchmark is evaluated in full on its held-out split.
W1. Novelty relative to harness-evolution work Harness-R1 and JIT-Agent, the two most related works, appeared on arXiv on 3 and 26 August 2026, a few weeks before the deadline, and we treat them as concurrent work. We cite and discuss both (Section 2 and Appendix E). Our work differs from them in four ways:
The meta-learning formulation is what makes the generalization across tasks claim precise. It specifies what is adapted (executable harness code), the adaptation rule (the proposer), and how transfer is measured (held-out tasks), so successive revision at test time becomes the inner loop of a learned adaptation procedure. Training uses immediate per-revision rewards, and successive improvement at test time already emerges from this single-revision training (Fig. 4). A faithful controlled reimplementation of the two concurrent methods is not feasible within the rebuttal period, because they use different harness interfaces, benchmarks, and target agents. We will extend Appendix E with the comparison above.