- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the distinction between a harness that improves immediate, supported execution and one that induces model capabilities that persist and transfer after training. It introduces a crossed training/deployment-harness evaluation protocol and trains a Feedback-Agent with token-level PPO to produce persistent executable harness revisions before each fixed-budget inner-training stage. The optimization signal is based on post-update transfer across a fixed family of reference harnesses, thereby assigning credit through subsequent model improvement rather than immediate execution success alone.
Critical strengths
-
The paper isolates an important evaluation confound: evaluating a training harness only while it remains attached conflates learned model capability with runtime assistance. The crossed-harness protocol directly measures this distinction across multiple models, domains, and deployment harnesses, making the evaluation conceptually valuable.
-
The empirical design is comparatively rigorous for harness optimization. Three independent Feedback-Agent fits, paired deployments, disjoint data splits, and targeted ablations of the reward, revision persistence, and credit-assignment mechanism provide meaningful support for the central claims.
Critical weaknesses
-
The reported gains are heterogeneous. In particular, TAME underperforms both the selected fixed teacher and the frozen reviser on Telecom, indicating that it is not a consistently dominant optimizer. Moreover, the transfer objective depends on a designer-specified reference family and assigns substantial weight to the minimal harness, which limits the generality of the transfer claim.
-
The method is computationally expensive and operationally complex. In the 100M-token ALFWorld setting, it more than doubles the reported GPU hours and model-call tokens relative to the selected fixed protocol. Reproducibility will also depend on releasing implementation details that are not currently specified or explicitly committed to in sufficient depth.
Questions for authors
-
Could the authors provide a consolidated table reporting common-harness, minimal-harness, held-out-harness, and attached-harness scores for TAME and every baseline across all model-domain settings, together with an analysis of the Telecom failure case?
-
How sensitive are the conclusions to the composition and weighting of the reference-harness family, particularly the one-half weight assigned to the minimal harness?
3 comments
Dear Reviewer Pk1i,
We would welcome your thoughts on the additional analyses in our November 13 response and revised paper.
Following your suggestions, we consolidated the endpoint results in Tables 14–17 and added reference-weight sensitivity and leave-one-reference-out retraining analyses (Table 10; Figure 21). The Telecom follow-up compares the methods from the same executive-harness initialization (Appendix D.7; Table 12), while Appendix D.4 separates teacher-construction costs from recurring deployment costs.
Do these additions help resolve your concerns about the scope of the gains and the practical tradeoff? If any concern remains, please let us know so that we can address it during the remaining discussion period.
Thank you again for your careful feedback.
Dear Reviewer Pk1i,
Thank you for your constructive suggestions.
- Our consolidated results show competitive, considerable gains with TAME over the fixed and frozen controls, including the selected fixed teacher and the frozen reviser, in most (11 of 12) settings throughout the tested reference-weight family, supported by leave-one-reference-out retraining.
- In the matched-initialization Qwen3-4B/Telecom follow-up, TAME competitively reaches 49.0 common transfer versus 45.4 and 46.7 for the fixed and frozen controls (Table 12).
- The revised cost accounting also makes the benefits of reviser reuse explicit.
Together, we believe that these findings strengthen the evidence for transferable improvements and address your questions about their scope and practical value. Would you kindly consider updating your recommendation to 3 (weak acceptance)?
Thank you so much for your consideration.
Best regards,
The authors
Response to Reviewer Pk1i
Thank you for the constructive review. We added a new analysis for each point.
W1/Q1: one consolidated table, and differences across settings. Tables 14–17 list common, minimal, Hheld, Hnew, and own-attached scores for every method in all 12 model–domain settings: selected fixed, information–executive mixture, one-shot, frozen reviser, execution search, evolved fixed, native-system reward, transfer-based search (where run), and TAME. Excluding search, TAME has the highest common transfer in 11 of 12 settings. These include Telecom with Llama-3.2-3B (+4.6 / +2.3 over fixed / frozen) and with Qwen2.5-3B (+3.6 / +2.4). The one exception is Qwen3-4B/Telecom (−1.7 / −1.9), discussed next. Native-system reward has the highest own-attached score in most settings, as expected, since it optimizes the attached system rather than the shared harnesses.
Telecom (App. D.7, Tables 11–12). On Qwen3-4B/Telecom, the validation-selected fixed teacher is Hexec. To test whether the starting harness explains the gap, we started TAME, the frozen reviser, and the fixed teacher from Hexec. The learned reviser stayed frozen as before, and nothing else changed (3 blocks × 5 deployments each):
From this start, TAME beats the fixed teacher by 3.6 points and the frozen reviser by 2.3, similar to its margins elsewhere, and it is ahead on every evaluator. Its revisions add 0.79 points on average over KEEP, against 0.20 for the frozen reviser (Fig. 6b). Across all four Qwen3-4B domains, TAME's revisions add 0.85–1.02 points over KEEP on average (Fig. 6c). The original gap therefore comes from the starting harness, not from the learned revision rule. The starting teacher is chosen on the selection split, not on test results.
W1/Q2: the reference family and the minimal-harness weight. Table 10 rescores all fixed endpoints with Jα=αJmin+41−α(Jdocs+Jinfo+Jexec+Jnull) for every α∈[0,1], and with each reference left out in turn. Against the fixed and frozen controls, the sign never changes in any of the 12 settings (TAME is ahead in 11). Against transfer-based search, the sign is stable in 6 of 7 settings; it flips only for Llama/ScienceWorld at small α. Within this family of weightings, the ½ minimal weight therefore does not drive the conclusions.
Rescoring keeps the trained models fixed, so we also retrained the reviser with each single reference removed from both reward and selection (Fig. 21, App. G.6). Common transfer drops by only 0.4–1.3 points, less than TAME's margin over both controls in each of the 11 settings, and Hheld drops by at most 0.9. Removing the minimal harness hurts most: the worst score across all eight harnesses falls by 1.5–2.6 points, which supports giving it the largest weight.
W2: cost and complexity. Table 29 now reports cost next to quality. We see transfer-based search as the relevant alternative, since it optimizes the same objective by selection instead of learning. It costs 148.2 GPU-h per deployment, against 106.5 for TAME with construction amortized over 20 deployments, and it still has lower common transfer in 5 of 7 settings. Because search pays its full cost every time, TAME is cheaper from the 13th deployment on. TAME's recurring cost per deployment (39.5 GPU-h) is close to that of the fixed (36.6) and frozen (40.9) arms. Its overhead over the fixed protocol therefore shrinks as the one-time cost is shared: 2.3× at 20 deployments, 1.4× at 100, and approaching 1.08× (App. D.4).
Reproducibility. The revision adds the Hnew specification with its hash, the partition procedure and counts, and the search protocol. We will release all code and implementation details, including Feedback-Agent training and deployment, the revision validator and editable harness surface, the reference and held-out harnesses, and the task-partition procedure.
We welcome any further questions.