- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to improve an agent’s prompts, skills, tools and workflow while keeping its model fixed. Harness-SFT chooses edits that make the agent follow teacher demonstrations more closely, while Harness-RL chooses edits using task rewards. The proposed method, STHT, first learns from demonstrations for tasks the initial agent failed, then continues with reward-based search. Experiments cover SpreadsheetBench, a DocVQA pilot, ALFWorld, and a coding subset of EvoAgentBench. STHT has the highest reported mean of the methods in Table 1 in all 8 settings, averaging 80.37% success across the 5 primary settings versus 77.28% with randomly selected demonstrations.
Critical strengths
-
The distinction between what an edit changes and how it's selected is useful. It's a clear way to organize methods that modify different parts of an agent. The authors are also clear that the comparison with model training is just an analogy. They don’t claim that editing a harness works like gradient-based training.
-
The experiments include a few different useful controls. The comparison with random task selection uses the same number of demonstrations, and the gains hold across different harnesses, reward optimizers, and teachers. On ALFWorld, targeting failures also beats selecting successful tasks or tasks whose executions differ most from the teacher’s. I appreciated the comparison between starting harnesses with almost identical performance. The supervised starting point still supports better subsequent search, which suggests that its benefit goes beyond just the higher initial score.
Critical weaknesses
-
The variation across runs is not clear. The main gains over random selection range from 1.78 to 4.64 percentage points, but the paper doesn't provide standard deviations, intervals, or statistical tests. The results are described as three-run averages, but Section 6.1 presents the ALFWorld result as 239/274 solved tasks, and Appendix E.3 refers to “14 of 32 and 21 of 32 groups in the reported run”. These counts could be consistent with averages, but the aggregation needs explaining.
-
I’m not sure the comparisons are fair on cost or on baseline strength. STHT gets to the validation threshold in fewer rollouts, but it costs 8.7% more overall than random selection, and a lot more than RL alone. It's not clear whether that cost includes generating and verifying teacher demonstrations. There's also no RL-only comparison given STHT’s total budget. The default optimizer, Parent Acceptance, is the weakest method in Table 12. Bandit Allocation reaches 91.61% on ALFWorld there, compared with STHT’s 87.23% in Table 1. The two use different training setups, so this isn’t a direct comparison. But the question of whether the budget would be better spent on a stronger search is still open.
-
Parts of the protocol need clarifying. Section 4 says training tasks choose candidate edits, but the SpreadsheetBench code sets the selection split to validation. If so, validation AUC and time to threshold are measured on selection data, but the test results are still held out. The same ALFWorld result also shows up as Table 11’s Success-only row, Table 12’s Parent Acceptance row, and Table 1’s RL-only entry, each described differently. Finally, it’s unclear what the modifier was told in each Table 11 condition, which matters for reading it as RL-like behavior.
Questions for authors
-
Could you report per-run results and uncertainty estimates for Tables 1-4? How do the counts in Section 6.1 and Appendix E.3 relate to the three-run averages, and does Appendix H use the same configuration as Table 1?
-
When deciding whether to keep a candidate edit, which tasks do you evaluate it on? Training tasks, the validation split named in the SpreadsheetBench code, or a separate selection fold like in Appendix H? How many tasks and rollouts go into each decision? Are those rollouts, and the initial failure-screening runs, counted in the 256-rollout budget and in time to threshold? How does validation AUC select a single checkpoint?
-
Does total cost include teacher generation and verification? Could you compare STHT with reward-only search at the same total cost?
-
Which editing scopes produced Table 11’s Success-only and Table 12’s Parent Acceptance rows? What reward instructions did the modifier get in each Table 11 condition, and what did the shared-candidate comparison described in Appendix D.2 show?
2 comments
A brief note on the response (Submission 54869)
Dear Reviewer Ro4A,
Thank you again for your detailed review of both the paper and the code. With the discussion period ending soon, we wanted to check whether our response has resolved your questions on the statistics, protocol, and cost comparisons:
- Uncertainty and aggregation (Tables 1–4, Appendix B.8 & H): added mean ± SD across the three seeds, explained the integer-count rounding rule (Eq. 33), and clarified the single-seed diagnostic setup in Appendix H (Table 18).
- Selection split and rollout accounting (Appendix B.4, Tables 11–13): confirmed that per-round retention uses only training tasks (not the validation split), clarified how , , and Stage-1 costs are accounted, and stated the checkpoint rule.
- Equal-cost reward-only search and teacher cost (Tables 5, 13) & Table 14–15 scopes/prompts (Appendix D.1–D.2).
Please let us know if anything remains unclear; we are happy to provide any further details.
Best regards, The Authors
We thank the reviewer for the careful reading of both the manuscript and the code.
Q1. Aggregation, uncertainty, and Appendix H
Table 2 (STHT vs. Uniform): successful-round rate 0.711±0.036 vs. 0.618±0.037; validation AUC 0.804±0.007 vs. 0.776±0.007. Table 3 (failure-targeted vs. Uniform): 0.716±0.118 vs. 0.625±0.121 rounds with success; AUC 0.817±0.015 vs. 0.790±0.015.
Q2. Retention split, rollout accounting, checkpoint selection
SELECTION_SPLIT_NAME. Inmodifier-system-v1this variable is only a label in the modifier's boundary warning, and its value "SpreadsheetBench validation" was misleading. We verified that the runner's retention step evaluates candidates only on training tasks, and we will correct the label in the repository. The modifier never sees validation tasks or traces; validation is used only for AUC, Tq, and selecting H⋆.Q3. Teacher cost, equal-cost reward-only search, optimizer strength
Q4. Scopes and modifier instructions for Tables 14–15
REWARD_CONTRACT(Appendix I), and the shifts compound to 9.80 tool calls, 8.20 steps, and 64.96% verification coverage on held-out tasks.