- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes HarnessWeaver, a continual agent-improvement framework that selects an executor and composes task-conditioned harness components, then validates and retains model-routing or harness revisions based on execution feedback. It evaluates first-encounter transfer and repeated-task recovery on ALFWorld and Omni-MATH, reporting improvements over a frozen reference and two external baselines.
Critical strengths
1.Separating first-encounter performance from repeated-task recovery makes the continual-improvement claims more informative than reporting final-round performance alone. 2. The paper includes reasonably clear visualizations, ablations, and cost accounting, which help assess the system’s behavior.
Critical weaknesses
- The main ingredients, model routing, compositional harnesses, failure-driven revision, and validation before deployment, are closely related to prior work. The paper does not make sufficiently clear what distinctive mechanism or insight their combination establishes.
- The full framework improves results, but Omni-MATH lacks harness-only and model-only controls, while its reported gains appear largely routing-driven. This makes it difficult to assess how much task-conditioned harness composition contributes beyond selecting a stronger executor.
Questions for authors
- What is the specific methodological contribution that would remain if model routing and harness adaptation were considered separately?
- Can the authors provide Omni-MATH adaptation-surface controls, or a matched comparison that isolates the benefit of harness composition from executor routing? This evidence could materially affect my assessment.
4 comments
W2 & Q2: Omni-MATH harness-only and model-only controls
Can the authors provide Omni-MATH adaptation-surface controls, or a matched comparison that isolates the benefit of harness composition from executor routing? This evidence could materially affect my assessment.
Thank you for suggesting these adaptation-surface controls. We have added and completed the Omni-MATH Harness-only and Model-only control experiments. They distinguish changing the executor from changing its execution mechanisms on the same 100-task manifest, with results shown in Section 4.3, Figure 5b.
| Adaptation surface | First encounter | Final |
|---|---|---|
| Harness-only, fixed 9B | 11% | 9% |
| Model-only, frozen initial components | 47% | 56% |
| Full | 55% | 65% |
Full obtains higher first-encounter and final-round success than both single-surface controls, exceeding Model-only by 8 and 9 percentage points, respectively. These completed-run comparisons support retaining both adaptation capabilities, with execution evidence determining which intervention to adopt for a given failure scope. Together with the ALFWorld controls, these results support our experimental conclusion that retaining both model selection and harness adaptation enables stronger continual improvement than restricting adaptation to either side in the evaluated runs.
I appreciate the authors’ response and their careful answers to my questions. I believe my initial score reflects my assessment of the paper, and therefore I would like to keep it.
Thank you for taking the time to review our responses and additional experiments. We appreciate your careful consideration. If there are any remaining concerns that we could help clarify, we would welcome the opportunity to address them. Thank you again for your time.
W1 & Q1: Distinctive methodological contribution
Thank you for distinguishing the evaluation protocol from the question of methodological contribution.
HarnessWeaver takes the perspective that both model selection and harness design can be learned continually from execution experience. Considered separately, the model side accumulates experience about which executor suits which tasks, allowing model choices to be updated from observed outcomes. The harness side accumulates experience about how tasks should be executed, retaining procedural support for search, tool use, and verification as selectable, reusable components. Both turn execution experience into support for subsequent tasks while keeping model weights fixed.
Combining the two introduces a central decision: for the same failure, should the system change the model, change the harness, or change both? HarnessWeaver proposes and compares these three types of candidates through execution on the same failure scope, retaining effective combinations with their applicable task conditions. Experience therefore guides both the selection of existing configurations and decisions about how the current configuration should change. On ALFWorld, Full achieves 58.6%/81.4% first-encounter/final-round success, exceeding Model-only at 48.6%/75.7% and Harness-only at 54.3%/62.9%. Subsequent reuse records show that one retained combination is selected 30 times with 24 successes. These results support the complementary value of the two adaptation capabilities and establish that retained experience is used in later execution.
Our contribution is thus to connect task-configuration selection, intervention after failure, and experience reuse into a continual-improvement process: the model side learns who should execute, the harness side learns how to execute, and their combination learns what should change after failure. Experience-guided search explores candidates, and execution validation determines which combinations to retain. The evaluation approach scores first encounters separately from repeated-task performance to examine whether accumulated experience improves subsequent execution.