- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether the effects of individual agent harness components can transfer across language models. The authors evaluate four harness components on airline, retail, and telecom tasks by comparing a full harness with configurations where one component is removed. They analyze how component effects vary across models and tasks, and whether observations from existing source models can predict component effects on a held-out target model.
The main findings are that 84.3% of the evaluated configurations show no change in binary task outcomes, active source-model sets differ across components in 68.3% of target-task cases, and 94.3% of nonzero target effects have at most one active source model. The authors also evaluate failure-profile-based weighting for cross-model prediction, finding only a small and inconclusive improvement over uniform averaging. Conservative screening reduces utility under equal benefit and harm costs.
Overall, the paper highlights the limited recurrence of observed harness component effects across models and motivates component-specific validation before reuse.
Critical strengths
1. Relevant and Practical Research Question
The paper addresses a practical question: whether harness components that work well on one model can be reused on another. Studying individual components rather than comparing only complete harnesses is a useful perspective for understanding cross-model differences and may help guide future harness design.
2. Empirical Analysis and Transparency
The paper provides an empirical analysis of component effects across different models and tasks, including effect sparsity, source-model variation, and cross-model recurrence. It also evaluates prediction and screening methods and reports negative or inconclusive results rather than overstating their effectiveness. The authors openly discuss several limitations in the appendix, which helps readers understand the boundaries of the findings.
Critical weaknesses
1. Unclear Experimental Choices
The paper does not explain why the four harness components (h0–h3) were selected or how representative they are of general harness designs.
The diagnostic tasks are also poorly explained. What are the 100 tasks? How were they selected? Why are they divided into five clusters, and what does each cluster represent?
These missing details make it difficult to assess the methodology, reproducibility, and generalizability of the findings.
2. Data and Experimental Consistency
Appendix A.2 reports inconsistencies between the summary matrix and raw experimental records, including:
- 19 binary-result discrepancies.
- 44 trial-mean discrepancies.
- 91 unresolved records.
Appendix A.3 also reports differences in user simulators and step limits between some reference and ablation runs.
Although the authors acknowledge these issues, they remain unresolved. Since the main findings rely on the measured component effects, these problems raise serious concerns about the reliability of the results. In particular, it is unclear whether the observed differences are caused by the harness components or by other changes in the experimental setup.
Questions for authors
1. Component Selection
- Why were these four harness components (h0–h3) selected?
- Were other components or configurations considered?
- How representative are these four components of general agent harness designs?
2. Data and Experimental Consistency
Appendix A.2 reports inconsistencies between the summary matrix and raw experimental records, including 19 binary discrepancies, 44 trial-mean discrepancies, and 91 unresolved records.
- Since the authors conducted the experiments and processed the data themselves, why are these inconsistencies still unresolved?
- What caused these discrepancies?
- Can the authors reproduce the main findings using verified raw records?
Appendix A.3 also reports differences in user simulators and step limits between reference and ablation runs.
- How can the authors ensure that the observed effects come from the harness components rather than these configuration differences?
- Do the main findings still hold under fully matched experimental settings?
3. Contribution and Practical Significance
The paper mainly provides a post-hoc analysis of harness component transferability. However, it is somewhat expected that different models respond differently to harness components. I am not convinced that reporting these differences alone is a strong enough contribution.
- What new insights does this analysis provide beyond confirming that harness effects vary across models?
- How can these findings guide practical harness reuse or improvement?
I also think it would be more meaningful to move beyond diagnosis toward improvement.
For example, it would be interesting to investigate whether existing harness optimization, self-evolution, or recursive self-improvement (RSI) methods can improve cross-model transferability.
Studying how these methods modify harness components, and why these changes transfer or fail to transfer across models, could provide deeper insights into the underlying mechanisms.
This could be more informative than manually evaluating components through leave-one-component-out ablations.
4. Diagnostic Tasks and Clustering
The diagnostic tasks and clustering process are not clearly explained.
- What are the 100 diagnostic tasks, and how were they selected?
- Why were they divided into five clusters?
- What does each cluster represent?
- How sensitive are the results to different clustering choices?
These details are important for understanding and evaluating the failure-profile similarity used for cross-model prediction.
5. Minor Formatting Issue
On page 10, the AI Use Statement, Reproducibility Statement, and Ethics Statement are numbered. However, the ICLR template appears to use unnumbered headings for these sections.