acceptodds
Withdrawn from ICLR 2027

Plug, Play, or Re-Evaluate: Harness Components Across Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An agent harness is the system around a language model that provides the instructions, tools, context, and controls needed for the model to execute a task. Reusing components of this system for new models could reduce repeated testing and tuning, but comparisons of complete harnesses do not reveal which component matters or whether its effect carries over between models. We study each component by comparing the full harness with versions in which one component is removed, then examine whether a component's effect observed on existing models recurs on a new model. We discover three patterns. First, component effects are sparse across model–task–component combinations: removing a component leaves the recorded result unchanged in 84.3% of 1,120 combinations. Second, components exhibit model-specific compatibility: in a separate four-model analysis, the models whose results change differ across components in 68.3% of 360 model–task cases. Third, cross-model recurrence is limited: among 176 cases in which removing a component changes the target model's result, 94.3% have at most one other model showing a change. These findings show that an effect observed on one model should not be assumed to recur on another, motivating component- and model-specific validation before reuse. Source code is available at https://anonymous.4open.science/r/harness-transfer-artifacts-2041/

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Transfer learning, label-free transfer prediction, failure-profile matching, large language model evaluation, LLM agent harnesses

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4B3nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether the effects of individual agent harness components can transfer across language models. The authors evaluate four harness components on airline, retail, and telecom tasks by comparing a full harness with configurations where one component is removed. They analyze how component effects vary across models and tasks, and whether observations from existing source models can predict component effects on a held-out target model.

The main findings are that 84.3% of the evaluated configurations show no change in binary task outcomes, active source-model sets differ across components in 68.3% of target-task cases, and 94.3% of nonzero target effects have at most one active source model. The authors also evaluate failure-profile-based weighting for cross-model prediction, finding only a small and inconclusive improvement over uniform averaging. Conservative screening reduces utility under equal benefit and harm costs.

Overall, the paper highlights the limited recurrence of observed harness component effects across models and motivates component-specific validation before reuse.

Critical strengths

1. Relevant and Practical Research Question

The paper addresses a practical question: whether harness components that work well on one model can be reused on another. Studying individual components rather than comparing only complete harnesses is a useful perspective for understanding cross-model differences and may help guide future harness design.

2. Empirical Analysis and Transparency

The paper provides an empirical analysis of component effects across different models and tasks, including effect sparsity, source-model variation, and cross-model recurrence. It also evaluates prediction and screening methods and reports negative or inconclusive results rather than overstating their effectiveness. The authors openly discuss several limitations in the appendix, which helps readers understand the boundaries of the findings.

Critical weaknesses

1. Unclear Experimental Choices

The paper does not explain why the four harness components (h0–h3) were selected or how representative they are of general harness designs.

The diagnostic tasks are also poorly explained. What are the 100 tasks? How were they selected? Why are they divided into five clusters, and what does each cluster represent?

These missing details make it difficult to assess the methodology, reproducibility, and generalizability of the findings.

2. Data and Experimental Consistency

Appendix A.2 reports inconsistencies between the summary matrix and raw experimental records, including:

  • 19 binary-result discrepancies.
  • 44 trial-mean discrepancies.
  • 91 unresolved records.

Appendix A.3 also reports differences in user simulators and step limits between some reference and ablation runs.

Although the authors acknowledge these issues, they remain unresolved. Since the main findings rely on the measured component effects, these problems raise serious concerns about the reliability of the results. In particular, it is unclear whether the observed differences are caused by the harness components or by other changes in the experimental setup.

Questions for authors

1. Component Selection

  • Why were these four harness components (h0–h3) selected?
  • Were other components or configurations considered?
  • How representative are these four components of general agent harness designs?

2. Data and Experimental Consistency

Appendix A.2 reports inconsistencies between the summary matrix and raw experimental records, including 19 binary discrepancies, 44 trial-mean discrepancies, and 91 unresolved records.

  • Since the authors conducted the experiments and processed the data themselves, why are these inconsistencies still unresolved?
  • What caused these discrepancies?
  • Can the authors reproduce the main findings using verified raw records?

Appendix A.3 also reports differences in user simulators and step limits between reference and ablation runs.

  • How can the authors ensure that the observed effects come from the harness components rather than these configuration differences?
  • Do the main findings still hold under fully matched experimental settings?

3. Contribution and Practical Significance

The paper mainly provides a post-hoc analysis of harness component transferability. However, it is somewhat expected that different models respond differently to harness components. I am not convinced that reporting these differences alone is a strong enough contribution.

  • What new insights does this analysis provide beyond confirming that harness effects vary across models?
  • How can these findings guide practical harness reuse or improvement?

I also think it would be more meaningful to move beyond diagnosis toward improvement.

For example, it would be interesting to investigate whether existing harness optimization, self-evolution, or recursive self-improvement (RSI) methods can improve cross-model transferability.

Studying how these methods modify harness components, and why these changes transfer or fail to transfer across models, could provide deeper insights into the underlying mechanisms.

This could be more informative than manually evaluating components through leave-one-component-out ablations.

4. Diagnostic Tasks and Clustering

The diagnostic tasks and clustering process are not clearly explained.

  • What are the 100 diagnostic tasks, and how were they selected?
  • Why were they divided into five clusters?
  • What does each cluster represent?
  • How sensitive are the results to different clustering choices?

These details are important for understanding and evaluating the failure-profile similarity used for cross-model prediction.

5. Minor Formatting Issue

On page 10, the AI Use Statement, Reproducibility Statement, and Ethics Statement are numbered. However, the ICLR template appears to use unnumbered headings for these sections.

Reviewer_pxmYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

My understanding of the paper: The paper studies how much we can transfer harness components across models. It does so by fixing all but one harness component, changing that component, and examining how performance changes across different model–task pairs. The paper concludes that harness components have limited transferability across models.

Critical strengths

None

Critical weaknesses
  1. Clarity. It is unclear what the paper’s goal is, how the method is implemented, and what the results show. In several places, the citations also do not seem to make sense. For example, around line 38, the authors write, “We therefore take a component-level view, evaluating one component at a time,” and cite the DeepSeek Harness GitHub repository. I don't see a connection between the statement and the citation. Moreover, Section 3.1, which defines the problem, is very hard to understand. For example, what does “Observed zeros remain in the pool; missing effects are omitted” mean?

  2. Results. Figure 2 appears to present a main result, but I cannot understand what I should conclude from it. The experimental setup is also very unclear. For example, it mentions 38/120 Qwen reference trials hitting the step cap and then dropping Qwen as a target, like what?? Overall, I find it difficult to draw concrete conclusions from the results.

Questions for authors

Honestly nothing could affect my assessment. I encourage the authors to first make the presentation clear: what the paper’s goal is, how it addresses that goal, and what the results and conclusions are.

Reviewer_VVtBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the effects of individual agent harness components transfer across language models. The authors evaluate four components on airline, retail, and telecom tasks using leave-one-component-out comparisons, and examine whether source-model effects predict effects on a held-out target. They report sparse and model-dependent outcome changes, limited cross-model recurrence, and no clear advantage from failure-profile weighting over simpler aggregation. They also analyze the benefit-harm trade-offs of retrospective component screening.

Critical strengths

The paper asks an important practical question: whether harness components need to be revalidated when the underlying model changes. The component-level perspective is useful, and examining both beneficial and harmful changes is more informative than comparing aggregate harness performance alone. The distinction between predictive AUC and decision-level utility is also useful.

Critical weaknesses

(1) The experiments do not seem to reliably isolate component effects. Appendix A reports different user simulators and step limits between reference and ablation runs, along with unresolved discrepancies between trial records and reported summaries. This makes it difficult to attribute outcome changes to the harness component, undermining the central conclusions about component effects and transfer. The key analyses could be repeated using verified trial records and identical evaluation settings except for the component being disabled.

(2) The evidence for model-specific effects needs stronger statistical support. With only three trials per setting, binary pass-at-three outcomes can hide changes in success probability and be sensitive to rollout noise. The findings also depend substantially on the outcome measure: using trial-mean scores reduces the proportion of nonzero target effects with at most one active source from 94.3% to 74.9% (Table 11). More repeated trials and a null comparison preserving each model's activity rates would help establish whether these patterns reflect reproducible model-specific behavior.

Questions for authors

Please see the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.