acceptodds
Under review as a conference paper at ICLR 2027

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Abstract

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation and reporting in future research on harness evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
harness, harness evolution, test-time scaling, recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6UxTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Harness-Delta Attribution (HDA), an evaluation framework for understanding performance gains from agent harness evolution. The authors aim to distinguish three sources of improvement: benchmark-specific overfitting, additional test-time computation, and generalizable improvements. HDA constructs a compute-matched baseline and a modified version of the evolved harness with detected benchmark-specific shortcuts removed. It then uses the performance differences to estimate the contribution of each source. Experiments across 11 executor models, three harness-evolution frameworks, and four benchmarks suggest that a large fraction of reported improvements comes from overfitting or additional computation rather than generalizable changes.

Critical strengths
  1. The paper addresses an important research problem. Understanding where the performance gains of automatically evolved agent harnesses come from is essential for evaluating these methods. Simply reporting higher benchmark scores is insufficient to demonstrate meaningful improvements.

  2. The paper provides interesting empirical findings across different models, frameworks, and benchmarks. In particular, the examples of benchmark-specific shortcuts in LiveMath and ALFWorld clearly illustrate the limitations of evaluating harness evolution based only on benchmark scores.

Critical weaknesses
  1. The proposed attribution method lacks a sound justification. The main concern is whether the proposed decomposition correctly measures the three sources of performance improvement. HDA defines overfitting and test-time scaling contributions through two separate interventions, and assigns the remaining gain to generalizable improvement. Although these quantities sum to the total improvement by construction, this does not establish that they represent the actual contributions of the three factors. In practice, these factors may interact. For example, an evolved harness may use additional LLM calls together with new rules that determine when and how those calls are made. The benefit of additional computation may therefore depend on the new rules. Similarly, removing a benchmark-specific rule may also affect other useful behaviors. The proposed decomposition does not account for these interactions, and simply assumes they are independent.

The paper also interprets the measured overfitting contribution as a lower bound and the residual as an upper bound on generalizable improvement. However, these interpretations are not sufficiently justified. Without clear assumptions about the interventions, the claimed bounds may not hold.

  1. The distinction between overfitting and generalizable improvement is subjective and insufficiently defined. HDA identifies overfitting mechanisms through LLM-assisted analysis and human review. However, the paper does not provide a sufficiently objective criterion for distinguishing benchmark-specific overfitting from useful domain knowledge. For example, the evolved ALFWorld harness learns environment-specific rules and replaces LLM calls with a deterministic controller. Such a controller may represent a valid and effective solution to the underlying task rather than overfitting. Whether this should be considered overfitting depends on the intended target distribution, which is not clearly defined. This ambiguity directly affects the reported attribution results. Moreover, the paper does not evaluate whether different reviewers or different reasonable ways of removing the identified rules would produce consistent results. The reliability of the proposed attribution therefore remains unclear.

  2. The compute-matching procedure does not necessarily isolate the contribution of test-time scaling. HDA estimates the contribution of additional computation by increasing the inference budget of the original harness. However, giving the original harness more LLM calls does not necessarily reproduce how the evolved harness uses additional computation. The effectiveness of additional calls depends on how they are allocated, how their results are combined, and how they interact with the harness's decision-making process. Different compute-matching strategies, such as majority voting or pass@k, can lead to substantially different results. Some strategies may also require information that is not available during actual deployment. Therefore, the measured improvement of a compute-matched baseline does not necessarily represent the contribution of additional computation within the evolved harness. The paper does not sufficiently validate this assumption or examine how sensitive the attribution is to different compute-matching choices.

Questions for authors
  1. What assumptions justify interpreting the proposed decomposition as an attribution of performance gains? In particular, how does HDA handle interactions between overfitting, test-time computation, and other harness improvements?

  2. How is overfitting objectively distinguished from useful domain-specific knowledge? What target task distribution is assumed, and how consistent are the attribution results across different reviewers or different neutralization choices?

Reviewer_rUmZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper raises an insightful question about the sources of score gains in agent harnesses, asking where those performance improvements actually come from. The authors break the gains down into three categories: genuine improvement, increased calls to the base model, and overfitting to a specific benchmark. They validate this across multiple base models and harness combinations on several benchmarks, ultimately proposing a framework for evaluating agent harnesses.

Critical strengths

Strengths:

  1. Their insights are strong, providing a quantifiable standard for how to evaluate and assess harnesses in the future.

  2. Their evaluation scope is broad, covering Qwen and Claude models, three harnesses, and a variety of benchmarks. I think their validation of this theoretical framework makes the assessment quite reliable.

Critical weaknesses

Weaknesses:

  1. There is no specific methodology or concrete solution for improving harness evolution. It only provides attribution and analysis, so its contribution to this field is limited.

  2. The three-part breakdown of harness evolution lacks theoretical backing. There is no detailed theory or verification regarding the relationships between these three parts and whether they are sufficiently universal. And how should we classify the evolution of a harness on a specific benchmark: is it normal learning that adapts to that bench, or is it overfitting to it? These two definitions aren't clearly distinct.

Questions for authors

Questions:

  1. Can you propose a method to optimize the harness evolution process, for instance, to avoid the issues mentioned during evolution?

  2. Are there clear ways to distinguish between overfitting and adaptation to the benchmark?

  3. Is there theoretical support for the three-part decomposition and the upper and lower bound interpretations, and do they require any conditions or assumptions?

  4. How are tasks divided between agents and manual review in HDA's overfitting identification and determination process? This is crucial for determining how reliable our overfitting detection really is.

Reviewer_j3yXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Harness-Delta Attribution, a semi-automated evaluation protocol that decomposes search-set performance gains in automated agent harness evolution into test-time compute scaling, benchmark-specific overfitting, and generalizable improvements.

Critical strengths
  1. It targets a major blind spot in agentic workflow optimization by exposing how impressive benchmark gains frequently mask compute inflation and shortcut exploitation

  2. The three-way attribution framework is conceptually sound, rigorously treating identified overfitting as an empirical lower bound and generalizable gain as a theoretical upper bound

  3. It provides vital insight by demonstrating that traditional held-out test splits and validation-based checkpoint selection fail to detect or prevent harness overfitting

Critical weaknesses
  1. The semi-automated shortcut neutralization step requires human verification, leaving ambiguous boundaries between legitimate engineering guardrails and illicit benchmark shortcuts

  2. The sequential attribution pipeline models compute scaling and shortcut exploitation additively, overlooking potential non-linear interactions between the two

  3. The estimated generalizable gain serves merely as an optimistic upper ceiling that does not reliably forecast true held-out generalization

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.