- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents Harness-Delta Attribution (HDA), an evaluation framework for understanding performance gains from agent harness evolution. The authors aim to distinguish three sources of improvement: benchmark-specific overfitting, additional test-time computation, and generalizable improvements. HDA constructs a compute-matched baseline and a modified version of the evolved harness with detected benchmark-specific shortcuts removed. It then uses the performance differences to estimate the contribution of each source. Experiments across 11 executor models, three harness-evolution frameworks, and four benchmarks suggest that a large fraction of reported improvements comes from overfitting or additional computation rather than generalizable changes.
Critical strengths
-
The paper addresses an important research problem. Understanding where the performance gains of automatically evolved agent harnesses come from is essential for evaluating these methods. Simply reporting higher benchmark scores is insufficient to demonstrate meaningful improvements.
-
The paper provides interesting empirical findings across different models, frameworks, and benchmarks. In particular, the examples of benchmark-specific shortcuts in LiveMath and ALFWorld clearly illustrate the limitations of evaluating harness evolution based only on benchmark scores.
Critical weaknesses
- The proposed attribution method lacks a sound justification. The main concern is whether the proposed decomposition correctly measures the three sources of performance improvement. HDA defines overfitting and test-time scaling contributions through two separate interventions, and assigns the remaining gain to generalizable improvement. Although these quantities sum to the total improvement by construction, this does not establish that they represent the actual contributions of the three factors. In practice, these factors may interact. For example, an evolved harness may use additional LLM calls together with new rules that determine when and how those calls are made. The benefit of additional computation may therefore depend on the new rules. Similarly, removing a benchmark-specific rule may also affect other useful behaviors. The proposed decomposition does not account for these interactions, and simply assumes they are independent.
The paper also interprets the measured overfitting contribution as a lower bound and the residual as an upper bound on generalizable improvement. However, these interpretations are not sufficiently justified. Without clear assumptions about the interventions, the claimed bounds may not hold.
-
The distinction between overfitting and generalizable improvement is subjective and insufficiently defined. HDA identifies overfitting mechanisms through LLM-assisted analysis and human review. However, the paper does not provide a sufficiently objective criterion for distinguishing benchmark-specific overfitting from useful domain knowledge. For example, the evolved ALFWorld harness learns environment-specific rules and replaces LLM calls with a deterministic controller. Such a controller may represent a valid and effective solution to the underlying task rather than overfitting. Whether this should be considered overfitting depends on the intended target distribution, which is not clearly defined. This ambiguity directly affects the reported attribution results. Moreover, the paper does not evaluate whether different reviewers or different reasonable ways of removing the identified rules would produce consistent results. The reliability of the proposed attribution therefore remains unclear.
-
The compute-matching procedure does not necessarily isolate the contribution of test-time scaling. HDA estimates the contribution of additional computation by increasing the inference budget of the original harness. However, giving the original harness more LLM calls does not necessarily reproduce how the evolved harness uses additional computation. The effectiveness of additional calls depends on how they are allocated, how their results are combined, and how they interact with the harness's decision-making process. Different compute-matching strategies, such as majority voting or pass@k, can lead to substantially different results. Some strategies may also require information that is not available during actual deployment. Therefore, the measured improvement of a compute-matched baseline does not necessarily represent the contribution of additional computation within the evolved harness. The paper does not sufficiently validate this assumption or examine how sensitive the attribution is to different compute-matching choices.
Questions for authors
-
What assumptions justify interpreting the proposed decomposition as an attribution of performance gains? In particular, how does HDA handle interactions between overfitting, test-time computation, and other harness improvements?
-
How is overfitting objectively distinguished from useful domain-specific knowledge? What target task distribution is assumed, and how consistent are the attribution results across different reviewers or different neutralization choices?