- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether performance gains achieved through harness optimization generalize, and introduces CHASE—a method that identifies and mitigates dependencies on specific protocols by altering environmental conditions within the benchmark. The core question addressed is: when a modification to a harness boosts benchmark scores, does that improvement stem from specialization tailored to the specific benchmark, or does it generalize to broader tasks? For instance, a harness might exploit fixed file formats or tool interfaces; the authors aim to determine whether the performance gains persist when these conditions change. Methodologically, CHASE keeps the problem and answer unchanged while seeking environmental variations that cause the harness's performance gains to vanish; it then uses this feedback to iteratively refine and select the harness. Theoretically, under certain assumptions, the paper analyzes the conditions under which this process yields a solution—within a finite number of iterations—that offers near-optimal performance and robustness against protocol variations. Experimentally, the authors evaluated the method using Syn-Ledger (a synthetic auditing task) and OfficeQA (a real-world financial Q&A task). In Syn-Ledger, CHASE achieved the best performance after the removal of artificially introduced "shortcut" cues. In OfficeQA, it excelled in terms of average performance, worst-case performance, and cross-corpus generalization, even though it did not achieve the highest raw benchmark score. The authors conclude that this approach helps retain optimization gains that are more likely to generalize to other protocol conditions.
Critical strengths
I believe the most significant contribution of this paper is that it raises a question well worth further investigation: exactly what should a harness optimize, and how do we determine whether such optimization is effective? A harness can serve different objectives: it might target a specific model to address weaknesses in particular tasks, or it might aim for general applicability across a wider range of tasks and environments. These goals are not identical. The paper highlights that even if an optimized harness yields performance gains on new problems, those improvements might still rely on formatting conventions or interface patterns shared across the benchmark. Consequently, improved performance on new tasks does not directly reveal the scope of the optimization's generalizability. This issue is crucial for understanding and evaluating the evolution of harnesses. From a broader perspective, the harness influences which tasks an agent can accomplish and how those tasks are executed. For instance, a CUA agent requires specific tools and interaction suites to gain the ability to operate a computer. Since practical application is inextricably linked to harness design, it is difficult to assess the merit of a specific harness modification in isolation from the model, tools, and environment. By holding the model and task content constant while varying certain protocol conditions, this paper investigates a specific facet of this broader issue: whether performance gains achieved through harness optimization persist when environmental conditions change. I strongly endorse this research direction. It prompts us to reflect on whether harness evolution should prioritize model-specific adaptation or broader generalization, and how both should be measured.
Critical weaknesses
In my view, the primary issue with this paper is the lack of a coherent link between the initial claims and the subsequent work. The problem raised at the beginning is that a harness might exploit specific patterns in a benchmark to inflate scores—meaning the scores fail to reflect the capabilities we actually care about—and that this evaluation distortion needs to be identified and addressed. This is a crucial issue, and it is precisely why I find the paper's research topic so compelling. However, the actual work performed later involves identifying protocol changes that negate the current harness's gains, incorporating these changes into the development pipeline, and then optimizing a harness capable of adapting to these new conditions. Even if this subsequent optimization succeeds, it does not demonstrate that the initial benchmark issue has been resolved.
Specifically, I have the following questions:
1. There is a missing link in the argument connecting "successful harness optimization" to a "more effective benchmark."
I had expected the authors to bridge these two concepts using , but I have not yet seen a sufficient argument to that effect. At least two steps are missing here.
First, why is the element being eliminated considered an illegitimate shortcut? The very purpose of a harness includes helping a model adapt better to the environment or addressing a specific model's weaknesses on certain tasks. The fact that an optimization relies on file organization, retrieval methods, or tool interfaces does not automatically imply that the optimization is unreasonable. The authors need to explain why such dependencies deviate from the capabilities the benchmark was originally intended to measure.
Second, why do scores become more representative of the capabilities we care about after these dependencies are removed? The current experiments demonstrate that the optimized harness retains more of its performance gains under the tested protocol changes. However, from an evaluation perspective, I need to know: what exactly is the modified benchmark measuring, and why is this score more reliable than the original one? Merely defining an ideal in theory is insufficient to answer these questions.
2. From an industrial perspective, it is unclear to me how this approach would ultimately be applied to evaluation.
Whether comparing different model-harness combinations (cross-sectional analysis) or the same system over time (longitudinal analysis), we need to align benchmarking conditions. If the benchmark protocol evolves alongside the system, how can we compare new results with old ones? When a new model or harness emerges and the protocol is modified again, how much additional evaluation is required for previous systems? How should these additional costs be borne?
My concerns go beyond just operational costs. Modifying a benchmark itself can alter what it measures, potentially compromising its comparability and interpretability. Therefore, I do not believe that "continuously identifying new variations and adapting the harness to them" inherently solves the evaluation problem.
I would be more receptive to an approach where CHASE is used prior to a benchmark's release to generate a "frozen" version, accompanied by a clear demonstration of how it differs from the original benchmark—specifically, which dependencies were eliminated, how system performance changed, and why these modifications enhance the evaluation's effectiveness. Alternatively, CHASE could be explicitly integrated into the harness development workflow, with the final result subsequently validated in an environment completely isolated from the development process. However, both approaches must be clearly linked to the evaluation objectives outlined earlier in the paper.
3. The scope of the current experiments is too limited to answer the aforementioned questions.
While Syn-Ledger represents a synthetic task with artificially introduced shortcuts and OfficeQA offers experiments based on real-world tasks (with additional results reported for OfficeQA Pro V2), the experiments overall remain confined to a limited range of task types, protocol variations, and relatively simple agent configurations. I would like to see more diverse benchmarks and harnesses that more closely resemble real-world applications.
For instance, one could develop the harness using multiple benchmarks, freeze it, and then test it on a completely separate "held-out" benchmark that played no role in the development phase. If the harness still yields performance gains in that context, I would be more inclined to believe it has learned transferable improvements rather than simply adapting further to the specific conditions of its initial development.
Furthermore, from an evaluation perspective, I am interested in the model's intrinsic capabilities, the specific contributions of the harness, and the ultimate performance of the model-harness combination. While these elements may not easily translate into three distinct scores, the paper should at least provide comparisons that allow for an understanding of these respective contributions. Current experiments focus primarily on harness optimization for fixed models and do not yet address these issues.
4. The specific scope and implementation of the harness require further clarification.
I still have questions regarding how the harness is utilized in the paper. Based on my industry experience, altering file structures or the content returned by tools would typically be classified as environment editing or benchmark editing. While the paper lists the components modified by the Proposer and the Challenger separately, it fails to adequately explain the conceptual and implementation boundaries between these modifications. In particular, the distinction regarding why modifications to the tool wrapper and the tool-output protocol fall under the "harness" and "environment" categories, respectively, needs to be clarified.
The supplementary code also presents issues requiring explanation. In the submitted OfficeQA execution path, the strategy logic does not receive actual retrieval results; consequently, conditions relying on retrieval coverage, document count, and evidence conflict cannot function as described. Certain actions—such as reserving a reading budget and halting when sufficient evidence is found—are included in the execution plan but are not utilized by the subsequent code. Furthermore, I could not locate a unified entry point for the CHASE workflow that integrates candidate generation, Challenger search, independent verification, and archive updates. These issues make it difficult for me to verify how the optimization process and harness behavior described in the paper correspond to the reported results.
I appreciate the research questions posed in this paper; however, the current evidence primarily supports a specific approach to harness development rather than substantiating the earlier claims regarding the resolution of benchmark evaluation distortion. This is the primary reason for my current rating. I would be willing to reconsider the score if the authors could bridge the gap in their argumentation, provide more compelling experiments on independent generalization, and address the aforementioned implementation issues.
Questions for authors
For Why Clarity Score 1
I believe the main issue with this paper is that the terminology, conceptual boundaries, and phrasing make it difficult to read. Some expressions differ from the relevant literature and industry practices I am familiar with; however, the authors do not explain these discrepancies upfront, which can lead readers to form expectations that do not align with the actual methodology.
- Some terms are prone to misunderstanding, and concepts from the literature are not clearly distinguished. Take the term "counterfactual-guided" in the title, for example. As I understand it, the paper primarily involves altering environmental configurations—such as redistributing the files containing evidence or changing the format of tool outputs—while keeping the problem statement and the answer unchanged. However, the term "counterfactual" tends to lead readers to expect a causal analysis. The authors need to explain more directly what this term specifically refers to in the context of this paper. Upon further research, I found precedents for similar usage in NeurIPS 2021 studies on counterfactual invariance, so the term is not entirely unfounded. However, lines 147–149 describe related work as "counterfactual-guided model repair," citing SpecRepair and the ICML 2023 paper A Robust Optimisation Perspective on Counterexample-Guided Repair of Neural Networks; yet, these actually discuss "counterexample-guided repair." A "counterexample" is distinct from a "counterfactual," and the ICML paper explicitly defines it as an input that causes the model to violate a specific rule. These two concepts must be distinguished and cannot be used interchangeably.
- The paper needs to clarify the specific scope of the "harness" earlier on. Based on my reading of the paper and the code, the actual modifications focus more on prompt engineering or prompt editing, with minimal structural changes to the "scaffold" itself. However, the title and earlier statements led me to expect a broader discussion on the evolution of the harness. The authors should clarify upfront what components constitute the harness in this context, which parts are actually being optimized, and how this relates to scaffold evolution. Similarly, the distinction between modifying the environment configuration and modifying the harness needs to be clearly defined. From my experience in industry, editing the benchmark environment itself is not automatically equivalent to modifying the harness. Even if the authors adopt a different conceptual scope, this should be explicitly stated so that readers understand exactly what is being modified at each step.
- The language is overly condensed, lacking sufficient foundational explanation. The paper devotes considerable space to theoretical proofs, yet fails to adequately explain some of the more fundamental background, definitions, and conceptual distinctions. Many sentences pack multiple concepts and conditions together, forcing readers to repeatedly deconstruct them. I suggest the authors include a clear introductory section outlining the object of study, the scope of modifiable elements, and the specific problem the method addresses, before proceeding to the formalization and proofs.
- Figures 1 and 2 do not sufficiently aid in understanding the method. I find the logic behind the presentation of these two figures—as well as the correspondence between the operations depicted and their actual implementation—to be unintuitive. For instance, I only(with my codex) grasped what "reassign documents" actually entailed after consulting the code. The diagrams should more clearly illustrate which steps modify the harness versus the environment, and how these changes feed into the next round of optimization.
I have a few additional questions
that I hope the authors can address, as the answers will influence my assessment of the paper.
- I am very interested to know how the authors define "shortcuts" and "generalization." Since a harness is inherently designed to help the model adapt to the environment, why should a modification that relies on file organization or tool interfaces be considered the exploitation of a shortcut? If a protocol change causes the benefits provided by the harness to disappear, how do we determine whether this exposes an illegitimate shortcut rather than disrupting a valid form of adaptation? Conversely, if the benefits persist after applying CHASE, why should this be interpreted as superior generalization or an indication that the score better reflects the capabilities we care about? I believe this is a point the paper needs to clarify.
- Regarding the practical application of this method, I would like the authors to provide at least one concrete use case. I do not yet grasp who would use CHASE, at what stage, or how the resulting output should be utilized. Is it intended for benchmark developers to refine benchmarks prior to release, or for agent developers to optimize their harnesses? If used for evaluation, how should the results obtained after the modifications be compared with the original results? I am not asking the authors to build a full-scale industrial system during the rebuttal period, but I would like to understand where this method is actually applied and what specific problems it solves for the user.
- Regarding OfficeQA, while examining the submitted code, My Agent and I noticed that the strategy logic did not receive the actual retrieval results. Furthermore, certain harness behaviors—such as reserving a reading budget and halting when sufficient evidence is found—were defined in the configuration or execution plan but were not utilized during actual execution. I am puzzled by this and would like the authors to explain: Did we overlook a specific execution entry point, or does the submitted code differ from the version used to generate the results reported in the paper? How exactly do these strategies take effect during actual experiments? If there is relevant code or execution logs available, I would appreciate it if the authors could point them out.
Reminder to Reviewer 1L86
Dear Reviewer 1L86, thank you again for your review. We would be grateful to know whether our response addresses your concerns. Since our reply, we have added an experiment that bears on W2: SWE-Bench Pro V2 [1], a benchmark for coding agents, with a third agent model, claude-sonnet-4-6 (Table 3, Appendix F). Certification on the 21 sealed tasks (accuracy in %):
CHASE attains the best score on each evaluation metric and improves on RawHarness by 11.43% [1.90, 22.86] under the released protocol. We are happy to clarify anything further before the discussion period ends.
References