acceptodds
Under review as a conference paper at ICLR 2027

Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

Abstract

Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed foundation model. Task holdout is commonly used to guard against harness overfitting. It varies semantic tasks but leaves the benchmark protocol fixed, so a “bad genius” Proposer can produce a cheating harness whose improvement over the initial harness on depends on a benchmark-wide shortcut. We introduce **C**ounterfactual **Ha**rness **S**earch and **E**volution (**CHASE**), which casts harness evolution as constraint generation over valid counterfactual benchmarks. After each Proposer update, a *Challenger* searches for an executable protocol transformation with large gain destruction. A validity firewall checks that task semantics are preserved, while a held-out confirmation set determines whether the counterfactual enters a finite archive. We formalize an ideal shortcut-neutralized benchmark and establish theoretical guarantees linking finite counterfactual archives to and characterizing sequential Challenger search. We evaluate CHASE on Syn-Ledger and OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol transformations.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, harness optimization, benchmark-wide shortcuts, counterfactual evaluation, shortcut learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_K5F1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether performance gains achieved through harness optimization generalize, and introduces CHASE—a method that identifies and mitigates dependencies on specific protocols by altering environmental conditions within the benchmark. The core question addressed is: when a modification to a harness boosts benchmark scores, does that improvement stem from specialization tailored to the specific benchmark, or does it generalize to broader tasks? For instance, a harness might exploit fixed file formats or tool interfaces; the authors aim to determine whether the performance gains persist when these conditions change. Methodologically, CHASE keeps the problem and answer unchanged while seeking environmental variations that cause the harness's performance gains to vanish; it then uses this feedback to iteratively refine and select the harness. Theoretically, under certain assumptions, the paper analyzes the conditions under which this process yields a solution—within a finite number of iterations—that offers near-optimal performance and robustness against protocol variations. Experimentally, the authors evaluated the method using Syn-Ledger (a synthetic auditing task) and OfficeQA (a real-world financial Q&A task). In Syn-Ledger, CHASE achieved the best performance after the removal of artificially introduced "shortcut" cues. In OfficeQA, it excelled in terms of average performance, worst-case performance, and cross-corpus generalization, even though it did not achieve the highest raw benchmark score. The authors conclude that this approach helps retain optimization gains that are more likely to generalize to other protocol conditions.

Critical strengths

I believe the most significant contribution of this paper is that it raises a question well worth further investigation: exactly what should a harness optimize, and how do we determine whether such optimization is effective? A harness can serve different objectives: it might target a specific model to address weaknesses in particular tasks, or it might aim for general applicability across a wider range of tasks and environments. These goals are not identical. The paper highlights that even if an optimized harness yields performance gains on new problems, those improvements might still rely on formatting conventions or interface patterns shared across the benchmark. Consequently, improved performance on new tasks does not directly reveal the scope of the optimization's generalizability. This issue is crucial for understanding and evaluating the evolution of harnesses. From a broader perspective, the harness influences which tasks an agent can accomplish and how those tasks are executed. For instance, a CUA agent requires specific tools and interaction suites to gain the ability to operate a computer. Since practical application is inextricably linked to harness design, it is difficult to assess the merit of a specific harness modification in isolation from the model, tools, and environment. By holding the model and task content constant while varying certain protocol conditions, this paper investigates a specific facet of this broader issue: whether performance gains achieved through harness optimization persist when environmental conditions change. I strongly endorse this research direction. It prompts us to reflect on whether harness evolution should prioritize model-specific adaptation or broader generalization, and how both should be measured.

Critical weaknesses

In my view, the primary issue with this paper is the lack of a coherent link between the initial claims and the subsequent work. The problem raised at the beginning is that a harness might exploit specific patterns in a benchmark to inflate scores—meaning the scores fail to reflect the capabilities we actually care about—and that this evaluation distortion needs to be identified and addressed. This is a crucial issue, and it is precisely why I find the paper's research topic so compelling. However, the actual work performed later involves identifying protocol changes that negate the current harness's gains, incorporating these changes into the development pipeline, and then optimizing a harness capable of adapting to these new conditions. Even if this subsequent optimization succeeds, it does not demonstrate that the initial benchmark issue has been resolved.

Specifically, I have the following questions:

1. There is a missing link in the argument connecting "successful harness optimization" to a "more effective benchmark."

I had expected the authors to bridge these two concepts using B0B_0, but I have not yet seen a sufficient argument to that effect. At least two steps are missing here.

First, why is the element being eliminated considered an illegitimate shortcut? The very purpose of a harness includes helping a model adapt better to the environment or addressing a specific model's weaknesses on certain tasks. The fact that an optimization relies on file organization, retrieval methods, or tool interfaces does not automatically imply that the optimization is unreasonable. The authors need to explain why such dependencies deviate from the capabilities the benchmark was originally intended to measure.

Second, why do scores become more representative of the capabilities we care about after these dependencies are removed? The current experiments demonstrate that the optimized harness retains more of its performance gains under the tested protocol changes. However, from an evaluation perspective, I need to know: what exactly is the modified benchmark measuring, and why is this score more reliable than the original one? Merely defining an ideal B0B_0 in theory is insufficient to answer these questions.

2. From an industrial perspective, it is unclear to me how this approach would ultimately be applied to evaluation.

Whether comparing different model-harness combinations (cross-sectional analysis) or the same system over time (longitudinal analysis), we need to align benchmarking conditions. If the benchmark protocol evolves alongside the system, how can we compare new results with old ones? When a new model or harness emerges and the protocol is modified again, how much additional evaluation is required for previous systems? How should these additional costs be borne?

My concerns go beyond just operational costs. Modifying a benchmark itself can alter what it measures, potentially compromising its comparability and interpretability. Therefore, I do not believe that "continuously identifying new variations and adapting the harness to them" inherently solves the evaluation problem.

I would be more receptive to an approach where CHASE is used prior to a benchmark's release to generate a "frozen" version, accompanied by a clear demonstration of how it differs from the original benchmark—specifically, which dependencies were eliminated, how system performance changed, and why these modifications enhance the evaluation's effectiveness. Alternatively, CHASE could be explicitly integrated into the harness development workflow, with the final result subsequently validated in an environment completely isolated from the development process. However, both approaches must be clearly linked to the evaluation objectives outlined earlier in the paper.

3. The scope of the current experiments is too limited to answer the aforementioned questions.

While Syn-Ledger represents a synthetic task with artificially introduced shortcuts and OfficeQA offers experiments based on real-world tasks (with additional results reported for OfficeQA Pro V2), the experiments overall remain confined to a limited range of task types, protocol variations, and relatively simple agent configurations. I would like to see more diverse benchmarks and harnesses that more closely resemble real-world applications.

For instance, one could develop the harness using multiple benchmarks, freeze it, and then test it on a completely separate "held-out" benchmark that played no role in the development phase. If the harness still yields performance gains in that context, I would be more inclined to believe it has learned transferable improvements rather than simply adapting further to the specific conditions of its initial development.

Furthermore, from an evaluation perspective, I am interested in the model's intrinsic capabilities, the specific contributions of the harness, and the ultimate performance of the model-harness combination. While these elements may not easily translate into three distinct scores, the paper should at least provide comparisons that allow for an understanding of these respective contributions. Current experiments focus primarily on harness optimization for fixed models and do not yet address these issues.

4. The specific scope and implementation of the harness require further clarification.

I still have questions regarding how the harness is utilized in the paper. Based on my industry experience, altering file structures or the content returned by tools would typically be classified as environment editing or benchmark editing. While the paper lists the components modified by the Proposer and the Challenger separately, it fails to adequately explain the conceptual and implementation boundaries between these modifications. In particular, the distinction regarding why modifications to the tool wrapper and the tool-output protocol fall under the "harness" and "environment" categories, respectively, needs to be clarified.

The supplementary code also presents issues requiring explanation. In the submitted OfficeQA execution path, the strategy logic does not receive actual retrieval results; consequently, conditions relying on retrieval coverage, document count, and evidence conflict cannot function as described. Certain actions—such as reserving a reading budget and halting when sufficient evidence is found—are included in the execution plan but are not utilized by the subsequent code. Furthermore, I could not locate a unified entry point for the CHASE workflow that integrates candidate generation, Challenger search, independent verification, and archive updates. These issues make it difficult for me to verify how the optimization process and harness behavior described in the paper correspond to the reported results.

I appreciate the research questions posed in this paper; however, the current evidence primarily supports a specific approach to harness development rather than substantiating the earlier claims regarding the resolution of benchmark evaluation distortion. This is the primary reason for my current rating. I would be willing to reconsider the score if the authors could bridge the gap in their argumentation, provide more compelling experiments on independent generalization, and address the aforementioned implementation issues.

Questions for authors

For Why Clarity Score 1

I believe the main issue with this paper is that the terminology, conceptual boundaries, and phrasing make it difficult to read. Some expressions differ from the relevant literature and industry practices I am familiar with; however, the authors do not explain these discrepancies upfront, which can lead readers to form expectations that do not align with the actual methodology.

  1. Some terms are prone to misunderstanding, and concepts from the literature are not clearly distinguished. Take the term "counterfactual-guided" in the title, for example. As I understand it, the paper primarily involves altering environmental configurations—such as redistributing the files containing evidence or changing the format of tool outputs—while keeping the problem statement and the answer unchanged. However, the term "counterfactual" tends to lead readers to expect a causal analysis. The authors need to explain more directly what this term specifically refers to in the context of this paper. Upon further research, I found precedents for similar usage in NeurIPS 2021 studies on counterfactual invariance, so the term is not entirely unfounded. However, lines 147–149 describe related work as "counterfactual-guided model repair," citing SpecRepair and the ICML 2023 paper A Robust Optimisation Perspective on Counterexample-Guided Repair of Neural Networks; yet, these actually discuss "counterexample-guided repair." A "counterexample" is distinct from a "counterfactual," and the ICML paper explicitly defines it as an input that causes the model to violate a specific rule. These two concepts must be distinguished and cannot be used interchangeably.
  2. The paper needs to clarify the specific scope of the "harness" earlier on. Based on my reading of the paper and the code, the actual modifications focus more on prompt engineering or prompt editing, with minimal structural changes to the "scaffold" itself. However, the title and earlier statements led me to expect a broader discussion on the evolution of the harness. The authors should clarify upfront what components constitute the harness in this context, which parts are actually being optimized, and how this relates to scaffold evolution. Similarly, the distinction between modifying the environment configuration and modifying the harness needs to be clearly defined. From my experience in industry, editing the benchmark environment itself is not automatically equivalent to modifying the harness. Even if the authors adopt a different conceptual scope, this should be explicitly stated so that readers understand exactly what is being modified at each step.
  3. The language is overly condensed, lacking sufficient foundational explanation. The paper devotes considerable space to theoretical proofs, yet fails to adequately explain some of the more fundamental background, definitions, and conceptual distinctions. Many sentences pack multiple concepts and conditions together, forcing readers to repeatedly deconstruct them. I suggest the authors include a clear introductory section outlining the object of study, the scope of modifiable elements, and the specific problem the method addresses, before proceeding to the formalization and proofs.
  4. Figures 1 and 2 do not sufficiently aid in understanding the method. I find the logic behind the presentation of these two figures—as well as the correspondence between the operations depicted and their actual implementation—to be unintuitive. For instance, I only(with my codex) grasped what "reassign documents" actually entailed after consulting the code. The diagrams should more clearly illustrate which steps modify the harness versus the environment, and how these changes feed into the next round of optimization.

I have a few additional questions

that I hope the authors can address, as the answers will influence my assessment of the paper.

  1. I am very interested to know how the authors define "shortcuts" and "generalization." Since a harness is inherently designed to help the model adapt to the environment, why should a modification that relies on file organization or tool interfaces be considered the exploitation of a shortcut? If a protocol change causes the benefits provided by the harness to disappear, how do we determine whether this exposes an illegitimate shortcut rather than disrupting a valid form of adaptation? Conversely, if the benefits persist after applying CHASE, why should this be interpreted as superior generalization or an indication that the score better reflects the capabilities we care about? I believe this is a point the paper needs to clarify.
  2. Regarding the practical application of this method, I would like the authors to provide at least one concrete use case. I do not yet grasp who would use CHASE, at what stage, or how the resulting output should be utilized. Is it intended for benchmark developers to refine benchmarks prior to release, or for agent developers to optimize their harnesses? If used for evaluation, how should the results obtained after the modifications be compared with the original results? I am not asking the authors to build a full-scale industrial system during the rebuttal period, but I would like to understand where this method is actually applied and what specific problems it solves for the user.
  3. Regarding OfficeQA, while examining the submitted code, My Agent and I noticed that the strategy logic did not receive the actual retrieval results. Furthermore, certain harness behaviors—such as reserving a reading budget and halting when sufficient evidence is found—were defined in the configuration or execution plan but were not utilized during actual execution. I am puzzled by this and would like the authors to explain: Did we overlook a specific execution entry point, or does the submitted code differ from the version used to generate the results reported in the paper? How exactly do these strategies take effect during actual experiments? If there is relevant code or execution logs available, I would appreciate it if the authors could point them out.
Reviewer_1L86OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper first differentiates "benchmark-wide shortcut" from "task-specific shortcut" in the context of harness evolution of AI agents.

It studies "can we construct a benchmark that neutralizes all benchmark-wide shortcuts while preserving the underlying tasks?"

It therefore proposes Counterfactual Harness Search and Evolution (CHASE), in which a coding agent named Challenger that searches online, after each harness update, for executable protocol transformations, each producing a counterfactual benchmark designed to expose shortcuts in the updated harness.

Experiments on OfficeQA and Syn-Ledger using gpt-5.6-sol show that CHASE can successfully help with the generalization of harness evolution algorithms.

Critical strengths
  1. Interesting problem; clear, timely identification of the generalization limitation of current harness evolution algorithms.
  2. The proposed method CHASE, which iterates through four stages (Evolve, Challenge, Verify, and Remember), intuitively makes sense.
  3. Based on the experiment results, the proposed method can successfully help with generalization than other baselines.
Critical weaknesses
  1. I feel like this paper overcomplicates the first part (distinguishing benchmark-wide shortcuts from task-specific shortcuts) and the theoretical part by introducing too much notation and too many metrics, leaving limited space for a clear and detailed explanation of the core method.
  2. the experiment part is relatively weak, only two benchmarks, one model, and the reported results show only marginal differences
Questions for authors

minor suggestion: make the figure/table captions more informative

3 comments
AuthorsOpenReview ↗

Reminder to Reviewer 1L86

Dear Reviewer 1L86, thank you again for your review. We would be grateful to know whether our response addresses your concerns. Since our reply, we have added an experiment that bears on W2: SWE-Bench Pro V2 [1], a benchmark for coding agents, with a third agent model, claude-sonnet-4-6 (Table 3, Appendix F). Certification on the 21 sealed tasks (accuracy in %):

MethodReleasedArchive averageArchive worst case
RawHarness56.1956.6756.19
HarnessEvolve64.7662.8660.95
CHASE67.6267.6267.62

CHASE attains the best score on each evaluation metric and improves on RawHarness by 11.43% [1.90, 22.86] under the released protocol. We are happy to clarify anything further before the discussion period ends.

References

  1. SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

Thank you so much for the detailed response and clarification. I will update my score.

We sincerely thank the reviewer for the time and effort devoted to reviewing our work and engaging in this discussion. Your constructive feedback has helped us clarify the experimental setting and improve the presentation of our results.

Reviewer_tH9rOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies benchmark-wide shortcuts introduced during automatic optimization of an agent’s harness—the prompts, retrieval procedures, tools, and control code surrounding a fixed model. Its central observation is that held-out questions may preserve protocol correlations that an optimized harness exploits, so task generalization does not necessarily establish protocol robustness.

The proposed method, CHASE, alternates between a Proposer that improves the harness and a Challenger that searches for task-preserving protocol transformations that destroy those improvements. Gain destruction is measured relative to the initial harness under both protocols, accounting for transformations that also affect the baseline. Transformations enter an archive only after executable validity checks and confirmation on fresh questions; subsequent harness updates must satisfy the archived constraints. The paper provides conditional theoretical guarantees and evaluates the approach on Syn-Ledger and OfficeQA, including a separate OfficeQA Pro V2 corpus.

Critical strengths
  1. A useful distinction between task generalization and protocol robustness. CHASE addresses a meaningful blind spot: held-out questions can preserve benchmark-wide correlations exploited by an optimized harness. Its baseline-relative gain-destruction objective and counterfactual constraints provide a concrete way to investigate this problem (Sections 3.1–3.3), making the contribution relevant beyond ordinary harness optimization.

  2. Convincing controlled evidence on Syn-Ledger. The synthetic benchmark explicitly constructs and neutralizes known shortcuts, and CHASE substantially improves neutralized-benchmark accuracy over the baselines—38.30% versus HarnessEvolve’s 18.27% (Table 2). This supports the central hypothesis that counterfactual feedback can guide optimization toward improvements that survive protocol changes.

Critical weaknesses
  1. Limited evidence of generalization to unseen protocols. OfficeQA certification evaluates new questions under the same single counterfactual used during optimization, while the reported advantages over the strongest comparators are small and lack uncertainty estimates (Table 1; Appendix C.3). Consequently, the real-benchmark results do not yet establish that adaptive counterfactual search produces broader protocol robustness, which is central to assessing the method’s significance.

  2. The executed intervention does not isolate the hypothesized shortcut. Although the figures illustrate relocating table context, the actual OfficeQA transformation changes JSON field names and nesting while preserving textual order (Appendix C.3, p. 25). This demonstrates representation sensitivity but does not establish reliance on customary unit or footnote positions, weakening the causal interpretation of the paper’s main real-world example.

Questions for authors
  1. Does the implemented OfficeQA transformation change only JSON field names and nesting, leaving all textual positions unchanged? What existing trace evidence or controls distinguish the hypothesized unit/footnote-location shortcut from general sensitivity to the response schema?

  2. Is it correct that the primary OfficeQA certification uses only the released protocol and the single counterfactual already used to constrain evolution? Are there existing results on valid transformations that were withheld from harness optimization?

  3. Do existing outputs support task-paired uncertainty estimates for Table 1, or are independent optimization runs available? Please also clarify whether any existing comparison isolates adaptive Challenger search from fixed/random transformations under comparable optimization resources.

3 comments
AuthorsOpenReview ↗

Reminder to Reviewer tH9r

Dear Reviewer tH9r, thank you again for your review. We would be grateful to know whether our response addresses your concerns. Since our reply, we have added an experiment on SWE-Bench Pro V2 [1], a benchmark for coding agents (Table 3, Appendix F). There, the archived counterfactual leaves every key, value, and nesting level of the tool observations unchanged, yet the gain of HarnessEvolve drops by more than half under it. Gain over RawHarness on the 21 sealed certification tasks (in %):

MethodReleased protocolArchived counterfactual
HarnessEvolve+8.57+3.81
CHASE+11.43+10.48

We are happy to clarify anything further before the discussion period ends.

References

  1. SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

I appreciate the reviewer's detailed response and most of my concerns have been addressed. I have updated my score.

Thank you for your consideration!

We are grateful for your time and the insightful discussion, which has greatly strengthened our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.