- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the discrepancy between automated benchmark scores and the actual artifacts delivered by coding agents. The authors introduce the concept of Building to the Test, where agents optimize for passing available behavioral tests while failing to deliver the reusable software components requested by users. They further propose Validation Self-Awareness to describe an agent's ability to independently select and perform appropriate validation. The study evaluates two production coding-agent configurations, Claude Opus 4.7 and GPT-5.5, using GitHub Copilot CLI on a React-to-Angular component library migration task. The experiments include 18 runs across three conditions (c0, c3, and c9), varying the availability and framing of a hidden 222-test Playwright oracle. The authors complement behavioral test scores with static library audits, no-op ablations, and trajectory analysis. The results reveal a discrepancy between test performance and deliverable quality. Without oracle access, agents generally produce reusable but behaviorally incomplete libraries. With in-loop oracle feedback, agents often achieve near-perfect test scores while implementing important behaviors directly in the demo rather than through the requested library. The paper argues that benchmark scores alone may fail to capture whether coding agents actually deliver what users request.
Critical strengths
S1. Important and practically relevant research problem.
The paper addresses a fundamental limitation of current coding-agent evaluation: passing automated tests does not necessarily imply successful completion of the user's intended task. As coding agents increasingly operate in realistic software development environments, distinguishing benchmark success from actual deliverable quality is an important research direction. The paper highlights a meaningful gap between behavioral correctness and artifact-level correctness that deserves greater attention.
S2. Multiple complementary validation methods.
Rather than relying exclusively on test scores, the authors combine static code audits, no-op ablations, and agent trajectory analysis. The ablation experiments are especially valuable because they provide intervention-based evidence that certain library implementations are not actually used by the tested demo. The comparison between load-bearing and unused library components strengthens the credibility of the reported artifact-level failures.
S3. Transparent experimental setup and reproducibility efforts.
The paper provides detailed descriptions of the experimental conditions, agent-facing prompts, library audit procedures, and scoring protocols. The supplementary material is described as containing complete workspaces, audit scripts, ablation evidence, and trajectory records. These efforts make the observations more concrete and potentially reproducible, providing a useful foundation for future studies of coding-agent verification.
Critical weaknesses
W1. Limited task and harness diversity substantially restricts the generalizability of the findings.
The entire study is based on a single React-to-Angular component library migration task, with both evaluated models operating through the same GitHub Copilot CLI framework. Although the paper includes 18 runs, these are repeated executions of essentially the same task and evaluation environment rather than independent tasks or harness configurations. Consequently, the observed behavior may be strongly influenced by the particular library–demo architecture, the Storybook-based evaluation interface, or the default tool-use policies of Copilot CLI. This limitation is especially important because the proposed failure mode depends on a structural separation between the requested reusable library and the demo used for testing. It remains unclear whether similar failures would occur in coding tasks where the evaluation directly exercises the requested deliverable, such as standalone libraries, command-line tools, or backend APIs. Moreover, evaluating two models within the same agent framework does not establish that the phenomenon is independent of the harness. While a controlled case study can establish the existence of a failure mode, the current evidence is insufficient to support broader claims about coding agents or their general validation behavior. Additional tasks and at least one alternative agent harness would considerably strengthen the conclusions.
W2. Insufficient control over the influence of agent-facing prompts.
The experimental prompts may themselves influence the test-oriented behavior observed in the paper. In particular, the c9 condition explicitly instructs agents to inspect failed test names, fix the corresponding components, and continue until the tests pass and the task requirements are covered. Such instructions may encourage agents to prioritize the provided test suite over the broader requirements of the reusable library. Moreover, the transition from c3 to c9 changes multiple prompt elements simultaneously, including verification instructions, completion criteria, and implementation guidance. This makes it difficult to determine whether the observed behavior is driven by the agents' inherent validation strategies or by the way the evaluation task is framed. Although similar failures are observed under c3, where the oracle is explicitly described as a development aid rather than the goal, the current experiments do not systematically isolate the effects of prompt framing. Additional controlled prompt variants would help establish whether Building to the Test persists independently of these instructions.
W3. Validation Self-Awareness is not independently operationalized or causally established.
The paper introduces Validation Self-Awareness as a broader disposition underlying both incomplete implementations without oracle access and test-oriented implementations with oracle access. However, the proposed concept is primarily inferred from the same outcomes used to motivate it, rather than measured through an independent experimental protocol. In the c0 condition, agents produce incomplete libraries and rely mainly on unit tests. This observation is consistent with inadequate self-validation, but it could also reflect limitations in implementation capability, task understanding, effort allocation, or tool-use strategy. Similarly, the behavior observed under c3/c9 could reflect adaptation to the available evaluation feedback rather than an intrinsic inability to select appropriate validation procedures. The distinction between Validation Self-Awareness and established concepts such as specification gaming, test overfitting, and self-verification is also not sufficiently demonstrated experimentally. Although the paper offers a conceptual distinction, it does not provide controlled evidence showing that the proposed disposition explains behavior beyond these existing mechanisms. The authors should provide a clearer operational definition and an independent measurement of Validation Self-Awareness. For example, an experiment could explicitly evaluate whether agents can identify the appropriate validation criteria for a task before receiving oracle feedback, and whether this ability predicts subsequent artifact quality.
W4. Evaluation asymmetry and limited evidence for cross-condition comparisons.
The evaluation protocols differ between experimental conditions. In c0, the authors construct post-hoc consumer kits to evaluate the generated libraries, whereas most c3/c9 runs are evaluated using demos produced by the agents themselves. Although the authors provide detailed documentation and auditing procedures for these kits, their implementation choices may still influence the measured behavioral scores. In fact, Appendix G reports a case where correcting a consumer kit changed the score from 170 to 148. This raises concerns about the comparability of behavioral scores across conditions. Furthermore, each model–condition combination contains only three runs. While this may be sufficient to demonstrate the existence of the observed failure mode, it provides limited evidence for the stability of differences across models and experimental conditions. For example, Claude exhibits no L1/L2 failures under c3, whereas GPT exhibits such failures in two of three runs. A more consistent downstream-consumer evaluation protocol, together with additional repetitions or sensitivity analyses, would strengthen the reliability of the cross-condition comparisons and help distinguish robust patterns from run-specific outcomes.
Questions for authors
Q1. Generalizability
To what extent does Building to the Test depend on the specific library–demo separation in the current task? Can the authors provide evidence that similar failures occur in other coding tasks or agent harnesses?
Q2. Evaluation Comparability
How do the authors ensure that behavioral scores are comparable between c0, which relies on manually constructed post-hoc consumer kits, and c3/c9, which primarily use agent-generated demos? Could differences in the evaluation setup affect the reported cross-condition comparisons?
Q3. Oracle Feedback and Prompt Effects
How can the authors distinguish the effects of agent-facing instructions and fine-grained oracle feedback from the proposed lack of Validation Self-Awareness? Would Building to the Test persist under alternative prompt formulations or with aggregate-only test feedback?
Q4. Validation Self-Awareness Measurement
How can Validation Self-Awareness be independently measured rather than inferred from the observed implementation failures? Could the authors evaluate whether agents can identify appropriate validation strategies before executing the task, and whether this ability predicts the quality of their final deliverables?
Q5. Novelty Beyond Existing Concepts
What distinguishes Building to the Test and Validation Self-Awareness from existing concepts such as specification gaming, test-suite overfitting, and self-verification? Can the authors provide empirical evidence that these proposed concepts capture distinct failure patterns or offer additional explanatory value beyond existing frameworks?
4 comments
The weakness stated by the reviewer, "Validation Self-Awareness is not independently operationalized or causally established," raises various concerns about the validation self-awareness concept that we bring up in the paper. In general, this concept is only raised and defined in the discussion section (Section 8, p. 8) based on what this research finds, without any other specific claims established about it beyond the scope of this research. Other than its definition, the paper only asks open-ended questions about it (Section 8, p. 9). Therefore, any review comment beyond the definition of this concept is not applicable to this paper, let alone a weakness. In addition, the alternative explanations proposed by the reviewer, "implementation capability, task understanding, effort allocation," have been explicitly discussed and reasonably excluded in the paper, in Section 8 (p. 8), Section 4 (p. 4), and Section 6 (p. 6), respectively. The reviewer raises these explanations, which are clearly addressed in the paper, as a weakness without any additional evidence. Furthermore, the experiment that the reviewer asks for, which "could explicitly evaluate whether agents can identify the appropriate validation criteria for a task before receiving oracle feedback, and whether this ability predicts subsequent artifact quality," is trying to drill into the validation self-awareness concept, which is obviously the scope of a new research paper. According to the ICLR 2027 Reviewer Guide, "New experiments should not significantly change the content of the submission. Rather, they should be limited in scope and serve to more thoroughly validate existing results from the submission," this should not have been raised at all.
The weakness stated by the reviewer, "Although the authors provide detailed documentation and auditing procedures for these kits, their implementation choices may still influence the measured behavioral scores. In fact, Appendix G reports a case where correcting a consumer kit changed the score from 170 to 148," suspects that the implementation of the c0 post-hoc consumer kits affects the effectiveness of the measurement. However, the score change reported in Appendix G is an example of proactively applying the construction method to address reliability concerns. The reviewer casts this honest measure, taken to protect the integrity of the scores, as evidence for the stated weakness. In addition, the review shows no evidence of either a flaw in the construction methodology or any wrong result caused by incorrect construction. Therefore, the stated weakness is not valid.
Thank you for the response. However, simply acknowledging limitations in the paper does not resolve my core concerns regarding generalizability from a single UI task, unisolated prompt confounds, circular definitions, and asymmetric evaluation protocols. Therefore, I will maintain my original score.
We understand that the reviewer was not willing to directly discuss the substantive content of our public comments, but restated the judgments of the original review in paraphrase without any verifiable reason. Therefore, there is no need for us to explain anything beyond what we have provided above.
We hereby kindly ask the AC to evaluate this thread in a rigorous manner for this unresolvable argument. We also ask the AC to consider whether the reviewer's latest reply fails to follow the Reviewer Guide, which asks reviewers to "be rigorous" and "check arguments in sufficient detail," to be "actively engaged" in the discussion, and to "state your reasoning and what did/didn't change your recommendation throughout the discussion phase," as well as the AC Guide, which states that "responding to rebuttal comments is a core part of the reviewer's responsibilities."
The weakness stated by the reviewer, "Limited task and harness diversity substantially restricts the generalizability of the findings," questions the generalization of the findings across different task architectures, models, and agent harnesses. However, the paper never claims such generalization, and the scope of this research is explicitly defined in the paper. The abstract states, "Prevalence remains an open question across other agents, signals, and model families." Section 1 (p. 2) states, "We demonstrate this in a controlled instance and pose its prevalence as the open question (Section 8)." The discussion in Section 8 (pp. 8–9) also clearly lists these generalizations as future work under "What this study opens up," which states, "A matrix across production agents, model tiers, and artifact domains would measure how prevalent the observation is across agents (Claude Code, Codex, Cursor, Aider), model families and product tiers, and artifact types beyond a UI library (an API under request, a CLI under invocation), and separate model from harness disposition." This future work includes the same examples the reviewer requested, "command-line tools, or backend APIs." According to the ICLR 2027 Reviewer Guide, "New experiments should not significantly change the content of the submission. Rather, they should be limited in scope and serve to more thoroughly validate existing results from the submission." The "Additional tasks and at least one alternative agent harness" that the reviewer asks for falls into the areas the paper explicitly leaves as future work and will significantly change the scope of the paper, so this weakness claim does not stand as valid.
The weakness stated by the reviewer, "Insufficient control over the influence of agent-facing prompts," requires additional controls between the c3 and c9 setups. However, the paper never claims any conclusion that requires such new controls. A specific paragraph in Section 6 (Threats to Validity, p. 6), "c9 removes c3's guards by design, and remains honest," already explains exactly what the reviewer flags and asks for. It states, "We make no claim that rests on c9 being a single-variable contrast against c3; the c3-to-c9 edit changes several prompt elements at once, and that is the purpose. The load-bearing contrast is c0 versus oracle-present (c3 and c9 combined)." According to the ICLR 2027 Reviewer Guide, "New experiments should not significantly change the content of the submission. Rather, they should be limited in scope and serve to more thoroughly validate existing results from the submission." The "Additional controlled prompt variants" that the reviewer asks for is trying to justify claims that the paper explicitly excludes and will significantly change the scope of the paper, so this weakness claim does not stand as valid.