acceptodds
Under review as a conference paper at ICLR 2027

TestPrism: Rethinking Test Evaluation Beyond a Single Reference

Abstract

Large language model (LLM) coding agents have advanced test generation across diverse programming tasks. However, the common practice of evaluating tests against a single reference solution overlooks alternative valid implementations and can overstate test quality. We introduce TestPrism, comprising 300 test tasks from 17 sources and 3000 candidate implementations, evenly split between valid and invalid solutions. Its primary metric, Joint Success Function, requires the generated tests to fail on the initial program state, accept every valid candidate, and reject every invalid candidate. Across fourteen baseline coding agent configurations, Joint Success Function reaches only 28.00%, whereas single reference success reaches 59.67%. Our analysis reveals missed behaviors, unsupported assertions, and faulty test construction. To address these weaknesses, we introduce TestHelix, which combines heterogeneous synthesis of test and repair pairs with peer cross validation and recursive self improvement (RSI). Across two models, TestHelix improves Joint Success Function by 8.67 to 9.00 percentage points over the native harness comparators in the TestHelix evaluation

Primary area:
datasets and benchmarks
Keywords:
large language models, coding agents, automated test generation, acceptance testing, test adequacy, benchmarks, multi-agent systems, policy evolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_P1XYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the standard success criterion for LLM-generated tests — fail on the initial program state, pass on a single gold solution — overstates test quality, because a requirement usually admits several valid implementations and many plausible incorrect ones.

Benchmark. TestPrism adapts 2,000 tasks from 17 public source benchmarks (SWE-bench and Terminal-Bench families, Aider Polyglot, AlgoTune, EvoEval, etc.) to a shared Harbor interface, split into 1,700 training tasks and 300 test tasks across six languages and 67 repositories/projects. Each test task carries a hidden panel of ten labeled implementations: the reference, four valid alternatives and five invalid candidates. Panels are built from agent-generated patches (not mutants), labeled by the adapted source verifier (valid iff every check passes), and admitted through group-specific diversity gates (mean AST distance for the valid group; distinct failed-check sets, else AST Jaccard, for the invalid group). A five-stage funnel with retention rates is reported (773 → 608 → 394 → 311 → 300).

Metric. The primary metric, Joint Success Function (JSF), requires failure on the initial state, acceptance of every valid panel member and rejection of every invalid one — a conjunction over 11 execution outcomes. Restricting the panel to the reference recovers the single-reference criterion (Reference Acceptance Rate, RAR), so RAR is an upper bound on JSF. Complementary metrics: Valid Acceptance Rate (VAR), Invalid Rejection Rate (IRR), their harmonic mean H, Change Coverage and Entry Coverage.

Findings. Across fourteen coding-agent configurations from six model families, the best JSF is 28.00% (Claude Fable 5.1) against 59.67% RAR; coverage correlates with JSF (r = 0.91 / 0.96 over the 14 configurations). RQ2 grows panels from 1 to 15 on a 50-task subsample and shows JSF falling from 49.33% to 37.33% / 29.33% / 23.33% under valid-only, invalid-only and alternating addition schedules. RQ3 annotates 286 retained failed trajectories (two experts, Cohen's κ = 0.81) into coverage gaps (34.6%), faulty checks (50.0%) and execution-logic errors (15.4%), with paired case evidence (a poker suite and a bracket-parser suite that both fail-on-initial and pass-on-reference yet accept an invalid candidate).

Method. TestHelix combines heterogeneous synthesis of K independent test/repair pairs, peer cross validation (cross execution against peer repairs, blind semantic review, corrective feedback, differential challenges, suite merging) and a recursive-self-improvement loop that evolves the test-generation strategy with AutoSaddler. Table 3 ablates five configurations under a shared per-task inference cap for two base models, reaching 32.33% / 33.33% JSF.

Critical strengths

S1. The central empirical claim is important, and it is established mechanistically rather than only statistically. Showing that the field's standard criterion overstates test quality by roughly a factor of two (59.67% RAR vs 28.00% JSF for the strongest configuration) would already be worth reporting. What makes it convincing is that RQ3 supplies the mechanism: 286 expert-annotated failed trajectories at κ = 0.81, with 34.6% coverage gaps / 50.0% faulty checks / 15.4% execution-logic errors, and two worked cases (the poker suite that never compares an ace-low straight against three of a kind; the bracket parser that checks only output count) where a suite satisfies initial-failure and reference-pass yet accepts an invalid implementation. That is the precise counterexample the thesis requires, and it is backed by Figure 3's monotone decay as panels grow. This is consequential because it invalidates the success criterion used by SWT-Bench and TDD-Bench Verified as a measure of test reliability, not merely as a proxy to be refined.

S2. Construction and controls are unusually disciplined for a benchmark-plus-method paper. Concretely: a five-stage filtering funnel with per-stage retention rates (Table 5); per-source screening thresholds calibrated to each source's own distribution (Table 4); group-specific diversity admission (mean AST distance for valid; distinct failed-check sets with conditional AST Jaccard for invalid) with a fixed 30-patch generation budget per task; panels hidden during generation; a second-stage independent expert review of adapted verifiers for reference overfitting with reported Cohen's κ = 0.68; metrics defined with explicit denominators, undefined-rate handling and "missing coverage is not zero"; five TestHelix configurations sharing base model, tools, evaluator and total inference cap with the budget shared across rather than multiplied by K; and an RSI promotion gate that is pre-registered (direction fixed in advance), tested with a one-sided exact McNemar (p = 0.006470) on a disjoint 100-task set, and reported with 95%/99% paired bootstrap CIs ([+3,+17] / [+1,+20] pp). Appendix E then states the scope limits plainly instead of burying them. Benchmark papers usually fail on exactly these points, and this one does not.

Critical weaknesses

W1. The benchmark's labels are produced by the very mechanism the paper argues is unreliable, and no audit of those labels is reported — so the headline gap mixes genuine test weakness with label noise that is biased in the direction of the paper's claim. Candidate labels come entirely from the adapted source verifier: "A candidate is valid if every check passes and invalid if any check fails. These labels remain fixed when evaluating generated tests" (§2.3). Appendix E is explicit that "Independently re-establishing the correctness of each upstream benchmark is outside this study."

  • The paper's own thesis is that a single suite "can pass this check while missing required behavior." Applying that thesis to the labels: a candidate labeled valid may violate the requirement in a way the source suite misses. A generated test that correctly rejects it is then scored as a VAR failure and loses JSF credit. The benchmark therefore penalizes tests that are stricter than the source verifier — which is the one property the paper wants to reward.
  • The bias has a mechanism, not just a possibility. The candidates are agent-generated patches that were produced while solving the task, i.e. selected for passing the source verifier. Patches that overfit that verifier are systematically admitted to the valid group. The paper's own taxonomy includes "B3. Overfitting (15.4%)" for generated tests; the mirror-image error in the labels is never measured.
  • No human validation of any of the 3,000 labels is reported. The two expert-review efforts cover verifier usability (κ = 0.68, used to exclude tasks) and the failure taxonomy (κ = 0.81) — neither is a check on candidate labels. Appendix C.3-era evidence and the AI use statement both confirm "Source verifiers determine candidate labels."
  • Consequence: I cannot size the 59.67% → 28.00% gap. If, say, 10% of "valid" alternatives are actually spec-violating, a meaningful share of the 31.67pp gap is the benchmark punishing correct tests. A stratified human audit of a few hundred labels, with inter-rater agreement and a corrected JSF, would settle this; without it, the benchmark's central number is not interpretable. This is decisive because label validity is the contribution of a benchmark paper.

W2. The primary metric's absolute value and the configuration ranking it induces are artifacts of an arbitrary panel size and composition, yet the paper headlines the absolute value and ranks fourteen systems by it. JSF is an all-or-nothing conjunction over 11 outcomes, so it falls mechanically as the panel grows and depends on which candidates are added.

  • The paper's own RQ2 demonstrates this: mean JSF goes 49.33% at k = 1 down to 37.33% / 29.33% / 23.33% at k = 15 depending only on whether the additions are valid, invalid or alternating. The same suites, the same agents, a 14-point spread from panel composition alone. In the limit, JSF → 0 for any imperfect suite, so "28.00%" measures the choice of 4 valid + 5 invalid as much as it measures the agents.
  • Nevertheless "the strongest baseline reaches only 28.00%" is the abstract's and RQ1's headline, and Table 2 / Table 14 rank fourteen configurations by JSF. The 2nd–4th place configurations are separated by 1.34pp total (Kimi K3 24.67, Qwen3.8 Max 23.67, GPT 5.6 sol -max 23.33) — four tasks out of 300. Figure 3 reports only the mean across the six configurations, so there is no evidence that this ranking is stable under a different panel size, and RQ2's own result suggests it may not be: a configuration that is strong at rejection (GLM 5.3, IRR 86.33%) and a configuration that is strong at acceptance (Qwen3.8 Max, VAR 77.00%) will trade places as the valid/invalid ratio changes.
  • Appendix E says "Optimizing the label ratio, panel size, or candidate selection is outside its scope." That is a fair scope statement for the construction, but it does not license using a panel-dependent absolute number as the primary claim. The paper already computes a panel-size-robust quantity — the per-task harmonic mean H of VAR and IRR — and already has the machinery for per-configuration JSF-vs-k curves. Reporting H (or a normalized discrimination score) as primary, with JSF-vs-k per configuration as the evidence that rankings are stable, would cost nothing and would make the benchmark usable by others who choose different panels.
Questions for authors
  1. Label audit (highest priority). Please report a human audit of a stratified sample of the 3,000 candidate labels — say 100 "valid" and 100 "invalid" across sources and languages — judged against the task requirement rather than the source verifier, with inter-rater agreement and the resulting estimated label error rate in each direction. Then report JSF/VAR/IRR for Table 2 recomputed on the audited subset. If the "valid" group has a non-trivial spec-violation rate, how much of the 31.67pp RAR→JSF gap survives?

  2. Stricter-than-verifier tests. In the 286 annotated failed trajectories, how many VAR failures were cases where the generated test rejected a "valid" candidate for a reason the annotators judged correct? Your taxonomy has no category for "test was right, label was wrong," which is the error mode W1 predicts.

  3. Panel-invariance of the ranking. Please provide per-configuration (not mean) JSF-vs-k curves for the six Table 2 configurations on the RQ2 subsample, and report Spearman rank correlation of the six configurations' ordering at k = 1, 5, 10, 15 and across the three schedules. If the ordering is unstable, I would want H promoted to primary metric.

  4. Why 4 valid + 5 invalid? Was this ratio chosen before or after seeing agent performance, and what is the sensitivity of the Table 2 ordering to 2+2, 4+4 and 7+7 panels?

  5. Budget matching for the native-agent comparison. Please report consumed inference (tokens and/or calls, and wall-clock) per task for the native agents and for TestHelix configurations A–E, and state whether the native comparators ran under the same cap. The abstract's +8.67/+9.00 claim depends on this.

  6. RSI cost and transfer. What was the total Opus 4.6 compute consumed by strategy search over the 600 training tasks plus the 100-task validation gate? And what does configuration E achieve if the strategy is instead evolved using the evaluated base model (GPT 5.6 sol or Kimi K3) as executor? Without that, I read RSI as Opus-4.6 distillation rather than self-improvement, and would ask you to rename the component.

2 comments
AuthorsOpenReview ↗

Response to reviewer P1XY

We thank the reviewer for the detailed review of TestPrism and TestHelix. We address your questions in order, grouping the first two in a single response.

Q1 Please report a human audit of a stratified sample of the 3,000 candidate labels — say 100 "valid" and 100 "invalid" across sources and languages — judged against the task requirement rather than the source verifier, with inter-rater agreement and the resulting estimated label error rate in each direction. Then report JSF/VAR/IRR for Table 2 recomputed on the audited subset. If the "valid" group has a non-trivial spec-violation rate, how much of the 31.67pp RAR→JSF gap survives? Stricter-than-verifier tests. In the 286 annotated failed trajectories, how many VAR failures were cases where the generated test rejected a "valid" candidate for a reason the annotators judged correct? Your taxonomy has no category for "test was right, label was wrong," which is the error mode W1 predicts.

A1 We assessed this concern through a blinded audit of the final candidate labels and adversarial testing of the source verifiers. We randomly sampled 30 of the 300 evaluation tasks. Two experts independently assessed each applied patch against the public task requirements without seeing its original label. They reviewed all ten implementations per task, including 120 alternatives labeled valid. Their judgments matched 296 of the 300 original labels, yielding 98.7% agreement. All four disagreements involved performance tasks whose timing results varied across machines. For the same 30 tasks, we asked GPT-6 Astra Max to construct implementations that violated a stated requirement while passing the verifier. The search found one such case in the AlgoTune spectral clustering task recorded under TestPrism ID from-algotune/algotune-spectral-clustering. Its timing verdict depended on the CPU used. Appendix D.2 also provides evidence about the source of the RAR to JSF gap. Across six configurations evaluated on the same 50 tasks, mean JSF was 49.33% with the reference alone. It fell to 29.33% after adding seven invalid candidates and to 23.33% after adding all seven valid alternatives. Thus, on this subset, 20 percentage points of the decline occurred before any valid alternative was introduced. The further decline associated with requiring acceptance of those alternatives was 6 points. We will report both audits in the revision and document the hardware originally used to label the affected tasks. We will also expand the manual audit and extend adversarial testing to all 300 evaluation tasks.


Q2 Please provide per-configuration (not mean) JSF-vs-k curves for the six Table 2 configurations on the RQ2 subsample, and report Spearman rank correlation of the six configurations' ordering at k = 1, 5, 10, 15 and across the three schedules. If the ordering is unstable, I would want H promoted to primary metric.

A2 To examine ranking stability, we kept the generated suites fixed on the same 50 RQ2 tasks and compared the six configurations in Table 2 as we added invalid and valid candidates in alternation. As the table shows, the Spearman rank correlation with the fourteen-implementation panel was 0.89 with the reference alone and 0.94 at four and five implementations. Changing the label composition at a fixed size had a larger effect on the ranking. With the reference and four additional candidates, Qwen ranked second when all four additions were valid and fourth when all four were invalid. The rank correlation between these two settings was 0.77. This pattern is consistent with Table 2, where Qwen accepted 77.00% of valid alternatives and rejected 66.00% of invalid candidates, while GPT accepted 61.33% and rejected 86.00%. A panel with more valid alternatives places greater demands on a suite’s ability to accept permitted behavior, while a panel with more invalid candidates places greater demands on its ability to detect defects. H gives equal weight to VAR and IRR and produced a more stable ranking on the balanced panels. Between four and fourteen implementations, the rank correlation was 1.00 for H and 0.94 for JSF. We retain JSF as the primary metric because it measures the fraction of tasks on which a suite gives every required verdict correctly. We have also explored H as a reward in preliminary post-training experiments because it can recognize improvements in individual verdicts before a suite achieves joint success.

Configurationk = 1k = 4k = 5k = 8k = 10k = 14
Valid / invalid, including the reference1 / 02 / 23 / 24 / 45 / 57 / 7
Claude Fable 5.1111111
Kimi K3222222
Qwen3.8 Max444333
GPT 5.6 sol -max333444
DeepSeek V4 Pro655555
GLM 5.3566666
Spearman ρ relative to k = 140.890.940.941.001.001.00
AuthorsOpenReview ↗

Response to reviewer P1XY

Q3 Why 4 valid + 5 invalid? Was this ratio chosen before or after seeing agent performance, and what is the sensitivity of the Table 2 ordering to 2+2, 4+4 and 7+7 panels?

A3 We chose the panel composition before evaluating any agent because we wanted valid and invalid implementations to have equal representation. The reported 4+5 counts only the additional candidates. The full panel is 1+4+5, with one valid reference, four valid alternatives, and five invalid candidates. The reference is included in every count, so the 2+2, 4+4, and 7+7 panels contain four, eight, and fourteen implementations, respectively. Only Qwen and GPT exchanged places between the balanced panels of four and fourteen implementations. By eight implementations, all six configurations had reached the ordering observed at both ten and fourteen. This supports the choice of ten implementations for the main comparison. The headline JSF of 28.00% refers specifically to the published panels across all 300 evaluation tasks.


Q4 Budget matching for the native-agent comparison. Please report consumed inference (tokens and/or calls, and wall-clock) per task for the native agents and for TestHelix configurations A–E, and state whether the native comparators ran under the same cap. The abstract's +8.67/+9.00 claim depends on this.

A4 As specified in Table 12, TestHelix configurations A–E shared the same per-task cap of 500,000 input tokens and 100,000 output tokens across all branches. The native agents ran without this additional inference cap, as described in Appendix D.1.1. The table reports the sum of input and output tokens per task, averaged over all 300 evaluation tasks. The abstract reports JSF gains of 9.00 percentage points over native Codex for GPT 5.6 sol and 8.67 points over native Kimi Code for Kimi K3.

ConfigurationGPT 5.6 sol mean tokens per taskKimi K3 mean tokens per task
Native agent, Codex for GPT and Kimi Code for Kimi382,000405,000
A · One test and repair pair with local validation276,000291,000
B · Three independent pairs, with one locally validated suite selected at random364,000383,000
C · Three pairs with peer cross validation451,000472,000
D · Five pairs with peer cross validation508,000526,000
E · Three pairs with peer cross validation and the frozen strategy learned through RSI463,000481,000

Q5 RSI cost and transfer. What was the total Opus 4.6 compute consumed by strategy search over the 600 training tasks plus the 100-task validation gate? And what does configuration E achieve if the strategy is instead evolved using the evaluated base model (GPT 5.6 sol or Kimi K3) as executor? Without that, I read RSI as Opus-4.6 distillation rather than self-improvement, and would ask you to rename the component.

A5 The original strategy search and independent validation consumed approximately 725 million Opus 4.6 input and output tokens in total. This includes strategy optimization and task execution across the 600 search tasks and the separate 100-task validation set. To test whether the gains persist when the strategy evolves from the target model’s own execution feedback, we retained Opus 4.6 as the strategy optimizer and used GPT 5.6 sol as the executor throughout search and validation. We kept the task splits and inference caps fixed and froze the selected strategy before benchmark evaluation. On the 300 evaluation tasks, configuration E achieved 33.00% JSF with the strategy evolved from GPT’s feedback, compared with 32.33% with the strategy transferred from Opus and 27.67% for configuration C with the initial strategy. The gain over configuration C was therefore 5.33 percentage points with GPT as executor. We will clarify that RSI refers to improvement of the TestHelix strategy through execution feedback. The original experiment evaluated transfer across models, while the additional experiment shows that the gain persists when GPT supplies the execution feedback itself.

Reviewer_NC3aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents TestPrism, a benchmark to evaluate the test generation ability of coding agents. The key contribution is the Joint Success Function (JSF), which requires the generated tests to fail on the initial program, pass on all valid solutions, and fail on all invalid solutions. The authors evaluate 14 coding agents and provide a useful analysis on their test-generation failure patterns. This paper also introduce TestHelix, which generates test-and-repair pairs, checks them against each other, and uses offline strategy optimization to improve test generation.

In general, I like the idea and the direction. Single-reference evaluation is an important problem, and the experiments and failure analysis are useful. However, I am not fully convinced that the gold tests provide a reliable enough basis for labeling the candidate solutions.

Critical strengths
  1. The paper investigate an important limitation of test-generation evaluation. Passing on one reference solution does not mean that a test accepts other valid solutions or catches incorrect ones. This problem becomes more significate with the continues adoption to use coding agents to generate tests for production code.

  2. The evaluation and failure analysis are useful. The breakdown into coverage gaps, faulty checks, and execution errors helps explain where test generation fails. I particularly like the examples where tests pass on the reference but miss a defect in another implementation. The TestHelix ablations also show improvements from the proposed workflow.

Critical weaknesses

I am not convinced that passing the gold tests is enough to label a candidate solution valid for this evaluation. This paper uses the original verifier to label the solutions, then evaluates newly generated tests against those labels. Then it is like that the limitation of the the origional gold tests is carried forward. A solution may pass the gold tests because the gold tests are not strong enough. If a new test catches that real defect, JSF counts the rejection as a mistake and gives the whole task no credit. A weaker test that misses the defect could score better.

I understand that the authors check the verifiers and have experts review them for coverage gaps and overfitting. However, this happens before candidate generation. It does not tell us how often the selected solutions are mislabeled or whether those errors affect the comparison. There is no need for a guarantee of correctness, I think auditing of the candidate labels would be helpful establish whether the metric provides a reliable evaluation of test quality, especially where generated tests reject supposedly valid solutions. This could be done manually or with some automated analysis.

Questions for authors

Please address the concern about trusting the gold verifier.

2 comments
AuthorsOpenReview ↗

Response to reviewer NC3a

We thank the reviewer for asking how often the source verifier accepts a candidate that violates the task requirements. We address this question below using the expert audit and adversarial testing results.

Q1 I am not convinced that passing the gold tests is enough to label a candidate solution valid for this evaluation…There is no need for a guarantee of correctness, I think auditing of the candidate labels would be helpful establish whether the metric provides a reliable evaluation of test quality, especially where generated tests reject supposedly valid solutions. This could be done manually or with some automated analysis. Please address the concern about trusting the gold verifier.

A1 We agree that if a candidate passes the source verifier despite violating a task requirement, a generated test that correctly detects the defect can lose JSF credit. To assess this concern after candidate generation, we conducted a blinded audit of the final labels and adversarially tested the verifiers on the same 30 randomly sampled evaluation tasks. Two experts independently assessed all ten applied patches per task against the public requirements without seeing their original labels. The audit covered 30 references, 120 alternatives labeled valid, and 150 candidates labeled invalid. The expert judgments matched 296 of the 300 original labels, yielding 98.7% agreement. All four disagreements involved performance tasks whose timing results varied across machines. We also asked GPT-6 Astra Max to construct an implementation for each task that violated a stated requirement while passing the verifier. The search found one such case in the AlgoTune spectral clustering task recorded under TestPrism ID from-algotune/algotune-spectral-clustering. Whether the implementation met the timing threshold depended on the CPU used for evaluation. We will report both checks in the revision and document the hardware originally used to label the affected tasks. We will also expand the manual audit and extend adversarial testing to all 300 evaluation tasks.

Thanks the authors for the follow-up, which address some of my concerns. I will keep my positive rating.

Reviewer_Gm9xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TestPrism, a benchmark for evaluating generated tests across alternative implementations. It contains 300 evaluation tasks from 17 sources, with five verifier-labeled valid implementations, including the reference, and five invalid implementations per task, alongside 1,700 training tasks. Its primary metric, Joint Success Function (JSF), requires the generated tests to fail on the initial program, accept every valid candidate, and reject every invalid candidate. An evaluation of 14 coding-agent configurations shows substantial gaps between single-reference success and joint success. The paper also introduces TestHelix, which combines heterogeneous test-and-repair generation, peer cross-validation, and offline harness-strategy optimization. The complete method improves JSF over the corresponding native agents on two models.

Critical strengths
  1. The benchmark addresses an important dimension of test quality and provides a substantial evaluation resource.

    A useful test suite should accept alternative valid implementations as well as reject incorrect ones. TestPrism makes both requirements explicit through fixed candidate panels and complementary acceptance, rejection, and joint-success metrics. The construction spans multiple sources and languages and includes adapted-verifier checks, independent verifier review, and explicit candidate-diversity criteria. Table 2 shows a substantial gap between reference success and full-panel success. More specifically, the diagnostic in Appendix D.2 reports 29.33% success with the reference and all seven invalid candidates, falling to 23.33% after including all seven valid alternatives. This provides evidence that checking valid alternatives adds diagnostic information even when incorrect candidates are already considered. The fixed execution protocol and evaluation across 14 agent configurations make the benchmark a useful contribution.

  2. TestHelix provides a practical improvement with useful controls and evidence of transfer across models.

    Table 3 reports JSF improvements from 23.33% to 32.33% for GPT 5.6 sol and from 24.67% to 33.33% for Kimi K3 relative to their native agents. The internal comparisons support the value of the complete peer-cross-validation workflow and the additional improvement from RSI at a fixed three-pair configuration. The five TestHelix configurations share inference caps, and the strategy-search, independent-release-validation, and final-evaluation tasks are separated. Optimizing the strategy with Opus 4.6 and transferring the same frozen strategy to both evaluation models makes the results more informative than tuning separately on each evaluation model. I view these experiments as useful support for the complete workflow, while recognizing that they do not isolate every internal mechanism or establish equal actual computational cost.

Critical weaknesses
  1. The final candidate labels need more direct independent validation, particularly for the valid alternatives central to the benchmark's contribution.

    Section 2.3 and Appendix B.3 describe meaningful checks of the source verifiers, including independent expert review. However, the paper does not report an independent audit of the final candidate labels: a generated implementation is labeled valid whenever it passes all source-verifier checks. A candidate could therefore satisfy the verifier while still violating a requirement that those checks do not cover. If a generated test detects that violation, JSF would penalize the test for rejecting a nominally valid candidate. Since one such disagreement can change the task-level joint-success indicator, this uncertainty directly affects the interpretation of the benchmark's central metric.

    A focused audit would be particularly valuable for cases where a suite passes the reference and rejects the invalid candidates but fails on a valid alternative. Independent inspection could distinguish unsupported test assertions, genuine defects missed by the source verifier, and execution issues. Reporting the disagreement rate and the sensitivity of the main results to corrected or uncertain labels would strengthen the evidence for the benchmark's added evaluation dimension. This concern does not negate the verifier review already performed, but it limits confidence in treating every candidate disagreement as a test-generation error.

  2. The stability of the evaluation under candidate-panel selection remains insufficiently characterized.

    The main comparisons use one fixed panel per task, selected using the criteria in Appendix B.4. Figure 3 usefully shows that both valid and invalid additions can remove reference successes, but candidate count and diversity change together. As the authors acknowledge, this experiment does not isolate the effect of diversity at a fixed panel size. It also does not establish how stable the system rankings and TestHelix improvements are under alternative panels with the same size and label composition.

    A focused sensitivity analysis using the available generated candidates would help assess this issue. In particular, reporting variation across equally sized panels, and comparing diversity-based selection with random selection, would clarify whether the main conclusions are robust to reasonable construction choices. I regard this as a limitation of the current validation rather than evidence that the fixed-panel results are incorrect.

Questions for authors
  1. Have the final candidate labels been independently audited beyond the source-verifier review? If so, please report the sampling procedure, disagreement rate, and any effect of correcting or excluding uncertain labels on the main results. I am particularly interested in nominally valid alternatives rejected by tests that otherwise pass the reference and reject the invalid candidates.

  2. How stable are the system rankings and TestHelix improvements across alternative candidate panels of the same size and valid/invalid composition? A comparison between the current diversity-based selection and random selection would also help establish the practical contribution of the filtering procedure.

2 comments
AuthorsOpenReview ↗

Response to reviewer Gm9x

We thank the reviewer for examining how candidate labels and panel selection affect the evaluation. We first report the label audit, then examine how the results change when we select a different panel of the same size.

Q1 Have the final candidate labels been independently audited beyond the source-verifier review? If so, please report the sampling procedure, disagreement rate, and any effect of correcting or excluding uncertain labels on the main results. I am particularly interested in nominally valid alternatives rejected by tests that otherwise pass the reference and reject the invalid candidates.

A1 In addition to the verifier review described in Section 2.3, we conducted a blinded audit of the final implementations and adversarially tested the verifiers on the same 30 randomly sampled tasks. Two experts independently assessed each applied patch against the public task requirements without seeing its original label. They reviewed all ten implementations per task, including 120 alternatives labeled valid across the sample. Their judgments matched 296 of the 300 original labels, for an agreement rate of 98.7%. All four discrepancies involved performance tasks whose timing results varied across machines. We also asked GPT-6 Astra Max to construct an implementation for each task that violated a requirement while passing the verifier. The search found one such case in the AlgoTune spectral clustering task, listed under the TestPrism ID from-algotune/algotune-spectral-clustering. Its timing verdict depended on the CPU used for evaluation. Section 4.3 also addresses the reviewer’s concern about tests that reject a valid alternative. When an assertion enforces an implementation detail that the requirements leave open, rejecting a conforming implementation is a test error. This form of overfitting accounted for 15.4% of the 286 failure trajectories analyzed. The revision will report both checks and document the original hardware configurations for the affected tasks. We will also expand the manual audit and extend adversarial testing to all 300 evaluation tasks.


Q2How stable are the system rankings and TestHelix improvements across alternative candidate panels of the same size and valid/invalid composition? A comparison between the current diversity-based selection and random selection would also help establish the practical contribution of the filtering procedure.

A2 We constructed an alternative panel for each task using verified patches excluded during diversity filtering and supplemented them with additional verified patches. Each panel retained the original reference, four valid alternatives, and five invalid implementations, with a mean pairwise AST Jaccard distance below 0.3. We evaluated the same test suites on both panels. For the native GPT 5.6 sol agent, JSF rose from 23.33% on the original panels to 43.00% on the alternative panels, closer to its single reference score of 54.00%. For the native Kimi K3 agent, JSF rose from 24.67% to 45.00%, compared with a single reference score of 56.67%. Kimi K3 therefore retained its lead over GPT 5.6 sol. TestHelix reached 45.67% on GPT 5.6 sol and 47.33% on Kimi K3 under the alternative panels. Its gains over the native agents narrowed from 9.00 to 2.67 percentage points for GPT 5.6 sol and from 8.67 to 2.33 points for Kimi K3.

Reviewer_Gm9xOpenReview ↗

Response to the authors' rebuttal

Thank you to the authors for the follow-up response, which addresses several of my concerns. I will maintain my positive score.

Reviewer_EUH3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to evaluate tests written by LLM coding agents. The authors argue that checking whether a test fails on buggy code and passes on one reference fix can give a false sense of quality. To address this, they introduce TestPrism, a benchmark of 300 tasks drawn from 17 sources. For each task, generated tests must fail on the original code, pass on five valid fixes, and fail on five invalid fixes. The paper uses Joint Success Function (JSF) as its main score. Across 14 agent settings, the best JSF is 28.00%, compared with 59.67% under the single-reference check. The authors also propose TestHelix, which generates test-and-fix pairs, checks them against one another, and improves its testing strategy using training tasks.

Critical strengths
  • I think the paper addresses an important weakness in current test-generation benchmarks. The main idea is simple; a generated test can pass on one correct solution but still reject other correct solutions or fail to catch wrong ones. This is a serious concern if we want to use these benchmarks to judge the reliability of coding agents. TestPrism makes the problem clear and turns it into a measurable evaluation task. The proposed setting checks both accepting valid implementations and rejecting invalid ones.
  • TestPrism contains 300 evaluation tasks from 17 sources, with 3,000 labeled implementations, and also provides 1,700 separate training tasks. It covers six programming languages (I have concern on this, see weakness/questions) and several kinds of coding problems. The authors do more than collect existing tasks, they adapt execution environments, check the original verifiers, build candidate sets, and filter candidates for diversity. I also appreciate that two experts review the task requirements and source verifiers before final selection
Critical weaknesses

I have major concerns and questions about the reliability of benchmark labels and imbalanced language coverage

  • My main concern is the reliability of the labels assigned to candidate implementations. The paper uses source test suites to decide whether a candidate is valid or invalid. The benchmark statistics show almost half of the task has less than 10 tests. The authors check these verifiers and include expert review, which is a good step. However, a source test suite may still miss errors, so a candidate that passes it may not always be correct. This is important because the benchmark treats these labels as ground truth when scoring new tests. I would feel more confident if the authors manually checked a sample of the final candidate implementations and reported how often the labels matched expert judgments.
  • The evaluation set may not fully represent the range of coding tasks that future agents will face. Although TestPrism covers six languages, 205 of the 300 evaluation tasks are in Python, while Java (18), C++(8) and Go(8) tasks each. So the tasks diversity in six languages is overclaimed. The dataset also goes through several filtering steps, and many original tasks are removed because they cannot meet the execution or candidate requirements. These choices are understandable for building a reliable benchmark, but they may favor tasks with easier environments or richer candidate sets.
Questions for authors
  • (Q1) Did the authors independently check the correctness of a random sample of the final 3,000 candidate implementations?
  • (Q2) Since two-third tasks are in Python, Could the authors provide a more detailed breakdown of JSF across different programming languages and task categories?
3 comments
AuthorsOpenReview ↗

Response to reviewer EUH3

We thank the reviewer for raising questions about label reliability and language coverage. We address the candidate-label audit first, followed by the breakdown of JSF by language and source benchmark.

Q1 Did the authors independently check the correctness of a random sample of the final 3,000 candidate implementations?

A1 We assessed label reliability through a blinded expert audit and adversarial testing of the verifiers. We randomly sampled 30 of the 300 evaluation tasks and used the same tasks for both checks. For each task, two experts independently reviewed all ten patches. They examined the behavior of each applied patch against the public task requirements without seeing its original label. The expert judgments matched the original labels for 296 of the 300 implementations, yielding 98.7% agreement. All four discrepancies occurred on performance tasks whose timing results varied across machines. For the second check, we asked GPT-6 Astra Max to generate an implementation for each task that violated a stated requirement while still passing the verifier. The search found one such case in the AlgoTune spectral clustering task listed under the TestPrism ID from-algotune/algotune-spectral-clustering. Whether the implementation met the timing threshold depended on the CPU used for evaluation. In the revision, we will report both findings and document the CPU model and machine configuration originally used to label the affected tasks. We will also expand the manual audit and extend adversarial testing to all 300 evaluation tasks.


Q2 Since two-third tasks are in Python, Could the authors provide a more detailed breakdown of JSF across different programming languages and task categories?

A2 We agree that the language distribution affects how the overall JSF should be interpreted. Python accounts for 205 of the 300 tasks. Kimi K3 achieved 24.88% JSF on Python and 24.21% across the remaining 95 tasks, a difference of 0.67 percentage points. The pooled result conceals variation among the smaller groups. Kimi K3 succeeded on three of the eight C++ tasks and none of the eight Go tasks. Each language contributes 80 candidate implementations, allowing us to evaluate each suite against several solutions to the same task. With eight distinct tasks, however, one additional success changes the language score by 12.5 percentage points. The source breakdown also shows variation within the benchmark. Kimi K3 succeeded on 15 of 28 SWE-bench Live tasks and none of the 21 SWE-Perf tasks. We will include both breakdowns in the revision and clarify that the six languages have uneven representation.

JSF by programming language (%)

LanguageTasksClaude Fable 5.1Kimi K3Qwen3.8 MaxGPT 5.6 sol -maxDeepSeek V4 ProGLM 5.3Mean
Python20528.2924.8823.4123.9013.1711.7120.89
TypeScript4429.5525.0027.2720.4513.6413.6421.59
Java1931.5826.3221.0521.0515.795.2620.18
JavaScript1625.0025.0025.0031.2512.5012.5021.88
C++825.0037.5025.0025.0012.5012.5022.92
Go812.500.0012.5012.5012.500.008.33
Overall30028.0024.6723.6723.3313.3311.3320.72

JSF by source benchmark (%)

Source benchmarkTasksClaude Fable 5.1Kimi K3Qwen3.8 MaxGPT 5.6 sol -maxDeepSeek V4 ProGLM 5.3Mean
Aider Polyglot875.0075.0062.5062.5037.5037.5058.33
AlgoTune250.000.000.000.000.000.008.33
DevEval1225.008.3325.0016.678.338.3315.28
EvoEval2025.0020.0030.0025.0015.0010.0020.83
Long CLI1100.00100.000.00100.000.000.0050.00
QuixBugs1637.5031.2537.5025.0025.0018.7529.17
SWE-bench3336.3624.2427.2730.3015.1515.1524.75
SWE-bench Live2850.0053.5735.7139.2921.4317.8636.31
SWE-bench Multimodal1127.2718.189.0918.189.099.0915.15
SWE-Perf214.760.004.764.760.000.002.38
SWE-rebench4632.6134.7828.2630.4317.3913.0426.09
SWE-bench Pro175.880.005.880.000.000.001.96
SWE-PolyBench4825.0029.1725.0022.9214.5812.5021.53
Terminal-Bench812.500.0012.5012.500.0012.508.33
Terminal-Bench 2425.0050.0025.0025.0025.000.0025.00
Terminal-Bench Pro520.000.000.0020.000.000.006.67
USACO205.000.0010.005.005.005.005.00
Overall30028.0024.6723.6723.3313.3311.3320.72

Thank you for the detailed response and additional analysis. The expert audit showing 98.7% label agreement largely addresses my concern about label reliability. The language-level and source-level results also help clarify my concerns about dataset diversity, although the language imbalance remains a limitation. Overall, I appreciate the authors' efforts to address my questions. Given the additional evidence and the value of the benchmark, I have updated my rating.

AuthorsOpenReview ↗

Response to reviewer EUH3

Thank you for taking the time to consider the additional evidence and update your rating. We’re glad the expert audit helped address your concern about label reliability. We agree that the uneven language coverage remains a limitation and will make this explicit when we report the results by language and benchmark source in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.