acceptodds
Withdrawn from ICLR 2027

Wrong Target or Failed Delivery? Diagnosing Scientific Coding Beyond Pass/Fail

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Strict scientific coding scores combine numerical computation, report construction, and final execution. A completion gap therefore need not represent an equally large difference in numerical content or program behavior. We examine this distinction through complete-reference matching, saved-content checks, and tests of frozen final programs linked to original execution records. In 43 historical starting-program pairs, strict completion differs by 65.1 percentage points, whereas the requested-vector match gap is bounded by 0–14.0 points. With reading rules fixed before model calls, GPT-5.4-mini and Qwen3.7 Plus show completion gaps of 66.7 and 75.0 percentage points, respectively, despite zero saved-content gaps. Haiku's completion gap exceeds its saved-content gap by at least 33.3 points; Gemini completes both starting conditions. A factorial design then varies computation and delivery support separately. Across six model configurations, the records distinguish correct numerical returns in invalid reports, numerical or return-type failures, and cases lacking qualifying final-source execution evidence. The prespecified Mini–Haiku replication yields an observed delivery-only versus computation-only completion difference of +20.8 points (). Joint evidence shows which completion differences accompany changes in tested computation and which coexist with matching numerical content and unfinished reporting or execution obligations.

Primary area:
datasets and benchmarks
Keywords:
large language models, scientific coding, coding agents, diagnostic evaluation, program testing, execution provenance

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_c1bhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents an empirical analysis of why scientific coding agents fail under strict pass/fail scoring. The authors argue that a single failed score can hide very different situations: the agent may compute a wrong quantity, compute the right numbers but save them in an invalid report structure, or write a test-passing program without a qualifying final execution record. To separate these cases, they combine three kinds of evidence: matching saved numerical content against references, running the frozen final program on five fixed test inputs, and checking the original execution records. Using small task collections and six model configurations, they report two main observations: (1) a correct starting program can greatly increase strict completion without a comparable increase in correct saved content, and (2) when computation support and delivery support are supplied separately, completion differences come from different unfinished requirements. However, the current version of this paper has serious problems in organization and writing. For example, the abstract and introduction do not explain the background, motivation, or what the study actually does. Key terms are used without definition, and there is no Background or Related Work section. The claimed contributions and findings are vague. Together with limited novelty, small-scale experiments on toy tasks, and weak statistical support, the paper is far from the standard of an ICLR paper.

Critical strengths
  • The paper focuses on a frontier and practically relevant topic: evaluating scientific coding agents beyond a single pass/fail score.

  • The figures are visually well-designed.

Critical weaknesses
  • Severe problems in writing and organization. The abstract and introduction give no background or motivation, key terms are used without definition, the paper has no Background or Related Work section. In addition, the two claimed contributions are vague observations. After reading the paper, it is still hard to tell what the study does, on which data, and what its findings mean.

  • Limited novelty with weak empirical support. The main findings (strict scoring mixes formatting failures with computation failures, and a correct starting program raises completion) have already been discussed in format-bias work [1-3] or are expected by design. The evidence is also weak: twelve simple tasks per collection, one session per cell, ceiling effects for strong models, a prespecified replication that is not statistically significant, selected historical subsets, and a post-hoc reading rule. Even with better writing, the paper would not provide enough new and reliable insight for ICLR.

[1] LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs. NAACL, 2025.

[2] Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. EMNLP, 2024.

[3] FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability. ACL, 2024.

Questions for authors
  1. What is the main contribution of the paper: a new evaluation protocol or metric, a benchmark, or an empirical study? What exactly can a benchmark designer take from the paper and use directly?
  2. Why was the saved-content reading rule fixed only after inspecting an undetermined output, and do the conclusions still hold under a rule fixed in advance?
  3. Please strengthen the statistical analysis and explain the data selection. (1) The abstract highlights the prespecified Mini–Haiku replication result of +20.8 points (p = 0.125) (Line 24), but the paper also states that the test does not reject at 0.05(Line 421). A non-significant result should not be presented as a main finding. (2) The historical comparison retains only 43 of 144 planned pairs (Line 234), and Table 1 notes that Historical pairs are a selected DeepSeek/GLM subset(Line 166). In the three-target study, 108 of 288 planned positions were never started (Line 214). (3) The historical support study spans protocol amendments, and the paper admits that Missing primary endpoints and protocol amendments leave the original common-protocol confirmation unresolved(Line 373). These issues make headline numbers such as the 65.1-point completion gap and the at least 51.2 points separation sensitive to selection. Please explain the selection criteria, report results on the full planned sets, and either increase the sample size for adequate statistical power or tone down the claims.
  4. What is the practical value of the work? The main message is that pass/fail scores hide different failure types, but the main text of this paper does not provide an actionable metric or evaluation protocol. As a result, benchmark designers do not know how to adopt the findings: which fields should be logged, how to compute the proposed evidence layers on an existing benchmark, and at what cost?
Reviewer_DoQpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows that pass/fail evaluation of scientific coding agents conflates incorrect computations, invalid reports, and incomplete execution. It proposes a layered diagnostic framework combining numerical reference matching, saved-content checks, frozen-program tests, and execution provenance. Across 43 historical paired comparisons, the strict completion gap between correct and neutral starting programs reaches 65.1 percentage points, while the numerical-content gap is only 0–14.0 points, with similar patterns in model replications. A factorial study across six model configurations further separates computation failures from delivery failures, showing that numerical correctness, report validity, and execution evidence should be reported alongside overall task success.

Critical strengths
  • Novel and important problem. The paper combines saved numerical content, frozen-program tests, and execution records to distinguish computational errors, reporting failures, and missing execution evidence. This provides a more informative account of agent failures than pass/fail scores alone. The finding that large completion gaps can coexist with small or zero numerical-content gaps offers concrete evidence that overall success rates should not be interpreted directly as measures of computational ability.
Critical weaknesses
  • Poor writing and organization. The paper is poorly organized and difficult to follow. The core distinction between computational correctness and delivery success is unclear; substantial restructuring is needed to clarify the two main comparisons and connect their results to the broader conclusions.
  • Sensitivity to the delivery contract is insufficiently established. The reported completion gaps depend on requirements such as exact JSON structure and a formal execution receipt linked to the final source, and violating these can produce strict failure even when the numerical values are correct or the program passes subsequent tests. A controlled comparison of alternative reporting and execution requirements, holding the underlying computation fixed, is needed to separate broadly relevant agent limitations from sensitivity to the particular evaluation interface.
Questions for authors
  1. Would the diagnosis of delivery failures remain unchanged if the frozen programs were evaluated on a broader set of unseen inputs and edge cases beyond the five fixed tests?
  2. Since using the supplied support is optional, how often do agents reuse, modify, or ignore the provided computation and delivery implementations, and how do these behaviors relate to the observed completion differences?
Reviewer_MmYYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of pass/fail evaluation in scientific coding tasks by distinguishing numerical correctness from report formatting and execution requirements. The authors propose a multi-layer evaluation framework combining saved numerical content, program tests, and execution records. Experiments across multiple LLMs show that large differences in task completion rates do not necessarily reflect differences in numerical correctness. The paper further examines the effects of computation and delivery support, revealing distinct failure modes related to incorrect calculations, invalid reports, and incomplete execution.

Critical strengths
  1. The paper addresses a practical limitation of pass/fail evaluation by distinguishing numerical errors from failures in report construction and execution. This provides a more detailed understanding of agent performance.

  2. The proposed evaluation combines numerical-content matching, program testing, and execution records to identify different failure modes, providing more informative diagnostics than a single success metric.

Critical weaknesses
  1. Previous studies have already explored the distinction between content correctness, format compliance, and task success. This paper mainly combines existing evaluation perspectives in scientific coding scenarios, offering limited conceptual novelty.

  2. The main factorial experiment does not establish a statistically significant advantage of delivery support over computation support (p = 0.125). Several models also exhibit ceiling effects, limiting the strength of the empirical conclusions.

  3. The experiments rely on a small number of controlled scientific coding tasks, with the same task roots repeatedly used across different conditions. It remains unclear whether the observed failure patterns generalize to more complex real-world scientific workflows.

  4. Some historical analyses rely on selected subsets of incomplete experimental records, and execution protocols changed during data collection.

Questions for authors
  1. I would like the authors to clarify the main contributions of this work. Previous studies [1,2] have already distinguished content correctness from format compliance and task performance. What new insights does this paper provide beyond combining these existing evaluation perspectives in scientific coding tasks?

  2. The experiments mainly use a small set of controlled scientific coding tasks. How well do the identified failure patterns generalize to more complex real-world scientific workflows involving multiple computation steps, dependencies, and intermediate artifacts?

  3. The paper distinguishes several failure modes, but does not evaluate whether these diagnoses improve subsequent task completion. Could the authors investigate whether providing agents with failure-specific feedback leads to more effective correction than generic pass/fail feedback?

References

[1] Xia, C., et al. FoFo: A Benchmark to Evaluate LLMs' Format-Following Capability. ACL 2024. https://aclanthology.org/2024.acl-long.40/

[2] Long, D. X., et al. LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs. NAACL 2025. https://aclanthology.org/2025.naacl-long.15/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.