- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical analysis of why scientific coding agents fail under strict pass/fail scoring. The authors argue that a single failed score can hide very different situations: the agent may compute a wrong quantity, compute the right numbers but save them in an invalid report structure, or write a test-passing program without a qualifying final execution record. To separate these cases, they combine three kinds of evidence: matching saved numerical content against references, running the frozen final program on five fixed test inputs, and checking the original execution records. Using small task collections and six model configurations, they report two main observations: (1) a correct starting program can greatly increase strict completion without a comparable increase in correct saved content, and (2) when computation support and delivery support are supplied separately, completion differences come from different unfinished requirements. However, the current version of this paper has serious problems in organization and writing. For example, the abstract and introduction do not explain the background, motivation, or what the study actually does. Key terms are used without definition, and there is no Background or Related Work section. The claimed contributions and findings are vague. Together with limited novelty, small-scale experiments on toy tasks, and weak statistical support, the paper is far from the standard of an ICLR paper.
Critical strengths
-
The paper focuses on a frontier and practically relevant topic: evaluating scientific coding agents beyond a single pass/fail score.
-
The figures are visually well-designed.
Critical weaknesses
-
Severe problems in writing and organization. The abstract and introduction give no background or motivation, key terms are used without definition, the paper has no Background or Related Work section. In addition, the two claimed contributions are vague observations. After reading the paper, it is still hard to tell what the study does, on which data, and what its findings mean.
-
Limited novelty with weak empirical support. The main findings (strict scoring mixes formatting failures with computation failures, and a correct starting program raises completion) have already been discussed in format-bias work [1-3] or are expected by design. The evidence is also weak: twelve simple tasks per collection, one session per cell, ceiling effects for strong models, a prespecified replication that is not statistically significant, selected historical subsets, and a post-hoc reading rule. Even with better writing, the paper would not provide enough new and reliable insight for ICLR.
[1] LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs. NAACL, 2025.
[2] Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. EMNLP, 2024.
[3] FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability. ACL, 2024.
Questions for authors
- What is the main contribution of the paper: a new evaluation protocol or metric, a benchmark, or an empirical study? What exactly can a benchmark designer take from the paper and use directly?
- Why was the saved-content reading rule fixed only after inspecting an undetermined output, and do the conclusions still hold under a rule fixed in advance?
- Please strengthen the statistical analysis and explain the data selection.
(1) The abstract highlights the prespecified Mini–Haiku replication result of +20.8 points (p = 0.125) (Line 24), but the paper also states that the test
does not reject at 0.05(Line 421). A non-significant result should not be presented as a main finding. (2) The historical comparison retains only 43 of 144 planned pairs (Line 234), and Table 1 notes thatHistorical pairs are a selected DeepSeek/GLM subset(Line 166). In the three-target study, 108 of 288 planned positions were never started (Line 214). (3) The historical support study spans protocol amendments, and the paper admits thatMissing primary endpoints and protocol amendments leave the original common-protocol confirmation unresolved(Line 373). These issues make headline numbers such as the 65.1-point completion gap and theat least 51.2 pointsseparation sensitive to selection. Please explain the selection criteria, report results on the full planned sets, and either increase the sample size for adequate statistical power or tone down the claims. - What is the practical value of the work? The main message is that pass/fail scores hide different failure types, but the main text of this paper does not provide an actionable metric or evaluation protocol. As a result, benchmark designers do not know how to adopt the findings: which fields should be logged, how to compute the proposed evidence layers on an existing benchmark, and at what cost?