- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UltrafastBench, an executable short-answer benchmark for ultrafast-optics prediction, inverse diagnosis, and constrained control. Its central claim is that replayable references, typed answer contracts, and field-level scoring can reveal the gap between a model’s “correct decision” and “complete answer,” and thereby inform discussions of scientific benchmark validity.
Critical strengths
-
The paper fills a genuinely existing evaluation slice: existing benchmarks such as SymPyBench, PRISM-Physics, and SciCode respectively emphasize parameterized physics problems, derivation evaluation, or program/repository work, whereas this paper focuses on short-answer tasks in ultrafast optics that require first identifying system parameters and then selecting control settings, supported by executable references and typed fields. This is of practical value to the scientific-reasoning evaluation community.
-
The paper separates three checks, whether the reference calculation can be replayed, whether the wording uniquely determines the reference answer, and whether the scorer accepts equivalent answers and openly audits the process using cases such as the BBO angle convention, the bare-JSON and FINAL-wrapper defects in 38 comb/CEP prompts, budget exhaustion, and unparsed responses. Such benchmark transparency is uncommon in comparable papers.
Critical weaknesses
-
The core extrapolation claim is structurally undermined by the experimental design. Although the 400 items are allocated 50 each across eight families, numeric normalization leaves only 68 prompt patterns; 19/24 families have a single pattern, 23/24 families have a single task label, and two pairs of families share core equations. Fresh v2 items also only vary parameters within already developed families. Task type is therefore heavily confounded with family, making it impossible to isolate general differences among prediction, inference, and control, or to support conclusions beyond parameter transfer within these families. Although the Limitations section acknowledges this, the Introduction and Conclusion still describe the contribution as covering this slice, creating a clear tension between claims and evidence. This is a structural flaw that additional experiments can mitigate but not fully resolve.
-
The evidence for benchmark validity is not closed. v3 changed 360 question IDs to clarify wording or answer contracts, with only 40 items retaining v2 wording, yet all fresh accuracy results remain based on v2. The four GPT-6 Pro packets flag 48 prompts, 3 keys, and 390 scorer contracts, all as unadjudicated labels, and do not audit saved v2 model responses. For a paper whose core contribution is benchmark validity, this leaves readers unable to determine whether the revised scores and conclusions are stable.
-
The five-model comparison is not a fair capability comparison and should not be presented as a ranking. Requested output caps are 32K for Luna/Sol, 64K for DeepSeek, and 128K for Spark; DeepSeek has 70 no-answer traces on the common subset, all exhausting 65,536 tokens, and Gemini reports output above its requested cap on 198/380 traces, 150 of which are correct, while the 32K cap for Luna/Sol is clearly tighter. Item screening was also influenced by prior model responses: 49 earlier items were replaced, and 20 retained items were selected after a Gemini miss. Although the authors state that this is not a general model ranking, Table 5 is still presented as the most prominent ranking table, and the one-item difference between Sol’s 284 and Spark’s 283 is described as having no ranking implication. Rerunning with unified budgets is costly, but at minimum a budget sensitivity analysis should be conducted and Table 5 should be downgraded to descriptive observations.
Questions for authors
see the weaknesses.