acceptodds
Under review as a conference paper at ICLR 2027

UltrafastBench: A Deterministic Benchmark for Predictive Experimental Reasoning

Abstract

We introduce UltrafastBench to test final-answer prediction, diagnosis, and constrained control in specified ultrafast-optics systems. Its 400 questions span 24 physical-model families, including 240 design tasks. Executable references and typed answer fields support deterministic scoring without an LLM judge. On 380 unchanged screened prompts, five models answer 88–284 completely. On a frozen 400-question fresh-parameter cohort, GPT-6 Luna and GPT-6 Sol answer 115 and 303 without tools; isolated Python access raises these scores to 339 and 399, rescuing 233/285 and 97/97 prior misses. In four category-plus-estimate families, Sol chooses all 80 categories but completes only 49 answers closed-book, rising to 80 with Python. Saved traces show both numerical rescues and tool-induced errors. The results establish sensitivity to this Python-enabled protocol on new parameter worlds within the developed families. Versioned questions, references, and traces permit field-level replay.

Primary area:
datasets and benchmarks
Keywords:
optics, ultrafast optics, benchmark, expert benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5AkyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UltrafastBench, an executable short-answer benchmark for ultrafast-optics prediction, inverse diagnosis, and constrained control. Its central claim is that replayable references, typed answer contracts, and field-level scoring can reveal the gap between a model’s “correct decision” and “complete answer,” and thereby inform discussions of scientific benchmark validity.

Critical strengths
  • The paper fills a genuinely existing evaluation slice: existing benchmarks such as SymPyBench, PRISM-Physics, and SciCode respectively emphasize parameterized physics problems, derivation evaluation, or program/repository work, whereas this paper focuses on short-answer tasks in ultrafast optics that require first identifying system parameters and then selecting control settings, supported by executable references and typed fields. This is of practical value to the scientific-reasoning evaluation community.

  • The paper separates three checks, whether the reference calculation can be replayed, whether the wording uniquely determines the reference answer, and whether the scorer accepts equivalent answers and openly audits the process using cases such as the BBO angle convention, the bare-JSON and FINAL-wrapper defects in 38 comb/CEP prompts, budget exhaustion, and unparsed responses. Such benchmark transparency is uncommon in comparable papers.

Critical weaknesses
  • The core extrapolation claim is structurally undermined by the experimental design. Although the 400 items are allocated 50 each across eight families, numeric normalization leaves only 68 prompt patterns; 19/24 families have a single pattern, 23/24 families have a single task label, and two pairs of families share core equations. Fresh v2 items also only vary parameters within already developed families. Task type is therefore heavily confounded with family, making it impossible to isolate general differences among prediction, inference, and control, or to support conclusions beyond parameter transfer within these families. Although the Limitations section acknowledges this, the Introduction and Conclusion still describe the contribution as covering this slice, creating a clear tension between claims and evidence. This is a structural flaw that additional experiments can mitigate but not fully resolve.

  • The evidence for benchmark validity is not closed. v3 changed 360 question IDs to clarify wording or answer contracts, with only 40 items retaining v2 wording, yet all fresh accuracy results remain based on v2. The four GPT-6 Pro packets flag 48 prompts, 3 keys, and 390 scorer contracts, all as unadjudicated labels, and do not audit saved v2 model responses. For a paper whose core contribution is benchmark validity, this leaves readers unable to determine whether the revised scores and conclusions are stable.

  • The five-model comparison is not a fair capability comparison and should not be presented as a ranking. Requested output caps are 32K for Luna/Sol, 64K for DeepSeek, and 128K for Spark; DeepSeek has 70 no-answer traces on the common subset, all exhausting 65,536 tokens, and Gemini reports output above its requested cap on 198/380 traces, 150 of which are correct, while the 32K cap for Luna/Sol is clearly tighter. Item screening was also influenced by prior model responses: 49 earlier items were replaced, and 20 retained items were selected after a Gemini miss. Although the authors state that this is not a general model ranking, Table 5 is still presented as the most prominent ranking table, and the one-item difference between Sol’s 284 and Spark’s 283 is described as having no ranking implication. Rerunning with unified budgets is costly, but at minimum a budget sensitivity analysis should be conducted and Table 5 should be downgraded to descriptive observations.

Questions for authors

see the weaknesses.

Reviewer_MdgQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces UltrafastBench, a benchmark of 400 ultrafast-optics questions spanning 24 physical-model families. The benchmark covers three tasks for a specified optical system: predicting a measurement, diagnosing a hidden setting, and choosing a feasible control under constraints. Each family defines its equations and validity regime, samples a parameter world, computes the reference answer with executable code before any model call, and renders a question with typed answer fields. Design questions use finite candidate sets, feasibility constraints, and tie rules so that references can be enumerated. Scoring is therefore deterministic and does not require an LLM judge. Question quality is checked through reference replay, independent numerical cross-checks for some families, scorer round trips, and model-assisted wording audits. On 380 screened prompts, five models answer between 88 and 284 questions fully correctly without tools. On a fresh cohort of 400 questions with new parameters, GPT-6 Luna and GPT-6 Sol answer 115 and 303 questions closed-book, rising to 339 and 399 with Python access. A field-level breakdown shows that Sol selects the correct physical category in all 80 prediction items but completes only 49 closed-book, suggesting that many errors are numerical rather than conceptual.

Critical strengths

Every reference answer is computed by executable code before any model call, and design questions are given finite candidate sets and tie rules so that the reference is unique. This allows grading without an LLM judge. Because questions are produced by parametric generators, new parameter worlds can be sampled to reduce contamination, as the fresh 400-item cohort demonstrates.

The field-level analysis goes beyond a single accuracy number. Separating correct physical choices from complete answers shows, for example, that Sol selects all 80 prediction categories correctly but completes only 49 closed-book, and that in two-stage SPM, identifying the model and choosing the controls fail at different rates. The paired closed-book and Python runs, together with the saved traces, show both numerical rescues and errors introduced by tool use.

Critical weaknesses

The main concern is the validity of the questions themselves. The deterministic scorer removes the LLM judge from grading, but checking whether each question’s wording actually determines its reference answer relies largely on model-assisted review. The wording and answer-contract audits were performed with GPT-6 Pro, their labels are described as unadjudicated leads, and the human two-reviewer pass keeps only consolidated verdicts without per-item records. Since GPT-6 Pro belongs to the same model family as two of the evaluated models, this may also bias which questions were flagged and revised. The defects that were confirmed, such as the undefined Δk convention in the BBO prompts and the parser conflict on 38 prompts, show that replaying the code alone does not guarantee a well-posed question. Moreover, the later v3 revision changed the wording or answer contracts of 360 of the 400 questions, yet all reported results are on v2.

The benchmark measures less than its title suggests. With Python access, Sol answers 399 of 400 fresh questions and Luna rises from 115 to 339. Sol’s closed-book errors are also mostly numerical, since it selects every prediction category correctly. Under a tool-enabled protocol, the benchmark therefore leaves almost no headroom and mainly tests numerical computation rather than experimental reasoning. Its diversity is also limited: after numeric normalization there are only 68 prompt patterns, 19 of the 24 families have a single pattern, and task type is largely confounded with family. The 400 questions are thus closer to parameter variations of a few dozen problems, and the paper does not argue why ultrafast optics is representative of experimental reasoning more broadly.

The cross-model comparison is difficult to interpret. The screened set was shaped by prior model responses: 49 earlier items were replaced, and 20 were retained after a Gemini miss. The runs also use different output budgets (32K to 128K tokens) and only one response per item. There is no human-expert or non-LLM baseline to calibrate difficulty, and the analysis of failure types is limited to a few individual traces. Meanwhile, a considerable portion of the main text is devoted to run-level provenance, such as request IDs, file names, and SHA-256 checksums, which would fit better in an appendix.

Questions for authors

Since v3 revises the wording or answer contracts of 360 of the 400 questions, could you report results on v3, or at least on the 40 questions whose wording is unchanged? Were any of the GPT-6 Pro flags adjudicated by human experts, and if so, how many were confirmed as real defects?

Given that Sol reaches 399/400 with Python, how do the authors see the benchmark remaining useful for tool-enabled models? Are there plans for held-out mechanisms or harder design variants, and could a human-expert baseline be provided to calibrate difficulty?

Could all five models be evaluated on the fresh cohort, whose worlds were not selected using model responses? This would show whether the screened-set ranking is affected by the replacement of 49 items and the retention of 20 items after a Gemini miss.

Reviewer_xTHLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UltrafastBench has 400 ultrafast-optics questions from 24 physical-model families: 90 prediction, 70 diagnosis and 240 design items. References are computed by code from sampled parameter worlds before any model call, and answers are scored deterministically against typed fields without an LLM judge. Five models are evaluated closed-book on 380 screened prompts (strict accuracy 88 to 284). GPT-6 Luna and GPT-6 Sol are also evaluated on a frozen 400-item fresh cohort, closed-book and with an isolated Python tool. Luna rises from 115 to 339 and Sol from 303 to 399. Field-level tables separate choices from numerical estimates. Scores are stated to measure agreement with declared references, not physical validation.

Critical strengths
  1. Judge-free scoring with paired, field-level accounting. On frozen items, the closed-book versus Python-enabled comparison gives 233 rescues against 9 regressions for Luna and 97 against 1 for Sol (Table 2, p.4 L200-204). That is far above noise, although it measures the whole protocol change, tool plus wrapper (p.3 L155; p.7 L336-337). Tables 3-4 separate correct decisions from complete answers, which is what shows where models fail. Every number I could recompute is consistent. This matters because results reported at field level can be diagnosed, not just aggregated.

  2. Candid validity accounting in a new domain slice. The paper:

    • separates reference, wording and scorer checks (Fig. 1);
    • excludes 20 BBO prompts with an undefined convention (p.2 L093-097);
    • reports a scorer defect both ways (Table 5);
    • describes a hashed archive with a generator and a replay script (p.4 L182-192), although that archive is not attached to the submission (Weakness 1).

    I located no prior LLM benchmark on ultrafast-optics prediction, diagnosis and control, nor items coupling identification to a control choice (Fig. 2; Table 4). This matters because it gives an auditable start in an uncovered domain.

Critical weaknesses
  1. The evaluated version's validity is not established, and the benchmark is neither presented nor attached.
    • All fresh-cohort results (Tables 2-4) are on v2. A later v3 changed 360 of 400 question IDs "to clarify wording or answer contracts", and no model run on v3 is reported, although the paper says the generator in its supplementary archive applies those corrections (p.3 L114-120, L135; p.4 L186).
    • v3 kept all worlds and references, and Sol with Python matches 399 of 400 references under the v2 wording (Table 2). So the items are computable as worded; uniqueness and answer equivalence are not established.
    • Validation is thin:
      • solver cross-checks are named for three families (p.2 L080-082);
      • the two-reviewer return has no votes, rationales or stated expertise (p.2 L103-105);
      • uncertain keys stay in the denominators (p.3 L118-119; p.4 L176);
      • the trace audit covers only Luna's closed-book unparsed answers (p.3 L108-112).
    • The paper itself omits the benchmark. The 24 families are never listed, no complete prompt with its contract appears (Fig. 2 is abridged, p.6 L279), and tolerances appear in one example only (p.5 L220). The main text ends at about 6.6 of 9 allowed pages, with no appendix (p.7 L353).
    • The paper places the versioned datasets, saved requests and answers, the scorer and generator source, and a hash-verified replay in an anonymous supplementary archive (p.3 L133-135; p.4 L182-192), but no supplementary material is attached to the submission, so none of it can be inspected or replayed.
    • For a benchmark paper this is decisive.
  2. Little headroom for the strongest model, and the scores do not isolate the titled construct.
    • With Python, Sol answers 399 of 400 items, and all 97 of its closed-book misses are rescued (Table 2; p.4 L209).
    • Closed-book, Sol picks 80 of 80 prediction categories but completes 49 (Table 3). It selects all two-zone controls on 17 of 20 items but completes 7 (Table 4). The paper calls this "a substantial numerical burden" (p.4 L211-213).
    • Categories are a weak signal: a post hoc majority choice gets 58 of 80, above Luna's 56 (p.5 L232).
    • Questions state the model and usually a finite candidate set (p.1 L029; p.2 L066-067), so strong-model completion largely tracks unaided numerical execution. The paper concedes it cannot separate optical reasoning from calculation (p.5 L253-254). Luna's 22 extra categories with Python (p.4 L212-214) show this is not the whole story for a weaker model.
    • The items reduce to 68 prompt patterns (p.2 L072-075), and regenerated cohorts stay in the same families (p.4 L186; p.7 L330-332).
    • The tool effect is the uncited program-aided reasoning effect (PAL, ICML 2023; Program of Thoughts, TMLR 2023; ToRA, ICLR 2024).
    • This matters because a benchmark earns its place by separating future models.
Questions for authors

Clarification requests (answers could change my assessment):

  1. v3 and eigenpulse. Can you report Luna and Sol on v3, or v2 scores on the 40 unchanged items, and classify the 360 edits as editorial, contract or meaning-changing? In the eigenpulse families, 21 of Luna's 31 residual answers match a post-gate rather than the specified pre-gate variance (p.4 L214-p.5 L217). Do Sol's traces there, which missed at most one item overall, use pre-gate?
  2. Primary protocol and headroom. Which protocol is the primary track, and what headroom remains beyond numerical execution? Decision-level results for all families, a harder or held-out split, or an expert-adjudicated stratified sample would address Weakness 2.
  3. Python loop and sampling. How was the Python loop implemented? Table 1 lists it under the messages wrapper at high reasoning, while OpenAI's model pages say Chat Completions supports function calling only with reasoning_effort none (https://developers.openai.com/api/docs/models/gpt-6-sol). Was it a text protocol, the Responses API, or a gateway? How often was Python used, and how often were caps hit? Which route and sampling settings served the rows requested as go-ds-flash, gemini-main and muse-spark-main (Table 1)?
  4. Rescues by failure type. Can you cross-tabulate rescues by closed-book failure type and add known-format scores to Table 2? 27 of Sol's 97 misses were unparsed and unaudited (p.4 L206-207). On the screened set, all 14 of Sol's known-format recoveries came from its 25 unparsed answers (Table 5; Fig. 3).
  5. Tolerances. What is the tolerance policy? The one shown (±0.015, p.5 L220) is near Sol's 0.017 median error in that family (Table 3). A replay at half and double tolerance would show the sensitivity.

Suggestions:

  • Run one non-OpenAI, ideally open-weight, model on the fresh cohort. Only GPT-6 models were run there, and the same vendor's tools assisted construction and audit (Table 1; p.7 L356-360).
  • Attach the supplementary archive the paper describes (p.4 L182-192), and add an appendix with families, complete prompts, contracts and tolerances.
  • Report confidence intervals: n = 20 per family, and the SPM 11 vs 6 gap rests on five discordant items (Table 4).
  • Position against NewtonBench (ICLR 2026), PHYBench (NeurIPS 2025) and PHYSICS (Findings of ACL 2025, which includes optics).
  • Do not headline the screened 88 to 284 range, which Section 4 says is not a ranking (p.6 L283-285).
  • State the release licence.

Rating rationale.

Not a clear rejection: no claim is false relative to the paper's caveats, the paired accounting is clean, and the domain slice is new. Not an acceptance:

  • fresh-cohort results are on a wording later revised in 90 percent of items, with no expert validation;
  • the benchmark is not in the paper, and the archive the paper relies on is not attached;
  • the strongest model is at 399 of 400 with tools.

v3 results, the archive, and a family, contract and tolerance appendix would resolve Weakness 1. Weak acceptance would also need evidence of headroom beyond numerical execution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.