acceptodds
Under review as a conference paper at ICLR 2027

FrontierChallenge: Evaluating Scientific Workflow Completion

Abstract

Scientific agents increasingly analyze data, execute code, and produce research artifacts, yet most benchmarks emphasize final answers, isolated programs, or a single domain. We introduce FrontierChallenge, a cross-domain benchmark comprising 300 end-to-end scientific workflows. In this paper, we release and evaluate 97 of these tasks, spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment. Each task provides fixed inputs and specifies a bundle of required scientific deliverables. We evaluate twelve frontier models with three agent scaffolds. Pass Rate measures the fraction of tasks satisfying the full-completion criterion, while Avg. Score captures partial progress. Each of the best-performing configurations completed only 20 of the 97 released tasks, yielding a Pass Rate of 20.6%. Partial progress translated especially poorly into complete delivery in analytical chemistry and electrochemistry/environment: Avg. Scores reached 87.6 and 94.9, but the highest Pass Rates were only 4% and 0%. Among non-passing Claude Code trajectories, 75.5% still ended with language claiming completion. Complementary HDS6 process scores correlate strongly with task outcomes, supporting FrontierChallenge as a benchmark of Heavy Duty Solver capabilities. These findings show that neither high partial scores nor confident claims of completion reliably indicate that a scientific task has been fully delivered, highlighting the need to evaluate end-to-end workflow execution and the completeness of scientific deliverables together.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Science Agent, Scientific Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r6gxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces FrontierChallenge, a benchmark that evaluates whether agents can complete scientific workflows end to end and deliver complete, mutually consistent artifact packages (code, tables, figures, and reports). The authors describe 300 workflows in total, of which 97 are released and evaluated, covering six domains: quantum chemistry, molecular dynamics, materials, analytical chemistry, life sciences, and electrochemistry. Each task has a task-specific grader, semantic rubric items are scored by GPT-5.6 Sol as a judge, and a task counts as passed only at a score of at least 99.9. Across 13 model–scaffold configurations, average scores range from 67.5 to 87.9, while the best pass rate is only 20.6%. The paper also reports HDS6 process scoring and an analysis of failure patterns.

Critical strengths
  1. The topic is valuable. Most existing benchmarks for scientific agents check only final answers or single programs. This paper targets the ability to deliver complete artifacts according to a contract, which is closer to real research and fills a clear gap.
  2. The task packages are of high quality. All 97 released tasks come with detailed instructions, fixed inputs, and environment configurations, covering six domains and 21 workflow families. This is a substantial construction effort with direct reuse value for the community.
  3. The core observation is insightful. The gap between high partial scores and low rates of complete delivery clearly exposes a weakness of current agents on long workflows.
  4. The evaluation is large and multi-faceted: 13 configurations, 3 scaffolds, and analyses of process scores, failure patterns, tool errors, and resource use. I recomputed the reported numbers, and they are internally consistent.
Critical weaknesses
  1. There is no evidence that the tasks are solvable at the 99.9 threshold, and the threshold itself is not justified. The paper reports neither the scores of reference solutions under the graders nor a human baseline. In electrochemistry, all 13 configurations pass zero tasks even though the best average score in that domain is 94.9. In addition, task_008 in the supplementary material tells the agent that a total score of at least 80 counts as a pass, which is inconsistent with the 99.9 criterion in the paper. I suggest reporting reference-solution scores, adding a sensitivity analysis at thresholds such as ≥95 and ≥90, and aligning the pass criteria that tasks show to agents with the paper.
  2. There is a risk of answer leakage, and the experimental configuration is unclear. In the supplementary material, all 97 tasks have network access set to public. In task_006, the authors themselves found that the source data overlapped about 94% with the graded answer and removed it, yet the agent-visible configuration file still contains the download URL of that data. I suggest rerunning with network access disabled, or auditing the agents' web access.
  3. The statistical support is weak, and there are several confounds. Only one trajectory is kept per configuration–task pair, and no confidence intervals are given (the 95% interval for 20/97 is 13.8%–29.7%). Domain differences are strongly confounded with difficulty (analytical chemistry and electrochemistry are almost entirely Hard tasks), with whether an LLM judge is used, and with whether the data are synthetic (52 of the 97 tasks use synthetic data, a proportion the paper does not disclose). I suggest running at least 3 trials with intervals and reporting results stratified by difficulty, judge vs. no judge, and real vs. synthetic data.

Minor:

  • The validity of the graders would benefit from additional evidence. There is no expert–grader agreement data, and the judge model, GPT-5.6 Sol, is also an evaluated model. I suggest a small expert agreement study, or cross-judging with a model that is not under evaluation, and releasing the rubrics.
  • The "300 workflows" in the abstract is not supported, since only 97 tasks are described and evaluated.
  • The heatmap cells in Figures 3 and 4(a) are low-resolution raster images, and Figure 4(a) has no value labels.
  • A label on the right side of Figure 1 is cropped.
  • Figure 7 is labeled as "unmodified" input, but it is a synthetic image.
Questions for authors
  1. What score does each task's reference solution receive under the grader? How many reach 99.9?
  2. How are the three judge calls aggregated, and what is the score variance when the same submission is re-evaluated?
  3. How many runs were actually performed per configuration–task pair, and by what criterion was the "valid" trajectory selected?
  4. The paper says 77 tasks use a judge, while only 71 tasks in the supplementary material configure one. Where does the difference come from? And what is the distribution of the 203 held-out tasks?
4 comments

Thank you for the thorough and constructive review. Below are our point-by-point responses.


Weaknesses

W1. Task Solvability at >=99.9, Threshold Rationale & Sensitivity, task_008 Alignment, and Zero Passes in Electrochemistry

1. Task Solvability and Reference Solutions

All 97 released tasks have been verified as solvable:

  • 87/97 tasks pass (>=99.9) directly using the expert-provided reference solutions.
  • 5/97 tasks (009, 010, 094, 099, 104) passed all core scientific, schema, and figure checks. Points were initially deducted solely because analysis.py, added during final specification normalization, was missing from the reference packages. We have now synced analysis.py into these 5 packages so they also reach >=99.9.
  • 5/97 remaining tasks have been verified as solvable by domain experts and achieve >=99.9 with minor adjustments to solve.py and the verification environment (e.g., ORCA/fixture dependencies or supplementary outputs).

2. Why >=99.9 Is the Pass Threshold

We set task_score >= 99.9 as the pass bar to separate high partial credit from complete, reliable workflow delivery:

  1. Full rubric satisfaction: We use >=99.9 as the operational criterion for full-contract completion. Each Judge semantic item is scored three times and averaged (with a 0.1-point margin below 100 for minor rounding/averaging variance), and we pair this strict bar with re-evaluation stability and multi-threshold sensitivity analyses below.
  2. High partial scores do not guarantee valid deliverables: In task_116 (EIS analysis), the highest-scoring submission (87.62) contradicts both itself and the raw inputs by claiming low-frequency impedance "grows" across scans while also reporting a drop from 9.7 Ω to 6.8 Ω (raw inputs show Re(Z) drops from 9.6964 Ω at 1.585 Hz to 6.7957 Ω, -29.9%). This input-verifiable error shows that high partial scores can mask real delivery failures. While not every deduction invalidates core conclusions, all rubric items form the delivery contract, with their scientific soundness ensured through independent rubric and Judge audits.
  3. Alignment with strict pass/fail benchmarks: Terminal-Bench and Terminal-Bench-Science similarly require passing all task checks without partial credit.

Therefore, Pass Rate (>=99.9) is the essential metric for evaluating end-to-end scientific workflow completion, rather than relying on high average partial scores.

3. Sensitivity Analysis Across Thresholds

Across 97 tasks × 12 configurations (1,164 historical frozen-verifier evaluations), cumulative pass rates at >=99.9, >=95, and >=90 are:

#Configuration>=99.9>=95>=90
1codex / gpt-5.6-sol20/97 (20.6%)60/97 (61.9%)68/97 (70.1%)
2claude-code / grok-4.620/97 (20.6%)54/97 (55.7%)65/97 (67.0%)
3claude-code / claude-opus-517/97 (17.5%)55/97 (56.7%)63/97 (64.9%)
4claude-code / kimi-k317/97 (17.5%)48/97 (49.5%)61/97 (62.9%)
5codex / gpt-5.6-terra15/97 (15.5%)49/97 (50.5%)64/97 (66.0%)
6claude-code / qwen3.8-max15/97 (15.5%)44/97 (45.4%)58/97 (59.8%)
7claude-code / deepseek-v4-pro13/97 (13.4%)47/97 (48.5%)56/97 (57.7%)
8claude-code / deepseek-v4-flash12/97 (12.4%)43/97 (44.3%)53/97 (54.6%)
9claude-code / gemini-3.7-flash10/97 (10.3%)38/97 (39.2%)51/97 (52.6%)
10claude-code / apodex-1.110/97 (10.3%)35/97 (36.1%)45/97 (46.4%)
11claude-code / qwen3.5-397b-a17b4/97 (4.1%)12/97 (12.4%)27/97 (27.8%)
12claude-code / glm-5.23/97 (3.1%)29/97 (29.9%)42/97 (43.3%)
AllOverall156/1164 (13.4%)514/1164 (44.2%)653/1164 (56.1%)
  • Overall pass rate rises from 13.4% at >=99.9 to 44.2% at >=95 and 56.1% at >=90, with 358/1,164 (30.8%) of runs falling in the 95–99.9 band. (Note: The 13th configuration in the paper, Apodex 1.1 + Frontier Agent Team v2, uses a distinct multi-agent reasoning paradigm and is excluded from this 12-configuration single-agent comparison.)
  • codex / gpt-5.6-sol ranks first at >=95 and >=90, and ties with claude-code / grok-4.6 at >=99.9.

4. task_008 Prompt Threshold (>=80 vs. >=99.9)

The >=80 wording in task_008's supplementary prompt is a legacy error that does not reflect the paper's >=99.9 criterion. We will remove this text in the updated release and further increase the objective score share. Under task_008's actual grader, requiring >=99.9 is well grounded:

  1. Objective checks (85/100 points): These hard checks are independently recomputed via Primer3:
Objective Check CategoryWeightSpecific Checks
Required deliverables6primers.csv, qc_summary.json, report.md, and README.md present and non-empty
CSV schema5Required fields present, 1–3 primer pairs, unique non-empty pair_id
QC semantics5Valid version, gene, recommended row, Primer3 version, background panel, specificity model, and hard-constraint flags
Sequence mapping & claimed values10Sequence/coordinate mapping (5), consistency of claimed Tm, GC, ΔG, coordinates, and off-targets with recomputation (5)
Primer thermodynamics15Length (2), Tm (4), ΔTm (2), GC (3), 3'-end quality (2), homopolymer (2)
Secondary structure12Hairpin (4), self-dimer (4), heterodimer (4)
Amplicon8Sequence & coordinate mapping (4), length (2), GC (2)
Junction / gDNA protection12Forward spanning J1 (5), >=5 nt flanks (4), Reverse in exon2 (3)
Pair specificity12Zero offline off-target products (8), claimed values matching recomputation (4)
Total Objective Score85Scored on the recommended pair, with all reported pairs passing hard constraints
  1. LLM reasoning (15/100 points): The remaining points enforce concrete reporting criteria (methodological rationale ~4.5, pair comparison ~3.75, limitations & validation plan ~3.75, no fabrication ~3.0), making full compliance (>=99.9) scientifically appropriate.

5. Zero Passes in Electrochemistry

Although the best configuration reaches an average score of 94.9 in Electrochemistry, Pass Rate requires each of its 6 tasks to pass >=99.9 individually. All 6 tasks fell short due to numerical/QC tolerance misses, figure consistency gaps, or scientific interpretation errors, as shown by the task_116 case (87.62) in Section 2 above.


W2. Answer Leakage Risk (task_006 URL) and Public Network Access

  1. Rationale for public network access: Realistic scientific workflows require installing packages, consulting public documentation, and using open-source tools. Keeping public network access uniform avoids arbitrary allowlist penalties and avoids shifting the evaluation toward offline dependency recovery. Upon formal dataset release, answers and graders will be isolated under strict authorization controls.
  2. Trajectory audit for task_006: Across all 12 historical ATIF trajectories for task_006, 12/12 read analysis_config.json, yet 0/12 invoked or downloaded the Matbench URL, and none used curl, wget, requests, urllib, httpx, WebFetch, or matminer.load_dataset to fetch raw Matbench data. Scores averaged 88.05 (max 93.1, 0/12 reaching >=95 or >=99.9), showing no evidence of leakage-driven inflation.
  3. Remediation: Retaining the download URL after removing the overlapping subset was an oversight. We will remove the URL from task_006's analysis_config.json (keeping only non-resolvable provenance text), add automated input-leakage regression checks and network logging, and re-run task_006 under public network access.

W3. Statistical Support and Experimental Confounds

  1. Single trials, run stability, and Wilson 95% CIs: Running 3 full independent trials across all 97 × 12 evaluations is infeasible under current time and compute constraints. However, for each task, we analyzed 3 runs of the same evaluated configurations from the closest pre-release versions (differing only slightly in prompt or wording) and observed score standard deviations consistently below 4.1 points. Below are the Wilson 95% confidence intervals for all 12 configurations. In the revision, we will focus on tier-level differences and the macro completion gap rather than overlapping top-tier ranks:
#ConfigurationPassedPass RateWilson 95% CI
1claude-code / grok-4.620/9720.6%13.8%–29.7%
1codex / gpt-5.6-sol20/9720.6%13.8%–29.7%
3claude-code / claude-opus-517/9717.5%11.2%–26.3%
3claude-code / kimi-k317/9717.5%11.2%–26.3%
5claude-code / qwen3.8-max15/9715.5%9.6%–24.0%
5codex / gpt-5.6-terra15/9715.5%9.6%–24.0%
7claude-code / deepseek-v4-pro13/9713.4%8.0%–21.6%
8claude-code / deepseek-v4-flash12/9712.4%7.2%–20.4%
9claude-code / apodex-1.110/9710.3%5.7%–17.9%
9claude-code / gemini-3.7-flash10/9710.3%5.7%–17.9%
11claude-code / qwen3.5-397b-a17b4/974.1%1.6%–10.1%
12claude-code / glm-5.23/973.1%1.1%–8.7%
  1. Confounding factors:
    • Difficulty: We neither pre-allocate difficulty quotas across domains nor curate hard tasks to depress scores. Rather, difficulty reflects the inherent multi-step complexity of real workflows in each discipline.
    • LLM-as-Judge: Judge usage depends strictly on deliverable type: programmatic numerical/schema tasks use deterministic checks alone, while open-ended reports and complex figure checks additionally use GPT-5.6-Sol.
    • Real vs. synthetic data: We will disclose the breakdown of real vs. synthetic/simulated tasks in the revision. Synthetic/simulated data protects unpublished datasets, reduces pretraining contamination risk, and provides exact ground truth, with every task reviewed by human domain experts to match realistic data characteristics and workflows. Accordingly, domain-level results reflect the natural task composition of each discipline rather than isolated single-factor effects.

Minor Weaknesses

MW1. Grader Validity, Expert Alignment, and Rubric Release

  1. Objective checks dominate scoring: Across 796/924 strictly decomposable Judge-task records, objective checks account for 89.97% of actual scores vs. 10.03% for the LLM Judge (45/77 verifiable tasks: median Judge weight 10/100, mean 11.36/100, max 20/100). Across 40 tasks completely decomposable over all 12 configurations, objective scores correlate with pre-cap total scores at Pearson 0.996 and Spearman 0.965. Domain experts reviewed Judge behavior task by task and confirmed that GPT-5.6-Sol (high) aligns well with expert judgment, and re-judging all Judge-evaluated tasks across the full submission set with GPT-6.1-Sol (high) / Claude Opus 5.5 (with fallback, neither among the 12 evaluated configurations) yielded a pass/fail reversal rate below 2%.
  2. Rubric release: We will release all 97 task rubrics under access control and encryption to prevent unauthorized downloads or answer leakage.

MW2. "300 Workflows" vs. 97 Released Tasks

As shown in our domain breakdown in Q4, the 203 held-out tasks span the six scientific domains, though licensing constraints prevent disclosing task-level details. To ensure every claim in the paper is fully backed by described and evaluated tasks, we will revise the abstract and main text to refer strictly to the 97 evaluated workflows rather than 300 workflows.

MW3. Figures 1, 3, 4(a), and 7

Figures 1, 3, and 4(a) were created as vector graphics, with minor rendering/margin artifacts during PDF export. In the revised manuscript, we will (1) re-export Figures 3 and 4(a) cleanly as vector graphics with value labels on 4(a), (2) fix the cropped right-side label in Figure 1, and (3) revise Figure 7's caption and label to accurately describe the input rather than labeling it as "unmodified."


Questions

Q1. Reference Solution Scores and Passability

As detailed in W1, all 97 tasks are verified solvable. Specifically, 87/97 pass directly (>=99.9), 5/97 (009, 010, 094, 099, 104) missed analysis.py sync initially (now fixed to reach >=99.9), and 5/97 reach >=99.9 with minor revisions to solve.py and the verification environment. We will include the complete per-task reference score table in the revised appendix.

Q2. Judge Aggregation and Re-Evaluation Variance

  1. Aggregation: Three independent Judge evaluations of the same submission are combined via a simple arithmetic mean, Jˉ=(J1+J2+J3)/3\bar{J} = (J_1 + J_2 + J_3)/3 (not best-of-3, majority voting, or max-score selection).
  2. Re-evaluation variance: Across 772 weight-verified historical submissions, median task-score sample variance is 0.051 (mean 0.240), with a single-call task-score SD median of 0.226 points (mean 0.319) and a 3-call mean SD (≈s/3\approx s/\sqrt{3}) median of 0.130 points (mean 0.184, P95 0.591) (raw 0–100 Judge subscore median variance 4.10, median SD 2.02). At >=99.9 (685 reconstructible submissions), 98.5% (675/685) yield identical pass/fail decisions across individual Judge calls (within-Judge single-call boundary flip rate 1.5%, i.e., <2% or 10 submissions, distinct from the cross-Judge re-evaluation in MW1).

Q3. Number of Runs and "Valid" Trajectory Selection

Close pre-release versions ran more than 3 times per task during development, whereas the paper reports 1 trajectory per configuration–task pair from completed runs of the frozen final version, never cherry-picked by task score.

Q4. Judge Count Discrepancy (77 vs. 71) and Held-Out Task Distribution

  1. 77 vs. 71: 71 tasks explicitly specify JUDGE_MODEL and JUDGE_API_ENDPOINT in task.toml, while the remaining 6 tasks (task_072_pt_co_adsorption, task_073_ion_diffusion_lammps, task_075_n2_nevpt2_pes, task_114_he_tumor_infiltration, task_116_eis_equivalent_circuit_analysis, task_117_cp2k_mgo_phonon) default to using the Judge model at runtime (so 77 tasks actually use an LLM Judge). We will add explicit declarations to these 6 task.toml files in the updated release.
  2. Held-out task distribution: Together with the 97 released tasks, the 203 held-out tasks span the six domains as shown below (data licensing prevents disclosing further task-level details, and the revised text will focus strictly on the 97 evaluated workflows):
DomainHeld-Out (203)Share of 203Total with Released 97 (300)
Quantum Chemistry4220.69%62
Molecular Dynamics94.43%25
Materials Characterization6833.50%90
Analytical Chemistry4321.18%66
Life Science2210.84%32
Electrochemistry199.36%25
Total203100%300

Thank you again for your detailed review. We will incorporate all of these clarifications and revisions into the updated manuscript.

Reviewer_YbFhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces FrontierChallenge, a benchmark for evaluating the ability of scientific agents to complete end-to-end scientific workflows. The benchmark contains 300 tasks across six scientific domains, with 97 tasks released and evaluated in this paper. Unlike benchmarks that primarily focus on final answers or individual programs, FrontierChallenge emphasizes the completeness and consistency of multiple scientific deliverables, including code, data, figures, and reports. The authors evaluate 12 frontier models across 13 model-scaffold configurations. The results show that even the best-performing configurations achieve only a 20.6% Pass Rate, despite relatively high average scores. The paper further investigates domain-specific performance differences and failure patterns, highlighting a substantial gap between partial task progress and complete scientific workflow execution.

Critical strengths
  1. Evaluating AI agents on realistic scientific workflows is an important and timely research direction, especially as agents are increasingly expected to handle complex, multi-step scientific tasks rather than isolated questions or coding problems. I appreciate the paper’s emphasis on end-to-end workflow completion and the reliability of scientific deliverables, which are important considerations for deploying scientific agents in real-world research settings.

  2. The benchmark covers diverse scientific domains and incorporates realistic task requirements and evaluation criteria. The experiments are extensive, covering 13 model-scaffold configurations, and the analyses of partial progress, completion rates, and failure patterns provide useful insights into current agent limitations. Overall, I find this a solid and well-executed research paper.

Critical weaknesses

While I appreciate the practical motivation, I am not fully convinced that FrontierChallenge introduces a sufficiently distinct evaluation setting compared to existing benchmarks. Conceptually, many tasks seem to resemble extensions of scientific data analysis tasks, such as those in ScienceAgentBench, into longer workflows involving multiple steps and deliverables. Existing benchmarks already cover important aspects of scientific agent capabilities: ScienceAgentBench evaluates data-driven scientific analysis, while PaperBench and NatureBench focus on more research-oriented tasks such as reproducing or improving scientific methods. Given these existing efforts, I find the unique positioning and necessity of FrontierChallenge as a standalone benchmark somewhat unclear. Although the emphasis on complete scientific deliverables is useful, I would appreciate a stronger justification of what fundamentally new capabilities or failure modes this benchmark captures that cannot be adequately evaluated by existing benchmarks.

Questions for authors
  1. How are the task-specific Graders constructed and validated? Were domain experts involved in designing or reviewing the evaluation criteria and reference solutions? I noticed that Appendix C describes package integrity checks and archived Judge controls, but these do not necessarily establish the scientific correctness of the evaluation scripts. Given the diversity and complexity of the scientific tasks, I would appreciate more details on how the authors ensure that the Graders reliably assess scientific correctness.

  2. Is the 99.9 threshold for task completion overly strict? While I understand the motivation for evaluating complete scientific deliverables, this threshold may penalize otherwise successful workflows for relatively minor omissions or formatting issues. Could the authors provide a sensitivity analysis using different thresholds (e.g., 90, 95, and 99.9)? It would be interesting to understand whether the main conclusions remain consistent under less stringent completion criteria.

  3. What are the main reasons behind the large gap between Avg. Score and Pass Rate? Many configurations achieve average scores around 80–90, while their Pass Rates remain below 20%. Are these failures primarily caused by missing deliverables, inconsistencies across artifacts, incorrect scientific reasoning, or execution errors? A more fine-grained analysis, ideally with concrete examples of high-scoring but non-passing trajectories, would help clarify the actual bottlenecks in current scientific agents.

4 comments

Thank you for your supportive and thoughtful review. Below are our responses to your comments and questions.


Weaknesses

W1. Distinctiveness Relative to ScienceAgentBench, PaperBench, and NatureBench

We agree that longer workflows or additional files alone do not define a new benchmark paradigm. Our goal is to evaluate whether a specified, technically intensive scientific workflow can be reliably delegated to an agent as a trustworthy assistant, requiring domain computation, quality control (QC), supporting evidence, and scientific interpretation to form a consistent, reusable deliverable bundle.

FrontierChallenge complements existing benchmarks across four distinct evaluation dimensions (which we will clarify in Section 2 and Table 1):

BenchmarkCore Evaluation GoalDomains & ToolingAcceptance GranularityPrimary Failure Mode Exposed
ScienceAgentBenchCode generation for data-driven analysis4 disciplines, Python librariesSingle program and its direct outputsErrors in single-program analysis and execution outputs
PaperBenchEnd-to-end ML paper reproductionMachine learningRepo execution and paper-result checksImplementation and reproduction of ML methods
NatureBenchMatching/surpassing SOTA on scientific ML tasksScientific machine learningHidden test metrics vs. paper SOTAWhether model performance reaches SOTA
FrontierChallengeReliable delegation of specified scientific workflows6 natural-science domains, domain software (ORCA, CP2K, LAMMPS, etc.)Cross-stage multi-artifact contract (evidence + QC + figures + report)High partial scores with broken evidence chains or cross-artifact contradictions

Multi-artifact contracts allow graders to verify cross-stage scientific dependencies rather than just counting files. For example, in our Thermodynamic Integration (TI) task, reporting a plausible free energy value is not enough: the grader independently re-integrates the hydration free energy from the submission's own per-lambda ∂V/∂λ\partial V / \partial \lambda simulation outputs. This design exposes concrete scientific-validity failures missed by single-output checks:

  • Disconnected computational evidence: In task_063 (Metadynamics, score 96.78), results.json disagreed with the FES file and the reweighted grid contained inf values.
  • Broken upstream-to-downstream QC propagation: In task_108 (LDH purification, score 97.85), flagged dilution wells from upstream QC were still included in downstream specific-activity calculations, corrupting summary metrics and final recommendations.
  • Self-contradictory interpretation: In task_116 (EIS analysis, score 87.62), the report claimed low-frequency impedance "grows" across scans while simultaneously reporting a drop from 9.7 Ω to 6.8 Ω.

Questions

Q1. Grader Construction, Expert Validation, and Scientific Correctness

Our task graders are designed and annotated directly by human domain experts alongside reference solvers (solve.py):

  1. Expert-authored checks with independent recomputation: Domain experts wrote and cross-reviewed the evaluation criteria, physical/chemical tolerances, and grade.py scripts. Across 796/924 strictly decomposable Judge-task records, deterministic objective checks account for 89.97% of actual scores vs. 10.03% for the LLM Judge (Pearson 0.996 with total scores), routinely recomputing metrics directly from submitted raw outputs.
  2. End-to-end validation with reference solvers: Validating all 97 graders against expert solve.py solutions shows:
    • 87/97 tasks pass (>=99.9) directly.
    • 5/97 tasks (009, 010, 094, 099, 104) passed all core scientific, schema, and figure checks. Points were initially deducted solely because analysis.py was not synced into the reference packages (now fixed so all 5 reach >=99.9).
    • 5/97 remaining tasks are expert-verified solvable and achieve >=99.9 with minor revisions to solve.py and the verification environment.
  3. Expert alignment on Judge rubrics: For the open-ended semantic criteria assessed by the LLM Judge (~10% of actual score), rubrics were annotated by domain experts, who reviewed Judge behavior task by task and confirmed GPT-5.6-Sol (high) aligns well with expert judgment. Re-judging all Judge-evaluated tasks across the full submission set with GPT-6.1-Sol (high) / Claude Opus 5.5 (with fallback, neither among the 12 evaluated configurations) yielded a pass/fail reversal rate below 2%.

Q2. Strictness of 99.9 and Sensitivity Analysis (90, 95, 99.9)

1. Rationale for >=99.9

We use task_score >= 99.9 as the operational criterion for full-contract completion:

  • Each Judge semantic item is evaluated three times and averaged (with a 0.1-point margin below 100 for minor rounding/averaging variance), and we pair this strict bar with the multi-threshold sensitivity analysis below.
  • High partial scores do not guarantee valid deliverables (as shown by task_116 at 87.62, task_063 at 96.78, and task_108 at 97.85), making Pass Rate (>=99.9) essential for evaluating reliable scientific handoff.

2. Sensitivity Analysis at >=99.9, >=95, and >=90

Across 97 tasks × 12 configurations (1,164 historical frozen-verifier evaluations), cumulative pass rates are:

#Configuration>=99.9>=95>=90
1codex / gpt-5.6-sol20/97 (20.6%)60/97 (61.9%)68/97 (70.1%)
2claude-code / grok-4.620/97 (20.6%)54/97 (55.7%)65/97 (67.0%)
3claude-code / claude-opus-517/97 (17.5%)55/97 (56.7%)63/97 (64.9%)
4claude-code / kimi-k317/97 (17.5%)48/97 (49.5%)61/97 (62.9%)
5codex / gpt-5.6-terra15/97 (15.5%)49/97 (50.5%)64/97 (66.0%)
6claude-code / qwen3.8-max15/97 (15.5%)44/97 (45.4%)58/97 (59.8%)
7claude-code / deepseek-v4-pro13/97 (13.4%)47/97 (48.5%)56/97 (57.7%)
8claude-code / deepseek-v4-flash12/97 (12.4%)43/97 (44.3%)53/97 (54.6%)
9claude-code / gemini-3.7-flash10/97 (10.3%)38/97 (39.2%)51/97 (52.6%)
10claude-code / apodex-1.110/97 (10.3%)35/97 (36.1%)45/97 (46.4%)
11claude-code / qwen3.5-397b-a17b4/97 (4.1%)12/97 (12.4%)27/97 (27.8%)
12claude-code / glm-5.23/97 (3.1%)29/97 (29.9%)42/97 (43.3%)
AllOverall156/1164 (13.4%)514/1164 (44.2%)653/1164 (56.1%)
  • Overall pass rate rises from 13.4% (>=99.9) to 44.2% (>=95) and 56.1% (>=90), with 358/1,164 (30.8%) in the 95–99.9 band. (Note: The 13th configuration in the paper, Apodex 1.1 + Frontier Agent Team v2, uses a distinct multi-agent reasoning paradigm and is excluded from this 12-configuration single-agent comparison.)
  • Main conclusions hold across thresholds: While absolute pass rates rise substantially under relaxed bars (e.g., the top configuration increases from 20.6% at >=99.9 to 61.9% at >=95 and 70.1% at >=90), even at >=90, 43.9% of all runs fail (and the top configuration falls short on ~30% of tasks), while overall capability tiers remain stable (codex / gpt-5.6-sol leads at >=95 and >=90 and ties for first at >=99.9).

Q3. Attribution of the Avg. Score vs. Pass Rate Gap and Concrete Examples

Auditing the non-passing trajectories (<99.9) shows that the gap arises because partial credit accumulates across sub-steps whereas complete delivery does not allow missed requirements to be compensated by high partial scores. Nearly half of all non-passing runs (497/1,008, or 49.3%) still score >=90. The following examples illustrate four observed numerical and scientific-consistency failure modes:

  1. Cross-artifact inconsistencies: In task_063 (Metadynamics, score 96.78), simulations completed, but results.json disagreed with the FES data file and the reweighted grid contained 498/1,296 inf values.
  2. Broken QC propagation and statistical reasoning: In task_108 (LDH purification, score 97.85), D80 dilution wells flagged for review in upstream QC were retained in downstream calculations (n_rate_wells=6 instead of 3), corrupting mean specific activity and recommendations. In task_036 (kinetic fitting, score 91.62), three 7-parameter fits were pooled while keeping k=7 instead of k=21, underestimating the AICc penalty by ~44.73.
  3. Scientific claims contradicting raw inputs: In task_116 (EIS analysis, score 87.62), the report claimed at line 195 that low-frequency impedance at SOC100 "grows" across scans while reporting a drop from 9.7 Ω to 6.8 Ω at line 199 (raw inputs show maximum Re(Z) drops from 9.6964 Ω to 6.7957 Ω, -29.9%).
  4. Local numerical tolerance misses: In task_042 (RRDE/ORR, score 98.82), 46/2,052 points in H2O2_percent and 43/2,052 points in electron_transfer_number exceeded tolerance despite otherwise complete analysis.

We will add these case analyses to the revised manuscript.


Thank you again for your helpful feedback. We will incorporate all of these clarifications into the revised version.

Reviewer_YbFhOpenReview ↗

Response to Authors' Rebuttal

Thank you for the detailed responses and additional analyses. The clarifications on grader validation, threshold sensitivity, and failure modes have addressed my main concerns. I maintain my original rating.

Thank you very much for your follow-up and support. We are glad that our clarifications addressed your main concerns, and we will follow your suggestions to carefully revise and refine the manuscript. Thank you for your valuable feedback again.

Reviewer_ggVxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FrontierChallenge tests end-to-end scientific workflows: each task fixes inputs and environment and requires a contract-specified deliverable bundle, with 97 of 300 tasks in six domains released. Graders mix deterministic checks and GPT-5.6 Sol judging, and passing needs >=99.9/100. Thirteen model / scaffold setups reach 3.1–20.6% Pass Rate versus 67.5–87.9 Avg. Score.

Critical strengths

Tasks define multi-file contracts on real software (ORCA, CP2K, LAMMPS) with separate, checksummed agent and evaluator packages (App. C), checking produced files rather than one final number.

The study spans 13 model / scaffold setups across six domains and three difficulty tiers (Table 2, Fig. 3a), and the authors openly note limits in App. A and App. C.

Critical weaknesses

The >=99.9 Pass Rate bar is uncalibrated, so the gap may reflect strict or noisy grading. Reference solutions are only promised, so we cannot verify that every task passes. Judge checks on p. 17 used another model with 14/63 hash mismatches and no human or threshold checks. GPT-5.6 Sol acts as Judge while topping Avg. Score, risking self-preference, as Judge-heavy domains (materials, analytical, and electrochemistry, 83.3–96.5% Judge-flagged failures, p. 9) get 10 passes in 663 runs.

Comparisons and diagnostics are thin. Single selected trials lack CIs (p. 12), so the top six (15.5–20.6%, CI about +/-8 pp) are likely within noise, and varying model and scaffold confounds comparisons (p. 5). On p. 9, passes show more completion claims (90.1% vs 75.5%) and tool errors (94.2% vs 80.7%) than failures, so those stats do not explain failures, and HDS6 (process score) is uncalibrated.

Minor:

Unreported: 203 held-out tasks, scaffold versions, budgets, network access, and reasoning effort. Reproducibility: proprietary Judge, ORCA not redistributed (16 tasks), and evaluator archives needing "authenticated decryption" (p. 17). Positioning: cite TheAgentCompany (2025), SciCode (2024), and EXP-Bench (2026), which report this gap, and state what is new.

Questions for authors

Do reference solutions reach >=99.9 under the exact official pipeline, i.e., is every task passable?

How does Pass Rate change at thresholds 95 and 98? Does the ranking hold if a judge outside the evaluated model families re-scores a task subset?

4 comments

Thank you for the thorough and constructive review. Below are our point-by-point responses.


Weaknesses

W1. Calibration of >=99.9 and Verification of Reference Solutions

1. Why >=99.9 Is the Pass Threshold

We define task_score >= 99.9 as full completion to distinguish high partial scores from complete, reliable delivery:

  1. Full rubric satisfaction: We use >=99.9 as the operational criterion for full-contract completion. Each Judge semantic item is scored three times and averaged (with a 0.1-point margin below 100 for minor rounding/averaging variance), and we pair this strict bar with re-evaluation stability and multi-threshold sensitivity analyses below.
  2. High partial scores do not imply valid deliverables: In task_116 (EIS analysis), the highest-scoring submission (87.62) contradicts both itself and the raw inputs by claiming low-frequency impedance "grows" across scans while also reporting a drop from 9.7 Ω to 6.8 Ω (raw inputs show Re(Z) drops from 9.6964 Ω at 1.585 Hz to 6.7957 Ω, -29.9%). This input-verifiable error shows that high partial scores can mask real delivery failures.
  3. Comparable benchmarks: Terminal-Bench and Terminal-Bench-Science similarly enforce strict pass/fail checks without partial credit. Therefore, Pass Rate (>=99.9) is the essential metric for evaluating end-to-end scientific workflow completion, rather than relying on high average partial scores.

2. Task Solvability and Reference Solutions

All 97 released tasks have been verified as solvable:

  • 87/97 tasks pass (>=99.9) directly using the expert-provided reference solutions.
  • 5/97 tasks (009, 010, 094, 099, 104) passed all core scientific, schema, and figure checks. Points were initially deducted solely because analysis.py was not synced into the reference packages (now fixed so all 5 reach >=99.9).
  • 5/97 remaining tasks have been verified as solvable by domain experts and achieve >=99.9 with minor adjustments to solve.py and the verification environment (e.g., ORCA/fixture dependencies or supplementary outputs).

3. Sensitivity Analysis Across Thresholds

Across 97 tasks × 12 configurations (1,164 historical frozen-verifier evaluations), cumulative pass rates at >=99.9, >=98, >=95, and >=90 are:

#Configuration>=99.9>=98>=95>=90
1codex / gpt-5.6-sol20/97 (20.6%)43/97 (44.3%)60/97 (61.9%)68/97 (70.1%)
2claude-code / grok-4.620/97 (20.6%)39/97 (40.2%)54/97 (55.7%)65/97 (67.0%)
3claude-code / claude-opus-517/97 (17.5%)35/97 (36.1%)55/97 (56.7%)63/97 (64.9%)
4claude-code / kimi-k317/97 (17.5%)32/97 (33.0%)48/97 (49.5%)61/97 (62.9%)
5codex / gpt-5.6-terra15/97 (15.5%)33/97 (34.0%)49/97 (50.5%)64/97 (66.0%)
6claude-code / qwen3.8-max15/97 (15.5%)29/97 (29.9%)44/97 (45.4%)58/97 (59.8%)
7claude-code / deepseek-v4-pro13/97 (13.4%)26/97 (26.8%)47/97 (48.5%)56/97 (57.7%)
8claude-code / deepseek-v4-flash12/97 (12.4%)26/97 (26.8%)43/97 (44.3%)53/97 (54.6%)
9claude-code / apodex-1.110/97 (10.3%)20/97 (20.6%)35/97 (36.1%)45/97 (46.4%)
10claude-code / gemini-3.7-flash10/97 (10.3%)16/97 (16.5%)38/97 (39.2%)51/97 (52.6%)
11claude-code / qwen3.5-397b-a17b4/97 (4.1%)6/97 (6.2%)12/97 (12.4%)27/97 (27.8%)
12claude-code / glm-5.23/97 (3.1%)8/97 (8.2%)29/97 (29.9%)42/97 (43.3%)
AllOverall156/1164 (13.4%)313/1164 (26.9%)514/1164 (44.2%)653/1164 (56.1%)

Broad capability tiers remain consistent across thresholds (358/1,164, or 30.8%, of runs fall in the 95–99.9 band), though specific adjacent ranks shift under relaxed bars (detailed in Q2). (Note: The 13th configuration in the paper, Apodex 1.1 + Frontier Agent Team v2, uses a distinct multi-agent reasoning paradigm and is excluded from this 12-configuration single-agent comparison.)


W2. Judge Verification Integrity (Appendix C, p. 17: 14/63 Hash Mismatches)

Thank you for pointing out this issue in Appendix C (p. 17). The 63 records in Appendix C were early packaging-stage positive/negative sanity controls run with qwen/qwen3.7-flash solely to verify that the Judge pipeline separated clearly valid from clearly invalid inputs. They were not an independent expert validation of the main gpt-5.6-sol Judge or a calibration of the 99.9 threshold. The 14/63 hash mismatches occurred because we subsequently refined those tasks multiple times to improve specification completeness and reduce unintended constraints on solving agents, reflecting our iterative curation. We will clarify the nature of these 63 archival control records in Appendix C.


W3. Judge Self-Preference Risk (GPT-5.6 Sol as Both Judge and Evaluated Model)

Four empirical findings address the risk of Judge self-preference and explain the low pass rates in Judge-heavy domains:

  1. Objective checks dominate scoring: Across 796/924 decomposable records, objective checks contribute 89.97% of actual scores vs. 10.03% for the LLM Judge (45/77 verifiable tasks: median Judge weight 10/100). Across 40 tasks completely decomposable over all 12 configurations, pure objective scores correlate with total scores at Pearson 0.996 and Spearman 0.965.
  2. Expert alignment: All semantic rubrics were annotated by domain experts, who reviewed Judge behavior task by task and confirmed GPT-5.6-Sol (high) aligns with expert judgment.
  3. Low reversal rate under external re-judging: We re-judged all Judge-evaluated tasks across the full submission set using GPT-6.1-Sol (high) / Claude Opus 5.5 (with fallback, neither among the 12 evaluated configurations), and the pass/fail reversal rate remained below 2%.
  4. No monopoly and clear non-Judge failures: At >=99.9, codex / gpt-5.6-sol (20/97, 20.6%) ties with claude-code / grok-4.6 (20/97, 20.6%), and failures in electrochemistry include clear deterministic/raw-input errors independent of Judge preferences (e.g., task_116 at 87.62 contradicting raw data, and task_042 at 98.82 exceeding curve-point tolerances).

W4. Single Trials, Stability, Confidence Intervals, and Model-Scaffold Confounding

  1. Single-trial selection, stability, and Wilson 95% CIs: The paper reports 1 trajectory per configuration–task pair from completed runs of the frozen final version, never cherry-picked by task score. For each task, we analyzed 3 runs of the same evaluated configurations from the closest pre-release versions (differing only slightly in prompt or wording) and observed score standard deviations consistently below 4.1 points. Below are the Wilson 95% confidence intervals for all 12 configurations, which support focusing on tier-level differences and the macro completion gap rather than overlapping top-tier ranks:
#ConfigurationPassedPass RateWilson 95% CI
1claude-code / grok-4.620/9720.6%13.8%–29.7%
1codex / gpt-5.6-sol20/9720.6%13.8%–29.7%
3claude-code / claude-opus-517/9717.5%11.2%–26.3%
3claude-code / kimi-k317/9717.5%11.2%–26.3%
5claude-code / qwen3.8-max15/9715.5%9.6%–24.0%
5codex / gpt-5.6-terra15/9715.5%9.6%–24.0%
7claude-code / deepseek-v4-pro13/9713.4%8.0%–21.6%
8claude-code / deepseek-v4-flash12/9712.4%7.2%–20.4%
9claude-code / apodex-1.110/9710.3%5.7%–17.9%
9claude-code / gemini-3.7-flash10/9710.3%5.7%–17.9%
11claude-code / qwen3.5-397b-a17b4/974.1%1.6%–10.1%
12claude-code / glm-5.23/973.1%1.1%–8.7%
  1. Model-scaffold pairing (p. 5): Each model was paired with its native or recommended scaffold (codex for OpenAI GPT-5.6 models, and claude-code uniformly across the remaining 10 models). Cross-scaffold comparisons therefore evaluate complete model-scaffold configurations, whereas within-scaffold comparisons across the 10 claude-code configurations hold the scaffold fixed.

W5. Diagnostic Power of Behavioral Metrics and Calibration of HDS6

We agree that the completion-claim (90.1% vs. 75.5%) and tool-error (94.2% vs. 80.7%) statistics on p. 9 are descriptive telemetry (which may reflect differences in execution length and error recovery in passing runs) rather than causal predictors of failure.

To explain why high-scoring runs fail, we will add a case-based failure audit from the 1,008 non-passing trajectories highlighting four numerical and scientific-consistency bottlenecks: (1) cross-artifact inconsistencies (task_063 at 96.78, where results.json mismatched FES data with 498/1,296 inf values), (2) broken QC propagation or statistical reasoning (task_108 at 97.85 retaining flagged D80 wells, and task_036 at 91.62 setting k=7 instead of k=21 in pooled AICc), (3) scientific claims contradicting raw inputs (task_116 at 87.62), and (4) local numerical tolerance misses (task_042 at 98.82 with 46/2,052 and 43/2,052 curve points out of tolerance).

Regarding HDS6: it evaluates six process capabilities, namely Coherence (deliverable inventory and cross-artifact consistency), Evidence (values grounded in actual execution and input inspection), Alternatives (method selection and input-defect verification), Scope (schema, boundaries, prohibitions), Tools (genuine analysis and clean reproduction), and Repair (pre-submission checks, diagnosis, and repair). Each task has 13 weighted fixed shared items and 6–10 task-specific items, each scored with explicit 0 / 1 / 2 anchors (missing / partial / well-executed). HDS6 serves as an auxiliary process-diagnostic rubric rather than a metric calibrated on an absolute human-expert scale. All primary conclusions rely on the task-level Pass Rate (>=99.9). We will include this definition and boundary in the revision.


Minor Weaknesses

MW1. Experimental Details, Reproducibility Constraints, and Related Work Positioning

  1. Experimental details and 203 held-out tasks:
    • Scaffolds, budgets, and reasoning effort: Scaffold versions were Claude Code 2.1.231 and Codex 0.147.0. Staging raised execution timeouts to a 24-hour floor (96 tasks at 24h, task_072 at 48h), with native model context limits (primarily 1M). Reasoning effort was set to max for gpt-5.6-terra, dynamic thinking for Gemini, and default for all other models. All 97 tasks ran under uniform public network access to support package installation and documentation lookup (task_006 audit confirmed 0/12 trajectories accessed the leaked URL, which will be removed).
    • 203 held-out tasks: Together with the 97 released tasks, the 203 held-out tasks span the six domains below (licensing constraints prevent disclosing task-level details, and the revised abstract and text will focus strictly on the 97 evaluated workflows):
DomainHeld-Out (203)Share of 203Total with Released 97 (300)
Quantum Chemistry4220.69%62
Molecular Dynamics94.43%25
Materials Characterization6833.50%90
Analytical Chemistry4321.18%66
Life Science2210.84%32
Electrochemistry199.36%25
Total203100%300
  1. Reproducibility: Authenticated decryption and access controls on evaluator archives prevent unauthorized agent access and contamination. For ORCA (16 tasks), third-party licensing prohibits binary redistribution, so we provide standardized setup instructions for licensed users. Meanwhile, deterministic checks supply 89.97% of actual scores with minimal Judge variance (median 0.051), reducing reliance on the proprietary Judge.
  2. Positioning (TheAgentCompany 2025, SciCode 2024, EXP-Bench 2026): We will cite and contrast all three works in Related Work. Beyond enterprise office tasks (TheAgentCompany), standalone scientific function synthesis (SciCode), or ML experiment replication (EXP-Bench), FrontierChallenge targets the reliable delegation of end-to-end natural-science workflows across six domains on specialized software stacks (ORCA, CP2K, LAMMPS), enforcing cross-stage multi-artifact contracts (computational evidence, QC propagation, figures, and reports) to expose high-partial-score scientific-validity failures.

Questions

Q1. Reference Solution Passability Under the Official Pipeline

Yes, all 97 released tasks are verified passable. Specifically, 87/97 pass directly (>=99.9), 5/97 (009, 010, 094, 099, 104) missed analysis.py sync initially (now fixed to reach >=99.9), and 5/97 achieve >=99.9 with minor revisions to solve.py and the verification environment.

Q2. Pass Rate at 95 and 98 and External-Judge Ranking Robustness

1. Pass Rate and Ranking Shifts at >=98 and >=95

As shown in W1, relaxing the bar from >=99.9 to >=98 and >=95 raises overall pass rate from 13.4% (156/1,164) to 26.9% (313/1,164) and 44.2% (514/1,164). Broad tiers remain consistent, while specific adjacent ranks shift:

  • At >=98: gpt-5.6-terra (33/97) passes kimi-k3 (32/97), and deepseek-v4-pro ties deepseek-v4-flash (26/97).
  • At >=95: claude-opus-5 (55/97) passes grok-4.6 (54/97), gpt-5.6-terra (49/97) passes kimi-k3 (48/97), deepseek-v4-pro (47/97) passes qwen3.8-max (44/97), and gemini-3.7-flash (38/97) passes apodex-1.1 (35/97).

2. Ranking Robustness Under External Re-Judging

Three lines of evidence indicate high scoring and ranking consistency across evaluation approaches:

  1. Low reversal rate under external re-judging: Re-judging all Judge-evaluated tasks across the full submission set with GPT-6.1-Sol (high) / Claude Opus 5.5 (with fallback, neither among the 12 evaluated configurations) yielded a pass/fail reversal rate below 2%.
  2. Agreement between objective and total scores: Across 796/924 decomposable records, the LLM Judge accounts for only 10.03% of actual scores (vs. 89.97% deterministic), and on 40 completely decomposable tasks across all 12 configurations, pure objective scores correlate with total scores at Spearman 0.965 (Pearson 0.996).
  3. Expert alignment: All semantic rubrics were annotated by domain experts, who reviewed Judge behavior task by task and confirmed GPT-5.6-Sol (high) aligns well with expert judgment.

Thank you again for your constructive feedback. We will incorporate all of these clarifications and revisions into the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.