- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces FrontierChallenge, a benchmark that evaluates whether agents can complete scientific workflows end to end and deliver complete, mutually consistent artifact packages (code, tables, figures, and reports). The authors describe 300 workflows in total, of which 97 are released and evaluated, covering six domains: quantum chemistry, molecular dynamics, materials, analytical chemistry, life sciences, and electrochemistry. Each task has a task-specific grader, semantic rubric items are scored by GPT-5.6 Sol as a judge, and a task counts as passed only at a score of at least 99.9. Across 13 model–scaffold configurations, average scores range from 67.5 to 87.9, while the best pass rate is only 20.6%. The paper also reports HDS6 process scoring and an analysis of failure patterns.
Critical strengths
- The topic is valuable. Most existing benchmarks for scientific agents check only final answers or single programs. This paper targets the ability to deliver complete artifacts according to a contract, which is closer to real research and fills a clear gap.
- The task packages are of high quality. All 97 released tasks come with detailed instructions, fixed inputs, and environment configurations, covering six domains and 21 workflow families. This is a substantial construction effort with direct reuse value for the community.
- The core observation is insightful. The gap between high partial scores and low rates of complete delivery clearly exposes a weakness of current agents on long workflows.
- The evaluation is large and multi-faceted: 13 configurations, 3 scaffolds, and analyses of process scores, failure patterns, tool errors, and resource use. I recomputed the reported numbers, and they are internally consistent.
Critical weaknesses
- There is no evidence that the tasks are solvable at the 99.9 threshold, and the threshold itself is not justified. The paper reports neither the scores of reference solutions under the graders nor a human baseline. In electrochemistry, all 13 configurations pass zero tasks even though the best average score in that domain is 94.9. In addition, task_008 in the supplementary material tells the agent that a total score of at least 80 counts as a pass, which is inconsistent with the 99.9 criterion in the paper. I suggest reporting reference-solution scores, adding a sensitivity analysis at thresholds such as ≥95 and ≥90, and aligning the pass criteria that tasks show to agents with the paper.
- There is a risk of answer leakage, and the experimental configuration is unclear. In the supplementary material, all 97 tasks have network access set to public. In task_006, the authors themselves found that the source data overlapped about 94% with the graded answer and removed it, yet the agent-visible configuration file still contains the download URL of that data. I suggest rerunning with network access disabled, or auditing the agents' web access.
- The statistical support is weak, and there are several confounds. Only one trajectory is kept per configuration–task pair, and no confidence intervals are given (the 95% interval for 20/97 is 13.8%–29.7%). Domain differences are strongly confounded with difficulty (analytical chemistry and electrochemistry are almost entirely Hard tasks), with whether an LLM judge is used, and with whether the data are synthetic (52 of the 97 tasks use synthetic data, a proportion the paper does not disclose). I suggest running at least 3 trials with intervals and reporting results stratified by difficulty, judge vs. no judge, and real vs. synthetic data.
Minor:
- The validity of the graders would benefit from additional evidence. There is no expert–grader agreement data, and the judge model, GPT-5.6 Sol, is also an evaluated model. I suggest a small expert agreement study, or cross-judging with a model that is not under evaluation, and releasing the rubrics.
- The "300 workflows" in the abstract is not supported, since only 97 tasks are described and evaluated.
- The heatmap cells in Figures 3 and 4(a) are low-resolution raster images, and Figure 4(a) has no value labels.
- A label on the right side of Figure 1 is cropped.
- Figure 7 is labeled as "unmodified" input, but it is a synthetic image.
Questions for authors
- What score does each task's reference solution receive under the grader? How many reach 99.9?
- How are the three judge calls aggregated, and what is the score variance when the same submission is re-evaluated?
- How many runs were actually performed per configuration–task pair, and by what criterion was the "valid" trajectory selected?
- The paper says 77 tasks use a judge, while only 71 tasks in the supplementary material configure one. Where does the difference come from? And what is the distribution of the 203 held-out tasks?
4 comments
4. task_008 Prompt Threshold (>=80 vs. >=99.9)
The >=80 wording in task_008's supplementary prompt is a legacy error that does not reflect the paper's >=99.9 criterion. We will remove this text in the updated release and further increase the objective score share. Under task_008's actual grader, requiring >=99.9 is well grounded:
- Objective checks (85/100 points): These hard checks are independently recomputed via Primer3:
| Objective Check Category | Weight | Specific Checks |
|---|---|---|
| Required deliverables | 6 | primers.csv, qc_summary.json, report.md, and README.md present and non-empty |
| CSV schema | 5 | Required fields present, 1–3 primer pairs, unique non-empty pair_id |
| QC semantics | 5 | Valid version, gene, recommended row, Primer3 version, background panel, specificity model, and hard-constraint flags |
| Sequence mapping & claimed values | 10 | Sequence/coordinate mapping (5), consistency of claimed Tm, GC, ΔG, coordinates, and off-targets with recomputation (5) |
| Primer thermodynamics | 15 | Length (2), Tm (4), ΔTm (2), GC (3), 3'-end quality (2), homopolymer (2) |
| Secondary structure | 12 | Hairpin (4), self-dimer (4), heterodimer (4) |
| Amplicon | 8 | Sequence & coordinate mapping (4), length (2), GC (2) |
| Junction / gDNA protection | 12 | Forward spanning J1 (5), >=5 nt flanks (4), Reverse in exon2 (3) |
| Pair specificity | 12 | Zero offline off-target products (8), claimed values matching recomputation (4) |
| Total Objective Score | 85 | Scored on the recommended pair, with all reported pairs passing hard constraints |
- LLM reasoning (15/100 points): The remaining points enforce concrete reporting criteria (methodological rationale ~4.5, pair comparison ~3.75, limitations & validation plan ~3.75, no fabrication ~3.0), making full compliance (>=99.9) scientifically appropriate.
5. Zero Passes in Electrochemistry
Although the best configuration reaches an average score of 94.9 in Electrochemistry, Pass Rate requires each of its 6 tasks to pass >=99.9 individually. All 6 tasks fell short due to numerical/QC tolerance misses, figure consistency gaps, or scientific interpretation errors, as shown by the task_116 case (87.62) in Section 2 above.
W2. Answer Leakage Risk (task_006 URL) and Public Network Access
- Rationale for public network access: Realistic scientific workflows require installing packages, consulting public documentation, and using open-source tools. Keeping public network access uniform avoids arbitrary allowlist penalties and avoids shifting the evaluation toward offline dependency recovery. Upon formal dataset release, answers and graders will be isolated under strict authorization controls.
- Trajectory audit for
task_006: Across all 12 historical ATIF trajectories fortask_006, 12/12 readanalysis_config.json, yet 0/12 invoked or downloaded the Matbench URL, and none usedcurl,wget,requests,urllib,httpx,WebFetch, ormatminer.load_datasetto fetch raw Matbench data. Scores averaged 88.05 (max 93.1, 0/12 reaching >=95 or >=99.9), showing no evidence of leakage-driven inflation. - Remediation: Retaining the download URL after removing the overlapping subset was an oversight. We will remove the URL from
task_006'sanalysis_config.json(keeping only non-resolvable provenance text), add automated input-leakage regression checks and network logging, and re-runtask_006under public network access.
W3. Statistical Support and Experimental Confounds
- Single trials, run stability, and Wilson 95% CIs: Running 3 full independent trials across all 97 × 12 evaluations is infeasible under current time and compute constraints. However, for each task, we analyzed 3 runs of the same evaluated configurations from the closest pre-release versions (differing only slightly in prompt or wording) and observed score standard deviations consistently below 4.1 points. Below are the Wilson 95% confidence intervals for all 12 configurations. In the revision, we will focus on tier-level differences and the macro completion gap rather than overlapping top-tier ranks:
| # | Configuration | Passed | Pass Rate | Wilson 95% CI |
|---|---|---|---|---|
| 1 | claude-code / grok-4.6 | 20/97 | 20.6% | 13.8%–29.7% |
| 1 | codex / gpt-5.6-sol | 20/97 | 20.6% | 13.8%–29.7% |
| 3 | claude-code / claude-opus-5 | 17/97 | 17.5% | 11.2%–26.3% |
| 3 | claude-code / kimi-k3 | 17/97 | 17.5% | 11.2%–26.3% |
| 5 | claude-code / qwen3.8-max | 15/97 | 15.5% | 9.6%–24.0% |
| 5 | codex / gpt-5.6-terra | 15/97 | 15.5% | 9.6%–24.0% |
| 7 | claude-code / deepseek-v4-pro | 13/97 | 13.4% | 8.0%–21.6% |
| 8 | claude-code / deepseek-v4-flash | 12/97 | 12.4% | 7.2%–20.4% |
| 9 | claude-code / apodex-1.1 | 10/97 | 10.3% | 5.7%–17.9% |
| 9 | claude-code / gemini-3.7-flash | 10/97 | 10.3% | 5.7%–17.9% |
| 11 | claude-code / qwen3.5-397b-a17b | 4/97 | 4.1% | 1.6%–10.1% |
| 12 | claude-code / glm-5.2 | 3/97 | 3.1% | 1.1%–8.7% |
- Confounding factors:
- Difficulty: We neither pre-allocate difficulty quotas across domains nor curate hard tasks to depress scores. Rather, difficulty reflects the inherent multi-step complexity of real workflows in each discipline.
- LLM-as-Judge: Judge usage depends strictly on deliverable type: programmatic numerical/schema tasks use deterministic checks alone, while open-ended reports and complex figure checks additionally use GPT-5.6-Sol.
- Real vs. synthetic data: We will disclose the breakdown of real vs. synthetic/simulated tasks in the revision. Synthetic/simulated data protects unpublished datasets, reduces pretraining contamination risk, and provides exact ground truth, with every task reviewed by human domain experts to match realistic data characteristics and workflows. Accordingly, domain-level results reflect the natural task composition of each discipline rather than isolated single-factor effects.
Minor Weaknesses
MW1. Grader Validity, Expert Alignment, and Rubric Release
- Objective checks dominate scoring: Across 796/924 strictly decomposable Judge-task records, objective checks account for 89.97% of actual scores vs. 10.03% for the LLM Judge (45/77 verifiable tasks: median Judge weight 10/100, mean 11.36/100, max 20/100). Across 40 tasks completely decomposable over all 12 configurations, objective scores correlate with pre-cap total scores at Pearson 0.996 and Spearman 0.965. Domain experts reviewed Judge behavior task by task and confirmed that GPT-5.6-Sol (high) aligns well with expert judgment, and re-judging all Judge-evaluated tasks across the full submission set with GPT-6.1-Sol (high) / Claude Opus 5.5 (with fallback, neither among the 12 evaluated configurations) yielded a pass/fail reversal rate below 2%.
- Rubric release: We will release all 97 task rubrics under access control and encryption to prevent unauthorized downloads or answer leakage.
MW2. "300 Workflows" vs. 97 Released Tasks
As shown in our domain breakdown in Q4, the 203 held-out tasks span the six scientific domains, though licensing constraints prevent disclosing task-level details. To ensure every claim in the paper is fully backed by described and evaluated tasks, we will revise the abstract and main text to refer strictly to the 97 evaluated workflows rather than 300 workflows.
MW3. Figures 1, 3, 4(a), and 7
Figures 1, 3, and 4(a) were created as vector graphics, with minor rendering/margin artifacts during PDF export. In the revised manuscript, we will (1) re-export Figures 3 and 4(a) cleanly as vector graphics with value labels on 4(a), (2) fix the cropped right-side label in Figure 1, and (3) revise Figure 7's caption and label to accurately describe the input rather than labeling it as "unmodified."
Questions
Q1. Reference Solution Scores and Passability
As detailed in W1, all 97 tasks are verified solvable. Specifically, 87/97 pass directly (>=99.9), 5/97 (009, 010, 094, 099, 104) missed analysis.py sync initially (now fixed to reach >=99.9), and 5/97 reach >=99.9 with minor revisions to solve.py and the verification environment. We will include the complete per-task reference score table in the revised appendix.
Q2. Judge Aggregation and Re-Evaluation Variance
- Aggregation: Three independent Judge evaluations of the same submission are combined via a simple arithmetic mean, (not best-of-3, majority voting, or max-score selection).
- Re-evaluation variance: Across 772 weight-verified historical submissions, median task-score sample variance is 0.051 (mean 0.240), with a single-call task-score SD median of 0.226 points (mean 0.319) and a 3-call mean SD () median of 0.130 points (mean 0.184, P95 0.591) (raw 0–100 Judge subscore median variance 4.10, median SD 2.02). At >=99.9 (685 reconstructible submissions), 98.5% (675/685) yield identical pass/fail decisions across individual Judge calls (within-Judge single-call boundary flip rate 1.5%, i.e., <2% or 10 submissions, distinct from the cross-Judge re-evaluation in MW1).
Q3. Number of Runs and "Valid" Trajectory Selection
Close pre-release versions ran more than 3 times per task during development, whereas the paper reports 1 trajectory per configuration–task pair from completed runs of the frozen final version, never cherry-picked by task score.
Q4. Judge Count Discrepancy (77 vs. 71) and Held-Out Task Distribution
- 77 vs. 71: 71 tasks explicitly specify
JUDGE_MODELandJUDGE_API_ENDPOINTintask.toml, while the remaining 6 tasks (task_072_pt_co_adsorption,task_073_ion_diffusion_lammps,task_075_n2_nevpt2_pes,task_114_he_tumor_infiltration,task_116_eis_equivalent_circuit_analysis,task_117_cp2k_mgo_phonon) default to using the Judge model at runtime (so 77 tasks actually use an LLM Judge). We will add explicit declarations to these 6task.tomlfiles in the updated release. - Held-out task distribution: Together with the 97 released tasks, the 203 held-out tasks span the six domains as shown below (data licensing prevents disclosing further task-level details, and the revised text will focus strictly on the 97 evaluated workflows):
| Domain | Held-Out (203) | Share of 203 | Total with Released 97 (300) |
|---|---|---|---|
| Quantum Chemistry | 42 | 20.69% | 62 |
| Molecular Dynamics | 9 | 4.43% | 25 |
| Materials Characterization | 68 | 33.50% | 90 |
| Analytical Chemistry | 43 | 21.18% | 66 |
| Life Science | 22 | 10.84% | 32 |
| Electrochemistry | 19 | 9.36% | 25 |
| Total | 203 | 100% | 300 |
Thank you again for your detailed review. We will incorporate all of these clarifications and revisions into the updated manuscript.
Thank you for the thorough and constructive review. Below are our point-by-point responses.
Weaknesses
W1. Task Solvability at >=99.9, Threshold Rationale & Sensitivity,
task_008Alignment, and Zero Passes in Electrochemistry1. Task Solvability and Reference Solutions
All 97 released tasks have been verified as solvable:
analysis.py, added during final specification normalization, was missing from the reference packages. We have now syncedanalysis.pyinto these 5 packages so they also reach >=99.9.solve.pyand the verification environment (e.g., ORCA/fixture dependencies or supplementary outputs).2. Why >=99.9 Is the Pass Threshold
We set
task_score >= 99.9as the pass bar to separate high partial credit from complete, reliable workflow delivery:task_116(EIS analysis), the highest-scoring submission (87.62) contradicts both itself and the raw inputs by claiming low-frequency impedance "grows" across scans while also reporting a drop from 9.7 Ω to 6.8 Ω (raw inputs show Re(Z) drops from 9.6964 Ω at 1.585 Hz to 6.7957 Ω, -29.9%). This input-verifiable error shows that high partial scores can mask real delivery failures. While not every deduction invalidates core conclusions, all rubric items form the delivery contract, with their scientific soundness ensured through independent rubric and Judge audits.Therefore, Pass Rate (>=99.9) is the essential metric for evaluating end-to-end scientific workflow completion, rather than relying on high average partial scores.
3. Sensitivity Analysis Across Thresholds
Across 97 tasks × 12 configurations (1,164 historical frozen-verifier evaluations), cumulative pass rates at >=99.9, >=95, and >=90 are: