acceptodds
Withdrawn from ICLR 2027

SciXbench exposes the fragility of LLMs in long-horizon scientific problem solving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A plausible scientific answer can conceal a flawed research process. We introduce SciXbench, a benchmark of 83 expert-curated, long-horizon research tasks across seven scientific subdomains, centered on chemistry. Beyond task construction, we make two methodological contributions to scientific evaluation: (1) Dynamic Scientific Rubrics (DSRs). We design a literature-grounded method that builds expert-reviewed subdomain frameworks and uses a multi-agent workflow to derive question-specific scoring criteria, scales, and weights while preserving criterion provenance. (2) Scientific Trace. We develop a non-intrusive approach to recording and evaluating scientific trajectories. It captures native runtime events, tool outputs, and generated artifacts without prescribing agent workflows, and links decisions, observations, and final results through their underlying evidence. DSR-based answer scores and trajectory scores are computed independently and analyzed jointly to diagnose execution errors, misinterpretation, and failures to carry evidence across research stages. On 15 FrontierScience chemistry problems, the selected DSR configuration achieves 76.03% pairwise agreement with expert-rubric judgments. Trace-derived supervision further improves strict top-5 retrosynthesis accuracy by 4.51% on a deduplicated ORDerly subset. SciXbench provides both challenging scientific tasks and a methodology for evaluating the outcomes and processes of scientific problem solving.

Primary area:
datasets and benchmarks
Keywords:
Large language models, Scientific benchmarks, Scientific Trace, Dynamic Scientific Rubrics, Chemistry and life sciences

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3tFbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SciXbench, a benchmark for evaluating LLM agents on long-horizon scientific tasks. It contains 83 expert-curated tasks across seven scientific subdomains, mainly focusing on chemistry. The authors introduce two evaluation components: Dynamic Scientific Rubrics (DSRs), which generate task-specific scoring criteria from expert-reviewed domain frameworks, and Scientific Trace, which records and evaluates intermediate decisions, tool executions, and supporting evidence. Experiments on eight LLMs reveal substantial gaps between final-answer quality and trajectory scores. The paper also examines the effects of different agent harnesses and explores using scientific traces as supervision for retrosynthesis tasks.

Critical strengths
  1. The paper studies a relevant problem. Evaluating scientific agents based on both their final answers and intermediate execution processes is useful, especially for tasks involving multiple tools and scientific decisions.

  2. The benchmark covers several scientific domains and includes tasks that require multiple steps of computation, analysis, and validation. These tasks go beyond conventional scientific question answering and provide a more realistic setting for evaluating agent capabilities.

  3. The authors have made a considerable experimental effort, evaluating eight LLMs and conducting additional experiments on rubric generation, agent harnesses, and trace-based fine-tuning.

Critical weaknesses
  1. The technical contributions of DSRs and Scientific Trace are not sufficiently clear. DSRs appear to combine existing practices such as literature-based rubric construction, expert review, and LLM-generated evaluation criteria. Similarly, Scientific Trace mainly records tool calls, intermediate decisions, and execution results, which are already common in agent systems. The paper introduces these components as major contributions but does not clearly explain what distinguishes them from existing approaches.

  2. The experiments do not provide sufficient evidence for the benefits of the proposed components. In Table 6, the best DSR configuration achieves 76.03% pairwise agreement, compared with 73.42% for the direct rubric generation baseline. The improvement is relatively small, and it is unclear whether it justifies the additional complexity. For Scientific Trace, the paper does not provide a direct comparison with standard agent logs to show whether it offers better error diagnosis or evaluation.

  3. The interpretation of the trajectory evaluation results is not convincing enough. The paper reports considerably higher trajectory scores than final-answer scores for several models. However, the two scores are based on different evaluation criteria, so the difference alone does not necessarily reveal a gap between scientific reasoning and task completion. More evidence is needed to show what the trajectory scores actually measure and whether they can reliably identify scientific errors.

  4. The presentation needs considerable improvement, particularly the main figures. Figures 1 and 2 contain many visual elements, but the key technical steps are not clearly explained. Figure 1 also makes it difficult to distinguish the rubric construction process from its validation procedure. More generally, the paper covers several components and experiments, but the connections between them are not always clear. This makes it difficult to understand how the proposed framework works as a whole and what each component contributes.

  5. Some experimental details and reported results need clarification. Section 4.1 reports 83 tasks, eight models, and three runs per task, which would correspond to 1,992 runs, but the paper reports only 1,764. In addition, the harness comparison in Table 2 is conducted on a single task, making it difficult to draw general conclusions about the effects of different agent harnesses. These issues raise questions about the completeness of the experimental reporting and the strength of the conclusions.

Questions for authors
  1. DSRs and Scientific Trace are presented as two main contributions, but their differences from existing methods are not clear. For DSRs, what is the main methodological difference from directly generating task-specific rubrics using LLMs? For Scientific Trace, what information does it capture beyond standard agent execution logs, and how is that information used to identify scientific errors?

  2. In Table 6, the best DSR configuration improves pairwise agreement from 73.42% to 76.03%. Is this difference statistically significant? Given the additional expert-reviewed domain frameworks and multi-agent generation process, what evidence shows that these components are necessary?

  3. The paper reports much higher trajectory scores than final-answer scores. Since these scores use different criteria, how can the authors determine whether this gap reflects a genuine difference in model capabilities rather than the scoring design? Has the reliability of trajectory scoring been checked against independent expert assessments of the actual execution processes?

  4. The paper describes Scientific Trace as a way to identify errors in scientific reasoning and evidence use. However, most of the evidence comes from case studies. Could the authors provide a systematic evaluation of its error-detection ability, including what types of errors it can identify and how accurately it does so compared with conventional execution logs?

  5. Section 4.1 reports 1,764 runs, whereas 83 tasks evaluated on eight models with three repetitions would give 1,992 runs. What accounts for the missing runs? Were any tasks or models excluded, and how were incomplete or failed executions handled in the reported averages?

Reviewer_LZYhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SciXbench, a benchmark of 83 expert-curated, long-horizon scientific tasks centered on chemistry. 45 of the tasks are retrosynthesis, built from three FDA-approved drugs and their analogues, and the rest cover computational chemistry. It proposes two evaluation methods: Dynamic Scientific Rubrics (DSRs), which use LLM agents to generate question-specific rubrics from literature-grounded subdomain frameworks, and Scientific Trace, which logs the agent's runtime events and scores the trajectory with a fixed trace rubric. Answers and traces are scored by two LLM judges. Evaluation results show relatively low "answer scores".

Critical strengths
  • The tasks are multi-stage research problems with real data and tools, curated by domain experts. Benchmarks like this can be useful for the community.
  • Scoring the process separately from the final answer make sense to me. The observation that answer scores and trace scores diverges is interesting.
Critical weaknesses
  1. The lack of justification and calibration of the LLM judges. All evaluation depends on two LLM judges, and the only reason given for choosing them is that they come from different providers than the evaluated models (page 7). There is limited discussion of whether they are aligned with human expert judgement on these tasks (for example, correlation with human experts).
  2. Following 1, the rubric design is not well explained, and its validation is weak. The answer rubric for each question is written by LLM. But there seems to be limited explanations on how domain expert would rate these rubrics.
  3. The trace score does not seem very discriminative (Figure 3). And there is limited understanding on why this is the case. How to understand that Fable 5.1 has the third highest answer score but the lowest trace score?
  4. The figures are hard to read due to small font sizes.
Questions for authors

See weaknesses.

Reviewer_nAv1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SciXbench has 83 expert-curated, multi-stage tasks (seven subdomains, 45 retrosynthesis) using real data and tools. LLM agents turn expert frameworks into per-question rubrics (DSRs, 76.03% expert agreement on 15 problems), and LLM judges score trajectories (Scientific Trace). Eight models score below 60 on answers versus 68.9–86.8 on traces, and trace SFT adds 4.51 ORDerly top-5 points.

Critical strengths

Multi-stage tasks come with detailed trace logs: 43.0% of 3,002 genomics tool steps were non-advancing rework or re-checks, showing where agents stall.

The rubric check tests tie-aware agreement against expert rubrics with no-framework and wrong-domain controls, and its item audit caught an expert-rubric unit error.

Critical weaknesses

Main comparisons are not on equal footing, so rankings are hard to trust. Rules for Claude-fable-5-1 refusals conflict across pp. 7, 15, and 44 (unreported, replaced, or an upper bound), while Glm-5.3 used low reasoning effort and 65,535-token caps versus Claude-fable's 8,192 (pp. 20, 45). Fig. 3's harness is unnamed, yet harness choice flips rankings (Tables 2–3).

DSR and trace scores lack broad checks, as DSRs are tested against humans only on 15 FrontierScience items where a no-framework control nearly matches (73.42% vs 76.03%, Table 6). In retrosynthesis, the answer rubric draws 80/100 points from the trace (pp. 50–51), blurring answer / trace independence, and ScoreST's fixed rubric (p. 112) conflicts with p. 7 without human checks.

Minor:

Run counts conflict (1,764 vs 8x83x3 = 1,992, p. 7), and missing variance, CIs, and judge agreement obscure small gaps. Trace-to-SFT is one run with no same-size control or target-grouped split stated (Table 7), so the 4.51-point gain may reflect size or overlap.

Uncited prior art (not a score factor): BiomniBench (2026) and Agent-as-a-Judge (2024) score trajectories, and Rubrics as Rewards (2025) builds instance rubrics.

Questions for authors

For each Fig. 3 model, what were the harness, output cap, reasoning setting and refusal treatment? Does the ranking hold under one consistent refusal policy?

Which rubric produced each domain's answer score: DSR-instantiated, or fixed like Table 24 (80/100 points trace-based)? How independent are answer and trace scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.