- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SciXbench, a benchmark for evaluating LLM agents on long-horizon scientific tasks. It contains 83 expert-curated tasks across seven scientific subdomains, mainly focusing on chemistry. The authors introduce two evaluation components: Dynamic Scientific Rubrics (DSRs), which generate task-specific scoring criteria from expert-reviewed domain frameworks, and Scientific Trace, which records and evaluates intermediate decisions, tool executions, and supporting evidence. Experiments on eight LLMs reveal substantial gaps between final-answer quality and trajectory scores. The paper also examines the effects of different agent harnesses and explores using scientific traces as supervision for retrosynthesis tasks.
Critical strengths
-
The paper studies a relevant problem. Evaluating scientific agents based on both their final answers and intermediate execution processes is useful, especially for tasks involving multiple tools and scientific decisions.
-
The benchmark covers several scientific domains and includes tasks that require multiple steps of computation, analysis, and validation. These tasks go beyond conventional scientific question answering and provide a more realistic setting for evaluating agent capabilities.
-
The authors have made a considerable experimental effort, evaluating eight LLMs and conducting additional experiments on rubric generation, agent harnesses, and trace-based fine-tuning.
Critical weaknesses
-
The technical contributions of DSRs and Scientific Trace are not sufficiently clear. DSRs appear to combine existing practices such as literature-based rubric construction, expert review, and LLM-generated evaluation criteria. Similarly, Scientific Trace mainly records tool calls, intermediate decisions, and execution results, which are already common in agent systems. The paper introduces these components as major contributions but does not clearly explain what distinguishes them from existing approaches.
-
The experiments do not provide sufficient evidence for the benefits of the proposed components. In Table 6, the best DSR configuration achieves 76.03% pairwise agreement, compared with 73.42% for the direct rubric generation baseline. The improvement is relatively small, and it is unclear whether it justifies the additional complexity. For Scientific Trace, the paper does not provide a direct comparison with standard agent logs to show whether it offers better error diagnosis or evaluation.
-
The interpretation of the trajectory evaluation results is not convincing enough. The paper reports considerably higher trajectory scores than final-answer scores for several models. However, the two scores are based on different evaluation criteria, so the difference alone does not necessarily reveal a gap between scientific reasoning and task completion. More evidence is needed to show what the trajectory scores actually measure and whether they can reliably identify scientific errors.
-
The presentation needs considerable improvement, particularly the main figures. Figures 1 and 2 contain many visual elements, but the key technical steps are not clearly explained. Figure 1 also makes it difficult to distinguish the rubric construction process from its validation procedure. More generally, the paper covers several components and experiments, but the connections between them are not always clear. This makes it difficult to understand how the proposed framework works as a whole and what each component contributes.
-
Some experimental details and reported results need clarification. Section 4.1 reports 83 tasks, eight models, and three runs per task, which would correspond to 1,992 runs, but the paper reports only 1,764. In addition, the harness comparison in Table 2 is conducted on a single task, making it difficult to draw general conclusions about the effects of different agent harnesses. These issues raise questions about the completeness of the experimental reporting and the strength of the conclusions.
Questions for authors
-
DSRs and Scientific Trace are presented as two main contributions, but their differences from existing methods are not clear. For DSRs, what is the main methodological difference from directly generating task-specific rubrics using LLMs? For Scientific Trace, what information does it capture beyond standard agent execution logs, and how is that information used to identify scientific errors?
-
In Table 6, the best DSR configuration improves pairwise agreement from 73.42% to 76.03%. Is this difference statistically significant? Given the additional expert-reviewed domain frameworks and multi-agent generation process, what evidence shows that these components are necessary?
-
The paper reports much higher trajectory scores than final-answer scores. Since these scores use different criteria, how can the authors determine whether this gap reflects a genuine difference in model capabilities rather than the scoring design? Has the reliability of trajectory scoring been checked against independent expert assessments of the actual execution processes?
-
The paper describes Scientific Trace as a way to identify errors in scientific reasoning and evidence use. However, most of the evidence comes from case studies. Could the authors provide a systematic evaluation of its error-detection ability, including what types of errors it can identify and how accurately it does so compared with conventional execution logs?
-
Section 4.1 reports 1,764 runs, whereas 83 tasks evaluated on eight models with three repetitions would give 1,992 runs. What accounts for the missing runs? Were any tasks or models excluded, and how were incomplete or failed executions handled in the reported averages?