- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem that “evaluating only final outcomes cannot reflect actual execution capabilities” in long-horizon agents. It proposes the RTD-Eval framework, which evaluates agents from two perspectives: final-report quality and trace–report consistency. It also constructs LongChemBench, a benchmark for long-horizon computational chemistry agents, and evaluates 11 models across a total of 550 task executions. Experiments show that relying solely on final reports may overestimate agent capabilities, while incorporating execution trajectories can improve evaluation reliability.
Critical strengths
-
The paper accurately identifies shortcomings in current agent evaluation systems, arguing that execution processes, tool calls, and intermediate verification are equally important for long-horizon tasks.
-
RTD-Eval decomposes agent capabilities into two dimensions: “report quality” and “execution consistency.” The evaluation framework is reasonably designed and can identify cases where “the report appears correct, but the execution process is unreliable.”
Critical weaknesses
-
The paper’s main contributions focus on the evaluation framework and benchmark design. It has not sufficiently demonstrated a substantial advance over existing methods, and its overall novelty is limited.
-
The paper relies on GPT-5.6-sol to assess trace–report consistency, which may be affected by the evaluator model’s capabilities and biases. Stability across different judge models requires further validation.
-
The experiments focus exclusively on computational chemistry agents, and the framework’s effectiveness has not yet been validated on other long-horizon tasks, such as coding agents.
Questions for authors
-
Is the paper’s contribution to redefine the agent evaluation framework by extending it from “outcome correctness” to “consistency between outcomes and execution evidence”?
-
Could comparisons across different LLM judge models be added to demonstrate that RTD-Eval does not depend on a single evaluator model, thereby establishing the framework’s robustness?
-
Could the framework be further extended to other long-horizon agents, such as coding agents, and serve as a training feedback signal to improve agent reliability?
2 comments
Thank you for clarifying. Expert validation provides valuable initial support for automatic evaluation, but has not directly addressed the robustness issue across LLM judgments. The innovation of existing work still needs further clarification. I will maintain my score.
Response to Reviewer GLDC
Our contribution is to distinguish the quality of an agent’s final report from whether its claims are supported by actual execution evidence, rather than to replace outcome-correctness evaluation. A plausible report does not necessarily faithfully describe the calculations, validations, or results produced during execution.
RTD-Eval makes this distinction explicit through separate report-quality and trace–report consistency scores (Section 3). Its practical relevance is illustrated by seven of eleven models changing rank between the two evaluation views (Section 5.2 and Figure 2). We therefore view the contribution as a targeted framework and empirical investigation of reporting fidelity in long-horizon scientific workflows.
Expert validation provides preliminary support for the automated evaluator. Two domain experts applied the same RTD-Eval rubric and scoring formula. On the representative validation task across ten models, automated and expert scores showed a Pearson correlation of 0.936 and a mean absolute difference of 4.37 points (Section 5.3 and Figure 3).
However, this comparison does not establish robustness across judge models or task types. We plan to add comparisons across different LLM judge models in future work to further assess RTD-Eval’s cross-judge robustness. Auditing lengthy execution traces and computational artifacts with additional judges entails further inference and verification costs.
The evaluation principle could extend to other long-horizon agents, including coding agents, by checking reported changes and test outcomes against execution logs and generated artifacts. Discrepancy annotations could also provide diagnostic or training feedback, although their effectiveness for improving agent reliability has not yet been tested.
These extensions require domain-specific tasks, rubrics, expert verification, and additional agent runs. Under our assumed token usage and pricing, running all 11 models on all 50 tasks for a full 12 hours per task would cost approximately USD 26,000 in model API fees alone, excluding expert annotation and computing infrastructure costs. Broader execution and training studies would therefore require substantial additional resources. We plan to explore cross-domain evaluation and training-feedback applications in future work.