acceptodds
Withdrawn from ICLR 2027

RTD-Eval and LongChemBench : Dual-View Evaluation of Long-Horizon Agent Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent progress in foundation models has enabled agents to use tools, interact with external environments, and pursue user-specified goals. However, their evaluation remains difficult because short-form benchmarks often fail to capture the multi-stage structure and verification demands of real-world workflows. We introduce LongChemBench, a long-horizon benchmark for computational chemistry, together with Report-Trajectory Dual-View Evaluation (RTD-Eval), a general grading framework for evaluating long-horizon agent tasks. RTD-Eval evaluates both the completeness and accuracy of the final report and the trace-report consistency. We instantiate this framework with 50 long-horizon open-ended computational chemistry tasks. When graded only on the final report, Qwen3.8-Max achieves the highest score, suggesting that it can produce comparatively complete and convincing answers. In the trace-report consistency view, most models exhibit substantial inconsistency, indicating that their final reports often include claims, values, or validation steps that are not fully supported by the actual execution process. This discrepancy highlights the limitation of final-report-only evaluation and motivates the need to evaluate both what an agent reports and how it arrived there.

Primary area:
datasets and benchmarks
Keywords:
evaluation methods, long-horizon, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GLDCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem that “evaluating only final outcomes cannot reflect actual execution capabilities” in long-horizon agents. It proposes the RTD-Eval framework, which evaluates agents from two perspectives: final-report quality and trace–report consistency. It also constructs LongChemBench, a benchmark for long-horizon computational chemistry agents, and evaluates 11 models across a total of 550 task executions. Experiments show that relying solely on final reports may overestimate agent capabilities, while incorporating execution trajectories can improve evaluation reliability.

Critical strengths
  1. The paper accurately identifies shortcomings in current agent evaluation systems, arguing that execution processes, tool calls, and intermediate verification are equally important for long-horizon tasks.

  2. RTD-Eval decomposes agent capabilities into two dimensions: “report quality” and “execution consistency.” The evaluation framework is reasonably designed and can identify cases where “the report appears correct, but the execution process is unreliable.”

Critical weaknesses
  1. The paper’s main contributions focus on the evaluation framework and benchmark design. It has not sufficiently demonstrated a substantial advance over existing methods, and its overall novelty is limited.

  2. The paper relies on GPT-5.6-sol to assess trace–report consistency, which may be affected by the evaluator model’s capabilities and biases. Stability across different judge models requires further validation.

  3. The experiments focus exclusively on computational chemistry agents, and the framework’s effectiveness has not yet been validated on other long-horizon tasks, such as coding agents.

Questions for authors
  1. Is the paper’s contribution to redefine the agent evaluation framework by extending it from “outcome correctness” to “consistency between outcomes and execution evidence”?

  2. Could comparisons across different LLM judge models be added to demonstrate that RTD-Eval does not depend on a single evaluator model, thereby establishing the framework’s robustness?

  3. Could the framework be further extended to other long-horizon agents, such as coding agents, and serve as a training feedback signal to improve agent reliability?

2 comments
AuthorsOpenReview ↗

Response to Reviewer GLDC

  1. Novelty and conceptual contribution (R3.1, R3.Q1)

Our contribution is to distinguish the quality of an agent’s final report from whether its claims are supported by actual execution evidence, rather than to replace outcome-correctness evaluation. A plausible report does not necessarily faithfully describe the calculations, validations, or results produced during execution.

RTD-Eval makes this distinction explicit through separate report-quality and trace–report consistency scores (Section 3). Its practical relevance is illustrated by seven of eleven models changing rank between the two evaluation views (Section 5.2 and Figure 2). We therefore view the contribution as a targeted framework and empirical investigation of reporting fidelity in long-horizon scientific workflows.

  1. Dependence on the evaluator and cross-judge robustness (R3.2, R3.Q2)

Expert validation provides preliminary support for the automated evaluator. Two domain experts applied the same RTD-Eval rubric and scoring formula. On the representative validation task across ten models, automated and expert scores showed a Pearson correlation of 0.936 and a mean absolute difference of 4.37 points (Section 5.3 and Figure 3).

However, this comparison does not establish robustness across judge models or task types. We plan to add comparisons across different LLM judge models in future work to further assess RTD-Eval’s cross-judge robustness. Auditing lengthy execution traces and computational artifacts with additional judges entails further inference and verification costs.

  1. Cross-domain extension and training feedback (R3.3, R3.Q3)

The evaluation principle could extend to other long-horizon agents, including coding agents, by checking reported changes and test outcomes against execution logs and generated artifacts. Discrepancy annotations could also provide diagnostic or training feedback, although their effectiveness for improving agent reliability has not yet been tested.

These extensions require domain-specific tasks, rubrics, expert verification, and additional agent runs. Under our assumed token usage and pricing, running all 11 models on all 50 tasks for a full 12 hours per task would cost approximately USD 26,000 in model API fees alone, excluding expert annotation and computing infrastructure costs. Broader execution and training studies would therefore require substantial additional resources. We plan to explore cross-domain evaluation and training-feedback applications in future work.

Thank you for clarifying. Expert validation provides valuable initial support for automatic evaluation, but has not directly addressed the robustness issue across LLM judgments. The innovation of existing work still needs further clarification. I will maintain my score.

Reviewer_u1FSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RTD-Eval, which assesses long-horizon agents from two separate perspectives: the quality of their final reports and the consistency of those reports with execution traces and computational artifacts. The authors instantiate the framework in LongChemBench, a set of 50 open-ended computational chemistry tasks, and evaluate 11 models across 550 runs. Their analysis finds report–trajectory discrepancies that report-only grading can miss, and shows that incorporating trace evidence changes some model rankings.

Critical strengths
  1. A useful distinction between deliverable quality and evidential support. A scientifically plausible report does not establish that the claimed calculations and checks were performed. Evaluating these properties separately addresses a meaningful limitation of report-only assessment.
  2. Concrete, auditable examples. The Diels–Alder case compares individual report claims with execution evidence, illustrating how unsupported numerical values, incomplete verification, and overstated conclusions can escape report-only grading.
Critical weaknesses
  1. The consistency score may favor sparse reporting. The metric divides the number of detected discrepancies by the number of report sentences (Section 3.3). An agent that omits a required calculation—and makes no claim about it—may avoid a discrepancy. Differences in report length and claim density could therefore affect cross-model comparisons. A high consistency score should not be interpreted as evidence of task completion.
  2. Validation is limited for the ranking conclusions. Expert comparison covers one representative task type; although the reported Pearson correlation is 0.936, the Spearman rank correlation across the 10 paired models is 0.576 (Section 5.3, Figure 3). Each model–task combination is also run only once. These results leave uncertainty about audit accuracy across task types and the stability of the observed ranking changes (Section 6.5).
Questions for authors
  1. How do the consistency scores and model rankings change if discrepancies are normalized by auditable claims rather than sentences? Can the authors quantify the effects of report length, claim density, and omitted required steps?
  2. Can the authors provide expert checks across additional task types, including false-positive and false-negative analyses, and uncertainty estimates or pairwise comparisons for the ranking changes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer u1FS

  1. Sentence-based normalization and reporting sparsity

Sentence normalization expresses discrepancy frequency relative to the number of report sentences, rather than comparing raw discrepancy counts (Section 3.3). We consider sentence count a practical choice because models differ in reporting style: a simple conclusion may be expressed in a much longer sentence by one model than another. Word- or token-based normalization would therefore be more directly affected by verbosity.

However, sentence normalization does not fully eliminate differences in claim density, and a high consistency score does not establish task completion. Omitted required content is assessed through the separate, task-specific report-quality rubric (Section 3.2). The two scores should therefore be interpreted together.

  1. Validation scope, ranking uncertainty, and evaluation costs

We agree that the current expert validation is limited in scope. With one evaluated run per model–task combination, the observed ranking changes should not be interpreted as a guaranteed stable ordering; these limitations are acknowledged in Section 6.5. However, repeated execution is costly. Under our assumed token usage and pricing, running all 11 models on all 50 tasks for a full 12 hours per task would cost approximately USD 26,000 in model API fees alone, excluding expert annotation and computing infrastructure costs.

Broader expert validation also requires substantial manual effort. Each task involves 11 models, and an average execution trace contains 452 interaction turns and 156 tool calls (Section 4.2). Checking reported claims against these traces and artifacts—especially identifying both false positives and missed discrepancies—is labor-intensive.

Thank you for the clarification. I agree that report quality and trace consistency should be interpreted together, and I appreciate the authors’ caution about ranking stability. My remaining concern is the sensitivity of the cross-model results to sentence-based normalization. A claim of comparison and expert checks on more task types would make the ranking evidence stronger.

Reviewer_Vds6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LongChemBench, a benchmark of 50 computational chemistry tasks, and RTD-Eval, a method for scoring consistency with the final report vs. the trace generated by the agent. The authors examine 11 models across the tasks and score consistency against the reports, finding very similar results for consistency across reports. The authors find that post hoc reports tend to misrepresent similar factors across the traces, often overclaiming. Finally, authors spend time discussing how trace-level evaluations are important for holistic evaluation of agentic benchmarks.

Critical strengths
  1. The problem identified by the authors, inconsistency between reports and trace validity, is an important one for long-horizon agentic systems.
  2. The benchmark proposed here seems to include many realistic and relevant tasks for computational chemistry, a challenging domain for agent systems.
  3. The authors ran many models against their benchmark, helping provide insights into model comparisons and sufficient samples to draw meaningful conclusions from their metrics.
Critical weaknesses
  1. There is no discussion in the paper of how LongChemBench was gathered. Were domain experts surveyed for difficult tasks? Were these tasks generated by AI? Without this information, it’s difficult to evaluate the contribution of the benchmark.
  2. The authors don’t release LongChemBench in this paper. This is unacceptable for review of a benchmark, and it’s impossible to evaluate the quality of the contribution without it.
  3. The paper lacks detail in several major sections. First, there is no description of how the experts performed their scoring in Section 5.3, nor is a description of the number of experts provided. Second, no prompts for the judges used for quality evaluation of the paper are given. Third, there is no description of the structures of the traces extracted from the models; the trace structure can be more or less granular depending on if the authors did explicit version control or not.
  4. The overall contribution of the paper is quite incremental. The central novelty of this benchmark is the trace-report consistency metric, but this is a relatively small contribution. The central quality evaluation here is dependent on an LLM judge (which is not described) and an auditing agent scanning the traces. This is a poor proxy for quality due to pitfalls with LLM judges, and a more deterministic metric for success would be more informative.
Questions for authors
  1. How is your dataset different than the other many scientific benchmarks stated in the related work? How is it different than say TerminalBench-Science (https://www.tbench.ai/news/terminal-bench-science-0-1)? It’s fine for the domains to be overlapping, but I don’t see why this benchmark is different or high-quality.
  2. What was the overall correctness of the models on the benchmark? How is correctness scored for the benchmark? The authors seem to only discuss the LLM judge validity rather than the correctness of the models solving these tasks.
  3. How would consistency performance change under different kinds of harnesses?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Vds6

We clarify the benchmark’s provenance, availability, distinctiveness, and evaluation below.

  1. Benchmark construction and availability. LongChemBench contains 50 tasks primarily adapted from undergraduate computational chemistry assignments. All scientific content was designed by domain experts; AI was used only to standardize formatting, not to generate scientific content. Experts reviewed scientific relevance, computational feasibility, difficulty, clarity, and suitability for long-horizon evaluation, removing or revising unsuitable tasks.

  2. Difference from existing scientific benchmarks. The following comparison clarifies its complementary role:

DimensionLongChemBenchTerminal-Bench-Science
Domain scopeFocused computational chemistry workflows, including quantum chemistry and molecular dynamicsWorkflows across multiple scientific disciplines
Task provenanceCourse assignments adapted and screened by domain expertsResearch workflows contributed and reviewed by scientists
Evaluation focusFinal-report quality and trace–report consistency, scored separatelyTask success assessed through artifacts and reproducible task-specific tests
Main analysisUnsupported claims, misreported values, omitted validation, and other report–execution discrepanciesOverall and domain-specific task-resolution performance
Core contributionDual-view evaluation separating deliverable quality from evidential supportA broad, continuously evolving scientific-agent benchmark

LongChemBench complements existing scientific benchmarks by treating reporting fidelity as a separate evaluation dimension. Its quality is supported by expert-designed content and rubrics, explicit task-selection criteria, a shared execution environment, and retained execution evidence.

  1. Model Correctness and Scoring

Model performance is reported in Section 5.2 and Figure 2, where average final-report quality scores range from 21.7 to 46.8 out of 100. As described in Section 3.2, reports are evaluated using expert-defined, task-specific rubrics covering method selection, calculations, validation, and the accuracy and completeness of reported results. Each report is scored independently three times, and the scores are averaged. Appendix B.3 provides a concrete example of the scoring rubric.

We clarify that these are rubric-based quality scores, not binary task-success rates; they should not be interpreted as the percentage of tasks correctly solved. Separately, trace–report consistency measures whether reported claims are supported by execution evidence (Section 3.3), rather than whether the underlying calculations are scientifically correct.

  1. Expert scoring and execution evidence.

Two domain experts evaluated trace–report consistency using the same RTD-Eval rubric. They identified discrepancies between the final report and execution evidence and converted the discrepancy counts into scores using Equation (1), following the evaluation procedure discussed in Section 5.3. Section 4.2 describes the retained execution evidence, including dialogue and tool-call sequences, command exit statuses, generated files, final reports, and environment metadata.

  1. LLM-Based Evaluation

As discussed in Sections 4.1 and 6.1, our open-ended computational chemistry tasks allow multiple valid computational approaches. Deterministic checks can verify specific outputs, but fixed scripts alone cannot fully assess methodological reasoning or whether reported procedures are supported by execution evidence. Rubric-based LLM evaluation is therefore a reasonable approach for these broader dimensions. Sections 3.2–3.3 describe how report quality is assessed using expert-defined, task-specific rubrics, while the auditing agent checks reported claims against traces and computational artifacts.

We nevertheless acknowledge the limitations of LLM judges. Our evaluation complements, rather than replaces, deterministic correctness checks, and trace–report consistency should not be interpreted as a comprehensive measure of scientific task success.

  1. Harness sensitivity

Our long-horizon computational chemistry tasks are costly to execute. Under our current cost assumptions, running all 11 models on all 50 tasks for a full 12 hours per task would cost approximately USD 26,000 in model API fees. Repeating this evaluation under additional agent harnesses would incur substantial additional costs. We have therefore not yet conducted cross-harness experiments and plan to investigate harness sensitivity in future work.

Reviewer_Vds6OpenReview ↗

Thank you, responses still lack adequate detail

Thank you for your response, this helps clarify parts of the paper. However, I still have major concerns about the work, and your provided descriptions still lack the detail needed to release this benchmark. Your descriptions also raise new questions, like are these proposed tasks available publicly on the internet? Would an LLM have been trained on these tasks? Further, do the tasks represent sufficiently-challenging tasks that will not quickly be saturated by new models? More levels of detail on your benchmark description might help answer some of these questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.