acceptodds
Under review as a conference paper at ICLR 2027

PaperEvaluation: Benchmarking Agent-Generated Scientific Papers with Execution Evidence

Abstract

Large language models are increasingly capable of performing complex, long-horizon research tasks. However, existing scientific benchmarks provide limited coverage of end-to-end workflows from data analysis and experimental validation to complete manuscript generation, and lack targeted evaluation that connects research execution to the final manuscript. We introduce PaperEvaluation, a benchmark for evaluating agent-generated scientific papers using execution evidence. The benchmark comprises 80 research task packages derived from published papers across 13 research domains. Each task provides a research question, the necessary data, and a controlled execution environment while withholding the source paper and its conclusions. Agents must independently conduct the research and produce a complete manuscript. To evaluate this long-horizon research process, we propose R3Eval, a framework with three dimensions: Research Completeness, Reporting Rigor, and Reasoning Alignment. Research Completeness assesses whether the required scientific work is supported by execution evidence. Reporting Rigor examines whether the manuscript accurately reports its methods and findings. Reasoning Alignment compares the generated and source papers for functional alignment of their scientific arguments. Finally, we evaluate 14 agent systems based on PaperEvaluation, revealing distinct performance profiles across these dimensions through an evaluation of long-horizon research that connects execution to the final manuscript.

Primary area:
datasets and benchmarks
Keywords:
Scientific Agents, Long-Horizon Agent Evaluation, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iMz3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PaperEvaluation, a benchmark for evaluating agent-generated scientific papers. It contains 80 research tasks derived from published papers across 13 domains. The authors propose an EvidenceFirst Scaffold to guide agents through research planning, execution, validation, and manuscript writing. They also introduce R3Eval, which evaluates Research Completeness (RC), Reporting Rigor (RR), and Reasoning Alignment (RA) using execution traces, analytical artifacts, and final manuscripts. Experiments on 14 Model–Harness systems are conducted to compare their performance.

Critical strengths
  • An interesting and timely problem. Evaluating the scientific research capabilities of AI agents is becoming increasingly important. I appreciate the attempt to go beyond evaluating final manuscripts and consider the actual research process.

  • A reasonably comprehensive evaluation framework. The combination of execution traces, analytical artifacts, and manuscripts is a useful direction. Separating RC, RR, and RA also provides different perspectives on agent performance.

  • Extensive experimental effort. The authors evaluate 14 Model–Harness systems and include expert agreement analysis, ablation studies, and controlled error injection experiments. These experiments provide some useful insights into the proposed evaluation framework.

Critical weaknesses

1. Concerns about benchmark validity.

I am concerned that the reported scores may not reflect the models' independent research capabilities. I checked the appendix and found that 62 out of 80 source papers were published before 2025. Some models may have already encountered these papers during pretraining, potentially inflating their scores. The authors briefly acknowledge this issue but provide no contamination analysis.

Also, the benchmark already seems close to saturation (e.g., GPT-6 achieves 94.46 RC and 98.86 RA), raising concerns about its future discriminative power. The uneven distribution of 80 tasks across 13 domains also makes it difficult to draw general conclusions about autonomous research capabilities.

2. Unclear task formulation and missing examples.

I had a hard time understanding what an actual task looks like. The authors define Q, D, E, O, T, F, and M, but a few real examples of complete task packages, including inputs and outputs, would be much more helpful.

More importantly, the exact Research Question matters a lot. Take ResNet as an example. Is the question "How can we address the degradation problem in deep networks?" or "Can shortcut connections improve the training of deeper networks?" Both are valid questions, but they test very different abilities. The latter already reveals much of the original idea.

The authors mention preserving "essential scientific mechanisms", but how much of the original solution is actually revealed? And what exactly is the reference answer? Without real task examples, it is hard to tell what the benchmark is actually measuring.

3. The name is somewhat misleading.

"PaperEvaluation" sounds like a framework for evaluating scientific papers in general. However, this work mainly evaluates agent-generated research using execution traces and artifacts. I suggest considering a name that includes "Trace" to better reflect its focus.

4. Minor presentation issue.

There seems to be an odd tau annotation in Figure 3. I suggest proofreading the figures and notation more carefully.

5. Several scoring rules seem ad hoc and insufficiently defined.

I find some parts of R3Eval unnecessarily complicated. For example, how exactly is the RC cap function in Eq. (3) calculated? In Eq. (7), what counts as a critical reasoning-chain failure, and what numerical cap is applied?

There is also an inconsistency: Section 3.4.1 defines stage credits as 0, 0.5, or 1, while Appendix C.1 allows any value between 0 and 1. Which one is actually used?

The RC calculation also involves several weights, caps, and applicability rules, plus an additional obligation-coverage measure. I am not convinced that all this complexity is necessary or well justified.

ICLR welcomes simple and insightful methodological contributions. Adding more formulas and heuristic rules does not necessarily make a method more rigorous. I would prefer clearer definitions and a sensitivity analysis to justify these design choices.

Questions for authors
  1. Data contamination: Given that 62 of the 80 source papers were published before 2025, how do the authors rule out the possibility that the evaluated models have already encountered these studies during pretraining? Could the authors provide a contamination analysis or report results separately for older and more recent papers?

  2. Actual task inputs: Could the authors provide several complete examples of the actual agent-visible task packages, especially the Research Questions and mechanism descriptions? How do the authors ensure that these inputs do not reveal the key methodological contributions of the source papers?

  3. Reference answers: What exactly is considered the reference answer for each task? How does R3Eval handle scientifically valid solutions that differ substantially from the original paper?

  4. Scoring rules: Could the authors clarify how the score caps in Eq. (3) and Eq. (7) are determined, and resolve the inconsistency in RC stage credit definitions between Section 3.4.1 and Appendix C.1? How sensitive are the reported rankings to these heuristic scoring choices?

  5. Benchmark saturation: Given the near-saturated RA scores, how do the authors expect PaperEvaluation to remain useful for evaluating stronger future models?

5 comments
AuthorsOpenReview ↗

Response to Reviewer iMz3 — Part 1

Thank you for your review. Please find our responses below. We have submitted the revised PDF, with changes and additions highlighted in blue.

1. Benchmark validity and interpretation of results

Thank you for your suggestions on benchmark validity. We address each concern below.

1.1 Source-paper publication year

To address your concern that models may have encountered older source papers, we group tasks by source-paper publication year: before 2025 and 2025 or later. We compare GPT-5.6 Sol and Claude-Sonnet-4.6 on the scored tasks in each group. This grouping addresses your request for a direct comparison across publication periods while keeping the evaluation criteria the same.

ModelPublication yearTasksRCRRRA
GPT-5.6 SolBefore 20256291.49687.03297.438
GPT-5.6 Sol2025 or later1890.97185.44497.181
Claude-Sonnet-4.6Before 20256281.0369.72684.196
Claude-Sonnet-4.62025 or later1877.8366.38984.007

GPT-5.6 Sol scores higher on RC, RR, and RA in both groups. These axes assess execution evidence, reporting accuracy, and inferential validity, respectively. Even a model familiar with the source study must therefore perform and validate the research required by the current task. Scoring rests on verifiable research work, and the performance difference between these two systems persists across the year groups.

Year-group results: Supplement §D.5.7 (Table S23); related limitations: §E. The end of main-text §4.9 points to Appendix D.

1.2 Saturation of the benchmark's discriminative ability

We understand your concern about distinguishing stronger systems in the future. Table 5 in the revision (Table 4 in the original submission) constructs flawed variants of the same submissions. Removing execution and validation evidence reduces RC by 13.07 and 10.08 points on average, respectively; corrupting reasoning reduces RA by 18.59 points, while language-only changes have small effects. This shows that the current scores respond to specific research defects, and high scores require corresponding evidence and valid arguments. These comparisons show why high overall scores and sensitivity to research failures are compatible: scores drop when the supporting work or reasoning is degraded, even though the task itself is unchanged.

We also agree that rapidly advancing agent capabilities raise the requirements for continued discrimination. We are developing V2 with tasks allowing more open methodological choices, requiring more validation conditions, and demanding more analyses across datasets. The aim is to evaluate more independent research decisions and complex validation, beyond increasing the number of similar tasks.

Revisions: main text §§4.2 (Table 2), 4.7 (Table 5); expansion of task numbers and types is discussed in Supplement §E.

1.3 Domain imbalance

We agree with your concern that domains with more tasks may dominate overall conclusions. We therefore reanalyzed results by domain and show two agents' performance in three selected domains below. This breakdown makes variation across research settings visible. Readers can examine whether a system difference appears in domains with different task counts and identify which evaluation dimensions vary with the domain.

DomainTasksModelRCRRRA
Social sciences/economics13GPT-5.6 Sol91.42088.69297.762
Social sciences/economics13Claude-Sonnet-4.679.77168.53884.966
Bioinformatics5GPT-5.6 Sol84.83679.40097.734
Bioinformatics5Claude-Sonnet-4.672.21066.60081.310
Energy1GPT-5.6 Sol96.54088.000100.000
Energy1Claude-Sonnet-4.687.50078.00087.050

GPT-5.6 Sol outperforms Claude-Sonnet-4.6 in all three domains despite their different task counts, while both have lower completion and reporting scores in bioinformatics. Evaluation thus captures system differences and challenges posed by different research requirements. R3Eval uses common evaluation principles while retaining each task's scientific requirements: RC checks required work, RR checks reporting against artifacts, and RA checks whether evidence supports conclusions. Cross-domain comparisons therefore use task-specific scientific obligations and evidence relationships, rather than directly comparing raw performance metrics from different research questions.

Domain results: Supplement §D.5.8 (Table S24); the end of main-text §4.9 points to Appendix D. Domain distribution and cross-task aggregation are described in main-text §3.2 (Figure 2) and Supplement §D.4, respectively.

AuthorsOpenReview ↗

Response to Reviewer iMz3 — Part 2

2. Task inputs, method disclosure, and evaluation references

Thank you for your suggestions on interpretability. We use the RNA-seq task to illustrate the inputs and outputs.

2.1 A complete task input/output example

The original research question for the RNA-seq differential-expression task is:

How robustly do RNA-seq differential-expression procedures control false discoveries and recover known signals across a real ERCC spike-in benchmark and simulations spanning sample size, DE prevalence, imbalance, dispersion and outlier regimes?

The actual inputs and candidate outputs correspond to the notation in §3.1 as follows:

ItemActual content
Q: Research questionCompare differential-expression methods in false-discovery control, recovery of known signals, and robustness across conditions
D: DataFrozen SEQC, KIRC, and Bottomly count matrices, sample metadata, ERCC ground truth, and simulation protocols
E: EnvironmentR/Python environment executable on CPUs
O: Submission requirementsData audits, method versions, metrics, sensitivity analyses, figures, tables, and manuscript
T: Execution traceExecution records for audit, analysis, and summary scripts
F: Analytical artifactsIndividual metrics, ERCC predictions, and a manifest recording simulation seeds and conditions
M: Final manuscriptA complete report of research methods, results, and interpretations

The input fields describe what the agent is given and required to deliver; the output fields show what it actually executed, produced, and reported. This distinction makes the task definition and the evidence used to assess it visible in the same example.

One actual submission ran three method families and 1,280 simulations with recorded seeds, producing an eight-page manuscript. Across 63 evaluable ERCC controls, count score and Wilcoxon achieved ROC AUCs of 0.903 and 0.989, and realized FDPs of 0.213 and 0.078, respectively, matching the manuscript's reported values. This provides a verifiable connection from visible inputs through execution to reported results.

Revisions: main text §3.1; Supplement §A.6 (Tables S4–S6: task inputs, mechanisms, and output files; the paragraph after Table S5 adds actual submission results).

2.2 Disclosure of source methods

We agree with the distinction in your ResNet example: specifying a mechanism changes what remains for the agent to discover. Our task descriptions therefore make the supplied mechanisms and the remaining research choices explicit. Tasks retain different degrees of research choice.

The RNA-seq task specifies count score, log-CPM Welch t, and Wilcoxon, together with BH correction, q<0.10q<0.10, and some simulation conditions. It tests whether agents correctly implement, validate, and interpret results under specified mechanisms. The task on player skin tone and referee red cards instead supplies an association question and constraints such as repeated observations, leaving models, covariates, and sensitivity analyses to the agent. It emphasizes research design and analytical choices.

This design covers two aspects of research capability: more prescribed tasks support comparisons of execution and validation rigor, while more open tasks test independent, scientifically justified analytical choices. Necessary scientific constraints keep alternative approaches focused on the same question. Explicitly stating the supplied mechanisms and remaining choices clarifies what each task measures.

Revisions: main text §§3.1–3.2; Supplement §A.6 (Tables S5, S6: RNA-seq and red-card task comparison).

2.3 What the reference answer is and how valid alternatives are handled

The reference consists of scientific obligations and reasoning references frozen before scoring, used to assess research completion and argument validity. AqSolDB illustrates the distinction:

  • Valid alternative: a submission reconstructs groups from released fields instead of rerunning an external service. Row-level artifacts still validate the consistency of the released rules, so the design receives functional-equivalence credit.
  • Reasoning error: another submission computes group-wise RMSE but interprets between-group differences as an underlying noise mechanism. The artifacts receive credit, while the inference exceeds their evidential support.

Alternatives receive credit for preserving the required scientific function; deductions identify missing evidence or invalid inferences. See our response to Reviewer KL3d, §2.2, for the local effects of credit mapping, and Reviewer pA76, §2.1, for detailed case judgments.

Revisions: main text §§3.1, 3.4.1–3.4.3; Supplement §§C.1, C.3 (Table S15: alternative cases and judgment criteria).

AuthorsOpenReview ↗

Response to Reviewer iMz3 — Part 3

3. Definitions and rationale for scoring rules

3.1 Eq. (3): RC caps

RC caps constrain credit for individual scientific obligations. We first aggregate decision, execution, validation, and reporting credits, then apply the cap determined by evidence conditions. A proposed and reported analysis without execution evidence for training and comparison should receive limited credit even if other stages score positively. For example, stage credits (1,0,0.5,1)(1,0,0.5,1) yield a weighted value of 0.45, capped at 0.25 because execution credit is zero. Available but unattempted obligations receive zero; missing artifact evidence imposes a 0.50 cap. Failed adaptations and substitutions that incompletely preserve the evidence function are capped at 0.25 and 0.35, respectively. When multiple conditions apply, the strictest cap is used.

High RC requires completion of the necessary research: planning and reporting earn credit but cannot compensate for missing essential execution evidence. Full trigger conditions and worked examples are provided in the Supplement.

Revisions: main text §3.4.1, Eq. (3); Supplement §C.1 (Tables S9–S11: stage credits, obligation states, and caps).

3.2 Eq. (7): Critical-chain failure

Critical chains are marked in the frozen rubric before scoring and represent arguments supporting the main conclusions. Suppose a submission's weighted RA is 85, but a critical chain scores 0.50, below the 0.55 threshold: final RA is capped at 60. At least two critical chains below the threshold impose a cap of 50. If any critical chain's analysis operation, evidence pattern, or inference is contradicted, the cap is 40. The strictest applicable cap is used.

A manuscript may complete many analyses yet base its main conclusion on an inference contradicting core evidence. High scores elsewhere should not give that argument a high overall RA. These caps make the reliability of central conclusions constrain the overall assessment, preventing numerous local alignments from concealing a serious flaw in the core argument.

Revisions: main text §3.4.3, Eq. (7); Supplement §C.3 (Table S14: critical chains, caps, and examples).

3.3 Stage credit, applicability, and coverage

Thank you for identifying this oversight. Stage credits are so∈{0,0.5,1}4s_o\in\{0,0.5,1\}^4, representing absent, partial, or sufficient evidence; weighted obligation credit can take other fractional values. We have aligned Supplement §C.1 and main-text §3.4.1 with this discrete definition.

Applicability includes available but unattempted obligations in the denominator with zero credit, and excludes genuinely unavailable, prohibited, or out-of-scope obligations. Coverage is the weighted share of assigned nonoptional obligations that are applicable, indicating how much of the task RC covers. Stage credits measure evidential sufficiency, applicability determines the work to assess, and coverage explains the score's scope.

Revisions: main text §3.4.1; Supplement §C.1 (Tables S9, S10 and the applicability/coverage formulas and explanations).

3.4 Do weights and caps determine the system comparison?

To address your sensitivity question, we fix submissions and atomic judgments while changing weights and caps separately.

ParameterAlternative settingMax. absolute score differenceMin. ρ / τ-b
Equal weightsEach parameter group separately3.8630.9879 / 0.9556
Local ±10%Multiply one weight by 0.9 or 1.1; normalize0.4961.0000 / 1.0000
Local ±20%Multiply one weight by 0.8 or 1.2; normalize1.0250.9879 / 0.9556
RC credit caps0.25/0.35/0.50, each ±0.050.1880.9879 / 0.9556
RA critical-chain threshold0.55 to 0.50 or 0.601.6050.9758 / 0.9111
RA score caps40/50/60, each ±5 points0.7531.0000 / 1.0000
Remove capsRemove RC or RA caps separately5.7460.9636 / 0.9111

Equal weights and local perturbations preserve high ranking agreement. Changes to RC caps, the RA critical-chain threshold, or RA caps produce limited local movements. The main system comparison does not depend on one precise weight or threshold; relative performance is robust across the tested reasonable alternative settings.

Revisions: Supplement pointer at the end of main-text §3.4; Supplement §§D.3 (Table S17), D.5.6 (Table S21).

4. Title and figures

Thank you for your suggestion. We intend to adopt PaperTrace: Benchmarking Agent-Generated Scientific Research with Execution Evidence as the title. This wording makes the role of execution evidence explicit. The submission title cannot be changed during the rebuttal phase, so we have not yet made this update. We sincerely apologize for this.

We have also enlarged some text in Figures 2–3 and corrected the rendering of τ in Figure 3. Thank you for your careful attention.

Thanks again, looking forward to further discussions with you.

Reviewer_pA76OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Evaluating agent-written scientific papers requires checking whether their claims are supported by experiments. PaperEvaluation provides 80 tasks across 13 domains and scores traces, artifacts, and manuscripts using three R3Eval dimensions. Across 14 systems, GPT-6 scores highest on all dimensions, while full execution evidence reduces research-completeness MAE from 15.3 to 3.4.

Critical strengths
  1. The paper studies an important problem: an agent may write a complete-looking paper without completing its experiments or supporting its conclusions. Checking the links between experiment records, analysis files, and the final paper has practical value.

  2. The paper reads smoothly and is easy to understand overall.

  3. The paper evaluates 14 systems and includes expert comparisons, repeated scoring, and controlled error tests.

Critical weaknesses
  1. The experiments do not isolate the benefit of the scoring method itself. Table 3 changes the materials available to the evaluator rather than comparing different scoring methods. It shows that experiment records help, but not whether R3Eval performs better than other scoring methods given the same materials.

  2. Section 3.4.3 allows alternative methods but does not concretely show how functional alignment is judged. Could the authors explain how the evaluator distinguishes a valid difference in design or conclusion from an actual reasoning error?

  3. The difference from earlier work needs to be more specific. MLR-Bench already uses historical information and execution logs to check consistency between research processes and outputs. While the cross-field tasks and three separate scores appear promising, could the authors clarify which problems these designs are able to identify that are not captured by existing verification mechanisms?

  4. Minor: (1) Some text in Figures 2 and 3 is too small and hard to see. (2) The \tau in Figure 3 is not rendered correctly as the Greek letter τ\tau.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer pA76 — Part 1

Thank you for your review. Please find our responses below. We have submitted the revised PDF, with changes and additions highlighted in blue.

1. Comparing scoring methods under identical evidence

We agree with your assessment: Table 3 establishes the value of execution evidence, while the scoring method itself requires a direct comparison under identical evidence. We fix submissions, expert reference scores, Judge settings, and task information, and compare a pointwise rubric with full R3Eval. A receives only the manuscript; B and C receive the same traces, analytical artifacts, and manuscript. The difference between B and C therefore directly reflects the scoring method.

The pointwise rubric uses the expert assessment criteria to score individual characteristics directly, without constructing reasoning chains. R3Eval connects the evidence to required work and the relations supporting each conclusion. This comparison therefore tests the value of structured assessment while holding the available materials fixed.

For each axis, we report mean absolute error (MAE) in points on the 0–100 scale: the average absolute difference between automated and expert scores. Lower values indicate closer agreement with experts.

SettingScoring methodRC MAERR MAERA MAE
AManuscript + pointwise rubric15.411.612.3
BFull evidence + pointwise rubric10.510.111.2
CFull R3Eval3.44.86.2

Full R3Eval is closer to expert judgments on all three axes under identical evidence. Organizing evidence around task obligations and reasoning chains, and making structured judgments, improves evaluation accuracy with the same inputs.

Revisions: main text §§4.5 (Table 3), 4.6 (Table 4); Supplement §§D.5.3, D.5.5 (Table S20).

2. Distinguishing valid alternatives from reasoning errors

Thank you for this suggestion. We understand that your concern is how the evaluator distinguishes valid differences in design or conclusions from actual reasoning errors. We explain this through concrete cases below.

2.1 Criteria and actual cases

When a design or conclusion differs, we check whether the submission answers the same research question, preserves the necessary evidence function, and supports its conclusions and their scope with actual evidence. The rubric, frozen before scoring, records reasoning components, adjacent relations, input visibility, and permitted alternatives. This lets the evaluator identify which scientific functions are preserved and where the argument has gaps.

Two AqSolDB cases illustrate the distinction:

  • Valid design difference. One submission reconstructs reliability groups from released fields instead of calling an external service again. Row-level comparison artifacts support the conclusion that the grouping rules are consistently implemented in the released data. The design is therefore judged functional_equivalent, with its evidence and conclusion receiving credit. Acceptance rests on preserving the scientific function of validating released-data consistency.
  • Actual reasoning defect. Another submission computes group-wise RMSE, so the evaluator judges the analysis-to-evidence relation aligned. However, it interprets between-group differences as an underlying noise mechanism, exceeding the noncausal scope supported by those results. This boundary is judged contradicted. The deduction identifies a specific gap between evidence and interpretation.

These cases separate two questions that your comment brings into focus: whether an alternative preserves the required scientific function, and whether the resulting claim is warranted by its evidence. An analysis can be correctly executed while its interpretation goes beyond what the results support. Judging these steps separately lets us credit the completed analysis and identify the exact inference that needs correction.

Valid differences receive credit for preserved scientific function and evidential support; reasoning errors are located through missing evidence or invalid inferences. Even when results differ from the source study, the question is whether the submission's own evidence supports its conclusions.

Revisions: main text §§3.4.3, 4.9; Supplement §C.3 (Table S15, cases and judgment criteria).

2.2 Credit for alternative paths

We also examine how alternative paths affect numerical scores. Giving full credit to functional equivalence increases credit for affected submissions while leaving current system rankings unchanged. Local credit differences can thus coexist with a robust overall comparison. The rationale and results are detailed in our response to Reviewer KL3d, §2.2.

Revisions: main text §3.4.3; Supplement §§D.3 (Table S17), D.5.6 (Table S22).

AuthorsOpenReview ↗

Response to Reviewer pA76 — Part 2

3. Specific differences from existing benchmarks

Thank you for this suggestion. We agree that MLR-Bench already uses historical information and execution logs to check consistency between the research process and its outputs. We therefore clarify which additional judgments are enabled by frozen scientific obligations and three separate scores. The task and evaluation differences are summarized below.

BenchmarkTask and evaluation focusCorresponding PaperEvaluation design
MLR-BenchOpen-ended machine learning research; staged, multidimensional evaluation using historical information and execution logsFrozen task obligations and source reasoning chains support separate diagnoses of completion, reporting, and reasoning
PaperBenchProvides the source paper and requires implementation and replication of its empirical contributions; evaluation uses independent execution and a hierarchical rubricProvides research questions, data, and mechanism constraints while withholding the source paper; evaluates the evidence and complete manuscript produced by the agent
CORE-BenchComputational reproduction using code and data from published studiesStarts from a visible research question and covers analysis, validation, the manuscript, and its argument
NatureBenchCross-disciplinary scientific tasks; quantitative comparison with published state-of-the-art results in standardized environmentsAssesses completion of scientific obligations, reporting of results, and the connection between evidence and conclusions

Your question concerns the evaluator's ability to locate a scientific failure within a research workflow. A common reference for required work and separate checks of reporting and inference serve different roles here. The frozen obligations specify what must be supported; the three axes identify how the submission meets or fails those requirements.

The following cases more directly answer your question about the additional value of the verification mechanism:

  1. The process and output agree, but required work is missing. If both the plan and manuscript omit a robustness check required by the task, they may remain mutually consistent. We freeze task obligations from the visible research question and scientific constraints, so that check still enters the RC assessment. This independent reference detects required work omitted from both the agent's plan and report. An agent can choose how to carry out the validation, but omitting it changes whether the scientific requirement has been fulfilled.
  2. The work is complete, but its reporting is inaccurate. In the Qwen3.8 case, artifacts support the core analyses, but the manuscript inconsistently reports mean and median LogS across different locations. RR continues the goal of checking consistency between process and output. The three-axis design further distinguishes this reporting error from the completed work recognized by RC, identifying reporting as the part requiring correction rather than rerunning the analysis. The distinction between mean and median matters because it changes which feature of the observed results the manuscript describes.
  3. The analysis and numbers are real, but the conclusion exceeds the evidence. In the AqSolDB case, the submission computes group-wise RMSE but interprets differences as an underlying noise mechanism. RA uses frozen functional reasoning chains to examine the analysis, evidence, and their connection to conclusions separately. The genuine analysis receives credit, while the unsupported interpretation is located at the relevant inferential step. A measured difference establishes a pattern in the data; explaining the mechanism that generated that pattern requires the corresponding evidential support.

R3Eval adds a task-obligation reference independent of the agent's plan and makes completion, reporting accuracy, and inferential validity separately checkable. This detects necessary work omitted from mutually consistent plans and manuscripts, distinguishes reporting errors from inferential defects, and identifies where the research workflow needs improvement. The shared evaluation principles make these diagnoses comparable across tasks while preserving the scientific obligations specific to each study.

Revisions: main text §2.1, Table 1, §§3.4, 4.9; Supplement §§C.1–C.3 (evaluation rules and cases, Table S15).

4. Figures and notation

Thank you for checking these details. We have enlarged some of the text in Figures 3 and corrected the rendering of \tau as the Greek letter τ in Figure 3.

Thank you again and I look forward to our further discussion.

Reviewer_KL3dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PaperEvaluation introduces 80 research tasks across 13 domains and evaluates 14 agent configurations. Its R3Eval framework combines execution traces, analytical artifacts, and manuscripts to assess research completeness, reporting rigor, and reasoning alignment with source studies.

Critical strengths
  • Evaluating manuscripts alongside execution evidence addresses a meaningful problem: a convincing paper does not necessarily demonstrate that the reported research was properly conducted.

  • The evidence ablation and controlled error injections provide useful initial validation. Adding execution evidence substantially reduces scoring error against expert assessments, particularly for research completeness.

Critical weaknesses
  • The methodological contribution is incremental. The novelty mainly lies in integrating task packages and evaluation dimensions.

  • The validity of Reasoning Alignment is questionable. Table S11 assigns lower credit to functionally equivalent reasoning than to aligned reasoning (0.85 versus 1.00), potentially favoring conformity to the source study over equally valid alternative approaches.

  • Key scoring choices lack sufficient justification. The paper specifies weights and credit mappings but provides little justification or sensitivity analysis. It is unclear whether system rankings remain stable under reasonable alternative settings.

Questions for authors
  • How does R3Eval compare with simpler rubric-based evaluation when both receive the same execution traces, artifacts, and manuscripts?

  • Why does functionally equivalent reasoning receive less credit, and how do scores and rankings change when equivalent approaches receive full credit or other scoring weights are varied?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KL3d — Part 1

Thank you for your review. Please find our responses below. We have submitted the revised PDF, with changes and additions highlighted in blue.

1. Novelty and the scoring method

Thank you for your comments on novelty and evaluation. We understand that your concerns are what substantive changes arise from integrating task packages and scoring dimensions, and whether R3Eval adds value over a pointwise rubric. We address these questions below.

1.1 Contributions of the task packages and evaluation protocol

Regarding your assessment that the contribution is incremental, we would like to clarify two aspects. First, our task packages provide a research question, data, scientific constraints, and an execution environment, and require agents to perform analyses, validate findings, and write a manuscript. Agents must therefore produce both research results and verifiable execution evidence, and use them to construct a complete scientific argument. We agree that task coverage and workflow integration include incremental advances. The central change is that jointly requiring research execution, evidence generation, and manuscript argumentation changes the capabilities being evaluated.

This task objective makes the relationship between an agent's decisions, the evidence it produces, and its final claims part of the evaluation. A successful submission must carry the same research question through analysis and validation into a manuscript whose claims can be checked against the resulting artifacts. This is why the task objective itself is part of the contribution.

Second, R3Eval establishes explicit criteria for assessing these outputs. It links execution traces, analytical artifacts, and manuscripts to task obligations and reasoning chains frozen before scoring, distinguishing whether work was completed, accurately reported, and validly interpreted. A required validation omitted from both the plan and manuscript still enters the RC assessment; a submission with genuine analytical artifacts may receive lower RA credit if its inference exceeds the evidence. Our focus is thus traceable distinctions and diagnoses of research completion, reporting quality, and reasoning in long-horizon tasks. The direct comparison below tests the value of this design.

Revisions: main text §§1, 3.1–3.4; Supplement §§A.6 (Tables S4–S6), C.1–C.3.

1.2 Comparing scoring methods under identical evidence

We agree that your proposed comparison is essential to assess R3Eval itself separately from the benefit of additional execution evidence. We fix candidate submissions, expert reference scores, and Judge settings, with all conditions sharing task and evaluation reference information. A receives only the candidate manuscript; B and C receive the same complete traces, analytical artifacts, and manuscript. Comparing B with C therefore isolates the scoring method under identical evidence.

The pointwise rubric follows the expert assessment criteria to judge individual characteristics directly. R3Eval connects these judgments through the frozen obligations and reasoning chains. The comparison thus tests whether explicitly relating the evidence to the required work and conclusions improves agreement with experts.

We measure agreement with experts using mean absolute error (MAE): the absolute difference between automated and expert scores for each submission, averaged over the same submission set. MAE is computed separately for RC, RR, and RA, in points on the 0–100 scale. Lower values indicate closer agreement with experts.

SettingScoring methodRC MAERR MAERA MAE
AManuscript + pointwise rubric15.411.612.3
BFull evidence + pointwise rubric10.510.111.2
CFull R3Eval3.44.86.2

The improvement from A to B shows the benefit of making execution evidence available. The further improvement from B to C shows the value of assessing that material through explicit obligations and evidence-to-conclusion relations. Because B and C use the same submissions and expert scores, this improvement reflects how the evidence is assessed.

These results directly address your question about the scoring method's added value: with identical execution evidence, full R3Eval is closer to expert judgments on all three axes. Organizing evidence around task obligations and reasoning chains makes research completion, reporting accuracy, and inferential validity individually checkable, improving evaluation accuracy under the same evidence conditions. This is the evaluation contribution we seek to emphasize.

Revisions: main text §§4.5 (Table 3), 4.6 (Table 4); Supplement §§D.5.3, D.5.5 (Table S20).

AuthorsOpenReview ↗

Response to Reviewer KL3d — Part 2

2. Scoring rationale and valid alternatives

2.1 Rationale for weights and credit mappings

In response to your concern about the scoring rationale, the weights reflect each stage's role in research reliability. RC emphasizes execution and validation because they directly establish completion. RR emphasizes accurate reporting of methods and results. RA assesses both argument components and their connections, linking analysis, evidence, and conclusions. We then use sensitivity analysis to test how the numerical choices affect our findings.

Revisions: main text §3.4; Supplement §§C.1–C.3, D.3.

2.2 Giving full credit to functional equivalence

We understand your concern that the 0.85 versus 1.00 mapping may disadvantage equally valid alternatives. The default mapping distinguishes arguments that directly correspond to the source study from those preserving the same scientific function through a different design. To test its impact, we raise functional-equivalence credit to 1.00 in this comparison and recompute scores and rankings.

Evaluation unitCountMean increaseMaximum increase
Affected submissions, uncapped RA810.3781.487
Affected submissions, final RA810.3331.487
Affected reasoning chains912.1275.280

Credit changes affect submissions using functionally equivalent paths, while aggregation and existing caps reduce their impact on overall means. We retain this distinction to recognize an alternative's scientific function while reflecting its correspondence to the source study's specific design and evidence sources. For example, an AqSolDB submission validates groups using released fields, supporting the consistency of the released data and earning functional-equivalence credit. This remains a different validation path from calling the external service again. Together with RA's aggregation and caps, the mapping records this local distinction while preserving the overall system comparison.

Revisions: main text §3.4.3; Supplement §§C.3, D.5.6.

2.3 Distinguishing valid alternatives from reasoning errors

Agents may change their implementation while answering the same research question, preserving the required evidence function, and grounding conclusions and their scope in the actual evidence. Acceptance therefore rests on scientific function; an error judgment must identify missing evidence or an invalid inference.

Across three tasks, we compare reference and permitted alternative paths with either supported or faulty inferences. The key question is whether a valid alternative can outperform a reference path with flawed reasoning. We score a single frozen reasoning chain on a 0–100 scale; higher raw scores indicate more complete functional correspondence to the reference argument.

TaskReference / supportedAlternative / supportedReference / faultyAlternative / faulty
196.394.286.281.5
297.396.179.982.1
397.995.283.685.4

Both supported paths score highly; inferential errors lower both scores, and valid alternatives outperform faulty reference paths. RA distinguishes valid alternatives from reasoning defects through scientific function and evidential support, allowing an alternative to receive more credit than a superficially similar but flawed reference path. This directly supports the validity of RA that you asked about.

Revisions: main text §§3.4.3, 4.8 (Table 6); Supplement §§C.3 (Table S15), D.5.4 (Table S19).

2.4 Weight sensitivity

To answer your question about ranking stability under reasonable alternative settings, we test equal weights, two local perturbation magnitudes, and component/relation shares. We fix atomic judgments and applicability sets, change one weight group at a time, and recompute system scores and rankings.

PerturbationChangeMax. absolute score differenceMin. ρ / τ-b
Equal weightsEach parameter group separately3.8630.9879 / 0.9556
Local ±10%Multiply one weight by 0.9 or 1.1; normalize0.4961.0000 / 1.0000
Local ±20%Multiply one weight by 0.8 or 1.2; normalize1.0250.9879 / 0.9556
Component/relation share0.70/0.30 to 0.50/0.50, 0.60/0.40, or 0.80/0.200.5340.9879 / 0.9556

All tested weight changes preserve highly consistent rankings, with movements mainly between adjacent positions. Rechecking common tasks using only final scores gives similar findings. Stability holds both near the default weights and when redistributing evaluation emphasis. Thus, the overall relative performance of systems remains stable across the tested reasonable scoring configurations; the main comparison is robust to these weight choices.

Revisions: Supplement pointer at the end of main-text §3.4; Supplement §§D.3, D.5.6.

Thank you once again. I look forward to discussing this further with you.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the AC and Reviewers: Summary of Revisions

Dear AC and Reviewers,

Thank you for the time you have invested in reviewing our submission and for the specific, constructive comments you provided. We have submitted a revised manuscript, in which all changes and additions are highlighted in blue.

Several concerns were raised by more than one reviewer, and we have addressed them directly. We clarify the contributions of the task packages and of R3Eval, and add a controlled comparison between R3Eval and a pointwise rubric under identical evidence. We set out the rationale and sensitivity analyses for the scoring weights, the capping rules, and the credit given to functional equivalence, and we distinguish valid alternative solutions from faulty reasoning. We strengthen the evidence for benchmark validity and interpretability through results grouped by source-paper publication year and by domain, a complete worked task example, a description of how much method information each task discloses, and a discussion of saturation and limitations. We have also revised the relevant figures, notation, and wording.

The location and justification of each change are given in our point-by-point responses. We thank you again for your comments, which helped us present the research design, the experimental evidence, and the scope of our conclusions more clearly. We would also welcome any further comments on these revisions and are glad to discuss them further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.