- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PaperEvaluation, a benchmark for evaluating agent-generated scientific papers. It contains 80 research tasks derived from published papers across 13 domains. The authors propose an EvidenceFirst Scaffold to guide agents through research planning, execution, validation, and manuscript writing. They also introduce R3Eval, which evaluates Research Completeness (RC), Reporting Rigor (RR), and Reasoning Alignment (RA) using execution traces, analytical artifacts, and final manuscripts. Experiments on 14 Model–Harness systems are conducted to compare their performance.
Critical strengths
-
An interesting and timely problem. Evaluating the scientific research capabilities of AI agents is becoming increasingly important. I appreciate the attempt to go beyond evaluating final manuscripts and consider the actual research process.
-
A reasonably comprehensive evaluation framework. The combination of execution traces, analytical artifacts, and manuscripts is a useful direction. Separating RC, RR, and RA also provides different perspectives on agent performance.
-
Extensive experimental effort. The authors evaluate 14 Model–Harness systems and include expert agreement analysis, ablation studies, and controlled error injection experiments. These experiments provide some useful insights into the proposed evaluation framework.
Critical weaknesses
1. Concerns about benchmark validity.
I am concerned that the reported scores may not reflect the models' independent research capabilities. I checked the appendix and found that 62 out of 80 source papers were published before 2025. Some models may have already encountered these papers during pretraining, potentially inflating their scores. The authors briefly acknowledge this issue but provide no contamination analysis.
Also, the benchmark already seems close to saturation (e.g., GPT-6 achieves 94.46 RC and 98.86 RA), raising concerns about its future discriminative power. The uneven distribution of 80 tasks across 13 domains also makes it difficult to draw general conclusions about autonomous research capabilities.
2. Unclear task formulation and missing examples.
I had a hard time understanding what an actual task looks like. The authors define Q, D, E, O, T, F, and M, but a few real examples of complete task packages, including inputs and outputs, would be much more helpful.
More importantly, the exact Research Question matters a lot. Take ResNet as an example. Is the question "How can we address the degradation problem in deep networks?" or "Can shortcut connections improve the training of deeper networks?" Both are valid questions, but they test very different abilities. The latter already reveals much of the original idea.
The authors mention preserving "essential scientific mechanisms", but how much of the original solution is actually revealed? And what exactly is the reference answer? Without real task examples, it is hard to tell what the benchmark is actually measuring.
3. The name is somewhat misleading.
"PaperEvaluation" sounds like a framework for evaluating scientific papers in general. However, this work mainly evaluates agent-generated research using execution traces and artifacts. I suggest considering a name that includes "Trace" to better reflect its focus.
4. Minor presentation issue.
There seems to be an odd tau annotation in Figure 3. I suggest proofreading the figures and notation more carefully.
5. Several scoring rules seem ad hoc and insufficiently defined.
I find some parts of R3Eval unnecessarily complicated. For example, how exactly is the RC cap function in Eq. (3) calculated? In Eq. (7), what counts as a critical reasoning-chain failure, and what numerical cap is applied?
There is also an inconsistency: Section 3.4.1 defines stage credits as 0, 0.5, or 1, while Appendix C.1 allows any value between 0 and 1. Which one is actually used?
The RC calculation also involves several weights, caps, and applicability rules, plus an additional obligation-coverage measure. I am not convinced that all this complexity is necessary or well justified.
ICLR welcomes simple and insightful methodological contributions. Adding more formulas and heuristic rules does not necessarily make a method more rigorous. I would prefer clearer definitions and a sensitivity analysis to justify these design choices.
Questions for authors
-
Data contamination: Given that 62 of the 80 source papers were published before 2025, how do the authors rule out the possibility that the evaluated models have already encountered these studies during pretraining? Could the authors provide a contamination analysis or report results separately for older and more recent papers?
-
Actual task inputs: Could the authors provide several complete examples of the actual agent-visible task packages, especially the Research Questions and mechanism descriptions? How do the authors ensure that these inputs do not reveal the key methodological contributions of the source papers?
-
Reference answers: What exactly is considered the reference answer for each task? How does R3Eval handle scientifically valid solutions that differ substantially from the original paper?
-
Scoring rules: Could the authors clarify how the score caps in Eq. (3) and Eq. (7) are determined, and resolve the inconsistency in RC stage credit definitions between Section 3.4.1 and Appendix C.1? How sensitive are the reported rankings to these heuristic scoring choices?
-
Benchmark saturation: Given the near-saturated RA scores, how do the authors expect PaperEvaluation to remain useful for evaluating stronger future models?
5 comments
Response to Reviewer iMz3 — Part 2
2. Task inputs, method disclosure, and evaluation references
Thank you for your suggestions on interpretability. We use the RNA-seq task to illustrate the inputs and outputs.
2.1 A complete task input/output example
The original research question for the RNA-seq differential-expression task is:
How robustly do RNA-seq differential-expression procedures control false discoveries and recover known signals across a real ERCC spike-in benchmark and simulations spanning sample size, DE prevalence, imbalance, dispersion and outlier regimes?
The actual inputs and candidate outputs correspond to the notation in §3.1 as follows:
| Item | Actual content |
|---|---|
| Q: Research question | Compare differential-expression methods in false-discovery control, recovery of known signals, and robustness across conditions |
| D: Data | Frozen SEQC, KIRC, and Bottomly count matrices, sample metadata, ERCC ground truth, and simulation protocols |
| E: Environment | R/Python environment executable on CPUs |
| O: Submission requirements | Data audits, method versions, metrics, sensitivity analyses, figures, tables, and manuscript |
| T: Execution trace | Execution records for audit, analysis, and summary scripts |
| F: Analytical artifacts | Individual metrics, ERCC predictions, and a manifest recording simulation seeds and conditions |
| M: Final manuscript | A complete report of research methods, results, and interpretations |
The input fields describe what the agent is given and required to deliver; the output fields show what it actually executed, produced, and reported. This distinction makes the task definition and the evidence used to assess it visible in the same example.
One actual submission ran three method families and 1,280 simulations with recorded seeds, producing an eight-page manuscript. Across 63 evaluable ERCC controls, count score and Wilcoxon achieved ROC AUCs of 0.903 and 0.989, and realized FDPs of 0.213 and 0.078, respectively, matching the manuscript's reported values. This provides a verifiable connection from visible inputs through execution to reported results.
Revisions: main text §3.1; Supplement §A.6 (Tables S4–S6: task inputs, mechanisms, and output files; the paragraph after Table S5 adds actual submission results).
2.2 Disclosure of source methods
We agree with the distinction in your ResNet example: specifying a mechanism changes what remains for the agent to discover. Our task descriptions therefore make the supplied mechanisms and the remaining research choices explicit. Tasks retain different degrees of research choice.
The RNA-seq task specifies count score, log-CPM Welch t, and Wilcoxon, together with BH correction, , and some simulation conditions. It tests whether agents correctly implement, validate, and interpret results under specified mechanisms. The task on player skin tone and referee red cards instead supplies an association question and constraints such as repeated observations, leaving models, covariates, and sensitivity analyses to the agent. It emphasizes research design and analytical choices.
This design covers two aspects of research capability: more prescribed tasks support comparisons of execution and validation rigor, while more open tasks test independent, scientifically justified analytical choices. Necessary scientific constraints keep alternative approaches focused on the same question. Explicitly stating the supplied mechanisms and remaining choices clarifies what each task measures.
Revisions: main text §§3.1–3.2; Supplement §A.6 (Tables S5, S6: RNA-seq and red-card task comparison).
2.3 What the reference answer is and how valid alternatives are handled
The reference consists of scientific obligations and reasoning references frozen before scoring, used to assess research completion and argument validity. AqSolDB illustrates the distinction:
- Valid alternative: a submission reconstructs groups from released fields instead of rerunning an external service. Row-level artifacts still validate the consistency of the released rules, so the design receives functional-equivalence credit.
- Reasoning error: another submission computes group-wise RMSE but interprets between-group differences as an underlying noise mechanism. The artifacts receive credit, while the inference exceeds their evidential support.
Alternatives receive credit for preserving the required scientific function; deductions identify missing evidence or invalid inferences. See our response to Reviewer KL3d, §2.2, for the local effects of credit mapping, and Reviewer pA76, §2.1, for detailed case judgments.
Revisions: main text §§3.1, 3.4.1–3.4.3; Supplement §§C.1, C.3 (Table S15: alternative cases and judgment criteria).
Response to Reviewer iMz3 — Part 3
3. Definitions and rationale for scoring rules
3.1 Eq. (3): RC caps
RC caps constrain credit for individual scientific obligations. We first aggregate decision, execution, validation, and reporting credits, then apply the cap determined by evidence conditions. A proposed and reported analysis without execution evidence for training and comparison should receive limited credit even if other stages score positively. For example, stage credits yield a weighted value of 0.45, capped at 0.25 because execution credit is zero. Available but unattempted obligations receive zero; missing artifact evidence imposes a 0.50 cap. Failed adaptations and substitutions that incompletely preserve the evidence function are capped at 0.25 and 0.35, respectively. When multiple conditions apply, the strictest cap is used.
High RC requires completion of the necessary research: planning and reporting earn credit but cannot compensate for missing essential execution evidence. Full trigger conditions and worked examples are provided in the Supplement.
Revisions: main text §3.4.1, Eq. (3); Supplement §C.1 (Tables S9–S11: stage credits, obligation states, and caps).
3.2 Eq. (7): Critical-chain failure
Critical chains are marked in the frozen rubric before scoring and represent arguments supporting the main conclusions. Suppose a submission's weighted RA is 85, but a critical chain scores 0.50, below the 0.55 threshold: final RA is capped at 60. At least two critical chains below the threshold impose a cap of 50. If any critical chain's analysis operation, evidence pattern, or inference is contradicted, the cap is 40. The strictest applicable cap is used.
A manuscript may complete many analyses yet base its main conclusion on an inference contradicting core evidence. High scores elsewhere should not give that argument a high overall RA. These caps make the reliability of central conclusions constrain the overall assessment, preventing numerous local alignments from concealing a serious flaw in the core argument.
Revisions: main text §3.4.3, Eq. (7); Supplement §C.3 (Table S14: critical chains, caps, and examples).
3.3 Stage credit, applicability, and coverage
Thank you for identifying this oversight. Stage credits are , representing absent, partial, or sufficient evidence; weighted obligation credit can take other fractional values. We have aligned Supplement §C.1 and main-text §3.4.1 with this discrete definition.
Applicability includes available but unattempted obligations in the denominator with zero credit, and excludes genuinely unavailable, prohibited, or out-of-scope obligations. Coverage is the weighted share of assigned nonoptional obligations that are applicable, indicating how much of the task RC covers. Stage credits measure evidential sufficiency, applicability determines the work to assess, and coverage explains the score's scope.
Revisions: main text §3.4.1; Supplement §C.1 (Tables S9, S10 and the applicability/coverage formulas and explanations).
3.4 Do weights and caps determine the system comparison?
To address your sensitivity question, we fix submissions and atomic judgments while changing weights and caps separately.
| Parameter | Alternative setting | Max. absolute score difference | Min. ρ / τ-b |
|---|---|---|---|
| Equal weights | Each parameter group separately | 3.863 | 0.9879 / 0.9556 |
| Local ±10% | Multiply one weight by 0.9 or 1.1; normalize | 0.496 | 1.0000 / 1.0000 |
| Local ±20% | Multiply one weight by 0.8 or 1.2; normalize | 1.025 | 0.9879 / 0.9556 |
| RC credit caps | 0.25/0.35/0.50, each ±0.05 | 0.188 | 0.9879 / 0.9556 |
| RA critical-chain threshold | 0.55 to 0.50 or 0.60 | 1.605 | 0.9758 / 0.9111 |
| RA score caps | 40/50/60, each ±5 points | 0.753 | 1.0000 / 1.0000 |
| Remove caps | Remove RC or RA caps separately | 5.746 | 0.9636 / 0.9111 |
Equal weights and local perturbations preserve high ranking agreement. Changes to RC caps, the RA critical-chain threshold, or RA caps produce limited local movements. The main system comparison does not depend on one precise weight or threshold; relative performance is robust across the tested reasonable alternative settings.
Revisions: Supplement pointer at the end of main-text §3.4; Supplement §§D.3 (Table S17), D.5.6 (Table S21).
4. Title and figures
Thank you for your suggestion. We intend to adopt PaperTrace: Benchmarking Agent-Generated Scientific Research with Execution Evidence as the title. This wording makes the role of execution evidence explicit. The submission title cannot be changed during the rebuttal phase, so we have not yet made this update. We sincerely apologize for this.
We have also enlarged some text in Figures 2–3 and corrected the rendering of τ in Figure 3. Thank you for your careful attention.
Thanks again, looking forward to further discussions with you.
Response to Reviewer iMz3 — Part 1
Thank you for your review. Please find our responses below. We have submitted the revised PDF, with changes and additions highlighted in blue.
1. Benchmark validity and interpretation of results
Thank you for your suggestions on benchmark validity. We address each concern below.
1.1 Source-paper publication year
To address your concern that models may have encountered older source papers, we group tasks by source-paper publication year: before 2025 and 2025 or later. We compare GPT-5.6 Sol and Claude-Sonnet-4.6 on the scored tasks in each group. This grouping addresses your request for a direct comparison across publication periods while keeping the evaluation criteria the same.
GPT-5.6 Sol scores higher on RC, RR, and RA in both groups. These axes assess execution evidence, reporting accuracy, and inferential validity, respectively. Even a model familiar with the source study must therefore perform and validate the research required by the current task. Scoring rests on verifiable research work, and the performance difference between these two systems persists across the year groups.
Year-group results: Supplement §D.5.7 (Table S23); related limitations: §E. The end of main-text §4.9 points to Appendix D.
1.2 Saturation of the benchmark's discriminative ability
We understand your concern about distinguishing stronger systems in the future. Table 5 in the revision (Table 4 in the original submission) constructs flawed variants of the same submissions. Removing execution and validation evidence reduces RC by 13.07 and 10.08 points on average, respectively; corrupting reasoning reduces RA by 18.59 points, while language-only changes have small effects. This shows that the current scores respond to specific research defects, and high scores require corresponding evidence and valid arguments. These comparisons show why high overall scores and sensitivity to research failures are compatible: scores drop when the supporting work or reasoning is degraded, even though the task itself is unchanged.
We also agree that rapidly advancing agent capabilities raise the requirements for continued discrimination. We are developing V2 with tasks allowing more open methodological choices, requiring more validation conditions, and demanding more analyses across datasets. The aim is to evaluate more independent research decisions and complex validation, beyond increasing the number of similar tasks.
Revisions: main text §§4.2 (Table 2), 4.7 (Table 5); expansion of task numbers and types is discussed in Supplement §E.
1.3 Domain imbalance
We agree with your concern that domains with more tasks may dominate overall conclusions. We therefore reanalyzed results by domain and show two agents' performance in three selected domains below. This breakdown makes variation across research settings visible. Readers can examine whether a system difference appears in domains with different task counts and identify which evaluation dimensions vary with the domain.
GPT-5.6 Sol outperforms Claude-Sonnet-4.6 in all three domains despite their different task counts, while both have lower completion and reporting scores in bioinformatics. Evaluation thus captures system differences and challenges posed by different research requirements. R3Eval uses common evaluation principles while retaining each task's scientific requirements: RC checks required work, RR checks reporting against artifacts, and RA checks whether evidence supports conclusions. Cross-domain comparisons therefore use task-specific scientific obligations and evidence relationships, rather than directly comparing raw performance metrics from different research questions.
Domain results: Supplement §D.5.8 (Table S24); the end of main-text §4.9 points to Appendix D. Domain distribution and cross-task aggregation are described in main-text §3.2 (Figure 2) and Supplement §D.4, respectively.