- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PaperRecon, a framework for evaluating scientific writing via paper reconstructions from structured summaries and additional resources. Reconstructions from different agents are benchmarked on 72 original papers based on their section-wise content coverage and detected contradictions. The results indicate that Claude-based agents achieve higher coverage, while GPT-based agents produce fewer contradictions.
Critical strengths
- I think it makes sense to evaluate different aspects of scientific writing (content coverage, hallucinations) individually to obtain insights, instead of trying to capture everything with a single reviewer score.
- Overall, I like the idea of paper reconstructions for helping decouple writing ability from research ability (idea generation, conducting experiments, ...).
- The human evaluations, while being small-scale, provide some evidence that the proposed evaluation makes sense.
- Some of the analyses are interesting / insightful (effect of overview length, harness vs. model effect).
Critical weaknesses
- It is unclear to me to what degree quality should be defined w.r.t. the original paper in your setting. I understand that sections are first mapped into common categories, which addresses some differences in organization. Still, some details might reasonably be mentioned in another section, or the focus could lie on slightly different points. These changes could lead to lower reconstruction scores without necessarily making the paper worse. I could imagine that scoring alternative (human-written) versions of the same paper with the existing rubrics could shed some more light on whether lower scores reflect reasonable editorial choices or missing scientific information.
- The hallucination score is based on claims that contradict information from the original paper and remain unsupported after checking the provided resources. Additional information that is absent from the original but does not contradict it is classified as neutral. I wonder whether this misses relevant cases of unsupported information. Furthermore, I wonder whether just comparing the number of hallucinations is a fair comparison when models make varying numbers of claims. Providing hallucination rates w.r.t. the overall number of claims, alongside the raw counts, could be an option here.
- The holistic human preference study is conducted by two reviewers on reconstructions of eight original papers. As far as I understand, the reviewers get different source papers assigned to them and therefore do not evaluate the same reconstructions. It remains unclear to what degree reviewers agree and whether results generalize beyond the considered eight original papers. The other human checks are useful, but do not resolve this question.
Questions for authors
- For the human preference evaluation, do you think it would be possible to ask several annotators to evaluate the same reconstructions in terms of clarity, coherence, scientific completeness, and factual accuracy, and report their agreement?
- I could not fully reconstruct how the Avg. score in Table 1 is calculated. Are all rubric items weighted equally, or are section-level scores averaged first?
6 comments
Authors' Rebuttal (2)
W3, Q1: Role of the Human Preference Study
Thank you for your comment. Our Human Preference Study is a supplementary experiment to examine the relationship between Rubric Scores and holistic human judgments. Since evaluators must carefully compare multiple full papers, large-scale evaluation is very costly.
To measure inter-annotator agreement, we conducted an additional evaluation where both evaluators judged the same reconstructions, achieving 70.8% agreement.
We also found that evaluators emphasized different aspects, such as missing tables, unclear equations, and methodological explanations. Such differences are inherent in subjective human judgments, highlighting the importance of systematically evaluating Content Coverage and Factual Contradictions.
Q2: Calculation of the Average Score in Table 1
Thank you for pointing this out. The Avg. score is calculated by first averaging all rubric items within each paper, and then averaging these scores across the 72 papers. We have clarified the calculation.
Thanks.
Thanks for the additional analyses. Your response answered some of my questions, but I still have two follow-ups:
- Coverage and alternative presentations: The full-paper check addresses content appearing in a different section. For items that still score 1–2, how do you distinguish an omission of essential scientific content from a reasonable choice about emphasis or level of detail? A few examples would help clarify what the score tells us about writing quality beyond faithful reconstruction.
2). Neutral claims: How were the 105 neutral claims selected across models and papers? What fraction of each model’s extracted claims was labeled neutral, and which models produced the three missed errors? This would help assess whether missed errors might affect the comparison between models.
Authors' Rebuttal (1)
We sincerely appreciate your careful review of our paper and your valuable suggestions for enhancing the clarity and soundness of our work. We have thoughtfully addressed your feedback and revised the manuscript accordingly, with all changes highlighted in brown. Below, we provide point-by-point responses to your comments:
Thank you for your comment. We agree that structural differences alone do not indicate lower paper quality. To address this concern, we designed PaperRecon to reduce the influence of structural differences and conducted an additional analysis to examine their impact on evaluation results.
First, our evaluation design reduces structural variations. All Writing Agents receive the same Overview, which defines the key scientific content to be reconstructed. Generated papers also follow the original section structure (L233–236). Furthermore, since we found that experimental subsection structures varied considerably across papers, we grouped them into one category to reduce their influence on evaluation.
Second, we conducted an additional analysis to measure the impact of structural differences. We re-evaluated all 6,023 text Rubric items scored 1–2, allowing the judge to search the entire generated paper rather than only the corresponding section.
We count an item as missing if it is still rated 1–2 when the judge reads the whole paper, and as found in another section if it is rated 4–5 with a supporting quote from a different section. The results show that most low-scoring items reflect missing content, and accounting for content in other sections does not change the model ranking. We retain section-level evaluation because presenting important information in appropriate sections is also part of faithful scientific reconstruction. We have included this additional analysis in the revised paper.
We address the definition of Hallucination and the normalization by the number of Claims below.
1. Hallucination Definition and Neutral Claim Validation
We agree that classifying non-contradictory Claims as Neutral may overlook unsupported information. However, exaggerations, overgeneralizations, and unsupported comparative claims (e.g., "best", "largest", or "consistently") are classified as Contradictory since they conflict with the original results. In our manual analysis of 200 detected Hallucinations, such overclaims were the most frequent category (43/200, 21.5%).
To further examine this concern, we manually verified 105 Claims classified as Neutral by the LLM in Stage 1, which only compares claims against the original paper. After a more detailed human verification using additional sources, including code, 97 (92.4%) were found to be supported, 5 (4.8%) were reasonable interpretations, and only 3 (2.9%) contained factual errors.
Exhaustively verifying all Neutral Claims would require substantial external knowledge and evaluation costs. Therefore, PaperRecon focuses on verifiable Factual Contradictions, balancing detection coverage and practicality.
2. Hallucination Rate per Claim
Following the suggestions, we calculated the Hallucination Rate normalized by the total number of Claims, in addition to the absolute number of Hallucinations.
Although Claude models generated fewer Claims than GPT models, their Hallucination Rates were approximately 3–4 times higher. Thus, our main findings remained unchanged after normalization.
To further investigate why Claude models produced more Hallucinations, we conducted additional analyses based on Reviewer iR2Y's feedback and added the results to Sec. 6 of the revised manuscript.