acceptodds
Under review as a conference paper at ICLR 2027

PaperRecon: Characterizing Scientific Writing Capabilities of AI Agents through Paper Reconstruction

Abstract

As coding agents become capable of generating entire research papers, assessing their actual writing quality and risks has become critical. Existing evaluations primarily rely on overall review scores, which assess acceptance-worthiness but provide limited insight into how individual agents write. We introduce (PaperRecon), a novel framework for characterizing scientific writing capabilities from multiple perspectives. PaperRecon formulates scientific writing as a reconstruction task. Given a structured overview and supporting resources derived from an existing paper, an agent reconstructs the full paper, which is then directly compared with the original. We evaluate two core aspects of scientific writing. measures how well the reconstructed paper captures the intended scientific content using section-specific rubrics, while analysis identifies claims that contradict the source paper and remain unsupported after checking the provided resources. We further conduct four diagnostic analyses to characterize the effects of the harness and backbone model, input information, citation behavior, and verbosity. To support this evaluation, we introduce , comprising 72 recent papers from top-tier venues in 2025 and 2026, filtered to reduce contamination from prior model knowledge. Our evaluation reveals clear differences across model families. Claude-based agents achieve higher content coverage than GPT-based agents but produce substantially more hallucinations. These results show that scientific writing capability cannot be adequately characterized by a single overall score and demonstrate how PaperRecon provides fine-grained insights into the strengths and weaknesses of current coding agents.

Primary area:
datasets and benchmarks
Keywords:
coding agents, AI-driven paper writing, risk

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LwJROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PaperRecon, a framework for evaluating scientific writing via paper reconstructions from structured summaries and additional resources. Reconstructions from different agents are benchmarked on 72 original papers based on their section-wise content coverage and detected contradictions. The results indicate that Claude-based agents achieve higher coverage, while GPT-based agents produce fewer contradictions.

Critical strengths
  • I think it makes sense to evaluate different aspects of scientific writing (content coverage, hallucinations) individually to obtain insights, instead of trying to capture everything with a single reviewer score.
  • Overall, I like the idea of paper reconstructions for helping decouple writing ability from research ability (idea generation, conducting experiments, ...).
  • The human evaluations, while being small-scale, provide some evidence that the proposed evaluation makes sense.
  • Some of the analyses are interesting / insightful (effect of overview length, harness vs. model effect).
Critical weaknesses
  • It is unclear to me to what degree quality should be defined w.r.t. the original paper in your setting. I understand that sections are first mapped into common categories, which addresses some differences in organization. Still, some details might reasonably be mentioned in another section, or the focus could lie on slightly different points. These changes could lead to lower reconstruction scores without necessarily making the paper worse. I could imagine that scoring alternative (human-written) versions of the same paper with the existing rubrics could shed some more light on whether lower scores reflect reasonable editorial choices or missing scientific information.
  • The hallucination score is based on claims that contradict information from the original paper and remain unsupported after checking the provided resources. Additional information that is absent from the original but does not contradict it is classified as neutral. I wonder whether this misses relevant cases of unsupported information. Furthermore, I wonder whether just comparing the number of hallucinations is a fair comparison when models make varying numbers of claims. Providing hallucination rates w.r.t. the overall number of claims, alongside the raw counts, could be an option here.
  • The holistic human preference study is conducted by two reviewers on reconstructions of eight original papers. As far as I understand, the reviewers get different source papers assigned to them and therefore do not evaluate the same reconstructions. It remains unclear to what degree reviewers agree and whether results generalize beyond the considered eight original papers. The other human checks are useful, but do not resolve this question.
Questions for authors
  1. For the human preference evaluation, do you think it would be possible to ask several annotators to evaluate the same reconstructions in terms of clarity, coherence, scientific completeness, and factual accuracy, and report their agreement?
  2. I could not fully reconstruct how the Avg. score in Table 1 is calculated. Are all rubric items weighted equally, or are section-level scores averaged first?
6 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1)

We sincerely appreciate your careful review of our paper and your valuable suggestions for enhancing the clarity and soundness of our work. We have thoughtfully addressed your feedback and revised the manuscript accordingly, with all changes highlighted in brown. Below, we provide point-by-point responses to your comments:

W1: Evaluating Scientific Content Independently of Writing Structure

Thank you for your comment. We agree that structural differences alone do not indicate lower paper quality. To address this concern, we designed PaperRecon to reduce the influence of structural differences and conducted an additional analysis to examine their impact on evaluation results.

First, our evaluation design reduces structural variations. All Writing Agents receive the same Overview, which defines the key scientific content to be reconstructed. Generated papers also follow the original section structure (L233–236). Furthermore, since we found that experimental subsection structures varied considerably across papers, we grouped them into one category to reduce their influence on evaluation.

Second, we conducted an additional analysis to measure the impact of structural differences. We re-evaluated all 6,023 text Rubric items scored 1–2, allowing the judge to search the entire generated paper rather than only the corresponding section.

AgentModelMissing from paperFound in another sectionScore: Original → Revised
CodexGPT-5.4-mini60%20%3.26 → 3.45
CodexGPT-5.6-luna60%22%3.37 → 3.58
CodexGPT-5.6-terra58%21%3.44 → 3.62
Claude CodeSonnet 4.651%23%3.66 → 3.82
Claude CodeSonnet 550%26%3.82 → 3.97

We count an item as missing if it is still rated 1–2 when the judge reads the whole paper, and as found in another section if it is rated 4–5 with a supporting quote from a different section. The results show that most low-scoring items reflect missing content, and accounting for content in other sections does not change the model ranking. We retain section-level evaluation because presenting important information in appropriate sections is also part of faithful scientific reconstruction. We have included this additional analysis in the revised paper.

W2: Validity of Neutral Claims and Hallucination Evaluation

We address the definition of Hallucination and the normalization by the number of Claims below.

1. Hallucination Definition and Neutral Claim Validation

We agree that classifying non-contradictory Claims as Neutral may overlook unsupported information. However, exaggerations, overgeneralizations, and unsupported comparative claims (e.g., "best", "largest", or "consistently") are classified as Contradictory since they conflict with the original results. In our manual analysis of 200 detected Hallucinations, such overclaims were the most frequent category (43/200, 21.5%).

To further examine this concern, we manually verified 105 Claims classified as Neutral by the LLM in Stage 1, which only compares claims against the original paper. After a more detailed human verification using additional sources, including code, 97 (92.4%) were found to be supported, 5 (4.8%) were reasonable interpretations, and only 3 (2.9%) contained factual errors.

Exhaustively verifying all Neutral Claims would require substantial external knowledge and evaluation costs. Therefore, PaperRecon focuses on verifiable Factual Contradictions, balancing detection coverage and practicality.

2. Hallucination Rate per Claim

Following the suggestions, we calculated the Hallucination Rate normalized by the total number of Claims, in addition to the absolute number of Hallucinations.

AgentModelClaims/paperHallucinations/paperRate (%)
CodexGPT-5.4-mini187.12.121.14
CodexGPT-5.6-luna195.52.421.24
CodexGPT-5.6-terra196.61.930.98
Claude CodeSonnet 4.6178.77.684.30
Claude CodeSonnet 5164.45.443.31

Although Claude models generated fewer Claims than GPT models, their Hallucination Rates were approximately 3–4 times higher. Thus, our main findings remained unchanged after normalization.

To further investigate why Claude models produced more Hallucinations, we conducted additional analyses based on Reviewer iR2Y's feedback and added the results to Sec. 6 of the revised manuscript.

AuthorsOpenReview ↗

Authors' Rebuttal (2)

W3, Q1: Role of the Human Preference Study

Thank you for your comment. Our Human Preference Study is a supplementary experiment to examine the relationship between Rubric Scores and holistic human judgments. Since evaluators must carefully compare multiple full papers, large-scale evaluation is very costly.

To measure inter-annotator agreement, we conducted an additional evaluation where both evaluators judged the same reconstructions, achieving 70.8% agreement.

We also found that evaluators emphasized different aspects, such as missing tables, unclear equations, and methodological explanations. Such differences are inherent in subjective human judgments, highlighting the importance of systematically evaluating Content Coverage and Factual Contradictions.

Q2: Calculation of the Average Score in Table 1

Thank you for pointing this out. The Avg. score is calculated by first averaging all rubric items within each paper, and then averaging these scores across the 72 papers.  We have clarified the calculation.

Reviewer_LwJROpenReview ↗

Thanks.

Thanks for the additional analyses. Your response answered some of my questions, but I still have two follow-ups:

  1. Coverage and alternative presentations: The full-paper check addresses content appearing in a different section. For items that still score 1–2, how do you distinguish an omission of essential scientific content from a reasonable choice about emphasis or level of detail? A few examples would help clarify what the score tells us about writing quality beyond faithful reconstruction.

2). Neutral claims: How were the 105 neutral claims selected across models and papers? What fraction of each model’s extracted claims was labeled neutral, and which models produced the three missed errors? This would help assess whether missed errors might affect the comparison between models.

Reviewer_iR2YOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PaperRecon, which evaluates scientific writing by asking agents to reconstruct papers from research summaries and supporting materials. Using 72 papers, it measures content coverage and factual contradictions across five agent configurations. Claude models achieve higher coverage but produce more contradictory claims than GPT models.

Critical strengths
  1. The evaluation checks both missing content and factual errors, providing more detail than a single overall paper score.

  2. The paper includes human validation and several useful analyses, including longer input summaries, alternative verifiers, and swapping models across agent harnesses.

Critical weaknesses
  1. The paper does not sufficiently establish the value of paper reconstruction as a benchmark task. Although it focuses on writing rather than experimental reproduction, the authors should explain what it reveals beyond existing writing and research-reproduction benchmarks such as PaperBench [1]. In particular, it is unclear whether the evaluation identifies general weaknesses of current models or mainly captures difficulties specific to this reconstruction setup. Showing which problems existing benchmarks miss and what new insights this task provides would make its contribution more convincing.

  2. The human verification process needs a clearer description. For the overviews, it is unclear how reviewers checked that the supplied materials contained enough information to satisfy the evaluation rubrics. For the rubrics, the criteria for labeling items as “necessary,” “acceptable,” or “unnecessary” are not explained. Section 3.2 and Appendix A.2 also give different descriptions of who verified the overviews. Providing explicit verification criteria and worked examples of overview revisions and rubric judgments would help readers understand how the checks were performed and assess their reliability.

  3. The experiments cover 72 papers and five configurations from two model families, with several analyses limited to 25 papers. The findings mainly describe performance differences between the tested models, without sufficiently explaining the underlying causes or showing whether these patterns generalize beyond the evaluated settings. For example, it remains unclear whether Claude’s higher coverage and greater number of contradictory claims reflect more claims being made, difficulty interpreting the materials, or a greater tendency to fill in missing details. Investigating these factors would provide more useful insights than model rankings alone.

[1] PaperBench: Evaluating AI’s Ability to Replicate AI Research. 2025.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1)

We sincerely thank you for your thorough review of our paper and your constructive feedback on how to make our work more rigorous and robust. We have carefully considered your suggestions and updated the manuscript accordingly, with changes highlighted in red. Below, we provide detailed responses to each of your points:

W1: Distinction from PaperBench and the Value of Scientific Writing Evaluation

While PaperBench evaluates experimental implementation and reproduction, PaperRecon independently evaluates the reliability of scientific writing. Automating paper writing is becoming a more practical and immediate application of AI, making its independent and rigorous evaluation increasingly important.

The question of whether PaperRecon can identify issues that existing evaluation methods struggle to capture was also raised by Reviewer QHan. To investigate this, we applied ScholarPeer [1] to the same 10 generated papers and compared its evaluations with those of PaperRecon. Of the 23 Contradictions detected by PaperRecon, ScholarPeer identified only 7 (30%).

These results demonstrate that, through its controlled reconstruction setting, PaperRecon can identify specific factual contradictions in scientific content, providing diagnostic information that existing review-based evaluations do not capture.

[1] Goyal+, ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review, arXiv2026

W2.1: Clarification of Overview Verifiers

Thank you for pointing it out. In Appendix A.2, we mistakenly used "the original authors" to refer to our paper's authors. As correctly stated in Section 3.2, three authors of our paper verified the Overviews of their assigned papers. We have corrected the wording in Appendix A.2 to ensure consistency between the two sections.

W2.2: Overview Verification and Rubric Judgment Criteria

Thank you for your valuable comment. We provide further details on the criteria and revisions involved in the human verification of Overviews and Rubrics.

1. Overview Verification

The purpose of the Overview is to concisely provide the key research information needed for reconstruction, rather than covering every detail of the original paper. Since Writing Agents also receive figures, tables, references, and code, the Overview alone does not need to cover all Rubric items. During human verification, we checked whether each Overview was consistent with the original paper and contained the key information necessary for reconstruction.

Regarding the specific revisions, we found that some Overviews lacked details, particularly regarding experimental settings and evaluation metrics, and revised them accordingly. For example, in MMDiff (paper_021), we added the numbers of training and evaluation samples and experimental pipeline settings, in addition to the numerical results. In WorMI (paper_063), we added definitions of the evaluation metrics (Success Rate and Pending Steps) to supplement the numerical results, which had been reported without explaining what each metric measures.

2. Rubric Verification

For Rubric verification, we asked the authors of the original papers to classify each item according to three criteria:

  • Necessary: Essential for accurately reconstructing the scientific content of the paper.
  • Acceptable: Not essential, but would improve the quality of the reconstructed paper if included.
  • Unnecessary: Contradictory to the original paper or unnecessary as an evaluation criterion.
AuthorsOpenReview ↗

Authors' Rebuttal (2)

W3.1: Extending the 25-Paper Analyses to All 72 Papers

Thanks for pointing it out. We extended the overview-length and harness-crossing analyses to all 72 papers. We observed that the conclusions are unchanged.

W3.2: Analysis of Coverage and Contradiction Trade-offs Across Models

We appreciate the reviewer's suggestion to investigate the factors underlying performance differences across models. We conducted two additional analyses: Claim Volume and Sources of Contradictions. Both analyses use the same 72-paper evaluation results as in Table 2.

1. Claim Volume

AgentModelClaims/PaperContradictions/PaperContradiction Rate (%)
CodexGPT-5.4-mini187.12.121.14%
CodexGPT-5.6-luna195.52.421.24%
CodexGPT-5.6-terra196.61.930.98%
Claude CodeSonnet 4.6178.77.684.30%
Claude CodeSonnet 5164.45.443.31%

The table reports the number of Claims extracted in Stage 1 and the contradiction rate.

Although Claude models generated fewer claims than GPT models, their Contradiction Rates were 3–4 times higher. Therefore, the difference in the number of Contradictions cannot be explained simply by differences in Claim Volume.

2. Sources of Contradictions

We classified the 1,411 detected Contradictions in all experiments into the following two categories using DeepSeek-V4-Pro:

  • Material Misinterpretation: Contradictions arising from incorrect interpretations of information provided in the source materials.
  • Unsupported Elaboration: Contradictions arising from information added by the Writing Agent that was absent from the provided materials and inconsistent with the original paper.

The classification was based on the Stage 2 verification results. The remaining 3 of the 1,411 Contradictions (2 for Sonnet 4.6 and 1 for GPT-5.4-mini) fit neither category.

AgentModelMisinterpretation/paperElaboration/paperElaboration Ratio
CodexGPT-5.4-mini1.810.3114.4%
CodexGPT-5.6-luna1.990.4317.8%
CodexGPT-5.6-terra1.610.3216.5%
Claude CodeSonnet 4.65.961.6922.1%
Claude CodeSonnet 54.171.2823.5%

Across all models, the majority of Contradictions (77–85%) were attributed to Material Misinterpretation. Compared with GPT models, Claude models produced approximately 3 times more Material Misinterpretations and 4 times more Unsupported Elaborations, with a higher proportion of the latter.

These results suggest that Claude models' higher number of Contradictions is not due to greater Claim Volume, but is associated with both misinterpretation of the provided materials and unsupported elaboration.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer iR2Y,

Thank you again for your time and constructive feedback on our submission.

We have carefully considered your comments and believe that we have addressed your main concerns through additional experiments, analyses, and clarifications presented in our rebuttal and revised manuscript.

In particular, we conducted a comparison with ScholarPeer, clarified the Overview and Rubric verification procedures, extended our analyses to all 72 papers, and investigated the factors underlying differences in Contradictions across models.

As the discussion period is coming to an end, we would be very grateful if you could take another look at our responses and reconsider your evaluation if your concerns have been addressed.

If you have any remaining questions or concerns, we would be happy to clarify them.

Best regards,

The Authors

Thank you for your reply. I think my concerns are addressed. I will increase my score.

Reviewer_QHanOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores how well AI agents can write scientific papers when the research itself is already finished. The authors introduce PaperRecon, a method that has agents rebuild papers from summaries and artefacts. They then check how closely the reconstructed papers match the originals and if they contain factual errors. In tests on PaperWrite-Bench, the authors find agents using Claude tend to cover more of the original content, but also make more contradictory claims than agents using GPT. The paper also looks at how the results change with more detailed input materials, different underlying models, and different agent frameworks.

Critical strengths

The use of human validation gives confidence to the evaluation accuracy. The authors manually check rubric scores and flagged contradictions, and compare rubric rankings with human preferences. These checks provide useful evidence that the automated judgments reflect meaningful differences in paper quality rather than relying entirely on LLM outputs.

The evaluation provides interpretable feedback. The paper-specific rubrics and explicit contradiction counts help identify which scientific details an agent omits or misrepresents. This makes the results useful for diagnosing writing behavior.

Critical weaknesses

My main concern is the incremental contribution over PaperOrchestra, cited in the related work. They already evaluate multiple aspects of writing and uses reviewers with evidence-checking capabilities. PaperRecon offers more explicit, paper-specific coverage checks and counts of verified contradictions, which could provide useful diagnostic detail. The comparison on a single score is not enough to show the value over prior works. A more detailed comparison on the same papers with examples of errors an omissions identified would be useful.

The contamination checks introduced in the paper are insufficient. Although the authors attempt to address this issue through title recall and sentence completion tests, these tests are insufficient to check that a paper was absent from the model’s training data. Models may not memorise specific sentence or titles, but still memorise ideas in the paper. Using papers from after the models release would be more reliable.

Questions for authors

Could the authors provide a more detailed comparison with PaperOrchestra’s evaluation suite, particularly its multidimensional assessments and ScholarPeer reviewer, ideally using the same generated manuscripts to demonstrate PaperRecon’s additional diagnostic value?

Could the authors compare when the papers were released with the model release date, where available, and provide results on a common subset of papers released after the models? This would be a better check for information leakage.

4 comments
AuthorsOpenReview ↗

Authors' Rebuttal

We sincerely thank you for your insightful and constructive feedback. Your comments are highly valued and critical to making our paper more robust. We have carefully considered your suggestions and updated the manuscript accordingly, with changes highlighted in magenta. Below, we provide detailed responses to each of your points:

W1: Comparison with PaperOrchestra and ScholarPeer on the Same Generated Papers

We agree that PaperOrchestra provides multidimensional evaluation and evidence verification through ScholarPeer. However, PaperOrchestra primarily focuses on assessing the overall quality of generated papers, whereas PaperRecon specifically aims to identify content omissions and factual contradictions against the original paper.

To demonstrate this distinction, we applied ScholarPeer to 10 papers generated by Sonnet 5 and GPT-5.6-terra.

1. Contradiction Detection Rate

For each contradiction detected by PaperRecon, we used DeepSeek-V4-Pro to determine whether ScholarPeer's review identified the same error.

Generated PaperContradictions Detected by PaperReconAlso Identified by ScholarPee
GPT-5.6-terra51
Sonnet 5186
Total237(30%)

ScholarPeer identified only 30% of the contradictions detected by PaperRecon. The identified errors were limited to internal inconsistencies, such as claims that conflict with tables in the generated paper itself. In contrast, errors requiring comparison with the original paper, such as incorrect descriptions of methods, were missed.

2. Comparison with PaperOrchestra's Multidimensional Assessments

We applied PaperOrchestra's SxS Overall Paper Quality and Literature Review Quality to the GPT-5.6-terra and Sonnet 5 papers for all 72 benchmark papers, using its prompts with DeepSeek-V4-Pro.

ModelSxS (Win / Tie / Loss)Literature Review QualityPaperRecon Rubric Score
GPT-5.6-terra10 / 16 / 4654.03.54
Sonnet 546 / 16 / 1062.53.93

Both evaluations show similar trends (the SxS winner has the higher Rubric Score in 80% of decided pairs), but only the Rubric provides an interpretable diagnosis. PaperOrchestra's assessments only indicate which paper is better, whereas the Rubric identifies what each paper misses (e.g., for CLEAR (paper_001), a tie in SxS, Sonnet 5's paper omits the dataset size and the hard-negative setting).

3. Example of Missing Scientific Content

ScholarPeer did not identify the missing experiment that PaperRecon detected, as its evaluation is not designed to explicitly identify such omissions.

For example, a paper generated by GPT-5.6-terra (DSeq-JEPA, paper_019) omitted the [CLS] token control experiment reported in the provided table. PaperRecon's rubric explicitly identified this omission. Although ScholarPeer also raised concerns about the [CLS] token, it only asked how the token was introduced and trained without identifying the experiment omission.

These results demonstrate that review-based evaluation alone does not reveal which specific factual errors or omissions each model makes, or how frequently they occur. PaperRecon provides this diagnostic information, complementing existing evaluation methods.

W2: Benchmark Contamination and Post-Cutoff Evaluation

Thank you for your valuable comment. While the reviewer suggested evaluating papers published after the models' release dates, only 2 of the 72 papers were first made public after the release of every evaluated model (the latest being GPT-5.6-luna and GPT-5.6-terra on July 9, 2026), which is too few for a meaningful comparison. We therefore used the training data cutoff instead. The latest training data cutoff among the evaluated models was February 16, 2026, for GPT-5.6-luna and GPT-5.6-terra.

We evaluated 28 papers published after this cutoff. Throughout, a paper's publication date is the date its first version (v1) appeared on arXiv.

AgentModelRubric Score (All 72 Papers)Rubric Score (28 Post-Cutoff Papers)Contradictions/Paper (All 72 Papers)Contradictions/Paper (28 Post-Cutoff Papers)
CodexGPT-5.4-mini3.343.352.122.46
CodexGPT-5.6-luna3.463.462.422.46
CodexGPT-5.6-terra3.543.541.932.21
Claude CodeSonnet 4.63.743.767.688.25
Claude CodeSonnet 53.933.935.446.43

The Rubric Scores changed by at most 0.02, and the model rankings remained unchanged. Contradictions per paper increased slightly on the 28-paper subset, with differences ranging from 0.04 to 0.99, but the overall trends remained consistent. In particular, the main finding that Claude models achieve higher Content Coverage than GPT models while producing more Contradictions remained unchanged. These results confirm that our main findings remain consistent.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer QHan,

Thank you again for your time and constructive feedback on our submission.

We have carefully considered your comments and believe that we have addressed your concerns regarding Benchmark Contamination/Post-Cutoff Evaluation and Comparison with PaperOrchestra and ScholarPeer through additional experiments and analyses presented in our rebuttal and manuscript.

As the discussion period is coming to an end, we would be very grateful if you could take another look at our responses and reconsider your evaluation if your concerns have been addressed.

If you have any remaining questions or concerns, we would be happy to clarify them.

Best regards, The Authors

I appreciate the author's response. I am keeping my score.

AuthorsOpenReview ↗

A Kind Request for Further Clarification on Remaining Concerns

Dear Reviewer QHan,

Thank you very much for taking the time to reconsider our rebuttal.

We believe that we have carefully addressed your concerns through additional experiments and supporting evidence. While we fully respect your decision, if you still feel that your score should remain unchanged, we would be very grateful if you could kindly clarify which concerns remain unresolved.

Your feedback would be greatly appreciated and would help us further improve our paper.

Best regards,

The Authors

Comments on the paper

3 comments

Dear reviewers,

Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.

Many thanks for your time and effort.

AuthorsOpenReview ↗

Minor Correction to Paper IDs

Dear Reviewer QHan and Reviewer iR2Y,

We noticed a minor inconsistency between the Paper IDs in the dataset provided through README.md in the supplementary materials and those used in our responses. We have just corrected the IDs in our responses. The correct IDs are CLEAR (paper_059 → paper_01), DSeq-JEPA (paper_064 → paper_19), MMDiff (paper_020 → paper_21), and WorMI (paper_053 → paper_63).

This was only a numbering issue, and all referenced papers are included in the benchmark.

We sincerely apologize for any confusion or inconvenience.

AuthorsOpenReview ↗

Summary of Reviews and Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We would like to briefly summarize the strengths highlighted in the reviews and how we addressed the main concerns raised.


1. Strength

The main contributions of this work are formulating scientific writing as a verifiable paper reconstruction task and designing a diagnostic evaluation for it. The reviewers recognized the value of this approach and highlighted the following strengths.

  • Verifiable task design (Reviewer LwJR). Paper reconstruction lets us assess writing ability separately from research ability and verify the generated text against the original research.
  • Diagnostic evaluation (All reviewers). Evaluating content omissions and factual errors separately shows what agents leave out and what they get wrong.
  • Human validation (All reviewers). Manual checks of rubric scores and flagged contradictions, along with comparisons to human preferences, support the automated evaluation.
  • Additional analyses (Reviewers iR2Y and LwJR). Both reviewers highlighted the analyses of overview length and the effects of models and agent harnesses.

2. Summary of Rebuttal

The reviews mainly asked for additional evidence to support these contributions and findings. During the rebuttal, we addressed the following points.
※ A ✅ indicates that the reviewer explicitly confirmed that the concern was addressed.

  • What does PaperRecon uniquely enable? (Reviewers QHan and ✅ iR2Y) We compared PaperRecon with PaperOrchestra and ScholarPeer on the same generated manuscripts. PaperRecon identified specific omissions, such as a missing control experiment, and ScholarPeer identified 7 of the 23 contradictions flagged by PaperRecon.
  • Training data contamination (Reviewer QHan). Our additional experiment showed that the main conclusion remained unchanged when we evaluated the 28 papers first released after the latest training data cutoff. To maintain the size and diversity of the benchmark and allow for future expansion, we screened papers using title recall and sentence completion tests. The consistent results between the 28-paper subset and the full 72-paper benchmark support our use of the full screened benchmark for the main evaluation.
  • Scope of the analyses (Reviewer ✅ iR2Y). We expanded the overview-length and model–harness analyses from 25 to all 72 papers. The conclusions of both analyses remained unchanged.
  • Relationship between claim count and contradictions (Reviewers ✅ iR2Y and ✅ LwJR). Our additional analysis showed that Claude models made fewer claims than GPT models, with a contradiction rate roughly 3–4 times as high. We also found that Claude models more frequently misinterpreted the source materials and added unsupported details.
  • Do low rubric scores reflect meaningful omissions? (Reviewer ✅ LwJR) We re-evaluated low-scoring items using the full manuscript and found that the model rankings stayed the same. We also provided examples of missing baselines and evaluation protocols that made the results harder to interpret.
  • How reliable are the evaluation judgments? (Reviewers ✅ iR2Y and ✅ LwJR) We clarified how we verified the overviews and rubrics, manually checked 105 claims labeled neutral, and measured agreement between human evaluators judging the same reconstructions.

3. Discussion Outcome and Remaining Concern

Following the discussion, Reviewers ✅ iR2Y and ✅ LwJR both raised their scores.

The remaining concern is training data contamination (Reviewer QHan). Our additional experiment showed that the main conclusion remained unchanged on the 28 papers first released after the latest training data cutoff. We believe this result highlights the scalability of our two memorization tests (Sections A.5.1 and A.5.2) as a strength of our approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.