acceptodds
Withdrawn from ICLR 2027

PaperSync: Reconstructing Research Evolution for Evidence-Gated Incremental Manuscript Synchronization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Research projects evolve continuously, and scientific manuscripts should evolve incrementally alongside them. This requires tracing long-term changes across code, experiments, manuscript versions, and related literature, identifying affected claims, and applying only revisions supported by evidence. Existing workflows are largely snapshot-based: they update manuscripts from the current project state without modeling how that state emerged or whether detected changes justify textual revision. We formulate incremental manuscript synchronization as a selective revision problem under long-term research evolution. We introduce PaperSync, an evidence-grounded framework that reconstructs research evolution paths and produces localized, reviewable manuscript patches. A versioned research evolution graph links changes in project artifacts and literature context to manuscript claims. PaperSync separates impact tracing, which identifies content that may require revision, from evidence authorization, which determines whether a revision is justified. An evidence contract gates candidate revisions according to evidential scope, version compatibility, and unresolved conflicts. The resulting synchronization plan categorizes candidate actions as supported revisions, justified preservation, or deferred decisions and guides the generation and verification of minimal patches. We evaluate PaperSync using a benchmark protocol spanning controlled omissions, genuine version transitions, and no-change cases. The protocol measures whether systems identify affected claims, perform warranted updates, avoid unnecessary edits, and cover relevant synchronization decisions. PaperSync reframes manuscript maintenance as tracing research evolution and selectively propagating evidence-supported changes into the manuscript.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Research Evolution, Manuscript Synchronization, Evidence-Gated Revision, Paper-Code Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_C4XJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

PaperSync addresses manuscript maintenance when code, experimental results, or relevant literature change. It proposes a versioned research graph, impact tracing to identify potentially affected claims, evidence contracts to authorize revisions, and localized patches. ManuSyncBench is described as a repository-grounded benchmark constructed through perturbations and omissions of author-written content. The manuscript presents comparisons against direct LLM editing and a general-purpose agent scaffold, together with component ablations.

Critical strengths

The task formulation identifies a meaningful distinction. A changed artifact can warrant inspection without invalidating an existing manuscript statement. Separating impact detection from revision authorization is therefore sensible. Requiring a replacement to justify superseding the original assertion is a particularly useful design principle.

The proposed benchmark design has promising elements. Author-written reference passages, hidden target locations, project-level separation, and exclusion of answer-bearing metadata could support a useful controlled evaluation. These are merits of the described design; its completed construction and experimental execution remain unclear.

Critical weaknesses
  1. The empirical record is internally inconsistent, preventing assessment of the central results. The abstract and Tables 1–2 present quantitative experimental comparisons and ablations. However, the AI Use Statement says, “The tables contain no generated or measured experimental results” (p. 10, lines 490–491). The Reproducibility Statement says “complete benchmark construction and empirical validation remain outstanding” (lines 508–509), describes comparisons as prospective, and contains unresolved appendix references. The appendices leave essential experimental settings—including sampling, repeated-run counts, endpoints, execution budgets, and consequential baseline permissions—as TBD (Tables 5–9). Section 6 describes confidence intervals, execution diagnostics, and human adjudication, but their corresponding results are not reported. The quantitative presentation also contains unresolved inconsistencies. The abstract reports GPT-5.6-sol exact-span recall of 4.26% versus 3.02%, whereas Table 1 reports 0.213 versus 0.178 for PaperSync and the agent scaffold. No stated denominator or normalization reconciles these figures. Table 4 contains literal placeholders and an inconsistent before/after/change entry. Outdated draft statements could explain these discrepancies. Nevertheless, the submitted document does not establish which results are measurements from completed experiments. Until this is reconciled, I cannot evaluate the claimed effectiveness or component contributions.

  2. The central evidence-authorization mechanism is insufficiently operationalized, and its selective-revision benefit is not demonstrated. Section 4 specifies meaningful evidence-contract requirements, including applicable versions, counterevidence, and operation-specific obligations. What remains unclear is the concrete procedure that determines whether these requirements are satisfied: how functional correspondences and impact paths are selected, how support and conflict are assessed, and how these assessments produce authorization decisions. The appendices mainly list enabled components and defer the full prompts to a future release. The evaluation also leaves the distinguishing selective-revision claim unresolved. The authors explicitly restrict the benchmark to verifiable required updates rather than exhaustive PRESERVE or DEFER labels. This is a defensible scope restriction, but update recall alone cannot establish avoidance of unnecessary or unsupported revisions. The paper defines Non-target Edit Rate and Strict Case Success to assess preservation and complete updates without collateral changes (Section 6.2, lines 360–367), yet reports neither in the main comparison nor the ablations. Document-quality and PDF-density diagnostics cannot substitute for these outcomes. Consequently, even if the recall tables are authenticated, the presented evaluation does not establish that evidence authorization and localized patching deliver their claimed control over unwarranted edits.

Questions for authors
  1. Which benchmark construction steps and experiments were completed for this submission? Please reconcile the abstract and Tables 1–2 with p. 10, identify the authoritative measured results, and provide the evaluated-case counts, configurations, run records, and uncertainty estimates supporting them.

  2. What operational procedure implements evidence authorization? Please trace an actual case from versioned sources through alignment and impact detection to the authorization decision and patch, explaining how evidential support, incompatible conditions, and conflicts are assessed.

  3. What completed evaluation establishes the selective-revision benefit? Please report the already-defined Non-target Edit Rate and Strict Case Success for the main comparison and relevant ablations, and delimit what the available evaluation establishes about preservation and deferral.

  4. I feel like the problem formulation is potentially useful, but the supplied version does not provide a coherent, sufficiently substantiated empirical record. The central authorization mechanism also I feel like it lacks enough operational detail and supporting evaluation to establish the claimed contribution. Could you please comment on this? A reconciled experimental record and evidence for the selective-revision benefit could materially change my assessment. Is it available?

Reviewer_A8kuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PaperSync addresses manuscript maintenance as research artifacts evolve. It constructs a versioned graph connecting manuscript claims with code, configurations, experiments, and literature; traces potentially affected claims; and applies evidence contracts to decide if to edit, preserve, or defer revision. The paper also introduces ManuSyncBench, which includes manually collected 203 cases from 128 papers and 331 atomic synchronization targets.

Critical strengths

I need to be blunt and say that I cannot identify a strength. I did not learn anything useful from this paper.

Critical weaknesses

I struggle with the problem statement and it motivation. A published paper describes a particular research state. Subsequent changes to code, experiments, or related literature do not necessarily require revising that paper. Unless, I am missing something.

There is no example in the paper to support the authors' problem statement. So, I completely miss the intuition of the proposed problem and solution. The benchmark itself deliberately alter claims or values, or remove manuscript passages. If the problem existed, I would expect the authors to find evidence in real instances.

The formalization section is superfluous, it is never again referenced.

The empirical study cannot be reproduced. The appendix includes placeholders. The paper claims to have a solution to decide whether to edit, preserve, or defer, but the benchmark focuses on cases requiring edits. So, it does not test its central premise.

There are some inconsistencies between reported numbers.

Questions for authors

Please see above.

Reviewer_o5ioOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PaperSync, a framework for synchronizing research manuscripts with evolving code, experiments, and literature. It constructs a versioned research graph to track changes, identifies potentially affected claims, and uses evidence authorization to decide whether to edit, preserve, or defer revisions.

The authors also introduce ManuSyncBench, containing 203 cases from 128 papers, mainly constructed by perturbing or removing existing manuscript content. The paper reports improvements over direct LLM editing and a general-purpose coding agent scaffold.

Critical strengths

Practical motivation. Keeping research papers consistent with evolving code and experimental results is a useful problem. The idea of tracing changes to specific claims rather than rewriting the entire manuscript is reasonable.

Structured editing workflow. Separating impact detection from evidence authorization is a sensible design. It could help reduce unnecessary or unsupported edits.

Critical weaknesses

Limited novelty and benchmark significance. I am not convinced this task requires a dedicated research benchmark. Much of the proposed workflow involves understanding repository changes, identifying inconsistencies, and updating documentation, which are already standard coding-agent capabilities. The paper does not clearly establish what new capability is being evaluated beyond existing repository understanding and document editing tasks.

Benchmark does not fully capture the proposed problem. ManuSyncBench is largely constructed by perturbing existing claims or removing author-written content and asking the agent to recover it. This seems substantially easier than handling actual research evolution, where agents need to determine whether new experiments, code changes, or literature genuinely justify modifying scientific claims. More importantly, the benchmark does not directly evaluate the proposed PRESERVE and DEFER decisions, despite these being central to the method.

Unclear experimental validity. The reported results need significant clarification. The AI Use Statement says the tables contain no measured experimental results, while the Reproducibility Statement indicates that empirical validation is still incomplete. The abstract also reports different numbers from Table 1, and many experimental configurations remain marked as TBD. These inconsistencies make it difficult to assess the reported improvements.

Questions for authors

What capability does ManuSyncBench measure that existing repository-level coding and documentation benchmarks do not? Could a standard coding agent with repository search and localized editing achieve similar results?

How does the current benchmark evaluate whether an agent correctly decides not to modify a manuscript? Without PRESERVE and DEFER cases, how can the evidence authorization mechanism be properly validated?

Are the results in Tables 1 and 2 actual measured experimental results? Could the authors clarify the contradictions with the AI Use and Reproducibility Statements and provide the missing experimental configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.