- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
PaperSync addresses manuscript maintenance when code, experimental results, or relevant literature change. It proposes a versioned research graph, impact tracing to identify potentially affected claims, evidence contracts to authorize revisions, and localized patches. ManuSyncBench is described as a repository-grounded benchmark constructed through perturbations and omissions of author-written content. The manuscript presents comparisons against direct LLM editing and a general-purpose agent scaffold, together with component ablations.
Critical strengths
The task formulation identifies a meaningful distinction. A changed artifact can warrant inspection without invalidating an existing manuscript statement. Separating impact detection from revision authorization is therefore sensible. Requiring a replacement to justify superseding the original assertion is a particularly useful design principle.
The proposed benchmark design has promising elements. Author-written reference passages, hidden target locations, project-level separation, and exclusion of answer-bearing metadata could support a useful controlled evaluation. These are merits of the described design; its completed construction and experimental execution remain unclear.
Critical weaknesses
-
The empirical record is internally inconsistent, preventing assessment of the central results. The abstract and Tables 1–2 present quantitative experimental comparisons and ablations. However, the AI Use Statement says, “The tables contain no generated or measured experimental results” (p. 10, lines 490–491). The Reproducibility Statement says “complete benchmark construction and empirical validation remain outstanding” (lines 508–509), describes comparisons as prospective, and contains unresolved appendix references. The appendices leave essential experimental settings—including sampling, repeated-run counts, endpoints, execution budgets, and consequential baseline permissions—as TBD (Tables 5–9). Section 6 describes confidence intervals, execution diagnostics, and human adjudication, but their corresponding results are not reported. The quantitative presentation also contains unresolved inconsistencies. The abstract reports GPT-5.6-sol exact-span recall of 4.26% versus 3.02%, whereas Table 1 reports 0.213 versus 0.178 for PaperSync and the agent scaffold. No stated denominator or normalization reconciles these figures. Table 4 contains literal placeholders and an inconsistent before/after/change entry. Outdated draft statements could explain these discrepancies. Nevertheless, the submitted document does not establish which results are measurements from completed experiments. Until this is reconciled, I cannot evaluate the claimed effectiveness or component contributions.
-
The central evidence-authorization mechanism is insufficiently operationalized, and its selective-revision benefit is not demonstrated. Section 4 specifies meaningful evidence-contract requirements, including applicable versions, counterevidence, and operation-specific obligations. What remains unclear is the concrete procedure that determines whether these requirements are satisfied: how functional correspondences and impact paths are selected, how support and conflict are assessed, and how these assessments produce authorization decisions. The appendices mainly list enabled components and defer the full prompts to a future release. The evaluation also leaves the distinguishing selective-revision claim unresolved. The authors explicitly restrict the benchmark to verifiable required updates rather than exhaustive PRESERVE or DEFER labels. This is a defensible scope restriction, but update recall alone cannot establish avoidance of unnecessary or unsupported revisions. The paper defines Non-target Edit Rate and Strict Case Success to assess preservation and complete updates without collateral changes (Section 6.2, lines 360–367), yet reports neither in the main comparison nor the ablations. Document-quality and PDF-density diagnostics cannot substitute for these outcomes. Consequently, even if the recall tables are authenticated, the presented evaluation does not establish that evidence authorization and localized patching deliver their claimed control over unwarranted edits.
Questions for authors
-
Which benchmark construction steps and experiments were completed for this submission? Please reconcile the abstract and Tables 1–2 with p. 10, identify the authoritative measured results, and provide the evaluated-case counts, configurations, run records, and uncertainty estimates supporting them.
-
What operational procedure implements evidence authorization? Please trace an actual case from versioned sources through alignment and impact detection to the authorization decision and patch, explaining how evidential support, incompatible conditions, and conflicts are assessed.
-
What completed evaluation establishes the selective-revision benefit? Please report the already-defined Non-target Edit Rate and Strict Case Success for the main comparison and relevant ablations, and delimit what the available evaluation establishes about preservation and deferral.
-
I feel like the problem formulation is potentially useful, but the supplied version does not provide a coherent, sufficiently substantiated empirical record. The central authorization mechanism also I feel like it lacks enough operational detail and supporting evaluation to establish the claimed contribution. Could you please comment on this? A reconciled experimental record and evidence for the selective-revision benefit could materially change my assessment. Is it available?