acceptodds
Under review as a conference paper at ICLR 2027

RippleKB: Finding What an Edit Changes Across Linked Documents

Abstract

Editing one fact can change a derived quantity, comparison, or other statement across linked documents. RippleKB evaluates whether systems can recover a constructed set of affected source units completely. Each item edits one source fact in a small set of linked documents. The proposed answer key is a hidden dependency closure, with unaffected units also present. A system returns a ranked list of source units, word for word, under a review rule tied to the constructed affected-set size. We regenerate every item from a public multi-hop corpus and test identifier-order, document-position, lexical, and supporting-fact rankings, alongside role and template concentration checks. We release the audit protocol under which the proposed closure is to be reviewed unit by unit. We report how far retrieval-style and language-model baselines get toward recovering the constructed closure completely on the machine-validated candidate set; human validation was not obtained. Complete recovery tests whether a ranking includes the entire constructed target, while exact-copy measures separately assess preservation of source text.

Primary area:
datasets and benchmarks
Keywords:
benchmark, edit-impact recovery, multi-document reasoning, dependency closure, information retrieval, exact-copy evaluation, language-model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K84YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

RippleKB asks systems to find every source statement affected by a numeric edit across linked documents. It constructs 150 items from MuSiQue sources, assigns each a proposed dependency closure, and asks systems to rank the original source units. Its main metric checks whether the whole closure appears within a review budget of twice the target-set size. The paper also measures partial recall and exact copying. The experiments compare structural and lexical rankings, embedding reranking, and language models. They show that a system can recover much of a proposed closure while still missing a required unit. The reported scores cover machine-validated candidates and construction-defined closures. The appendix sets out a procedure for semantic review.

Critical strengths
  1. Complete recovery is a meaningful evaluation target. The results show that high partial recall does not necessarily imply recovery of the full affected set.
  2. The evaluation protocol is transparent. The paper clearly reports metric definitions, shortcut checks, parser behavior, and the current limits of the benchmark.
Critical weaknesses
  1. The proposed affected sets have not been semantically established. Section 3.3 reports scores against construction-defined closures. Appendix D describes independent review and adjudication but gives no completed outcomes. Graph and span checks cannot determine whether a statement truly changes under the edit or whether another affected statement has been omitted. Either error can change the all-or-nothing score. The present results therefore measure recovery of constructed labels; treating them as scores for actual edit impact requires reviewed targets and, where membership changes, revised scores.
  2. A construction cue remains untested. Affected downstream units must begin with their annotated entity; unaffected units have no such requirement. The reported identifier, position, lexical, and supporting-fact controls do not test this cue. The paper acknowledges it. Still, part of the language-model advantage could come from recognizing how target sentences were generated. Its predictive strength and effect on the model comparisons remain unknown.
Questions for authors
  1. Have the 150 proposed closures undergone the independent review and adjudication described in Appendix D? If so, what are the resulting agreement rates and how do the adjudicated targets affect the main scores?
  2. What performance is obtained by a baseline using sentence-initial entity features, and do the main results persist when this construction cue is balanced or removed?
2 comments

We thank the reviewer for the review. To address your two concerns, we report: for the Appendix D review (Q1), its outcomes and inter-reviewer agreement rates from three model reviewers on all 150 items (item 1 below) and every system re-scored against the adjudicated closures beside its submitted scores (item 2 below); for the construction cue (Q2), sentence-initial entity baselines and a re-evaluation on cue-balanced items (Part 2). The cause of the review outcomes, worked examples, and the corrected-construction pilot are in the general response.

1. Appendix D review outcomes and agreement rates (Q1). Three model reviewers from different providers (Claude Fable 5.1, GPT-6 Astra, Grok 4.7) executed both stages of Appendix D on all 150 items (450 reviews; see the general response).

  • Blind recovery vs constructed closures: exact-set match / mean Jaccard / mean blind-set size (edited-unit-only items) are 9% / 0.35 / 1.99 units (101) for Claude Fable 5.1, 4% / 0.29 / 1.63 units (112) for GPT-6 Astra, and 3% / 0.26 / 1.40 units (121) for Grok 4.7.
  • Inter-reviewer agreement: in blind recovery across 2875 non-edited units, pairwise mean Jaccard / Cohen's kappa are 0.89 / 0.69 (Claude Fable 5.1 / GPT-6 Astra), 0.88 / 0.47 (GPT-6 Astra / Grok 4.7), and 0.85 / 0.50 (Claude Fable 5.1 / Grok 4.7). In closure review (reject vs accept across all 831 constructed members), pairwise Cohen's kappa is 0.82, 0.78, and 0.73 for those same pairs (62 member and 2 omission single-vote disagreements; 581 of 681 downstream members [85%] removed, 0 of 150 direct removed, 1 omission added, 618 of 866 edges rejected).

2. Scores on constructed vs reviewed closures (Q1). Re-scoring every retained response against the reviewed closures (111 of 150 containing only the edited sentence, so scores mostly reflect placing that sentence within the budget) gives:

SystemComplete@2G (constructed)Recall@2G (constructed)Complete@2G (reviewed)Recall@2G (reviewed)
BM2520% [14, 27]0.66 [0.63, 0.70]79% [73, 85]0.90 [0.86, 0.93]
Token overlap13% [7, 18]0.68 [0.65, 0.71]49% [41, 57]0.60 [0.53, 0.67]
Identifier ascending / descending3% [1, 7] / 3% [1, 7]0.55 [0.52, 0.59] / 0.56 [0.52, 0.59]5% [2, 9] / 5% [2, 9]0.14 [0.10, 0.19] / 0.14 [0.10, 0.19]
Render / Random ordering0% [0, 0] / 1% [0, 3]0.57 [0.54, 0.59] / 0.54 [0.51, 0.57]7% [3, 11] / 7% [3, 11]0.17 [0.13, 0.22] / 0.14 [0.10, 0.19]
BM25 + embeddings13% [8, 19]0.68 [0.65, 0.71]35% [27, 43]0.44 [0.37, 0.51]
Qwen3 32B53% [45, 61]0.72 [0.66, 0.78]76% [69, 83]0.78 [0.71, 0.84]
Llama 3.3 70B65% [57, 73]0.91 [0.88, 0.93]95% [91, 99]0.98 [0.96, 0.99]
Claude Sonnet 4.5 / Grok 4.60% [0, 0] / 3% [1, 7]0.00 [0.00, 0.00] / 0.03 [0.01, 0.07]0% [0, 0] / 3% [1, 7]0.00 [0.00, 0.00] / 0.03 [0.01, 0.07]
Claude Sonnet 582% [76, 88]0.95 [0.93, 0.97]94% [90, 97]0.97 [0.94, 0.99]
Claude Opus 50% [0, 0]0.79 [0.78, 0.81]0% [0, 0]0.17 [0.13, 0.22]
Claude Fable 5.149% [41, 57]0.89 [0.87, 0.91]54% [46, 62]0.62 [0.55, 0.70]
Cue rule (i) / (ii) (added during the discussion period)24% [17, 31] / 27% [19, 34]0.75 [0.72, 0.78] / 0.66 [0.61, 0.70]81% [74, 87] / 78% [71, 85]0.92 [0.90, 0.95] / 0.89 [0.85, 0.92]
Grok 4.7 (added during the discussion period)2% [0, 5]0.03 [0.01, 0.05]3% [1, 5]0.03 [0.01, 0.05]
Llama 3.3 70B / Sonnet 5, two-pass (added during the discussion period)65% [57, 73] / 85% [79, 90]0.91 [0.88, 0.93] / 0.93 [0.89, 0.96]96% [93, 99] / 97% [93, 99]0.98 [0.97, 1.00] / 0.97 [0.93, 0.99]

3. Sentence-initial entity baselines and cue-balanced evaluation (Q2). See the general response for the sentence-opening cue shares (75% of 681 affected downstream units vs 45% of 2194 unaffected units open with a multi-word capitalized name, AUC 0.65; 89% vs 65% for any name, AUC 0.62; template score AUC 0.55) and the rewriting procedure that balances entity-initial openings across unaffected units (130 of 150 items pass all mechanical checks, 20 excluded; post-balancing opening share 90% vs 100%, AUC 0.45). Across all 150 items, the two cue-only rankings (placing the edited sentence first) achieve Complete-closure@2G 24% [17, 31] (Recall@2G 0.75 [0.72, 0.78]) for Cue rule (i) (entity-initial opening) and 27% [19, 34] (Recall@2G 0.66 [0.61, 0.70]) for Cue rule (ii) (template features), beside BM25 (20% [14, 27] / 0.66 [0.63, 0.70]), Token overlap (13% [7, 18] / 0.68 [0.65, 0.71]), Llama 3.3 70B (65% [57, 73] / 0.91 [0.88, 0.93]), and Claude Sonnet 5 (82% [76, 88] / 0.95 [0.93, 0.97]).

Re-running the cue-only rankings, BM25, and four language models on the same 130 items before and after balancing under the same prompt, budget, and parser against the constructed dependency closures gives:

SystemComplete@2G originalComplete@2G balancedRecall@2G originalRecall@2G balanced
Cue rule (i)25% [18, 33]11% [6, 16]0.75 [0.71, 0.78]0.66 [0.63, 0.70]
Cue rule (ii)27% [19, 35]8% [3, 12]0.66 [0.61, 0.71]0.50 [0.45, 0.54]
BM2518% [12, 25]27% [19, 35]0.65 [0.61, 0.69]0.73 [0.69, 0.77]
Qwen3 32B54% [45, 62]48% [40, 57]0.72 [0.65, 0.79]0.70 [0.63, 0.77]
Llama 3.3 70B66% [58, 74]65% [56, 73]0.91 [0.89, 0.93]0.91 [0.88, 0.93]
Claude Sonnet 582% [75, 88]82% [75, 88]0.95 [0.93, 0.97]0.94 [0.92, 0.97]
Claude Fable 5.148% [40, 57]52% [43, 60]0.89 [0.86, 0.91]0.89 [0.87, 0.91]

On the balanced items, the language models' scores stay within their intervals while the cue-only rankings fall.

Reviewer_bLXiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

RippleKB introduces 150 automatically constructed items for identifying statements affected by a numeric edit across linked documents. Systems rank candidate units and reproduce their original text. Evaluation measures partial recall and complete recovery against constructed dependency closures, with separate copying-fidelity metrics. The results illustrate that high recall can coexist with incomplete recovery.

Critical strengths
  • Complete recovery is a useful objective for document maintenance, where missing an affected statement can leave an inconsistency unresolved. Llama reaches 0.91 recall but completes only 65% of the constructed targets, a distinction that average recall alone would not capture.

  • The evaluation distinguishes selection errors from copying errors and examines formatting failures through rescoring of the same responses. The authors also disclose construction revisions, later model additions, and changes to the evaluation population.

Critical weaknesses
  • The main concern is the semantic validity of the target closures. The OpenReview abstract explicitly acknowledges that human validation was not obtained, and the paper provides a review protocol without completed validation results. Mechanical checks verify structural properties without establishing that every included statement changes under the particular edit, or that every affected statement is included. Since complete recovery depends on these membership decisions, it is unclear whether completion scores reliably measure recovery of the affected statements.

  • A construction cue also remains untested. Affected downstream units must begin with their annotated entity, whereas unaffected units have unconstrained openings. The authors acknowledge this potential cue, but the reported controls do not measure its predictive value. It therefore remains unclear how much performance depends on tracing edit consequences rather than recognizing construction-specific sentence patterns.

  • Finally, the supplied materials are insufficient for independent result recomputation. Appendices A and K identify missing item-level records and executable components needed for replay. Rebuilding figures from saved aggregates does not verify the underlying scoring or uncertainty calculations.

Questions for authors
  1. Is completed independent semantic validation of the target closures available? If so, what incorrectly included units, omissions, and reviewer disagreements were observed?
  2. Has a baseline using only sentence-opening or template features been evaluated? How much affected-set recovery can it achieve?
  3. Are the benchmark items, target closures, retained model responses, and executable scorer available for independent inspection and recomputation?
2 comments

We thank the reviewer for the review. To address your three concerns, we report: for the semantic validity of the target closures (Q1), the released closure-review protocol applied to all 150 items by three model reviewers, with the removed members, omissions, and reviewer disagreements (item 1 below); for the construction cue (Q2), two cue-only rankings and a cue-balanced re-evaluation (Part 2); and for independent recomputation (Q3), the replay archive and what its replay reproduces (Part 2). The cause of the review outcomes, the scores against the reviewed closures, and the corrected-construction pilot are in the general response.

1. Model-based review under the released protocol: removed members, omissions, and disagreements (Q1). Three model reviewers from different providers (Claude Fable 5.1, GPT-6 Astra, and Grok 4.7) applied the released protocol to all 150 items (450 reviews, all parsed; see the general response for the cause, worked examples, and reviewed-closure scores). Under a two-of-three rule, 581 of 681 downstream members (85%) are removed, 0 of 150 directly edited sentences are removed, 1 omission is added, and 618 of 866 proposed edges are rejected by at least two reviewers (144 of 150 closures change and 6 are unchanged; adjudicated closures average 1.67 units vs 5.54 constructed, and 111 of 150 reduce to the edited sentence alone):

ReviewerBlind set sizeEdited unit onlyExact match / JaccardMember reject rateVALID verdicts / Edges rejected
Claude Fable 5.11.991019% / 0.350.652 / 560
GPT-6 Astra1.631124% / 0.290.710 / 599
Grok 4.71.401213% / 0.260.750 / 641
Adjudicated (2 of 3)1.67111—0.70— / 618

Disagreements (exactly one vote) occur on 62 constructed members and 2 proposed omissions (pairwise closure-review Cohen's kappa over 831 constructed members is 0.82 for Claude Fable 5.1 / GPT-6 Astra, 0.78 for GPT-6 Astra / Grok 4.7, and 0.73 for Claude Fable 5.1 / Grok 4.7; blind-recovery mean Jaccard / kappa over 2875 non-edited units are 0.89 / 0.69, 0.88 / 0.47, and 0.85 / 0.50; leave-one-out removals excluding each reviewer are 570 of 681 [142 items change], 532 of 681 [136 change], and 535 of 681 [138 change]).

2. Sentence-opening and template baselines (Q2). Across all 681 affected downstream units and 2194 unaffected units, 75% of affected downstream units vs 45% of unaffected units open with a multi-word capitalized name (89% vs 65% with single-word names included; membership-classifier AUC over non-edited units is 0.65 for multi-word name, 0.62 for any name, and 0.55 for the template score). Two cue-only rankings (placing the edited sentence first, as named by the edit) on all 150 items against the constructed dependency closures give:

RankingComplete@2GRecall@2G
Cue rule (i): entity-initial opening (added during the discussion period)24% [17, 31]0.75 [0.72, 0.78]
Cue rule (ii): template features (added during the discussion period)27% [19, 34]0.66 [0.61, 0.70]
BM25 (submitted)20% [14, 27]0.66 [0.63, 0.70]
Token overlap (submitted)13% [7, 18]0.68 [0.65, 0.71]
Llama 3.3 70B (submitted)65% [57, 73]0.91 [0.88, 0.93]
Claude Sonnet 5 (submitted)82% [76, 88]0.95 [0.93, 0.97]

On the 130 cue-balanced items where unaffected units are rewritten to open with their named entity (90% vs 100% capitalized-name openings, AUC 0.45; see the general response), the cue-only rankings fall from 25% [18, 33] to 11% [6, 16] and from 27% [19, 35] to 8% [3, 12] Complete-closure@2G while the language models' scores stay within their intervals.

3. Archive for independent inspection and recomputation (Q3). We assembled a replay archive (383 files) that contains every component Appendices A, E.2, and K list as absent, and also holds the 30 pilot items with their constructed and reviewed closures, the pilot review records, and the pilot scores. We will release it with the next version of the paper. Its contents:

  • Item-level benchmark records and target closures: all 150 candidate items, full private source mappings and span annotations, original source-isolation records, and both the constructed dependency closures and the reviewed closures.
  • Retained raw model responses and closure-review records: every retained raw response across all evaluated systems and all 450 model-review records from the protocol execution.
  • Executable code and one-command replay: the executable scorer, the 10,000-replicate source-cluster bootstrap, both the primary and fence-stripping parsers, the construction code, and a one-command replay script. Running the replay from the archive reproduces the primary-recovery table (Table 10; all 14 systems item by item) and the copying-fidelity table (Table 15) exactly.
Reviewer_1e78OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RippleKB studies change-impact retrieval across linked documents. The benchmark contains 150 machine-validated English items derived from MuSiQue, each mixing affected and unaffected units across three or four documents, and includes structural, positional, lexical, and supporting-fact controls.

Critical strengths

The paper addresses a meaningful challenge in knowledge maintenance: identifying all statements affected by a factual edit across linked documents. This goes beyond conventional knowledge editing and multi-hop question answering.

Critical weaknesses

Insufficient validation of the ground truth. The benchmark constructs dependency graphs before generating the corresponding statements, and graph reachability determines the ground-truth impact set. However, a dependency edge does not necessarily imply that an edit changes the truth or content of a downstream statement. Although the authors provide a human-review protocol, no completed human validation results are reported. This is particularly concerning because the central metric, complete-closure recovery, depends entirely on the correctness and completeness of the ground truth.

Limited difficulty for frontier LLMs. Claude Sonnet 5, released just four months ago, already achieves 82% complete recovery and 95% recall with a single inference pass, without task-specific optimization or iterative verification. Given the small candidate sets (14–26 units) and short dependency chains, the benchmark may be approaching saturation. The absence of stronger, more recent models or reasoning-based baselines makes it difficult to establish whether RippleKB remains challenging for frontier LLMs.

Questions for authors

Have the authors conducted human validation of the constructed dependency closures? What proportion of generated dependencies are semantically valid, and how often do annotators identify missing or incorrectly included affected statements?

Given that Sonnet 5 already achieves 82% complete recovery, how much headroom remains for stronger frontier models? Can the authors evaluate more recent LLMs and iterative reasoning or self-verification approaches to demonstrate that RippleKB is not already close to saturation?

How can the authors rule out the possibility that models exploit systematic differences between affected and unaffected statements? Would performance remain comparable if both categories were generated using identical stylistic and structural constraints?

How would the proposed evaluation extend to larger document collections, non-numerical edits, and dependencies that are not explicitly represented in the text?

3 comments

We thank the reviewer for the review. To address your concerns about the validity of the constructed closures (Q1) and the benchmark's difficulty for frontier models (Q2), we report the released closure-review protocol applied to all 150 items by three model reviewers (item 1), and, for headroom, two-pass self-verification for Claude Sonnet 5 and Llama 3.3 70B, a run of Grok 4.7, and scores against the reviewed closures (item 2). Part 2 covers systematic differences between affected and unaffected units (Q3), the scope of the construction (Q4), and terminology. The cause of the review outcomes, worked examples, and the corrected-construction pilot are in the general response.

1. Closure review and dependency validity (Q1). Three model reviewers from different providers (Claude Fable 5.1, GPT-6 Astra, and Grok 4.7) applied the released protocol to every constructed closure across all 150 items (see the general response for agreement, leave-one-out checks, cause, worked examples, and reviewed-closure scores). Under two-of-three adjudication across the 450 reviews, 581 of 681 constructed downstream members (85%) are removed, 0 of 150 directly edited sentences are removed, 1 omission is added, and 618 of 866 proposed edges are rejected by at least two reviewers (2 VALID verdicts of 450 reviews; single-vote disagreements on 62 constructed members and 2 omissions). Overall, 144 of 150 closures change and 6 are unchanged; adjudicated closures average 1.67 units (constructed 5.54), and 111 of 150 reduce to the edited sentence alone.

2. Headroom, newer model, self-verification, and misses (Q2). Against the constructed closures, Claude Sonnet 5 reaches 82% [76, 88] Complete-closure@2G (Recall@2G 0.95 [0.93, 0.97]) in a single pass and 85% [79, 90] (0.93 [0.89, 0.96]) under two-pass self-verification (where the model sees its own ranking, checks every unit, and re-ranks); against the reviewed closures (111 of 150 consisting of the edited sentence alone), the task is near-trivial (94% [90, 97] single-pass and 97% [93, 99] two-pass for Claude Sonnet 5; 95% [91, 99] single-pass and 96% [93, 99] two-pass for Llama 3.3 70B). We also ran Grok 4.7 under the submitted prompt, 4,096-token cap, and primary parser:

SystemParsedComplete@2GRecall@2G
Grok 4.7 (added during the discussion period)42% [0, 5]0.03 [0.01, 0.05]
Claude Sonnet 5, single pass (submitted)15082% [76, 88]0.95 [0.93, 0.97]
Claude Sonnet 5, two-pass (added during the discussion period)14585% [79, 90]0.93 [0.89, 0.96]
Llama 3.3 70B, single pass (submitted)15065% [57, 73]0.91 [0.88, 0.93]
Llama 3.3 70B, two-pass (added during the discussion period)15065% [57, 73]0.91 [0.88, 0.93]

Across the 150 items (14–26 candidate units; maximum depth 42 at 3, 72 at 4, 36 at 5; ∣G∣|G| of 4 on 23 items, 5 on 56, 6 on 38, 7 on 33), 17 of Claude Sonnet 5's incomplete items miss one unit and 10 miss two or more, while the directly edited sentence falls outside the budget on 45% of items for Claude Fable 5.1 and 100% for Claude Opus 5:

SystemComplete@2G, size 4 (23)Complete@2G, size 5 (56)Complete@2G, size 6 (38)Complete@2G, size 7 (33)Edited sentence missed
Claude Sonnet 578%82%76%91%6%
Llama 3.3 70B52%66%66%73%0%
Claude Fable 5.165%48%45%42%45%
BM259%20%18%30%0%

3. Ruling out systematic stylistic and structural differences (Q3). See the general response for the sentence-opening cue measurements (75% of 681 affected downstream units vs 45% of 2194 unaffected units open with a multi-word capitalized name, AUC 0.65; cue-only rankings reach 24% [17, 31] and 27% [19, 34] Complete-closure@2G) and the rewriting procedure that enforces entity-initial openings across unaffected units (130 of 150 items pass all mechanical checks, 20 excluded; post-balancing entity-initial share 90% affected downstream vs 100% unaffected, AUC 0.45). Re-evaluating on the same 130 items before and after balancing against the constructed dependency closures gives:

SystemComplete@2G originalComplete@2G balancedRecall@2G originalRecall@2G balanced
Cue rule (i)25% [18, 33]11% [6, 16]0.75 [0.71, 0.78]0.66 [0.63, 0.70]
Cue rule (ii)27% [19, 35]8% [3, 12]0.66 [0.61, 0.71]0.50 [0.45, 0.54]
BM2518% [12, 25]27% [19, 35]0.65 [0.61, 0.69]0.73 [0.69, 0.77]
Qwen3 32B54% [45, 62]48% [40, 57]0.72 [0.65, 0.79]0.70 [0.63, 0.77]
Llama 3.3 70B66% [58, 74]65% [56, 73]0.91 [0.89, 0.93]0.91 [0.88, 0.93]
Claude Sonnet 582% [75, 88]82% [75, 88]0.95 [0.93, 0.97]0.94 [0.92, 0.97]
Claude Fable 5.148% [40, 57]52% [43, 60]0.89 [0.86, 0.91]0.89 [0.87, 0.91]

On the balanced items, the language models' scores stay within their intervals while the cue-only rankings fall.

4. Extension to larger collections, non-numerical edits, and unstated dependencies (Q4). The current construction generates 150 items—75 with 3 documents and 75 with 4 documents—containing 14–26 candidate units across eight dependency shapes (Section 3.4 and Appendix B.3). Every edit replaces one number in one source sentence with the smallest larger integer whose rendering is absent from the sentence (an increment of one on 145 items, two on 4, and three on 1; 71 edited values are years). Proposed dependencies are hidden graph edges (866 in total, 716 between units); the documents state them in prose and systems see only the prose. This is the scope the construction generates today: numeric edits in three- and four-document collections whose dependencies are stated in the text (Section 7).

5. Presentation and terminology clarity. In the next version, Section 2 will define the intended impact set and the constructed dependency closure once, and the empirical results in Sections 1–7 will refer directly to the constructed dependency closure, with the reviewed closures reported beside them.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.

However, I am still quite concerned about the benchmark's validity and difficulty. The new results actually seem to reinforce my original concerns.

First, 144 out of 150 closures were changed after review, and 85% of the downstream dependencies were removed. This is a pretty substantial issue with the original ground truth, especially since the review was conducted entirely by LLMs rather than human annotators.

Second, after correcting the ground truth, Sonnet 5 already achieves 97% complete recovery, and even Llama 3.3 70B reaches 95% in a single pass. More importantly, 111 out of 150 items no longer require any downstream reasoning.

So I'm still not convinced that RippleKB provides a sufficiently reliable or challenging benchmark. I think the authors should at least report performance on the remaining 39 items with actual downstream dependencies and provide human validation of those closures.

Comments on the paper

2 comments

We thank the reviewers for their reviews. All three asked whether the constructed dependency closures are semantically valid and whether systems can exploit how affected and unaffected units were written; Reviewer 1e78 also asked about headroom and scope, and Reviewer bLXi whether the results can be recomputed independently. To address these concerns we report, all from the discussion period: the released closure-review protocol (Appendix D) applied to all 150 items by three model reviewers from different providers (Claude Fable 5.1, GPT-6 Astra, Grok 4.7; 450 reviews, all parsed), with its outcomes, their cause, and the scores on the reviewed closures (items 1–2 below); a 30-coordinate corrected-construction pilot, sentence-opening cue baselines with a cue-balanced re-evaluation, and the replay archive (Part 2); and a Grok 4.7 run and two-pass self-verification (our response to Reviewer 1e78). The next version of the paper will report the review outcomes and their cause in the abstract, the results, and the limitations, and will describe every submitted score as recovery of the constructed dependency closures.

1. Closure review outcomes, reviewer overlap checks, and cause. Each model reviewer applied blind recovery followed by unit-by-unit and edge-by-edge closure review to all 150 items. Under a two-of-three rule (a constructed member leaves when two of three reject it; a unit enters when two of three propose it), 581 of 681 downstream members (85%) are removed, 0 of 150 directly edited sentences are removed, 1 omission is added, and 618 of 866 proposed edges are rejected by at least two reviewers (2 VALID verdicts of 450 reviews; single-vote disagreements on 62 constructed members and 2 proposed omissions). Overall, 144 of 150 closures change and 6 are unchanged; adjudicated closures average 1.67 units (constructed 5.54), and 111 of 150 reduce to the edited sentence alone. Because Claude Fable 5.1 is an evaluated system in the primary-recovery table (Table 10) and Grok 4.7 is a discussion-period row while GPT-6 Astra is not evaluated, leave-one-out pairs counting a member as removed only when both remaining reviewers reject it remove 570 of 681 downstream members (142 items change) for GPT-6 Astra and Grok 4.7, 532 of 681 (136 items change) for Claude Fable 5.1 and GPT-6 Astra, and 535 of 681 (138 items change) for Claude Fable 5.1 and Grok 4.7.

Most downstream members are removed because every edit adds the smallest integer increment absent from the sentence (one on 145 items, two on 4, three on 1; 71 edited values are years), while constructed dependents are stated at a coarser grain (status, period, rank, threshold), so an edit of one rarely changes them; the 6 unchanged closures are those whose dependents refer to the value itself (a threshold at the edited value, an address digit, the following season). For example, editing "Also, its drummer was Seb Rochford, who won the BBC Rising Star Jazz Award in 2004 and leads the Mercury Prize-nominated experimental-jazz group Polar Bear." (2004 → 2005) leaves "Seb Rochford is recognized as a recipient of the BBC Rising Star Jazz Award." true, and all three reviewers reject it; conversely, editing "In Canada, the community of Swedish ancestry is 330,000 strong..." (330,000 → 330,001) flips "The Swedish-Canadian diaspora constitutes a population group that exceeds the three-hundred-thirty-thousand threshold." from false to true, and all three reviewers accept it.

2. Scores on reviewed closures. All submitted scores on the constructed dependency closures remain unchanged; our response to Reviewer K84Y (Part 1/2) lists every system's score on the reviewed closures beside them. The reviewed-closure column is a diagnostic, not a ranking: against reviewed closures (111 of 150 consisting of the edited sentence alone), the task reduces largely to placing the edited sentence within the review budget, which is why a lexical ranking scores high (BM25 Complete-closure@2G 20% [14, 27] constructed to 79% [73, 85] reviewed; Recall@2G 0.66 [0.63, 0.70] to 0.90 [0.86, 0.93]). Across the 14 systems in the primary-recovery table (Table 10), 10 pairwise complete-closure point-estimate orderings reverse on the reviewed closures, including Claude Fable 5.1 (49% [41, 57] to 54% [46, 62]) and Qwen3 32B (53% [45, 61] to 76% [69, 83]) vs BM25 (20% to 79%), and Claude Sonnet 5 (82% [76, 88] to 94% [90, 97]) vs Llama 3.3 70B (65% [57, 73] to 95% [91, 99]).

  • Corrected-construction pilot (30 coordinates, never pooled with the 150 items): Rebuilding the same 30 selected coordinates with two changes only—each edit moves the value far enough to cross a stated boundary, and each affected statement must be true before and false after the edit by a rule written in the documents—the same three-model review and two-of-three rule remove 70 of 133 constructed downstream members (53%, vs 105 of 133, 79%, on the original items of the same coordinates) and change 22 of 30 pilot closures (vs 28 of 30; mean closure size 5.43 to 3.17 on the pilot vs 5.43 to 1.93 on the original items; 10 vs 18 reviewed closures reduced to the edited sentence alone). Complete-closure@2G against each set's own constructed closures (original → pilot) is: BM25 23% [10, 40] → 7% [0, 17]; Token overlap 3% [0, 10] → 10% [0, 20]; Render order 0% [0, 0] → 0% [0, 0]; Random ordering 0% [0, 0] → 3% [0, 10]; Claude Fable 5.1 43% [27, 60] → 67% [50, 83] (30 of 30 pilot responses parsed); Claude Opus 5 0% [0, 0] → 3% [0, 10] (30 of 30 parsed); Claude Sonnet 5 77% [60, 90] → 90% [80, 100] (29 of 30 parsed); Qwen3 32B 50% [33, 67] → 40% [23, 57] (resting on 15 of 30 pilot responses parsed by the primary parser); and Llama 3.3 70B 63% [47, 80] → 80% [67, 93] (30 of 30 parsed).

3. Sentence-opening cue and balanced items. Multi-word capitalized names open 75% of affected downstream units (681) vs 45% of unaffected units (2194; AUC 0.65; with single-word names included: 89% vs 65%, AUC 0.62; template score: AUC 0.55). Two cue-only rankings (placing the edited sentence first) reach Complete-closure@2G 24% [17, 31] (Recall@2G 0.75 [0.72, 0.78]) and 27% [19, 34] (Recall@2G 0.66 [0.61, 0.70]). Rewriting unaffected units without capitalized-name openings to open with their named entity via an unevaluated model family (747 of 769 targeted units accepted under mechanical checks requiring the same numbers, the edit's values absent, and most content words kept; 130 of 150 items pass every check, 20 items with a unit that failed three attempts excluded) yields 90% vs 100% capitalized-name openings (AUC 0.45). On these 130 items, the language models' scores stay within their intervals on the balanced items (Qwen3 32B 54% [45, 62] to 48% [40, 57], Llama 3.3 70B 66% [58, 74] to 65% [56, 73], Claude Sonnet 5 82% [75, 88] to 82% [75, 88], Claude Fable 5.1 48% [40, 57] to 52% [43, 60]) while the cue-only rankings fall (25% [18, 33] to 11% [6, 16] and 27% [19, 35] to 8% [3, 12]).

4. Replay archive. We assembled a replay archive (383 files) that holds the 150 items, constructed and reviewed closures, retained raw responses, the scorer, the bootstrap, both parsers, the construction code, the closure-review records, the 30 pilot items with their constructed and reviewed closures, the pilot review records, and the pilot scores, and a one-command replay that reproduces the primary-recovery table (Table 10; all 14 systems item by item) and the copying-fidelity table (Table 15) exactly. We will release it with the next version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.