- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
RippleKB asks systems to find every source statement affected by a numeric edit across linked documents. It constructs 150 items from MuSiQue sources, assigns each a proposed dependency closure, and asks systems to rank the original source units. Its main metric checks whether the whole closure appears within a review budget of twice the target-set size. The paper also measures partial recall and exact copying. The experiments compare structural and lexical rankings, embedding reranking, and language models. They show that a system can recover much of a proposed closure while still missing a required unit. The reported scores cover machine-validated candidates and construction-defined closures. The appendix sets out a procedure for semantic review.
Critical strengths
- Complete recovery is a meaningful evaluation target. The results show that high partial recall does not necessarily imply recovery of the full affected set.
- The evaluation protocol is transparent. The paper clearly reports metric definitions, shortcut checks, parser behavior, and the current limits of the benchmark.
Critical weaknesses
- The proposed affected sets have not been semantically established. Section 3.3 reports scores against construction-defined closures. Appendix D describes independent review and adjudication but gives no completed outcomes. Graph and span checks cannot determine whether a statement truly changes under the edit or whether another affected statement has been omitted. Either error can change the all-or-nothing score. The present results therefore measure recovery of constructed labels; treating them as scores for actual edit impact requires reviewed targets and, where membership changes, revised scores.
- A construction cue remains untested. Affected downstream units must begin with their annotated entity; unaffected units have no such requirement. The reported identifier, position, lexical, and supporting-fact controls do not test this cue. The paper acknowledges it. Still, part of the language-model advantage could come from recognizing how target sentences were generated. Its predictive strength and effect on the model comparisons remain unknown.
Questions for authors
- Have the 150 proposed closures undergone the independent review and adjudication described in Appendix D? If so, what are the resulting agreement rates and how do the adjudicated targets affect the main scores?
- What performance is obtained by a baseline using sentence-initial entity features, and do the main results persist when this construction cue is balanced or removed?
2 comments
3. Sentence-initial entity baselines and cue-balanced evaluation (Q2). See the general response for the sentence-opening cue shares (75% of 681 affected downstream units vs 45% of 2194 unaffected units open with a multi-word capitalized name, AUC 0.65; 89% vs 65% for any name, AUC 0.62; template score AUC 0.55) and the rewriting procedure that balances entity-initial openings across unaffected units (130 of 150 items pass all mechanical checks, 20 excluded; post-balancing opening share 90% vs 100%, AUC 0.45). Across all 150 items, the two cue-only rankings (placing the edited sentence first) achieve Complete-closure@2G 24% [17, 31] (Recall@2G 0.75 [0.72, 0.78]) for Cue rule (i) (entity-initial opening) and 27% [19, 34] (Recall@2G 0.66 [0.61, 0.70]) for Cue rule (ii) (template features), beside BM25 (20% [14, 27] / 0.66 [0.63, 0.70]), Token overlap (13% [7, 18] / 0.68 [0.65, 0.71]), Llama 3.3 70B (65% [57, 73] / 0.91 [0.88, 0.93]), and Claude Sonnet 5 (82% [76, 88] / 0.95 [0.93, 0.97]).
Re-running the cue-only rankings, BM25, and four language models on the same 130 items before and after balancing under the same prompt, budget, and parser against the constructed dependency closures gives:
| System | Complete@2G original | Complete@2G balanced | Recall@2G original | Recall@2G balanced |
|---|---|---|---|---|
| Cue rule (i) | 25% [18, 33] | 11% [6, 16] | 0.75 [0.71, 0.78] | 0.66 [0.63, 0.70] |
| Cue rule (ii) | 27% [19, 35] | 8% [3, 12] | 0.66 [0.61, 0.71] | 0.50 [0.45, 0.54] |
| BM25 | 18% [12, 25] | 27% [19, 35] | 0.65 [0.61, 0.69] | 0.73 [0.69, 0.77] |
| Qwen3 32B | 54% [45, 62] | 48% [40, 57] | 0.72 [0.65, 0.79] | 0.70 [0.63, 0.77] |
| Llama 3.3 70B | 66% [58, 74] | 65% [56, 73] | 0.91 [0.89, 0.93] | 0.91 [0.88, 0.93] |
| Claude Sonnet 5 | 82% [75, 88] | 82% [75, 88] | 0.95 [0.93, 0.97] | 0.94 [0.92, 0.97] |
| Claude Fable 5.1 | 48% [40, 57] | 52% [43, 60] | 0.89 [0.86, 0.91] | 0.89 [0.87, 0.91] |
On the balanced items, the language models' scores stay within their intervals while the cue-only rankings fall.
We thank the reviewer for the review. To address your two concerns, we report: for the Appendix D review (Q1), its outcomes and inter-reviewer agreement rates from three model reviewers on all 150 items (item 1 below) and every system re-scored against the adjudicated closures beside its submitted scores (item 2 below); for the construction cue (Q2), sentence-initial entity baselines and a re-evaluation on cue-balanced items (Part 2). The cause of the review outcomes, worked examples, and the corrected-construction pilot are in the general response.
1. Appendix D review outcomes and agreement rates (Q1). Three model reviewers from different providers (Claude Fable 5.1, GPT-6 Astra, Grok 4.7) executed both stages of Appendix D on all 150 items (450 reviews; see the general response).
2. Scores on constructed vs reviewed closures (Q1). Re-scoring every retained response against the reviewed closures (111 of 150 containing only the edited sentence, so scores mostly reflect placing that sentence within the budget) gives: