- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RCT-PPR adapts Personalized PageRank's restart distribution and outgoing transitions during iterative retrieval for multi-hop QA. An LLM controller proposes seeds and a retrieval direction; entropic mirror descent accumulates relation–target feedback through multiplicative-weights updates. The graph remains fixed during a query, while retrieved evidence is accumulated for final chunk selection. A local static-regret bound characterizes the proxy alignment loss without providing an end-to-end retrieval guarantee. On three multi-hop QA collections, reported macro-F1 is 53.3 versus 46.9 for the evaluated HippoRAG 2 configuration, with additional topology, component, retrieval-round and model comparisons.
Critical strengths
-
Coherent formulation with bounded theoretical claims (PDF pp. 4–6 and 18–20, Algorithm 1 and Appendix D). The paper makes the feedback ordering explicit: a retrieval round precedes the controller feedback that updates transitions for the next round. It derives a normalized multiplicative-weights rule on each node's outgoing simplex and explains how cumulative target alignment enters the distribution. This provides a clear mathematical description of the adaptation, including the possibility of retaining misleading feedback. The standard regret result applies to the realized sequence of bounded proxy losses and a fixed comparator. The manuscript explicitly distinguishes that characterization from controller correctness, changing optimal routes, and successful evidence-chain recovery. This precision makes the formulation useful without requiring a stronger guarantee than the paper claims.
-
Multiple empirical checks support the complete system and several design choices (PDF pp. 7–9, Tables 3–8). The study goes beyond a headline benchmark table: it varies graph topology, removes components, sweeps retrieval rounds, evaluates a second LLM, reports paired full-system F1 comparisons, and inspects 50 unsuccessful cases. Relative to full-system F1 of 53.3, a single round yields 48.1 and removing transition adaptation yields 51.3. These are separate interventions, not additive decompositions of the total gain. The round sweep further shows that most average improvement arrives within three rounds, while the topology comparison supports separating entity diffusion from chunk selection in this implementation. Table 7 provides uncertainty estimates for full-system comparisons with HippoRAG 2 and IRCoT. The failure analysis usefully identifies graph, embedding and controller limitations, while appropriately treating its counts as diagnostic observations rather than population estimates.
Critical weaknesses
-
The headline gain does not yet establish the incremental value of cumulative adaptation (PDF p. 14, Appendix A.3; pp. 5, 8–9 and 19, Tables 4/7 and Appendix D.2). A.3 describes the evaluated HippoRAG 2 propagation as entity–chunk only, whereas the original method also includes relation and synonym edges (PDF pp. 4–5, Figure 2, Sections 3.1–3.2). It is important to establish whether this reflects the implementation or an incomplete description. If the implemented graph excludes those connections, the headline comparison also changes the available propagation structure; its gain would therefore not isolate the proposed adaptive retrieval mechanism. I cannot determine the size or direction of that configuration effect from the reported results.
Separately, Table 4 establishes that updating transitions helps relative to no update, but it does not isolate accumulating past targets from a memoryless target-conditioned rule. Both could benefit from the controller's current direction. The distinctive question is whether retaining earlier feedback improves retrieval beyond that simpler adaptation, particularly because previous targets can also be misleading. D.2 already explains the cumulative-versus-latest-target distinction mathematically, and the limitations acknowledge the need to compare transition rules. The local regret result does not resolve the empirical comparison: it evaluates proxy losses on the realized feedback sequence, not the alternative controller trajectory under another retrieval policy. Table 7's full-system statistics likewise do not quantify uncertainty in the 2.0-point update ablation. A latest-target softmax comparison with the same graph, controller, seeding rules and round budget would directly address this remaining attribution gap.
-
Several evaluation details are needed to interpret and reproduce the comparison (PDF pp. 6–7, Tables 1–2; pp. 12, 16–17 and 20, Appendices A.1/C.2/D.4). With 200 MuSiQue queries, one unweighted mean of binary EM outcomes would have 0.5-point increments, whereas Table 2 reports 29.2 and 26.3. A different denominator or run aggregation may explain this; the concern is that the actual within-dataset calculation is unclear, not that these values demonstrate an error. The stated macro-average across datasets does not resolve that question.
C.2 specifies three independent annotators, expert adjudication and the CFC aggregation formula, but not the actual assessed-answer counts and sampling for the reported comparisons. Those counts are needed to understand the coverage of this supplementary evaluation; Table 8's 50 failed cases describe a separate analysis. The numerical chunk length/overlap and units, fusion weight, and experimental transition-temperature setting are also not explicit. D.4 supplies theoretical temperature rules, so the missing information is which rule and numerical settings produced the reported results. Clarifying these points would make the evidence auditable without requiring a new evaluation protocol.
I lean toward weak rejection. The coherent formulation and substantial system-level evidence make this a meaningful contribution candidate. My main reservation is the strength of the evidence for cumulative adaptation beyond a simpler adaptive retriever, once the baseline configuration is established. A favorable matched memoryless comparison, with paired uncertainty for that comparison and the existing update ablation, could change my assessment. If the simpler rule performs similarly, the system's usefulness would remain supported, but the empirical case for the distinctive cumulative mechanism would be narrower. The reporting questions are relevant to interpreting the evidence and should be resolved, but are not evidence of incorrect results or the primary basis for my rating.
Questions for authors
- For W1 — baseline clarification: Does A.3 accurately describe the implemented HippoRAG 2 graph? If it is restricted to entity–chunk propagation, how does the original relation/synonym/context graph perform under the same corpus and generation setup? If the description is incomplete, please clarify it.
- For W1 — main contribution test: How does cumulative MWU compare with latest-target softmax under the same graph, controller, seeding rules and round budget? Paired uncertainty for this comparison and the existing no-update ablation would help establish the incremental benefit beyond iteration itself.
- For W2: Please specify MuSiQue's actual EM denominator and aggregation, including normalization/reference handling and run averaging if applicable; the CFC sample counts/sampling; and the numerical chunk, fusion and transition-temperature settings used for the reported results.
16 comments
3. HippoRAG 2 Graph Configuration and Fairness (W1, Q1)
We appreciate the reviewer's careful observation regarding the graph connectivity described in the original HippoRAG 2 paper.
As described in Appendix A.3, the evaluated HippoRAG 2 configuration uses entity–chunk connections for propagation, whereas the original method includes additional relation and synonym connections.
We acknowledge that this distinction may affect PPR propagation and therefore complicates attribution of the performance difference.
The reported 6.4-point improvement over the submitted HippoRAG 2 configuration should not be interpreted as an isolated effect of cumulative MWU, since the two systems also differ in graph representation, iterative retrieval, and other components.
To investigate the impact of graph configuration, our submitted Table 3 already evaluates several diffusion structures:
| Graph Configuration | HotpotQA | MuSiQue | 2Wiki | Avg. |
|---|---|---|---|---|
| Directed, entities + chunks | 58.7 | 41.1 | 57.2 | 52.3 |
| Directed, entity-only | 60.0 | 41.5 | 58.5 | 53.3 |
| Undirected, entities + chunks | 56.7 | 39.9 | 56.3 | 51.0 |
| Undirected, entity-only | 58.9 | 40.7 | 56.1 | 51.9 |
These experiments suggest that RCT-PPR maintains relatively strong performance under different graph topologies. In particular, the directed entities-plus-chunks configuration achieves 52.3 average F1, compared with 53.3 under the default entity-only configuration.
Nevertheless, we agree that these experiments do not substitute for a faithful evaluation of the original HippoRAG 2 graph.
Additional graph-configuration comparison
| Method / Configuration | HotpotQA F1 | MuSiQue F1 | 2Wiki F1 | Avg. F1 |
|---|---|---|---|---|
| HippoRAG 2 — submitted configuration | 54.4 | 32.7 | 53.7 | 46.9 |
| HippoRAG 2 — matched entity graph | 55.8 | 33.5 | 54.2 | 47.8 |
| HippoRAG 2 — original full graph | 56.2 | 34.8 | 55.1 | 48.7 |
| RCT-PPR — entity-only | 60.0 | 41.5 | 58.5 | 53.3 |
The additional experiments evaluate both matched entity-graph connectivity and the original full-graph configuration. The full-graph baseline achieves 48.7 average F1, compared with 53.3 for RCT-PPR, leaving a 4.6-point difference.
Matching entity-graph connectivity reduces the average performance gap from 6.4 to 5.5 F1 points. This indicates that graph connectivity explains part of the difference under these configurations, while a substantial system-level gap remains.
We agree that the original graph should be evaluated under the same corpus, query set, final generator, and output-context budget. We will make the distinction between the submitted, matched, and original configurations explicit, and avoid attributing system-level gains exclusively to the transition-update mechanism.
4. Clarification of EM Denominators and Evaluation Aggregation (W2, Q3)
We sincerely thank the reviewer for identifying the inconsistency between the nominal number of MuSiQue queries and the reported EM values.
We agree that, for a single-run binary Exact Match evaluation over exactly 200 questions, EM percentages should occur in increments of 0.5. Therefore, values such as 26.3% and 29.2% require clarification.
After inspecting our archived evaluation records and scripts, we identified an issue in the EM aggregation procedure.
Specifically, some archived prediction files contain fewer records than the nominal query count. The evaluation code can omit queries whose processing returns None, and the EM aggregation script additionally excludes entries with empty predictions or reference answers. The examples below contain missing records rather than empty saved predictions.
As a result, the effective EM denominator was sometimes slightly smaller than the nominal dataset size.
For example, our audit identified the following archived records:
| Method / Dataset | Exact Matches | Recorded Predictions | Computed EM |
|---|---|---|---|
| Contriever / HotpotQA | 113 | 299 | 37.79% |
| GTR / HotpotQA | 129 | 299 | 43.14% |
| Contriever / 2Wiki | 105 | 298 | 35.23% |
| HippoRAG 2 / MuSiQue | 52 | 198 | 26.26% |
These examples reproduce the corresponding reported EM values after rounding.
In particular, the archived HippoRAG 2 MuSiQue evaluation contains 52 exact matches among 198 recorded predictions:
This explains why the reported value does not follow the expected 0.5-point increments for 200 questions.
For the reproduced examples, this discrepancy is explained by the effective denominator; it does not require a different definition of Exact Match or non-binary partial credit. This finding does not establish that every anomalous table entry has the same source.
The denominator audit must also cover the other metrics. The inspected scripts average saved F1 and semantic-similarity scores over the available records, so omitted records can affect those denominators as well. Human-assessed CFC coverage must be checked against its annotation records separately. We therefore do not characterize this as an EM-only issue or claim that the other metrics are unaffected.
We acknowledge that excluding unsuccessful predictions from EM can introduce an inconsistency in evaluation coverage. We will correct this behavior by applying a consistent evaluation denominator and explicitly accounting for missing predictions.
We have not yet reconciled the reported RCT-PPR MuSiQue EM of 29.2% with a confirmed source prediction file; the precise source of the main-table value still requires confirmation. We will reconcile the original prediction records before reporting any corrected EM value.
This correction concerns evaluation coverage and aggregation. It does not require redefining Exact Match, but the affected metrics must be recomputed consistently from the appropriate prediction and annotation records.
We appreciate the reviewer for pointing out this issue and helping us improve the transparency and reproducibility of the evaluation.
5. CFC Annotation Coverage and Implementation Details (W2, Q3)
We thank the reviewer for identifying the remaining details required for reproducibility.
5.1. CFC annotation coverage
As described in Appendix C.2, the CFC evaluation uses three independent annotators, with correctness scores of 0, 0.5, or 1. Cases with disagreement are resolved through expert adjudication.
We agree that the existing description should also report the actual number of assessed answers and the sampling procedure.
The revised evaluation documentation should explicitly include:
- Number of assessed answers per dataset and method: not available in the supplied paper/rebuttal; retrieve from CFC annotation logs.
- Sampling protocol and annotation coverage: not established by the available records; verify the actual evaluation set and selection rule.
- Number of cases requiring adjudication: retrieve from original annotator disagreements and adjudication logs.
- Pre-adjudication inter-annotator agreement: calculate from independent pre-adjudication labels; cannot be reconstructed from adjudicated aggregates alone. We also clarify that the 50 unsuccessful cases in Table 8 belong to a separate qualitative failure analysis and should not be interpreted as the CFC annotation sample.
5.2. Chunking and fusion
We clarify the numerical settings and the distinction between the preprocessing units. The checked preprocessing code first constructs page-like segments using a regular-expression word/punctuation splitter with a target length of 500 units and a limited extension toward a sentence boundary. Within each segment, whitespace is normalized and the text is sliced into overlapping character windows. Thus, the final retrieval chunks are measured in characters, not model tokens or words.
| Parameter | Implementation setting |
|---|---|
| Maximum chunk length | 500 Unicode characters after whitespace normalization |
| Chunk overlap | 80 characters between consecutive chunks within a segment |
| Effective stride | 420 characters for full-length windows |
| Final-window tokenizer | None; Python string slicing |
| Preliminary segment splitter | Regex word/punctuation units; target 500 units with bounded sentence-boundary extension |
| Fusion weight | 0.5 for graph score and 0.5 for dense score |
| Dense score | Cosine similarity between the original query and candidate-chunk embeddings |
| Score normalization | Separate Min-Max normalization of graph and dense scores over the final candidate pool |
| Equal-score normalization | A constant score vector is mapped to an all-ones vector |
| Maximum retrieval rounds | 5 in the checked current retrieval implementation |
| Final selected chunks | Up to 5 |
| LLM decoding temperature | 0 for the reported online reasoning and answer-generation setting |
| Transition temperature | Scheduled in the checked current implementation; detailed in Section 5.3 |
The parameter named stride=80 in the chunking function actually specifies the overlap: the next window starts at the previous window's end minus 80 characters. Its effective full-window stride is therefore 500 - 80 = 420 characters. The last chunk of a segment can be shorter than 500 characters, and overlap is not imposed across segment boundaries.
We sincerely thank the reviewer for the careful, detailed, and constructive evaluation of our work. We particularly appreciate the recognition of our coherent mathematical formulation, appropriately scoped theoretical analysis, and extensive empirical evaluation.
We agree that the central question is whether cumulative transition adaptation provides additional benefits beyond simpler memoryless adaptation, particularly under a sufficiently faithful baseline configuration. We also appreciate the reviewer's careful examination of the evaluation details, which helped us identify an inconsistency in the EM aggregation procedure.
We address these concerns below, incorporating the existing results and additional controlled comparisons.
1. Incremental Benefit of Cumulative MWU over Current-target Softmax (W1, Q2) We thank the reviewer for identifying the need to distinguish historical feedback accumulation from memoryless transition reweighting. We agree that the existing w/o OCO ablation alone cannot fully establish the distinctive contribution of cumulative MWU.
The key difference is whether previous reasoning directions continue to influence subsequent graph diffusion.
Current-target Softmax computes transition probabilities using only the current semantic direction:
wt+1,j(v)=∑kexp(η⟨ev,k,rt⟩)exp(η⟨ev,j,rt⟩). In contrast, our cumulative MWU update preserves the previous transition distribution:
wt+1,j(v)=∑kwt,k(v)exp(η⟨ev,k,rt⟩)wt,j(v)exp(η⟨ev,j,rt⟩). With uniform initialization and a constant learning rate, this can be rewritten as:
wt+1,j(v)∝exp(η∑s=1t⟨ev,j,rs⟩). Thus, unlike memoryless Softmax, MWU accumulates semantic feedback across retrieval rounds, allowing previously identified directions to influence subsequent evidence exploration.
Controlled comparison of transition-update strategies
We conducted the comparison using identical graph connectivity, LLM controller, seed-update policy, retrieval-round budget, and final answer-generation setting, with the same temperature-tuning budget for the two adaptive rules.
The first and third rows are reported in Table 4; the additional current-target Softmax experiment isolates the value of accumulating historical feedback.
Current-target Softmax improves average F1 by 0.7 points over no transition adaptation, while cumulative MWU provides a further 1.3-point improvement.
The dataset-level differences between cumulative MWU and Softmax are +0.9, +0.8, and +2.4 F1 on HotpotQA, MuSiQue, and 2Wiki, respectively.
These results suggest that retaining previous feedback can provide benefits beyond adapting transitions to the current direction alone, particularly when multi-hop retrieval requires integrating information discovered across successive reasoning steps.
We also acknowledge the reviewer's observation that accumulating previous directions may preserve misleading feedback. Consequently, we do not claim that cumulative MWU must outperform memoryless Softmax in every retrieval setting.
Furthermore, the standard local static-regret guarantee applies to the realized sequence of semantic-alignment losses. It does not directly compare the different controller trajectories induced by alternative transition-update policies. The controlled empirical comparison is therefore essential for establishing the incremental practical value of cumulative adaptation.
2. Paired Statistical Analysis of Transition Adaptation (W1, Q2) We agree that the paired significance tests in Table 7 evaluate full-system differences against external baselines, rather than the incremental benefit of cumulative transition adaptation.
Our existing component ablation reports:
These results show consistent positive improvements, with the largest contribution on 2Wiki. However, as the reviewer correctly notes, positive aggregate differences do not necessarily establish statistical significance.