- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
DhiRAG builds a KNN graph over chunks and trains a GCN assigner with a differentiable structural-entropy (SE) loss to get soft hierarchical chunk-to-community affiliations. These are reused to (i) form community prototypes without LLM summaries and (ii) give chunks a structural prior that is fused with dense+lexical scores (γ=0.2). It reports the best average accuracy on PopQA, HotpotQA, 2Wiki, and MuSiQue.
The efficiency-oriented direction is good and the performance looks nice. However, the novelty is limited (essentially GNN + SeRAG), no code is available, the paper is not rigorously written and contains many numerical errors, and important ablations are missing.
Critical strengths
- Summary-free indexing gives low token cost (Table 7).
- Using the same soft affiliations for both prototypes and scoring is a clean design, and single-pass ranking avoids hard-filtering errors.
- It achieves the best average accuracy across four benchmarks, and the paper includes retrieval-level metrics and run-to-run variance.
Critical weaknesses
Major
- Limited novelty. SeRAG already does SE-based, summary-free hierarchical indexing, and the differentiable SE formulation follows LSEnet/HyperSED. The contribution reduces to adding a GNN and soft assignments.
- Missing ablations.
- No γ=0 result (hybrid scoring only). Since 80% of the score comes from the dense+lexical term, it is unclear whether the hierarchy helps at all.
- No ablation of the lexical/NER term, and SeRAG is not run with the same hybrid scoring.
- No comparison of the SE objective against Leiden or k-means.
- No sensitivity analysis for λ, τ, H, K, or γ, even though the paper claims sensitivity analyses.
- "Soft" is unverified. The orthogonality regularizer and low-τ Gumbel-Softmax both push assignments toward one-hot, and the paper never reports how soft the learned S actually is.
- Possible unfair comparison. A 1,200-token chunk merges about 8–9 HotpotQA passages (about 1,073 chunks vs. 9,221 passages), so top-3 retrieval and Hit@3 may not be comparable with passage-level baselines.
- No code, and key details are missing: the lexical/NER scoring, which level is used in Eq. 10, N_h, λ, and τ.
- Efficiency is overclaimed. DhiRAG is slower than SeRAG in both indexing (939 s vs. 771 s) and retrieval (0.433 s vs. 0.188 s).
Minor
- Table 1 caption says "three datasets"; there are four.
- all-MiniLM-L6-v2 is 384-d, but Table 5 says 768.
- HotpotQA passages: 9,221 (Table 4) vs. 9,811 (A.6).
- A.3 says pools range from 8,676 to 11,656, but 2Wiki has 6,119. A.3 also mentions "conversational contexts" that are not evaluated.
- Indexing time: 939.5 s (Table 7) vs. about 15 min + 78 s (Table 8).
- "HOC dilemma" (RQ2) is undefined in the main text.
- Overclaims: "guarantees", "purely attributable", "sets a new standard" (the method is second on HotpotQA).
- LaTeX errors ("Equation equation 1"); the indexing in Eq. 3 is confusing.
References
- DiffPool (Ying et al., 2018); MinCutPool (Bianchi et al., 2020): the CᵀAC pooling and orthogonality regularizer closely mirror these.
- SEP (Wu et al., ICML 2022): structural-entropy-guided graph pooling.
- Leiden (Traag et al., 2019): a natural clustering baseline.
Questions for authors
- What are the results with γ=0 and without the lexical/NER term? How does SeRAG perform with the same hybrid scoring?
- What is the entropy or overlap of the learned S per level?
- Is "w/o Soft Assignment" the argmax of the trained S, or retrained with hard assignments?
- Which level(s) are used in Eq. 10, and how is N_h chosen?
- What is the retrieval unit and context length for each baseline?
- Will the code be released?