DhiRAG: Learning Soft Hierarchical Indices for Retrieval-Augmented Generation

Hierarchical indices can provide corpus-level context for retrieval-augmented generation, but their usefulness depends on both how communities are constructed and how community information is combined with passage-level relevance. We present DhiRAG, a framework that learns soft hierarchical affiliations and uses them to contextualize evidence ranking. Starting from a semantic graph over text chunks, a graph neural assignment network is optimized using a differentiable structural-entropy objective. The learned affiliations serve two connected roles: they produce community prototypes through weighted aggregation of chunk embeddings, and they transfer query–community relevance to the original chunks. The resulting structural scores are combined with local matching signals to retrieve textual evidence, without generating natural-language community summaries. On PopQA, HotpotQA, 2WikiMultiHopQA, and MuSiQue, DhiRAG achieves the highest average Str-Acc and LLM-Acc among the evaluated methods, improving over the strongest average baseline by $2.4$ and $2.7$ percentage points, respectively. Retrieval-level evaluation and component ablations provide complementary evidence for the effectiveness of the learned index. These results support soft hierarchical context as a useful complement to local evidence matching.