- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies two fundamental flaws in existing graph-based RAG systems: an intent-blind knowledge utilization gap where under-specified queries retrieve evidence misaligned with latent user needs, and a query-blind knowledge organization gap where static structurally-defined communities ignore downstream query distributions. To address these, the authors propose EchoRAG, a self-evolving graph RAG framework with two core mechanisms. First, query-guided collaborative community evolution starts from an initial Leiden-based community partition, continuously tracks query demand distribution, and triggers selective community updates (refresh or evolve) when demand shifts exceed thresholds, using a joint structural-semantic objective with LLM calls only when signals conflict. Second, intent-harmonized knowledge utilization infers latent user intent from each query, generates an intent blueprint and hypothetical answer template, combines it with the explicit query, and retrieves and refines evidence into grounded, intent-aligned evidence cards anchored to source spans.
Critical strengths
- Formalization and theoretical grounding of two distinct problems (query-blind organization, intent-blind utilization) with regret bounds, providing rare theoretical rigor in the RAG field.
- Closed-loop adaptive framework that unifies knowledge organization and utilization into a self-reinforcing system, representing an important break from the static one-time indexing paradigm.
- Cost-aware practical design that minimizes expensive LLM calls, using lightweight signals for routine decisions and invoking LLM only for conflict resolution, with tiered maintenance policies well-suited for real deployment.
Critical weaknesses
- Evolution trigger thresholds are key parameters but not deeply analyzed. How they should be tuned across different datasets, query distributions, and update frequencies, and how sensitive performance is to suboptimal settings, remain unclear.
- Limited long-term evolution analysis: self-evolution analysis shows only a few rounds; performance over longer time horizons with continuously shifting query distributions is not established.
- LLM dependency of intent inference: the intent harmonization step adds per-query LLM calls, increasing latency and cost. The paper shows cost advantages over some baselines but does not quantify per-query latency overhead.
- The paper ablates the two main modules but does not compare against simpler baselines such as query expansion for intent or demand-based community reweighting without structural changes.
Questions for authors
- How sensitive is the community evolution mechanism to the threshold parameters τ_D and τ_H? Were these tuned per dataset, and can you comment on their generalizability across different corpora and query workloads?
- The self-evolution analysis shows results for only a few rounds. How does EchoRAG behave over longer time horizons with continuously shifting query distributions? Is there convergence or a steady state?
- How much latency does the intent induction step add per query compared to standard graph RAG retrieval? Has the latency-performance tradeoff been evaluated across different query types?
- Have you considered simpler baselines for each mechanism, such as query expansion for intent or demand-based community reweighting without structural changes? How much of the total gain comes from the full evolution mechanism versus simpler adaptations?
3 comments
- We quantify both complete-pipeline and isolated intent-induction overhead: EchoRAG takes 33.12 seconds per MultiHop-RAG query, while intent induction averages 2.60 seconds over 500 queries. We would like to clarify that Appendix Table 8 already reports per-query latency. For a directly aligned comparison, we combine the MultiHop-RAG timing results with the token-consumption results from main-text Fig. 4 (time in seconds):
| Method | Index Time | Query Latency (s) | Prompt Tokens | Completion Tokens | Total Tokens |
|---|---|---|---|---|---|
| RAPTOR | 11,118 | 3.75 | 696,373 | 85,888 | 782,261 |
| ArchRAG | 39,669 | 5.16 | 1,945,896 | 100,440 | 2,046,336 |
| LinearRAG | 6,997 | 11.79 | 3,452,010 | 212,170 | 3,664,180 |
| HippoRAG 2 | 26,909 | 11.53 | 5,042,912 | 226,114 | 5,269,026 |
| LightRAG | 32,304 | 17.97 | 28,042,361 | 184,104 | 28,226,465 |
| DA-RAG | 3,075 | 162.61 | 159,901,691 | 371,799 | 160,273,490 |
| HyperRAG | 25,292 | 364.94 | 41,515,090 | 11,320,998 | 52,836,088 |
| CogRAG | 10,574 | 31.51 | 34,275,895 | 1,084,603 | 35,360,498 |
| GraphRAG | 516,419 | 30.13 | 12,763,383 | 374,964 | 13,138,347 |
| EchoRAG | 33,145 | 33.12 | 39,589,093 | 1,533,244 | 41,122,337 |
To isolate the overhead specifically attributable to intent induction, we additionally profiled 500 queries and report both average and tail latency across question types:
| Question Type | # Queries | Intent-Induction Latency (s) | P95 (s) |
|---|---|---|---|
| Temporal | 113 | 2.55 | 3.75 |
| Inference | 152 | 2.62 | 3.74 |
| Comparison | 168 | 2.57 | 3.55 |
| Null | 67 | 2.70 | 4.10 |
| All | 500 | 2.60 | 3.75 |
The mean intent-induction latency remains within 2.55--2.70 seconds across question types, with an overall mean of 2.60 seconds and P95 of 3.75 seconds. EchoRAG is not the lowest-overhead system: it incurs higher query latency and token consumption than lightweight baselines, yet uses fewer total tokens and substantially less query time than DA-RAG and HyperRAG, while delivering stronger performance. We therefore view EchoRAG as a performance-cost trade-off rather than claiming uniform efficiency. We will add the intent-induction profiling protocol and these results to the revised paper. Complete timing results for HotpotQA and NarrativeQA remain available in Appendix Table 8.
- Under the same 1,000-question protocol, EchoRAG consistently outperforms both Query Expansion and non-structural Demand Reweighting. Our original ablation focused on isolating the two proposed modules and therefore did not include alternative, non-structural adaptation strategies. Following the reviewer's suggestion, we added two controlled baselines: Query Expansion (QE) uses an LLM to generate semantically equivalent queries, while Demand Reweighting (DR) accumulates community demand and periodically reweights community retrieval without modifying the knowledge structure. All methods use the same 1,000-question subsets, initial index, generator, retrieval budget, prompts, and evaluator.
| Dataset | Method | Acc. | F1 | BLEU-1 | METEOR | Div. | Overall |
|---|---|---|---|---|---|---|---|
| MultiHop-RAG | EchoRAG | 73.7 | 69.1 | -- | -- | 67.8 | 58.0 |
| QE | 68.0 | 65.3 | -- | -- | 62.4 | 48.2 | |
| DR | 66.0 | 62.3 | -- | -- | 60.6 | 45.5 | |
| HotpotQA | EchoRAG | 85.1 | 91.6 | -- | -- | 78.3 | 69.8 |
| QE | 76.0 | 81.7 | -- | -- | 67.6 | 56.8 | |
| DR | 77.0 | 78.6 | -- | -- | 68.0 | 62.6 | |
| NarrativeQA | EchoRAG | 51.2 | -- | 10.2 | 17.3 | 47.9 | 34.6 |
| QE | 42.0 | -- | 8.5 | 13.3 | 39.3 | 31.0 | |
| DR | 45.0 | -- | 9.1 | 14.7 | 43.2 | 29.6 |
EchoRAG consistently outperforms both simpler alternatives, showing that query reformulation or demand-based retrieval reweighting alone does not reproduce the gains from structural evolution and intent-aware evidence utilization. We will report these baselines, their implementation details, and the above results in the revised paper.
Thank you for the additional experiments and clarifications. The intent-latency profiling and simpler baselines substantially solve my concerns. However, threshold sensitivity is evaluated only on MultiHop-RAG, and cumulative performance on a single 1,000-query stream does not yet establish robustness under sustained distribution shifts.I also noticed discrepancies between the QE/DR results in the rebuttal and Table 10 of the revised paper,please also clarify the corresponding configurations.
The results remain stable without performance collapse even under substantial deviations from the default. These thresholds jointly control maintenance frequency: lower values trigger more frequent updates, whereas higher values update more conservatively. We therefore use the intermediate (8,25)×10−3 setting across the main experiments to balance adaptation and maintenance, rather than tuning thresholds to individual test queries.
The long-term evolution shows stable, selective long-horizon behavior rather than uncontrolled reorganization: only 13 of 20 checkpoints trigger structural changes, totaling 36 node movements. We agree that examining longer sequential execution is valuable. Beyond Section 4.5, we tracked 1,000 sequential queries in their original order, recording community adjustments and cumulative QA performance at every 50-query checkpoint.
Across 20 checkpoints, only 13 trigger structural changes, totaling 36 node movements, while global modularity remains close to its initial value (0.9187 → 0.9171) and cumulative QA performance remains stable. The alternating active and inactive checkpoints show that EchoRAG continues to respond selectively as demand changes without repeatedly reorganizing the knowledge base. We will add this longer-horizon analysis and clarify that it demonstrates empirical stability on the observed stream rather than formal convergence under arbitrary future shifts.