- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WARP-G selectively builds regional GraphRAG indexes using workload information. It forms passage regions, probes regional HippoRAG2 indexes with evidence-annotated design queries, and retains combinations with positive conditional retrieval utility. Full-corpus passage retrieval remains available when graph augmentation is unhelpful or unavailable. The experiments use 800 design and 200 held-out test queries per dataset across four QA workloads. WARP-G retains zero to two graphs and reports 38.7–67.2% lower first-use graph-token cost than Full Graph, including discarded probes and design evaluations. Selection and partition controls examine the resulting retrieval quality, retained-index size, and design costs.
Critical strengths
-
A useful deployment problem with careful graph-cost accounting (PDF pp. 5–8, Sections 3.4–4.3; pp. 15–18, Appendix A.3 and Tables 6–8). The method treats graph construction as an optional investment while preserving passage access over the full corpus. This is a useful formulation because a deployment can decline graph augmentation without giving up its basic retrieval coverage. The reported first-use accounting includes discarded probes and conditional evaluations, counts reused indexes once, and distinguishes logical token counts from physical usage after caching. These savings therefore account for design work even when the resulting probe indexes are not retained. The cost boundary is also explicit: it covers graph design and online graph retrieval for 200 test queries, while shared passage indexing, reranking, and answer generation are excluded. Within that scope, the 38.7–67.2% savings are meaningful evidence that selective regional provisioning can substantially reduce graph expenditure relative to constructing a full-corpus graph. They should be interpreted as graph-token savings rather than an equivalent reduction in all end-to-end serving costs.
-
Transparent results that expose workload-dependent trade-offs (PDF pp. 7–9 and 19–20, Tables 1–2 and 9–11). On 2Wiki, the retained graph uses 8.2% of Full Graph's construction tokens and obtains answer F1 of 64.21, compared with Hybrid's 63.06 and Full Graph's 62.47. The paper reports zero-graph outcomes on HotpotQA and NQ rather than assuming that graph construction must always help. Frequency-only, gain-only, and partition controls further show that smaller retained indexes, lower first-use cost, and higher retrieval scores are different objectives. Section 4.5 explains the distinction between filling an externally given capacity and spending design effort to select capacity automatically. The retrieval-depth and routing diagnostics also help explain why reaching an evidence region need not translate into better answer quality. This reporting makes the method's intended use and its limitations substantially easier to assess.
Critical weaknesses
-
The incremental value of choosing graph capacity automatically is not yet established by an independent capacity comparison (PDF pp. 5, 8–9 and 18–20, Sections 3.4/4.5 and Tables 2/9). The frequency-only, gain-only, and random-selection controls inherit WARP-G's retained-region count. These are informative tests of which regions to select at a given capacity, but they do not independently evaluate how many graphs should be retained. This distinction matters because choosing capacity, including choosing zero, is one of the principal benefits claimed for the probing and conditional-selection procedure.
Frequency-only obtains nearby ER@10 values with lower first-use costs: 2.786 versus 6.713 million tokens on 2Wiki, and 3.826 versus 8.429 million on MuSiQue. However, it retains larger indexes, and its stated cost excludes the work needed to determine the capacity inherited from WARP-G. These results therefore do not establish that frequency-only dominates the proposed method. The authors already explain the two deployment settings; the unresolved issue is quantitative evidence for the additional automatic decision. A practical, independently specified or selected capacity rule would provide that comparison if any selection overhead were counted. The savings against Full Graph support selective provisioning, but do not by themselves show how much value this particular automatic selection procedure adds over a simpler deployment rule.
-
One completed design sample per workload leaves uncertainty about the usefulness of the selected deployment (PDF p. 5, Section 3.4; pp. 15–16 and 20, Appendices A.3/B). Although the overall design workload has 800 queries, the final conditional-selection stage compares up to 16 graph sets using at most 32 shared queries and accepts the largest strictly positive estimated marginal gain. Each dataset has one completed design/test evaluation with seed 42. The paper explicitly discloses this limitation and provides held-out test-query bootstrap intervals; those intervals address uncertainty conditional on the selected deployment, not the effect of changing the design sample that produced it.
This matters because the method's output is a deployment decision. Different selection samples might choose different graph sets yet yield similarly useful quality–cost trade-offs, which would be reassuring. Conversely, materially different held-out utility or cost would limit the reliability of automatic provisioning. The current results do not establish either outcome, so I do not interpret them as evidence of observed instability or overfitting. A limited resampling analysis of the final decision stage, reusing existing probes and holding the partition and shortlist fixed, would address part of this uncertainty without rerunning the entire design pipeline. The reported +1.15/+1.10 answer-F1 gains over Hybrid are point estimates; paired uncertainty would also clarify their strength without treating interval overlap as a significance test.
I lean toward weak rejection. The optional-index formulation, transparent workload analysis, and design-inclusive graph-cost accounting are valuable. My reservation is whether the evidence is sufficient for the automatic selection contribution: its capacity decision lacks an independent comparison, and the usefulness of that decision has not been assessed across selection samples. A focused comparison and limited resampling study showing a repeatable, useful quality–cost trade-off would materially improve my assessment. WARP-G need not minimize every cost or choose identical graphs across samples; it should demonstrate why its additional design effort produces a useful deployment decision.
Questions for authors
- For W1 — main priority: Can an independently specified or selected capacity rule be compared under the same partition, retrieval backend, and full-corpus passage access? Please explain how the capacity is determined without inheriting WARP-G's output, count any selection overhead, and report retrieval quality, retained-index cost, and first-use cost separately. This would clarify the deployment conditions under which automatic selection justifies its design effort.
- For W2: With the existing probed indexes, partition, and shortlist fixed, how do resamples of the final selection queries affect selected capacity, cost, and held-out quality? The useful outcome is the distribution of quality–cost choices, not necessarily agreement on the exact graph set. Please distinguish this targeted check of the final decision stage from variability in the entire design pipeline.
- For W2: If per-query scores are available, can paired uncertainty for the answer-F1 changes against Hybrid supplement the existing retrieval intervals and gain-only test? This would clarify the strength of the answer-quality evidence without requiring a new benchmark or a different generation setup.
3 comments
Thank you for showing that WARP-G uses fewer graph tokens than the validation-based selector even after accounting for selection costs. I maintain my rating of 2 (weak rejection) because the fixed k=1 policy used fewer graph tokens than WARP-G and achieved higher F1 point estimates on both datasets, and answer F1 remains unreported across the 20 selection resamples.
Thank you for the follow-up. Our contribution is deciding where graphs are useful and how many to retain, using limited probing, conditional selection, and index reuse. Fixed k = 1 sets capacity in advance; it cannot decide when to retain no graph or multiple graphs. Its success on two workloads does not establish a generally suitable capacity.
Choosing capacity also has a cost: the evaluated validation search uses 18.491M/15.811M graph tokens, versus WARP-G’s 6.068M/7.538M design tokens on 2Wiki/MuSiQue, including probing and selection. Fixed k = 1 has higher F1 point estimates and lower first-use cost, but WARP-G has higher ER@10: 80.00 vs. 79.13, and 73.62 vs. 73.08. The k = 1 results are strong, but for a new workload we still need a way to decide which graphs to retain and how many, without spending too much on that decision.
We have now completed the missing answer-quality evaluation for the 20 selection resamples:
| Dataset | F1, mean ± SD | First-use graph tokens (M), mean ± SD |
|---|---|---|
| 2Wiki | 62.41 ± 0.47 | 6.491 ± 0.683 |
| MuSiQue | 44.51 ± 1.06 | 7.751 ± 0.394 |
Only the selection-query sample changes; the partition, probed indexes, shortlist, and test queries remain fixed. Here we check how much answer quality and cost change when we use a different query sample to select the graphs.
Response to Reviewer dsKN
Thank you for the clear and practical suggestions. Your point about choosing the graph count, rather than only choosing regions for a given count, was especially helpful. We ran the capacity and repeat-sampling tests you asked for, and added confidence intervals for answer F1.
Q1 / W1: Choosing the graph count and paying for that choice
In the paper, frequency-only used the same graph count as WARP-G. We now also test frequency ranking with fixed counts k=0,1,2,4,8, and a method that chooses k using 200 labeled validation queries. Its score is 0.5ER@10+0.5CE@10. We count all the graph construction and validation graph retrieval used to make that choice.
The WARP-G numbers are from the paper. For frequency at the matching graph count, the cost and retrieval scores are from the paper and F1 comes from saved original-run answers. Other counts are from new runs. Costs are in millions of graph tokens (M).
First-use cost includes graph construction and online graph retrieval. WARP-G's design and online costs are 6.068M + 0.646M (2Wiki) and 7.538M + 0.891M (MuSiQue), as reported in the paper. The validation-based capacity search alone uses 18.491M / 15.811M tokens.
A small fixed count can work very well. The hard part is picking that count before seeing test results. WARP-G uses a small set of probes, reuses the graphs it builds, and stops when another graph no longer adds value. This is also why it selects zero graphs on HotpotQA and NQ, while keeping full passage retrieval.
Q2 / W2: Changes across selection samples
We ran the test you suggested: 20 samples of 32 selection queries (seeds 1000–1019). We kept the regions, probed graphs, shortlist, and test queries unchanged.
The exact graph IDs can change, but every run keeps at most two graphs. We also reran the full pipeline with different data splits and regions. Compared with Hybrid on each matching split, WARP-G changed answer F1 by 0.00 and +0.19 points on 2Wiki (seeds 43/44), and +1.36 points on MuSiQue (seed 43). This gives us a second view of how the method behaves when the design data changes.
Q3 / W2: Confidence intervals for answer F1
We added 10,000 paired bootstrap samples using the original paper's 200 test queries:
We agree with your point that the exact region IDs matter less than the final quality and cost. In the paper's 2Wiki result, WARP-G keeps one region costing just 8.2% of Full Graph construction tokens, with 64.21 F1 (Hybrid: 63.06; Full Graph: 62.47). Across four workloads, WARP-G reduces first-use graph tokens by 38.7–67.2%, even after counting the work done to choose the graphs.
Thank you again for suggesting these tests. We would be happy to discuss any other questions.