- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LADDER, which integrates graph retrieval into the denoising process of a diffusion language model for multi-hop QA. Motivated by the observation that about 88% of final entity spans appear in the intermediate draft of Dream-7B before they are committed, with mean lead times of 5.7 to 9.6 steps, LADDER triggers retrieval only when a new graph entity emerges in the draft, and fine-tunes GFM-RAG on these entity events to retrieve both missing and previously found evidence. On HotpotQA, 2WikiMultihopQA and MuSiQue, with MuSiQue held out from retriever training, it reports an average EM increase from 39.6% to 45.2% and a 4.1× latency reduction over a multi-round Dream-7B baseline, and also compares against six autoregressive retrieval pipelines. Ablations cover refresh schedules, training objectives and the effect of evidence quality on denoising steps.
Critical strengths
-
Diffusion-specific retrieval timing. Entities in the final response usually appear in the intermediate draft before they are committed, and the paper turns this into a retrieval trigger without a learned gate.
-
Effective refresh policy. Under a fixed backbone and retriever, entity-triggered refresh outperforms retrieving once, fixed-interval and confidence-based schedules at similar latency.
-
Retrieval as a source of decoding speed. Better evidence in the prompt reduces the number of denoising steps, a property of diffusion decoding that standard autoregressive decoding lacks.
-
Favorable quality-latency trade-off. LADDER obtains the best EM on all three benchmarks with lower latency than all iterative baselines.
Critical weaknesses
-
Source of the speedup. The 4.1× gain is measured against a Dream-7B baseline that runs multiple IRCoT rounds, while retrieving once within a single trajectory is already slightly faster than LADDER. The efficiency gain therefore comes mainly from single-pass decoding rather than from entity-triggered retrieval.
-
Source of the accuracy gains. The main comparison changes the decoding loop, the trigger and the retriever training at once, and no experiment isolates the effect of retriever fine-tuning. On held-out MuSiQue, LADDER is roughly on par with the strongest baseline, and no significance tests are reported.
-
Evidence for the acceleration claim. Random passages already remove a large share of denoising steps, which may partly reflect the generator's fine-tuning format rather than evidence quality. The step-count experiment also uses fixed evidence, so it does not show that refreshing retrieval during denoising speeds up convergence.
Questions for authors
-
Could the authors report LADDER with the pretrained GFM-RAG retriever, and the IRCoT baseline with the fine-tuned retriever? This would separate the contribution of retriever fine-tuning from that of the decoding scheme.
-
Could the authors report the number of denoising steps for each refresh schedule, and for a generator not fine-tuned exclusively with documents in the prompt? This would show whether entity-triggered refresh itself speeds up convergence, and why random passages already reduce the step count.
-
Could the authors add an autoregressive baseline with a single retrieval round, and describe the inference setup used for the autoregressive models? This would clarify how LADDER's latency compares with an efficient autoregressive pipeline.
-
The paper could discuss retrieval-based speculative decoding, which also uses external text to accelerate autoregressive generation, to better position its efficiency argument.
4 comments
Response (Part 2/2)
[W3,Q2] Dynamic convergence and the generator's training format
Convergence under dynamic refresh. Manuscript Table 11 measures convergence with the evidence fixed at the start. Table K3 reports denoising steps and GNN forwards under four refresh policies within a single trajectory. We hold the Dream-7B and event-tuned GFM-RAG checkpoints, confidence threshold (τc = 0.85), and output budget fixed.
Table K3. Denoising, retrieval, and accuracy by refresh policy. EM and total time are macro-averages from manuscript Table 8. LADDER’s GNN-forward count is the macro-average of manuscript Table 6.
| Schedule | EM ↑ | Mean denoising steps | GNN forwards/question | s/question ↓ |
|---|---|---|---|---|
| Retrieve once | 37.9 | 24.5 | 1.00 | 2.55 |
| Every step | 44.5 | 21.9 | 21.37 | 4.38 |
| Confidence drop | 42.9 | 22.8 | 2.97 | 2.76 |
| LADDER | 45.2 | 21.8 | 3.20 | 2.79 |
LADDER uses 2.7 fewer denoising steps than retrieve-once (21.8 versus 24.5). It uses almost the same number of steps as per-step refresh (21.8 versus 21.9), with 3.20 versus 21.37 GNN forwards per question and 2.79 versus 4.38 s/question. Compared with confidence-drop refresh, LADDER gains 2.3 EM points at an additional 0.03 s/question.
Generator training format. The no-evidence result in manuscript Table 11 may reflect both the generator’s training format and Appendix H’s instruction to answer only from supplied facts. On 2Wiki, we compare the current checkpoint with one trained using mixed-context SFT. Both use the same base model, training questions, targets, and number of updates. Mixed-context SFT leaves the evidence slot empty in 30% of training examples. We evaluate both on the same questions with a shared prompt that allows answers when the evidence slot is empty. The random-passage and graph-evidence prompts have matched token lengths.
Table K4. Matched training-format control on 2Wiki. Values are mean denoising steps / EM.
| Evidence | Current checkpoint, matched prompt | Mixed-context SFT, matched prompt |
|---|---|---|
| No evidence | 39.2 / 32.0 | 30.8 / 34.2 |
| Random passages | 24.4 / 32.9 | 26.1 / 34.1 |
| Fixed t=0 GFM-RAG passages | 15.5 / 57.3 | 16.3 / 56.8 |
With the prompt held fixed, mixed-context SFT reduces the no-evidence step count from 39.2 to 30.8 and narrows the gap between no evidence and random passages from 14.8 to 4.7 steps. This suggests that the training format contributes to the slowdown with empty evidence. With the mixed-context checkpoint, random passages yield nearly the same EM as no evidence (34.1 versus 34.2). Replacing them with graph-retrieved passages reduces the mean step count from 26.1 to 16.3 and raises EM from 34.1 to 56.8.
[Q4] Retrieval-based speculative decoding
Following your suggestion, we have added a discussion of REST to the related work in the final version. REST retrieves candidate continuations for verification by an autoregressive model. LADDER retrieves supporting passages that condition subsequent parallel denoising in multi-hop QA. The revised text ties the efficiency claims to the systems and inference settings in each comparison. REST, NAACL 2024
Table K5. End-to-end comparison with REST. Both IRCoT variants pair Qwen2.5-7B with HippoRAG 2. LADDER uses Dream-7B. The IRCoT and LADDER rows use results from manuscript Tables 1–2.
| Method | Average EM ↑ | Average F1 ↑ | R@7 ↑ | s/question ↓ |
|---|---|---|---|---|
| IRCoT | 41.3 | 49.2 | 62.6 | 11.65 |
| IRCoT + REST | 41.3 | 49.2 | 62.6 | 9.05 |
| LADDER | 45.2 | 54.6 | 67.1 | 2.79 |
With REST, the Qwen2.5-7B IRCoT system takes 9.05 s/question versus 11.65 for standard IRCoT, with the same EM, F1, and R@7. Compared with IRCoT + REST, LADDER achieves a 3.24× end-to-end speedup and scores 3.9 points higher in EM and 5.4 points higher in F1.
Thank you for the authors’ response and the additional experiments. I will maintain my current decision for weak acceptance.
Thank you for your great support and for taking the time to consider our responses and additional experiments.
Response (Part 1/2)
Thank you for the questions about efficiency attribution, retriever adaptation, and convergence. Manuscript table numbers refer to the submitted PDF. Times are seconds per question, and averages give equal weight to HotpotQA, 2WikiMultihopQA, and MuSiQue.
[W1,Q3] Source of the speedup and a single-retrieval AR baseline
The 4.1× speedup in manuscript Table 2 compares multi-round Dream-7B IRCoT with pretrained GFM-RAG (11.30 s/question) against LADDER (2.79 s/question). Both systems commit tokens in parallel within each denoising step. With the pretrained retriever held fixed, switching to a single trajectory with entity-triggered refresh reduces latency from 11.30 to 2.93 s/question. Following your suggestion, we have named the 4.1× comparator in the abstract and TL;DR.
Within one trajectory and with the event-tuned retriever fixed, manuscript Table 3 provides the schedule comparison. LADDER improves EM from 37.9 to 45.2 over retrieve-once for an additional 0.24 s/question (2.55 to 2.79). Compared with per-step refresh, it reduces latency by 36.3% (4.38 to 2.79 s/question) and improves EM from 44.5 to 45.2.
Table K1. Single-retrieval AR and DLM comparison. LADDER results are from manuscript Tables 1–2. The two HippoRAG 2 configurations use the same top-7 passages retrieved from the question alone.
The single-retrieval Qwen2.5-7B baseline takes 1.64 s/question versus 2.79 for LADDER. LADDER scores 7.8 points higher in EM and 8.6 points higher in F1. Appendix B describes the matched generator supervision. Manuscript timings use a single NVIDIA H20 and a warmed graph index. The two retrieve-once runs use batch size 1, with identical warm-up and timing boundaries covering online retrieval and generation.
[W2,Q1] Retriever tuning and MuSiQue
We agree that a checkpoint swap is the direct control. Manuscript Table 4 compares training objectives, with 41.9 EM for retrieve-only training and 45.2 for the full objective. Both of those checkpoints are tuned. Table K2 instead crosses execution mode with pretrained versus event-tuned GFM-RAG. Within each mode, the generator, controller, query and seed construction rules, evidence budget, and evaluator are fixed.
Table K2. Execution mode and GFM-RAG checkpoint on Dream-7B. The pretrained IRCoT and event-tuned entity-refresh results are from manuscript Tables 1–2.
With the execution mode fixed, event-conditioned tuning improves EM by 1.3 points under IRCoT and 2.8 points under entity refresh. With the checkpoint fixed, switching to single-trajectory entity refresh improves EM by 2.8 points with pretrained weights and 4.3 points with tuned weights. Within entity refresh, tuning also raises R@7 from 63.8 to 67.1 and reduces latency from 2.93 to 2.79 s/question.
MuSiQue was held out from retriever optimization and model selection. In manuscript Table 2, LADDER takes 3.95 s/question versus 14.24 for Youtu-GraphRAG on Qwen2.5-7B. The accuracy point estimates differ by +0.6 EM points (18.8 versus 18.2) and −0.4 F1 points (30.7 versus 31.1) for LADDER.