- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FastRAG speeds up Search-R1-style multi-round RAG by generating the search query from intermediate layers. An exit trigger (an adapter checked every two layers) decides per token whether to stop, and a token calibrator maps the intermediate hidden state into the space of the final norm and LM head. At inference, a speculative query whose mean token entropy is below γ goes straight to the retriever. Otherwise the query is regenerated at full depth. Training (SpecGRPO) adds a query-consistency reward (E5 similarity to the full-depth query, thresholded at τ) and an early-exit reward to GRPO, plus a KL loss for the calibrator. On Qwen2.5-3B/7B-Instruct and seven QA benchmarks, the 7B model reportedly cuts average latency from 3.673 s to 1.382 s compared with Search-R1, while EM rises from 43.1 to 47.3.
Critical strengths
- The evaluation is broad. It covers seven QA benchmarks and many baselines from four families of RAG acceleration (retrieval-call control, pipelined/speculative retrieval, context compression, reasoning-side methods), and the appendix has extensive sensitivity analysis.
- Entropy gating is simple and cheap. Table 8 shows it keeps almost all the accuracy of always generating the full-depth query (47.3 vs. 47.6) at about half the latency.
Critical weaknesses
1. Calibrator and trigger architecture, and inference details. §3 describes C_ψ and G_φ only as "lightweight adapters". What is their architecture (number of layers, hidden size, how the layer embedding e_ℓ is injected), and how many parameters do they have? The trigger runs at every candidate layer for every token, so its cost directly affects latency. Two inference details are also missing:
- Are only query tokens early-exited, or
<think>tokens too? §3.1 talks about "using intermediate-layer tokens for reasoning", but the entropy gate only checks the query. - When a token exits early, the layers above it have no KV entries for it, yet later tokens that go deeper still attend to it. How is this handled?
2. What data are Figs. 7 and 8 computed on? Which dataset, how many samples, and which split? Fig. 7 names only the model (3B). In Fig. 8, accuracy without fallback approaches 100% in the lowest-entropy region, against 37.8% overall. How many samples fall in that region? If it is a small, easy subset, the figure overstates how well entropy predicts correctness. A 7B version of Fig. 7 would also help, since the latency results are on 7B.
3. Where does the latency reduction come from? According to Fig. 2, retrieval is only 10.8% of end-to-end latency on NQ. Even if FastRAG ran all retrieval in parallel with reasoning, that would save at most about 10.8%. Yet Table 2 reports a (3.314 − 1.247)/3.314 ≈ 62.4% reduction over Search-R1 on NQ. Layer skipping does not obviously close the gap. From Fig. 7 the mean exit layer is about 18.8 of 36. So even if every token exited at that depth and the adapters cost nothing, reasoning would get at most about 1.9x faster, and combined with removing retrieval entirely the reduction would top out around 53%. (Figs. 2 and 7 use 3B and Table 2 uses 7B, but the gap remains unless the 7B model behaves very differently.) Two further observations point the same way. In Table 8, the "standard verification" variant generates the full-depth query in every round, yet it is still about 1.1 s faster than Search-R1. And as written, the inference procedure never overlaps retrieval with computation; retrieval simply starts earlier. Please give a latency breakdown (prefill, decode, retrieval), the per-question number of search rounds and generated tokens for FastRAG and Search-R1, and the fallback rate.
4. Where does the accuracy gain come from? Could the authors briefly explain? At best a speculative query matches the full-depth one, so speculation by itself should not raise EM. The variant without verification (45.1, Table 8) still beats Search-R1 (43.1). That suggests the gain comes from the different training recipe (extra rewards and an extra advantage term), possibly through fewer search rounds or shorter reasoning. Evaluating the plain GRPO model from Fig. 9 for both EM and latency, with the same backbone, data and training steps, would answer this and point 3.
5. Limited model scope. All experiments use Qwen2.5-3B/7B with a single retriever and corpus, which the paper acknowledges in Appendix F. At least one other model family and one larger model would make the claims more convincing. Appendix F also mentions training overhead: each rollout runs both a full-depth and an early-exit path. That overhead should be quantified.
6. The "verification" is a confidence heuristic, and the consistency signal behind it is lenient. At test time, accepted queries are not verified; they are trusted because their entropy is low. Calling this "self-verification" invites a comparison with speculative decoding, which guarantees exact outputs. There is no such guarantee here, and even the lowest-entropy bin in Fig. 4c has only an 84.7% verification success rate. The consistency check is also loose. Because E5 is trained with a very low InfoNCE temperature, its cosine scores are concentrated in roughly 0.7 to 1.0 regardless of relevance (see the FAQ in the E5 model card), so τ = 0.78 may accept query pairs that retrieve different documents. The nearly flat Table 5 (46.8 to 47.3 EM for τ between 0.74 and 0.82) fits this reading. A retrieval-level criterion, such as top-k overlap between the documents retrieved by q^spe and by q^ref, would be more meaningful. Please also report the test-set acceptance rate and EM for accepted vs. fallback rounds.
7. Missing related work and baseline. Decoding intermediate layers through the final head via a learned translator is essentially the tuned lens (Belrose et al., 2023). Learned per-token exit is central to CALM (Schuster et al., 2022), and self-speculative decoding is the subject of LayerSkip (Elhoushi et al., 2024) and Draft & Verify (Zhang et al., 2024). None of these are cited. The most natural baseline is also missing: lossless speculative decoding applied to Search-R1, for example with a Qwen2.5-0.5B draft model or LayerSkip-style self-speculation.
8. Evaluation protocol.
- All thresholds and weights (γ, τ, λ_spe, λ_cal and the reward weights in Tables 4 to 7) are selected by average EM on the same seven benchmarks used for testing, with no held-out dev set. No seeds or variances are reported, and most differences in Tables 3 to 7 are within about one EM point.
- How were the baselines reproduced? Do they all use the same backbone, retriever and corpus?
- Table 9 puzzles me. For 10 of the 11 methods, the seven-dataset average latency divided by the NQ latency in Table 2 is the same 1.108 (e.g. Search-R1 3.673/3.314, Adaptive-RAG 2.838/2.561, FastRAG 1.382/1.247). Only SearchAgent-X differs (about 1.21). These methods behave very differently on multi-hop questions, and the percentile columns do not follow the same pattern, so I would not expect this. Please explain how Table 9 was computed and report per-dataset latencies.
Minor
- Fig. 5 notation: the downward arrow from Reference Query should be labeled q_m^ref and the upward arrow from Speculation Query q_m^spe; the two labels are swapped. The red/green verification conditions should compare q_m^spe with q_m^ref, but the figure writes V_m(q_m^spe, q_m^spe).
- Fig. 6: the update arrow pointing to the exit trigger is labeled "ψ Update". Per the text it should be "φ Update"; ψ denotes the calibrator's parameters.
- The introduction says intermediate-layer queries "often remain consistent" with full-depth ones (Fig. 3, Fig. 4a). But for the untrained model, Fig. 4a shows a 0% query match rate at layer 30 and 44% at layer 35 of 36. The redundancy is created by training rather than observed beforehand.
- D_{i,m} (Eq. 7) and D_i (Eq. 12) clash, and the first sentence of the conclusion is broken ("retrieval queriefull-depth generation").
Questions for authors
See weaknesses