- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines the "competence-policy transfer gap" in tool-using retrieval-augmented generation (RAG): fine-tuning an LLM on full gold evidence (Full-Evidence SFT) improves evidence-conditioned reader accuracy but harms its interactive query formulation policy when deployed in an interactive search environment. The authors observe that simply pairing the inherited unadapted Base acquisition policy with the adapted reader (Base+FE) substantially recovers interactive performance.
To improve upon this baseline, the authors introduce K2D-ALIGN (Knowledge-to-Decision Alignment). K2D-ALIGN learns a specialized first-step retrieval action (root proposal) via Base-relative supervision and implements a deployment-time admission controller that executes parallel "shadow continuations" (evaluating both candidate and Base trajectories) before deciding whether to admit the proposed action or fall back to the Base policy. The method is primarily evaluated on Qwen3-4B using BM25 retrieval over Wikipedia across HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.
Critical strengths
Empirical Documentation of Reader-Acquisition Decoupling: The paper systematically documents that full-evidence SFT degrades interactive tool-use while improving reader competence, and clearly quantifies that simple functional decoupling (using Base acquisition with an FE reader) serves as a surprisingly effective and compute-free recovery baseline.
Critical weaknesses
-
Conceptual Obsolescence Relative to Modern Agentic RL Paradigms: The fundamental motivation of this paper is severely dated. Modern interactive information seeking and multi-step tool use have largely transitioned to unified Agentic Reinforcement Learning (e.g., Search-R1, ReSearch, Tool-RL), where search, reasoning, and reflection are jointly optimized end-to-end. Trying to patch an SFT-induced policy degradation via an ad-hoc heuristic pipeline—freezing an adapted reader, falling back to base acquisition, and performing dual-branch shadow rollouts solely to modify the first search action—offers minimal scientific or practical insight to the modern research community. Operating strictly in a toy BM25 retrieval environment over legacy 2018 multi-hop QA benchmarks (HotpotQA) further widens this gap from practical web-agent and dense-retrieval realities.
-
Severe Empirical Flaws, Missing Call Traces, and Disproportionate Compute Overhead: The submission exhibits critical experimental compromises and poor efficacy:
- Missing Traces and Unreliable Reporting: In Table 3 and Figure 2d, the authors explicitly state: "The 52.34-EM K2D run lacks a complete call trace; Figure 2d uses a separate 50.78-EM audit." Submitting a primary claim where the corresponding compute trace was lost, substituting a lower-performing run to plot compute-quality trade-offs, and leaving baseline quantities blank ("unrecorded quantities are left blank, not imputed") is unacceptable for a top-tier venue.
- Disproportionate Cost for Marginal/Negative Gains: K2D-ALIGN triples inference compute (model calls jump nearly threefold from 6.08 to 17.71), yet the macro-average EM across four benchmarks improves by only 1.9 points over the compute-free Base+FE baseline (30.7 to 32.6). Crucially, on the most challenging multi-hop benchmark, MuSiQue, K2D-ALIGN actually regresses performance from 13.2 to 12.6 EM (-0.6 points). Combined with tiny sample slices (root supervision from merely 24 pairs; Hermes evaluated on 32 cases with 0.00% gain) and failure to replicate on Qwen2.5-7B, the empirical claims are fragile.
Questions for authors
- Given that modern multi-hop retrieval and information seeking have largely transitioned to end-to-end Agentic RL (e.g., Search-R1, ReSearch) and dense/hybrid retrieval, what foundational insights or practical advantages does this piecewise SFT decoupling and dual-branch heuristic patch offer over integrated RL approaches?
- How can the empirical validity of K2D-ALIGN be guaranteed when the primary 52.34-EM run lacks a complete call trace, trade-off curves are plotted using a separate lower-performing run, and the method exhibits negative transfer on MuSiQue (-0.6 EM) despite tripling inference compute?
6 comments
Thanks For Reply
Thank you for the detailed rebuttal and for running the additional controls (visible-reasoning ReAct/IRCoT, the native-index MuSiQue slice, and the dense/hybrid and SearchGym diagnostic checks). In recognition of these added analyses and the removal of the 50.78-EM substitution, I have raised my Informativeness score from 1 to 2.
However, my core concerns remain unresolved, so I am maintaining my overall rating of 2 (Weak Rejection):
- Trace Provenance & Compute–Quality Trade-off: While removing the mismatched 50.78-EM substitution improves clarity, the 16.34 calls/case for the 52.34-EM run is still a post-hoc reconstruction from deterministic replay rather than original runtime logs, and token/latency measurements remain N/A. Meanwhile, across the full 1,625-case benchmark suite, K2D-ALIGN still requires ~2.7–2.9× model calls for a +1.9 macro EM gain, while its larger improvements rely on small slices (e.g., 24 states / 5 repairs on HotpotQA dev, 128 cases on native MuSiQue).
- Positioning Relative to Agentic RL: Characterizing end-to-end Agentic RL as an unattributable scalar-return black box is not convincing. Separating retrieval hit/recall from reader accuracy, reward shaping, and reference-policy/counterfactual advantage estimation are standard in RL. Moreover, a zero-cost component-substitution diagnostic does not justify paying ~3× inference compute for an inference-time shadow-rollout patch—especially since SearchGym-Qwen3-4B was only tested for diagnostic compatibility on 48 cases without reporting a direct accuracy and cost comparison against K2D-ALIGN.
Thank you for the follow-up.
Cost and provenance. The 16.34 figure is a reconstructed model-call count for the historical operating point; token and latency costs remain unmeasured. The full-suite result therefore describes an accuracy–inference trade-off. The smaller matched study addresses learning attribution within its evaluated setting. The component-substitution recovery itself uses Base acquisition with the adapted reader; the shadow-rollout cost belongs to the subsequent repair experiment.
Relation to RL. Retrieval metrics, reward shaping and counterfactual advantages are established tools. Our contribution concerns a specific adaptation-induced transfer phenomenon and its identification through controlled component substitution. K2D tests whether intervening on the identified acquisition component can realize additional competence. The 48-case SearchGym experiment extends the diagnostic to a trained agent’s evidence interface.
Thank you for recognizing the compute-free Base+FE recovery baseline and raising direct questions on agentic-RL positioning, trace provenance, and compute–quality trade-offs.
1. Foundational Insight Relative to End-to-End Agentic RL (Q1, W1)
End-to-end RL (Search-R1, ReSearch, ToolRL, SearchGym) optimizes a scalar return coupling evidence acquisition (π) and evidence-conditioned reading (r). Scalar return changes alone cannot identify which sub-function improved or degraded after domain post-training, distillation, or continual updates.
Component substitution isolates both interfaces at zero extra inference compute: holding evidence fixed measures reader transfer (Oracle EM: 58.6 → 68.0 on HotpotQA), while holding the adapted reader fixed isolates acquisition loss (32.0 EM) and recovers 46.1 EM at 5.52 calls/case (Base+FE, Table 2 [orig. Table 3]; 26.1 → 30.7 macro EM across four benchmarks at 6.08 calls/case, Table 4 [orig. Table 5]).
K2D-ALIGN then formulates Base-relative credit assignment: a candidate root is supervised only when it improves terminal correctness over the inherited action from the same state under a shared reader and continuation (ΔEMi>0). Under matched initialization, 24 states, 48 updates, and identical admission (Table 1 [orig. Table 2]), K2D-positive rises from 48.44 (zero-update) to 52.34 EM (5 repairs, 0 regressions; paired 95% CI [+0.78, +7.81]), whereas Absolute-outcome (rewarding actions merely co-occurring with success), Direct, and Relation stay at 48.44. Functional attribution and Base-relative credit thus complement joint RL.
Manuscript: Sections 1–2; Table 1; Appendix B (Table 7).
2. Rebuttal Controls: Visible Reasoning & Modern Interfaces (W1)
Unconstrained visible-reasoning ReAct and IRCoT (new rebuttal results; prompt protocol in Appendix H). Removing scratchpad suppression, we ran 256 native-interface evaluations (128 × 2) allowing two explicit reasoning sentences per turn (384-token cap) retained across turns under the shared Qwen3-4B backbone, FE reader, BM25 index, and 5-search cap (all invalid actions retained):
Descriptive paired EM differences on this roster are +14.84 pp (95% CI [+7.03, +22.66]) and +11.72 pp ([+5.47, +18.75]), testing scratchpad visibility under matched search caps with K2D's call count shown explicitly. A blinded LLM-judge check across all 48 EM-discordant pairs (without method labels) attributes 22 wins / 1 loss vs. ReAct and 16 wins / 0 losses vs. IRCoT to substantive correctness rather than formatting.
Dense/hybrid & RL-agent diagnostics (rebuttal-only). Across 48 BrowseComp-Plus questions under BM25, official dense, and hybrid retrieval with byte-identical Base/FE reader swaps (288 runs), plus 48 bounded 8-action episodes and 96 identical-evidence reader replays on RL-trained SearchGym-Qwen3-4B, the substitution diagnostic executes cleanly across modern retrievers and a trained search agent.
3. Same-Branch Cost Provenance for 52.34 EM (Q2, W2)
We removed the 50.78-EM substitution and incomplete H872 rows. Replaying the exact saved terminal branches yielding 52.34 EM / 64.41 F1 gives 13.41 pre-decision plus 2.94 selected-path calls—16.34 model calls and 4.63 backend requests alongside 1.47 committed searches per case—verified case-by-case against 128 instrumented traces under deterministic read-only continuation. Unmeasured token/latency cells remain N/A.
Manuscript: Table 2 [orig. Table 3]; Appendix D.3 (Tables 15–16, Figure 3).
4. MuSiQue Corpus Condition & Compute Trade-off (Q2, W2)
In the 1,625-case suite (Table 4 [orig. Table 5]), all benchmarks query Wikipedia-HotpotQA—an out-of-index test where missing MuSiQue paragraphs cap every method at 9.0–13.2 EM (K2D: 12.6 EM / 20.8 F1 vs. Base+FE: 13.2 / 20.4, while preventing unguarded Scaled Root-DPO's drop to 11.4 EM and 26.2 macro EM via Base fallback).
On MuSiQue's native paragraph-pool index (128 cases, Table 24), K2D-ALIGN raises Base+FE from 19.53 to 26.56 EM (+7.03 pp, [+1.56, +12.50]; 11 repairs, 2 regressions), alongside 40.83 → 43.12 EM on H872 (+2.29 pp, [+0.92, +3.78]; 31 repairs, 11 regressions, Table 3).
Base+FE gives compute-free recovery; K2D-ALIGN trades ~2.7–2.9× calls for selective repair.
Manuscript: Sections 3.4, 4.5–4.6; Tables 3–4, 8, 24; Appendices A.1, C.