- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tackles the "over-search" issue in agentic search, where parametric knowledge is already sufficient for an intermediate evidence piece without needing extra search calls. It proposes SOCAL, which first uses an LLM-as-Judge proxy to diagnose over-search steps, and then performs advantage calibration on the corresponding tokens to penalize such behaviors. Experiments on multi-hop and single-hop QA benchmarks based on Qwen2.5-3B/7B show its effectiveness.
Critical strengths
- The overall idea of calibrating the advantage is a more moderate practice (at least to me) compared with direct reward shaping, which can easily lead to unexpected reward-hacking behaviors.
- The figures are vivid, and beyond reporting performance gains, the authors provide training dynamics to show how the shaped policy evolves toward more efficient and purposeful search invocation.
Critical weaknesses
W1 The real necessity of addressing "over-search" and the mismatched testing scenarios:
- I am not sure whether "over-search" is a pervasive issue in modern LLMs. Based on my experience with models like Gemma-4, Kimi-K2, and DeepSeek-V3.2, these models tend to under-search (or perform shallow search) and rarely engage in persistent search to collect all necessary information pieces. Even when extra searches occur, they are not necessarily bad behavior: it remains an open problem to judge whether an exploratory query truly contributes to the final answer. While strictly redundant searches should be penalized, persistent search across diverse information pieces to pinpoint relevant evidence should be encouraged.
- Although the paper is motivated by "over-search", all downstream evaluations are conducted on relatively simple QA benchmarks using Qwen2.5-3B/7B backbones capped at 4 search calls, which is insufficient to justify the method's effectiveness in realistic long-horizon search scenarios.
W2 The performance gains appear to come primarily from an existing training trick (dynamic sampling from DAPO) rather than the proposed methodology itself:
- The authors use dynamic sampling in Section 4.3, whereas the HiPRAG baseline in Table 1 does not. In Table 1, the 3B HiPRAG baseline achieves 65.4, while SOCAL-3B achieves 66.9. However, in the Table 2 ablation, SOCAL w/o dynamic sampling drops to 65.1, which cannot even outperform the HiPRAG baseline (65.4), yet dynamic sampling is a standard optimization trick rather than the core contribution of this paper.
W3 The computational overhead and complexity introduced by this method are completely neglected:
- Omitting training cost makes the comparison with baselines unfair. In practice, SOCAL introduces substantial overhead at every search step, including external LLM-as-Judge calls, additional forward passes for hindsight log-probability computation, etc.
Questions for authors
Please see above weaknesses
2 comments
Dear Reviewer,
Thank you again for your thoughtful comments. As the discussion period is ending soon, we wanted to check whether our responses addressed your concerns. In brief: SOCAL does not penalize search in general (it searches more than HiPRAG while achieving higher CEM); raising the search cap to 6 or 8 at evaluation time barely changes accuracy or search volume; without dynamic sampling and at a matched 400-step budget, SOCAL still outperforms HiPRAG by 1.18 points; and we now report full training and inference cost.
We would be grateful for any further feedback.
We thank the reviewer for the thoughtful comments, and for noting that advantage calibration is a more moderate practice than direct reward shaping.
W1.1. Is over-search a real problem, and should extra searches be penalized?
We agree that extra searches are not inherently bad, and SOCAL is designed precisely not to penalize search in general. Our claim is narrower: under outcome-only RL, an unnecessary search inside a successful trajectory receives the same positive credit as a useful one, so RL can amplify it. Figures 1(b) and 3 show this in our setting: Search-R1's OSR and Search/Q both rise during training. Over-search has also been documented independently [1–3].
SOCAL only modifies steps where the policy could answer the local query without retrieval, and leaves every other step's credit untouched; it does not discourage persistent or exploratory search for information the model lacks. Empirically, SOCAL searches more than HiPRAG (1.86 vs. 1.74 per question) while achieving higher CEM (66.9 vs. 65.4) — this is what we mean by "calibration, not suppression." We agree that for models that predominantly under-search the benefit may be smaller, and will state this in the Limitations section.
W1.2. Simple QA, 3B/7B backbones, four-call cap.
Over-search concerns whether a retrieval was necessary, not trajectory length; Figure 3 shows the credit-assignment problem already arises under a four-call cap. Our evaluation includes multi-hop QA, where SOCAL's gains are largest, including three multi-hop datasets unseen in training (*):
The four-call cap is also rarely binding: average Search/Q is 1.74–2.33 for all trained models. Raising the cap at evaluation time (full protocol, no retraining) barely changes SOCAL's behavior:
Neither accuracy nor search volume changes materially, so the cap does not constrain SOCAL. Nevertheless, we agree that multi-hop QA does not establish effectiveness in long-horizon, open-web search; in the revision we will scope our claims to knowledge-intensive QA and list long-horizon agentic search as future work.
W2. Are the gains due to dynamic sampling?
Two new controls separate DS from the calibration mechanism.
(a) DS held fixed. In a new constant-coefficient control, c=0 is exactly GRPO + DS with no calibration, trained in the same pipeline for 320 steps. It reaches 65.85 CEM; SOCAL reaches 66.90 (+1.05) and lowers OSR from 4.17 to 3.49. Of this gain, 0.60 comes from the adaptive coefficient.
(b) DS removed, budget matched. Without DS, fewer groups have nonzero advantage, so each step carries fewer effective updates. Training SOCAL w/o DS for the same 400 steps:
At the same budget and without DS, SOCAL outperforms HiPRAG by 1.18 points and DS-only training (c=0) by 0.73. DS is therefore not the source of the gain; it mainly lets SOCAL reach a similar level in fewer steps. We will add these rows to Table 2.
W3. Computational overhead.
We agree cost should be reported and have measured it. SOCAL's per-step overhead consists of the no-search probe (≈70 s), net judge waiting (≈18 s; judge calls run concurrently) and hindsight-conditioned scoring (≈15 s; a forward pass on already-sampled tokens, with no regeneration and no writer model). On 4×H100, a SOCAL step (including DS) takes 465.8 s vs. 362.5 s for Search-R1-style training (+28.5%) and 451.7 s for EviSD (+3.1%). Because SOCAL converges in 320 instead of 400 steps, total training compute is 165.6 vs. 161.1 GPU-hours (+2.8%). At inference, SOCAL adds no overhead: it issues fewer searches than Search-R1 (1.86 vs. 2.22) and has 10.7% lower latency. We will add a cost table to the paper.
We hope these new results address the reviewer's concerns, and we would be happy to clarify any remaining questions.
References (all cited in the paper)