acceptodds
Under review as a conference paper at ICLR 2027

SOCAL: Step-Specific Over-Search Calibration for Search Agents via Self-Distilled Hindsight

Abstract

Search agents are prone to over-search when they fail to accurately assess whether their internal knowledge is sufficient, resulting in unnecessary external search. Outcome-based RL can further amplify this miscalibration, as trajectory-level credit does not distinguish between useful and unnecessary search decisions. Existing process-aware methods incorporate process-level feedback on search necessity, yet the degree of correction applied to individual search decisions remains largely determined by predefined reward formulations or advantage-rescaling rules. This raises a natural question: can the agent itself provide an adaptive, decision-specific signal for calibrating its search behavior? To this end, we introduce SOCAL (Step-Specific Over-Search Calibration for Search Agents via Self-Distilled Hindsight), which transforms over-search diagnoses into step-specific hindsight and leverages self-distillation for adaptive search calibration. Specifically, SOCAL first performs knowledge probing over sampled trajectories to extract hindsight reflecting the model's knowledge sufficiency. For each diagnosed step, it contrasts the policy's support for the same sampled step under the original and hindsight-conditioned views. The resulting likelihood discrepancy serves as a self-distilled signal for reshaping the corresponding outcome-derived credit, allowing the correction strength to adapt to each individual search step. Extensive experiments on seven knowledge-intensive question-answering benchmarks demonstrate the effectiveness of SOCAL, yielding 1.5–2.9 percentage-point gains in CEM over the strongest baselines and the lowest over-search rates across both 3B and 7B model settings. Our code is publicly available at https://anonymous.4open.science/r/SOCAL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Search Agents, Reinforcement Learning, Search Calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qqyzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles the "over-search" issue in agentic search, where parametric knowledge is already sufficient for an intermediate evidence piece without needing extra search calls. It proposes SOCAL, which first uses an LLM-as-Judge proxy to diagnose over-search steps, and then performs advantage calibration on the corresponding tokens to penalize such behaviors. Experiments on multi-hop and single-hop QA benchmarks based on Qwen2.5-3B/7B show its effectiveness.

Critical strengths
  1. The overall idea of calibrating the advantage is a more moderate practice (at least to me) compared with direct reward shaping, which can easily lead to unexpected reward-hacking behaviors.
  2. The figures are vivid, and beyond reporting performance gains, the authors provide training dynamics to show how the shaped policy evolves toward more efficient and purposeful search invocation.
Critical weaknesses

W1 The real necessity of addressing "over-search" and the mismatched testing scenarios:

  • I am not sure whether "over-search" is a pervasive issue in modern LLMs. Based on my experience with models like Gemma-4, Kimi-K2, and DeepSeek-V3.2, these models tend to under-search (or perform shallow search) and rarely engage in persistent search to collect all necessary information pieces. Even when extra searches occur, they are not necessarily bad behavior: it remains an open problem to judge whether an exploratory query truly contributes to the final answer. While strictly redundant searches should be penalized, persistent search across diverse information pieces to pinpoint relevant evidence should be encouraged.
  • Although the paper is motivated by "over-search", all downstream evaluations are conducted on relatively simple QA benchmarks using Qwen2.5-3B/7B backbones capped at 4 search calls, which is insufficient to justify the method's effectiveness in realistic long-horizon search scenarios.

W2 The performance gains appear to come primarily from an existing training trick (dynamic sampling from DAPO) rather than the proposed methodology itself:

  • The authors use dynamic sampling in Section 4.3, whereas the HiPRAG baseline in Table 1 does not. In Table 1, the 3B HiPRAG baseline achieves 65.4, while SOCAL-3B achieves 66.9. However, in the Table 2 ablation, SOCAL w/o dynamic sampling drops to 65.1, which cannot even outperform the HiPRAG baseline (65.4), yet dynamic sampling is a standard optimization trick rather than the core contribution of this paper.

W3 The computational overhead and complexity introduced by this method are completely neglected:

  • Omitting training cost makes the comparison with baselines unfair. In practice, SOCAL introduces substantial overhead at every search step, including external LLM-as-Judge calls, additional forward passes for hindsight log-probability computation, etc.
Questions for authors

Please see above weaknesses

2 comments

We thank the reviewer for the thoughtful comments, and for noting that advantage calibration is a more moderate practice than direct reward shaping.

W1.1. Is over-search a real problem, and should extra searches be penalized?

We agree that extra searches are not inherently bad, and SOCAL is designed precisely not to penalize search in general. Our claim is narrower: under outcome-only RL, an unnecessary search inside a successful trajectory receives the same positive credit as a useful one, so RL can amplify it. Figures 1(b) and 3 show this in our setting: Search-R1's OSR and Search/Q both rise during training. Over-search has also been documented independently [1–3].

SOCAL only modifies steps where the policy could answer the local query without retrieval, and leaves every other step's credit untouched; it does not discourage persistent or exploratory search for information the model lacks. Empirically, SOCAL searches more than HiPRAG (1.86 vs. 1.74 per question) while achieving higher CEM (66.9 vs. 65.4) — this is what we mean by "calibration, not suppression." We agree that for models that predominantly under-search the benefit may be smaller, and will state this in the Limitations section.

W1.2. Simple QA, 3B/7B backbones, four-call cap.

Over-search concerns whether a retrieval was necessary, not trajectory length; Figure 3 shows the credit-assignment problem already arises under a four-call cap. Our evaluation includes multi-hop QA, where SOCAL's gains are largest, including three multi-hop datasets unseen in training (*):

Dataset3B HiPRAG3B SOCAL7B HiPRAG7B SOCAL
HotpotQA57.462.762.467.7
2WikiMQA*67.470.171.774.0
MuSiQue*24.135.434.138.6
Bamboogle*41.651.252.854.4

The four-call cap is also rarely binding: average Search/Q is 1.74–2.33 for all trained models. Raising the cap at evaluation time (full protocol, no retraining) barely changes SOCAL's behavior:

Search capCEM (%)Search/Q
466.901.86
666.832.02
867.142.07

Neither accuracy nor search volume changes materially, so the cap does not constrain SOCAL. Nevertheless, we agree that multi-hop QA does not establish effectiveness in long-horizon, open-web search; in the revision we will scope our claims to knowledge-intensive QA and list long-horizon agentic search as future work.

W2. Are the gains due to dynamic sampling?

Two new controls separate DS from the calibration mechanism.

(a) DS held fixed. In a new constant-coefficient control, c=0c=0 is exactly GRPO + DS with no calibration, trained in the same pipeline for 320 steps. It reaches 65.85 CEM; SOCAL reaches 66.90 (+1.05) and lowers OSR from 4.17 to 3.49. Of this gain, 0.60 comes from the adaptive coefficient.

(b) DS removed, budget matched. Without DS, fewer groups have nonzero advantage, so each step carries fewer effective updates. Training SOCAL w/o DS for the same 400 steps:

MethodDynamic samplingStepsCEM (%)
HiPRAGNo40065.40
GRPO + DS (c=0c=0)Yes32065.85
SOCAL w/o DSNo32065.11
SOCAL w/o DSNo40066.58
SOCALYes32066.90

At the same budget and without DS, SOCAL outperforms HiPRAG by 1.18 points and DS-only training (c=0c=0) by 0.73. DS is therefore not the source of the gain; it mainly lets SOCAL reach a similar level in fewer steps. We will add these rows to Table 2.

W3. Computational overhead.

We agree cost should be reported and have measured it. SOCAL's per-step overhead consists of the no-search probe (≈70 s), net judge waiting (≈18 s; judge calls run concurrently) and hindsight-conditioned scoring (≈15 s; a forward pass on already-sampled tokens, with no regeneration and no writer model). On 4×H100, a SOCAL step (including DS) takes 465.8 s vs. 362.5 s for Search-R1-style training (+28.5%) and 451.7 s for EviSD (+3.1%). Because SOCAL converges in 320 instead of 400 steps, total training compute is 165.6 vs. 161.1 GPU-hours (+2.8%). At inference, SOCAL adds no overhead: it issues fewer searches than Search-R1 (1.86 vs. 2.22) and has 10.7% lower latency. We will add a cost table to the paper.

We hope these new results address the reviewer's concerns, and we would be happy to clarify any remaining questions.

References (all cited in the paper)

  • [1] Xie et al., 2026b. Over-searching in search-augmented large language models. arXiv:2601.05503.
  • [2] Wu et al., 2026. HiPRAG: Hierarchical process rewards for efficient agentic retrieval augmented generation. arXiv:2510.07794.
  • [3] Zhang et al., 2026b. To search or not to search: Aligning the decision boundary of deep search agents via causal intervention. arXiv:2602.03304.

Dear Reviewer,

Thank you again for your thoughtful comments. As the discussion period is ending soon, we wanted to check whether our responses addressed your concerns. In brief: SOCAL does not penalize search in general (it searches more than HiPRAG while achieving higher CEM); raising the search cap to 6 or 8 at evaluation time barely changes accuracy or search volume; without dynamic sampling and at a matched 400-step budget, SOCAL still outperforms HiPRAG by 1.18 points; and we now report full training and inference cost.

We would be grateful for any further feedback.

Reviewer_vYx5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies over-search in search-augmented agents, where trajectory-level outcome rewards may reinforce unnecessary searches within successful trajectories. SOCAL uses knowledge probing to identify potentially unnecessary search steps, constructs localized hindsight, and adjusts step advantages using the same policy’s likelihood discrepancy between the original and hindsight-conditioned contexts. Experiments on two model sizes and seven QA benchmarks show improvements in answer quality and over-search metrics.

Critical strengths
  1. An important problem and a coherent methodological response. The paper clearly explains how outcome-based rewards can reinforce redundant searches. Knowledge probing, localized hindsight, and step-level credit adjustment form a consistent approach to addressing this problem.
  2. Positive empirical results. SOCAL improves average CEM over HiPRAG by 1.5 and 2.9 percentage points at 3B and 7B, respectively, while reducing OSR and the proportion of trajectories containing over-search. The ablations also support the localized hindsight design.
Critical weaknesses
  1. The methodological contribution over HiPRAG appears limited. SOCAL retains HiPRAG’s step structure and knowledge-probing diagnosis, and primarily changes how the diagnosis enters optimization: constructing local hindsight and using a teacher–student likelihood discrepancy to adjust step advantages. Given existing work on privileged-context self-distillation and likelihood-based credit adjustment, this appears to be a relatively straightforward adaptation of established ideas. The improvements over HiPRAG are encouraging, but the paper does not sufficiently establish what substantive methodological insight SOCAL adds. This is my primary reservation and the main reason for my current recommendation.
  2. The computational trade-off is incompletely characterized. SOCAL introduces hindsight-conditioned scoring and dynamic sampling without quantifying their training overhead. The paper notes that obtaining nonzero-advantage groups becomes increasingly expensive later in training. SOCAL also makes more searches per question than HiPRAG. Given the motivation concerning unnecessary searches and their associated costs, reporting total training time and inference costs would help establish the practical value of the improvements.
Questions for authors
  1. What substantive methodological insight does SOCAL add to HiPRAG beyond applying privileged-context self-distillation to local credit assignment? Please clarify how its design advances existing approaches such as EviSD and SD-Search. Where experimental settings or problem scopes differ, an explanation of those differences and their implications for comparison would be sufficient.
  2. What is the additional computational cost relative to HiPRAG? Could the authors report total training time or GPU-hours from existing runs, identify the overhead of hindsight scoring and dynamic sampling, and clarify the inference-cost trade-off?

I currently lean toward weak rejection because the methodological contribution over HiPRAG appears limited. I recognize the importance of the problem, the coherent design, and the positive results. If the rebuttal convincingly establishes the value of SOCAL’s specific contribution and clarifies the computational trade-off, I would consider raising my score, including toward acceptance.

5 comments

We thank the reviewer for recognizing the importance of the problem and the coherence of the design, and for clearly stating what would change the assessment.

Q1. Contribution relative to HiPRAG, EviSD and SD-Search.

We do not claim to be the first to use hindsight or same-model privileged scoring. Our contribution answers a question that prior process-aware methods leave open: once a step is (noisily) diagnosed as over-search, how much should its credit be corrected? HiPRAG answers with a fixed reward formula aggregated at the trajectory level; EviSD with a likelihood shift induced by dataset-provided evidence. SOCAL answers with the shift induced by the agent's own knowledge state at that step. This has three consequences:

  1. The diagnosis becomes conditioning information, not a reward. In HiPRAG a diagnosis enters as a fixed bonus or penalty on the whole trajectory. In SOCAL it becomes hindsight, and the policy itself determines how strongly that hindsight disagrees with the sampled step: a diagnosed step the policy remains confident about receives little correction, one that the hindsight clearly contradicts receives more.
  2. The privileged signal is model-specific. EviSD's evidence and gold answer are identical for every policy. Whether a search was unnecessary depends on what the current policy already knows, which only an on-policy probe reveals.
  3. The correction is localized and gated. Only diagnosed steps are modified, and only toward less support (Δ<0\Delta<0); all other steps keep their outcome advantage. This is what lets SOCAL search more than HiPRAG while over-searching less.
MethodMechanismDifference from SOCAL
HiPRAGStep-level necessity judgments aggregated into a trajectory-level process rewardSOCAL localizes the correction to the diagnosed step and sets its strength from the policy's own two-view likelihood gap
EviSDSame-model two-view scoring; evidence (search) / gold answer (answer) as privileged context; bounded adjustment on all action tokensPrivileged context is dataset-provided and not tied to over-search; no diagnosis gate
SD-SearchHindsight from sibling rollouts' queries and outcomes; auxiliary token-level JSD distillation loss at search-query positionsSOCAL uses a diagnosis-gated advantage correction instead of an extra loss, targeting the credit of unnecessary actions rather than query generation

We ran new experiments that test these claims directly (3B, same pipeline and evaluation protocol as the paper).

Constant-strength control (only ci,kc_{i,k} in Eq. 10 changes; diagnosis, step mask, DS and 320 steps fixed):

ci,kc_{i,k}CEMOSRSearch/Q
c=0c=0 (no correction; GRPO + DS)65.854.172.01
c=1c=1 (uniform attenuation)66.133.761.81
batch-mean cc66.303.621.88
adaptive cc (SOCAL)66.903.491.86

Comparison with EviSD and privileged-source swap:

Method (3B)Pooled CEMMacro CEMOSR
HiPRAG65.457.293.58
EviSD (full method, our pipeline)66.1059.634.37
SOCAL, hindsight → supporting evidence66.4660.094.42
SOCAL66.9061.273.49

From these results:

  • Adaptive strength matters. With all else fixed, adaptive cc beats the best constant cc by 0.60 CEM, 57% of the gain of calibration over no calibration.
  • The privileged source matters. Inside SOCAL's mechanism, replacing evidence with our hindsight lowers OSR by 0.93 and raises CEM by 0.44; full EviSD is 0.80 CEM below SOCAL with 0.88 higher OSR.
  • Searching less is not enough. Action-only hindsight (Table 2) gives the lowest Search/Q (1.70) but the worst CEM (63.95) and OSR (5.24).
3B variant (Table 2)CEMOSRSearch/Q
Trajectory-prefix hindsight65.354.862.06
Global hindsight64.754.631.97
Action-only hindsight63.955.241.70
Token-level adjustment66.623.661.91
Full SOCAL66.903.491.86

We did not run SD-Search because we could not obtain an implementation we could faithfully reproduce; EviSD has a public implementation, so we compared against it directly.

Q2. Computational cost.

Training. SOCAL's extra training computation consists of (i) a gradient-free no-search probe by the current actor for each search step, (ii) a binary equivalence judgment by an external LLM, and (iii) hindsight-conditioned scoring of diagnosed steps. The hindsight is template-rendered (no writer model), and scoring is a forward pass over already-sampled tokens (no regeneration). Measured per-step wall-clock time on 4×H100 (Qwen2.5-3B, batch 512, G=5G=5, micro-batch 32, identical retrieval and generation limits); SOCAL's time includes DS resampling:

MethodTime / step (s)GPU-h / stepStepsTotal GPU-h
Search-R1-style362.50.4028400161.1
EviSD451.70.5019300150.6
SOCAL465.80.5176320165.6

Each method uses its own training budget: 400 steps for Search-R1-style training (following HiPRAG), the official 300-step setting for EviSD, and 320 steps for SOCAL.

GPU-h = step time × 4 / 3600, including GPU time spent waiting on the judge; API fees and the separate retrieval server are excluded. Of SOCAL's 103 s per-step overhead over Search-R1, about 70 s is probe generation, 18 s is net (concurrent) judge waiting, and about 15 s is hindsight-conditioned scoring. SOCAL is 28.5% slower per step than Search-R1 and 3.1% slower than EviSD. In total training compute, SOCAL is within 3% of a 400-step Search-R1-style run (165.6 vs. 161.1 GPU-h) and about 10% above EviSD (165.6 vs. 150.6 GPU-h), which is trained for fewer steps.

Relative to HiPRAG: HiPRAG's process reward relies on the same kind of no-search probe and equivalence judgment, so SOCAL's additional overhead over HiPRAG is mainly hindsight scoring (about 15 s per step) and DS.

Inference. At test time SOCAL runs no probe, judge or self-distillation; it is a standard search agent. SOCAL issues 1.86 searches per question, versus 1.74 for HiPRAG (+6.9%) and 2.22 for Search-R1 (−16.2%), with higher CEM than both. Latency on 100 questions randomly sampled from the seven datasets (single H100, BF16, TP1, concurrency 1, greedy decoding, CPU retrieval, ≤ 4 searches; excluding initialization, warm-up and offline judging):

Method (3B)Latency (s / question)
Search-R14.749
EviSD4.510
SOCAL4.243

SOCAL is 10.7% faster than Search-R1. Relative to HiPRAG it makes slightly more searches; we view this as a deliberate accuracy–cost trade-off (+1.5 CEM for +0.12 searches per question), consistent with our goal of calibrating rather than suppressing retrieval. We will add these numbers to the paper.

We hope these new results address the reviewer's concerns, and we would be happy to clarify any remaining questions.

Dear Reviewer,

Thank you again for clearly stating what would change your assessment. As the discussion period is ending soon, we wanted to check whether our new experiments addressed these points: a constant-strength control isolating the contribution of adaptive calibration, a direct comparison against EviSD (+0.80 pooled CEM and −0.88 OSR for SOCAL), and measured training and inference cost (SOCAL is within 3% of a 400-step Search-R1-style run in total GPU-hours, and 10.7% faster than Search-R1 at inference).

We would greatly appreciate your feedback and are happy to answer any remaining questions.

Reviewer_NC2WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses over-search in RL-trained search agents. When an agent misjudges whether its own parametric knowledge suffices, it calls the retriever unnecessarily; because outcome-based RL assigns one credit to an entire trajectory, a successful answer also reinforces the unnecessary searches inside it. The authors show this empirically (Search-R1's over-search rate rises during training) and argue that existing process-aware fixes still decide how much to correct with a predefined reward formula or advantage-rescaling rule.

Their method, SOCAL, makes that correction strength decision-specific, in three stages:

Diagnosis and hindsight. For each search step, the policy answers the step's local query without retrieval, and an LLM judge marks the step as over-search if that answer matches the post-search conclusion. Each such step gets its own hindsight note (information need + knowledge-sufficiency evidence + decision guidance), with the probe answer, retrieved context and original conclusion deliberately excluded. Self-distilled calibration. The same pre-update policy scores the sampled step twice, with and without the hindsight prepended; the mean per-token log-likelihood gap is the step's calibration signal. Credit reshaping. The negative part of that gap, squashed through tanh, sets how strongly to attenuate the step's outcome advantage (scaled by |A|), leaving other steps untouched. Training is clipped GRPO with dynamic sampling. Validation uses Qwen2.5-3B/7B, trained on NQ + HotpotQA and evaluated with CEM on seven QA benchmarks (51,713 examples, five unseen). SOCAL reaches 66.9 / 70.1 average CEM, +1.5 / +2.9 over HiPRAG (the strongest process-aware baseline); at 7B it beats HiPRAG on all seven sets. Over-search rate drops from 4.92→3.49% (3B) and 5.69→3.92% (7B), and the share of trajectories containing any over-search roughly halves. The authors argue this is calibration rather than suppression, since SOCAL searches more than HiPRAG while over-searching less. Ablations vary hindsight scope (step vs global), placement (at the step vs at the prefix), its content (with vs without the knowledge-state evidence), and credit granularity (step vs token).

Critical strengths
  1. The problem framing and the "calibration, not suppression" argument are both well executed. The paper does not just claim that over-searching hurts; it shows Search-R1's OSR rising during training (Figure 3) as a direct consequence of trajectory-level credit, which is a concrete mechanism rather than a conjecture. Then, importantly, it pre-empts the obvious alternative explanation for its own gains: SOCAL issues more searches than HiPRAG yet has lower OSR and higher CEM, and the Search/Q decomposition shows the extra searches are judged necessary. That is the right way to defend a retrieval-efficiency claim, and it is why I take the result seriously despite the modest headline delta.
  2. The ablation set is targeted at the mechanism rather than at the hyperparameters, and the negative results are reported. Moving hindsight to the prefix (65.35), sharing one global hindsight (64.75), and dropping the knowledge-state evidence (63.95, with the lowest Search/Q at 1.70 but the highest OSR at 5.24) each isolate a distinct design commitment. The action-only row is especially informative: it demonstrates that a policy can be pushed to search less without becoming better calibrated, which is exactly the failure mode a naive "penalize search" reward would produce. That row alone justifies the hindsight design.
Critical weaknesses
  1. The paper's central claim — that the correction strength should adapt per step — is never tested against a constant-strength control. Everything else in the pipeline (hindsight construction, step-level localization, credit reshaping on top of GRPO) is a variant of existing ingredients, and the paper itself positions the contribution as: existing methods' "degree of correction ... remains largely determined by predefined reward formulations or advantage-rescaling rules", which SOCAL replaces with a decision-specific signal. But Table 2 contains no row where c_{i,k} is replaced by a fixed constant (c=1, i.e. a uniform attenuation λ|A_i| on every diagnosed over-search step). Such a row would directly measure the value of the tanh(β[−Δ]_+) machinery, which is the novelty. Without it, the reported +1.5/+2.9 over HiPRAG is equally consistent with "any bounded attenuation of credit on diagnosed over-search steps works", and the sophisticated per-step Δ_k computation would be unjustified overhead.

The nearest ablation, token-level versus step-level adjustment (66.62 vs 66.90 CEM, 3.66 vs 3.49 OSR), is a 0.28-point and 0.17-point difference with no reported variance, so it does not substitute for the missing control — and it is arguably evidence that granularity within the step barely matters, which cuts against the paper's own "trajectory-level credit is too coarse" motivation.
2. The closest prior work is not compared, and the over-search metric shares a signal family with the training objective. (a) EviSD (Xie et al., 2026a, cited in the paper's own related work) already converts token-level likelihood differences under privileged context into bounded adjustments of action-token advantages. SOCAL's mechanism is the same construction with a different privilege source (diagnosed-over-search hindsight instead of evidence/gold context) and a coarser granularity (step instead of action token). The paper discusses EviSD in one sentence and never compares to it, even though it is the baseline that would isolate what the hindsight framing adds. HiPRAG and LeTS are process-reward methods; EviSD is the method whose mechanics SOCAL reuses.

(b) OSR is computed by a fixed qwen3.7-flash "retrieval necessity" evaluator, and the training-time diagnosis is also an LLM semantic-equivalence judgment over a no-search probe. These are not the same judge, but they encode the same notion of necessity. SOCAL is explicitly optimized to reduce credit on steps that this notion flags, so a lower OSR is partly a statement about alignment with that notion rather than an independent measurement of search quality. The CEM gains are not affected by this, which is why I do not weight it more heavily — but the OSR and OS-Traj numbers should not be read as independent evidence.

Also missing: no variance or significance anywhere (one training run per setting), the pooled-example averaging of "Avg. CEM" makes the headline number sensitive to benchmark size composition (the unweighted seven-dataset mean is 61.3/64.6 rather than 66.9/70.1), and the baseline budget is mismatched in SOCAL's favour (SOCAL 320 steps vs baselines 400), which the paper should state as such rather than leave in Implementation Details.

Questions for authors
  1. Constant-strength control. What is CEM/OSR/Search-Q if c_{i,k} is fixed at 1 (uniform λ|A_i| attenuation on all diagnosed over-search steps), and if it is fixed at its mean over the batch? This is the single experiment I would most like to see; if adaptivity contributes most of the gain, my rating changes.
  2. EviSD comparison. Can you run EviSD's bounded action-token advantage adjustment in your pipeline, and a variant of SOCAL with EviSD's privilege source? That would separate "hindsight is the right privilege signal" from "privileged self-distillation is the right mechanism".
  3. Hindsight construction. Which model writes H_k, with what prompt, and is it the same model as the Eq. 4 judge? Is H_k generated once per diagnosed step at the pre-update policy, and does it become stale as the policy shifts over 320 steps?
  4. Diagnosis validity. The probe answers the local query q_k without retrieval, whereas the conclusion z_k follows from retrieved context. How often does probe agreement occur because both are wrong in the same way, or because the local sub-query is simply easier than the original question? Do you have a human-annotated sample of O?
  5. Metric independence. How much do OSR and OS-Traj move if the necessity evaluator is replaced by a different model (e.g., a 7B judge or a human spot-check on 200 steps)? Are the reported OSR comparisons stable under that swap?
  6. Variance and budget. Can you report at least two additional seeds for the 3B and 7B averages, and a HiPRAG run matched to 320 steps (or SOCAL to 400), so the comparison is budget-matched?
  7. Missing pieces. There is no limitations section and no appendix. Please add a limitations discussion (the diagnosis depends on the probe's calibration, which is itself the thing being trained) and the λ/β sensitivity. Please also report the macro-average over the seven datasets alongside the pooled average, and clarify whether the abstract's "1.5–2.9 percentage-point gains in CEM" refers to the pooled or the macro number.
4 comments

We thank the reviewer for the detailed and constructive review, and for recognizing the "calibration, not suppression" argument and the mechanism-targeted ablations.

Q1. Constant-strength control.

We agree this is the most direct test of our central claim and have run it. All variants share the same pipeline (diagnosis, step mask, DS, 320 steps); only ci,kc_{i,k} in Eq. 10 changes. c=0c=0 removes the correction (GRPO + DS); c=1c=1 applies a uniform λ∣Ai∣\lambda\lvert A_i\rvert attenuation to every diagnosed over-search step; batch-mean cc replaces each ci,kc_{i,k} by its batch mean.

ci,kc_{i,k}CEM (%) ↑OSR (%) ↓Search/Q
c=0c=0 (no correction; GRPO + DS)65.854.172.01
c=1c=1 (uniform attenuation)66.133.761.81
batch-mean cc66.303.621.88
SOCAL66.903.491.86

Any bounded attenuation helps (c=1c=1 vs. c=0c=0: +0.28 CEM, −0.41 OSR), but the adaptive coefficient is clearly better than the best constant (+0.60 CEM and −0.13 OSR over batch-mean cc; +0.77 CEM over c=1c=1). Of SOCAL's total +1.05 CEM gain over no correction, 0.60 points (57%) come from adaptivity rather than from the mere presence of attenuation. Notably, uniform attenuation (c=1c=1) yields the fewest searches (1.81) but not the best CEM or OSR: applying the same strength to every diagnosed step suppresses search, whereas the adaptive coefficient calibrates it. For scale, the SD over repeated evaluations of the SOCAL checkpoint is 0.22 (Q6).

On token- vs. step-level adjustment: that ablation varies granularity within an already-localized step. Our motivation contrasts trajectory-level with step-level credit, i.e., c=0c=0 (outcome credit only) or HiPRAG (trajectory-aggregated process reward) vs. SOCAL. We will clarify this.

Q2. Comparison with EviSD.

Both methods score sampled actions under a privileged and an unprivileged view of the same actor, but with different privileged information. EviSD conditions search actions on supporting evidence and answer actions on the gold answer, and adjusts all action tokens without any over-search gate. SOCAL's privileged information is the agent's own knowledge state at a given step, which only an on-policy probe can reveal, and the correction applies only to diagnosed over-search steps. We ran both suggested comparisons:

Method (3B)Pooled CEMMacro CEMOSR
Search-R162.754.964.92
HiPRAG65.457.293.58
EviSD (full method, our pipeline)66.1059.634.37
SOCAL, hindsight → supporting evidence66.4660.094.42
SOCAL66.9061.273.49

The EviSD row uses its bounded action-token advantage adjustment with both branches (search/evidence, answer/gold). The fourth row keeps SOCAL's diagnosis, step mask, DS and update rule fixed and swaps only the privileged source.

  • Mechanism (EviSD → SOCAL mechanism with evidence): +0.36 CEM; OSR essentially unchanged (4.37 → 4.42).
  • Privileged source (evidence → hindsight): +0.44 CEM, −0.93 OSR.

Overall SOCAL improves over EviSD by 0.80 pooled / 1.64 macro CEM and lowers OSR by 0.88. Evidence tells the policy what the answer is, not whether retrieval was needed; the over-search reduction comes almost entirely from the knowledge-state hindsight. We do not claim this ordering for all tasks, only that for over-search calibration a diagnosis-gated knowledge-state signal is the more appropriate privileged context.

Q3. Hindsight construction.

Hk\mathcal{H}_k is not written by any model. It is rendered from a fixed template (Appendix C.3) whose only slot is the local query qkq_k; the "observed basis" and "decision guidance" fields are fixed text triggered by a positive diagnosis. It contains no probe answer, original conclusion, or retrieved context. The Eq. 4 judge only outputs a binary semantic-equivalence label (prompt in Appendix C.2); it is Qwen3.7-Flash, the same model as the OSR evaluator. Because the two share a model, we validated the diagnosis with a second evaluator and with human annotation (Q4–Q5).

Hk\mathcal{H}_k is neither cached nor stale. At every training step and for every newly sampled rollout, the no-search probe is generated by the current pre-update policy πθold\pi_{\theta_\text{old}} (Algorithm 1), the diagnosis and hindsight are recomputed, and both views are scored by the same πθold\pi_{\theta_\text{old}} (Algorithm 2). The calibration signal thus always reflects the policy's current knowledge state.

Q4–Q5. Diagnosis validity and metric independence.

Human annotation. We randomly sampled 100 search steps with complete probe outputs from fixed evaluation trajectories. Two annotators, blind to the judge's labels, independently judged whether the no-search answer and the post-search conclusion were semantically equivalent (the judge's criterion); disagreements were adjudicated.

Human \ JudgeOver-searchNecessaryTotal
Over-search42446
Necessary25254
Total4456100

Agreement is 94% (Cohen's κ=0.88\kappa=0.88); the judge's over-search labels have 95.5% precision and 91.3% recall.

This validates the equivalence judgment, but not the case where probe and conclusion are both wrong in the same way. Two design choices limit its impact: the correction is bounded (c∈[0,1)c\in[0,1), so the advantage changes by at most λ∣Ai∣=0.2∣Ai∣\lambda\lvert A_i\rvert=0.2\lvert A_i\rvert), and it is applied only when the hindsight-conditioned view itself lowers the policy's support for the step (Δ<0\Delta<0). A spurious diagnosis on which the policy remains confident therefore yields little or no correction. We will discuss this in the Limitations section.

Evaluator swap. Keeping checkpoints, trajectories and probe outputs fixed, we re-judged every search step with a second evaluator from a different model family, GPT 5.6-Luna. CEM and Search/Q are unaffected by construction.

3BCEMOSR (Qwen3.7-Flash)OSR (GPT 5.6-Luna)
Search-R162.74.924.76
HiPRAG65.43.583.45
SOCAL66.93.493.40

OSR changes by at most 0.16 points, and the ranking Search-R1 > HiPRAG > SOCAL is preserved. SOCAL's reduction relative to Search-R1 is large and stable (−1.43 vs. −1.36). Its margin over HiPRAG is small under both evaluators (−0.09 vs. −0.05); we will state this explicitly and present SOCAL's advantage over HiPRAG primarily as higher CEM at a slightly lower OSR. We agree OSR should not be read as fully independent evidence and will note this in the paper.

Q6. Macro average, budget, and variance.

Macro average (unweighted over the seven datasets in Table 1):

Method3B Pooled3B Macro7B Pooled7B Macro
Search-R162.754.9665.759.39
HiPRAG65.457.2967.261.67
SOCAL66.961.2770.164.64
SOCAL − HiPRAG+1.5+3.99+2.9+2.97

The "1.5–2.9 points" in the abstract refers to the pooled average; SOCAL's advantage is preserved (at 3B enlarged) under macro averaging. We will report both.

Budget-matched comparison. We trained SOCAL without DS for 400 steps, matching the baselines in both step budget and sampling procedure:

MethodDynamic samplingStepsCEM (%)
HiPRAGNo40065.40
SOCAL w/o DSNo32065.11
SOCAL w/o DSNo40066.58
SOCALYes32066.90

Under an identical budget and without DS, SOCAL outperforms HiPRAG by 1.18 points. DS mainly lets SOCAL reach a similar level in fewer steps. Total training compute is also comparable: 320 SOCAL steps (including DS) take 165.6 GPU-hours on 4×H100, versus 161.1 for a 400-step Search-R1-style run (+2.8%).

Variance. We evaluated each fixed 3B checkpoint 10 times with stochastic (temperature-based) decoding, keeping datasets, prompts, retrieval corpus and evaluator fixed:

Method (3B)Pooled CEM, mean ± SDMacro CEM, mean ± SD
Search-R162.72 ± 0.1355.01 ± 0.16
HiPRAG65.44 ± 0.4757.26 ± 0.38
SOCAL66.88 ± 0.2261.26 ± 0.29

SOCAL exceeds Search-R1 and HiPRAG by 4.16 and 1.44 pooled points (6.25 and 4.00 macro), several times the evaluation SD. These runs capture evaluation variance, not training-seed variance; we report each setting from a single training run and will list this as a limitation.

Q7. Limitations, appendix, and λ/β sensitivity. A 15-page supplementary appendix (A–G) was submitted with the paper, containing pseudocode (B), all prompt contracts (C), metric definitions (D), the results ledger (E), case studies (F) and the λ/β sensitivity (G). We will reference it explicitly from the main text. Table A2 (3B; β=5\beta=5 when varying λ\lambda, λ=0.2\lambda=0.2 when varying β\beta):

SettingCEM (%)OSR (%)
λ=0.1\lambda=0.166.723.60
λ=0.4\lambda=0.466.683.44
β=1\beta=166.653.64
β=3\beta=366.813.55
β=7\beta=766.243.66
Default (λ=0.2\lambda=0.2, β=5\beta=5)66.903.49

CEM varies by at most 0.22 over λ\lambda and 0.66 over β\beta, and every setting outperforms HiPRAG (65.4). Stronger correction is not always better: λ=0.4\lambda=0.4 gives the lowest OSR but lower CEM than the default.

The new Limitations section will cover: (1) the diagnosis relies on the probe's calibration, which is itself being trained, and on an LLM equivalence judge; (2) the added training cost of probing, judging and hindsight scoring; (3) evaluation limited to 3B/7B models on knowledge-intensive QA with a four-call budget; (4) single training runs per setting.

We hope these new results address the reviewer's concerns, and we would be happy to clarify any remaining questions.

Dear Reviewer,

Thank you again for your detailed and constructive review. As the discussion period is ending soon, we wanted to briefly check whether our responses addressed your concerns. In particular, we ran the constant-strength control you suggested (the adaptive coefficient outperforms the best constant by +0.60 CEM, 57% of the calibration gain), a direct comparison with EviSD together with a privileged-source swap, human validation of the diagnosis (94% agreement, Cohen's κ = 0.88), and a cross-family evaluator swap for OSR. We also reported macro averages, a budget-matched comparison without dynamic sampling, and evaluation variance.

We would be grateful for any further feedback, and are happy to clarify anything that remains unclear.

Comments on the paper

1 comment

We thank the reviewers for their feedback. For convenience, here is a short summary of the new results in our responses:

  • Calibration mechanism. A constant-strength control shows that the adaptive coefficient, not attenuation alone, accounts for most of the gain.
  • Comparison with EviSD. We compare against EviSD directly and swap SOCAL's privileged source for supporting evidence.
  • Diagnosis and OSR. Human annotation of the over-search judge, and OSR re-judged with a second evaluator from a different model family.
  • Robustness. Macro averages, evaluation variance, a budget-matched run without dynamic sampling, and a search-cap sweep at evaluation time.
  • Cost. Measured training cost per step and in total, and inference latency.

We also clarified how the hindsight is constructed, and pointed to the λ/β sensitivity analysis already in Appendix G. We are happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.