acceptodds
Under review as a conference paper at ICLR 2027

Stop or Continue? Counterfactual Timing Supervision for Search Agents

Abstract

Multi-turn search agents must learn not only what to search for, but also when further interaction is worth its cost. Poor timing leads to two opposite errors: stopping prematurely with incomplete evidence, or continuing to search after sufficient evidence has already been acquired. Direct supervision of this timing decision requires a counterfactual comparison between the utility of continuing and that of stopping from the same decision context, whereas a standard rollout reveals only the outcome of the chosen action. We introduce Action-Timing, a cost-aware reinforcement learning framework with an explicit STOP/CONTINUE timing decision. Using main-trajectory branches together with matched same-action and counterfactual rollouts organized in a knowledge-state graph, Action-Timing estimates the Marginal Utility of Continuation (MUC), defined as the difference between the expected cost-adjusted downstream utilities of continuing and stopping at that decision point. MUC directly supervises whether further interaction is worthwhile, while continuation plans receive credit relative to the same stopping reference, separating learning when to continue from learning how to continue. Experiments on search-agent benchmarks show improved task performance over competing methods. Trajectory analyses further illustrate bidirectional timing correction, showing that Action-Timing can continue beyond premature stopping points to recover correct answers while terminating redundant searches earlier without sacrificing answer correctness.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Agentic Search, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yjMcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the trade-off challenge between answer accuracy and search efficiency in search agents. To jointly optimize these two objectives, the authors propose an agentic RL framework that leverages counterfactual rollouts to estimate the utility difference between continuing and stopping the search. Furthermore, the search planning policy is optimized by comparing the downstream utility of continuation plans against the utility of immediate stopping. To improve the stability of utility estimation and group-level advantage assignment, the authors organize trajectories reaching equivalent knowledge states into groups and aggregate their returns to provide more reliable training signals. Extensive experiments with existing methods and ablation studies demonstrate the effectiveness of the proposed method.

Critical strengths

Well-Motivated and Effective Methodology: The paper aims to address the trade-off between answer accuracy and search efficiency in search agents. It tackles this challenge through counterfactual utility comparisons between stopping and continuing. The key insight is to use immediate stopping as a common reference to optimize both when to search and how to search. This intuitive and coherent design enables more effective credit assignment, improving answer accuracy while reducing unnecessary searches.

Critical weaknesses
  1. Insufficient Ablations and Hyperparameter Analysis: The paper lacks a controlled ablation without counterfactual resampling. Although the existing ablations support the effectiveness of several components, the framework involves multiple loss terms and hyperparameters, with limited sensitivity analysis to assess their robustness.

  2. Limited Novelty: The core idea of counterfactual supervision for search timing has already been explored in DAS. Although this work introduces cost-aware utility estimation and separate timing and planning optimization, the overall novelty appears incremental.

Questions for authors
  1. Could the authors clarify how the loss weights (e.g., λT\lambda_T, λA\lambda_A, and γAKL\gamma_{\mathrm{AKL}}) and reliability thresholds were selected? Given the multiple objectives and heuristic thresholds, a sensitivity analysis would help assess the robustness of the proposed framework.
  2. Could the authors provide a more comprehensive efficiency comparison with existing methods, including EM, ASQ, OSR, USR, training time, and GPU-hours? Although Figure 3 reports some search efficiency metrics, a unified comparison across strong baselines would better demonstrate the accuracy–efficiency trade-off and whether the performance gains justify the additional computational overhead of counterfactual rollouts.
5 comments
AuthorsOpenReview ↗

Rebuttal [1/3]

Response to Reviewer yjMc

We thank the reviewer for the careful assessment and for recognizing the value of using immediate stopping as a shared reference for timing and planning. We address the concerns below.

W1. Counterfactual Resampling and Controlled Ablations

Counterfactual resampling serves a distinct role from increasing the number of rollouts for an observed action. Same-action resampling provides additional outcomes for the selected action at a fixed decision context, whereas counterfactual resampling supplies outcomes for the otherwise unobserved alternative action. Since MUC-based Timing supervision and STOP-referenced Planning credit rely on comparing STOP and CONTINUE utilities from the same context, removing counterfactual branches therefore changes not only the sampling procedure but also the value evidence available for constructing Timing and Planning supervision. Thus, removing counterfactual resampling while keeping the proposed Timing and Planning objectives unchanged is not a well-defined ablation, since the required same-context action values would no longer be available.

Our experiments provide two complementary comparisons: one isolates the contributions of Timing and Planning objectives under fixed counterfactual resampling, while the other tests whether increasing conventional GRPO sampling alone can explain the performance gains.

First, Table 2 holds counterfactual resampling, graph construction, and action-cost settings fixed. Replacing the Timing objective, Planning objective, or both with GRPO updates reduces average EM from 45.9 to 43.1, 41.7, and 40.1, respectively. These controlled ablations establish the contribution of the proposed Timing and Planning objectives under identical counterfactual sampling conditions.

Second, we evaluate whether increasing conventional GRPO sampling can reproduce Action-Timing's gains under a comparable generation-token budget on Qwen2.5-3B-Instruct.

Both methods use the same Qwen2.5-3B-Instruct initialization, training data, STOP/CONTINUE protocol, cost-aware reward, and generation/retrieval settings. GRPO-16 follows the standard Search-R1 GRPO objective with 16 rollouts per question.

MethodOutput tokens (×10⁴)Sampling time (min)Avg. EM
GRPO-5 (ours)——31.9
GRPO-1646.1021.0833.8
Action-Timing45.6714.0142.8

Increasing GRPO from 5 to 16 rollouts improves average EM from 31.9 to 33.8, whereas Action-Timing achieves 42.8 with a closely matched generation-token budget and lower sampling time.

Together, these experiments demonstrate the benefits of dedicated Timing–Planning supervision under fixed counterfactual resampling and show that additional conventional GRPO rollouts do not reproduce Action-Timing's gains under a comparable generation-token budget.

W2. Methodological Distinction from DAS

While DAS and Action-Timing both employ counterfactual interventions, the key distinction lies in the learning problem formulated from their outcomes. DAS constructs sequence-level preference pairs from factual and counterfactual trajectories, jointly influencing search timing and subsequent response content. Action-Timing instead uses matched utility estimates to establish a shared decision criterion for two related but distinct objectives: whether further search is worthwhile, and whether a particular continuation plan improves upon answering immediately.

A better continuation is not necessarily a worthwhile continuation. A search plan may outperform alternative plans yet still yield less utility than stopping. Action-Timing explicitly captures this distinction by using immediate stopping as a common value reference: MUC directly supervises the STOP/CONTINUE decision through the difference between continuation and stopping utilities, while STOP-referenced Planning credit evaluates each continuation plan against the same stopping value. This connects action selection and plan-level credit assignment through a shared utility criterion.

Our controlled ablations substantiate this design. Table 2 shows that replacing the dedicated Timing objective, Planning objective, or both with GRPO updates reduces average EM from 45.9 to 43.1, 41.7, and 40.1, respectively. More directly, Table 8 retains the complete Timing objective and changes only the Planning reference from immediate STOP utility to the CONTINUE-group mean. This reduces average EM from 45.9 to 42.8, while OSR increases from 25.2% to 26.1% and USR from 44.2% to 47.1%.

These results show that direct Timing supervision does not subsume the role of immediate stopping as a Planning reference: even with the Timing objective held fixed, replacing the STOP reference degrades performance. The central contribution is therefore a shared value-based optimization principle that connects when to search with which search plans are worth reinforcing.

AuthorsOpenReview ↗

Rebuttal [2/3]

Q1. Optimization Coefficients and Reliability Thresholds

We set the optimization coefficients to balance the relative contributions of the training signals according to their intended roles in the joint objective. We set λT=0.05\lambda_T=0.05 to give direct MUC-based Timing supervision a relatively small external scale, controlling its influence alongside other training objectives. For Analysis, λA=0.5\lambda_A=0.5 introduces knowledge-state-relative corrections at a reduced scale while retaining the question-level GRPO advantage as the primary signal. Action-level KL regularization constrains changes in the STOP/CONTINUE distribution relative to the frozen rollout policy, including eligible decisions without accepted MUC targets. We set γAKL=2.0\gamma_{\mathrm{AKL}}=2.0, yielding an effective Action KL coefficient of λT×γAKL=0.10\lambda_T \times \gamma_{\mathrm{AKL}} = 0.10 in the total actor loss.

We further evaluate the sensitivity of these coefficients on Qwen2.5-7B-Instruct, reporting macro-averaged results across seven QA benchmarks. Each variant changes one effective supervision or regularization strength while holding the others fixed. When increasing λT\lambda_T to 0.10, we accordingly reduce γAKL\gamma_{\mathrm{AKL}} to 1.0 to preserve the effective Action KL coefficient.

Variant (λT\lambda_T, λA\lambda_A, γAKL\gamma_{\mathrm{AKL}})EM (%) ↑ASQOSR (%) ↓USR (%) ↓
Full (0.05, 0.50, 2.0)45.92.2225.244.2
Timing (0.10, 0.50, 1.0)45.52.0924.945.3
Analysis (0.05, 1.00, 2.0)45.82.2425.344.4
Action KL (0.05, 0.50, 1.0)45.22.1925.545.1

The tested variations yield average EM between 45.2% and 45.8%, compared with 45.9% under the default configuration. Doubling the Analysis correction coefficient changes EM by only 0.1 points, while doubling direct Timing supervision or halving the effective Action KL strength reduces EM by 0.4 and 0.7 points, respectively. Across the tested coefficient variations, average EM remains within 0.7 percentage points of the default configuration, indicating limited sensitivity to these local changes.

Reliability thresholds control which estimated utility comparisons are used for supervision. The STOP reliability threshold τS=0.70\tau_S=0.70 requires at least three correct answers among four STOP evaluations before a negative MUC can reinforce stopping. Among qualifying STOP evaluation pools, 80% achieve 4/4 correctness. Teacher refinement requires at least three supporting child returns, correctness of at least 0.80, a utility improvement exceeding 0.05, and an eligible positive-advantage Plan. In our training runs, Teacher refinement was activated for only 3–5% of accepted nonzero Timing targets.

The Timing weight cap wmax⁡=0.25w_{\max}=0.25 limits the influence of large estimated utility gaps. In our training logs, 44.4% of accepted nonzero targets contribute 84.1% of the uncapped supervision weight; the cap reduces their share to 68.5% without changing target directions.

We further test the weight cap and STOP reliability threshold through controlled single-factor variations on Qwen2.5-7B-Instruct, holding other settings fixed.

ConfigurationEM (%) ↑ASQOSR (%) ↓USR (%) ↓
Full (wmax⁡=0.25w_{\max}=0.25, STOP 3/4)45.92.2225.244.2
wmax⁡=0.50w_{\max}=0.5045.52.1325.045.2
STOP reliability 4/445.62.3725.943.9

Increasing the weight cap to 0.50 changes EM by −0.4 points, while tightening STOP reliability from 3/4 to 4/4 changes EM by −0.3 points and increases ASQ from 2.22 to 2.37. These controlled variations indicate limited EM sensitivity over the tested settings.

AuthorsOpenReview ↗

Rebuttal [3/3]

Q2. Accuracy–Efficiency Trade-off and Training Cost

We provide a unified accuracy–search-efficiency comparison for Search-R1, DAS, ReSeek, and Action-Timing under the same evaluation protocol. We restrict this comparison to baselines with publicly available checkpoints that can be evaluated consistently, as several other methods have not released their trained checkpoints. Results are macro-averaged across seven QA benchmarks using the evaluated 7B models.

MethodEM (%) ↑ASQOSR (%) ↓USR (%) ↓
Search-R138.52.4929.748.5
DAS44.42.4229.248.0
ReSeek42.32.2027.557.8
Action-Timing45.92.2225.244.2

Action-Timing achieves the highest average EM and the lowest OSR and USR among the four methods, with comparable or fewer search calls. Relative to DAS, EM improves by 1.5 percentage points while ASQ decreases by 8.3%, OSR by 4.0 points, and USR by 3.8 points. Compared with ReSeek, Action-Timing improves EM by 3.6 points and reduces USR by 13.6 points, with nearly identical search usage (2.22 versus 2.20 ASQ). These results demonstrate a favorable accuracy–search-efficiency trade-off under the shared evaluation protocol.

For Qwen2.5-7B-Instruct, the average per-step training time is approximately 29.18 minutes, comprising 6.60 minutes for main-trajectory sampling, 9.57 minutes for matched resampling, and 13.01 minutes for other training operations. Across 200 steps on eight NVIDIA A100 GPUs, this yields an estimated training time of 97.27 hours and 778.1 allocated GPU-hours per run. Matched resampling accounts for approximately 32.8% of the reported per-step training time.

To assess the computational trade-off, we additionally compare Action-Timing against GRPO-16 on Qwen2.5-3B-Instruct under closely matched generated-token budgets. The following table reports average per-step costs and the corresponding allocated GPU-hours over 200 training steps on eight NVIDIA A100 GPUs. Output-token counts include all sampled outputs, including matched counterfactual branches for Action-Timing, while sampling and other training times cover non-overlapping intervals.

MethodOutput tokens (×10⁴)Sampling (min)Other (min)Total (min)Allocated GPU-hours / runAvg. EM
GRPO-1646.1021.086.3027.38730.133.8
Action-Timing45.6714.019.1023.11616.342.8

Under comparable generated-token budgets, Action-Timing achieves 9.0 percentage points higher average EM with approximately 15.6% lower reported per-step training time.

Together, these results demonstrate a favorable accuracy–search-efficiency trade-off, report the measured time spent on matched resampling, and show that increasing conventional GRPO sampling under a comparable generation-token budget does not reproduce Action-Timing's performance gains.

We hope these additional experiments and clarifications address your concerns regarding the methodological novelty and empirical support for Action-Timing. We respectfully ask you to reconsider your assessment of our work in light of this evidence.

Reviewer_SaUmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Action-Timing jointly learns when to stop searching and which search action to take. Its central move is to evaluate continuation relative to answering immediately from the same state, using a counterfactual rollout graph. This provides a shared reference for timing and planning, rather than treating search quality and stopping as separate problems.

Critical strengths
  • The STOP reference is a useful methodological contribution. A query can be better than other queries but still worse than answering now; the proposed objective explicitly captures this distinction.

  • The controlled ablations are convincing about the value of joint training. Table 2 keeps the protocol, rollout graph, resampling, and cost setting fixed while replacing disabled regions with GRPO. The separate losses from removing timing and planning provide stronger evidence than the benchmark improvements alone.

Critical weaknesses
  • The meaning of the continuation target deserves closer examination. MUC averages equally over observed successor evidence groups, rather than weighting them by the deployed policy's probability of reaching them. This avoids overcounting repeated branches, but it can reward useful continuations that the current planner rarely realizes. Table 3 already compares group-balanced and branch-averaged targets; the remaining question is not which wins on aggregate, but whether the balanced target reliably identifies states where continuing actually helps.

  • The experiments establish an effective training system more clearly than they establish accurate stopping decisions. Final accuracy mixes timing, query selection, and answer generation. In particular, an incorrect stopped answer does not by itself show premature stopping: additional search may not have recovered it. A direct intervention at held-out stopping states would make the central timing claim substantially sharper.

Questions for authors
  1. At held-out decision states, how well does the sign of MUC predict the realized utility difference between stopping and continuing under the learned policy? States where group-balanced and frequency-weighted targets disagree would be especially informative.

  2. When the policy stops with an incorrect answer, how often does forcing an additional search actually recover the answer under the same remaining budget? Conversely, how much utility is lost by forcing an earlier stop? These interventions would separate timing quality from general answer quality.

1 comment

Response to Reviewer SaUm

We thank the reviewer for the constructive feedback and for recognizing the value of our shared STOP reference and controlled ablations. We address the concerns regarding group-balanced MUC estimation and timing-decision quality through two complementary held-out counterfactual evaluations.

W1 & Q1. Group-Balanced MUC and Its Decision Relevance

Regarding Weakness 1, group-balanced MUC assigns equal weight to distinct observed successor evidence groups when estimating continuation utility (Sections 3.2–3.4). This design reflects the structure of matched resampling, where continuation exploration discovers successor groups and value completion allocates additional samples to refine their return estimates. Consequently, branch frequencies reflect deliberate sampling allocation as well as policy behavior, rather than directly representing the policy's successor-state probabilities. Group balancing allows additional samples to refine within-group utility estimates without assigning greater aggregation weight to an existing successor group solely because it receives more samples.

We randomly select 400 held-out decision states with valid nonzero MUC estimates, estimate MUC using matched resampling, and independently evaluate forced STOP/CONTINUE outcomes. For evaluation, only the current timing action is forced; the learned policy generates subsequent Plan/Search and timing decisions naturally, without group balancing or value completion. Sign agreement is computed against the sign of the independently estimated cost-adjusted utility gap (CONTINUE − STOP).

AggregationSign agreement ↑Correct in disagreements
Group-balanced95.25%16/18
Branch-averaged91.75%2/18

The two aggregation methods disagree in predicted sign on 18/400 states (4.50%). Group-balanced MUC also correlates with independent utility gaps (Pearson r = 0.905). These results support its decision relevance, including when the two aggregation methods disagree.

W2 & Q2. Direct Evaluation of Timing Decisions

To directly evaluate STOP/CONTINUE decisions, we independently draw another 400 held-out decision states without conditioning on the policy's timing choice or answer correctness. At each state, we preserve the analysis prefix, evidence, and remaining budget, and independently sample four STOP answers and eight CONTINUE trajectories. Only the current timing action is forced; subsequent decisions follow the learned policy. We report forced-action accuracy and the mean cost-adjusted utility difference (ΔU = CONTINUE − STOP). For naturally selected STOP states, the correct/incorrect distinction refers to the original answer, rather than the independently resampled STOP outcomes.

Natural decisionStatesForced STOP EM (%)Forced CONT. EM (%)ΔU
CONTINUE21321.3637.68+0.1247
STOP, correct7796.7584.25−0.1814
STOP, incorrect1104.5520.11+0.1298

The mean continuation gain at naturally selected CONTINUE states has a 95% CI of [0.0800, 0.1718]. At states where the policy naturally stops incorrectly, forced CONTINUE trajectories achieve 20.11% EM, compared with 4.55% for resampled STOP answers. These state-matched interventions quantify both the benefits and costs of continuation: further search improves utility on average at naturally selected CONTINUE states, reduces utility on average at states where the policy naturally stops with a correct answer, and can recover some incorrect STOP outcomes. Together, these results help distinguish the effects of search timing from final-answer correctness alone.

Reviewer_Sx4nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an explicit STOP/CONTINUE action for multi-turn search agents. At selected decision points, it freezes the analysis prefix and samples matched forced-stopping and forced-continuation branches. Continuation branches are grouped by the evidence they retrieve first. The Marginal Utility of Continuation (MUC) is the difference in cost-adjusted utility between continuing and stopping. It serves as a weighted cross-entropy target for the timing token and as a STOP-referenced advantage for search plans. On seven Wikipedia QA benchmarks with Qwen2.5-3B/7B, the method achieves the best average EM (45.9 vs. 44.4 for DAS at 7B) and lower over-/under-searching rates.

Critical strengths
  1. The paper frames search timing clearly as a counterfactual problem. The explicit decision boundary with a frozen prefix allows both actions to be compared from the same context.
  2. Learning when to continue is separated from learning how to continue, with a shared stopping reference. The ablations support that both components contribute.
  3. The appendices specify the method in detail, code is provided, and the reported numbers are internally consistent.
Critical weaknesses
  1. The gains are small, concentrated in-domain, and not compute-matched. No variance or significance tests are reported. Recomputing from Table 1, the 7B margin over DAS is +3.5 EM in-domain (NQ, HotpotQA) but only +0.7 EM on the five held-out sets. At 3B, the margin over Tree-GRPO is 0.34 EM. Each decision point can cost up to ~15 extra rollouts, yet the paper reports neither compute figures nor equal-budget baselines.
  2. Many hand-tuned heuristics with no sensitivity analysis. Target construction and sampling rely on numerous fixed thresholds and quotas, e.g., a stopping-accuracy gate of 0.7, a 0.25 weight cap, and 4/6/6 branch quotas. Several loss coefficients are not reported. Apart from λc, none of these settings is varied. The Teacher refinement adds substantial complexity but covers only 3–5% of targets and is never ablated. Performance also appears sensitive to design details: changing only the aggregation or the planning reference lowers EM to 42.8–45.1, below DAS (Tables 3, 8).
  3. The method combines ideas from cited prior work: branched rollouts for step-level credit (Tree-GRPO), state-matched grouping (GiGPO), and counterfactual timing supervision (CaRT; DAS). DAS in particular already applies forced-answer and forced-search interventions at decision points. The paper distinguishes itself from DAS mainly at the objective level. However, the two methods also differ in training regime and signal type, the ablations compare only against GRPO, and the DAS reproduction is not described, so it is unclear which difference drives the gain.
Questions for authors
  1. Could you report standard deviations across the three seeds, separately for the in-domain and out-of-domain benchmarks? Could you also report the per-batch training overhead relative to GRPO (resampled decision points, extra rollouts, tokens, and wall-clock time)? If feasible, an equal-rollout-budget comparison at 3B would help.
  2. What are the values of the unreported coefficients, and how sensitive are the results to τ_S, w_max, and the branch quotas? What happens if the Teacher refinement is removed?
  3. How was DAS reproduced (starting checkpoint, training data, LoRA vs. full fine-tuning, budget)? Could you discuss, or if feasible test, which difference from DAS accounts for the gain, e.g., by training with DAS-style preference pairs built on the same matched branches?
5 comments
AuthorsOpenReview ↗

Rebuttal [1/3]

Response to Reviewer Sx4n

We thank the reviewer for the constructive feedback and for recognizing our counterfactual timing formulation and the shared STOP reference connecting when to continue with how to continue. We address the concerns regarding performance robustness, computational cost, hyperparameter sensitivity, and methodological novelty with new experiments and clarifications.

W1 & Q1. OOD Performance, Training Variability, and Sampling Cost

(1) OOD Performance and Search Behavior.

Regarding the +0.7 EM margin over DAS on the five held-out benchmarks, average EM alone does not fully capture the accuracy–search behavior trade-off. Under the DAS diagnostic protocol (Figure 3), Action-Timing achieves lower OSR and USR across all five held-out benchmarks. On TriviaQA, EM ties at 67.4 while ASQ decreases and OSR falls from 56.4% to 43.6%. On MuSiQue, ASQ increases while EM improves from 20.6 to 21.4 and USR decreases from 74.0% to 69.1%. On PopQA, 2WikiMultihopQA, and Bamboogle, EM improves while ASQ decreases. These results reflect dataset-dependent search behavior consistent with learning when further search justifies its cost.

(2) Training Variability.

Across three independent 7B training runs (seeds 42, 24, and 2026), the macro-averaged EM scores are:

Benchmark groupEM (%) ± SD
ID (2 benchmarks)47.51 ± 0.21
OOD (5 benchmarks)45.31 ± 0.18
Overall (7 benchmarks)45.94 ± 0.19

The three OOD run scores are 45.19, 45.52, and 45.23, all exceeding the evaluated DAS checkpoint's 44.60 EM.

(3) Sampling Cost and Compute-Matched Comparison.

The following table reports per-batch sampling averages for the 7B configuration.

Sampling stageRolloutsOutput tokens (×10⁴)Time (min)
Main trajectories64014.406.60
Matched resampling~3,85241.549.57
Total~4,49255.9416.17

Each batch resamples approximately 472 decision occurrences, averaging 8.16 additional rollouts per occurrence (~1,719 STOP and ~2,133 CONTINUE branches). The total sampling time is 2.45× that of the main-trajectory sampling phase. Including 13.01 minutes for other training operations, the measured average total training time is 29.18 minutes per step.

We further compare Action-Timing against GRPO with increased sampling on Qwen2.5-3B-Instruct, using the same training setup and evaluation protocol.

MethodRollouts/questionOutput tokens (×10⁴)Sampling time (min)Total time (min/step)Avg. EM
GRPO5———31.9
GRPO1646.1021.0827.3833.8
Action-Timing5 + resampling45.6714.0123.1142.8

Under a closely matched generation-token budget, Action-Timing achieves a 9.0-point higher average EM than GRPO with 16 rollouts and outperforms it on all seven benchmarks, with lower sampling time. The corresponding total per-step training times are 23.11 and 27.38 minutes, respectively, including both sampling and other training operations. This indicates that increasing conventional GRPO sampling alone does not reproduce the gains of Action-Timing.

AuthorsOpenReview ↗

Rebuttal [2/3]

W2 & Q2. Optimization Settings and Sensitivity

(1) Optimization Settings and Teacher Ablation.

The following optimization coefficients are shared by the 3B and 7B models.

ComponentSetting
TimingλTλ_T = 0.05; γAKL\gamma_{\mathrm{AKL}} =2.0 (effective Action KL coefficient 0.10)
Regularizationββ = 0.001; ηη = 0.001; λPλ_P = 0.01
Local / PPOλAλ_A = 0.5; clocc_{\mathrm{loc}} = 1.0; εclipε_{\mathrm{clip}} = 0.2

The stopping threshold τSτ_S = 0.70 requires at least 3/4 correct STOP evaluations before a negative MUC can reinforce STOP. Among qualifying STOP evaluation pools, 80% have 4/4 correct answers and 20% have 3/4.

The weight cap wmaxw_{\mathrm{max}} = 0.25 controls the concentration of Timing supervision. In our training logs, 44.4% of accepted nonzero Timing targets account for 84.1% of uncapped supervision weight. After capping, their share decreases to 68.5%, while 55.6% of targets retain their original weights. The cap limits the influence of large utility gaps without changing target directions.

Teacher refinement uses existing matched-rollout evidence to refine supported CONTINUE targets, without additional rollouts or teacher-model inference. It activates on only 3–5% of accepted nonzero Timing targets. We additionally conduct a controlled Teacher-off ablation, disabling only Teacher target refinement while retaining base MUC timing supervision, STOP-referenced Planning credit, and the original auxiliary Plan selection.

VariantAvg. EM ↑ASQOSR ↓USR ↓
Action-Timing (Full)45.92.2225.244.2
w/o Teacher refinement45.72.1125.144.5

These modest changes are consistent with Teacher's limited activation, while the main performance gains of Action-Timing are retained without Teacher refinement.

(2) Hyperparameter and Resampling Sensitivity.

We additionally conduct controlled single-factor sensitivity tests for the Timing weight cap, STOP reliability, and resampling quotas, holding other settings fixed. The default uses wmax⁡=0.25w_{\max}=0.25, a 3/4 STOP reliability requirement, and 4/6/6 resampling quotas (STOP / CONTINUE exploration / value completion).

ConfigurationAvg. EM ↑ASQOSR ↓USR ↓
Full (4/6/6)45.92.2225.244.2
wmax⁡=0.50w_{\max}=0.5045.52.1325.045.2
STOP reliability 4/445.62.3725.943.9
Reduced exploration (4/4/6)44.92.4628.546.6
Reduced completion (4/6/4)45.32.2626.143.7

Changing the cap or STOP reliability requirement changes EM by at most 0.4 points. Reducing value completion yields a 0.6-point EM decrease; reducing exploration yields a larger 1.0-point decrease, with OSR and USR increasing by 3.3 and 2.4 points, respectively. These results indicate limited EM sensitivity to the tested cap and reliability threshold, with greater sensitivity to exploration quota under the tested budgets.

Regarding the design sensitivity noted in Tables 3 and 8, these experiments examine structural choices rather than numerical hyperparameters. In Table 3, individually changing utility aggregation or plan value sharing yields 45.1 and 44.7 EM, respectively, both above DAS (44.4). The STOP-reference ablation in Table 8 (42.8 EM) is discussed in W3.

AuthorsOpenReview ↗

Rebuttal [3/3]

W3 & Q3. Methodological Contribution and DAS Comparison

(1) Methodological distinction. Regarding the novelty concern in Weakness 3, a better continuation is not necessarily a worthwhile continuation. A search plan may outperform alternative plans yet still provide less utility than answering immediately. This distinction motivates Action-Timing to use the value of stopping as a common counterfactual reference, directly supervising whether to continue and assigning local credit according to whether each continuation plan improves upon stopping.

Finer-grained credit assignment within search does not establish whether the search was worth undertaking. Tree-GRPO and GiGPO improve credit attribution through branched trajectories and grouped states, but do not explicitly connect timing and planning through a shared stopping reference. DAS introduces counterfactual answer/search comparisons, yet converts them into sequence-level preference pairs rather than assigning continuation plans numerical credit relative to the same STOP value. Action-Timing bridges this gap by making the value of not searching a common basis for both deciding when to search and learning which search plans are worth reinforcing.

(2) Controlled experimental evidence. Under the same explicit timing protocol, graph construction, matched resampling, and action-cost settings, replacing the Timing or Planning objective with GRPO updates reduces average EM from 45.9 to 43.1 and 41.7, respectively, while replacing both yields 40.1 (Table 2).

More directly, Table 8 retains the full Timing objective and changes only the Planning reference. Replacing the STOP value with the CONTINUE-group mean reduces EM from 45.9 to 42.8. This shows that directly supervising when to continue does not substitute for evaluating continuation plans against the value of stopping.

Together, these controlled ablations demonstrate the complementary contributions of direct Timing supervision and STOP-referenced Planning credit within Action-Timing, supporting a value-based learning mechanism distinct from DAS's sequence-level preference optimization.

(3) DAS evaluation protocol. Regarding the DAS baseline, Table 1 reports our evaluation of the officially released DAS LoRA adapter on its Search-R1 base checkpoint across all seven benchmarks, without retraining or further fine-tuning. We used shared general evaluation settings while retaining the official DAS-specific configuration.

We hope these additional experiments and clarifications strengthen the empirical evidence for Action-Timing and further clarify its methodological contributions. We respectfully ask you to reconsider your overall assessment in light of this additional evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.