- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the trade-off challenge between answer accuracy and search efficiency in search agents. To jointly optimize these two objectives, the authors propose an agentic RL framework that leverages counterfactual rollouts to estimate the utility difference between continuing and stopping the search. Furthermore, the search planning policy is optimized by comparing the downstream utility of continuation plans against the utility of immediate stopping. To improve the stability of utility estimation and group-level advantage assignment, the authors organize trajectories reaching equivalent knowledge states into groups and aggregate their returns to provide more reliable training signals. Extensive experiments with existing methods and ablation studies demonstrate the effectiveness of the proposed method.
Critical strengths
Well-Motivated and Effective Methodology: The paper aims to address the trade-off between answer accuracy and search efficiency in search agents. It tackles this challenge through counterfactual utility comparisons between stopping and continuing. The key insight is to use immediate stopping as a common reference to optimize both when to search and how to search. This intuitive and coherent design enables more effective credit assignment, improving answer accuracy while reducing unnecessary searches.
Critical weaknesses
-
Insufficient Ablations and Hyperparameter Analysis: The paper lacks a controlled ablation without counterfactual resampling. Although the existing ablations support the effectiveness of several components, the framework involves multiple loss terms and hyperparameters, with limited sensitivity analysis to assess their robustness.
-
Limited Novelty: The core idea of counterfactual supervision for search timing has already been explored in DAS. Although this work introduces cost-aware utility estimation and separate timing and planning optimization, the overall novelty appears incremental.
Questions for authors
- Could the authors clarify how the loss weights (e.g., , , and ) and reliability thresholds were selected? Given the multiple objectives and heuristic thresholds, a sensitivity analysis would help assess the robustness of the proposed framework.
- Could the authors provide a more comprehensive efficiency comparison with existing methods, including EM, ASQ, OSR, USR, training time, and GPU-hours? Although Figure 3 reports some search efficiency metrics, a unified comparison across strong baselines would better demonstrate the accuracy–efficiency trade-off and whether the performance gains justify the additional computational overhead of counterfactual rollouts.
5 comments
Rebuttal [2/3]
Q1. Optimization Coefficients and Reliability Thresholds
We set the optimization coefficients to balance the relative contributions of the training signals according to their intended roles in the joint objective. We set to give direct MUC-based Timing supervision a relatively small external scale, controlling its influence alongside other training objectives. For Analysis, introduces knowledge-state-relative corrections at a reduced scale while retaining the question-level GRPO advantage as the primary signal. Action-level KL regularization constrains changes in the STOP/CONTINUE distribution relative to the frozen rollout policy, including eligible decisions without accepted MUC targets. We set , yielding an effective Action KL coefficient of in the total actor loss.
We further evaluate the sensitivity of these coefficients on Qwen2.5-7B-Instruct, reporting macro-averaged results across seven QA benchmarks. Each variant changes one effective supervision or regularization strength while holding the others fixed. When increasing to 0.10, we accordingly reduce to 1.0 to preserve the effective Action KL coefficient.
| Variant (, , ) | EM (%) ↑ | ASQ | OSR (%) ↓ | USR (%) ↓ |
|---|---|---|---|---|
| Full (0.05, 0.50, 2.0) | 45.9 | 2.22 | 25.2 | 44.2 |
| Timing (0.10, 0.50, 1.0) | 45.5 | 2.09 | 24.9 | 45.3 |
| Analysis (0.05, 1.00, 2.0) | 45.8 | 2.24 | 25.3 | 44.4 |
| Action KL (0.05, 0.50, 1.0) | 45.2 | 2.19 | 25.5 | 45.1 |
The tested variations yield average EM between 45.2% and 45.8%, compared with 45.9% under the default configuration. Doubling the Analysis correction coefficient changes EM by only 0.1 points, while doubling direct Timing supervision or halving the effective Action KL strength reduces EM by 0.4 and 0.7 points, respectively. Across the tested coefficient variations, average EM remains within 0.7 percentage points of the default configuration, indicating limited sensitivity to these local changes.
Reliability thresholds control which estimated utility comparisons are used for supervision. The STOP reliability threshold requires at least three correct answers among four STOP evaluations before a negative MUC can reinforce stopping. Among qualifying STOP evaluation pools, 80% achieve 4/4 correctness. Teacher refinement requires at least three supporting child returns, correctness of at least 0.80, a utility improvement exceeding 0.05, and an eligible positive-advantage Plan. In our training runs, Teacher refinement was activated for only 3–5% of accepted nonzero Timing targets.
The Timing weight cap limits the influence of large estimated utility gaps. In our training logs, 44.4% of accepted nonzero targets contribute 84.1% of the uncapped supervision weight; the cap reduces their share to 68.5% without changing target directions.
We further test the weight cap and STOP reliability threshold through controlled single-factor variations on Qwen2.5-7B-Instruct, holding other settings fixed.
| Configuration | EM (%) ↑ | ASQ | OSR (%) ↓ | USR (%) ↓ |
|---|---|---|---|---|
| Full (, STOP 3/4) | 45.9 | 2.22 | 25.2 | 44.2 |
| 45.5 | 2.13 | 25.0 | 45.2 | |
| STOP reliability 4/4 | 45.6 | 2.37 | 25.9 | 43.9 |
Increasing the weight cap to 0.50 changes EM by −0.4 points, while tightening STOP reliability from 3/4 to 4/4 changes EM by −0.3 points and increases ASQ from 2.22 to 2.37. These controlled variations indicate limited EM sensitivity over the tested settings.
Rebuttal [3/3]
Q2. Accuracy–Efficiency Trade-off and Training Cost
We provide a unified accuracy–search-efficiency comparison for Search-R1, DAS, ReSeek, and Action-Timing under the same evaluation protocol. We restrict this comparison to baselines with publicly available checkpoints that can be evaluated consistently, as several other methods have not released their trained checkpoints. Results are macro-averaged across seven QA benchmarks using the evaluated 7B models.
| Method | EM (%) ↑ | ASQ | OSR (%) ↓ | USR (%) ↓ |
|---|---|---|---|---|
| Search-R1 | 38.5 | 2.49 | 29.7 | 48.5 |
| DAS | 44.4 | 2.42 | 29.2 | 48.0 |
| ReSeek | 42.3 | 2.20 | 27.5 | 57.8 |
| Action-Timing | 45.9 | 2.22 | 25.2 | 44.2 |
Action-Timing achieves the highest average EM and the lowest OSR and USR among the four methods, with comparable or fewer search calls. Relative to DAS, EM improves by 1.5 percentage points while ASQ decreases by 8.3%, OSR by 4.0 points, and USR by 3.8 points. Compared with ReSeek, Action-Timing improves EM by 3.6 points and reduces USR by 13.6 points, with nearly identical search usage (2.22 versus 2.20 ASQ). These results demonstrate a favorable accuracy–search-efficiency trade-off under the shared evaluation protocol.
For Qwen2.5-7B-Instruct, the average per-step training time is approximately 29.18 minutes, comprising 6.60 minutes for main-trajectory sampling, 9.57 minutes for matched resampling, and 13.01 minutes for other training operations. Across 200 steps on eight NVIDIA A100 GPUs, this yields an estimated training time of 97.27 hours and 778.1 allocated GPU-hours per run. Matched resampling accounts for approximately 32.8% of the reported per-step training time.
To assess the computational trade-off, we additionally compare Action-Timing against GRPO-16 on Qwen2.5-3B-Instruct under closely matched generated-token budgets. The following table reports average per-step costs and the corresponding allocated GPU-hours over 200 training steps on eight NVIDIA A100 GPUs. Output-token counts include all sampled outputs, including matched counterfactual branches for Action-Timing, while sampling and other training times cover non-overlapping intervals.
| Method | Output tokens (×10⁴) | Sampling (min) | Other (min) | Total (min) | Allocated GPU-hours / run | Avg. EM |
|---|---|---|---|---|---|---|
| GRPO-16 | 46.10 | 21.08 | 6.30 | 27.38 | 730.1 | 33.8 |
| Action-Timing | 45.67 | 14.01 | 9.10 | 23.11 | 616.3 | 42.8 |
Under comparable generated-token budgets, Action-Timing achieves 9.0 percentage points higher average EM with approximately 15.6% lower reported per-step training time.
Together, these results demonstrate a favorable accuracy–search-efficiency trade-off, report the measured time spent on matched resampling, and show that increasing conventional GRPO sampling under a comparable generation-token budget does not reproduce Action-Timing's performance gains.
We hope these additional experiments and clarifications address your concerns regarding the methodological novelty and empirical support for Action-Timing. We respectfully ask you to reconsider your assessment of our work in light of this evidence.
Rebuttal [1/3]
Response to Reviewer yjMc
We thank the reviewer for the careful assessment and for recognizing the value of using immediate stopping as a shared reference for timing and planning. We address the concerns below.
W1. Counterfactual Resampling and Controlled Ablations
Counterfactual resampling serves a distinct role from increasing the number of rollouts for an observed action. Same-action resampling provides additional outcomes for the selected action at a fixed decision context, whereas counterfactual resampling supplies outcomes for the otherwise unobserved alternative action. Since MUC-based Timing supervision and STOP-referenced Planning credit rely on comparing STOP and CONTINUE utilities from the same context, removing counterfactual branches therefore changes not only the sampling procedure but also the value evidence available for constructing Timing and Planning supervision. Thus, removing counterfactual resampling while keeping the proposed Timing and Planning objectives unchanged is not a well-defined ablation, since the required same-context action values would no longer be available.
Our experiments provide two complementary comparisons: one isolates the contributions of Timing and Planning objectives under fixed counterfactual resampling, while the other tests whether increasing conventional GRPO sampling alone can explain the performance gains.
First, Table 2 holds counterfactual resampling, graph construction, and action-cost settings fixed. Replacing the Timing objective, Planning objective, or both with GRPO updates reduces average EM from 45.9 to 43.1, 41.7, and 40.1, respectively. These controlled ablations establish the contribution of the proposed Timing and Planning objectives under identical counterfactual sampling conditions.
Second, we evaluate whether increasing conventional GRPO sampling can reproduce Action-Timing's gains under a comparable generation-token budget on Qwen2.5-3B-Instruct.
Both methods use the same Qwen2.5-3B-Instruct initialization, training data, STOP/CONTINUE protocol, cost-aware reward, and generation/retrieval settings. GRPO-16 follows the standard Search-R1 GRPO objective with 16 rollouts per question.
Increasing GRPO from 5 to 16 rollouts improves average EM from 31.9 to 33.8, whereas Action-Timing achieves 42.8 with a closely matched generation-token budget and lower sampling time.
Together, these experiments demonstrate the benefits of dedicated Timing–Planning supervision under fixed counterfactual resampling and show that additional conventional GRPO rollouts do not reproduce Action-Timing's gains under a comparable generation-token budget.
W2. Methodological Distinction from DAS
While DAS and Action-Timing both employ counterfactual interventions, the key distinction lies in the learning problem formulated from their outcomes. DAS constructs sequence-level preference pairs from factual and counterfactual trajectories, jointly influencing search timing and subsequent response content. Action-Timing instead uses matched utility estimates to establish a shared decision criterion for two related but distinct objectives: whether further search is worthwhile, and whether a particular continuation plan improves upon answering immediately.
A better continuation is not necessarily a worthwhile continuation. A search plan may outperform alternative plans yet still yield less utility than stopping. Action-Timing explicitly captures this distinction by using immediate stopping as a common value reference: MUC directly supervises the STOP/CONTINUE decision through the difference between continuation and stopping utilities, while STOP-referenced Planning credit evaluates each continuation plan against the same stopping value. This connects action selection and plan-level credit assignment through a shared utility criterion.
Our controlled ablations substantiate this design. Table 2 shows that replacing the dedicated Timing objective, Planning objective, or both with GRPO updates reduces average EM from 45.9 to 43.1, 41.7, and 40.1, respectively. More directly, Table 8 retains the complete Timing objective and changes only the Planning reference from immediate STOP utility to the CONTINUE-group mean. This reduces average EM from 45.9 to 42.8, while OSR increases from 25.2% to 26.1% and USR from 44.2% to 47.1%.
These results show that direct Timing supervision does not subsume the role of immediate stopping as a Planning reference: even with the Timing objective held fixed, replacing the STOP reference degrades performance. The central contribution is therefore a shared value-based optimization principle that connects when to search with which search plans are worth reinforcing.