- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Selective-Inference Policy Optimization (SIPO), a framework for improving credit estimation in tree-structured reinforcement learning for search agents. SIPO introduces three components: (1) a scale-free branch criterion that normalizes generation scores before applying sibling-count penalties; (2) exchangeable branching that samples multiple fresh continuations from selected parent states while preserving the overall leaf budget; and (3) an order-statistic correction that adjusts retained incumbent values based on their selection ranks and the estimated association between generation scores and outcomes. Experiments across seven QA benchmarks using Qwen3-4B, Qwen3-8B, and Qwen2.5-7B demonstrate improvements over both tree-free and tree-based baselines. Component ablations and branch-level diagnostics further provide empirical support for the proposed components.
Critical strengths
-
The paper identifies a potentially important source of bias in adaptive tree-based RL: selected incumbent branches and fresh continuations have different selection histories, even when they share the same parent context. Explicitly accounting for this asymmetry provides a useful perspective on credit estimation in search-agent training.
-
The ablation study evaluates the three components individually and in combination. In addition, the selected–fresh and fresh–fresh comparisons provide empirical evidence consistent with the proposed selection-history asymmetry, while the rank-model diagnostics examine whether the correction captures the direction of the observed value displacement.
Critical weaknesses
-
The correction models the relationship between generation scores and outcomes using a linear conditional-mean assumption and an order-statistic approximation. However, actual rollout candidates may exhibit heterogeneous score–outcome relationships across prompts, parent states, and training stages. The reported diagnostics also show that the correction can overestimate the magnitude of the observed displacement for some candidate-count groups. It remains unclear how robust the correction is when these modeling assumptions are substantially violated.
-
The improvements over across all three backbones are generally modest. The paper does not clearly establish the statistical significance of the main accuracy improvements across independent training runs. More rigorous uncertainty estimates would help distinguish consistent improvements from run-to-run variability. Moreover, the experiments focus on text-based QA agents using a particular retrieval setup. It is unclear whether the approach transfers to more complex agentic workflows, different tool environments, or substantially longer interaction horizons.
Questions for authors
-
The correction assumes a particular relationship between standardized generation scores and outcomes, while the actual rollout data may exhibit nonlinear or heterogeneous relationships across prompts and parent states. How sensitive is SIPO to violations of this assumption? Could the authors evaluate its performance under different score–outcome correlations or compare the current correction with alternative, less restrictive estimators?
-
Can the authors report uncertainty estimates or statistical tests based on independent training runs for the main results, particularly the aggregate accuracy improvements? Could the authors provide sensitivity analyses for the OSC correction strength () and the sibling-penalty coefficient ()?
2 comments
Response to Reviewer 3 (2/2)
R3.3 Correction-strength and sibling-penalty sensitivity
Could the authors provide sensitivity analyses for the OSC correction strength ... and the sibling-penalty coefficient ...?
Response. The intermediate correction-strength run, , and the lower sibling-penalty run, , have both completed. Each starts from the original Qwen2.5-7B model, changes only the indicated algorithm parameter relative to full SIPO, and preserves the 22-leaf budget, 240 updates, and full evaluation every 20 updates.
| OSC strength , fixed | Best EM (%) | Best update | Update-240 EM (%) |
|---|---|---|---|
| 0 | 44.78 | 240 | 44.78 |
| 0.5 | 44.15 | 180 | 33.99 |
| 1 | 46.58 | 220 | 44.87 |
| Sibling penalty , fixed | Best EM (%) | Best update | Update-240 EM (%) |
|---|---|---|---|
| 0.025 | 41.71 | 100 | 30.05 |
| 0.05 | 46.58 | 220 | 44.87 |
The reported setting, and , outperforms these two alternative settings at both their best evaluations and the common final update. The alternative settings also show late degradation: invalid responses reach 36.80% for and 42.39% for at update 240. These observations reveal parameter sensitivity and support the reported setting over the tested alternatives in these runs.
The third sibling-penalty setting, at fixed , is currently training under the same protocol. Its completed-run result is pending.
R3.4 Uncertainty across independent runs
Can the authors report uncertainty estimates or statistical tests based on independent training runs for the main results, particularly the aggregate accuracy improvements?
Response. Each end-to-end training run in our agentic RL setting takes approximately 2–3 days. Repeating the experiments with three independent training seeds would therefore require substantially more GPU time. We plan to extend the main experiments to three independent seeds as additional computational resources become available, and report the mean performance and variability across runs.
Response to Reviewer 3 (1/2)
We have completed a weaker-assumption correction baseline and additional sensitivity runs. We added an empirical-mean correction control to compare the training performance of OSC with a correction that applies the same adjustment regardless of candidate rank. The parameter results identify where the reported configuration performs better and where late-training degradation occurs.
R3.1 Heterogeneity and a less restrictive correction
Response. We completed a less restrictive estimator comparison in the actual training process. The empirical correction uses the preceding batch's observed mean incumbent–fresh displacement. It requires neither Gaussian order statistics nor a linear score–outcome relationship, while preserving SFC+EXB, K=3,B=2, the 22-leaf budget, correction eligibility, and the training/evaluation schedule.
OSC exceeds the empirical-average estimator by 1.51 points at the best evaluation and 1.04 points at update 240. Previously reported configurations retain their manuscript values; new experiments report question-weighted EM over 22,523 questions. EM differences are calculated from the displayed scores.
R3.2 Interpreting the observed overcorrection
Response. The diagnostic measures calibration error, while the training ablation measures downstream utility; we report both. In Figure 4(b–c), the mean observed displacement is −0.0778 and the prediction is −0.1054 across 1,536 pairs from 256 Base trees. For candidate counts 21–30, the residual is +0.0450 with a 95% tree-bootstrap interval of [0.0042, 0.0839], confirming magnitude overestimation in that group.
These are early-training, final-round diagnostics in raw reward units. They identify a concrete departure from the reference approximation. The completed training comparison nevertheless favors OSC over the empirical mean-displacement estimator at both the best evaluation and the common final update. We will present the grouped residuals together with that downstream comparison.