acceptodds
Under review as a conference paper at ICLR 2027

SIPO: Selective-Inference Policy Optimization for Tree-Structured Agentic RL

Abstract

Tree-structured reinforcement learning trains search agents by comparing alternative continuations and propagating terminal rewards to intermediate decisions. Adaptive expansion, however, creates a statistical asymmetry: an incumbent is selected using its own generation statistic, whereas fresh siblings are sampled after selection. When that statistic is associated with return, branch values can reflect selection history as well as continuation quality, even for a shared parent. We propose Selective-Inference Policy Optimization (SIPO), which incorporates this distinction into tree-based credit estimation. Its scale-free branch criterion keeps generation scores and sibling penalties on a consistent relative scale; exchangeable branching supplies multiple fresh continuations from each selected parent; and order-statistic correction adjusts retained incumbent values using selection rank and the estimated score–outcome association. These mechanisms preserve the leaf budget and the host policy optimisation objective. Across seven QA benchmarks using Qwen3-4B, Qwen3-8B, and Qwen2.5-7B, SIPO achieves the highest reported multi-hop and single-hop averages among the compared methods. On Qwen3-8B, it improves these averages over AT2PO by and percentage points, respectively, and ranks first on six of seven benchmarks. Component ablations evaluate the individual and combined changes, while early-training paired diagnostics show a selected–fresh value gap alongside a near-zero fresh–fresh reference. Together, these results support accounting for selection history when constructing and evaluating search-agent rollouts. We will open-source our code upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Tree Search, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZhmtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective-Inference Policy Optimization (SIPO), a framework for improving credit estimation in tree-structured reinforcement learning for search agents. SIPO introduces three components: (1) a scale-free branch criterion that normalizes generation scores before applying sibling-count penalties; (2) exchangeable branching that samples multiple fresh continuations from selected parent states while preserving the overall leaf budget; and (3) an order-statistic correction that adjusts retained incumbent values based on their selection ranks and the estimated association between generation scores and outcomes. Experiments across seven QA benchmarks using Qwen3-4B, Qwen3-8B, and Qwen2.5-7B demonstrate improvements over both tree-free and tree-based baselines. Component ablations and branch-level diagnostics further provide empirical support for the proposed components.

Critical strengths
  • The paper identifies a potentially important source of bias in adaptive tree-based RL: selected incumbent branches and fresh continuations have different selection histories, even when they share the same parent context. Explicitly accounting for this asymmetry provides a useful perspective on credit estimation in search-agent training.

  • The ablation study evaluates the three components individually and in combination. In addition, the selected–fresh and fresh–fresh comparisons provide empirical evidence consistent with the proposed selection-history asymmetry, while the rank-model diagnostics examine whether the correction captures the direction of the observed value displacement.

Critical weaknesses
  • The correction models the relationship between generation scores and outcomes using a linear conditional-mean assumption and an order-statistic approximation. However, actual rollout candidates may exhibit heterogeneous score–outcome relationships across prompts, parent states, and training stages. The reported diagnostics also show that the correction can overestimate the magnitude of the observed displacement for some candidate-count groups. It remains unclear how robust the correction is when these modeling assumptions are substantially violated.

  • The improvements over AT2PO\texttt{AT}^2\texttt{PO} across all three backbones are generally modest. The paper does not clearly establish the statistical significance of the main accuracy improvements across independent training runs. More rigorous uncertainty estimates would help distinguish consistent improvements from run-to-run variability. Moreover, the experiments focus on text-based QA agents using a particular retrieval setup. It is unclear whether the approach transfers to more complex agentic workflows, different tool environments, or substantially longer interaction horizons.

Questions for authors
  • The correction assumes a particular relationship between standardized generation scores and outcomes, while the actual rollout data may exhibit nonlinear or heterogeneous relationships across prompts and parent states. How sensitive is SIPO to violations of this assumption? Could the authors evaluate its performance under different score–outcome correlations or compare the current correction with alternative, less restrictive estimators?

  • Can the authors report uncertainty estimates or statistical tests based on independent training runs for the main results, particularly the aggregate accuracy improvements? Could the authors provide sensitivity analyses for the OSC correction strength (ww) and the sibling-penalty coefficient (λ\lambda)?

2 comments

Response to Reviewer 3 (1/2)

We have completed a weaker-assumption correction baseline and additional sensitivity runs. We added an empirical-mean correction control to compare the training performance of OSC with a correction that applies the same adjustment regardless of candidate rank. The parameter results identify where the reported configuration performs better and where late-training degradation occurs.

R3.1 Heterogeneity and a less restrictive correction

The correction models the relationship between generation scores and outcomes using a linear conditional-mean assumption and an order-statistic approximation.

How sensitive is SIPO to violations of this assumption? Could the authors evaluate its performance under different score–outcome correlations or compare the current correction with alternative, less restrictive estimators?

Response. We completed a less restrictive estimator comparison in the actual training process. The empirical correction uses the preceding batch's observed mean incumbent–fresh displacement. It requires neither Gaussian order statistics nor a linear score–outcome relationship, while preserving SFC+EXB, K=3,B=2K=3, B=2, the 22-leaf budget, correction eligibility, and the training/evaluation schedule.

Estimator with SFC+EXBBest EM (%)Update-240 EM (%)
No correction44.7844.78
Empirical mean displacement45.0743.83
OSC46.5844.87

OSC exceeds the empirical-average estimator by 1.51 points at the best evaluation and 1.04 points at update 240. Previously reported configurations retain their manuscript values; new experiments report question-weighted EM over 22,523 questions. EM differences are calculated from the displayed scores.

R3.2 Interpreting the observed overcorrection

The reported diagnostics also show that the correction can overestimate the magnitude of the observed displacement for some candidate-count groups.

Response. The diagnostic measures calibration error, while the training ablation measures downstream utility; we report both. In Figure 4(b–c), the mean observed displacement is −0.0778 and the prediction is −0.1054 across 1,536 pairs from 256 Base trees. For candidate counts 21–30, the residual is +0.0450 with a 95% tree-bootstrap interval of [0.0042, 0.0839], confirming magnitude overestimation in that group.

These are early-training, final-round diagnostics in raw reward units. They identify a concrete departure from the reference approximation. The completed training comparison nevertheless favors OSC over the empirical mean-displacement estimator at both the best evaluation and the common final update. We will present the grouped residuals together with that downstream comparison.

Response to Reviewer 3 (2/2)

R3.3 Correction-strength and sibling-penalty sensitivity

Could the authors provide sensitivity analyses for the OSC correction strength ... and the sibling-penalty coefficient ...?

Response. The intermediate correction-strength run, w=0.5w=0.5, and the lower sibling-penalty run, λ=0.025\lambda=0.025, have both completed. Each starts from the original Qwen2.5-7B model, changes only the indicated algorithm parameter relative to full SIPO, and preserves the 22-leaf budget, 240 updates, and full evaluation every 20 updates.

OSC strength ww, fixed λ=0.05\lambda=0.05Best EM (%)Best updateUpdate-240 EM (%)
044.7824044.78
0.544.1518033.99
146.5822044.87
Sibling penalty λ\lambda, fixed w=1w=1Best EM (%)Best updateUpdate-240 EM (%)
0.02541.7110030.05
0.0546.5822044.87

The reported setting, w=1w=1 and λ=0.05\lambda=0.05, outperforms these two alternative settings at both their best evaluations and the common final update. The alternative settings also show late degradation: invalid responses reach 36.80% for w=0.5w=0.5 and 42.39% for λ=0.025\lambda=0.025 at update 240. These observations reveal parameter sensitivity and support the reported setting over the tested alternatives in these runs.

The third sibling-penalty setting, λ=0.10\lambda=0.10 at fixed w=1w=1, is currently training under the same protocol. Its completed-run result is pending.

R3.4 Uncertainty across independent runs

Can the authors report uncertainty estimates or statistical tests based on independent training runs for the main results, particularly the aggregate accuracy improvements?

Response. Each end-to-end training run in our agentic RL setting takes approximately 2–3 days. Repeating the experiments with three independent training seeds would therefore require substantially more GPU time. We plan to extend the main experiments to three independent seeds as additional computational resources become available, and report the mean performance and variability across runs.

Reviewer_njWEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies selection asymmetry in tree-structured RL applied to LLM search search agents. In this context, the authors attack the sampling credit assignment problem in which continuations from intermediate states. The proposed SIPO modifies AT^2PO in three ways: scale-free branch selection, exchangable sampling of fresh continuation and an OSC of selected branch values before credit propagation. The method is evaluated using 3 Qwen models on 7 benchmarks, showing a small improvement over AT^2PO

Critical strengths
  1. Relevant and interesting problem formulation. The paper provides a useful perspective on credit assignment and estimation in tree-structured RL.
  2. The experiment a broad benchmark with improvements over the best reported baseline and provides a component ablation study.
Critical weaknesses
  1. Tab. 2 shows that adding OSC improves performance, but the authors provide no comparison with a simpler correction, such as a constant adjustment of selected branch values.
  2. The OSC in (3) is derived assuming independent fresh outcomes while the algorithm uses fresh continuations from selected parents.
Questions for authors
  1. What is the exact estimation target of OSC? Why should we consider the observed fresh return difference as an estimation bias and not a difference in action quality?
  2. How does the OSC compare with a more simple rank-independent correction, for example subtracting the average observed fresh displacement?
  3. Why is the accuracy in Tab. 2 lower than the AT^2PO result in Tab. 1? What are the differences in training setup?
  4. The improvement w.r.t. the best baselines is limited and, to my understanding, only the performance for one training seed is provided. Could you provide results including confidence curves over multiple training seeds?
2 comments

Response to Reviewer 2 (1/2)

We completed the suggested empirical constant-correction baseline. OSC exceeds it by 1.51 percentage points at the best evaluation, with the same rollout budget and training schedule. We first clarify the estimation target, then give the controlled comparison.

R2.1 The target of OSC and the role of action quality

What is the exact estimation target of OSC? Why should we consider the observed fresh return difference as an estimation bias and not a difference in action quality?

Response. The target in Eq. (3) is the expected outcome displacement associated with rank selection relative to a specified independent reference. Let (Zi,Yi)(Z_i,Y_i) be i.i.d. candidate pairs, with Zi∼N(0,1)Z_i\sim\mathcal N(0,1) and E[Yi∣Zi]=a0+a2Zi\mathbb E[Y_i\mid Z_i]=a_0+a_2Z_i. If Y[r]Y_{[r]} is the outcome attached to the rr-th largest score and Y0Y_0 is an independent reference outcome with mean a0a_0, then

Δ(r,n)=E[Y[r]−Y0]=a2E[Z(r:n)]≈a2h(r,n).\Delta(r,n)=\mathbb E[Y_{[r]}-Y_0]=a_2\mathbb E[Z_{(r:n)}]\approx a_2h(r,n).

This target captures a selection-associated displacement. An incumbent's realized generation score participates in selecting an expansion location; a fresh continuation is generated after that location is selected. Their different sampling histories motivate the adjustment before credit propagation.

Action quality can contribute to the observed contrast. For a fixed parent state ss and incumbent action segment aa, let Q(s,a)=E[R∣s,a]Q(s,a)=\mathbb E[R\mid s,a] and V(s)=Ea′∼π[Q(s,a′)]V(s)=\mathbb E_{a'\sim\pi}[Q(s,a')]. The terminal-return difference decomposes as

Rinc−Rfresh=[Q(s,a)−V(s)]+[Rinc−Q(s,a)]−[Rfresh−V(s)].R_{\rm inc}-R_{\rm fresh}=[Q(s,a)-V(s)]+[R_{\rm inc}-Q(s,a)]-[R_{\rm fresh}-V(s)].

The first term is an action-quality difference; selection may also change the conditional return fluctuations.

Response to Reviewer 2 (2/2)

R2.3 Comparison with an empirical constant correction

How does the OSC compare with a more simple rank-independent correction, for example subtracting the average observed fresh displacement?

Response. We implemented and trained this empirical-average control. OSC achieves the highest best-evaluation EM among the three correction variants below.

Correction with SFC+EXBBest EM (%)
No correction44.78
Empirical constant correction45.07
OSC46.58

OSC's advantage over the constant correction is 1.51 percentage points at the best evaluation. The control retains Qwen2.5-7B, SFC+EXB, K=3,B=2K=3, B=2, 22 leaves per prompt, 64 prompts per update, 240 updates, and complete evaluation every 20 updates. It preserves correction eligibility and value propagation, replacing the adjustment magnitude alone.

For final-round expansion event ee, we record the incumbent leaves before fresh attachment and calculate the unadjusted displacement in the same standardized value units:

De=V‾inc,e−V‾fresh,e,ct−1=1∣Et−1∣∑e∈Et−1De.D_e=\overline V_{{\rm inc},e}-\overline V_{{\rm fresh},e},\qquad c_{t-1}=\frac{1}{|\mathcal E_{t-1}|}\sum_{e\in\mathcal E_{t-1}}D_e.

Fresh branches are averaged within each event, and events receive equal weight. In the next batch, every eligible value in a rollout worker receives the same adjustment, V~=V−ct−1\widetilde V=V-c_{t-1}, independent of rank and candidate count. The estimate is worker-local, matching the historical implementation; the first batch has no correction, and evaluation does not update this state.

R2.4 Published results and local ablations

Why is the accuracy in Tab. 2 lower than the AT^2PO result in Tab. 1? What are the differences in training setup?

Response. Several local baseline reproductions achieve lower scores than their published results. We retain the published baseline scores in Table 1, where AT²PO is 45.83%, while Table 2 reports our local component ablation, whose AT²PO Base is 42.98%. For the Qwen2.5-7B multi-hop QA comparison in this paper, SIPO achieves state-of-the-art (SOTA) performance with an aggregate EM of 46.58%, whether compared against published baseline results or our local reproductions. The difference between the two AT²PO values therefore reflects different experimental records.

R2.5 Independent training seeds

The improvement w.r.t. the best baselines is limited and, to my understanding, only the performance for one training seed is provided. Could you provide results including confidence curves over multiple training seeds?

Response. Each end-to-end training run in our agentic RL setting takes approximately 2–3 days. Repeating the experiments with three independent training seeds would therefore require substantially more GPU time. We plan to extend the main experiments to three independent seeds as additional computational resources become available, and report the mean performance and variability across runs.

Reviewer_yLgFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes to improve credit estimation in tree-structured agentic RL through introducing a scale-free branch criterion, exchangeable branching and order-statistic correction. The authors motivate their approach by showing that when expanding the tree early in training, under the baseline configuration, the incumbent's advantage values are lower compared to freshly sampled siblings from the same parent. The paper compares the method, Selective-Inference Policy Optimization (SIPO) to trajectory- and tree-based baseline methods on multi-hop and single-hop QA benchmarks using three different backbones. They use two training runs per backbone: one for multi-hop, and one for single-hop QA. For each method, they choose the checkpoint that has the best-so-far performance on average on the evaluation benchmarks. The authors show that for a single training run, SIPO achieves the highest score among all methods on 5 of 7 benchmarks for Qwen2.5-7B backbone, 6 of 7 benchmarks for Qwen3-8B backbone, and 4 of 7 benchmarks for Qwen3-4B backbone.

Critical strengths
  • The fresh-fresh control as a diagnostic is clever.
  • The leaf budget is matched across methods.
  • The paper identifies reasonable problem with credit assignment in tree-structured agentic RL.
  • The suggested components improve performance on the multi-hop and single-hop QA benchmarks.
Critical weaknesses
  • Writing is not clear, e.g. Figure 1 contains unexplained "Shipped" labels.
  • Only one training run per method is conducted, so it is hard to judge significance of observed improvements
  • Using the average performance on the test sets for checkpoint selection is suboptimal compared to using a held-out validation set.
  • The best-so-far performance of baselines in Figure 2 does not correspond with the reported average values in Table 1. The best-so-far evaluation performance for GSPO in Figure 2 is ~43, but in Table 1 it is 44.4. For AT2PO, the best-so-far evaluation performance in Figure 2 is ~43, but in Table 1 it is 45.83.
  • Tree-GRPO is not in Figure 2.
  • Adding SFC+OSC at K=6 B=1 to the ablation would clarify whether EXB contributes to the OSC gain in the full configuration.
  • The measured training costs for the Qwen2.5-7B backbone in Table 4 only show costs for the Base method and SIPO.
Questions for authors
  • What are the training costs for other baselines in your experimental protocol?
  • Is the base method AT2PO?
  • Why is the performance of AT2PO in Table 1 different from its best-so-far evaluation performance in Figure 2?
  • What is the training set for the single-hop QA task family?
  • Why does SIPO have a higher token and time training budget compared to Base in Table 4?
  • What is the performance of SFC+OSC at K=6 and B=1?
4 comments

Response to Reviewer 1 (1/3)

We have completed the requested SFC+OSC ablation, expanded the cost comparison to all six completed local methods, and added results at the common final update. These results strengthen the component and baseline comparisons. Below we address each concern and clarify the provenance of the reported numbers.

R1.1 Figure 1 terminology and the Base configuration

Writing is not clear, e.g. Figure 1 contains unexplained "Shipped" labels.

Is the base method AT2PO?

Response. Yes. Base is our local AT²PO host configuration with SFC, EXB, and OSC disabled, using (M,L,K,B)=(10,2,6,1)(M,L,K,B)=(10,2,6,1). “Shipped” refers to that reference implementation. We will replace the label with “AT²PO reference implementation”, use “SIPO” for our method, and define Base consistently in the captions of Figure 1 and Tables 2 and 4.

R1.2 Evidence across independent training runs

Only one training run per method is conducted, so it is hard to judge significance of observed improvements

Response. Each end-to-end training run in our agentic RL setting takes approximately 2–3 days. Repeating the experiments with three independent training seeds would therefore require substantially more GPU time. We plan to extend the main experiments to three independent seeds as additional computational resources become available, and report the mean performance and variability across runs.

R1.3 Checkpoint selection

Using the average performance on the test sets for checkpoint selection is suboptimal compared to using a held-out validation set.

Response. SIPO's advantage over the strongest completed local baseline remains at the common final update. At update 240, SIPO achieves 44.87% EM and DAPO achieves 43.98%, a difference of 0.89 percentage points. Under the best-evaluation protocol, the corresponding scores are 46.58% and 44.95%, a difference of 1.63 points. The full local comparison appears below.

The original protocol selects the highest question-weighted EM on the benchmark evaluation/test sets. The supplemental update-240 comparison uses the same endpoint for every method and therefore removes separate peak selection from that comparison.

Response to Reviewer 1 (2/3)

R1.4 Published results and local reproductions

The best-so-far performance of baselines in Figure 2 does not correspond with the reported average values in Table 1.

Why is the performance of AT2PO in Table 1 different from its best-so-far evaluation performance in Figure 2?

Response. Several local baseline reproductions achieve lower scores than their published results. We retain the published baseline scores in Table 1 and report our local training trajectories in Figure 2. For the Qwen2.5-7B multi-hop QA comparison in this paper, SIPO achieves state-of-the-art (SOTA) performance with an aggregate EM of 46.58%, whether compared against published baseline results or our local reproductions. Figure 2 shows the score at each scheduled evaluation, rather than a running-best envelope.

MethodTable 1 published reference (%)Local best EM (%)Best updateLocal update-240 EM (%)
AT²PO Base45.8342.9818041.41
GRPO44.4844.2514038.37
GSPO44.4042.284040.10
AEPO44.5143.4310035.71
DAPO44.9144.9510043.98
SIPO—46.5822044.87

Previously reported local configurations retain their manuscript values throughout this response. Published reference scores and local results are labeled separately.

R1.5 Tree-GRPO coverage

Tree-GRPO is not in Figure 2.

Response. We will add Tree-GRPO to Figure 2.

R1.6 The requested SFC+OSC ablation

Adding SFC+OSC at K=6 B=1 to the ablation would clarify whether EXB contributes to the OSC gain in the full configuration.

What is the performance of SFC+OSC at K=6 and B=1?

Response. The requested experiment is complete: SFC+OSC at K=6,B=1K=6, B=1 reaches 45.85% EM at update 200. This is a 1.42-point improvement over SFC alone under the best-evaluation protocol. The complete four-way comparison is:

ConfigurationKBBest EM (%)Update-240 EM (%)
SFC6144.4343.63
SFC+OSC6145.8540.26
SFC+EXB3244.7844.78
SFC+EXB+OSC3246.5844.87

Thus, the observed peak-performance gain from OSC is 1.42 points without EXB and 1.80 points with EXB; full SIPO exceeds SFC+OSC by 0.73 points. OSC's peak benefit is present in both branching configurations. Both use the same leaf budget:

N=M+LKB=10+2×6×1=10+2×3×2=22.N=M+LKB=10+2\times6\times1=10+2\times3\times2=22.

EXB changes the allocation between the number of expansion locations and fresh continuations per location. The backbone, training data, batch size, update count, reward, and evaluation settings remain fixed in this component comparison.

Response to Reviewer 1 (3/3)

R1.7 Training costs and the rollout budget

The measured training costs for the Qwen2.5-7B backbone in Table 4 only show costs for the Base method and SIPO.

What are the training costs for other baselines in your experimental protocol?

Why does SIPO have a higher token and time training budget compared to Base in Table 4?

Response. We have extended the cost table to all six completed local methods, using all 240 training-step records from each run and the same accounting as Table 4.

MethodTraining-sequence tokens (M)Training time (h)Allocated GPU-hours
AT²PO Base568.1922.36178.89
GRPO683.3126.35210.77
GSPO774.5332.07256.58
AEPO765.1429.36234.90
DAPO704.9725.88207.06
SIPO680.3523.46187.65

Base and SIPO share the prescribed budget of 22 leaves per prompt, 64 prompts per update, and 240 updates. Branch allocation and subsequent tool interactions affect sequence lengths, so equal leaf counts do not imply equal realized token or time costs. SIPO's observed overhead relative to Base is 19.74% in training-sequence tokens and 4.90% in training time. Its observed training time is lower than that of the other four completed local baselines.

R1.8 Single-hop training data

What is the training set for the single-hop QA task family?

Response. Single-hop training uses the Natural Questions training split: 79,168 processed examples. TriviaQA and PopQA are evaluation sources for this task family. The evaluation contains 3,610 NQ, 11,313 TriviaQA, and 14,267 PopQA questions, totaling 29,190. We will add the training source and these counts to the experimental setup.

Thank you for the detailed response and the additional experiments. The clarifications on the Base configuration, the Table 1 results, the single-hop training data, and the extended cost comparison answer several of my questions, and I appreciate that the SFC+OSC ablation was completed.

However, my main concerns remain, and I therefore maintain my score:

  • Local baselines are only reported for one of the six backbone–task settings (Qwen2.5-7B multi-hop). For the other five, SIPO is compared only with published results obtained under a different setup. In the one setting where both are available, the local AT2PO reproduction is 2.85 points below its published result, more than SIPO’s margin over the published AT²PO in any setting (0.51–1.45 Points over the published AT2PO averages, Table 1). It therefore remains unclear whether SIPO improves over its baseline.

  • Checkpoints are selected on the test sets, and the training curves are volatile. For example, SFC+OSC reaches 45.85% at its best checkpoint (update 200) but 40.26% at update 240. Selecting the maximum over 12 test-set evaluations favours configurations whose curves fluctuate more. Consistent with this, OSC’s gains at the best checkpoint (+1.42 without EXB, +1.80 with EXB) do not hold at update 240 (−3.37 and +0.09).

  • All results come from a single training run per configuration, so the significance of the reported improvements, most of which are below 2 points, cannot be assessed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.