acceptodds
Under review as a conference paper at ICLR 2027

Agentic Policy Improvement through Value-guided Informative Search

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. However, its learning signal is limited by the trajectories discovered by the current policy. When group rollouts repeatedly follow similar or low-value paths, the resulting verified signal provides little guidance for learning better behavior. Existing search-based rollout methods can increase trajectory diversity, but diversity alone neither guarantees a better rollout policy nor ensures that search improvements are transferred back to the model. We therefore study how to make search during RLVR a principled policy-improvement operator. We propose APIVIS, a rollout-and-learning framework that applies value-guided search to domain-native language-model decisions and combines searched and direct trajectories within each rollout group. APIVIS further distills search-improved decisions, preserving an improvement-aligned learning signal even when group-relative returns are uninformative. We prove statewise and trajectory-level policy improvement under exact values, together with an explicit degradation bound under bounded value errors. Across established benchmarks in mathematical reasoning and multi-turn interaction, APIVIS outperforms established RLVR and search-based baselines, demonstrating the effectiveness of search as a learnable policy-improvement operator.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards, Value-Guided Search, Agentic Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BBMZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes APIVIS, a training framework that combines value-guided search with group-relative reinforcement learning. It mixes directly sampled and search-generated trajectories, then uses InfoSFT to selectively supervise decisions identified as improvements by a frozen value model. Its theoretical contribution establishes policy improvement for the search-induced rollout policy under exact values, with a degradation bound under bounded estimation errors. Experiments cover mathematical reasoning, ALFWorld, and WebShop, evaluating trained policies without inference-time search. Results show improvements over the evaluated RL and search-based baselines, with ablations examining the search operator, InfoSFT, and search budgets.

Critical strengths

The paper’s empirical evidence supports its central idea that training-time search can improve the policy itself. Comparisons with MCTS-α under matched search settings demonstrate the benefit of the proposed search operator, while the InfoSFT ablation shows additional gains from explicitly supervising selected decisions. Improvements across mathematical reasoning and interactive tasks are retained without inference-time search, strengthening the case for the method’s practical relevance and applicability across domains.

Critical weaknesses
  1. The ablation experiments show that adding InfoSFT improves performance, but it remains unclear whether the benefit comes from additional supervised learning or the value-based selection mechanism. Please compare RL + InfoSFT with RL + standard SFT using the same search-generated data and the same RL objective, where standard SFT supervises all searched decisions from successful trajectories. A random-selection baseline with the same number of supervised tokens would further test whether value-based selection identifies better training targets. A baseline trained only with SFT on the search-generated data would also help assess the contribution of the RL objective.

  2. The value model guides both search and supervision selection, but its reliability has not been fully validated. Please explain how the value model was trained and evaluate whether its scores predict actual continuation success at different training stages. This would clarify whether the frozen value model remains reliable as the policy evolves.

  3. Table 2 increases the number of candidates CC and simulations SS simultaneously, leaving their individual effects unclear. Please vary CC and SS separately while keeping KK fixed to distinguish the benefits of exploring more candidates from evaluating them more thoroughly.

Questions for authors

Please see weaknesses.

3 comments

We sincerely thank the reviewer for the careful reading, constructive feedback, and patience. We have conducted substantial additional experiments and analyses to address the points raised. Below, W denotes a weakness and Q denotes a question. We hope these results and clarifications resolve the reviewer's concerns.

W1 — Standard-SFT, token-matched, and SFT-only controls

Weakness 1. The ablation experiments show that adding InfoSFT improves performance, but it remains unclear whether the benefit comes from additional supervised learning or the value-based selection mechanism. Please compare RL + InfoSFT with RL + standard SFT using the same search-generated data and the same RL objective, where standard SFT supervises all searched decisions from successful trajectories. A random-selection baseline with the same number of supervised tokens would further test whether value-based selection identifies better training targets. A baseline trained only with SFT on the search-generated data would also help assess the contribution of the RL objective.

For the matched RL arms, we hold search, mixed rollouts, GRPO, training budget, and validation-only selection fixed, and vary only the SFT data and value gate. SFT-only separately tests removing RL.

ArmSFT dataValue gateRLMath Avg@16
apivis_referenceAllYesYes40.62±0.96
all_ungatedAllNoYes39.78±0.62
success_ungatedsuccess-onlyNoYes38.54±2.15
no_sftNone—Yes37.68±2.64
sft_onlysuccess-onlyNoNo41.39±1.14

Among matched RL arms, value-gated InfoSFT outperforms all-span SFT, success-only SFT, and no SFT. The strong SFT-only result shows that search-generated supervision is itself effective, while the RL controls isolate the additional benefit of selective targets. It also motivates future work on simpler objectives that directly distill search-generated successes.

W2 — Frozen value-model training and reliability

Weakness 2. The value model guides both search and supervision selection, but its reliability has not been fully validated. Please explain how the value model was trained and evaluate whether its scores predict actual continuation success at different training stages. This would clarify whether the frozen value model remains reliable as the policy evolves.

We keep the value models frozen throughout policy training: DeepSeek-VM-1.5B for Math and AgentRM-3B for interactive tasks. We will specify their training data, objectives, and exact checkpoints in the revision. To test whether value guidance remains effective as the policy evolves, we compare terminal rewards from value-guided search with direct rollouts from the same policy during early, middle, and late training. Rewards are normalized to [0,1].

Domain / training methodSeedsDirectSearchOverall liftEarlyMiddleLate
Math / APIVIS30.2970.354+0.057+0.077+0.050+0.047
WebShop / APIVIS+GRPO30.6340.662+0.028+0.043+0.027+0.017
WebShop / APIVIS+GiGPO30.6410.672+0.031+0.028+0.037+0.028

Despite step-level noise, every run and training phase shows positive mean lift. Thus, frozen-value-guided search remains effective as the policy evolves on both Math and WebShop; Figure 2 of the revised paper shows the full terminal-reward-lift curves across training. The lift nevertheless tends to attenuate later in training, motivating future work on jointly adapting the value model as the policy distribution evolves.

W3 — Separating candidate count and simulation budget

Weakness 3. Table 2 increases the number of candidates (C) and simulations (S) simultaneously, leaving their individual effects unclear. Please vary (C) and (S) separately while keeping (K) fixed to distinguish the benefits of exploring more candidates from evaluating them more thoroughly.

Following Gumbel MuZero (Danihelka et al., 2022), C and S are coupled by the sequential-halving schedule: C defines the initial candidate pool, while S allocates repeated simulations as candidates are pruned. In our implementation, setting S=C gives each candidate only one evaluation, removing successive refinement and approaching best-of-K rather than preserving the same search algorithm. We will therefore describe Table 2 as a total-search-budget sweep and avoid claiming an independent causal decomposition of C and S.

Danihelka, Ivo, et al. "Policy improvement by planning with Gumbel." International Conference on Learning Representations. 2022.

Thank you for the substantial effort in providing additional experiments and detailed explanations. My concerns have been adequately addressed, and I maintain my rating of 3.

Thank you for the positive feedbacks, we will continue to refine this work.

Reviewer_g7bkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes APIVIS RL algorithm for mathematic reasoning and agentic optimization, which combines value-guided rollout search, mixed direct and search rollout groups, and selective distillation through InfoSFT. The experiments report gains on mathematical reasoning and agent tasks without requiring search at inference time.

Critical strengths
  1. The paper proposes a coherent framework that connects search-based rollout generation with learning from the selected decisions.
  2. The trained models achieve a consistent math improvements at two model sizes, with direct decoding at evaluation time.
Critical weaknesses
  1. The paper title focuses on agentic RL while the evidence is less convincing: the ALFWorld gain over GiGPO is small, and WebShop reuses the validation goals for final evaluation.
  2. Root trajectories and optimizer steps are matched, but total training compute is not, leaving the gains over ordinary GRPO at comparable cost unclear.
  3. Table 2 shows that adding InfoSFT helps, but does not isolate its selective gate from ordinary SFT on successful searched decisions or compare against MCTS-α with the same added supervision.
  4. The framework closely follows existing search-and-learn approaches, so its contribution is mainly in the implementation recipe of specific search, rollout mixing, and distillation choices.
Questions for authors
  1. What are the total generated tokens, value-model calls, environment calls, and GPU-hours, and how do the methods compare at matched compute?
  2. With search fixed, how does InfoSFT compare with ordinary SFT on successful searched decisions, and does MCTS-α also benefit from the same supervision?
  3. How reliably do the frozen value models rank candidate continuations as the policy changes during training?
2 comments

We sincerely thank the reviewer for the careful reading, constructive feedback, and patience. We have conducted substantial additional experiments and analyses to address the points raised. Below, W denotes a weakness and Q denotes a question. We hope these results and clarifications resolve the reviewer's concerns.

W1 — Agentic evidence and the WebShop protocol

Weakness 1. The paper title focuses on agentic RL while the evidence is less convincing: the ALFWorld gain over GiGPO is small, and WebShop reuses the validation goals for final evaluation.

ALFWorld is close to saturation: GiGPO reaches 91.29±1.72 overall success, with five of six task types at or above 90%.

ALFWorld task typeGiGPO success rate
Pick80.56±10.49
Look100.00±0.00
Clean90.32±3.23
Heat94.20±2.51
Cool92.06±7.27
Pick294.12±5.88

This ceiling limits the observable aggregate margin; APIVIS+GiGPO remains competitive at 91.79±1.49 under validation-rank-1 selection. To address the WebShop leakage concern directly, we reran all methods with disjoint train/validation/test splits and exactly one validation-selected checkpoint per seed.

MethodPaper task scoreNew task scoreΔ meanPaper success rateNew success rateΔ mean
GRPO80.59±4.5379.45±2.75-1.1468.40±7.7569.38±6.43+0.98
MCTS-α+GRPO85.17±0.4784.75±2.36-0.4274.10±4.3874.60±3.34+0.50
APIVIS+GRPO85.43±0.2886.48±2.29+1.0575.20±2.5577.53±3.16+2.33
GiGPO83.37±1.3483.69±0.74+0.3273.00±4.7071.48±3.55-1.52
APIVIS+GiGPO84.69±2.0886.70±0.82+2.0176.27±2.6677.67±1.47+1.40

Under this corrected protocol, APIVIS+GRPO outperforms GRPO and MCTS-α+GRPO, while APIVIS+GiGPO outperforms GiGPO, on both task score and success rate. The agentic gains therefore persist without validation/test overlap.

W2 / Q1 — Compute accounting and compute-matched GRPO

Weakness 2. Root trajectories and optimizer steps are matched, but total training compute is not, leaving the gains over ordinary GRPO at comparable cost unclear.

Question 1. What are the total generated tokens, value-model calls, environment calls, and GPU-hours, and how do the methods compare at matched compute?

We first report complete accounting for strict-protocol WebShop over three seeds. Value-model calls count scored candidates; environment calls include executed and search-only transitions; GPU-hours include scheduled validation.

MethodGenerated tokens (M)Value-model callsEnvironment callsGPU-hours
GRPO2.55±0.660101,590±5,09984.08±7.44
GiGPO3.66±1.48082,147±5,53996.80±5.44
MCTS-α+GRPO18.00±2.09158,411±10,068345,120±25,715187.68±43.60
APIVIS+GRPO17.75±1.39153,768±6,323332,465±17,477167.28±14.40
APIVIS+GiGPO17.33±0.92149,644±3,768322,117±12,113157.52±12.72

At comparable search-side compute, APIVIS outperforms MCTS-α under the strict protocol.

We apply the same accounting to the completed three-seed Math runs; environment calls do not apply to this non-interactive domain.

MethodRunsGenerated tokens (M)Value-model callsGPU-hours
GRPO351.67±2.470120.56±4.32
LATR356.74±4.590614.40±24.32
TS-LLM3325.74±8.4341,574±632447.60±17.52
APIVIS3301.51±11.2139,827±375204.80±5.92

Generated tokens count returned training generations, and GPU-hours sum logged step time across eight A100s, including scheduled validation. APIVIS and TS-LLM use comparable generation and value-model budgets, while APIVIS requires substantially fewer GPU-hours.

Finally, we isolate generation budget by extending GRPO to 600 steps, matching 98.70% of APIVIS's generated tokens.

Training settingRunsGenerated tokens (M)Relative to GRPO meanAPIVIS-budget coverage
GRPO, 100 steps351.67±2.471.00×17.10%
APIVIS, 100 steps3301.51±11.215.84×100%
GRPO-longer, 600 steps3297.58±5.495.75×98.70%
Training settingCheckpoint selectionMath Avg@16
GRPO-longer, 600 stepsStep-30036.30±4.52
GRPO-longer, 600 stepsStep-60035.14±3.83
GRPO-longer, 600 stepsValidation top-136.57±2.78
APIVIS, 100 stepsValidation top-140.62±0.96

GRPO-longer remains below APIVIS at step 300, step 600, and after validation-only checkpoint selection. Together, these results show that APIVIS's gains are not explained by additional rollout compute alone.

W3 / Q2 — Selective InfoSFT and MCTS-α with matched supervision

Weakness 3. Table 2 shows that adding InfoSFT helps, but does not isolate its selective gate from ordinary SFT on successful searched decisions or compare against MCTS-α with the same added supervision.

Question 2. With search fixed, how does InfoSFT compare with ordinary SFT on successful searched decisions, and does MCTS-α also benefit from the same supervision?

To investigate the efficacy of InfoSFT, we fix the search operator and budget, mixed rollouts, GRPO objective, training budget, and validation-top-1 checkpoint selection. Then we test different learning objectives based on the same setting.

Training objectiveMath Avg@16Δ vs APIVIS+InfoSFT
APIVIS + InfoSFT40.62±0.96—
APIVIS + success-only SFT38.54±2.15-2.08
APIVIS without SFT37.68±2.64-2.94
MCTS-α + InfoSFT37.49±1.38-3.13

APIVIS+InfoSFT outperforms success-only SFT and no SFT across three seeds; the completed MCTS-α+InfoSFT seed0 control also remains below APIVIS.

W4 — Contribution beyond existing search-and-learn methods

Weakness 4. The framework closely follows existing search-and-learn approaches, so its contribution is mainly in the implementation recipe of specific search, rollout mixing, and distillation choices.

Different from previous search-based works, APIVIS contributes a policy-improvement framework for LLM reasoning. It formalizes the rollout policy jointly induced by policy-sampled, variable-length candidates and value-guided search, proving improvement with exact values and a Vπ(s)−2ϵV^{\pi}(s)-2\epsilon guarantee under bounded value error. Matched-budget results across mathematical and agentic tasks validate the predicted search improvement, while gains under search-free evaluation show that this improvement transfers into the trained policy.

Q3 — Frozen value-model reliability during policy training

Question 3. How reliably do the frozen value models rank candidate continuations as the policy changes during training?

To test reliability as the policy evolves, we compare the terminal rewards of value-guided search and direct trajectories from the same policy during early, middle, and late training. Rewards are normalized to [0,1].

Domain / training methodSeedsDirectSearchOverall liftEarlyMiddleLate
Math / APIVIS30.2970.354+0.057+0.077+0.050+0.047
WebShop / APIVIS+GRPO30.6340.662+0.028+0.043+0.027+0.017
WebShop / APIVIS+GiGPO30.6410.672+0.031+0.028+0.037+0.028

Despite step-level noise, every run and training phase has positive mean lift. Frozen-value-guided search therefore remains effective throughout training on both Math and WebShop; Figure 2 of the revised paper shows the full terminal-reward-lift curves across training. The lift nevertheless tends to attenuate later in training, motivating future work on jointly adapting the value model as the policy distribution evolves.

Reviewer_6VVQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes APIVIS, which combines direct and value-guided search rollouts during RLVR training. An additional InfoSFT objective trains the policy on selected search decisions. The authors establish policy-improvement guarantees under idealized assumptions and bounds under value-estimation errors. Experiments on mathematical reasoning, ALFWorld, and WebShop report improvements when evaluating the trained policy without inference-time search.

Critical strengths
  1. The gains carry over to direct inference. Final evaluation uses no search. For the 4B model, average mathematical accuracy increases from 36.82% with GRPO to 40.02%, which is a useful improvement.
  2. The search component has some supporting ablations. At matched C/S/K settings, Table 2 shows an advantage over MCTS-α. This suggests that the choice of search operator matters beyond simply increasing the search budget.
Critical weaknesses
  1. Final evaluation is used for checkpoint selection. Section 5.1 reports the best full-evaluation result among validation-shortlisted checkpoints. Appendix B.1.3 also states that WebShop validation and final evaluation use the same 500 goals. This introduces selection bias and weakens the evidence for generalization. Checkpoints should be selected using separate validation data before final evaluation.
  2. The benefit of the InfoSFT gate is not sufficiently isolated. Its gain in Table 2 is 0.70 percentage points, with only one mathematical training seed. There is also no comparison against supervising all successful searched decisions without the gain/margin gate. It remains unclear whether the proposed filtering improves over ordinary success-based supervision.
Questions for authors

See weaknesses above, I have another two questions:

  1. Evaluation clarification: How many shortlisted checkpoints were evaluated for each method? Do the main gains remain when selecting a single checkpoint using an independent validation set?
  2. Additional evidence: With the same searched trajectories, does InfoSFT consistently outperform supervision on all successful searched decisions without gain/margin filtering?
2 comments

We sincerely thank the reviewer for the careful reading, constructive feedback, and patience. We have conducted substantial additional experiments and analyses to address the points raised. Below, W denotes a weakness and Q denotes a question. We hope these results and clarifications resolve the reviewer's concerns.

W1 / Q1 — Independent validation and checkpoint selection

Weakness 1. Final evaluation is used for checkpoint selection. Section 5.1 reports the best full-evaluation result among validation-shortlisted checkpoints. Appendix B.1.3 also states that WebShop validation and final evaluation use the same 500 goals. This introduces selection bias and weakens the evidence for generalization. Checkpoints should be selected using separate validation data before final evaluation.

Question 1. How many shortlisted checkpoints were evaluated for each method? Do the main gains remain when selecting a single checkpoint using an independent validation set?

We updated the protocol in two ways: each run exposes exactly one validation-rank-1 checkpoint to the test set, and WebShop uses disjoint train, validation, and test splits.

OriginalRevised
Test-evaluated checkpoints/runTop two validation checkpointsExactly one
SelectionBetter full-evaluation resultValidation rank 1; earliest tie
WebShop splitSame 500 validation/test goalsTrain [628,end), validation [500,628), test [0,500)

Math Avg@16 after validation-top-1 reselection

ModelMethodReportedValidation top-1Change
Qwen3-4BGRPO36.8236.820.00
Qwen3-4BLATR36.5434.41-2.13
Qwen3-4BTS-LLM36.1536.150.00
Qwen3-4BAPIVIS+InfoSFT40.0240.020.00
Qwen3-1.7BGRPO25.4225.26-0.16
Qwen3-1.7BLATR24.8622.59-2.27
Qwen3-1.7BTS-LLM26.1525.10-1.05
Qwen3-1.7BAPIVIS+InfoSFT27.7927.790.00

ALFWorld success rate after validation-top-1 reselection

MethodReportedValidation top-1Change in mean
GRPO74.88±9.4574.88±9.450.00
GiGPO91.29±1.7291.29±1.720.00
MCTS-α+GRPO84.08±0.4383.58±1.29-0.50
APIVIS+GRPO+InfoSFT88.81±1.9787.56±0.43-1.25
APIVIS+GiGPO+InfoSFT91.79±1.4991.79±1.490.00

WebShop with independent validation and validation-top-1 checkpoint per seed

MethodPaper task scoreNew task scoreΔ meanPaper success rateNew success rateΔ mean
GRPO80.59±4.5379.45±2.75-1.1468.40±7.7569.38±6.43+0.98
MCTS-α+GRPO85.17±0.4784.75±2.36-0.4274.10±4.3874.60±3.34+0.50
APIVIS+GRPO85.43±0.2886.48±2.29+1.0575.20±2.5577.53±3.16+2.33
GiGPO83.37±1.3483.69±0.74+0.3273.00±4.7071.48±3.55-1.52
APIVIS+GiGPO84.69±2.0886.70±0.82+2.0176.27±2.6677.67±1.47+1.40

Across Math, ALFWorld, and WebShop, the stricter protocol leaves the main conclusions unchanged: APIVIS retains its mathematical and agentic advantages. We will revise the protocol description and release the split definitions, selected-checkpoint manifests, and per-seed results.

W2 / Q2 — Value-gated InfoSFT versus ordinary success-only SFT

Weakness 2. The benefit of the InfoSFT gate is not sufficiently isolated. Its gain in Table 2 is 0.70 percentage points, with only one mathematical training seed. There is also no comparison against supervising all successful searched decisions without the gain/margin gate. It remains unclear whether the proposed filtering improves over ordinary success-based supervision.

Question 2. With the same searched trajectories, does InfoSFT consistently outperform supervision on all successful searched decisions without gain/margin filtering?

To isolate value gating, we hold search, mixed rollouts, GRPO, training budget, and validation-only checkpoint selection fixed, and vary only the SFT target-selection rule. All Math results use three seeds.

Training objectiveMath Avg@16Δ vs reference
InfoSFT40.62±0.96—
SFT success-only38.54±2.15-2.08
No SFT37.68±2.64-2.94

Ordinary success-only SFT trails InfoSFT on Math. We repeat the controlled comparison on WebShop using disjoint splits and one validation-selected checkpoint per seed.

WebShop training objectiveTask scoreSuccess rate
InfoSFT86.48±2.2977.53±3.16
SFT success-only84.95±2.1176.47±0.81
Δ+1.53+1.06

Across both domains, value-gated InfoSFT consistently outperforms ordinary success-only SFT under the same search and RL setup.

Qwen3-4B-Base Table 1 robustness. To address the single-seed concern more broadly, we replicate the full Qwen3-4B-Base comparison with validation-only checkpoint selection and three seeds.

MethodSeedsMATH-500AIME24AIME25OlympiadBenchAvg@16
APIVIS + InfoSFT381.30±0.6018.26±1.6815.07±1.0747.83±0.9840.62±0.96
GRPO378.97±1.0912.78±2.0510.34±1.8944.33±1.3636.61±1.55
LATR378.57±1.0311.60±0.7311.18±3.2443.31±0.1736.16±0.86
TS-LLM377.57±0.7613.20±1.349.79±1.0842.84±0.7035.85±0.70

APIVIS remains best on every benchmark and in aggregate across three seeds. Given the rebuttal-scale compute budget, we will report the corresponding three-seed Qwen3-1.7B-Base results in the camera-ready version.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your time and insightful reviews. As the discussion period is drawing to a close, we wanted to politely follow up on our rebuttal.

We have conducted extensive additional experiments and provided detailed responses to answer your concerns. We would greatly appreciate it if you could share any brief feedback on whether our revisions have resolved your concerns. Thank you for your consideration.

Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.