- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes APIVIS, a training framework that combines value-guided search with group-relative reinforcement learning. It mixes directly sampled and search-generated trajectories, then uses InfoSFT to selectively supervise decisions identified as improvements by a frozen value model. Its theoretical contribution establishes policy improvement for the search-induced rollout policy under exact values, with a degradation bound under bounded estimation errors. Experiments cover mathematical reasoning, ALFWorld, and WebShop, evaluating trained policies without inference-time search. Results show improvements over the evaluated RL and search-based baselines, with ablations examining the search operator, InfoSFT, and search budgets.
Critical strengths
The paper’s empirical evidence supports its central idea that training-time search can improve the policy itself. Comparisons with MCTS-α under matched search settings demonstrate the benefit of the proposed search operator, while the InfoSFT ablation shows additional gains from explicitly supervising selected decisions. Improvements across mathematical reasoning and interactive tasks are retained without inference-time search, strengthening the case for the method’s practical relevance and applicability across domains.
Critical weaknesses
-
The ablation experiments show that adding InfoSFT improves performance, but it remains unclear whether the benefit comes from additional supervised learning or the value-based selection mechanism. Please compare RL + InfoSFT with RL + standard SFT using the same search-generated data and the same RL objective, where standard SFT supervises all searched decisions from successful trajectories. A random-selection baseline with the same number of supervised tokens would further test whether value-based selection identifies better training targets. A baseline trained only with SFT on the search-generated data would also help assess the contribution of the RL objective.
-
The value model guides both search and supervision selection, but its reliability has not been fully validated. Please explain how the value model was trained and evaluate whether its scores predict actual continuation success at different training stages. This would clarify whether the frozen value model remains reliable as the policy evolves.
-
Table 2 increases the number of candidates and simulations simultaneously, leaving their individual effects unclear. Please vary and separately while keeping fixed to distinguish the benefits of exploring more candidates from evaluating them more thoroughly.
Questions for authors
Please see weaknesses.
3 comments
Thank you for the substantial effort in providing additional experiments and detailed explanations. My concerns have been adequately addressed, and I maintain my rating of 3.
Thank you for the positive feedbacks, we will continue to refine this work.
We sincerely thank the reviewer for the careful reading, constructive feedback, and patience. We have conducted substantial additional experiments and analyses to address the points raised. Below, W denotes a weakness and Q denotes a question. We hope these results and clarifications resolve the reviewer's concerns.
W1 — Standard-SFT, token-matched, and SFT-only controls
For the matched RL arms, we hold search, mixed rollouts, GRPO, training budget, and validation-only selection fixed, and vary only the SFT data and value gate. SFT-only separately tests removing RL.
apivis_referenceall_ungatedsuccess_ungatedno_sftsft_onlyAmong matched RL arms, value-gated InfoSFT outperforms all-span SFT, success-only SFT, and no SFT. The strong SFT-only result shows that search-generated supervision is itself effective, while the RL controls isolate the additional benefit of selective targets. It also motivates future work on simpler objectives that directly distill search-generated successes.
W2 — Frozen value-model training and reliability
We keep the value models frozen throughout policy training: DeepSeek-VM-1.5B for Math and AgentRM-3B for interactive tasks. We will specify their training data, objectives, and exact checkpoints in the revision. To test whether value guidance remains effective as the policy evolves, we compare terminal rewards from value-guided search with direct rollouts from the same policy during early, middle, and late training. Rewards are normalized to [0,1].
Despite step-level noise, every run and training phase shows positive mean lift. Thus, frozen-value-guided search remains effective as the policy evolves on both Math and WebShop; Figure 2 of the revised paper shows the full terminal-reward-lift curves across training. The lift nevertheless tends to attenuate later in training, motivating future work on jointly adapting the value model as the policy distribution evolves.
W3 — Separating candidate count and simulation budget
Following Gumbel MuZero (Danihelka et al., 2022), C and S are coupled by the sequential-halving schedule: C defines the initial candidate pool, while S allocates repeated simulations as candidates are pruned. In our implementation, setting S=C gives each candidate only one evaluation, removing successive refinement and approaching best-of-K rather than preserving the same search algorithm. We will therefore describe Table 2 as a total-search-budget sweep and avoid claiming an independent causal decomposition of C and S.
Danihelka, Ivo, et al. "Policy improvement by planning with Gumbel." International Conference on Learning Representations. 2022.