acceptodds
Under review as a conference paper at ICLR 2027

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) derives its learning signal from binary rewards via on-policy exploration and becomes a powerful paradigm for improving the reasoning of large language models. Yet it faces a learning cliff on problems beyond model's current ability: when every rollout fails, the samples reveal nothing about a correct solution, and group-relative estimators (e.g., GRPO, RLOO) collapse the update to zero advantage. Expert trajectories offer a remedy, but they are incompatible with the current policy and often lie in the student's low-probability regions, so the surviving updates demand large parameter shifts and decay into surface-level imitation rather than transferable reasoning. We argue that a useful learning signal on hard problems must be both verifiably correct and sufficiently compatible with the current policy. We propose Policy-Compatible Reinforcement Learning (**PICO**), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding trajectories that are verified and policy-compatible by construction. These trajectories are optimized jointly with on-policy rollouts under a mixed-behavior objective that mitigates their distribution mismatch via importance sampling. Experiments across reasoning diverse datasets and agentic benchmarks demonstrate that PICO consistently outperforms standard RLVR and prior off-policy baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Off-policy Optimization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GHnXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses all-failure rollout groups in reinforcement learning with verifiable rewards, where group-relative advantages provide no learning signal. PICO generates an additional trajectory through teacher–student joint decoding: a teacher conditioned on expert guidance selects candidate tokens, while sampling preserves the student’s relative probabilities within the retained support. A retained-probability-mass constraint controls the local sampling divergence. Successfully verified trajectories replace failed rollouts and are incorporated through a mixed-behavior, clipped optimization objective with token-level importance ratios. Experiments cover Qwen3-4B/8B reasoning models and Qwen3-4B-Instruct on agentic tasks, with comparisons against expert-guided baselines and component ablations.

Critical strengths
  1. A concrete intervention for a real training bottleneck. PICO directly targets groups lacking successful rollouts. Separating teacher-based support selection from student-based relative probabilities gives a distinguishable sampling mechanism with an explicit local divergence constraint and fallback rule. This provides a clear, testable technical contribution beyond simply inserting expert demonstrations.

  2. Consistent improvements with meaningful component comparisons. PICO improves the six-benchmark macro average over the strongest overall baseline by 2.2 and 1.8 percentage points for the 4B and 8B models, respectively, and improves the agentic evaluation average by 4.9 points. Evaluation uses the student without teacher assistance. The direct-teacher, fixed-k, and importance-ratio ablations provide useful evidence that the particular construction contributes to the gains.

Critical weaknesses
  1. The incremental value over closely related guided-training approaches is not sufficiently established. Expert-guided recovery from unsuccessful exploration and correction for distribution mismatch have already been studied, notably in OC-GRPO. PICO’s contribution therefore rests on the additional value of its joint-decoding construction, but there is no direct comparison with this particularly relevant alternative. Resource comparability also remains unresolved: Table 7 reports approximately 54% higher time per update than GRPO for the 4B setting, without a matched-compute performance comparison. Furthermore, the provenance of the expert solutions used as guidance is insufficiently specified. These gaps make it difficult to determine whether the additional construction offers a robust advantage under comparable information and computational budgets.

  2. The evidence for the proposed compatibility mechanism is weaker than the evidence for improved performance. The retained-mass constraint establishes local sampling proximity, but does not establish that the resulting successful trajectories are more learnable or explain the downstream gains. The ablations are informative, yet simultaneously change rescue success, trajectory content and length, distribution mismatch, or update weighting. Similarly, if Figure 5 includes fallback steps, increasing average retained mass could partly reflect more fallback rather than more effective teacher guidance. The current experiments support a useful configuration, but provide limited characterization of when the compatibility constraint helps, particularly as student capability and problem difficulty change. This limits the strength of the proposed general learning principle.

Questions for authors
  1. What exactly constitutes the privileged guidance gg: a complete solution, a solution prefix, or the final answer? The official DAPO dataset provides questions and answers, whereas the manuscript describes expert demonstrations. Please identify the source or construction process for these demonstrations and clarify whether the relevant baselines use the same guidance. Are the joint-decoding length limits, truncation handling, and verification rules also held fixed in the direct-teacher and fixed-k ablations?

  2. Can existing training logs or accuracy-versus-time results establish whether PICO retains its advantage at comparable total training compute? A comparison with OC-GRPO or a similarly close guided-training alternative would be particularly informative. Please also clarify the total updates and checkpoint-selection rule underlying the reported results.

  3. Do Figure 2’s reward and solve-none statistics describe the original student rollouts or the reconstructed groups? Does Figure 5 include fallback steps? Available analyses connecting fallback frequency, rescue success, and retained mass to subsequent independent student performance—ideally across difficulty levels—would help determine whether compatibility explains the gains.

Reviewer_aJNPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

RLVR with group-relative estimators stalls when every rollout in a group fails, since the advantage collapses to zero and no learning signal survives. This paper proposes PICO, which on all-failed groups performs distribution-constrained joint decoding: the teacher (conditioned on an expert solution) and the student intersect their top-k token sets, the student's relative probabilities are preserved within the consensus set, and a dynamic top-k mechanism enforces a retained-mass constraint with an explicit trajectory-level KL bound. The verified joint trajectory replaces one failed rollout and is optimized under a mixed-behavior PPO objective with a closed-form importance ratio. PICO consistently outperforms GRPO and off-policy baselines across math, general-reasoning, and agentic benchmarks.

Critical strengths
  1. The method design is remarkably clean: the top-k intersection, the hard policy, and the dynamic retained-mass mechanism all admit closed-form solutions, with no extra heuristics.
  2. Proposition 1's KL decomposition reveals a clean separable structure between support restriction and soft reweighting, from which explicit KL bounds follow naturally at both the token and trajectory levels.
  3. Joint decoding is triggered only on all-failed rollout groups, leaving the regular on-policy regime untouched; rollout utilization and utilization-per-time remain essentially on par with GRPO.
  4. The importance ratio for the joint trajectory admits a closed-form ZtZ_t correction, yielding conditional action-level exact probability correction on the sampled tokens.
Critical weaknesses
  1. The trigger rule fires only on fully failed groups and does not adapt to weaker signal regimes such as a single success in the group, making the activation criterion somewhat coarse-grained.
  2. The trajectory-level KL bound scales linearly with Tmax⁡T_{\max} and becomes essentially vacuous in long-response settings.
  3. The experimental scale stops at 8B-Base, with no validation on larger models or instruct-tuned math specialists.
Questions for authors

Please see my comments on weaknesses.

Reviewer_u4FeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PICO, a method for learning from hard problems where all student rollouts fail. It uses a teacher with access to expert solutions to guide decoding. At each step, it intersects the teacher’s and student’s top-k token sets and samples according to the student’s relative probabilities. A dynamic top-k rule limits the per-step distribution shift. Verified joint trajectories replace failed rollouts and are used for mixed-behavior policy optimization with token-level importance ratios. Experiments on reasoning and agentic tasks show improvements over GRPO and several expert-guided baselines.

Critical strengths
  1. The paper is generally easy to follow, and the method is straightforward to implement. The algorithm and implementation details make it accessible and easy to build on.

  2. The experiments cover both reasoning and agentic tasks. The paper compares against several baselines and includes component ablations. This breadth provides useful evidence that the method can work across different settings.

Critical weaknesses
  1. The paper contains several presentation errors and inconsistent statements.
  • Figure 2 labels the proposed method as “PoCAL,” while the text calls it “PICO.”
  • Equation references repeatedly use phrases such as “Eq. equation 6.”
  • Table 5 also misplaces the temperature entry. The training time needs clarification as well.
  • Appendix C reports 1.5–2 days for math training, but the 43.6 minutes per step in Table 7 would imply about three days for the 100 steps shown in Figure 2.
  1. In Figure 2, SolveNone drops quickly for all methods during early training. PICO’s advantage mainly appears later, when its response length also becomes longer. Meanwhile, the training curves do not show that GRPO has converged. So I suspect that the gains may partly come from longer cot induced by teacher guidance. These longer cot may help the model reach the reasoning depth needed to solve hard problems. So, my question is,If GRPO were trained longer, could it also produce longer CoTs, solve these problems, and achieve a similar SolveNone rate to PICO?

  2. The core guarantee is that each decoding step retains at least Zmin⁡=0.8Z_{\min}=0.8 of the student’s probability mass. This bounds the per-step KL divergence. The mathematical result itself appears sound. However, KL divergence accumulates over the full trajectory. Proposition 2 gives the bound

\le T_{\max}(-\log Z_{\min}).$$ With a maximum joint-decoding length of 10,240 tokens, this bound is about 2,285 nats. Although finite, it does not establish that long trajectories remain close to the student’s distribution in practice. At each step, the student predicts from a prefix already shaped by teacher guidance. A token having high probability under this prefix does not mean the student can reach the prefix independently from the original question. Thus, the same concern the authors raise about prefix/hint methods still apply to PICO, though in a different form. 4. The paper claims to expand exploration on hard problems. However, in Table 4, 8B PICO and RLRT have identical Pass@16 scores on all three math benchmarks, while PICO has higher avg@16 scores. This is consistent with answering solvable problems correctly more often. It does not establish that PICO expands the range of problems the model can solve. 5. Since expert solutions are available for these training problems, could SFT followed by RL solve more hard problems? How does PICO compare with this baseline under the same training compute budget?
Questions for authors

Please see the weaknesses above. My main concern is whether the proposed mechanism enables the model to solve harder problems. The results in Table 4 do not yet establish this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.