- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses all-failure rollout groups in reinforcement learning with verifiable rewards, where group-relative advantages provide no learning signal. PICO generates an additional trajectory through teacher–student joint decoding: a teacher conditioned on expert guidance selects candidate tokens, while sampling preserves the student’s relative probabilities within the retained support. A retained-probability-mass constraint controls the local sampling divergence. Successfully verified trajectories replace failed rollouts and are incorporated through a mixed-behavior, clipped optimization objective with token-level importance ratios. Experiments cover Qwen3-4B/8B reasoning models and Qwen3-4B-Instruct on agentic tasks, with comparisons against expert-guided baselines and component ablations.
Critical strengths
-
A concrete intervention for a real training bottleneck. PICO directly targets groups lacking successful rollouts. Separating teacher-based support selection from student-based relative probabilities gives a distinguishable sampling mechanism with an explicit local divergence constraint and fallback rule. This provides a clear, testable technical contribution beyond simply inserting expert demonstrations.
-
Consistent improvements with meaningful component comparisons. PICO improves the six-benchmark macro average over the strongest overall baseline by 2.2 and 1.8 percentage points for the 4B and 8B models, respectively, and improves the agentic evaluation average by 4.9 points. Evaluation uses the student without teacher assistance. The direct-teacher, fixed-k, and importance-ratio ablations provide useful evidence that the particular construction contributes to the gains.
Critical weaknesses
-
The incremental value over closely related guided-training approaches is not sufficiently established. Expert-guided recovery from unsuccessful exploration and correction for distribution mismatch have already been studied, notably in OC-GRPO. PICO’s contribution therefore rests on the additional value of its joint-decoding construction, but there is no direct comparison with this particularly relevant alternative. Resource comparability also remains unresolved: Table 7 reports approximately 54% higher time per update than GRPO for the 4B setting, without a matched-compute performance comparison. Furthermore, the provenance of the expert solutions used as guidance is insufficiently specified. These gaps make it difficult to determine whether the additional construction offers a robust advantage under comparable information and computational budgets.
-
The evidence for the proposed compatibility mechanism is weaker than the evidence for improved performance. The retained-mass constraint establishes local sampling proximity, but does not establish that the resulting successful trajectories are more learnable or explain the downstream gains. The ablations are informative, yet simultaneously change rescue success, trajectory content and length, distribution mismatch, or update weighting. Similarly, if Figure 5 includes fallback steps, increasing average retained mass could partly reflect more fallback rather than more effective teacher guidance. The current experiments support a useful configuration, but provide limited characterization of when the compatibility constraint helps, particularly as student capability and problem difficulty change. This limits the strength of the proposed general learning principle.
Questions for authors
-
What exactly constitutes the privileged guidance : a complete solution, a solution prefix, or the final answer? The official DAPO dataset provides questions and answers, whereas the manuscript describes expert demonstrations. Please identify the source or construction process for these demonstrations and clarify whether the relevant baselines use the same guidance. Are the joint-decoding length limits, truncation handling, and verification rules also held fixed in the direct-teacher and fixed-k ablations?
-
Can existing training logs or accuracy-versus-time results establish whether PICO retains its advantage at comparable total training compute? A comparison with OC-GRPO or a similarly close guided-training alternative would be particularly informative. Please also clarify the total updates and checkpoint-selection rule underlying the reported results.
-
Do Figure 2’s reward and solve-none statistics describe the original student rollouts or the reconstructed groups? Does Figure 5 include fallback steps? Available analyses connecting fallback frequency, rescue success, and retained mass to subsequent independent student performance—ideally across difficulty levels—would help determine whether compatibility explains the gains.