acceptodds
Under review as a conference paper at ICLR 2027

MindPRO: Accelerating LLM Reinforcement Learning via Robust Partial-Rollout Sampling

Abstract

Reinforcement learning with verifiable rewards (RLVR) typically waits for all responses in a rollout batch before each policy update, allowing a small number of long responses to delay the entire synchronous training step. We introduce MindPRO, a prompt-preserving partial-rollout scheduler that mitigates this long-tail barrier without continuously overlapping rollout generation and policy optimization. Building on quota-based partial-rollout scheduling, MindPRO over-provisions prompt groups, advances once a target quota becomes trainable, and manages never-launched prompts and reusable interrupted trajectories through two checkpointable buffers. Prompt preservation here refers to maintaining prompt identity and eligible partial work across cutoff and recovery, while all selected, terminal, fresh, and partial outcomes remain explicitly accounted for. Rollout collection and optimization remain phase-serialized, while optional stability safeguards mitigate the localized policy mismatch and completion-order bias introduced by prefix reuse. Experiments on mathematical reasoning tasks show verified mean-step reductions of 39.8%–48.6% across three MindRLHF workloads, while final-checkpoint evaluations generally retain or improve downstream reasoning quality. We will publicly release our code to facilitate further development of the MindSpore ecosystem.

Primary area:
reinforcement learning
Keywords:
RLVR, partial-rollout

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TQHtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims to mitigate the rollout barrier caused by long-tail response generation in conventional synchronous RL. The authors propose MindPRO, which does not wait for all prompt groups to finish rollout, but instead selects a subset matching the target training batch size to proceed with training. The unfinished groups are left to later rounds to continue generation. To address the policy mismatch across steps, they introduce prefix importance sampling, length reweighting, and zero-variance group salvage. Evaluations on mathematical reasoning tasks demonstrate the efficiency of MindPRO.

Critical strengths
  1. The motivation is intuitive and the overall method is sound.
Critical weaknesses
  1. The research problem is not new. Many previous works have explored this issue, as already listed in the related work section (e.g., APRIL, CoPRIS, RollPacker). However, this study only compares against standard GRPO and a naive APRS variant, which are too weak as baselines to support the claimed novelty and superiority of the proposed approach.
  2. The experimental results in Table 1 are questionable. The reported accuracies on the Qwen2.5 series are much lower than official reports. More importantly, in several rows, the accuracy on GSM8K is significantly lower than on MATH, which suggests potential bugs in the evaluation pipeline or column misalignments.
  3. Although three stability safeguards are introduced, Table 3 shows no clear benefit from combining them together. Removing redundant components would make the method cleaner and more convincing.
  4. The evaluation is limited to mathematical reasoning tasks. Evaluating on more diverse domains (such as code generation or general agentic tasks) is recommended.
Questions for authors

Please refer to the weaknesses above

Reviewer_KFgaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes MindPRO, a partial-rollout scheduler for synchronous RLVR training. Following APRIL, MindPRO over-provisions αN\alpha N prompt groups, stops generation once N groups are trainable, and aborts the remaining requests. Unfinished prompts are kept in two checkpointed buffers (pending-fresh and partial-trajectory), and three optional safeguards (prefix importance sampling, length reweighting, salvage loss) handle the off-policy prefixes.

Critical strengths
  • The long-tail rollout problem is practical and important for RLVR training.
Critical weaknesses
  • The novelty is limited. Over-provisioning, quota cutoff, and resuming interrupted trajectories are already proposed in APRIL, Kimi k1.5, and CoPRIS. The main additions (dual buffers, checkpointing, “closed accounting”) are engineering features of a framework rather than a research contribution

  • Lack of baselines. MindPRO is only compared with synchronous GRPO while many related works are mentioned in the paper.

  • Step time is not a suitable metric since the method changes which samples are trained on. The authors should report time-to-target accuracy (wall-clock vs. accuracy curves). For example, with the 1.5B model on DAPO-Math-17K, MindPRO is 3.1x faster per step, but GSM8K drops from 42.99 to 33.97 (Table 1). The 7B MATH run has a similar drop (35.9 to 29.3).

  • Table 1 is confusing. It mixes validation reward and accuracy, the MATH score of MindPRO (1.5B, MATH) is missing, and some numbers are very low without explanation (3.34, 5.08 for MindRLHF GSM8K).

Questions for authors
  • How does the speedup change with α and the maximum response length?
  • What is the distribution of the resume depth k in practice?
  • Selecting the first N finished groups may favor short or easy prompts. Does it shift the training prompt distribution over time?
  • How does the re-prefill overhead scale with longer contexts (Figure 3 shows ~90% token utilization at 1.5B)?
Reviewer_swveOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MindPRO, a partial-rollout scheduling framework designed to accelerate reinforcement learning with verifiable rewards (RLVR) by mitigating the long-tail latency of rollout generation. Building upon existing over-provisioning and early-cutoff approaches, MindPRO launches additional prompt groups and proceeds with policy optimization once a sufficient number of complete groups have been collected, avoiding synchronization delays caused by straggling responses. To improve the practicality and robustness of this approach, the authors introduce a dual-buffer mechanism that preserves unlaunched prompts and interrupted trajectories for subsequent training iterations, along with checkpoint recovery and explicit sample accounting. The framework additionally incorporates optional stability mechanisms, including prefix importance sampling, response-length reweighting, and salvage loss. Experiments conducted on Qwen models ranging from 0.6B to 7B across mathematical reasoning benchmarks demonstrate substantial reductions in training-step time, with reported speedups of up to 3.10×, while generally maintaining comparable downstream performance.

Critical strengths
  1. Clearly written plots are drawn clearly.
  2. The speedup number is empirical and seems to be ample, given empirical strong results. The system implementation will be a great contribution to the community.
Critical weaknesses
  1. Technique-wise, very similar to APRIL/APRS that rely on overprovisioning. The contribution seems overwhelmingly engineering
  2. The evaluation primarily focuses on the Qwen2.5 model family, except the 0.6B Qwen3 model. This limits the generalizability of the empirical results, as these models typically produce relatively short reasoning trajectories compared to more recent long-reasoning models. Evaluating MindPRO on models that generate substantially longer rollouts would better reflect practical RL training scenarios and provide stronger evidence for the claimed efficiency improvements. Furthermore, the experiments are restricted to mathematical reasoning tasks, primarily GSM8K and MATH, which further limits the diversity and representativeness of the evaluation.
  3. Part of the techniques proposed is heavily dependent on epoching. In practice, when data is ample, it won't be useful.
  4. Lack of evaluation on tasks that aren't math. Other domain evaluation would be helpful to justify the generalization of the techniques proposed.
Questions for authors
  1. Comparison with APRIL: Given the substantial methodological overlap with APRIL (Zhou et al., 2025), could the authors clarify the key technical contributions beyond engineering improvements? In particular, how does MindPRO compare against APRIL under identical experimental settings in terms of training efficiency, model quality, and implementation overhead?
  2. Does this technique achieve greater speedup on reasoning models that generate longer responses, like Qwen3 family models (greater than 0.6B)?
  3. Does the proposed method generalize to domains except Math?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.