- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how the order of supervised and verifier-guided training affects learning on a finite corpus. It uses a simplified attention regression model: supervised training updates both the key and value using noisy scalar labels, while GRPO updates only the key using rewards for correct position selection. The authors derive dynamical mean-field equations describing repeated data use and objective switching, and validate selected training trajectories through synthetic experiments. With matched update counts and verifier budgets, the experiments show that the relative advantage of staged versus randomly interleaved training changes with label noise and sequence length.
Critical strengths
-
The training-order comparisons are carefully controlled. The schedules use the same update counts and verifier budgets, and repeated paired experiments strengthen the credibility of the comparisons.
-
The theoretical analysis accounts for both repeated use of training data and the effect of model changes on subsequent action sampling. Together with the numerical validation, this provides some technical value.
Critical weaknesses
-
The two training procedures differ not only in their update rules but also in their learning tasks. Supervised training learns the final scalar prediction and updates both the key and value, whereas GRPO learns position selection using exact position rewards and freezes the value. It is therefore difficult to determine how much of the observed training-order effect comes from alternating supervised learning and policy optimization, and how much comes from differences in tasks, feedback quality, and the parameters being updated.
-
The overall setup differs substantially from models and training tasks with practical applications. The model contains only two trainable vectors, the key and value, and the task is to locate a token in a synthetic Gaussian sequence and predict a scalar. The paper does not sufficiently explain whether these simplifications preserve the key factors that determine which training order works better, or how its findings can guide training choices for practical models. Its practical guidance therefore remains unclear.
Questions for authors
-
How can the training-order effects arising from policy sampling be distinguished from those arising from differences in the tasks, feedback, and parameters updated by the two procedures? As a possible diagnostic comparison, would the main conclusions still hold if GRPO were replaced with supervised training on the correct position?
-
Which key factors governing the relative effectiveness of training orders in practical models are preserved by the simplifications in the model, data, and task? Based on the current analysis, what specific training recommendations can the authors offer, and under what conditions do they apply?
Response to Reviewer f6h8
We thank the reviewer for recognizing the technical value of our treatment of repeated data use and the influence of model updates on subsequent action sampling.
1. Tasks, feedback and updated parameters (Weakness 1 and Question 1).
Thank you for suggesting position supervision as a diagnostic. MSE updates the key and value from noisy scalar labels, while GRPO updates the key from binary correctness rewards on sampled positions.
As suggested, we replace GRPO with correct-position cross-entropy (CE), retaining the position task and key-only updates. CE supplies more information than sampled binary verification. We also show that, as the group size G→∞, GRPO updates align with the CE descent direction, with a different effective step length (Appendix F.7).
Two additional settings with different signal powers exhibit ordering reversals while both schedules substantially improve prediction. We report the CE control in Appendix H.5 and the parameter comparisons in Appendix H.6.
2. Model simplifications and practical implications (Weakness 2 and Question 2).
Thank you for asking about the connection to practical large-model training. We build on the single-location regression task introduced by Marion et al. (2025) to study how training objectives interact under data reuse. The model retains nonlinear softmax competition and policy-dependent verifier feedback. Earlier updates change the representations and sampling probabilities used by later updates, so training order can affect learning.
Gaussian inputs and a single informative location provide a minimal, tractable setting for studying objective order and sample reuse. Our experiments show that some settings exhibit noise-dependent ordering reversals, while other allocations favor interleaving at every sampled noise level. We have clarified the modeling tradeoff in Section 2.3.
Our analysis does not yet establish general schedule recommendations for large models. A qualitative intuition is that supervised learning can build representations that help later GRPO when successful verifier samples are sparse. Our MSE stage illustrates this idea, with token-level SFT serving as a motivating analogue in LLMs. Extending the theory and validating this intuition in practical models remain future work. We have stated this limitation in the Conclusion.