acceptodds
Under review as a conference paper at ICLR 2027

Finite-Corpus Dynamics of Supervised and Verifier-Guided Softmax Attention

Abstract

Large-model training commonly separates pretraining and supervised learning from subsequent reinforcement learning with verifiable rewards, yet the dynamical advantage of such stage separation is not well understood theoretically. We study a solvable finite-corpus attention model that combines mean-squared-error (MSE) training with Group Relative Policy Optimization (GRPO). In a single-location regression task, MSE jointly learns a key and value representation, while GRPO uses exact position feedback to refine only the key. In the high-dimensional limit, we derive a dynamical mean-field theory that reduces the finite-corpus dynamics to a self-consistent single-channel process. The theory accurately tracks learning across an abrupt MSE-to-GRPO stage switch and explains verifier-guided refinement after supervised learning. We further compare hard staging with the random interleaving strategy under matched supervision and verifier budgets. Paired experiments reveal systematic schedule crossovers with label noise and sequence length. An exact endpoint decomposition and local force analysis show that these crossovers reflect a competition between clean co-adaptation under interleaving and its continued exposure to noisy supervised updates.

Primary area:
learning theory
Keywords:
dynamical mean-field theory, statistical physics, nerual network training dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f6h8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the order of supervised and verifier-guided training affects learning on a finite corpus. It uses a simplified attention regression model: supervised training updates both the key and value using noisy scalar labels, while GRPO updates only the key using rewards for correct position selection. The authors derive dynamical mean-field equations describing repeated data use and objective switching, and validate selected training trajectories through synthetic experiments. With matched update counts and verifier budgets, the experiments show that the relative advantage of staged versus randomly interleaved training changes with label noise and sequence length.

Critical strengths
  1. The training-order comparisons are carefully controlled. The schedules use the same update counts and verifier budgets, and repeated paired experiments strengthen the credibility of the comparisons.

  2. The theoretical analysis accounts for both repeated use of training data and the effect of model changes on subsequent action sampling. Together with the numerical validation, this provides some technical value.

Critical weaknesses
  1. The two training procedures differ not only in their update rules but also in their learning tasks. Supervised training learns the final scalar prediction and updates both the key and value, whereas GRPO learns position selection using exact position rewards and freezes the value. It is therefore difficult to determine how much of the observed training-order effect comes from alternating supervised learning and policy optimization, and how much comes from differences in tasks, feedback quality, and the parameters being updated.

  2. The overall setup differs substantially from models and training tasks with practical applications. The model contains only two trainable vectors, the key and value, and the task is to locate a token in a synthetic Gaussian sequence and predict a scalar. The paper does not sufficiently explain whether these simplifications preserve the key factors that determine which training order works better, or how its findings can guide training choices for practical models. Its practical guidance therefore remains unclear.

Questions for authors
  1. How can the training-order effects arising from policy sampling be distinguished from those arising from differences in the tasks, feedback, and parameters updated by the two procedures? As a possible diagnostic comparison, would the main conclusions still hold if GRPO were replaced with supervised training on the correct position?

  2. Which key factors governing the relative effectiveness of training orders in practical models are preserved by the simplifications in the model, data, and task? Based on the current analysis, what specific training recommendations can the authors offer, and under what conditions do they apply?

1 comment
AuthorsOpenReview ↗

Response to Reviewer f6h8

We thank the reviewer for recognizing the technical value of our treatment of repeated data use and the influence of model updates on subsequent action sampling.

1. Tasks, feedback and updated parameters (Weakness 1 and Question 1).

Thank you for suggesting position supervision as a diagnostic. MSE updates the key and value from noisy scalar labels, while GRPO updates the key from binary correctness rewards on sampled positions.

As suggested, we replace GRPO with correct-position cross-entropy (CE), retaining the position task and key-only updates. CE supplies more information than sampled binary verification. We also show that, as the group size G→∞G\to\infty, GRPO updates align with the CE descent direction, with a different effective step length (Appendix F.7).

Two additional settings with different signal powers exhibit ordering reversals while both schedules substantially improve prediction. We report the CE control in Appendix H.5 and the parameter comparisons in Appendix H.6.

2. Model simplifications and practical implications (Weakness 2 and Question 2).

Thank you for asking about the connection to practical large-model training. We build on the single-location regression task introduced by Marion et al. (2025) to study how training objectives interact under data reuse. The model retains nonlinear softmax competition and policy-dependent verifier feedback. Earlier updates change the representations and sampling probabilities used by later updates, so training order can affect learning.

Gaussian inputs and a single informative location provide a minimal, tractable setting for studying objective order and sample reuse. Our experiments show that some settings exhibit noise-dependent ordering reversals, while other allocations favor interleaving at every sampled noise level. We have clarified the modeling tradeoff in Section 2.3.

Our analysis does not yet establish general schedule recommendations for large models. A qualitative intuition is that supervised learning can build representations that help later GRPO when successful verifier samples are sparse. Our MSE stage illustrates this idea, with token-level SFT serving as a motivating analogue in LLMs. Extending the theory and validating this intuition in practical models remain future work. We have stated this limitation in the Conclusion.

Reviewer_EYeBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main theoretical contribution of the paper is a dynamical mean-field theory (DMFT) for a solvable toy model trained first with supervised learning and then with RL, on a fixed, reused dataset. The genuinely new piece is a score-function term that tracks how RL’s own sampled actions feed back into training. The evidence for the theory is in good agreement with simulations in Figs. 2–3.

Critical strengths
  • Prior high-dimensional dynamics work covers online SGD on single-index, attention-like models, and multi-pass SGD in multi-index models. None of these handles a training update whose data randomness is policy-dependent: the history score term in the paper, which propagates how earlier parameter perturbations reweight later sampled GRPO actions, appears to be new, and is the right mechanism to capture on-policy RL theoretically. This is a nontrivial theoretical contribution.
  • Why multi-stage training works, and when interleaving beats staging, is actively debated empirically. A solvable model in which the staging-vs-interleaving question can be posed with exactly matched update counts, verifier budgets, data, and sampling randomness is valuable to draw insights. The paper's claim (Figure 4) that staging wins when supervision is noisy, interleaving wins when it is clean, is a testable prediction for real pipelines.
Critical weaknesses
  • The fact that GRPO refines the key while preserving the learned value vector (Sec. 4.2) is a by-hand construction, not a finding. Likewise, the staging-vs-interleaving mechanism follows directly from this construction. In real RLVR, the reward depends on the generated output, so value/readout parameters do receive gradient updates. Hence, the paper's central tradeoff (interleaving allows continued value co-adaptation but continued noise exposure over GRPO's only key update) may not survive in realistic situations.
  • The crossover locations in Figure 4 are measured from small-scale synthetic experiments and have not been proved theoretically (not even perturbatively near the crossover). Comparing DMFT with and without the action-history contribution would show how important that is.
  • The paper is not carefully written. For example, in Figure 1,2 legend is coming on top of the curves. The size of Figure 4 left and right is not the same.
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer EYeB

We thank the reviewer for recognizing our DMFT analysis of GRPO and multi-stage training, particularly the action-history response contribution.

1. Model construction and scope.

Thank you for highlighting the distinction between our solvable model and practical GRPO. We agree that value freezing is a modeling choice arising from the key-dependent position policy. In output policies that depend on attention value parameters, these parameters can receive updates through the log-policy derivative.

The preferred training order depends on the parameters and protocol. Selected settings favor staging, while others favor interleaving. Extending the analysis to more complex policies and establishing implications for practical training remain future work. We have clarified this limitation in the Conclusion.

2. Perturbative crossover prediction.

Thank you for requesting the perturbative analysis near the crossover. We did it. We derive first and second derivatives of terminal test risk with respect to label-noise amplitude σy\sigma_y from the closed DMFT. On a smooth finite-time branch with integrable derivative moments, these derivatives give local linear and quadratic predictions near a simple crossing.

In Figure 9 in the appendix, expansion at σy=2.1\sigma_y=2.1 (in Figure 4(a) setting) gives first- and second-order crossing estimates of 2.2262 and 2.2274, respectively. The DMFT grid gives 2.2278, compared with 2.2350 from SGD at dimension p=768p=768. We have added the derivation and validation in Appendix G.6.

3. Action-history ablation.

Thank you for suggesting the history comparison. We compare self-consistent DMFT predictions with and without the cross-window action-history response against the same SGD process. Omitting the term underestimates endpoint test risk by 11.88% and normalized key squared norm by 17.35%, relative to SGD means. Full DMFT reduces these errors to 0.16% and 0.46% in magnitude. We report this comparison in Appendix H.4.

4. Presentation.

Thank you for the layout suggestions. We have reorganized the model and update descriptions, introduced the mean-field idea before its definitions, and clarified the effective process and closed recursion in Sections 2–3. Figure 6 in Appendix D.5 illustrates one macroscopic update.

Reviewer_1D4yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the training dynamics of a simplified attention model on a single-location regression task, using both mse and grpo. MSE training uses noisy regression labels and updates both key and value vectorss, whereas grpo uses rewards for picking the correct token and updates only key. They study the dynamics in the proportional high-dimensional regime, and compare MSE-only training, switching from mse to grpo, and interleaving the two objectives.

Critical strengths

The work is on a relevant topic these days and developing theoretical intuitions on toy tractable models is a good step.

Critical weaknesses

The paper in general is very hard to read. Starting from the learning model, there’s no intuition why the model only has a trainable key and value vector, why are they studying a simplifying attention model (probably comes from previous works but there’s no explanation or discussion). The text doesn’t say what the goal is, which is for kk to learn k∗k_* and vv to learn v∗v_*. The reader instead has to put the pieces together.

Section 2.2 reads very dense and super hard to parse, especially lines 126-141. There’s no explanation of the basic mechanism, text straight away dives into GRPO terminology. This continues into the rest of the paper, Sec 3, first para reads so dense, loads of terminology just thrown there. Even figure 1, it’s just there. There’s no discussion

I have major doubts regarding the main message of the paper: “mse learns representations and grpo helps with localization”.

MSE training goes through noisy labels, where you train both k and v, whereas grpo training you only train k, why is that? Next, during grpo training you have access to the correct position (through rewards) of course it will help localization. You could train with cross entropy to predict the correct position and it would do that as well, what is special about grpo here? I feel the advantage here that we see in Figure 3 is very artificial. I have similar doubts on figure 4 results re comparison of staged vs switching.

There is a LOT of missing related work on attention dynamics. See Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection and related work in there.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1D4y (1/2)

We thank the reviewer for recognizing the value of tractable models and for requesting clearer motivation and theoretical explanations. We have substantially revised Sections 2–3, redesigned Figure 1 to explain the training mechanisms, and added Figures 5–6 as a visual guide to DMFT.

1. Model motivation and learning goals.

Thank you for requesting the modeling intuition. Single-location regression couples identifying an informative token with predicting its noise-free target. Following Marion et al. (2025) and Duranthon et al. (2026), we use a shared query and scalar output to obtain two effective trainable vectors while retaining nonlinear softmax competition.

We study MSE and GRPO on a reused corpus as a tractable analogue of token-level SFT followed by verifier-guided GRPO in LLM post-training. We have expanded this motivation and the learning goals in Sections 2.1–2.3 and Figure 1.

2. Training mechanisms and parameter updates.

Thank you for pointing out the dense math of Section 2.2. We add new Figure 1(b)–(c) illustrates the MSE and GRPO mechanisms.

Specifically, GRPO samples positions from the current policy, receives binary correctness rewards, and uses group-relative advantages to encourage or discourage the sampled positions.

MSE updates both key and value because the prediction depends on both. Our position policy depends only on the key, making value freezing a model choice. In LLMs, GRPO can update attention value weights through their influence on the output policy. We clarify these operations in Section 2.2 and derive them in Appendix A.3.

3. Mean-field explanation.

Thank you for asking for a clearer entry to DMFT. We add Figure 6 for better visualize those update equations. A general picture of DMFT is follwoing. SGD evolves two pp-dimensional vectors in 2p2p parameter coordinates. As p→∞p\to\infty, DMFT describes this trajectory through overlap, correlation and response matrices whose channel dimensions remain fixed, while retaining their time histories. These quantities determine test risk and student–teacher cosine similarities.

We rewrite Sections 3.1–3.4 and add Figure 6 in Appendix D.5 to show one window update of the closed macroscopic recursion in Eq. (10). Figure 5 in Appendix C.4 illustrates the single-channel student-field process in Eq. (9).

4. Verifier information and position cross-entropy.

Thank you for suggesting direct position supervision. Position cross-entropy (CE) receives the correct-position label, while GRPO receives binary correctness rewards for sampled positions.

At fixed input and parameters, GRPO updates align with the CE descent direction as the group size G→∞G\to\infty, with a different policy-dependent step length. We derive and validate this relation in Appendices F.7 and H.5 (Figure 13).

Our training comparisons use small, finite groups. Feedback covers only the sampled positions, and identical group rewards produce no update, so learning signals can be sparse. This reflects how verifier-guided LLM post-training obtains feedback from sampled outputs, while CE provides stronger direct supervision. We clarify this distinction in Section 2.2 and Appendix H.5.

5. Interpretation and fairness of Figures 3–4.

Thank you for questioning the comparisons. Figure 3 compares continued MSE with MSE followed by GRPO, corresponding to rows 1–2 of Figure 1(d). The runs receive different supervision after the common MSE stage. The further improvement with verifier feedback does not establish a general advantage of GRPO over MSE. We clarify this interpretation in Section 4.2.

Figure 4 compares staging and interleaving, rows 2–3 of Figure 1(d). The schedules share the corpus, supervision access and numbers of MSE and GRPO updates, differing only in temporal order. Sampled positions and rewards can differ as the policies evolve. The matched protocol is described in Sections 2.3 and 4.3.

Additional experiments show that the preferred order depends on the parameters. Selected settings favor staging, while other allocations favor interleaving at every sampled noise level. These comparisons appear in Appendix H.6 (Figure 15 and Table 6).

We also derive local perturbative crossover predictions from first- and second-order DMFT noise sensitivities. The derivation and validation appear in Appendix G.6 (Figures 8–9 and Table 1).

AuthorsOpenReview ↗

Response to Reviewer 1D4y (2/2)

6. Related work.

Thank you for identifying the missing attention literature. We have added ten related papers, listed below.

The new discussion includes Julistiono et al. (2026), Ataee Tarzanagh et al. (2023) and Vasudeva et al. (2025), relating their optimization geometry to our finite-corpus stochastic dynamics. We have expanded Section 5 and the bibliography, with the single-location motivation also added in Section 2.1.

References added to the manuscript (excluded from response word count)

  1. Marion, P., Berthier, R., Biau, G., and Boyer, C. (2025). Attention Layers Provably Solve Single-Location Regression. ICLR.
  2. Dohmatob, E. (2025). Understanding Softmax Attention Layers: Exact Mean-Field Analysis on a Toy Problem. NeurIPS.
  3. Ataee Tarzanagh, D., Li, Y., Zhang, X., and Oymak, S. (2023). Max-Margin Token Selection in Attention Mechanism. NeurIPS.
  4. Vasudeva, B., Deora, P., and Thrampoulidis, C. (2025). Implicit Bias and Fast Convergence Rates for Self-attention. TMLR.
  5. Julistiono, A. K., Ataee Tarzanagh, D., and Azizan, N. (2026). Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection. JMLR, 27(36), 1–61.
  6. Oymak, S., Rawat, A. S., Soltanolkotabi, M., and Thrampoulidis, C. (2023). On the Role of Attention in Prompt-tuning. ICML.
  7. Tian, Y., Wang, Y., Chen, B., and Du, S. S. (2023). Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer. NeurIPS.
  8. Deora, P., Ghaderi, R., Taheri, H., and Thrampoulidis, C. (2024). On the Optimization and Generalization of Multi-head Attention. TMLR.
  9. Chen, S., Sheen, H., Wang, T., and Yang, Z. (2024). Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality (extended abstract). COLT.
  10. Nichani, E., Damian, A., and Lee, J. D. (2024). How Transformers Learn Causal Structure with Gradient Descent. ICML.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all three reviewers for their constructive suggestions. Following their recommendations, we have made the following major revisions.

1. New theoretical analysis

1.1 Perturbative crossover analysis. We derive first- and second-order noise sensitivities from the DMFT closure and use them to predict local crossover locations. The derivation and validation against DMFT and SGD appear in Appendix G.6 (Figures 8–9 and Table 1). (EYeB)

1.2 GRPO and position cross-entropy. As the group size Ggrp→∞G_{\rm grp}\to\infty, GRPO ascent and position cross-entropy descent yield key vector updates in the same direction, with different effective step sizes. We provide the derivation in Appendix F.7 and experimental validation in Appendix H.5 (Figure 13). (1D4y, f6h8)

2. Additional experiments

2.1 Action-history ablation. A self-consistent ablation shows that retaining the cross-window action-history response reduces the absolute relative error in endpoint test risk from 11.88% to 0.16%, using SGD as the reference. This comparison appears in Appendix H.4 (Figure 12 and Table 5). (EYeB)

2.2 Position supervision using cross-entropy. We add a training-order control that replaces GRPO with cross-entropy supervision of the correct position in Appendix H.5 (Figure 14). (1D4y, f6h8)

2.3 Additional parameter comparisons. We add paired experiments at two signal strengths and compare SGD measurements with DMFT predictions in Appendix H.6 (Figure 15). (f6h8)

3. Exposition and figures

3.1 Model motivation and training mechanisms. We redesign Figure 1 and clarify the model motivation and the MSE and GRPO mechanisms in Sections 2.1–2.3 and Appendix A.3, with new visualizations of the training procedures in Figure 1(b)–(c). (1D4y, EYeB, f6h8)

3.2 Mean-field overview and DMFT diagrams. We rewrite Section 3 to introduce the mean-field idea and clarify the DMFT construction. Two new diagrams in Appendices C.4 and D.5 illustrate the single-channel process and one iteration of the macroscopic closure (Figures 5 and 6). (1D4y, EYeB)

3.3 Clarified experimental comparisons. We clarify the change in supervision in Figure 3 and the matched supervision access and budgets in Figure 4. These explanations appear in Sections 2.3 and 4.2–4.3, with the schedules identified in Figure 1(d). (1D4y, f6h8)

4. Related work and figure layout

4.1 Expanded related work. We add ten references and expand the discussion of attention optimization, training dynamics and single-location regression in Section 5. (1D4y)

4.2 Figure layout. We move the Figure 2 legends outside the data regions and match the panel sizes in Figure 4. (EYeB)

The revised manuscript PDF also includes the appendices.

Detailed point-by-point responses to each reviewer are provided below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.