- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a shared-trajectory meta-learning framework for joint SFT and RL training of LLMs. It combines SFT and RL updates within a shared inner loop and applies a Reptile-style outer update to improve gradient alignment between the two objectives.
Critical strengths
The method does not require explicit second-order gradient computation and is relatively straightforward to implement. The experiments cover multiple benchmarks, compare against several recent hybrid SFT–RL methods. The authors also apply SFT warm-up to the reproduced hybrid baselines, helping to make the comparisons fairer by accounting for the effect of initialization.
Critical weaknesses
(1) Some derivations have obvious gaps. Specifically, the step from Equation (9) to Equation (10) does not clearly consider how inner-loop parameters depend on initial parameters. This creates a mismatch between the update direction and the gradient of the target function. The authors may explain the assumptions for this derivation and add its full detailed derivation in the appendix.
(2) The current experimental evidence does not sufficiently establish whether the proposed method achieves gradient alignment between SFT and RL, or whether such alignment contributes to the observed performance gains.
Questions for authors
(1) Could the authors clarify the transition from Equation (9) to Equation (10)? Consider a scalar parameter and two inner steps, with After factoring out the common learning-rate multiplier, the update direction in Equation (9) is However, directly differentiating the objective in Equation (10) gives The additional chain-rule term is generally nonzero. How is this difference accounted for when interpreting Equation (9) as optimizing Equation (10)?
(2) Adding meta-updates improves RL-only training from 56.9 to 58.1 (+1.2) and joint SFT–RL training from 57.5 to 58.9 (+1.4). These similar gains are also consistent with meta-updates themselves improving optimization, leaving the independent contribution of cross-objective alignment unclear. Could the authors provide additional evidence to distinguish these two explanations?
7 comments
W2 & Q2: Thank you for the comment. First, we want to clarify that we have already provided evidence of improved cross-objective gradient matching. Figure 3(c) evaluates SFT–RL gradient compatibility, showing that our method achieves the highest proportion of positive gradient similarities.
Although we have demonstrated the presence of gradient matching, isolating its contribution remains challenging, as gradient matching is not the only mechanism of meta-learning. As discussed in Sections 3.2 and 3.3, the benefits of ensembling and flat minima also extend to single-objective training. Thus, meta-learning can yield improvements even in the absence of cross-objective interactions.
To isolate the effect of cross-objective gradient matching, an experiment must satisfy three conditions: (1) include both SFT and RL data, (2) train within the meta-learning framework, and (3) allow cross-objective gradient matching to be controlled independently. To this end, we design the following experiment: we separate SFT and RL data at the iteration level, such that each iteration performs meta-learning using either SFT data or RL data exclusively. In this way, gradient matching occurs only within each objective, while cross-objective gradient matching is eliminated.
| Initialization | conf | AIME24 | AIME25 | AMC | MATH-500 | Minerva | Olympiad | Avg. |
|---|---|---|---|---|---|---|---|---|
| SFT | 33.4 | 27.0 | 75.2 | 88.6 | 41.9 | 59.9 | 54.3 | |
| SFT | within | 38.1 | 29.3 | 78.8 | 92.0 | 43.4 | 64.8 | 57.7 |
| SFT | cross | 41.8 | 30.5 | 79.7 | 92.0 | 43.8 | 63.6 | 58.6 |
| Base | 16.6 | 8.2 | 49.3 | 70.4 | 27.2 | 34.3 | 34.3 | |
| Base | within | 33.8 | 27.8 | 74.6 | 89.6 | 46.0 | 60.1 | 55.3 |
| Base | cross | 38.5 | 31.4 | 77.7 | 92.8 | 43.8 | 62.5 | 57.8 |
Thank you for acknowledging and correcting the derivation. Could you clarify the assumptions needed for the symmetry identity in the corrected proof, particularly whether it holds for both random objective selection and the fixed alternating schedule?
W1 & Q1:
Thank you for carefully examining our equations and providing this, which helps us improve the quality of the paper. Our response and correction include two points:
This issue does not affect the conclusion, as established techniques for proving gradient matching already exist in previous literature [1]. Our alternative proof unfortunately overlooked this flaw. We sincerely appreciate your careful observation.
Eqs. (10) and (13) can be removed, and Eq. (9) can instead be analyzed by substituting ∂θr∂Li for L in Eq. (21), without changing the main structure of our proof.
The gradient matching has an established analysis, which appears on pages 5–7 of [1]. Particularly relevant is Eq. (32), giving the following identity:
E![∇2Li(θ)∇Lj(θ)]![∇2Lj(θ)∇Li(θ)]![∇Li(θ)⋅∇Lj(θ)].=E=21∇θE
In our manuscript, we extend this algorithm from within-objective to cross-objective gradient matching to address conflicting signals in joint SFT–RL training. The most direct correction would be to follow the analysis in [1] by only replacing the objectives to SFT and RL. However, we would like to preserve the distinctive structure of our induction-based argument. The corrected proof is given below.
We first recall the induction argument in Appendix A. We show that
\mathcal L(\theta_i) = \mathcal L(\theta_1) -\alpha\sum_{j=1}^{i-1} \nabla\mathcal L_j(\theta_1)\cdot\nabla\mathcal L(\theta_1) +O(\alpha^2), \tag{21}
where
θj+1=θj−α∇Lj(θj).
For i=1, the statement is immediate. For i=2, Taylor expansion gives
\mathcal L(\theta_2) = \mathcal L(\theta_1) -\alpha\nabla\mathcal L_1(\theta_1)\cdot\nabla\mathcal L(\theta_1) +O(\alpha^2). \tag{22}
Assume Eq. (21) holds for i=k. The inductive step is
\begin{aligned} \mathcal L(\theta_{k+1}) &= \mathcal L(\theta_k) -\alpha\nabla\mathcal L_k(\theta_k)\cdot\nabla\mathcal L(\theta_k) +O(\alpha^2)\\\\ &= \mathcal L(\theta_1) -\alpha\sum_{j=1}^{k-1} \nabla\mathcal L_j(\theta_1)\cdot\nabla\mathcal L(\theta_1)\\\\ &\quad -\alpha \bigl(\nabla\mathcal L_k(\theta_1)+O(\alpha)\bigr) \cdot \bigl(\nabla\mathcal L(\theta_1)+O(\alpha)\bigr) +O(\alpha^2)\\\\ &= \mathcal L(\theta_1) -\alpha\sum_{j=1}^{k} \nabla\mathcal L_j(\theta_1)\cdot\nabla\mathcal L(\theta_1) +O(\alpha^2). \end{aligned} \tag{23}
This completes the induction. Setting L=Li recovers Eq. (12).
We now connect this result to the actual outer update:
\theta \leftarrow \theta-\varepsilon\alpha \sum_{i=1}^{n}\nabla\mathcal L_i(\theta_i). \tag{9}
For each parameter component θr, we apply Eq. (21) with L=∂θr∂Li. Combining the resulting componentwise expansions yields
∇Li(θi)=∇Li(θ)−α∑j=1i−1∇2Li(θ)∇Lj(θ)+O(α2).
Using the symmetry identity from [1], we obtain
E![i=1∑n∇Li(θi)]![i=1∑n∇Li(θ)−αi=1∑nj=1∑i−1∇2Li(θ)∇Lj(θ)]+O(α2)![i=1∑nLi(θ)−2αi=1∑nj=1∑i−1∇Li(θ)⋅∇Lj(θ)]+O(α2).=E=∇θE
Thus, the expected update admits a first-order interpretation involving the loss sum and a gradient matching term. When steps i and j use different objectives, this term includes interactions between SFT and RL gradients.
We will remove the unsupported transition from Eq. (9) to Eq. (10), retain the valid induction argument, and add the derivation above. Thank you again for identifying this issue.
[1] Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms.