acceptodds
Under review as a conference paper at ICLR 2027

Shared-Trajectory Meta-Learning for Joint SFT and RL Optimization in LLMs

Abstract

Supervised fine-tuning (SFT) and reinforcement learning (RL) are two synergistic paradigms for large language models, typically applied in a SFT then RL pipeline. However, the staged schedule can cause RL to overwrite the gains acquired from SFT. Jointly optimizing both is challenging because their gradients may conflict, and existing methods usually consider which signal should dominate while overlooking their interaction. We propose a meta-learning framework that places SFT and RL on a shared inner trajectory and applies a Reptile-style outer update. This implicitly encourages alignment across their gradients, guiding the model towards the regions where progress on one objective is less likely to hinder the other. Our method outperforms SFT, RL, and hybrid methods across multiple backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Supervised Fine-tuning, Reinforcement Learning, Meta-Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jVGjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a shared-trajectory meta-learning framework for joint SFT and RL training of LLMs. It combines SFT and RL updates within a shared inner loop and applies a Reptile-style outer update to improve gradient alignment between the two objectives.

Critical strengths

The method does not require explicit second-order gradient computation and is relatively straightforward to implement. The experiments cover multiple benchmarks, compare against several recent hybrid SFT–RL methods. The authors also apply SFT warm-up to the reproduced hybrid baselines, helping to make the comparisons fairer by accounting for the effect of initialization.

Critical weaknesses

(1) Some derivations have obvious gaps. Specifically, the step from Equation (9) to Equation (10) does not clearly consider how inner-loop parameters depend on initial parameters. This creates a mismatch between the update direction and the gradient of the target function. The authors may explain the assumptions for this derivation and add its full detailed derivation in the appendix.

(2) The current experimental evidence does not sufficiently establish whether the proposed method achieves gradient alignment between SFT and RL, or whether such alignment contributes to the observed performance gains.

Questions for authors

(1) Could the authors clarify the transition from Equation (9) to Equation (10)? Consider a scalar parameter and two inner steps, with θ2(θ)=θ−αLSFT′(θ).\theta_2(\theta)=\theta-\alpha L'_{\mathrm{SFT}}(\theta). After factoring out the common learning-rate multiplier, the update direction in Equation (9) is LSFT′(θ)+LRL′(θ2).L'_{\mathrm{SFT}}(\theta)+L'_{\mathrm{RL}}(\theta_2). However, directly differentiating the objective in Equation (10) gives ddθ[LSFT(θ)+LRL(θ2(θ))]=LSFT′(θ)+LRL′(θ2)dθ2dθ=LSFT′(θ)+LRL′(θ2)[1−αLSFT′′(θ)]=LSFT′(θ)+LRL′(θ2)−αLSFT′′(θ)LRL′(θ2)⏟additional term absent from Eq. (9).\begin{aligned}\frac{d}{d\theta}\left[L_{\mathrm{SFT}}(\theta)+L_{\mathrm{RL}}(\theta_2(\theta))\right] &= L'_{\mathrm{SFT}}(\theta)+L'_{\mathrm{RL}}(\theta_2)\frac{d\theta_2}{d\theta} \\\\ &= L'_{\mathrm{SFT}}(\theta)+L'_{\mathrm{RL}}(\theta_2)\left[1-\alpha L''_{\mathrm{SFT}}(\theta)\right] \\\\ &= L'_{\mathrm{SFT}}(\theta)+L'_{\mathrm{RL}}(\theta_2)-\underbrace{\alpha L''_{\mathrm{SFT}}(\theta)L'_{\mathrm{RL}}(\theta_2)}_{\text{additional term absent from Eq. (9)}}.\end{aligned} The additional chain-rule term is generally nonzero. How is this difference accounted for when interpreting Equation (9) as optimizing Equation (10)?

(2) Adding meta-updates improves RL-only training from 56.9 to 58.1 (+1.2) and joint SFT–RL training from 57.5 to 58.9 (+1.4). These similar gains are also consistent with meta-updates themselves improving optimization, leaving the independent contribution of cross-objective alignment unclear. Could the authors provide additional evidence to distinguish these two explanations?

7 comments

W1 & Q1:

Thank you for carefully examining our equations and providing this, which helps us improve the quality of the paper. Our response and correction include two points:

  1. This issue does not affect the conclusion, as established techniques for proving gradient matching already exist in previous literature [1]. Our alternative proof unfortunately overlooked this flaw. We sincerely appreciate your careful observation.

  2. Eqs. (10) and (13) can be removed, and Eq. (9) can instead be analyzed by substituting ∂Li∂θr\frac{\partial\mathcal L_i}{\partial\theta_r} for L\mathcal L in Eq. (21), without changing the main structure of our proof.

The gradient matching has an established analysis, which appears on pages 5–7 of [1]. Particularly relevant is Eq. (32), giving the following identity:

E![∇2Li(θ)∇Lj(θ)]=E![∇2Lj(θ)∇Li(θ)]=12∇θE![∇Li(θ)⋅∇Lj(θ)].\begin{aligned} \mathbb E\\!\left[ \nabla^2\mathcal L_i(\theta)\nabla\mathcal L_j(\theta) \right] &= \mathbb E\\!\left[ \nabla^2\mathcal L_j(\theta)\nabla\mathcal L_i(\theta) \right]\\\\ &= \frac12\nabla_\theta\mathbb E\\!\left[ \nabla\mathcal L_i(\theta)\cdot\nabla\mathcal L_j(\theta) \right]. \end{aligned}

In our manuscript, we extend this algorithm from within-objective to cross-objective gradient matching to address conflicting signals in joint SFT–RL training. The most direct correction would be to follow the analysis in [1] by only replacing the objectives to SFT and RL. However, we would like to preserve the distinctive structure of our induction-based argument. The corrected proof is given below.

We first recall the induction argument in Appendix A. We show that

\mathcal L(\theta_i) = \mathcal L(\theta_1) -\alpha\sum_{j=1}^{i-1} \nabla\mathcal L_j(\theta_1)\cdot\nabla\mathcal L(\theta_1) +O(\alpha^2), \tag{21}

where

θj+1=θj−α∇Lj(θj).\theta_{j+1} = \theta_j-\alpha\nabla\mathcal L_j(\theta_j).

For i=1i=1, the statement is immediate. For i=2i=2, Taylor expansion gives

\mathcal L(\theta_2) = \mathcal L(\theta_1) -\alpha\nabla\mathcal L_1(\theta_1)\cdot\nabla\mathcal L(\theta_1) +O(\alpha^2). \tag{22}

Assume Eq. (21) holds for i=ki=k. The inductive step is

\begin{aligned} \mathcal L(\theta_{k+1}) &= \mathcal L(\theta_k) -\alpha\nabla\mathcal L_k(\theta_k)\cdot\nabla\mathcal L(\theta_k) +O(\alpha^2)\\\\ &= \mathcal L(\theta_1) -\alpha\sum_{j=1}^{k-1} \nabla\mathcal L_j(\theta_1)\cdot\nabla\mathcal L(\theta_1)\\\\ &\quad -\alpha \bigl(\nabla\mathcal L_k(\theta_1)+O(\alpha)\bigr) \cdot \bigl(\nabla\mathcal L(\theta_1)+O(\alpha)\bigr) +O(\alpha^2)\\\\ &= \mathcal L(\theta_1) -\alpha\sum_{j=1}^{k} \nabla\mathcal L_j(\theta_1)\cdot\nabla\mathcal L(\theta_1) +O(\alpha^2). \end{aligned} \tag{23}

This completes the induction. Setting L=Li\mathcal L=\mathcal L_i recovers Eq. (12).

We now connect this result to the actual outer update:

\theta \leftarrow \theta-\varepsilon\alpha \sum_{i=1}^{n}\nabla\mathcal L_i(\theta_i). \tag{9}

For each parameter component θr\theta_r, we apply Eq. (21) with L=∂Li∂θr\mathcal L=\frac{\partial\mathcal L_i}{\partial\theta_r}. Combining the resulting componentwise expansions yields

∇Li(θi)=∇Li(θ)−α∑j=1i−1∇2Li(θ)∇Lj(θ)+O(α2).\nabla\mathcal L_i(\theta_i) = \nabla\mathcal L_i(\theta) -\alpha\sum_{j=1}^{i-1} \nabla^2\mathcal L_i(\theta)\nabla\mathcal L_j(\theta) +O(\alpha^2).

Using the symmetry identity from [1], we obtain

E![∑i=1n∇Li(θi)]=E![∑i=1n∇Li(θ)−α∑i=1n∑j=1i−1∇2Li(θ)∇Lj(θ)]+O(α2)=∇θE![∑i=1nLi(θ)−α2∑i=1n∑j=1i−1∇Li(θ)⋅∇Lj(θ)]+O(α2).\begin{aligned} \mathbb E\\!\left[ \sum_{i=1}^{n}\nabla\mathcal L_i(\theta_i) \right] &= \mathbb E\\!\left[ \sum_{i=1}^{n}\nabla\mathcal L_i(\theta) -\alpha\sum_{i=1}^{n}\sum_{j=1}^{i-1} \nabla^2\mathcal L_i(\theta)\nabla\mathcal L_j(\theta) \right] +O(\alpha^2)\\\\ &= \nabla_\theta\mathbb E\\!\left[ \sum_{i=1}^{n}\mathcal L_i(\theta) -\frac{\alpha}{2} \sum_{i=1}^{n}\sum_{j=1}^{i-1} \nabla\mathcal L_i(\theta)\cdot\nabla\mathcal L_j(\theta) \right] +O(\alpha^2). \end{aligned}

Thus, the expected update admits a first-order interpretation involving the loss sum and a gradient matching term. When steps ii and jj use different objectives, this term includes interactions between SFT and RL gradients.

We will remove the unsupported transition from Eq. (9) to Eq. (10), retain the valid induction argument, and add the derivation above. Thank you again for identifying this issue.

[1] Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms.

W2 & Q2: Thank you for the comment. First, we want to clarify that we have already provided evidence of improved cross-objective gradient matching. Figure 3(c) evaluates SFT–RL gradient compatibility, showing that our method achieves the highest proportion of positive gradient similarities.

Although we have demonstrated the presence of gradient matching, isolating its contribution remains challenging, as gradient matching is not the only mechanism of meta-learning. As discussed in Sections 3.2 and 3.3, the benefits of ensembling and flat minima also extend to single-objective training. Thus, meta-learning can yield improvements even in the absence of cross-objective interactions.

To isolate the effect of cross-objective gradient matching, an experiment must satisfy three conditions: (1) include both SFT and RL data, (2) train within the meta-learning framework, and (3) allow cross-objective gradient matching to be controlled independently. To this end, we design the following experiment: we separate SFT and RL data at the iteration level, such that each iteration performs meta-learning using either SFT data or RL data exclusively. In this way, gradient matching occurs only within each objective, while cross-objective gradient matching is eliminated.

InitializationconfAIME24AIME25AMCMATH-500MinervaOlympiadAvg.
SFT33.427.075.288.641.959.954.3
SFTwithin38.129.378.892.043.464.857.7
SFTcross41.830.579.792.043.863.658.6
Base16.68.249.370.427.234.334.3
Basewithin33.827.874.689.646.060.155.3
Basecross38.531.477.792.843.862.557.8

Thank you for acknowledging and correcting the derivation. Could you clarify the assumptions needed for the symmetry identity in the corrected proof, particularly whether it holds for both random objective selection and the fixed alternating schedule?

Reviewer_C6MROpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to jointly leverage supervised fine-tuning (SFT) and reinforcement learning (RL) in LLM post-training. In a conventional SFT→RL pipeline, subsequent RL may overwrite capabilities acquired through SFT, while directly mixing the two losses may introduce gradient interference. The authors propose Shared-Trajectory Meta-Learning: SFT and RL updates are interleaved along a shared inner trajectory, followed by a Reptile-style outer update that moves the initial parameters toward the inner-loop endpoint. Through a first-order expansion, the authors argue that this procedure implicitly encourages alignment between SFT and RL gradients, steering the model toward regions where the two objectives are more compatible. The experiments cover Qwen2.5-Math-7B, Qwen3-4B-Base, Qwen3-8B-Base, and Llama-3.1-8B-Instruct, with mathematical reasoning benchmarks and out-of-distribution (OOD) evaluations such as ARC, GPQA, and MMLU. On Qwen2.5-Math-7B, the authors report an average mathematics score of 58.6 and an OOD average of 69.1, exceeding their reproduced SFT, GRPO, SFT→GRPO, LUFFY, ReLIFT, SRFT, and Prefix-RFT baselines. Ablations show that meta-learning already benefits a single objective, and placing SFT and RL on a shared trajectory yields further improvements.

Critical strengths

The paper makes a concrete technical contribution to cross-objective optimization. The shared inner trajectory and Reptile-style outer update place SFT/RL interaction inside the update rule rather than merely tuning fixed loss weights. This is the central distinction from ordinary alternating training and existing hybrid post-training methods.

Why this matters for the overall assessment: If the gains remain under matched compute, this mechanism provides the main reason to accept the paper.

Critical weaknesses
    • The method’s gain is not yet fully separated from training budgets and baseline implementations. The main experiments include SFT warm-up, and the optimizers, data, training steps, and rollout budgets should be matched more strictly across methods.

    • Why this matters for the overall assessment: This is the empirical risk most likely to change the recommendation. If the advantage disappears under matched tokens, rollouts, and GPU time, the central claim would be substantially weakened.

    • The necessity of jointly optimizing SFT and RL is not sufficiently established by Table 3. The paper is motivated by the claim that jointly using SFT and RL is better than relying on a single fine-tuning objective. However, Table 3 shows that the meta-learning + RL configuration already achieves strong performance, so the current evidence does not demonstrate that SFT provides an indispensable benefit during the final joint optimization stage.

    • Why this matters for the overall assessment: This directly challenges the motivation of the paper. If meta-learning + RL, or another normalized/stabilized RL method, reaches comparable performance under the same budget, the additional complexity and scientific contribution of jointly optimizing SFT and RL would be substantially reduced.

Questions for authors
  1. Under identical SFT warm-up, RL rollout counts, optimizer update counts, and GPU-time budgets, how much does the shared-trajectory method improve over simple alternating training?

  2. How robust is the method to the inner learning rate, SFT/RL ratio, random scheduling probability, and outer learning rate? Could the authors report means, standard deviations, and failure cases across training seeds?

  3. Could the authors directly measure the relationship between SFT/RL gradient inner products, gradient angles, and final performance to test whether the proposed gradient-matching mechanism occurs during training?

  4. How does the method differ from SRFT, UFT, LUFFY, Prefix-RFT, and ReLIFT? Are any of these methods special cases, or do they use fundamentally different mechanisms?

6 comments

Thank you for your comments. However, we believe that some of the weaknesses do not accurately reflect the facts.

  1. The first weakness is that our setup differs from compared methods, which is incorrect. In fact, the experimental settings are identical. We not only report the results from their original papers,but also ensure a fair comparison by implementing these methods on our baseline which yields even higher results. (see Hybrid Method (w/ SFT warm-up) in Table 1).

  2. The second weakness attributes to us a motivation that we never made, that jointly using SFT and RL is better than relying on a single objective. Rather, we position our work as an exploratory study. As stated in Line 40 of our manuscript, "Jointly leveraging heterogeneous training sources is a commonly discussed topic in machine learning." Our initial goal is simply to investigate whether this idea can also be beneficial in LLM post-training.

  3. The reviewer also treats one of our positive findings as a weakness. The fact that RL combined with meta-learning alone already yields performance improvements is a strength of our meta-learning approach. Yet, the reviewer uses this result to argue that combining SFT, RL, and meta-learning is unnecessary, while overlooking the further performance gains achieved by this.

Q1: Our method uses the same setup as others. Specially, SFT warm-up is to obtain a strong starting model. Existing methods often report relatively low performance when initialized from the base model, as the underdeveloped chain-of-thought is difficult to improve through RL. We use SFT to provide a stronger foundation for subsequent SFT+RL to further elicit its reasoning capabilities. For a fair comparison, we also apply SFT warm-up to all hybrid baselines that we reproduce.

The results on Qwen2.5-Math-7B under both initialization are shown below. (Same as Table 1 in our main text.)

From base:

MethodAIME24AIME25AMCMATH500MinervaOlympiadAvg.
LUFFY30.722.566.286.841.255.350.4
ReLIFT28.322.965.187.4—57.8—
SRFT35.3—74.389.839.758.3—
Prefix-RFT31.826.468.288.440.355.751.8
Ours38.531.477.792.843.862.557.8

From SFT:

MethodAIME24AIME25AMCMATH-500MinervaOlympiadAvg.
LUFFY39.530.277.090.843.464.557.6
ReLIFT39.730.178.891.440.463.457.3
SRFT34.225.075.991.044.161.955.4
Prefix-RFT38.229.278.190.242.662.856.9
Ours41.830.579.792.043.863.658.6

Q2: We did not specifically tune the ratio between SFT and RL. The fact that randomly choosing either objective already works well is an evidence of robustness.

For the inner-loop learning rate, we follow the strong baseline[1]. We set the outer-loop learning rate to 0.3 to keep the update scale equal to that of a single inner-loop step: 0.3×3=0.9≈1,0.3 \times 3 = 0.9 \approx 1, where 3 is the number of inner-loop steps.

During the early stages of this work, we did conduct some ablation on the inner-loop learning rate using from-base SFT meta-learning. The outer-loop learning rate β\beta was adjusted accordingly to keep the update scale consistent.

Inner lrOuter lrAIME24AIME25AMCMATH500MinervaOlympiadAvg.
5×10−55\times10^{-5}0.335.930.176.091.642.761.156.2
5×10−45\times10^{-4}0.0335.128.477.491.841.262.656.1
5×10−35\times10^{-3}0.00335.728.977.088.640.461.655.4
5×10−25\times10^{-2}0.000333.426.976.390.243.060.155.0

[1] Limozin, Alexis, et al. "SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning." arXiv preprint arXiv:2604.23747 (2026).

The results of three training runs are reported below.

RunAIME24AIME25AMCMATH500MinervaOlympiadAvg.
041.830.579.792.043.863.658.57
140.929.680.591.845.664.558.82
241.030.178.692.044.964.558.52
Mean ± SD41.2 ± 0.530.1 ± 0.579.6 ± 1.091.9 ± 0.144.8 ± 0.964.2 ± 0.558.63 ± 0.16

Q3: Please refer to Figure3(c) for more details.

Q4: Please refer to Line 53 of our manuscript, the Related Work section, and the Strengths section above.

I have carefully read the authors’ response and have increased my score to 2. However, my central concern remains unresolved: given that meta-learning + RL already achieves strong performance, is it necessary to jointly optimize SFT and RL during the final stage?

Could the authors provide additional ablations comparing meta-learning + RL with meta-learning + RL + SFT under matched data, training-step, and compute budgets? Alternatively, if meta-learning + RL already accounts for most of the improvement, the authors should clarify the positioning and contribution of the paper.

Reviewer_mndcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to jointly optimize supervised fine tuning and reinforcement learning for LLM post training. The authors argue that staged SFT followed by RL may lead to forgetting, while directly combining the two objectives can suffer from conflicting gradients.

The proposed method places SFT and RL updates on a shared inner trajectory and applies a Reptile style outer update. Through a first order analysis, the authors argue that this implicitly encourages cross objective gradient alignment and moves the model toward parameter regions where improvements on one objective are less likely to harm the other.

Experiments on several LLM backbones and both mathematical reasoning and out of distribution benchmarks show improvements over SFT only, RL only, staged SFT to RL, and several recent hybrid methods. The paper also provides ablations on the inner loop design and an analysis of SFT and RL gradient cosine similarity.

Critical strengths
  1. The paper presents a simple and well motivated way to coordinate SFT and RL without explicitly choosing a fixed weighting between the two objectives. The shared trajectory formulation also provides a clear connection to cross objective gradient compatibility.

  2. The empirical evaluation is reasonably broad. The method is tested on multiple model backbones and compared with several strong SFT, RL, staged, and hybrid baselines, with consistent gains on both math and out of distribution benchmarks.

Critical weaknesses
  1. The related work and novelty positioning could be strengthened. In particular, the paper does not discuss several closely related directions, including BRIDGE [1], DFT [2], and LearnAlign [3]. BRIDGE is especially relevant because it also uses meta-level optimization to coordinate SFT and RL, making the distinction from the proposed shared-trajectory formulation less clear. DFT further shows that the SFT objective itself can be reformulated from an RL perspective, raising the question of whether the observed gradient conflict is fundamentally due to SFT–RL incompatibility or partly due to the use of vanilla SFT. LearnAlign is also relevant to the paper’s emphasis on gradient alignment. A clearer comparison with these works would help establish the paper’s novelty more convincingly.

  2. The empirical gains over the strongest hybrid baselines are often modest, while the paper does not report results across multiple random seeds or provide uncertainty estimates. Given the variance typically associated with RL training, it is difficult to assess whether some of the reported improvements are robust. Reporting standard deviations or confidence intervals for the strongest baseline comparisons would make the empirical conclusions more convincing.

[1] Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning.

[2] On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification.

[3] LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment.

Questions for authors
  1. Could the authors clarify the distinction from BRIDGE, which also uses meta level optimization to coordinate SFT and RL? I think this comparison is important for understanding the novelty of the proposed shared trajectory formulation.

  2. How would the proposed method interact with rectified SFT objectives such as DFT? In particular, does the SFT RL gradient conflict remain similarly strong when vanilla SFT is replaced by an SFT objective whose gradient structure has already been modified from an RL perspective?

  3. Are the reported improvements stable across different random seeds? If multiple runs are available, could the authors provide variance or standard deviation for the comparisons with the strongest hybrid baselines?

5 comments

Compare with BRIDGE:

Thank you for your comment. We agree that BRIDGE is closely related to our setting, and we will expand the discussion to clarify its relationship to our work. However, we would like to point out that meta-learning is a broad class of methods. Although both algorithms are named meta-learning, our implementations are totally different.

BRIDGE is a bi-level optimization approach, that means it has multiple sets of parameters and optimizes them alternately. Specifically, each iteration involves three updates.

First, update the main model by minimizing its combined SFT and RL loss:

θk+1=θk−α∇θ[(1−λ)LSFT(θk,wk)+λLRL(θk,wk)].\theta^{k+1}=\theta^k-\alpha\nabla_{\theta}\left[(1-\lambda)\mathcal{L}_{\mathrm{SFT}}(\theta^k,w^k)+\lambda\mathcal{L}_{\mathrm{RL}}(\theta^k,w^k)\right].

Second, update an auxiliary copy of the model by minimizing its RL loss:

θ^k+1=θ^k−α∇θ^LRL(θ^k,wk).\hat{\theta}^{k+1}=\hat{\theta}^{k}-\alpha\nabla_{\hat{\theta}}\mathcal{L}_{\mathrm{RL}}(\hat{\theta}^{k},w^k).

Third, update the LoRA parameters to decrease the main model's loss while increasing the auxiliary model's RL loss:

wk+1=wk−β∇w[(1−λ)LSFT(θk,wk)+λLRL(θk,wk)−λLRL(θ^k,wk)].\begin{aligned}w^{k+1}=w^k-\beta\nabla_w\Big[&(1-\lambda)\mathcal{L}_{\mathrm{SFT}}(\theta^k,w^k)+\lambda\mathcal{L}_{\mathrm{RL}}(\theta^k,w^k)\\\\&-\lambda\mathcal{L}_{\mathrm{RL}}(\hat{\theta}^k,w^k)\Big].\end{aligned}

The motivation of this design is a penalty-based relaxation of its bilevel problem. The auxiliary RL model approximates the lower-level optimum, and the teacher update incorporates the RL loss gap between the two models. However, narrowing this gap does not guarantee better performance of the main model, since increasing the auxiliary model's loss can also reduce the gap.

In comparison, the relationship between our motivation and first-order meta-learning approach is more explicit:

  1. The optimization objectives of SFT and RL may conflict.
  2. We therefore seek regions in parameter space where their gradients do not conflict.
  3. Such regions can be characterized by smaller angles between the two gradients.
  4. Directly optimizing this alignment would require second-order derivatives, which are computationally expensive.
  5. We therefore use first-order meta-learning to encourage gradient matching without explicitly computing second-order derivatives.

Our method is also more efficient. BRIDGE requires SFT and RL optimization on the main model, together with additional RL optimization on an auxiliary copy. Our method incorporates both SFT and RL updates into a single parameter trajectory, without requiring a separately optimized auxiliary model.

The experimental results also show that our method outperforms BRIDGE. The reported results on Qwen3-8B-Base are listed below:

BenchmarkBRIDGEOurs (from base)Ours
AIME2416.739.146.9
AMC70.079.580.8
MATH50079.091.892.4
Minerva39.745.649.3
Olympiad44.062.564.5

Compare with DFT and LearnAlign:

We are familiar with DFT and consider it a very interesting work. However, we understand its contribution as bringing certain characteristics of RL into supervised learning, rather than enabling supervised learning to replace the functionality of RL.

Consider its derivation, let p=πθ(y⋆∣x)p=\pi_\theta(y^\star\mid x). The DFT loss and its gradient are:

LDFT=−sg⁡(p)log⁡p,gDFT=−∇θLDFT=p,∇θlog⁡p.\mathcal L_{\mathrm{DFT}} = -\operatorname{sg}(p)\log p, \qquad g_{\mathrm{DFT}} = -\nabla_\theta\mathcal L_{\mathrm{DFT}} = p\\,\nabla_\theta\log p.

They then consider an artificially constructed RL task with the following reward: A response that exactly matches the demonstration y⋆y^\star receives a reward of 11, otherwise 00.

The expected reward of this RL task is the probability of generating the demonstration:

JRL=Ey∼πθ(⋅∣x)[1[y=y⋆]]=p.J_{\mathrm{RL}} = \mathbb E_{y\sim\pi_\theta(\cdot\mid x)} \left[\mathbf 1[y=y^\star]\right] = p.

Its gradient is therefore:

gRL=∇θp=p,∇θlog⁡p.g_{\mathrm{RL}} = \nabla_\theta p = p\\,\nabla_\theta\log p.

Thus, the two update directions are equal.

This connection is mathematically interesting. However, in practice, we would not design such a restrictive reward objective, because the probability of rolling out an entire reference response is near to zero.

Setting aside this derivation and examining the implementation, DFT reweights the SFT loss by each token's predicted probability. It therefore remains a weighted variant of supervised fine-tuning. This modification does not imply that DFT reproduces RL's most special capacity for self-improvement through rollout, thus it does not make DFT's gradient structure imitate that of RL.

For LearnAlign, the role of gradient alignment is different from that in our method.

1.First, LearnAlign considers gradient alignment among different training examples within the RL objective. Our method focuses on cross-objective alignment between SFT and RL.

2.Even within a single objective, our RL-only meta-learning variant also differs from LearnAlign. LearnAlign takes a subtractive approach: it evaluates gradient similarity among RL training examples and selects a subset, filtering out examples with lower alignment scores. As a result, some training data are not fully utilized. Rather than manually designed rules, our approach instead guides optimization toward regions of parameter space, where gradients of different objectives become better aligned. In this way, we not only preserve the original structure of the gradients, but also avoid discarding training data and make full use of the entire dataset.

Compare with classic baseline across seeds:

We report three training runs of our method and LUFFY on Qwen2.5-Math-7B:

MethodRunAIME24AIME25AMCMATH500MinervaOlympiadAvg.
LUFFY039.530.277.090.843.464.557.57
138.829.779.591.243.062.657.45
241.229.879.891.241.263.557.78
Mean ± SD39.8 ± 1.229.9 ± 0.378.8 ± 1.591.1 ± 0.242.5 ± 1.263.5 ± 0.957.60 ± 0.16
Ours041.830.579.792.043.863.658.57
140.929.680.591.845.664.558.82
241.030.178.692.044.964.558.52
Mean ± SD41.2 ± 0.530.1 ± 0.579.6 ± 1.091.9 ± 0.144.8 ± 0.964.2 ± 0.558.63 ± 0.16

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We have prepared a revised version, which corrects the error in the original derivation and highlights our motivation.

During the rebuttal discussions, we have also addressed questions regarding the assumptions, related work, and additional results. As the rebuttal period is about to close, we will refine and incorporate them into the next revision of our paper.

We sincerely thank the Area Chair and reviewers for their time and constructive feedback. The discussions have helped us improve the clarity and rigor of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.