acceptodds
Under review as a conference paper at ICLR 2027

Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation

Abstract

On-Policy distillation (OPD) in large language models is shifting from full-trace KL supervision toward more selective training paradigms. Recent OPD methods increasingly focus on selecting which trajectories to learn from, which tokens are most informative, and which supervision signals are most reliable. Motivated by this trend, we rethink optimization granularity of OPD and propose \fireicon FiRe-OPD (Filter, then Reweight), which jointly adjusts supervision signals at both trajectory and token levels. In details, FiRe-OPD first filters trajectories to remove low-quality rollout samples, and then applies soft reweighting within the retained trajectories to emphasize informative tokens. Compared with hard token selection, FiRe-OPD leverages a soft-weighting mechanism to effectively mitigate information loss and enhance optimization stability, thereby achieving finer-grained OPD optimization. We validate the effectiveness of FiRe-OPD across strong-to-weak, single-teacher, and multi-teacher settings, and demonstrate its superiority over recent token-level OPD methods ( (e.g., +6.25 on AIME 2024 in strong-to-weak, +18.81 on Miner in multi-teacher). Our code is available at https://anonymous.4open.science/r/FiRe-OPD-7B28/README.md.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Post-training, Large Language Model, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZGedOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FiRe-OPD combines trajectory filtering and token reweighting for on-policy distillation. It discards student rollouts with low length-normalized teacher likelihood, then scales token advantages using teacher confidence and student uncertainty. The paper evaluates strong-to-weak, same-size single-teacher, and math/code multi-teacher distillation with a Qwen3-4B student, and reports component and hard/soft ablations.

Critical strengths
  1. FiRe-OPD provides a simple way to select student trajectories and redistribute supervision within them without an external verifier. The component ablation supports combining the two levels: the full method reaches 60.83, compared with 59.30 for trajectory filtering alone.
  2. The experiments cover three distillation relationships and report improved aggregate performance in each. The multi-teacher code result is particularly encouraging, with an average of 64.16 versus ExOPD's 62.53.
Critical weaknesses
  1. The paper does not sufficiently establish the specific limitation of existing selective OPD methods that its combination resolves. TIP (Xu et al., 2026) already considers student uncertainty and teacher–student disagreement, while SCOPE (Zheng et al., 2026) uses teacher perplexity to adjust trajectory supervision. FiRe's verifier-free, two-level design differs, but the introduction presents separate treatment of the two granularities as a key limitation without directly diagnosing its consequences. The method then treats low teacher likelihood as evidence of unreliable supervision, although this score alone does not establish either teacher–student divergence or correction reliability. Appendix B supports the default filtering rule empirically, but does not establish the proposed explanation in terms of noisy gradients. These gaps weaken the claimed insight beyond a combination of existing selection and weighting ideas.
  2. The experiments support the tested recipe more clearly than the claimed optimization principle or a robust advantage over strong alternatives. Table 6 compares all four hard/soft combinations, but the manuscript does not specify the alternative trajectory-weighting function, token-selection threshold, normalization, or comparable tuning effort. Their ranking therefore cannot be attributed confidently to granularity alone. Against the strongest reported aggregate baseline in each setting, the gains are 0.67 for strong-to-weak math, 0.53 for single-teacher math, 0.85 for multi-teacher math, and 1.63 for code. No independent training variation or validation-based parameter-selection procedure is reported. Avg@8 averages sampled answers and does not establish robustness across training runs.

References

  • Hou, W., Peng, S., Wang, W., et al. (2026). Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe. arXiv preprint arXiv:2605.03677.
  • Xu, Y., Sang, H., Zhou, Z., et al. (2026). TIP: Token Importance in On-Policy Distillation. arXiv preprint arXiv:2604.14084.
  • Yang, W., Liu, W., Xie, R., et al. (2026). Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv preprint arXiv:2602.12125.
  • Zheng, B., Ma, X., Liang, Y., et al. (2026). SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting. arXiv preprint arXiv:2604.10688.
Questions for authors
  1. Could the authors provide evidence relating teacher likelihood to correction reliability or the advantage distribution on retained and discarded trajectories, to substantiate the interpretation of filtering?
  2. Could the authors provide the exact alternative objectives and selection rates in Table 6, including normalization and how tuning and optimization budgets were made comparable?
  3. Could the authors clarify parameter and checkpoint selection, provide uncertainty for the main baseline comparisons, and explain why Uni-OPD (Hou et al., 2026) is listed in §4.1 but absent from the result tables?
3 comments

We thank the reviewer for the detailed assessment and the constructive suggestions on methodological positioning, experimental controls, and the transparency of model selection and baseline reporting.

Response to W1

The contribution is not “using two confidence signals,” but allocating supervision at two different units: trajectory gating decides whether a complete student path participates in updates; token weighting decides the relative emphasis within a retained path, without binary removal of individual tokens. The two operations serve different purposes and are not interchangeable.

The difference from existing concurrent methods is concrete. The concurrent work TIP performs hard token selection based on student uncertainty and teacher–student divergence; other concurrent work SCOPE routes trajectories by correctness and applies teacher-perplexity-weighted KL on incorrect trajectories. FiRe-OPD differs in two respects: (i) at the trajectory level, it filters only by teacher likelihood, without an external verifier; (ii) at the token level, it uses soft multiplicative reweighting that preserves all token gradients, whereas TIP irreversibly discards tokens. The 2×2 factorial supports this choice: hard token selection (C, 58.23) is substantially worse than soft token reweighting (A, 60.83), especially on AIME25 (+6.67) and HMMT Nov (+6.25).

Response to q1

Teacher likelihood measures the teacher’s relative support for a student-generated path; it is not an answer-correctness classifier. At a fixed prefix, expected teacher NLL decomposes into student entropy plus student-to-teacher KL, so low likelihood reflects distributional mismatch rather than a quality label.

We directly measured what the filter discards on a batch of student-generated trajectories. We applied the length-normalized 20th-percentile rule to trajectories from the initial Qwen3-4B, scored on identical tokens by Qwen3-30B-A3B-Instruct-2507 (temperature/top-p 1.0/1.0, response cap 16,384).

Selection outcomeAllRetainedDiscarded
Trajectory share100%81.25%18.75%
Accuracy75.0%84.6%33.3%
Mean token advantage−0.379−0.355−0.471
Token advantage std1.3461.3231.427
Negative-advantage token fraction37.9%34.9%49.5%

The correct-discard rate is 8.3%; 91.7% of correct trajectories are retained. The retained group has higher accuracy (84.6% vs. 33.3%), and discarded trajectories carry more negative and more dispersed token advantages. This demonstrates that the filter mainly removes incorrect trajectories whose supervision is more negative and less consistent, while also ensuring that the retained set keeps a high accuracy.

We directly measured gradients on the same trajectory pool used for the selection diagnostic. We drew four microbatches from the full pool and four from the likelihood-retained pool, using whole trajectories with matched token counts (paired difference below 5%).

Gradient diagnosticUniform-random poolLikelihood-filtered pool
Mean gradient norm5.6795.249
Gradient variance trace5.5702.152
Response tokens per microbatch15,826–16,20715,683–16,207

The mean gradient norm drops from 5.679 to 5.249, and the gradient variance trace.

E∥g∥2−∥Eg∥2\mathbb{E}\|g\|^2-\|\mathbb{E}g\|^2

drops from 5.570 to 2.152—a 61.4% reduction on the retained pool. After filtering, microbatch gradients fluctuate substantially less. This matches the design motivation: the trajectories removed by the filter are exactly those whose updates are larger and less consistent, so dropping them stabilizes the local update.

We include the above findings in Appendix. We have also revised the relevant discussion in the main text to clarify the rationale for trajectory filtering. Specifically, we frame teacher likelihood as a signal of teacher–student alignment along a student-generated path, rather than a direct measure of answer correctness or supervision quality. We also connect this motivation to the gradient analysis in Appendix, which shows lower local gradient variation on the retained pool. This revision more clearly explains the motivation and empirical support for allocating supervision differently across trajectories instead of supervising all paths uniformly.

Response to q2

The exact definitions are summarized below. Here, tctc denotes teacher confidence and scsc denotes student confusion.

LevelModeMathematical formImplementation / threshold
TrajectoryHardtraj_weight = 1{norm_teacher_logprob >= τ_traj}Bottom N% discarded; the rest receive weight 1. τ_traj is the batch-level traj_skip_percentile (default N=20%).
TrajectorySofttraj_weight = sigmoid(k * (norm_teacher_logprob - τ_traj))k=10 (default traj_soft_k); uses the same τ_traj as the hard gate, matching the cutoff location.
TokenSofttoken_weight = (1 + α * tc) * (1 + β * sc)α=β=1.0; tc is teacher confidence and sc is student confusion.
TokenHardtoken_mask = 1{tc > τ_tc and sc > τ_sc}τ_tc and τ_sc are batch-level 70th percentiles, chosen so that the retained fraction is comparable to the soft gate's effective mass.

The four groups A (Hard+Soft), B (Soft+Soft), C (Hard+Hard), and D (Soft+Hard) share the same optimizer, batch size, rollout budget, and evaluation protocol. The details of the soft/hard selection ablations have been added to the appendix.

Response to q4

Parameter and checkpoint selection. Details are in §4.1.

Uni-OPD. We acknowledge Uni-OPD as valuable concurrent work that presents a systematic dual-perspective framework for improving student exploration and teacher supervision, supported by extensive evaluations across tasks and modalities. Uni-OPD combines data balancing with outcome-guided margin calibration, whereas FiRe-OPD focuses on verifier-free supervision allocation through teacher-likelihood-based trajectory filtering and soft token-level reweighting. The two methods therefore address related challenges from different methodological perspectives. Due to computational constraints, we did not evaluate Uni-OPD experimentally. Its inclusion in the list of evaluated baselines was a writing error; we will correct this and retain a discussion of its contributions and differences from our method in the related work.

Response to W2

We report three independent test runs:

BenchmarkSeed ASeed BSeed CMean ± Std (n=3)
AIME2459.1762.0860.8360.69 ± 1.46
AIME2553.3351.6252.9252.62 ± 0.89
MATH50093.6593.7793.7393.72 ± 0.06
AMC202392.5093.5293.1393.05 ± 0.51
OlympiadBench70.3870.3870.4770.41 ± 0.05
MinervaMATH43.0143.8043.4743.43 ± 0.40
HMMT Feb32.5030.4232.0831.67 ± 1.10
HMMT Nov39.3341.2540.0040.19 ± 0.97
Average60.4860.8660.8360.72 ± 0.21

Three-seed mean 60.72 ± 0.21, all three runs above OPD (58.70) and ExOPD (60.16), margins +2.02 and +0.56 (95% CI [60.21, 61.23]).

Thanks for the detailed rebuttal and new experiments. Clarifying the roles of the trajectory filter and token weights—along with the diagnostics showing higher correctness (84.6% vs. 33.3%) and lower gradient variance—gives much stronger empirical motivation for the filter, and the note on Uni-OPD resolves my baseline question.

However, two main concerns keep me at a 2 (Weak Reject):

  1. Generalizability of Table 6: Switching from soft trajectory weighting to hard filtering helps under soft token weights (58.68 → 60.83) but slightly hurts under hard token selection (58.55 → 58.23). While this supports your setup here, it leaves open when this design transfers elsewhere. Reporting the exact fraction of retained tokens and loss normalization details would help isolate the effect of supervision volume.

  2. Significance of the strong-to-weak gains: While FiRe-OPD's three runs (60.72 ± 0.21) beat ExOPD (60.16), "independent test runs" is ambiguous as to whether seeds varied during training or only evaluation. Clarifying this and reporting baseline variance under the same protocol is needed to gauge that 0.56-point margin.

Reviewer_WzcjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper discuss a point that the student model should not learn from the teacher model on the following cases: the teacher model is not good at the current sampling via a low likelihood, and those tokens with low probability. To solve these issues, authors propose to filter the sampling before the training and reweight the token importance to learn the fine-grained knowledge in OPD. Several empirical studies are conducted to validate the proposed method in the practical training.

Critical strengths
  1. The paper provides an interesting idea in OPD learning and may bring practical benefits in practice.

  2. The paper is easy to understand and well-organized, leading to a clear instruction for readers.

Critical weaknesses
  1. The author posits that assigning a low probability to a trajectory implies a significant gap between the teacher and student distributions. In reality, however: E[log⁡(πT(y))]=−H(πθ)−DL(πθ∥πT)E[\log(\pi_T(y))]=-H(\pi_\theta)-DL(\pi_\theta\Vert\pi_T). Therefore, the teacher likelihood is influenced by both student entropy and KL divergence. Even if the two model distributions are identical, it is still possible to generate trajectories with low likelihood.

  2. The fact that teachers assign a low probability to erroneous trajectories may indicate that these trajectories contain clear signals for error correction. Filtering out such trajectories could mean discarding valuable negative supervision. The authors do not directly verify whether the filtered-out trajectories actually exhibit higher gradient variance or poorer supervision quality.

  3. Trajectory filtering alters the distribution of samples actually used for updates. The current version does not provide a deeper, principle-based explanation, but only relies on heuristic techniques to construct the sample selection process.

  4. The experiments show that the proposed method may result in negative growth compared to standard OPD; this indicates that the criteria used by this approach for sample selection are neither stable nor reliable.

Questions for authors
  1. The low confidence level in a teacher model’s response remains dependent on the input. For instance, if a student provides erroneous information, subsequent reasoning will inevitably lead to low-probability outcomes. Given such scenarios, how can one rely solely on confidence levels to assess the quality of a sample?

  2. The results in Table 3 are very strange. On the Olymp. dataset, after OPD training the performance become lower than the vanilla student model. How did this happen? Could the authors provide details regarding the training process?

  3. The adaptive method utilizes training data of mixed difficulty, but the baseline results appear to follow the original data configuration; this requires verification.

  4. The comparison process is not strictly fair. The current comparison is based on the OPD optimizer. But the refined trick includes the token and sample selection. The authors should provide the independent comparisons with different update methods.

  5. The authors claim that the method reduces noise and improves stability, yet it lacks empirical analysis regarding gradient variance, gradient norm, and the quality of the filtered trajectories. Could authors provide more studies to validate the statements and major points proposed in this paper?

  6. The observed improvements involve a degree of randomness. Could the authors provide statistical error estimates to characterize the magnitude of these improvements?

3 comments

We thank the reviewer for the careful examination of the experimental comparisons and for the constructive suggestions on supervision quality, gradient behavior, and statistical uncertainty.

Response to q1

Teacher likelihood measures the teacher’s relative support for a student-generated path; it is not an answer-correctness classifier. At a fixed prefix, expected teacher NLL decomposes into student entropy plus student-to-teacher KL, so low likelihood reflects distributional mismatch rather than a quality label.

We directly measured what the filter discards on a batch of student-generated trajectories. We applied the length-normalized 20th-percentile rule to trajectories from the initial Qwen3-4B, scored on identical tokens by Qwen3-30B-A3B-Instruct-2507 (temperature/top-p 1.0/1.0, response cap 16,384).

Selection outcomeAllRetainedDiscarded
Trajectory share100%81.25%18.75%
Accuracy75.0%84.6%33.3%
Mean token advantage−0.379−0.355−0.471
Token advantage std1.3461.3231.427
Negative-advantage token fraction37.9%34.9%49.5%

The correct-discard rate is 8.3%; 91.7% of correct trajectories are retained. The retained group has higher accuracy (84.6% vs. 33.3%), and discarded trajectories carry more negative and more dispersed token advantages. This demonstrates that the filter mainly removes incorrect trajectories whose supervision is more negative and less consistent, while also ensuring that the retained set keeps a high accuracy.

We include the above findings in Appendix. We have also revised the relevant discussion in the main text to clarify the rationale for trajectory filtering. Specifically, we frame teacher likelihood as a signal of teacher–student alignment along a student-generated path, rather than a direct measure of answer correctness or supervision quality. We also connect this motivation to the gradient analysis in Appendix, which shows lower local gradient variation on the retained pool. This revision more clearly explains the motivation and empirical support for allocating supervision differently across trajectories instead of supervising all paths uniformly

Response to q2

Thank you for catching this. Table 3 contains a typographical error: OlympiadBench and Minerva entries were swapped in the affected trained-model rows. We correct the entries and the corresponding differences; the reported averages are unchanged.

Response to q3

The adaptive variant uses a mixture of level-5 and level-6 examples, whereas the fixed reference uses level-6 examples only. However, the total number of training examples is the same in both settings. Appendix E presents a supplementary exploratory experiment on difficulty-aware distillation and serves only as a comparison of approaches in the context of the current method.

Response to q4

We do not claim that the filtered update is an unbiased estimate of the original uniformly supervised OPD objective. The precise statement is that the method defines a selective surrogate: the trajectory mask controls whether a token loss participates, and continuous weights allocate the retained supervision. Its value is an implementable allocation rule, not an importance correction that restores the original distribution. This differs from PPO’s old-to-current ratio, which does not remove trajectory-selection bias.

Table 5 keeps the OPD update formulation fixed and compares no selection/weighting, selection only, weighting only, and both; this isolates the allocation components while controlling the update rule. Transfer to other update objectives is outside the scope of these experiments.

Response to q5

We directly measured gradients on the same trajectory pool used for the selection diagnostic. We drew four microbatches from the full pool and four from the likelihood-retained pool, using whole trajectories with matched token counts (paired difference below 5%).

Gradient diagnosticUniform-random poolLikelihood-filtered pool
Mean gradient norm5.6795.249
Gradient variance trace5.5702.152
Response tokens per microbatch15,826–16,20715,683–16,207

The mean gradient norm drops from 5.679 to 5.249, and the gradient variance trace.

E∥g∥2−∥Eg∥2\mathbb{E}\|g\|^2-\|\mathbb{E}g\|^2

drops from 5.570 to 2.152—a 61.4% reduction on the retained pool. After filtering, microbatch gradients fluctuate substantially less. This matches the design motivation: the trajectories removed by the filter are exactly those whose updates are larger and less consistent, so dropping them stabilizes the local update.

The quality-related concerns are addressed in our response to Q1.

We include the above findings in Appendix.

Response to q6

We report three independent test runs:

BenchmarkSeed ASeed BSeed CMean ± Std (n=3)
AIME2459.1762.0860.8360.69 ± 1.46
AIME2553.3351.6252.9252.62 ± 0.89
MATH50093.6593.7793.7393.72 ± 0.06
AMC202392.5093.5293.1393.05 ± 0.51
OlympiadBench70.3870.3870.4770.41 ± 0.05
MinervaMATH43.0143.8043.4743.43 ± 0.40
HMMT Feb32.5030.4232.0831.67 ± 1.10
HMMT Nov39.3341.2540.0040.19 ± 0.97
Average60.4860.8660.8360.72 ± 0.21

Three-seed mean 60.72 ± 0.21, all three runs above OPD (58.70) and ExOPD (60.16), margins +2.02 and +0.56 (95% CI [60.21, 61.23]).

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to clarify the filtering mechanism and provide empirical evidence regarding trajectory quality and gradient variance.

However, several concerns remain. The NLL decomposition holds in expectation and does not establish that individual low-likelihood trajectories provide unreliable supervision. And the gradient variance analysis is based on only four microbatches, and lower variance does not necessarily imply better optimization, particularly when discarded trajectories may contain useful corrective signals. The difficulty-aware comparison still involves different training data distributions, which raises concerns about fairness. While the additional evidence strengthens the empirical motivation, the fundamental justification and experimental reliability remain insufficiently established. I therefore maintain my original rating.

Reviewer_RM1DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FiRe-OPD, an on-policy distillation framework that combines trajectory-level filtering with token-level soft reweighting. The method first filters low-likelihood trajectories based on teacher predictions and then assigns token-level weights using teacher confidence and student confusion, estimated through entropy. The goal is to improve the quality of distillation signals while preserving useful token-level information. Experiments across strong-to-weak, single-teacher, and multi-teacher settings on mathematical reasoning and code generation benchmarks show improvements over standard OPD and several recent baselines. Ablation studies and hyperparameter analyses further examine the contributions of the proposed components.

Critical strengths
  1. Practical and easy-to-integrate framework. FiRe-OPD combines trajectory-level filtering with token-level soft reweighting in a simple framework. It does not require additional reward models or external verifiers, making it practical to integrate into existing OPD training pipelines. This simplicity is a meaningful strength of the proposed approach.
  2. Evaluation across multiple distillation settings. The paper evaluates FiRe-OPD in strong-to-weak, single-teacher, and multi-teacher settings, covering both mathematical reasoning and code generation tasks. The results show improvements over standard OPD, while the ablation studies provide evidence that the two components contribute to performance. This broad evaluation supports the practical usefulness of the method, although the robustness of some smaller improvements remains uncertain.
Critical weaknesses
  1. Limited methodological novelty. The core components of FiRe-OPD, trajectory filtering based on teacher likelihood and token reweighting based on entropy, are relatively straightforward heuristic strategies. Although their combination is reasonable, the paper does not sufficiently clarify what new methodological insight the dual-granularity framework provides beyond integrating existing ideas. This limits the significance of the technical contribution.
  2. Insufficient validation of the core assumptions and performance gains. The method assumes that low teacher likelihood indicates unreliable trajectory-level supervision and that teacher and student entropy can effectively identify informative tokens. However, these assumptions are not directly validated. Furthermore, the improvements over the strongest baseline are relatively small in some settings (e.g., 0.67 points in Table 2), and the paper does not report results across multiple random seeds or statistical uncertainty. These issues make it difficult to assess the robustness of the reported improvements and weaken the evidence supporting the proposed method.
Questions for authors
  1. Clarification of methodological novelty: What is the key methodological contribution of FiRe-OPD beyond combining trajectory-level filtering and token-level reweighting? How does the proposed dual-granularity design differ from existing confidence-based or adaptive OPD methods?
  2. Justification of trajectory filtering: In Eq. (4), the authors use teacher log-likelihood to identify unreliable trajectories. Is there empirical evidence that low teacher likelihood correlates with poor supervision quality? In particular, what proportion of correctly solved trajectories is discarded by the filtering mechanism?
  3. Validation of token-level weighting: What motivates the multiplicative formulation in Eq. (7) over simpler alternatives, such as additive weighting? Can the authors provide a controlled comparison to demonstrate that this specific formulation is necessary for the observed improvements?
2 comments

We thank the reviewer for the thoughtful comments on methodological novelty, the filtering and weighting mechanisms, and the robustness of the reported results.

Response to q1

The contribution is not “using two confidence signals,” but allocating supervision at two different units: trajectory gating decides whether a complete student path participates in updates; token weighting decides the relative emphasis within a retained path, without binary removal of individual tokens. The two operations serve different purposes and are not interchangeable.

The difference from existing concurrent methods is concrete. The concurrent work TIP performs hard token selection based on student uncertainty and teacher–student divergence; other concurrent work SCOPE routes trajectories by correctness and applies teacher-perplexity-weighted KL on incorrect trajectories. FiRe-OPD differs in two respects: (i) at the trajectory level, it filters only by teacher likelihood, without an external verifier; (ii) at the token level, it uses soft multiplicative reweighting that preserves all token gradients, whereas TIP irreversibly discards tokens. The 2×2 factorial supports this choice: hard token selection (C, 58.23) is substantially worse than soft token reweighting (A, 60.83), especially on AIME25 (+6.67) and HMMT Nov (+6.25).

Response to q2

Teacher likelihood measures the teacher’s relative support for a student-generated path; it is not an answer-correctness classifier. At a fixed prefix, expected teacher NLL decomposes into student entropy plus student-to-teacher KL, so low likelihood reflects distributional mismatch rather than a quality label.

We directly measured what the filter discards on a batch of student-generated trajectories. We applied the length-normalized 20th-percentile rule to trajectories from the initial Qwen3-4B, scored on identical tokens by Qwen3-30B-A3B-Instruct-2507 (temperature/top-p 1.0/1.0, response cap 16,384).

Selection outcomeAllRetainedDiscarded
Trajectory share100%81.25%18.75%
Accuracy75.0%84.6%33.3%
Mean token advantage−0.379−0.355−0.471
Token advantage std1.3461.3231.427
Negative-advantage token fraction37.9%34.9%49.5%

The correct-discard rate is 8.3%; 91.7% of correct trajectories are retained. The retained group has higher accuracy (84.6% vs. 33.3%), and discarded trajectories carry more negative and more dispersed token advantages. This demonstrates that the filter mainly removes incorrect trajectories whose supervision is more negative and less consistent, while also ensuring that the retained set keeps a high accuracy.

We include the above findings in Appendix. We have also revised the relevant discussion in the main text to clarify the rationale for trajectory filtering. Specifically, we frame teacher likelihood as a signal of teacher–student alignment along a student-generated path, rather than a direct measure of answer correctness or supervision quality. We also connect this motivation to the gradient analysis in Appendix, which shows lower local gradient variation on the retained pool. This revision more clearly explains the motivation and empirical support for allocating supervision differently across trajectories instead of supervising all paths uniformly

Response to q3

The multiplicative form is designed to strengthen the interaction between teacher confidence and student uncertainty. To characterize its effect, we compare additive and multiplicative weighting on a fixed Qwen3-4B checkpoint, using the same retained trajectories, four paired microbatches, the original advantages, and the same normalization.

Fixed-checkpoint metricUniformAdditiveMultiplicative
Mean normalized weight1.00001.00001.0000
Weight std (mean across microbatches)00.04490.0786
p90 weight (range across batches)11.0444–1.04671.1084–1.1114
Mean gradient norm5.34385.71616.1490
Relative gradient variance0.077850.078320.07890
Held-out surrogate gradient cosine0.497890.495250.49453

Stronger token-weight differentiation. The within-microbatch weight standard deviation is 0.0438–0.0467 for additive and 0.0771–0.0794 for multiplicative; the corresponding p90 weight increases from 1.044–1.047 for additive to 1.108–1.111 for multiplicative. This shows that, under the same mean weight, multiplication produces sharper token-level relative emphasis rather than approximately uniform amplification of all tokens.

Larger local update scale with similar direction. The mean gradient norm over four paired microbatches increases from 5.716 for additive to 6.149 for multiplicative, about +7.6%; the paired gradient cosine between additive and multiplicative is 0.99953–0.99960. In this fixed-checkpoint diagnostic, multiplication affects local gradient scale more than overall direction.

These two observations show that multiplication is a stronger non-uniform modulation than addition, consistent with the design motivation of sharpening token-level supervision differences.

We include the above findings in Appendix.

Response to W2

We report three independent test runs:

BenchmarkSeed ASeed BSeed CMean ± Std (n=3)
AIME2459.1762.0860.8360.69 ± 1.46
AIME2553.3351.6252.9252.62 ± 0.89
MATH50093.6593.7793.7393.72 ± 0.06
AMC202392.5093.5293.1393.05 ± 0.51
OlympiadBench70.3870.3870.4770.41 ± 0.05
MinervaMATH43.0143.8043.4743.43 ± 0.40
HMMT Feb32.5030.4232.0831.67 ± 1.10
HMMT Nov39.3341.2540.0040.19 ± 0.97
Average60.4860.8660.8360.72 ± 0.21

Three-seed mean 60.72 ± 0.21, all three runs above OPD (58.70) and ExOPD (60.16), margins +2.02 and +0.56 (95% CI [60.21, 61.23]).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.