acceptodds
Under review as a conference paper at ICLR 2027

Flux-OPD: On-Policy Distillation with Evolving Contexts

12 views

Abstract

Large language model training in open-ended domains lacks verifiable rewards, making task preferences difficult to formalize as effective supervision. Contexts derived from task feedback can provide richer supervision, yet fixed contexts cannot adapt as the student evolves, motivating contexts that evolve with student performance. However, directly using evolving contexts as in-training supervision results in unstable distillation targets and conflicting distributions, requiring mechanisms to stabilize the target and downweight conflicts. In this paper, we characterize context distillation through a reverse-KL decomposition, showing that the student is distilled toward the geometric mean of context-conditioned teachers, while the objective contains a conflict term measuring disagreement among these teachers. Motivated by this characterization, we propose Flux-OPD, an OPD paradigm that uses evolving contexts as in-training supervision to incorporate contextual signals derived from task feedback. Flux-OPD treats differences between context-conditioned and context-free teachers as contextual difference signals, injects them as contextual corrections into the context-free teacher, and weights the correction strength using the conflict term. Experiments on open-ended tasks show that Flux-OPD achieves the best reported aggregate results among the evaluated OPD paradigms, highlighting the potential of combining teacher supervision with evolving contexts as in-training supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Context Distillation, Open-Ended

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3yqQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Flux-OPD, an iterative on-policy distillation framework that uses evolving contexts extracted from student trajectories as in-training supervision. It anchors the distillation target to a context-free teacher, introduces a correction derived from multiple context-conditioned teachers, and adjusts its strength using a reverse-KL-derived conflict score. Experiments on prompt optimization and medical question answering show improved aggregate performance over the evaluated OPD variants.

Critical strengths
  • The evaluation covers distinct open-ended tasks, multiple student scales and downstream generators, together with relatively comprehensive ablations of the main components.
  • The paper presents an interesting empirical finding: contextual supervision is more effective when applied as a controlled correction to a stable teacher and attenuated when contextual teachers disagree.
  • The paper is clearly written, with well-organized and informative figures and tables.
Critical weaknesses
  • The evaluation is limited to Qwen-family students of at most 4B, leaving scalability and broader generalizability unclear.
  • The context-extraction procedure relies on a very small set of randomly sampled trajectories, raising concerns about the representativeness and robustness of the extracted supervision.
  • The additional computational cost appears substantial relative to the reported gains, while the resulting cost–performance trade-off is not sufficiently evaluated.
Questions for authors
  • Given the greater degradation at 4B than at 2B, can the authors evaluate Flux-OPD on students of 8B or larger?
  • With only eight randomly sampled trajectories and no explicit selection for failures, is there sufficient error coverage to derive representative and comprehensive guidance? Could context extraction be improved by jointly synthesizing insights from all eight trajectories, with deduplication and conflict resolution, rather than accumulating them sequentially?
  • How sensitive are the results to extraction-set size, update frequency, and trajectory order?
  • How do the baselines perform when given the same additional compute as Flux-OPD? Do the modest gains justify the added cost, and could alternative configurations offer a better performance–compute trade-off?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3yqQ

We appreciate your constructive feedback and positive assessment of our empirical findings, comprehensive ablations, and presentation. We address the concerns below.


W1/Q1. Student Scalability

Thank you for your valuable suggestions. We evaluate an 8B student with a 32B teacher on VBench across two video generators:

MethodVACECogVideoXAverage ↑Δ over Student
Teacher82.6277.9980.31−0.24
Student81.9979.1080.55–
+ OPD82.1478.6080.37−0.18
+ Flux-OPD82.7079.1580.93+0.38

Flux-OPD improves the 8B student on both generators and outperforms OPD, showing scalability to larger students.


W2/Q2/Q3. Context Extraction

Thank you for your thoughtful question. Our core contribution is how to use evolving contexts for distillation instead of how to extract them. Our extraction procedure largely follows prior works (OPCD and OEL), with minimal task-specific engineering. We provide additional analyses using a 2B student on VBench with CogVideoX.

1. Trajectory Sampling and Extraction-Set Size

We clarify that 8 fresh student trajectories are sampled at each context update instead of reusing the same trajectories throughout training. We also evaluate extraction-set size:

Extraction-Set SizeVBench ↑Gain
Student76.48–
476.68+0.20
8 (default)77.10+0.62
1676.91+0.43

All extraction-set sizes improve over the initial student, while sizes beyond 8 provide no further benefit. This supports our default choice as a favorable performance–cost trade-off.

2. Sequential Accumulation vs. Joint Synthesis

We clarify that sequential accumulation is not simple concatenation. Our extraction prompt (Fig.6) explicitly instructs the teacher to avoid redundant or conflicting experience items when incorporating new items.

Joint synthesis has a limitation: processing all trajectories in a single input is challenging, while sequential accumulation scales better with the number of trajectories. This is particularly useful for multimodal tasks such as video prompt optimization, where jointly processing eight video trajectories is difficult.

3. Sensitivity to Context Update Frequency

We evaluate different context update intervals:

Update Interval (Steps)# UpdatesVBench ↑Gain
Student–76.48–
600277.08+0.60
400577.21+0.73
300 (default)677.10+0.62
200977.15+0.67

Flux-OPD improves over the initial student at all update frequencies. Updating every 600 steps still yields +0.60, suggesting frequent updates are unnecessary for efficient training.

4. Sensitivity to Trajectory Order

Starting from the same initial student, we extract contexts from an identical trajectory set in three orders, then train each variant for 300 steps.

Trajectory OrderVBench ↑Gain
Student76.48–
Original76.84+0.36
Reversed76.76+0.28
Shuffled76.89+0.41

All three orders yield improvements, suggesting limited sensitivity to trajectory ordering in the evaluated initial-context setting.

Overall, these results support the effectiveness and practical robustness of our simple extraction procedure, without requiring larger extraction sets, frequent updates, or carefully optimized trajectory orders.


W3/Q4. Computational Cost and Trade-Off

Thank you for your valuable suggestions. We provide additional cost analysis and compute-matched comparisons to evaluate the performance–compute trade-off.

1. Training and Context Extraction Overhead

Although Flux-OPD requires 4 teacher forward passes per step, its training GPU-hour overhead is considerably smaller than 4×:

TaskOPDFlux-OPD
Video Prompt Optimization (2B)~8~11
Medical QA~47~56

Context extraction additionally adds ~14 and ~9 GPU-hours, respectively. The higher extraction cost for video prompt optimization mainly stems from video generation and understanding.

2. Compute-Matched OPD Comparison

We evaluate OPD under larger compute budgets on VBench (2B, CogVideoX):

MethodTeacher ForwardsTraining GPU-hoursVBench ↑Gain
Student––76.48–
OPD1×~876.45−0.03
OPD (Training-Matched)~2×~1176.33−0.15
OPD (4× Forwards)4×~2376.79+0.31
Flux-OPD4×~1177.10+0.62

Flux-OPD outperforms OPD at matched training GPU-hours. Even including context extraction (~25 total GPU-hours), Flux-OPD surpasses OPD with 4× teacher forwards (~23 GPU-hours).

Thus, Flux-OPD's gains are not solely due to additional computation, but also benefit from more effective use of contexts.

3. More Efficient Configurations

Our additional analyses indicate that less frequent context updates can reduce extraction overhead, while smaller context histories (Appendix C.4) can reduce computation while maintaining competitive performance, enabling flexible trade-offs.

Thanks for the additional experiments. They address several of my concerns. I still have two reservations: the 8B results only cover VBench, without VideoVerse or the number of independent runs; and the compute comparison excludes the additional context extraction cost, which makes the overall cost relatively high for modest gains. Overall, the rebuttal is helpful, but these concerns still prevent me from increasing my score.

Thank you for your follow-up. We would like to further clarify two points:

1. 8B Results

We additionally evaluate the 8B model on VideoVerse, with the following results:

MethodVACECogVideoX
Student11231125
+ OPD11021118
+ Flux-OPD11321129

The 8B experiments follow the same setting as Table 1, with one independent run.

2. Cost and Performance

As reported in our Compute-Matched OPD Comparison, even including context extraction (~25 GPU-hours in total), Flux-OPD outperforms OPD with 4× teacher forwards (~23 GPU-hours). This demonstrates that the improvements cannot be attributed solely to additional teacher computation.

Moreover, VBench is a relatively saturated benchmark where absolute score differences are typically small. Our method achieves performance competitive with or even surpassing the teacher in most settings, while other baselines often exhibit degradation. In comparison, VideoVerse shows larger absolute gains (~10–25 points over the OPD baseline), further demonstrating the effectiveness of our method.

Reviewer_xFaCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Flux-OPD, an on-policy distillation framework that incorporates evolving contextual supervision into language model training for open-ended tasks. The authors identify two challenges in directly distilling from context-conditioned teachers: unstable targets as contexts evolve and conflicting supervision across different contexts. They derive a reverse-KL decomposition showing that context distillation implicitly targets the normalized geometric mean of context-conditioned teacher distributions, with an additional term measuring distributional disagreement.

Based on this observation, Flux-OPD introduces contextual correction, which interpolates between a context-free teacher and the geometric-mean context-conditioned teacher in log-probability space, and contextual weighting, which adjusts the interpolation strength according to teacher disagreement. Contexts are periodically regenerated from student trajectories and task-specific feedback during a single training run.

The method is evaluated on video-generation prompt optimization and medical question answering using Qwen3 and Qwen3-VL student-teacher configurations. Results on VBench, VideoVerse, and HealthBench show consistent improvements over the evaluated OPD variants. Ablations investigate evolving contexts, contextual correction, and conflict-aware weighting, while additional analyses examine training stability and performance across repeated runs.

Critical strengths
  1. A clear and practically motivated approach to stabilizing evolving contextual supervision. The paper identifies meaningful limitations of directly applying evolving contexts to on-policy distillation. The proposed combination of a context-free teacher anchor, contextual corrections, and conflict-aware weighting is simple, interpretable, and supported by a clear mathematical formulation. This provides a useful extension of existing OPD and OPCD approaches.

  2. Consistent improvements across different tasks with informative ablations. Flux-OPD improves performance on video-generation prompt optimization and medical question answering across multiple student-teacher configurations. Table 3 separates the contributions of evolving contexts, contextual correction, and adaptive weighting. The training-stability analysis and repeated-run experiments provide additional evidence that the method can alleviate instability associated with directly using evolving contexts.

Critical weaknesses
  1. The performance gains are not established under comparable computational budgets. Appendix C.3 explicitly states that Flux-OPD requires four teacher forward passes per optimization step, compared with one for vanilla OPD, in addition to periodic context extraction. However, the main comparisons match training steps rather than total teacher computation. This distinction is important because the improvements over OPD are relatively modest: the average VBench gains are approximately 0.45 and 0.54 points for the two student configurations, and the HealthBench improvement is 0.57 points without rubrics. Figure 4 reports only three independent runs, without establishing the statistical significance of these differences. Consequently, it remains unclear whether the benefits arise specifically from the proposed contextual mechanisms or could be achieved by allocating comparable computation to simpler OPD variants. This substantially limits the strength of the empirical contribution.

  2. The theoretical characterization does not fully justify the proposed conflict-aware correction rule. Proposition 1 provides a valid decomposition of the reverse-KL objective into a geometric-mean distillation target and a disagreement term. However, this decomposition is an algebraic identity under fixed histories and does not establish that exponentially reducing the contextual correction according to disagreement is optimal or necessarily beneficial. The proposed target is effectively a log-space interpolation between the context-free teacher and the geometric-mean contextual teacher, while the mapping from conflict to interpolation strength remains heuristic. Although Table 3 supports adaptive weighting, Table 4 shows considerable sensitivity to the scaling factor, and the advantage over JSD-based weighting in Appendix Table 7 is relatively small. Stronger evidence connecting the conflict measure to the reliability of contextual corrections would better substantiate the claimed methodological contribution.

Questions for authors
  1. Computational fairness: Since Flux-OPD uses four teacher forward passes per step while vanilla OPD uses one, how should the improvements be interpreted under a fixed total training-compute budget? Are any compute-normalized comparisons or total GPU-hour measurements available? In particular, Appendix Table 6 suggests that reducing the number of contexts can substantially lower computation with limited performance degradation.

  2. Hyperparameter selection: How were the values of the conflict scaling factor β selected? Appendix Table 5 uses different values across downstream video generators and tasks, while Table 4 shows that performance can change noticeably with β. Were these values determined using separate validation data?

  3. Interpretation of conflict-aware weighting: The reverse-KL decomposition establishes that the conflict term measures disagreement but contributes no direct gradient under fixed histories. What evidence indicates that this disagreement reliably predicts whether contextual corrections are beneficial? Could the authors clarify why the exponential mapping is preferable to other weighting functions beyond the reported comparison with entropy and JSD?

1 comment
AuthorsOpenReview ↗

Response to Reviewer xFaC

We appreciate your constructive feedback and positive assessment of our clear motivation, methodological contribution, and informative ablations. We address your concerns below.


W1/Q1. Computational Fairness

Thank you for your valuable suggestions. We provide additional computational cost analysis and compute-matched comparisons.

1. Training and Context Extraction Overhead

Although Flux-OPD requires 4 teacher forward passes per step, its training GPU-hour overhead is considerably smaller than 4×:

TaskOPDFlux-OPD
Video Prompt Optimization (2B)~8~11
Medical QA~47~56

Context extraction additionally adds ~14 and ~9 GPU-hours, respectively. The higher extraction cost for video prompt optimization mainly stems from video generation and understanding.

2. Compute-Matched OPD Comparison

We evaluate OPD under larger compute budgets on VBench (2B, CogVideoX):

MethodTeacher ForwardsTraining GPU-hoursVBench ↑Gain
Student––76.48–
OPD1×~876.45−0.03
OPD (Training-Matched)~2×~1176.33−0.15
OPD (4× Forwards)4×~2376.79+0.31
Flux-OPD4×~1177.10+0.62

Flux-OPD outperforms OPD at matched training GPU-hours. Even including context extraction (~25 total GPU-hours), Flux-OPD surpasses OPD with 4× teacher forwards (~23 GPU-hours).

Thus, Flux-OPD's gains are not solely due to additional computation, but also benefit from more effective use of evolving contexts.


W2/Q2. Hyperparameter Selection

Thank you for your thoughtful question. All hyperparameters were selected on a held-out validation set, comprising 1% of the data reserved before training, without using the final evaluation benchmarks.

β controls the correction strength, with larger β values more strongly suppressing conflicting corrections. Different tasks may require different β values due to their distinct characteristics, which is intuitive.

We evaluated β ∈ {8, 10, 30, 50}, taking β = 30 as the reference for its moderate correction strength, balancing contextual supervision and target stability. For each setting, Gemini 3.8 Flash performed pairwise comparisons against the reference on the same validation samples, assessing generated video quality for prompt optimization and answer quality for Medical QA. Each comparison was conducted in both A/B orders to mitigate position bias. Wins, ties, and losses were scored as 1, 0.5, and 0, respectively. The β with the highest average preference score was selected, with the reference assigned a nominal score of 50%.

Representative validation results (preference scores, %) are shown below:

Settingβ=8β=10β=30 (Ref.)β=50Selected β
Prompt Optimization (4B, VACE)46.7445.6550.0045.0030
Prompt Optimization (4B, CogVideoX)54.7846.9650.0051.968
Medical QA47.9147.4950.0051.4050

W2/Q3. Contextual Weighting

1. Effectiveness of the Conflict Term

Thank you for your thoughtful question. We clarify that Proposition 1 characterizes contextual teacher disagreement, without establishing the optimality of our weighting rule. Our motivation is to downweight conflicting contextual corrections for training stability. Beyond the task performance improvements in Table 3, we provide additional analyses:

  • Training Stability. We add training loss and gradient norm curves for the ablation without contextual weighting (Appendix E). The results show that conflict-aware weighting leads to more stable training dynamics.

  • Teacher Inference. We evaluate contextual corrections on VBench using two context pools with the 8B teacher and CogVideoX. Averaged across both pools, conflict-aware weighting achieves 78.02, compared with 77.71 for full correction (without weighting) and 77.68 for the context-free teacher.

Together with Table 3, these findings provide further empirical support for conflict-aware weighting, while we do not claim that conflict directly determines the optimal correction strength.

2. Choice of Exponential Mapping

We acknowledge that the exponential mapping is a motivated heuristic without theoretical optimality guarantees. We choose it for two practical reasons: (1) it provides smooth, bounded, and monotonically decreasing correction strengths; (2) it strongly downweights high-conflict corrections without completely suppressing contextual supervision, unlike truncated linear weighting, which may zero out corrections abruptly.

Reviewer_84LkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Flux-OPD, an on-policy distillation method that leverages evolving contexts from student trajectories. The key idea includes contextual correction and conflict-aware weighting, motivated by a reverse-KL decomposition analysis. Experiments on video prompt optimization and medical question answering show improvements over existing baselines.

Critical strengths
  • Theoretical analysis: The paper addresses the evolving context challenge in on-policy distillation through an analysis of the objective function. The reverse-KL decomposition provides insights into the effects of conflicting term, which inspires the design of the proposed method.

  • Good empirical results: The experiments demonstrate improvements over existing baselines across the two evaluated tasks. The ablation studies further support the effectiveness of the proposed components.

Critical weaknesses
  • The paper is not very well written. The central concept of "context" is not explained formally, with few concrete examples to help readers understand what it represents and how it is constructed. The mathematical presentation is also not easy to follow. For example, in Section 3, what is the difference between p_\theta and \pi_theta, what is q_0 and \pi_T. Readers can probably guess by conventions, but it's better the properly define to be self-contained.

  • Reverse KL can be sensitive to low-probability teacher outputs, but the paper provides limited discussion of it, and, maybe contradictory, it argues it helps stablizing the training.

  • What are the selection criteria of the tasks for experiments? some long context tasks, such as deep research might be a better fit? as it's also a popular tasks that people use rubrics to serve as reward surrogate and it needs iterative refinement, e.g. DR Tulu: Reinforcement learning with evolving rubrics for deep research, 2025 and RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards, 2026

Questions for authors

Please see the critical weaknesses above. In particular, the clarification on the definitions, numerical stability of reverse KL, and the rationale behind the experimental task selection would be appreciated.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 84Lk

We appreciate your constructive feedback and positive assessment of our theoretical analysis and empirical results. We address your concerns regarding context and notation, reverse KL, and task selection below.


W1. Clarification of Context and Notation

Thank you for your valuable suggestions. We agree that the definitions of contexts and mathematical notation could be clearer.

The original manuscript describes context construction in Sec. 4.1 and Fig. 3, with concrete examples in Figs. 11–12 (Appendix B). To improve clarity, we have explicitly defined contexts as collections of task-general experience items in the Introduction and clarified the distinction between student/teacher policies and their corresponding next-token distributions in Sec. 3.1. We have also added explicit references to the context examples in Sec. 4.1.

All revisions are highlighted in blue in the updated manuscript.


W2. Clarification of Reverse KL

  1. Thank you for your question. We clarify that we do not claim reverse KL itself stabilizes training. Our focus is on the instability introduced by evolving contexts (Fig. 1).

  2. Thank you for your valuable suggestions. We agree that further discussion of reverse KL would be beneficial.

    • First, we acknowledge its sensitivity to low-probability teacher outputs, an inherent property of reverse-KL-based distillation that does not undermine our core contribution.
    • Second, Flux-OPD does not amplify low-probability penalties beyond those of its reference teachers. Our corrected teacher is a normalized geometric interpolation between the context-free teacher and the geometric-mean context-conditioned teacher, ensuring that no token receives a probability lower than both references.
    • Third, our implementation mitigates numerical instability. We use log-softmax to avoid numerical underflow, compute the distillation loss over renormalized top-64 target tokens, and calculate contextual conflict using full-vocabulary log-probabilities (Appendix C.1). While these choices do not eliminate the inherent sensitivity of reverse KL, they mitigate numerical issues in practice, consistent with the stable training losses and gradient norms in Fig. 5.

W3. Task Selection

Thank you for your thoughtful question. Our task selection is primarily motivated by the nature of supervision signals, not context length. Flux-OPD is particularly suitable for tasks where task preferences are difficult to capture through evaluation rubrics, but can be learned from dense feedback and used to directly guide model behavior.

Video prompt optimization exemplifies this setting, as prompt quality depends on downstream video generation and generator-specific preferences that are difficult to assess from text alone. Medical QA provides a complementary setting to examine whether Flux-OPD also generalizes to more general end-to-end tasks.

We agree that deep research is a promising extension. However, context length is not our defining criterion. We consider deep research a valuable direction for future work on evolving contexts.

Thanks for the response. It addressed some of questions, while I share similar options with Reviewer 3yqQ. Thanks.

Thank you for your follow-up. We have further addressed these concerns in our response to Reviewer 3yqQ, including 8B student scalability, sensitivity to context extraction settings, and compute-matched comparisons. We hope these additional results help address your remaining concerns.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.