acceptodds
Under review as a conference paper at ICLR 2027

CV-OPSD: Self-Evolving Long-Context Reasoners via Zero-Annotation Clean-View On-Policy Self-Distillation

Abstract

Models often answer correctly when the relevant clip or span is presented in isolation, yet fail when the same evidence is embedded in a long input. We study this gap as a problem of evidence localization under bounded computation. Our method, clean-view on-policy self-distillation (CV-OPSD), enables a long-context reasoner to self-evolve by pairing a full-input student with a live, stop-gradient teacher instantiated from the same model. The student generates the rollout; the teacher evaluates the same prefixes from a privileged view that reveals where the evidence lies without providing an external answer. As the shared model improves, its clean-view posterior supplies an evolving learning signal without requiring a stronger teacher. In the fully self-generated setting, the examples and their evidence views are produced without human annotation. The view is adapted to each modality: a question-relevant temporal interval for video or marked supporting spans in the identical document. We formalize why such views can increase information accessible to a bounded predictor even when they do not add answer information. Across 9B and 4B backbones, CV-OPSD improves performance across video and long-text evaluations, with gains of up to 4.8 and 5.0 points, respectively. Controlled comparisons with hard SFT, same-view voting, and alternative teacher inputs show that the useful signal comes from transferring an evidence-conditioned posterior on states visited by the full-input policy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long context, On-policy self distillation, self-evolving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_pNQyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CV-OPSD trains a long-context student using an evidence-conditioned teacher with the same current weights and a stop-gradient branch. The student generates prefixes from the full input; the teacher scores those prefixes using a relevant video interval or the full text with supporting facts marked. A forward KL objective transfers the teacher's distributions without answer labels in the loss. Evaluation spans two backbones, long-video and long-text benchmarks, with both annotated and self-produced video intervals. The reported gains and localization diagnostics support the usefulness of evidence-conditioned training, while leaving the scope of annotation-free results and the contributions of individual design choices incompletely established.

Critical strengths
  1. The text teacher preserves the document and distractors while marking supporting facts, making the evidence-location intervention particularly informative. Inference needs neither marks nor a teacher. Table 1 reports improvements across both backbones: BABILong thinking-mode results rise by 8.8 and 8.9 points, alongside positive AA-LCR and LongBench v2 results. The video experiments include self-produced questions and intervals, extending the evidence beyond the annotated-interval setting. This breadth makes the approach worth studying despite the remaining validation gaps.

  2. Fig. 5 reports a Charades-STA grounding-score increase from 51.93 to 54.48. The 4B model's LVBench advantage decreases from 1.50 points with full videos to 0.32 when both models receive the correct interval. These observations support improved localization as an explanation and provide useful information beyond aggregate answer accuracy. Because supplying an interval also changes the input, they do not quantify localization's causal share of the gain.

Critical weaknesses
  1. Table 1(a)'s 4.8-point LVBench improvement, from 43.8 to 48.6, uses 6,015 examples constructed from human-annotated intervals. The 579 self-produced examples improve the same 9B setting to 46.2, a useful 2.4-point gain; the 4B gain is 1.5 points. The abstract should associate the maximum result with its annotated-interval condition and distinguish absence of answer labels from absence of human localization data. The generation and filtering procedures, and how often a selected interval omits necessary information, also matter: the loss contains no correctness check on the evidence-conditioned teacher. This limits how broadly the current results support the advertised training setting.

  2. Fig. 3's online teacher reaches 48.6 on LVBench versus 43.2 for a frozen teacher, but Fig. 4's hard-target SFT comparison changes both target distributions and training prefixes. Its BABILong comparison, 57.30 versus 64.90, therefore cannot isolate soft targets from prefix alignment. The referenced Appendix A is absent from the supplied PDF, leaving the top-k KL implementation, sampling/context settings, and other reproduction details unspecified. Independent training-run variation is unreported; the smaller Video-MME and LongBench v2 gains need that context, while LVBench and AA-LCR additionally require judge prompts and assessment of judging variability. These fixable gaps, together with the claim-scope issue above, are why I currently lean toward weak rejection despite the positive results. For literature context, Clue-OPSD and Video-OPSD also teach student prefixes from evidence views, with different teacher and divergence choices (K. Wang et al. 2026; Z. Wang et al. 2026); OPSDL uses a relevant short-text teacher (Zhang et al. 2026). These preprints' omission or lack of direct comparison is not a rejection reason. The manuscript already distinguishes the solution-conditioned Self-Distilled Reasoner (Zhao et al. 2026).

References

Wang, Kaishen, Dongdi Zhao, Yijun Liang, Dingqiang Ye, Ruibo Chen, Heng Huang, and Di Fu. 2026. “Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding.” arXiv:2608.25356. https://arxiv.org/abs/2608.25356.

Wang, Ziyue, Shiqi Huang, Weiwen Xu, Bihan Wen, and Xudong Jiang. 2026. “Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models.” arXiv:2608.27065. https://arxiv.org/abs/2608.27065.

Zhang, Xinsen, Zhenkai Ding, Tianjun Pan, Run Yang, Chun Kang, Xue Xiong, and Jingnan Gu. 2026. “OPSDL: On-Policy Self-Distillation for Long-Context Language Models.” arXiv:2604.17535. https://arxiv.org/abs/2604.17535.

Zhao, Siyan, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. 2026. “Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.” arXiv:2601.18734. https://arxiv.org/abs/2601.18734.

Questions for authors
  1. What generation and filtering rules produced the 579 self-produced question–interval pairs, and what evidence-quality checks establish that the intervals contain the information needed to answer? This would affect my assessment of the annotation-free setting.

  2. Can you provide the referenced Appendix A, including the top-k KL support and tail treatment, video sampling/context settings, LVBench and AA-LCR judge prompts, and training cost? These details are necessary to assess reproducibility and the reliability of the reported gains.

  3. Can you provide a control that holds the examples, evidence view, and student prefixes fixed while changing soft versus hard targets, together with training-run and evaluation variability for the smaller gains? This would help determine which design choices are supported by the current improvements.

5 comments

Weakness 1 — Annotation scope and evidence quality

Table 1(a)'s 4.8-point LVBench improvement, from 43.8 to 48.6, uses 6,015 examples constructed from human-annotated intervals. The 579 self-produced examples improve the same 9B setting to 46.2, a useful 2.4-point gain; the 4B gain is 1.5 points. The abstract should associate the maximum result with its annotated-interval condition and distinguish absence of answer labels from absence of human localization data. The generation and filtering procedures, and how often a selected interval omits necessary information, also matter: the loss contains no correctness check on the evidence-conditioned teacher. This limits how broadly the current results support the advertised training setting.

Abstract and introduction. The abstract now reports the fully self-produced result: 9B LVBench improves from 43.8 to 46.2, a 2.4-point gain from 579 training examples, and AA-LCR improves by 5.0 points. Section 5 reports the same construction.

Generation and filtering. We first segment each source video into candidate clips, then prompt the model to write one question answerable from a clip, its short answer, and the clip's start and end timestamps. Because the question is written from the clip and the answer is produced from the same clip, the clip contains the evidence needed to answer. Each retained pair carries the clip boundaries as its evidence annotation. A pair is retained when a second pass answers the question correctly from the clip alone, the question is not answerable from the question text alone, and the source video is disjoint from every evaluation video. The round-trip check discards pairs whose clip omits required evidence, so the retained clip supports the paired question by construction. The full-video student still recovers the same evidence without the clip at inference. Section 4, Section 5, and Appendix B give the construction and the prompt.

Evidence quality. The question and the answer are both produced from the same clip, and the round-trip check requires the answer to be recoverable from that clip alone. A clip that omits necessary evidence fails the check and is filtered, and the question-alone condition removes pairs that do not depend on the video. The retained clip therefore carries the evidence path for the paired question, and the full-video student must recover the same evidence without the clip. A manual spot-check of the retained pairs found them to be of good quality. The check was diagnostic only; it did not modify, rewrite, or filter any question.

Residual interval noise. Training distills the teacher's full next-token distribution on student prefixes with a forward KL, not a decoded one-hot answer. If a retained clip is still imperfect, the teacher's posterior is less concentrated and its mass spreads over plausible continuations, so the KL gradient on any single token is smaller. No hard label is imposed, and the student still sees the full video on the same prefix, which keeps the update anchored to the full-view policy. An imperfect clip therefore weakens the signal instead of injecting a wrong answer.

We note that no human annotation enters the self-produced setting: the questions, answers, and evidence intervals are all model-produced. At inference, no selector, marker, or teacher is used, and the deployed model receives the same full-context input as its initialization. The revised abstract leads with this 579-example result.

Weakness 2 — Factor isolation, reproducibility, variability, and related work

Fig. 3's online teacher reaches 48.6 on LVBench versus 43.2 for a frozen teacher, but Fig. 4's hard-target SFT comparison changes both target distributions and training prefixes. Its BABILong comparison, 57.30 versus 64.90, therefore cannot isolate soft targets from prefix alignment. The referenced Appendix A is absent from the supplied PDF, leaving the top-k KL implementation, sampling/context settings, and other reproduction details unspecified. Independent training-run variation is unreported; the smaller Video-MME and LongBench v2 gains need that context, while LVBench and AA-LCR additionally require judge prompts and assessment of judging variability. These fixable gaps, together with the claim-scope issue above, are why I currently lean toward weak rejection despite the positive results. For literature context, Clue-OPSD and Video-OPSD also teach student prefixes from evidence views, with different teacher and divergence choices (K. Wang et al. 2026; Z. Wang et al. 2026); OPSDL uses a relevant short-text teacher (Zhang et al. 2026). These preprints' omission or lack of direct comparison is not a rejection reason. The manuscript already distinguishes the solution-conditioned Self-Distilled Reasoner (Zhao et al. 2026).

Target form versus prefix source. Table 2 crosses prefix source with target form at matched data. The two on-policy rows use the same examples, the same evidence views, and the same student-generated prefixes, and differ only in the target. On BABILong, soft targets reach 64.90 and hard targets reach 63.05. The soft target is 1.85 points above the hard target and 7.6 points above off-policy SFT at 57.30. The soft target preserves the teacher's full distribution and gives the strongest result; prefix alignment and target form both contribute. Section 6.2 reports this control.

Appendix and implementation. The complete appendix is now included in the revised PDF. Appendix A covers implementation, data, evaluation protocols, result breakdowns, output dynamics, and formal supplements. Appendix B gives every prompt. Appendix C gives the additional controls and repeated evaluations. The implementation specifies top-100 forward KL with the residual tail retained as a single aggregate term, the student and teacher sampling protocol, the token limits, and eight H100 GPUs for 20–24 optimization steps with no separate teacher.

Training-run variation. Table 6 reports the requested measurements. On 4B Video-MME, our model averages 64.79 against a 63.63 base across three runs, and the run-to-run standard deviation is 0.24. Three independent LongBench v2 retrains give 1.10. Repeated LVBench judging gives standard deviations of 0.42 and 0.16, with 98.67% unanimous item-level decisions; AA-LCR gives 0.00 on 100 items with 97.0% unanimity. The BABILong effects of +8.85 and +8.95 exceed these estimates by an order of magnitude. Appendix C gives the tables.

Related work. Table 3 compares CV-OPSD with Clue-OPSD, Video-OPSD, and OPSDL along modalities, evidence view, training signal, divergence, and whether the text view is length-preserving. CV-OPSD is the only method in the table that covers video and text, obtains its training signal from a self-produced pool, and keeps the full document and all distractors in the text view. OPSDL's teacher receives a short extracted context, so teacher and student differ in input length; CV-OPSD's text teacher sees the identical document. The three methods are cited at the point where the shared evidence-view principle is introduced.

Question 1 — How were the 579 pairs generated and filtered?

What generation and filtering rules produced the 579 self-produced question–interval pairs, and what evidence-quality checks establish that the intervals contain the information needed to answer? This would affect my assessment of the annotation-free setting.

The 579 pairs are generated in three steps. First, each source video is segmented into candidate clips. Second, a question and its short answer are generated from one clip, together with the clip's start and end timestamps. Third, a round-trip check re-asks the question from the clip alone; only pairs answered correctly from the clip are retained. A pair is also rejected when the question is answerable from the question text alone or when the source video appears in an evaluation set. The retained pool contains 579 question–answer–interval triples. Appendix B gives the generation prompt and the retention rule.

Evidence quality follows from the construction. Each question and its answer are produced from the same clip, and the round-trip check requires the answer to be recoverable from that clip alone. A clip that omits necessary evidence fails the check and is filtered, and the question-alone condition removes pairs that do not depend on the video. The retained clip supports the paired question by construction, and the full-video student must recover the same evidence without the clip. A manual spot-check of the retained pairs found them to be of good quality. The check was diagnostic only; it did not modify, rewrite, or filter any question. The self-produced arm reaches 46.2 on 9B and 46.2 on 4B, which shows that the construction supplies usable supervision on both backbones. Section 4, Section 5, and Appendix B report the construction and the measured results.

Residual interval noise is handled by the soft objective. Training distills the teacher's full next-token distribution on student prefixes with a forward KL, not a decoded one-hot answer. When a clip is imperfect, the teacher's posterior is less concentrated and its mass spreads over plausible continuations, so the KL gradient on any single token is smaller. No hard label is imposed, and the student still sees the full video on the same prefix, which keeps the update anchored to the full-view policy. An imperfect clip therefore weakens the signal instead of injecting a wrong answer.

Question 2 — Appendix A, KL implementation, prompts, and cost

Can you provide the referenced Appendix A, including the top-k KL support and tail treatment, video sampling/context settings, LVBench and AA-LCR judge prompts, and training cost? These details are necessary to assess reproducibility and the reliability of the reported gains.

The full appendix is now part of the revised PDF.

Top-k KL support and tail. The forward KL is computed over the teacher's 100 highest-probability tokens. The residual tail mass is retained as a single aggregate term, so the objective sums over the full vocabulary rather than renormalizing over the support. Appendix A gives the implementation.

Sampling and context. Appendix A gives the student and teacher sampling protocol, the video and text token limits, and the LVBench and Video-MME decoding settings. The student receives the full video and the teacher receives the selected interval in every run.

Judge prompts. Appendix B gives the exact LVBench and AA-LCR judge prompts. All checkpoints and all repeated judgments use the same prompt with gpt-4.1-mini-2025-04-14.

Training cost. Each run uses one node of eight H100 GPUs, batch size 16, four rollouts per prompt, and 20–24 optimization steps. No separate teacher is trained. Appendix A and Appendix C report the cost and the repeated-run measurements.

Reviewer_Ewa3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CV-OPSD trains a long-context student model using a shared-weight teacher that has privileged access to key evidence. The student processes the full raw input, while the teacher observes either an isolated relevant video interval or a document with ground-truth spans highlighted. A forward KL divergence loss transfers the teacher’s output distribution over student-generated prefixes. The authors evaluate both human-annotated and self-generated evidence views across video and long-text benchmarks on two Qwen backbones. The analysis includes comparisons against frozen teachers, standard SFT, voting, and answer-conditioned distillation, alongside dedicated evidence-localization tests.

Critical strengths
  1. The text-view design provides a clean baseline comparison: the teacher receives the full document and distractors with only the supporting spans tagged. This rules out input length truncation as a confounder and isolates the specific benefit of guided evidence access.

  2. The localization experiments offer mechanistic support: the full-video margin shrinks from 1.50 points to 0.32 once both models are conditioned on the correct ground-truth interval, coupled with gains on Charades-STA. These diagnostics support the author's localization hypothesis.

Critical weaknesses

Ambiguous evidence for teacher "self-evolution." While Figure 3(a) confirms that updating the teacher helps compared to keeping it frozen, Figure 3(b) leaves key details open: the screening set is undefined, and it is unclear whether the plot tracks the privileged teacher or the student. As a result, it does not directly demonstrate the teacher improving dynamically over the course of training. In addition, Section 6.2 discusses token-level KL concentration qualitatively without providing the underlying quantitative curves or data. These gaps weaken the mechanistic claims.

Questions for authors
  1. What exactly does “screening accuracy” in Figure 3(b) track? Which branch (privileged teacher or full-input student) and dataset were evaluated, and was this set held static during training? If it tracks student performance, could you share the corresponding metrics for the privileged teacher?

  2. Could you provide the exact teacher prompts used in the evidence- vs. answer-conditioned comparisons, as well as the quantitative token-level KL distribution described in Section 6.2? Specifically, how were answer-related tokens operationalized, and how stable is this trend across tasks?

3 comments

Weakness 1 — Teacher self-evolution and quantitative token evidence

Ambiguous evidence for teacher "self-evolution." While Figure 3(a) confirms that updating the teacher helps compared to keeping it frozen, Figure 3(b) leaves key details open: the screening set is undefined, and it is unclear whether the plot tracks the privileged teacher or the student. As a result, it does not directly demonstrate the teacher improving dynamically over the course of training. In addition, Section 6.2 discusses token-level KL concentration qualitatively without providing the underlying quantitative curves or data. These gaps weaken the mechanistic claims.

Figure 3(b) in the original submission tracked the student, so it did not directly show teacher improvement. The revision separates the two branches: Figure 3(a) tracks the privileged teacher on a fixed held-out set, and Figure 3(b) reports the student. The teacher score rises from 57.89 at initialization to 61.11 at step 20.

A direct probe on the same held-out set gives a 10.78-point endpoint gain. The probe uses the teacher branch with the privileged view, and a fixed judge scores the answer under two criteria, with the conservative intersection reported. The log-probability diagnostic tracks the teacher's probability of the reference answer and improves in all five adjacent checkpoint intervals, with positive slopes on 93.0% of items.

Figure 5 adds the quantitative token-level distribution: the top 5% of tokens carry 68.0% and 70.9% of the evidence- and answer-conditioned signal on video, and 62.4% and 64.1% on text. The answer-fed view places up to 3.1 percentage points more mass on the highest-gap tokens, and Appendix A gives the per-token cases.

Question 1 — What does screening accuracy measure?

What exactly does “screening accuracy” in Figure 3(b) track? Which branch (privileged teacher or full-input student) and dataset were evaluated, and was this set held static during training? If it tracks student performance, could you share the corresponding metrics for the privileged teacher?

Figure 3(a) tracks the privileged teacher, not the student. It is evaluated on a fixed held-out screening set drawn from the training distribution, and the same items are used at every checkpoint. The curve measures teacher improvement rather than a changing evaluation set.

The student metrics are in Figure 3(b), where the live-teacher student reaches 48.6 on LVBench, 5.4 points above the frozen-teacher student. The corresponding teacher metrics are the step-level curve from 57.89 to 61.11, the direct probe on the same held-out set, and the log-probability trend across checkpoints. Section 6.1 reports the student and teacher branches side by side.

Question 2 — Exact prompts, token definition, and stability

Could you provide the exact teacher prompts used in the evidence- vs. answer-conditioned comparisons, as well as the quantitative token-level KL distribution described in Section 6.2? Specifically, how were answer-related tokens operationalized, and how stable is this trend across tasks?

Exact prompts. Appendix B gives the complete templates for the evidence-conditioned video teacher, the answer-conditioned video teacher, the evidence-conditioned text teacher, the answer-conditioned text teacher, the self-produced video data, and both judges. The video teacher receives the question-relevant interval. The text teacher receives the identical document with every supporting span enclosed in [[KEY EVIDENCE]] and [[/KEY EVIDENCE]]. The answer-conditioned teacher receives the model's own answer appended to the full input. The student always receives the deployment input with no markers.

Answer-related tokens. A decoded token is counted as answer-related when its normalized word overlaps the reference answer span in the response. In the archived text aggregate this yields 61 answer tokens, whose mean positive gaps are 0.2827 against 0.1170 for other tokens under evidence marking, and 0.5158 against 0.1504 under answer conditioning. The ratios are 2.42 and 3.43. Section 6.2 and Appendix A report the definition.

Quantitative distribution and stability. The statistic is the positive teacher-minus-student log-probability gap on each observed token. The top-5% shares are 68.0 and 70.9 on video and 62.4 and 64.1 on text, and Figure 5 gives the cumulative gap between the two views. The pattern holds across both modalities: the same small fraction of tokens carries most of the signal in video and text, and the answer-fed view is higher in both. Across the 30 video examples, the median top-5% share is 64.6%, with an interquartile range of 53.5% to 73.5%, so the concentration is present in every example while its exact size varies by question. The per-token cases in Appendix A span different question types and show the same direction of separation.

As the discussion deadline approaches, we would appreciate it if you could take a moment to review our rebuttal and let us know whether it addresses your concerns. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

Thank you for the responses and additional experiments which address most of my concerns. I decide to raise my score to 3.

Comments on the paper

2 comments

We thank both reviewers for recognizing the paper's main design choices. Both highlighted the length-preserving text view, which keeps the full document and all distractors while marking only the supporting spans and rules out input-length effects. Both also highlighted the localization diagnostics, where Charades-STA improves and the LVBench advantage shrinks once both models receive the correct interval, supporting localization as the mechanism.

Reviewer pNQy additionally recognized the breadth of the evaluation across 9B and 4B backbones and across BABILong, AA-LCR, LongBench v2, and the self-produced video setting, as well as the fact that inference needs neither marks nor a teacher.

The revision keeps these strengths central and addresses the remaining weaknesses without changing the design the reviewers recognized.

We thank both reviewers for their careful reading.

Note on the appendix and revision marking. The appendix waswas provided in the supplementary material and is nownow merged into the revised paper. In the marked PDF, blue denotes changes made in response to Reviewer Ewa3, red denotes changes made in response to Reviewer pNQy, and purple denotes changes that address both.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.