acceptodds
Withdrawn from ICLR 2027

From Decomposition to Integration: Stage-Specialized On-Policy Distillation for Video Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained video diagnosis requires identifying the relevant subject, discovering subtle differences, and deciding which differences constitute a defect. We present a framework that turns these interdependent capabilities into structured supervision and integrates them within one model. A multipass annotation pipeline decomposes reference-conditioned diagnosis into Subject Binding, Multidimensional Comparison, and Issue Filtering, producing compact traces that connect visual evidence to diagnostic decisions. We then introduce Stage-Specialized On-Policy Distillation (Stage OPD): the student generates a complete trajectory, and each specialist provides feedback for its stage under the student's own preceding outputs. Annotated-trace replay supplies complementary supervision for content, stage transitions, and diagnostic decisions. With 2,160 training traces, the resulting 9B student improves both precision and recall on a human-reviewed benchmark, raising issue-detection from 36.68% to 41.88%. The student exceeds a three-call SFT chain in and approaches the 27B reference within 0.31 points while retaining a single 9B model at inference. Overall, Stage OPD organizes specialized supervision through decomposition and integrates the resulting diagnostic capabilities into one model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Multi-stage reasoning, Vision-language models, Video consistency evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oUBCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies subject consistency in generated videos. It breaks diagnosis into subject binding, attribute comparison, and issue filtering. Stage-specific teachers then transfer these skills to a single 9B model. Training uses 2,160 structured traces, including unchanged attributes and changes allowed by the prompt. On a 50-video benchmark, the method improves F1 from 36.68% to 41.88%. This exceeds the three-call fine-tuned pipeline and the other 9B distillation methods evaluated.

Critical strengths

1.The task decomposition is well motivated. The paper distinguishes visible differences from defects that should actually be reported. It also retains unchanged attributes and permitted changes. This supervision fits the task well and could help reduce false alarms. 2.Providing stage-specific feedback along the same student-generated trajectory is a meaningful extension. The distillation methods start from the same student checkpoint. The three-call baseline also reuses the same weights. These controls help assess the value of the integrated training approach.

Critical weaknesses

1.The main results need stronger validation. The benchmark contains only 50 videos. Checkpoints are selected by monitored F1, but it is unclear whether monitoring uses a separate validation set. The gain over Video-OPD is just 0.78 F1 points, with no uncertainty analysis. Also, under the stated protocol, the per-seed scores in Appendix D do not reconcile with the pooled counts in the main table. The authors need to check and explain this discrepancy. 2.The contribution of stage specialization is not yet clear. Table 3 provides some support, but the effects of teacher configuration, additional training, and loss reweighting remain difficult to separate. The paper does not report a continued fine-tuning baseline with the same starting checkpoint, supervision, and a comparable training budget. Without this control, it is hard to tell how much the on-policy stage feedback adds.

Questions for authors

1.Was the final evaluation set also used to select the best model? 2.Could you clarify how the results in Table 2 relate to Appendix D? Do they use the same model and evaluation setup? 3.The gain over Video-OPD is small. Does it hold on a separate set of videos? 4.What happens if you keep the same supervised losses and training budget, but leave out the teacher feedback? 5.How were disagreements resolved during annotation review? Were the automated matching decisions checked by human reviewers?

Reviewer_TBXcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Stage-Specialized On-Policy Distillation for reference-conditioned video diagnosis. The authors invented a 3 stage multi-pass diagnosis pipeline, and then uses large models such as gemini3.5 flash and qwen 3.8 max to generate some reference trajectories. These trajectories were used to train 3 specialist 9B models and 1 student 9B model, then during distillation, the student generates a complete trajectory, and each teacher provides feedback on its assigned stage under the student-generated history. On a self-constructed benchmark of 50 videos, the authors reported an improvement of F1 from 36.68 to 41.88.

Critical strengths
  1. Finding defects in AI generated Videos is an important and timely problem.
  2. The 3-stage pipeline and the OPD method is of potential deployment value.
  3. The authors constructed a new benchmark.
Critical weaknesses

W1 The design of the 3-stage pipeline seems arbitary

The three-stage decomposition is intuitively motivated, but the paper does not directly ablate the choice of stages or the necessity of specializing each stage. Actually, I feel like this part may be interesting.

W2 The improvement is not reliable

The reported 3 seeds is the seeds for decoding from the same checkpoint, and the variance for the SFT decoding is very high. The empirical improvement is ~5.2 pp, however, OPD baseline and SFT-chain baseline already achieve > 80% of it. The self-constructed benchmark has only 50 videos. The codebase and the benchmark are not open-source. The description for how the benchmark was constructed is insufficient.

W3 Limited methodological novelty

(a) Task decomposition and multi-teacher OPD are well-established. The distinctive contribution appears to be assigning different teachers to different stages within one student-generated trajectory. This can be a reasonable extension, but its advantage over simpler alternatives is not directly shown.

(b) The mechanism analysis also remains largely suggestive, lacking behaviour analysis. Decreasing decision NLL on replay targets demonstrates improved fitting of those targets, but does not confirm a better diagnostic generalization. Relative to SFT chain, Stage OPD reduces false positives from 58 to 38 while true positives decrease from 52 to 49. This supports improved reporting selectivity, but does not identify whether the improvement comes from what behaviors.

Questions for authors

See the above weakness, I will reconsider the rating based on the authors' response

Reviewer_LwmNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reference-conditioned video diagnosis, aiming to detect unintended identity or appearance changes of a referenced subject in generated videos. The authors first develop a multipass annotation pipeline (Subject Binding → Multidimensional Comparison → Issue Filtering) using Gemini-3.5-Flash and Qwen3.8-Max, and convert the outputs into 2,160 compact diagnostic traces. They then introduce Stage-Specialized On-Policy Distillation (Stage OPD). Three stage specialists are trained from the same 9B base model. The student generates a complete trajectory, while each specialist provides reverse-KL supervision for tokens belonging to its assigned stage. This is combined with annotated-trace replay that assigns higher weights to evidence and decision tokens. On a 50-video human-reviewed benchmark, the 9B student improves F1 from 36.68% to 41.88%. The authors report that it outperforms a three-call SFT chain (40.47%) and falls within 0.31 points of the 27B SFT model.

Critical strengths
  • Practical and well-defined task. Fine-grained subject consistency checking (e.g., shirt numbers, hairstyles, and logos) is an important need in generated-video quality control that existing VBench-style metrics do not adequately address. Issue-level matching also makes both false alarms and missed defects measurable.

  • Informative process supervision. The compact traces preserve subject bindings, paired evidence, empty comparison blocks (60.4% of all blocks), excluded prompt-permitted changes, and change-to-event links. This provides substantially richer supervision than a final issue list alone, as clearly illustrated by the worked example in Appendix F.

  • Natural idea of stage-routed teachers. All specialists evaluate the same student-generated history, allowing the student to maintain a shared reasoning context while receiving stage-specific feedback. This differs clearly from prompt-level routing approaches such as MOPD. The authors also provide TP/FP counts and per-seed results, making the reported results easier to verify.

Critical weaknesses
    1. Checkpoint and hyperparameter selection on the test benchmark. The paper states that each configuration uses the checkpoint with the highest monitored F1, but does not mention a separate validation set. E, g, η, and the replay fraction (Table 3, Figure 4) also appear to have been selected on the same 50 videos. Figure 6 shows that F1 fluctuates between roughly 25 and 42 across checkpoints, so selecting the maximum introduces a clear optimistic bias. It is also unclear whether the baselines received comparable tuning budgets. Results using a held-out validation split or a fixed final checkpoint are needed.
    1. Small benchmark, single training run, and no significance testing. The benchmark contains only 50 videos and 49 issues, including just 9 negative cases.
    • Most performance gaps correspond to only 1–2 additional matches: 0.78 F1 over Video-OPD, 1.41 F1 over the SFT chain, and the ablation differences in Table 3.

    • Decoding alone produces F1 scores of 43.90, 32.50, and 29.63 for the 9B SFT model across three seeds, a spread of about 14 points.

    • All models are trained with a single training seed (42), with no standard deviations, confidence intervals, or bootstrap significance tests.

    A larger benchmark and results across multiple training seeds (mean ± std) are needed to establish whether the reported ranking is statistically meaningful.

  • 3, Missing key controls. The three specialists share the same base model and training data as the student, differing only in which tokens they supervise. Without the following controls, it remains unclear whether the improvements come from stage specialization or simply from additional reweighted training:

    • Replay-only (ρ=1): Continue SFT using only the weighted replay losses, without the on-policy term.

    • Single 9B teacher: Use a single teacher trained on full traces under the same stabilized training recipe.

    • Specialist chain: Report detection P/R/F1 when the three specialists are called sequentially. This is the most natural reference point for Stage OPD, but the paper only reports structure validity and candidate counts (Figure 5a/b).

Additionally, the untuned Qwen3.6-27B achieves 45.33 F1, outperforming the 27B SFT model (42.19) and all 9B methods. The choice of the weaker 27B SFT model as the "27B reference" therefore requires justification. This result also raises concerns about the quality of the compact traces.

Questions for authors

Please refer to the weakness sec.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.