acceptodds
Withdrawn from ICLR 2027

Seeing the Evidence: Spatiotemporal-Privileged On-Policy Self-Distillation for Long-Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models still struggle to identify and integrate fine-grained spatiotemporal evidence for long-video question answering. We compare different combinations of temporal and spatial evidence and find that joint spatiotemporal views improve the same model's answer accuracy over standard full-video inputs. Building on this study, we propose ST-OPSD (Spatiotemporal On-Policy Self-Distillation), a framework that transfers the model's own predictive advantage under privileged spatiotemporal views to its full-video policy. Specifically, we construct the privileged visual input by combining temporal evidence frames, bounding-box-marked keyframes, and local crops, preserving event context while highlighting spatial details. A frozen teacher initialized from the same pretrained model as the student provides token-level supervision under this view along student-generated response trajectories. We train ST-OPSD with a forward-KL distillation objective on 3,200 carefully selected examples from CG-Bench, which we annotate with spatial bounding boxes. Experiments show that ST-OPSD achieves strong performance across four video understanding benchmarks using Qwen3.5 models at different scales.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video understanding, OPSD

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_hTfkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ST-OPSD, which applies on-policy self-distillation to long-video QA. The student model sees 256 frames sampled across the video, while a frozen teacher initialized from the same model bounding-box-marked keyframes and provides token-level supervision. Experiments show that the approach achieves strong performance across four video understanding benchmarks on 4B and 9B models.

Critical strengths
  1. It conducts experiments on different scales. The 4B and 9B achieve gain across all four benchmarks, with average increases of 3.26 and 2.39 respectively.
  2. It analyzes the effectiveness of which view should be presented to the teacher model. It ablades for three different views and find joint spatiotemporal view is the best.
Critical weaknesses
  1. The main table (Table 2) is unfinished. It has randomly generated values as placeholder, as mentioned in the caption "The 4B and 9B SFT rows contain illustrative placeholder values pending measured evaluation results." This make the experimental results hard to follow and questionable.
  2. The teacher input is unclear. Section 3.2.2 describes several marked keyframes and crops. Section 4.3 says one keyframe and one crop. Which was used in Table 3? Teacher frame counts, image sizes, and box annotation prompts are also missing.
  3. Teacher quality is only tested for 9B. The training data is selected using 9B, then also used for 2B and 4B. Teacher accuracy on the training examples is not reported for each model size. This makes the mixed 2B results hard to explain.
Questions for authors
  1. How does ST-OPSD compare with directly training grounding on the same backbone and evidence annotations? 2, At each model scale, what is teacher accuracy on the training examples? How often is the ordinary view correct while the privileged teacher is wrong?
  2. Does Table 3 use one keyframe or all annotated keyframes? What are the teacher frame counts and resolutions, and what inputs and prompts did the annotation agent receive?
1 comment
AuthorsOpenReview ↗

Response to Reviewer hTfk

Q1

We evaluated AnchorSR SFT-v4, a grounding SFT baseline initialized from Qwen3.5-9B and trained on the same CG-Bench examples and evidence annotations as ST-OPSD. The baseline is trained to predict question-relevant evidence locations. We tested its checkpoint on four video question-answering benchmarks to compare direct grounding supervision with privileged-view self-distillation. ST-OPSD outperforms the grounding SFT baseline on all four benchmarks.

Accuracy is reported as a percentage. Δ denotes ST-OPSD minus Grounding SFT in percentage points.

MethodVideo-MMEMLVU (M-Avg)LVBenchVideo-Holmes
Grounding SFT (AnchorSR SFT-v4)69.7873.7152.4247.74
ST-OPSD (Ours)73.5981.2356.8853.40
Δ (pp)+3.81+7.52+4.46+5.66

Q2

At each model scale, we evaluated the same frozen checkpoint under ordinary and privileged views on all 3,200 training examples. The table reports both accuracies and the percentage of all training examples for which the ordinary view produces a correct answer but the privileged teacher does not.

Model ScaleOrdinary-View Accuracy (%)Privileged Teacher Accuracy (%)Gain (pp)Ordinary Correct, Teacher Wrong (%)
2B28.0344.56+16.537.66
4B45.7262.91+17.196.53
9B47.2563.25+16.007.72

Q3

Table 3 uses all annotated keyframes retained after selection and deduplication by the annotation agent. The temporal view contains 16 frames uniformly sampled within the ground-truth temporal intervals. The spatial view contains the retained annotated keyframes and their corresponding crops, with counts varying across examples. The joint view combines these two components. The maximum pixel count per image is 112,896.

We will correct the singular wording in the manuscript and provide the prompts used by the annotation agent in the revised version.

Reviewer_fV3dOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Propose ST-OPSD, a frozen teacher model has access to annotated temporal intervals, keyframes with bounding box and local crops during training, while the student only observes sampled frames from raw videos. And via self-distillation, the teacher’s predictive capability with additional evidence is transferred to the student, and such annotations are no longer provided at test time.

Critical strengths

Teacher and student are initialized from the same pretrained model, with different visual inputs allowing the teacher to provide stronger supervision under explicit spatiotemporal guidance. On-policy self-distillation transfers this predictive advantage to the student operating on ordinary video inputs, imbue the model with spatio-temporal content perception for video question answering, without requiring manually written or externally generated chain-of-thought supervision.

Critical weaknesses
  1. Table 2's caption say "The 4B and 9B SFT rows contain illustrative placeholder values pending measured evaluation results." The reported results seem not to come from real experiments. And the content is likely AI generated.
  2. The joint view adds frames and enlarged crops, so its gains cannot be clearly attributed to spatiotemporal complementarity rather than additional observations or magnification.
Questions for authors
  1. Can you report measured SFT results with the same trainable modules as ST-OPSD?
  2. Does the joint view advantage persist under matched teacher-side visual-token budgets and resolution settings?
1 comment
AuthorsOpenReview ↗

Response to Reviewer fV3d

Q1

We would like to clarify that the SFT results reported in Table 2, including those for the 4B and 9B models, were obtained from actual experiments and are not illustrative placeholders. The SFT baselines use the same trainable modules as ST-OPSD. We sincerely apologize for the incorrect statement left in the caption and the concerns it raised about the credibility of our experimental results.

When we were preparing the main results table, the SFT results for the 4B and 9B models had not yet been entered into the table. At that stage, we used AI assistance to polish the caption, which contained a temporary note indicating that those results were pending. We subsequently filled in the results obtained from actual experiments but inadvertently failed to remove that sentence and the corresponding † markers before submission. We also explicitly disclosed our use of AI assistance for language polishing in the submitted manuscript.

Q2

Effect of the teacher visual-token budget.

We first measured the average number of teacher-side visual tokens under different privileged-view configurations. Adding spatial crops to the same 16 temporal frames increases the average token count from 1,526 to 1,651, an increase of 8.2%.

Teacher viewAverage visual tokens per example
Temporal: 16 uniformly sampled frames1,526
Joint: the same 16 frames + spatial crops1,651
Temporal: 2 FPS, up to 256 frames4,057

To examine whether denser temporal sampling can achieve comparable student performance, we evaluated a temporal-only teacher view sampled at 2 FPS, capped at 256 frames. This configuration uses approximately 2.46 times as many visual tokens as the joint view.

The resulting student accuracies are reported below. ST-OPSD achieves higher accuracy than the denser temporal-only baseline on all four benchmarks.

Teacher viewVideo-MMEMLVU (M-Avg)LVBenchVideo-Holmes
Temporal: 2 FPS, up to 256 frames72.8978.8055.7152.80
ST-OPSD (Ours)73.5981.2356.8853.40

These results support the effectiveness of the joint evidence design under the evaluated settings. Increasing temporal sampling density and allocating a substantially larger visual-token budget to temporal frames alone does not match the student performance achieved with the joint spatiotemporal teacher view.

Reviewer_9vvKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduced Spatiotemporal On-Policy Self-Distillation (ST-OPSD) which is basically:

  • 1\ a data engine
    • starting from cg bench, the pipeline first leverages qwen3.5 9b to filter samples, and then gpt 6 astra to label timestamped bbox.
    • ended with 3200 spatiotemporal training data.
  • 2\ ST-OPSD
    • basically standard OPSD where the extra info to the teacher now coming from
      • gt temporal intervals only (i.e. not given full 256 frames)
      • frames with bbox overlaid (the gpt 6 labeld)
      • and cropped frames based on bbox
  • 3\ eval
    • Then the work has been evaluated on videomme, mlvu, lvben, video-homes
Critical strengths
  • The motivation and methodology is clear that aligning to a spatiotemporal aware distribution could help several video benchmarks. By trained on enhanced cg bench data, but improved on other video benchmarks, this work shows it can be generalizable.
Critical weaknesses
  • Limited novelty
    • Something -opsd has been used so widely and provided limited insights to the community. This work is basically distilling gpt 6 astra and some tool calls (crop) formed distribution into the student model. Practically, when the model scaling from 4b to 8b, the gains are diminishing from +3.26 -> +2.39, indicating that the additional benefits of spatiotemporal are not that necessary for larger models. So if both limited novelty and not practically useful when we scale, this method's contribution to the community becomes a question mark.
  • Missing major comparisons
    • This work only compares with the classic opsd, not comparing to other -opsd work. for example vision-opd (which is essentially a self-distillation work), although the paper cited this work, it didn't compare to its performance, which has a same design idea to have a cropped image for the teacher model to distill.
Questions for authors
  • 1\ for the gpt 6 generated timestamped annotations, does the gpt 6 get the gt timestamp input from the cg bench?
  • 2\ "The 4B and 9B SFT rows contain illustrative placeholder values pending measured evaluation results."? how possible we put fake numbers in this important results table? Please explain.
    • this even looks like a violation - one key reason for clear rejection or score could be higher
    • if no results then just remove that row or put "-" is even better then putting illustrative values. this all of sudden makes the whole paper numbers not trustful anymore.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9vvK’s Concerns

We sincerely apologize for the incorrect statement left in the caption of Table 2 and the concerns it raised about the credibility of our experimental results.

When we were preparing the main results table, the SFT results for the 4B and 9B models had not yet been entered into the table. At that stage, we used AI assistance to polish the caption, which contained a temporary note indicating that those results were pending. We subsequently filled in the results obtained from actual experiments but inadvertently failed to remove that sentence and the corresponding † markers before submission. We also explicitly disclosed our use of AI assistance for language polishing in the submitted manuscript.

We would like to clarify that the reported SFT values are actual experimental results, not illustrative placeholders.

W1:

We would like to clarify both the source of supervision and the scope of our contribution.

First, we do not distill GPT-6-astra’s output distribution. GPT-6-astra is used only for offline spatial annotation: it selects keyframes and annotates bounding boxes within clips extracted using CG-Bench’s temporal clue intervals. The distillation teacher is a frozen copy of the same pretrained Qwen3.5 checkpoint used to initialize the student. Its next-token distributions are computed under the privileged visual view along student-generated prefixes.

Second, our contribution concerns the construction and evaluation of privileged visual evidence for long-video self-distillation. Specifically, we combine interval-sampled full frames that preserve temporal context with bounding-box-marked keyframes and local crops that highlight spatial details. We evaluate how these complementary forms of evidence affect both the frozen model’s predictions and the resulting student’s performance.

Third, we did not train an 8B model in this work. The 9B model consistently improves over its pretrained backbone across all four benchmarks. Moreover, the 9B ablation study in Table 3 shows that the joint spatiotemporal view achieves an average accuracy 4.43 and 2.27 percentage points higher than the temporal-only and spatial-only views, respectively. These results demonstrate that joint spatiotemporal evidence continues to provide additional benefits at the larger scale evaluated. A smaller improvement alone therefore does not establish a lack of practical value.

W2:

Thank you for highlighting Vision-OPD. We cited it because it shares the principle of distilling privileged visual evidence into an ordinary-view policy. However, Vision-OPD was developed and evaluated for fine-grained image understanding, and its paper does not report results on the four video benchmarks used in our study.

We evaluated the released Vision-OPD-4B and Vision-OPD-9B checkpoints on all four benchmarks using the same video evaluation protocol as for our models. The results are shown below.

ModelVideo-MMEMLVULVBenchVideo-Holmes
Vision-OPD-4B70.6373.9654.9446.38
ST-OPSD-4B71.1578.0655.6150.41
Vision-OPD-9B72.9677.6655.7150.52
ST-OPSD-9B73.5981.2356.8853.40

Q1:

We extract video clips using the ground-truth temporal clue intervals provided by CG-Bench and provide these clips to an annotation agent powered by GPT-6-astra. The agent selects keyframes from these clips and annotates spatial bounding boxes in the selected frames. Both the number of selected keyframes and the number of bounding boxes per frame are determined by the agent.

Q2:

Thank you for raising this concern. Please refer to our clarification in the opening paragraphs of this response.

we do not distill GPT-6-astra’s output distribution

Sorry for the confusion. Here, distillation has a broader definition, not just distillation algorithm. In general we can consider any kinds of data labeling from stronger models and then train those data into other models is a distillation. And here we are doing this.

illustrative placeholder

Thanks for the clarification. This is Table 2. The key results table. The explanation is not trustful any more.

I'll keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.