acceptodds
Under review as a conference paper at ICLR 2027

Distill Video Datasets into Images

Abstract

Dataset distillation aims to synthesize compact yet informative datasets that allow models trained on them to achieve performance comparable to training on the full dataset. While this approach has shown promising results for image data, extending dataset distillation methods to video data has proven challenging and often leads to suboptimal performance. In this work, we first identify the core challenge in video set distillation as the substantial increase in learnable parameters introduced by the temporal dimension of video, which complicates optimization and hinders convergence. To address this issue, we observe that a single frame is often sufficient to capture the discriminative semantics of a video. Leveraging this insight, we propose Single-Frame Video set Distillation (SFVD), a framework that distills videos into highly informative frames for each class. Using differentiable interpolation, these frames are transformed into video sequences and matched with the original dataset, while updates are restricted to the frames themselves for improved optimization efficiency. To further incorporate temporal information, the distilled frames are combined with sampled real videos from real videos during the matching process through a channel reshaping layer. Extensive experiments on multiple benchmarks demonstrate that SFVD substantially outperforms prior methods, achieving improvements of up to 5.3% on MiniUCF, thereby offering a more effective strategy to video dataset distillation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dataset distillation, video recognition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QQaXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SFVD, a framework for reducing the optimization complexity of video dataset distillation by representing each synthetic video with a single learnable image. The authors argue that the increased number of learnable parameters introduced by the temporal dimension makes conventional video dataset distillation difficult to optimize. Experiments on several dataset demonstrate improvements in classification accuracy and computational efficiency over the reported baselines.

Critical strengths
  • SFVD and SFVD-T outperform the reported baselines across the four evaluated benchmarks.
  • SFVD significantly reduces the distillation cost.
  • The single-frame design considerably reduces the storage requirements of SFVD.
  • The authors demonstrate Performance improvements when transferring distilled datasets to other architecture.
Critical weaknesses
  • The idea of exploiting temporal redundancy and representing videos using a small number of learnable frames has already been investigated in prior work (e.g., Dancing with Still Images-CVPR'24, PRISM: Video Dataset Condensation with Progressive Refinement and Insertion for Sparse Motion-CVPR'26). Although the authors have compared their work with CVPR'24 work they have not mentioned about CVPR'26 work and properly compared with them. Furthermore, PRISM is particularly relevant because it also addresses the redundancy and optimization cost of video dataset condensation. However, PRISM is neither discussed nor experimentally compared in the submitted manuscript. This omission makes it difficult to asses the novelty and competitiveness of SFVD under comparable accuracy, storage, and computational budget.
  • The authors attribute the difficulty of conventional video distillation primarily to the increased number of learnable parameters. However, the experiments do not adequately distinguish parameter dimensionality from other factors, including temporal information, optimization objectives, and representation constraints. Figure 5(b) shows that increasing the number of distilled frames reduces accuracy, but this observation alone does not establish that optimization dimensionality is the underlying cause.
  • SFVD-T incorporates sampled real videos into the distilled representations, which raises questions about the contribution of the distilled frames relative to the additional real video information. The paper does not sufficiently isolate the benefit of the proposed fusion mechanism from that of simply including real video samples.
Questions for authors
  • How does SFVD differ fundamentally from PRISM (CVPR 2026), particularly in its use of sparse learnable representations and interpolation? Could the authors provide comparisons under matched IPC, storage, and computational budgets?
  • What experimental evidence establishes that the number of learnable parameters, rather than the optimization objective or temporal representation, is the primary source of convergence difficulties?
  • In SFVD-T, how much of the performance improvement comes from the additional real video samples rather than the proposed channel reshaping module or the distilled images themselves?
  • Since the implementation duplicates a single synthetic frame across the temporal dimension, what benefit does differentiable interpolation provide in this specific setting?
Reviewer_H2e1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Single-Frame Video Dataset Distillation (SFVD), which optimizes synthetic images through trajectory matching between repeated-frame synthetic clips and an interpolated real dataset, and extends it to SFVD-T by incorporating video samples through a learnable fusion module. The method achieves consistent accuracy improvements across four benchmarks. Its main strengths are the simplicity of the representation and the effectiveness of matching through the target video architecture.

Critical strengths
  • The observation that a single frame retains substantial discriminative information provides a good motivation for simplifying video dataset distillation. The paper connects this observation to an actionable design: optimizing a compact static representation while still training and evaluating through a video model.

  • The use of the target video architecture during distillation is well motivated. Matching repeated-frame clips through a video model addresses the mismatch between learning synthetic images for an image classifier and subsequently using them to train a video classifier.

Critical weaknesses
  • The methodological novelty relative to VDSD[1] is narrower than the framing text. Prior work already identifies weak temporal learning during distillation and uses a static-memory stage followed by dynamic compensation; SFVD’s more specific difference is matching repeated-frame clips by a video model and changing the temporal integration method. These changes are not separated through controlled component swaps.

  • The experiments for preserving temporal information remain limited. Since SFVD repeats a single frame, its classification gains do not by themselves demonstrate the retention of motion or temporal order. SFVD-T provides additional improvements on SSv2, but the absence of frame-shuffling or motion-sensitive class analyses leaves the temporal contribution unclear.

[1] Wang, Ziyu, et al. "Dancing with still images: Video distillation via static-dynamic disentanglement." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Questions for authors
  • The central claim that an enlarged parameter space is the main cause of video dataset-distillation failure is not isolated. SFVD changes both the synthetic representation and the expert-training dataset, replacing original videos with interpolated clips. Consequently, the convergence comparison in Fig. 2 mixes parameter reduction with a change in the matching target.

  • Can you cross single-frame versus full-video synthetic representations with interpolated versus original expert-training data, while controlling label supervision and tuning budgets?

  • Which parts of SFVD-T are optimized? The video pixels, the fusion module, or both? Are video samples fixed after initialization? Could you compare replacing only VDSD’s static stage with SFVD against replacing only its temporal module, and clarify exactly what must be used during training?

  • Does SFVD-T’s advantage persist on SSv2 classes requiring temporal order, and how does performance change when its distilled videos are shuffled or reversed?

Reviewer_VMAsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies video dataset distillation. The authors propose Single-Frame Video set Distillation (SFVD), which distills video datasets into compact sets of synthetic images through trajectory matching, reducing the number of learnable parameters. SFVD-T is an extended variant that combines the distilled images with sampled videos to incorporate temporal information. Experiments on four video benchmarks show improvements over existing distillation methods.

Critical strengths

Distilling video datasets into images is a useful approach that reduces storage requirements and the number of parameters.

Critical weaknesses
  1. My main concern is the explanation for the performance drop when more frames are distilled. The authors suggest that this drop is related to convergence difficulty, but Fig. 5b only reports final accuracy. I think this point needs further clarification.
  2. The ablation study and efficiency analysis are limited to MiniUCF. I think extending this ablation to larger datasets such as SSv2 would help justify the single-frame design.
  3. The specific contribution of the distilled images within SFVD-T seems unclear. I wonder how performance would change if the distilled images were replaced with sampled real frames while keeping the other settings unchanged.
  4. I am concerned about the applicability of single-frame distillation to longer videos or videos with substantial temporal variation. I think analyzing how performance changes with video duration and temporal variation would help clarify the scope of applicability of the single-frame design.
Questions for authors
  1. Which dataset is used in Fig. 3? This experiment supports the main motivation, but the dataset seems not specified in Sec. 3.2 or the figure caption.
  2. Fig. 4 indicates that SFVD uses interpolated videos as targets, while SFVD-T uses original videos. However, Sec. 3.4 describes training the target model on original videos when introducing the objective for both variants. Does this description apply only to SFVD-T?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.