acceptodds
Withdrawn from ICLR 2027

Octree-based video representation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Video models commonly use uniform grids even though visual complexity varies substantially across space and time. We introduce OctVideo, which approximates a video clip with an octree. This hierarchy recursively partitions a spatio-temporal volume into eight subvolumes, so that smooth regions remain coarse while detailed regions receive finer cells. Each leaf stores local RGB values and spatio-temporal gradients, supplemented by a lightweight learned residual. For reconstruction, a Conv1D VAE maps the serialized cells to a regular latent grid and selectively refines details during decoding. Our VAE achieves 36.12 dB PSNR with 38.2M parameters and 189.4 GFLOPs per clip on Kinetics-400 (K400). It also generalizes zero-shot to the high-resolution Densely Annotated VIdeo Segmentation (DAVIS) 2016 dataset with reconstruction quality comparable to the best evaluated models. On both datasets, it requires the fewest model FLOPs and achieves the fastest encoding and decoding among the evaluated models. OctVideo also supports video understanding, achieving competitive recognition performance with few input tokens when trained from scratch. By exploiting the redundancy already present in video signals and efficiently processing sparse structures, OctVideo provides an efficient representation for video.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
octree video efficient

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YVEfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OctVideo, an adaptive spatiotemporal octree representation whose leaves encode local RGB values and first-order gradients. Error-guided subdivision allocates tokens to complex regions, while sparse processing supports video reconstruction and recognition. A video VAE combines this representation with a regular latent grid and hierarchical decoding. Experiments demonstrate favorable reconstruction–compute trade-offs, supported by controlled architectural ablations and a complementary action-recognition evaluation.

Critical strengths
  • A coherent representation with useful analytical structure. The first-order leaf descriptors support direct reconstruction, and reconstruction error provides a natural criterion for adaptive subdivision.
  • Strong internal evidence for the design choices. The controlled ablations distinguish the effects of gradient descriptors, adaptive allocation, sparse decoding, and residual pathways. In particular, comparison with larger dense models supports a meaningful reconstruction–compute advantage.
Critical weaknesses
  • The scope of the practical efficiency advantage needs tighter qualification. The reported encoder timing excludes octree construction, which is important considering the small encoding-time difference from LeanVAE. The high-resolution evaluation also uses different native frame counts and execution protocols across methods. Moreover, OctVideo has higher peak memory than LeanVAE and worse DAVIS LPIPS/rFVD despite faster inference. These results support a useful trade-off, but do not yet establish a uniformly preferable operating point.
  • All compared baselines are video VAEs built for latent video generation, so a good latent space should be not only accurate in reconstruction but also suitable for generation. The paper trains no diffusion model on the OctVideo latent and provides no evaluation of latent-space quality, leaving the suitability of the representation for its baselines' primary use case unverified.
Questions for authors
  • Could the authors provide additional analysis of the latent space's suitability for generation? I would like to see a small-scale generation experiment on the latent if possible.
  • What are the complete input-to-reconstruction latency and peak memory, including tree construction and descriptor preparation, under a matched device, precision, and batch size?
  • The compared classifiers differ in clip resolution, temporal extent, and evaluation views, so token count alone provides incomplete evidence for the computational benefit in recognition. What are the total inference FLOPs and measured throughput for action recognition when all evaluation views are counted, ideally under a matched clip/view protocol?
Reviewer_EdaLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an octree representation for video data. Starting from a fixed-size HxWxT pixel tensor, the authors propose to recursively partition it in a fixed-depth octree, by following reconstruction heuristics: specifically, any node whose aggregated features do not represent their grid cell well enough gets further partitioned. Such method naturally settles on low depth leaves for regions with less spatio temporal information and on mode deep leaves in regions with color or motion high frequencies. The resulting octree is then serialized in a deterministic manner, resulting into a variable-length sequence of tokens representing the videos. The authors further propose architectures for reconstruction-based and recognition-based tasks, and validate the representation on Kinetics and UCF101 benchmarks.

Critical strengths
  • The idea of using octrees is new to my knowledge, and it is an interesting one. The partitioning scheme that the authors propose is simple and sound, and allows to dynamically focus on relevant sections of the video, at least for reconstruction purposes.
  • The dynamic tokenization algorithm could be useful, especially when operating on video streams under a token budget, and it might prove more effective than other efficient tokenization approaches.
Critical weaknesses
  • The main weakness with the paper is the model's evaluation. Although the authors provide a number of experiments, exploring reconstruction / compute cost tradeoffs, architectural ablations and zero-shot performances, the benchmarks in use do not help assessing the quality of the proposed representations. Indeed, the paper mainly assesses reconstruction performance and cost. But reconstruction for its own sake is not a relevant task (even when combined with cost analysis). To prove the usefulness of the proposed representations in practice, evidence should be provided showcasing that octree representations can be used to solve an useful use-case. For example, can we sample representations from the VAE prior to generate videos? Or can the tokenized octree representation be used for competitive neural compression? The problem with relying on reconstruction metrics only is that they do not inform anything about the quality of the representations. For example, an under-regularized autoencoder can excell at a reconstruction task while having meaningless latents. Even more so, an identity function, having perfect reconstruction and 0 flops, would outperform all methods on Fig. 3, Tab. 1, Tab. 2. Summarizing, the presented metrics and benchmarks do not inform the reader about the quality of the representations.
  • The video classification experiment is more sound, but it is conducted on outdated and saturated datasets, such as Kinetics and UCF101. The authors should consider showcasing results on more recent benchmarks, such as Activity100M [1]. Moreover, on the UCF101 dataset the proposed model and representation scores similarily to R(2+1)D (Table 7), an 8 years old model, which doesn't testify for the practical usefulness of the proposal.
  • Some aspects of the VAE architecture are not clear. For instance, it seems that the sparse octree representation gets then densified for VAE encoding. But what is then the benefit of sparsifying in the first place?

References:

[1] Chen, Delong, et al. "Action100m: A large-scale video action dataset." arXiv preprint arXiv:2601.10592 (2026).

Questions for authors

None

Reviewer_KyBxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OctVideo, which partitions a video clip into a spatio-temporal octree. Each leaf stores least-squares RGB values and gradients for its 2x4x4 subregions. A cell splits into eight children when its first-order L∞L_\infty approximation error exceeds a depth-dependent threshold. Leaves are serialized in Morton order, encoded with depthwise Conv1D blocks, and mapped onto an 8x32x32 latent grid. The decoder refines only the cells chosen by a split head. On K400, the VAE uses 38.2M parameters and 189.4 GFLOPs, reaches 36.12 dB PSNR, and reports the fewest FLOPs and fastest encoding and decoding among compared VAEs. Zero-shot DAVIS 2016 gives 32.76 dB at 0.540 s per clip. UCF101 ablations isolate the gradients, error-guided placement, and selective decoding, and a from-scratch classifier reaches 80.1% K400 Top-1 with about 4,200 tokens per view.

Critical strengths
  • The ablation design is well controlled. All variants are trained from scratch on UCF101 split 1 with the same data, latent size, optimization, and model selection, so the component claims are tested under matched conditions.
  • The VAE design is new. Mixed-depth octree leaves are connected to a regular latent grid through Conv1D encoding and copy/pool mapping, and the input octree supervises selective decoding.
Critical weaknesses
  • My main concern is that the quality comparisons are not matched. OctVideo is trained and evaluated on K400, while the baselines' training data and objectives are reported nowhere and "frozen K400 checkpoints" (Section 4.2.1) is ambiguous. Frame counts differ (29, 32, 33), and on DAVIS only Wan2.1 runs full-frame.
  • A second concern is the scope of the efficiency claims. FLOPs, parameters, and the 2.3 times decoding advantage over LeanVAE on K400 hold regardless of construction. The encoding times, however, exclude octree construction (under 6.5 ms on an RTX 4090, Section 4.2), and the Abstract's "fastest encoding and decoding" carries no caveat.
Questions for authors
  • What GPU, batch size, precision, and repetition count were used for the runtimes in Tables 1 and 2, and what is the end-to-end encoding time including octree construction?
  • What are the training data and objectives (including any adversarial loss) of the baseline checkpoints, and does "frozen K400 checkpoints" in Section 4.2.1 refer only to OctVideo?
  • As a clarification, what value range does the threshold τ=(0.7,0.8,1.0)\tau = (0.7, 0.8, 1.0) in Eq. (3) assume? Appendix A.2 gives [0, 1] only for metrics, where these values would rarely split cells. Please report K400 leaf depth ratios; finest-depth leaves have single-voxel subregions and no gradient information.
  • Full decoder expansion uses 4.4 times the compute (835.9 GFLOPs) yet reaches 0.28 dB lower PSNR (Table 3). Does this difference persist over repeated runs, and what explains it?
  • As a clarification, how do the classifier's GFLOPs and throughput compare with MViTv1-B? As a suggestion, a uniform tubelet ViT with about 4,096 tokens trained with the same recipe would be a fair baseline. The answer decides whether the token savings carry over to compute efficiency.
  • Was any latent diffusion generation run with OctVideo latents, even at small scale? The latent grid keeps the same 4x8x8 compression as the baselines, so diffusion-side cost is unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.