- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a video encoder trained with pixel and feature reconstruction objectives. By reducing redundant information, it accommodates more video frames within a limited token budget, enhancing the model's performance on several video tasks. The approach is evaluated on temporal grounding, video understanding, and the proposed BallBench benchmark under varying token budgets and frame rates.
Critical strengths
- The construction of token-length supervision is interesting. The method evaluates reconstruction PSNR at different token-retention ratios and selects the smallest ratio that meets a quality threshold relative to a full-frame reconstruction. This provides automatic per-frame length labels without manual annotation.
Critical weaknesses
-
Predicting the minimum token-retention ratio needed for each P-frame is a central contribution of this paper. However, Table 1 shows inconsistent benefits over the fixed 1/16 ratio. The proposed predicted-length method improves accuracy on BallBench and Video-Holmes by 2.1% and 1.6%, but reduces performance on the other five benchmarks. The prediction stage also introduces additional computation for generating oracle labels and training the predictor, as well as a small inference-time overhead. The paper does not analyze the reasons for the performance decline.
-
Table 2 shows that performance can vary with frame rate, although the trends differ across models. However, table 1 evaluates NativeCodec at 16 fps while other baselines with their default frame rate. Therefore, we cannot determine whether the improvements in table 1 stem from the proposed method or from an increase in FPS.
Questions for authors
-
The title emphasizes Encoding Every Video Frame, whereas NativeCodec processes inputs at up to 16 fps. Is the claim Encoding Every Video Frame consistent with the 16 fps?
-
Could the authors report the class distribution of the oracle retention-ratio labels used to train the Stage 2 predictor, together with the distribution of its predictions? This would help determine whether certain compression ratios dominate the supervision and whether the predictor disproportionately favors particular ratios.
-
NativeCodec is trained on the BallBench training split, whereas the baselines in Table 2 are not reported to receive equivalent training. Could the authors add the Qwen3.5-4B model fine-tuned on the same data?
-
Since a fixed retention ratio of 1/16 already achieves strong results, does mixed-ratio training provide additional benefits over training only at 1/16 ratio? Also, does dynamic anchor placement outperform inserting I-frames at fixed intervals?
3 comments
Reply for uAhk - part 2/2
W2. What are the oracle label and predictor output distributions, and do particular retention ratios dominate?
We thank the reviewer for this suggestion and report oracle label and predictor output distributions across frame rates. The oracle labels are not uniform. At 16 fps, the proportions of 1/16, 1/32, and 1/64 are 24%, 34%, and 33%, totaling 91%. Small retention ratios therefore dominate supervision, although the labels are not concentrated in a single class.
Oracle
| FPS | 1 | 1/2 | 1/4 | 1/8 | 1/16 | 1/32 | 1/64 | Retention |
|---|---|---|---|---|---|---|---|---|
| 16 | 0 | 0 | 0.02 | 0.07 | 0.24 | 0.34 | 0.33 | 8.5 |
| 8 | 0 | 0.08 | 0.14 | 0.19 | 0.21 | 0.23 | 0.15 | 21.7 |
| 4 | 0.05 | 0.07 | 0.21 | 0.33 | 0.18 | 0.09 | 0.07 | 32.5 |
| 2 | 0.10 | 0.12 | 0.43 | 0.11 | 0.09 | 0.06 | 0.09 | 45.0 |
Predictor
| FPS | 1 | 1/2 | 1/4 | 1/8 | 1/16 | 1/32 | 1/64 | Retention |
|---|---|---|---|---|---|---|---|---|
| 16 | 0 | 0.01 | 0.01 | 0.03 | 0.18 | 0.30 | 0.47 | 7.6 |
| 8 | 0.03 | 0.04 | 0.10 | 0.14 | 0.19 | 0.24 | 0.26 | 20.8 |
| 4 | 0.08 | 0.05 | 0.13 | 0.22 | 0.18 | 0.16 | 0.18 | 31.1 |
| 2 | 0.12 | 0.07 | 0.30 | 0.12 | 0.11 | 0.12 | 0.16 | 41.0 |
As the frame rate decreases, the oracle distribution shifts toward larger retention ratios. This trend is consistent with the explanation that larger temporal intervals between frames generally require greater capacity for incremental representations.
The predictor tends to favor smaller retention ratios compared with the oracle. For example, at 16 fps, the predictor selects 1/64 in 47% of cases, compared with 33% for the oracle. Its average P-frame retention ratio is 3.92%, which is lower than the oracle's 4.45%. At 8, 4, and 2 fps, the predictor's average retention ratio is also lower than that of the oracle.
This bias may be related to the limited capacity of the predictor and the limited information available in its input. In future work, we will explore richer contextual inputs and predictor designs to improve the approximation of the oracle policy.
W3. Could you add a Qwen3.5-4B baseline fine-tuned on the same BallBench training data?
The Qwen3.5 4B SFT model in Table 1 was trained on the same data. We use this model to evaluate the effect of different input frame rates on its performance on BallBench, with the results shown below.
| FPS | 16 fps | 8 fps | 4 fps | 2 fps |
|---|---|---|---|---|
| Qwen3.5 4B | 12.2 | 13.2 | 8.6 | 10.4 |
| Qwen3.5 4B SFT | 22.3 | 20.2 | 18.3 | 15.3 |
| NativeCodec-4B | 26.1 | 24.3 | 23.5 | 22.9 |
Its lower performance than ours at high frame rates follows the explanation in replay for Q2: long, high-resolution videos produce excessively long token sequences, which degrade performance. On BallBench, low frame rates provide insufficient information, while high frame rates produce excessive tokens that can degrade performance. This motivates a compression scheme such as ours.
W4. Does mixed-ratio training offer advantages over fixed 1/16 training, and is dynamic anchor placement better than fixed I-frame intervals?
Mixed-ratio training primarily supports variable-length frame representations. Under dynamic allocation, different frames use different retention ratios, requiring the model to organize and use representations of varying lengths within the same context. Training only at 1/16 does not guarantee effectiveness at other ratios or on sequences mixing them. We therefore use mixed-ratio training to accommodate this variable-length interface. Under this training setting, oracle allocation achieves better downstream performance than fixed 1/16 retention while retaining fewer tokens, as shown below, suggesting the potential of adaptive allocation to improve both performance and efficiency.
| Method | AN | CH | QV | Retention |
|---|---|---|---|---|
| Fixed 1/16 | 54.5 | 53.8 | 67.7 | 12.1% |
| Predictor | 52.7 | 51.4 | 66.2 | 7.6% |
| Oracle | 55.3 | 54.6 | 68.2 | 8.6% |
In addition, we believe that dynamic anchor placement and inserting I-frames at fixed intervals would achieve similar downstream performance under their respective matched training settings, as training can compensate for this gap. We choose dynamic anchors primarily to accommodate the variable-length inference interface. Different P-frames retain different numbers of tokens, so fixed frame intervals do not correspond to a stable cumulative token cost. The dynamic rule determines when to introduce a complete I-frame reference based on the cumulative P-frame token count, allowing GOP boundaries to adjust to the actual allocated lengths.
If our responses have addressed your concerns, we kindly ask you to consider raising your score.
Follow-up on our rebuttal to Reviewer uAhk
Dear Reviewer uAhk,
Thank you again for your time and feedback. We posted our two-part rebuttal on November 11, including the same-data 16-fps control, BallBench SFT comparisons across frame rates, oracle/predictor distributions, and clarifications on mixed-ratio training and anchor placement.
Could you please revisit both parts and let us know whether our responses address your concerns, or which points need further clarification? Even a brief reply before the November 18 deadline would be very helpful, and we would welcome the opportunity to answer any remaining questions.
Best regards, The Authors
Reply for uAhk - part 1/2
We thank reviewer uAhk for appreciating our paper's method and insights.
Below, we address your questions in turn.
We agree that the current predictor does not consistently outperform fixed 1/16 retention.
We trained a downstream LLM following the oracle search rules. When the reconstruction oracle selects the ratio for each frame, performance is slightly better than that of the fixed retention ratio scheme, suggesting that content-adaptive allocation has the potential to yield further improvements.
However, this oracle determines the allocated length by searching over reconstruction decoding results for multiple candidate ratios, which incurs substantial computational overhead during both training label construction and testing and makes it unsuitable for practical inference. Our design moves this search process to the offline supervision construction stage and trains a lightweight predictor to predict the ratio directly during inference, thereby avoiding online reconstruction search.
At present, the lightweight predictor still does not adequately approximate the oracle allocation policy. Since the reconstruction of subsequent frames depends on previously retained representations, length prediction errors in early frames, particularly excessive compression, may degrade the reconstruction quality of subsequent frames and accumulate over time, limiting the extent to which the prediction scheme can realize the performance gains of the oracle.
We would like to clarify that NativeCodec aims to preserve denser temporal evidence within a limited visual-token budget, with its objective extending beyond improving representations at the same frame rate. It compresses the incremental representation of each frame relative to the history, which reduces the token cost per frame and allows the model to receive more frames within the same budget. The higher effective input frame rate is thus a capability enabled by this compression interface and forms part of the overall benefit we aim to evaluate; it should not be treated as an additional advantage unrelated to the method.
However, we agree that the comparison in Table 1 alone cannot separate the representational gains from incremental coding from the gains brought by denser input. Using 16 fps produces a very large number of tokens. For example, a 60-second video at 16 fps and a resolution of 512×512 produces approximately 250k tokens, so we do not limit the number of input tokens for the 16 fps version here. We additionally trained a Qwen3.5 4B model with 16 fps input on the same data, and the results are shown below.
Because small models have limited attention capacity, even after training, their performance at 16 fps is worse than at 2 fps. This also explains why we need a scheme that reduces the number of tokens produced by videos at their native frame rates. NativeCodec is designed for this purpose. By compressing the incremental representation of each frame, it preserves high frame rate input within a limited token budget, allowing the model to use denser temporal evidence without receiving the full long sequence produced by the native interface.
The frame rate of 16 fps is the operating point used in the current experiments, and it does not define the upper frame rate limit of the NativeCodec coding mechanism. We emphasize ”encoding every video frame” because the mechanism controls token cost by compressing the incremental representation of each frame, without relying on low frame rate input.
For original videos at 24/30/60 fps, the full frame sequence can likewise be fed into NativeCodec, with I-frames providing complete references and P-frames encoding incremental information relative to the history. The additional cost brought by a higher frame rate is controlled through the incremental representation of each frame and the token allocation mechanism.
Thus, “encoding every video frame” in the title describes how the method encodes video, while 16 fps describes the operating setting of the current experiments. The two are consistent.