acceptodds
Under review as a conference paper at ICLR 2027

NativeCodec: Encoding Every Video Frame with Variable-Length Causal Visual Tokens

10 views

Abstract

Reasoning about motion and object identity often depends on brief interactions that sparse frame sampling can miss. Increasing temporal resolution preserves more of this evidence, but encoding each frame independently introduces substantial visual redundancy into multimodal large language models. The central challenge is to retain temporal detail without repeatedly allocating tokens to content already represented by earlier frames. Because the amount of new information varies across frames, a compact video representation should adapt its capacity to each frame's contribution relative to its history. **We introduce NativeCodec**, a causal visual encoder for video inputs at up to 16 fps. Inspired by predictive coding, it combines complete anchor frames with variable-length incremental representations of subsequent frames. Pixel and feature reconstruction at multiple truncation lengths encourages the encoder to concentrate information complementary to its history into retained token suffixes. An offline reconstruction-based teacher supervises a per-frame length predictor, and cumulative suffix lengths determine anchor placement. **We also introduce BallBench**, which tests identity tracking through collisions between visually identical objects. With a 4B base model and a fixed suffix ratio, NativeCodec reaches 56.4 mIoU on TimeLens-Bench, 25.0 mIoU on MomentSeeker, and 26.1% accuracy on BallBench, and with predicted suffix lengths it reaches 48.7% accuracy on Video-Holmes. Relative to its Qwen3.5-4B base model, it **gains 13.9 points on BallBench, 7.6 on Video-Holmes, 6.5 on MomentSeeker, and 5.7 mIoU on TimeLens-Bench**. NativeCodec is ahead of the native pathway of its base model at every budget, and the margin widens as the budget tightens, reaching 14.2 mIoU at 4k tokens. At a suffix ratio of 1/16, NativeCodec obtains 54.4 mIoU from 16k visual tokens, while the native pathway of its base model obtains 50.9 from 64k tokens, four times the budget. The code and model will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, Video Understanding, Codec

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uAhkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a video encoder trained with pixel and feature reconstruction objectives. By reducing redundant information, it accommodates more video frames within a limited token budget, enhancing the model's performance on several video tasks. The approach is evaluated on temporal grounding, video understanding, and the proposed BallBench benchmark under varying token budgets and frame rates.

Critical strengths
  1. The construction of token-length supervision is interesting. The method evaluates reconstruction PSNR at different token-retention ratios and selects the smallest ratio that meets a quality threshold relative to a full-frame reconstruction. This provides automatic per-frame length labels without manual annotation.
Critical weaknesses
  1. Predicting the minimum token-retention ratio needed for each P-frame is a central contribution of this paper. However, Table 1 shows inconsistent benefits over the fixed 1/16 ratio. The proposed predicted-length method improves accuracy on BallBench and Video-Holmes by 2.1% and 1.6%, but reduces performance on the other five benchmarks. The prediction stage also introduces additional computation for generating oracle labels and training the predictor, as well as a small inference-time overhead. The paper does not analyze the reasons for the performance decline.

  2. Table 2 shows that performance can vary with frame rate, although the trends differ across models. However, table 1 evaluates NativeCodec at 16 fps while other baselines with their default frame rate. Therefore, we cannot determine whether the improvements in table 1 stem from the proposed method or from an increase in FPS.

Questions for authors
  1. The title emphasizes Encoding Every Video Frame, whereas NativeCodec processes inputs at up to 16 fps. Is the claim Encoding Every Video Frame consistent with the 16 fps?

  2. Could the authors report the class distribution of the oracle retention-ratio labels used to train the Stage 2 predictor, together with the distribution of its predictions? This would help determine whether certain compression ratios dominate the supervision and whether the predictor disproportionately favors particular ratios.

  3. NativeCodec is trained on the BallBench training split, whereas the baselines in Table 2 are not reported to receive equivalent training. Could the authors add the Qwen3.5-4B model fine-tuned on the same data?

  4. Since a fixed retention ratio of 1/16 already achieves strong results, does mixed-ratio training provide additional benefits over training only at 1/16 ratio? Also, does dynamic anchor placement outperform inserting I-frames at fixed intervals?

3 comments
AuthorsOpenReview ↗

Reply for uAhk - part 1/2

We thank reviewer uAhk for appreciating our paper's method and insights.

Below, we address your questions in turn.

W1. Why does length prediction underperform fixed 1/16 retention on several benchmarks, despite its additional training and inference costs?

We agree that the current predictor does not consistently outperform fixed 1/16 retention.

We trained a downstream LLM following the oracle search rules. When the reconstruction oracle selects the ratio for each frame, performance is slightly better than that of the fixed retention ratio scheme, suggesting that content-adaptive allocation has the potential to yield further improvements.

MethodANCHQV
Fixed 1/1654.553.867.7
Predictor52.751.466.2
Oracle55.354.668.2

However, this oracle determines the allocated length by searching over reconstruction decoding results for multiple candidate ratios, which incurs substantial computational overhead during both training label construction and testing and makes it unsuitable for practical inference. Our design moves this search process to the offline supervision construction stage and trains a lightweight predictor to predict the ratio directly during inference, thereby avoiding online reconstruction search.

At present, the lightweight predictor still does not adequately approximate the oracle allocation policy. Since the reconstruction of subsequent frames depends on previously retained representations, length prediction errors in early frames, particularly excessive compression, may degrade the reconstruction quality of subsequent frames and accumulate over time, limiting the extent to which the prediction scheme can realize the performance gains of the oracle.

W2. Do the gains reflect NativeCodec's compression or its higher input frame rate compared with the baselines?

We would like to clarify that NativeCodec aims to preserve denser temporal evidence within a limited visual-token budget, with its objective extending beyond improving representations at the same frame rate. It compresses the incremental representation of each frame relative to the history, which reduces the token cost per frame and allows the model to receive more frames within the same budget. The higher effective input frame rate is thus a capability enabled by this compression interface and forms part of the overall benefit we aim to evaluate; it should not be treated as an additional advantage unrelated to the method.

However, we agree that the comparison in Table 1 alone cannot separate the representational gains from incremental coding from the gains brought by denser input. Using 16 fps produces a very large number of tokens. For example, a 60-second video at 16 fps and a resolution of 512×512 produces approximately 250k tokens, so we do not limit the number of input tokens for the 16 fps version here. We additionally trained a Qwen3.5 4B model with 16 fps input on the same data, and the results are shown below.

MethodANCHQV
Fixed 1/1654.553.867.7
Qwen3.5 4B SFT 2 fps51.749.566.6
Qwen3.5 4B SFT 16 fps49.846.364.8

Because small models have limited attention capacity, even after training, their performance at 16 fps is worse than at 2 fps. This also explains why we need a scheme that reduces the number of tokens produced by videos at their native frame rates. NativeCodec is designed for this purpose. By compressing the incremental representation of each frame, it preserves high frame rate input within a limited token budget, allowing the model to use denser temporal evidence without receiving the full long sequence produced by the native interface.

Q1. Is the title's claim of “encoding every video frame” consistent with evaluation at 16 fps?

The frame rate of 16 fps is the operating point used in the current experiments, and it does not define the upper frame rate limit of the NativeCodec coding mechanism. We emphasize ”encoding every video frame” because the mechanism controls token cost by compressing the incremental representation of each frame, without relying on low frame rate input.

For original videos at 24/30/60 fps, the full frame sequence can likewise be fed into NativeCodec, with I-frames providing complete references and P-frames encoding incremental information relative to the history. The additional cost brought by a higher frame rate is controlled through the incremental representation of each frame and the token allocation mechanism.

Thus, “encoding every video frame” in the title describes how the method encodes video, while 16 fps describes the operating setting of the current experiments. The two are consistent.

AuthorsOpenReview ↗

Reply for uAhk - part 2/2

W2. What are the oracle label and predictor output distributions, and do particular retention ratios dominate?

We thank the reviewer for this suggestion and report oracle label and predictor output distributions across frame rates. The oracle labels are not uniform. At 16 fps, the proportions of 1/16, 1/32, and 1/64 are 24%, 34%, and 33%, totaling 91%. Small retention ratios therefore dominate supervision, although the labels are not concentrated in a single class.

Oracle

FPS11/21/41/81/161/321/64Retention
16000.020.070.240.340.338.5
800.080.140.190.210.230.1521.7
40.050.070.210.330.180.090.0732.5
20.100.120.430.110.090.060.0945.0

Predictor

FPS11/21/41/81/161/321/64Retention
1600.010.010.030.180.300.477.6
80.030.040.100.140.190.240.2620.8
40.080.050.130.220.180.160.1831.1
20.120.070.300.120.110.120.1641.0

As the frame rate decreases, the oracle distribution shifts toward larger retention ratios. This trend is consistent with the explanation that larger temporal intervals between frames generally require greater capacity for incremental representations.

The predictor tends to favor smaller retention ratios compared with the oracle. For example, at 16 fps, the predictor selects 1/64 in 47% of cases, compared with 33% for the oracle. Its average P-frame retention ratio is 3.92%, which is lower than the oracle's 4.45%. At 8, 4, and 2 fps, the predictor's average retention ratio is also lower than that of the oracle.

This bias may be related to the limited capacity of the predictor and the limited information available in its input. In future work, we will explore richer contextual inputs and predictor designs to improve the approximation of the oracle policy.

W3. Could you add a Qwen3.5-4B baseline fine-tuned on the same BallBench training data?

The Qwen3.5 4B SFT model in Table 1 was trained on the same data. We use this model to evaluate the effect of different input frame rates on its performance on BallBench, with the results shown below.

FPS16 fps8 fps4 fps2 fps
Qwen3.5 4B12.213.28.610.4
Qwen3.5 4B SFT22.320.218.315.3
NativeCodec-4B26.124.323.522.9

Its lower performance than ours at high frame rates follows the explanation in replay for Q2: long, high-resolution videos produce excessively long token sequences, which degrade performance. On BallBench, low frame rates provide insufficient information, while high frame rates produce excessive tokens that can degrade performance. This motivates a compression scheme such as ours.

W4. Does mixed-ratio training offer advantages over fixed 1/16 training, and is dynamic anchor placement better than fixed I-frame intervals?

Mixed-ratio training primarily supports variable-length frame representations. Under dynamic allocation, different frames use different retention ratios, requiring the model to organize and use representations of varying lengths within the same context. Training only at 1/16 does not guarantee effectiveness at other ratios or on sequences mixing them. We therefore use mixed-ratio training to accommodate this variable-length interface. Under this training setting, oracle allocation achieves better downstream performance than fixed 1/16 retention while retaining fewer tokens, as shown below, suggesting the potential of adaptive allocation to improve both performance and efficiency.

MethodANCHQVRetention
Fixed 1/1654.553.867.712.1%
Predictor52.751.466.27.6%
Oracle55.354.668.28.6%

In addition, we believe that dynamic anchor placement and inserting I-frames at fixed intervals would achieve similar downstream performance under their respective matched training settings, as training can compensate for this gap. We choose dynamic anchors primarily to accommodate the variable-length inference interface. Different P-frames retain different numbers of tokens, so fixed frame intervals do not correspond to a stable cumulative token cost. The dynamic rule determines when to introduce a complete I-frame reference based on the cumulative P-frame token count, allowing GOP boundaries to adjust to the actual allocated lengths.

If our responses have addressed your concerns, we kindly ask you to consider raising your score.

AuthorsOpenReview ↗

Follow-up on our rebuttal to Reviewer uAhk

Dear Reviewer uAhk,

Thank you again for your time and feedback. We posted our two-part rebuttal on November 11, including the same-data 16-fps control, BallBench SFT comparisons across frame rates, oracle/predictor distributions, and clarifications on mixed-ratio training and anchor placement.

Could you please revisit both parts and let us know whether our responses address your concerns, or which points need further clarification? Even a brief reply before the November 18 deadline would be very helpful, and we would welcome the opportunity to answer any remaining questions.

Best regards, The Authors

Reviewer_GTcUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes NativeCodec, a causal visual encoder that reduces redundant video tokens while preserving temporal information for video understanding. Inspired by predictive coding, it combines complete anchor frames with variable-length representations of subsequent frames, using reconstruction objectives and a learned predictor to determine how much information to retain. The paper also introduces BallBench, a benchmark for tracking object identities through collisions. Experiments show that NativeCodec consistently outperforms the Qwen3.5-4B baseline across several video understanding benchmarks, particularly under limited token budgets.

Critical strengths
  • The paper is well written, with clear motivation and a well-structured methodology that is easy to follow.
  • The proposed approach is technically interesting, particularly the use of reconstruction-based training to learn variable-length frame representations and an adaptive length predictor to reduce redundant visual tokens.
  • The experiments are extensive, covering multiple benchmarks and token budgets, and demonstrate consistent improvements over the baseline, especially under tight token constraints.
Critical weaknesses
  • One interesting aspect of the paper is the use of reconstruction quality to determine the number of tokens retained for each frame. However, it is unclear whether reconstruction quality is a reliable indicator of the information needed for downstream reasoning. For example, small visual changes may be critical for understanding object interactions but contribute little to reconstruction error?
  • The method involves several design decisions, including the reconstruction threshold, cumulative-token anchor placement param, and the use of both pixel-level and feature-level reconstruction. More ablation studies would help justify these choices and clarify their individual contributions.
  • It is unclear where the inference speedup mainly comes from. Is it primarily due to the reduced number of visual tokens passed to the MLLM, or does the parallel length prediction and incremental encoder architecture also contribute to computational efficiency?
Questions for authors

Overall, I see the paper is as interesting. I only have two major questions about the reconstruction quality and the factor that brings the inference speedup.

4 comments
AuthorsOpenReview ↗

Reply for GTcU

We thank reviewer GTcU for appreciating our paper's writing, experiments, and insights.

We address each question below.

Does reconstruction quality reliably indicate the information needed for subsequent reasoning?

RGB video is the model's original visual evidence. We use reconstruction quality as a measurable retention criterion requiring no task annotations, aiming to preserve original observations in compressed representations. Under the same token budget, reconstruction quality and downstream accuracy generally decline as retention decreases, supporting this proxy for capacity allocation. Global RGB error may miss small regions crucial for interaction reasoning. We will explore locally weighted or task-relevant metrics to capture these subtle but important changes.

Ratio11/21/41/81/161/321/64
AN55.155.555.755.054.553.450.9
CH55.555.555.054.653.851.748.9
QV68.768.468.067.367.767.364.5
PSNR18.7618.5718.4418.2718.1817.8017.54

Could you provide more ablation experiments?

Reconstruction threshold

We fix Stage 1 and retrain only the Stage 2 length predictor for each tolerance δ, comparing predicted and sequential-oracle retention and compression, and the PSNR gap between predicted reconstruction and the full-grid teacher.

Tolerance δ0.0 dB0.3 dB0.5 dB0.8 dB
Predicted token retention16.00%11.20%5.80%4.28%
Oracle token retention19.06%12.80%8.50%5.50%
Predicted compression6.25×8.93×17.24×23.36×
Oracle compression5.25×7.81×11.76×18.18×
PSNR gap0.28 dB0.23 dB0.31 dB0.25 dB

As δ rises from 0 to 0.8 dB, predicted retention falls from 16.00% to 4.28% and compression rises from 6.25× to 23.36×; the PSNR gap remains 0.23–0.31 dB.

Pixel and feature reconstruction

We retain the auxiliary loss for embedding-layer feature alignment and omit pixel-only training. We train an auxiliary-only model, compare auxiliary loss across steps, and evaluate downstream performance across ratios under a 64K visual token budget. At smaller ratios, joint training lowers auxiliary loss faster and reaches a lower final loss, while improving downstream performance.

Auxiliary loss by step

RatioTraining135040506750945012150
1Both (pix + aux)0.360.190.160.150.15
1Aux-only0.320.180.150.150.15
1/2Both (pix + aux)0.520.310.280.260.26
1/2Aux-only0.480.310.270.260.25
1/4Both (pix + aux)0.610.410.370.350.34
1/4Aux-only0.570.410.370.350.34
1/8Both (pix + aux)0.660.480.430.420.41
1/8Aux-only0.630.490.460.440.43
1/16Both (pix + aux)0.720.550.500.490.48
1/16Aux-only0.700.600.560.540.53
1/32Both (pix + aux)0.770.600.550.540.53
1/32Aux-only0.750.670.640.620.61
1/64Both (pix + aux)0.790.620.570.560.55
1/64Aux-only0.780.700.680.670.67
Ratio11/21/41/81/161/321/64
AN55.155.555.755.054.553.450.9
AN-AUX Only55.155.455.654.753.251.949.8
CH55.555.555.054.653.851.748.9
CH-AUX Only55.655.555.153.951.151.047.3
QV68.768.468.067.367.767.364.5
QV-AUX Only68.668.368.166.966.265.763.2

Does speedup come mainly from token reduction, or also from parallel length prediction and incremental encoding?

The main speedup comes from fewer visual tokens passed to the MLLM. NativeCodec uses previous frames to represent later frames as shorter incremental sequences, reducing prefill computation, attention cost, and KV-cache memory. Shorter visual context also reduces attention cost during autoregressive generation.

Parallel length prediction is also substantially faster than oracle search, but adds latency compared with a fixed threshold. A lightweight feedforward network predicts P-frame ratios in parallel, without length search or repeated reconstruction. This enables adaptive allocation at little additional cost, avoiding substantial length-selection latency.

The incremental encoder adds visual-side computation by causally aggregating features across frames to incorporate history, compared with passing per-frame features directly to the MLLM.

For long, redundant sequences with substantial compression, MLLM savings exceed the cost of incremental encoding.

If our responses have addressed your concerns, we kindly ask you to consider raising your score.

AuthorsOpenReview ↗

Follow-up on our rebuttal to Reviewer GTcU

Dear Reviewer GTcU,

Thank you again for your time and feedback. We posted our rebuttal on November 11, including additional reconstruction and downstream analyses, loss and tolerance ablations, and a clarification of where the computational savings come from.

Could you please revisit our response and share whether these results and explanations address your questions? Even a brief reply before the November 18 deadline would be very helpful. We would be happy to clarify any points you would like to discuss further.

Best regards, The Authors

I thank the authors for the rebuttal. After carefully reading all reviews and the rebuttal, I would keep my initial judgment (weak accept), as the rebuttal addressed my main concerns.

Thank you very much for your careful re‑evaluation. We greatly appreciate your recognition of our revisions.

Reviewer_BL9yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes NativeCodec, a video encoder for MLLMs. It encodes the first frame of each group in full (I frame), and later frames only as a few tokens that capture what changed (P frames). This allows input at up to 16 fps with far fewer visual tokens. A learned predictor can also choose how many tokens each P frame gets. The paper also introduces BallBench, a synthetic benchmark for tracking identical balls through collisions.

Critical strengths
  1. Well-motivated design. The paper combines I/P-frame GOPs and a budget-driven GOP rule into a causal, variable-length token stream that matches the autoregressive LLM interface.
  2. Strong results under tight budgets. NativeCodec leads the native pathway by 14.2 mIoU at 4k tokens, and at 16k it exceeds the native pathway at 64k.
  3. Useful diagnostic benchmark. BallBench is synthetic with exact ground truth, can be rendered at any frame rate, and uses separate seeds for its train and evaluation splits.
Critical weaknesses
  1. Would simple pooling work just as well? In Table 1, NativeCodec runs at 16 fps while the baselines use their default frame rates. In Fig. 3, the baseline is the base model's native pathway, which is not trained on the same data. So it is hard to tell whether the gain comes from the I/P codec, or simply from seeing more frames with fewer tokens each. A natural control is to train the same base on the same data at 16 fps, and shrink each frame to the same token count with plain pooling.
  2. The learned length allocation does not seem to help. The paper argues that token lengths should be learned rather than hand-set. However, all headline results (Table 1, Figs. 3–4) use a fixed 1/16 ratio. The learned version is worse on three of five benchmarks (e.g., 54.4 vs 56.4 on TimeLens-Bench). It also appears to use fewer tokens (4.28% vs 6.25%, Appendix E), so the comparison is not at the same budget. Could the authors compare the two at the same token count?
  3. Limited generalization. P-frame tokens are newly learned features, not ViT tokens, so the LLM cannot use them without SFT. Unlike training-free token pruning, every new backbone needs the codec and the LLM to be retrained. The SFT data also overlaps in task type with most evaluations. On Video-MME-v2, the benchmark furthest from the training data, NativeCodec is only 0.5 above the untrained base, and the learned-length version is below it (23.7 vs 24.1).
Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Reply for BL9y

We sincerely thank the reviewer BL9y for recognizing our design rationale and insights.

We address your questions below.

  1. Could simple pooling achieve the same effect?

We apply the simplest pooling scheme: 4 × 4 spatial mean pooling to post-ViT tokens within each P-frame. This retains 1/16 of P-frame tokens. We train the same base model on these data at 16 fps.

MethodANCHQV
Fixed 1/16 at 16 fps54.553.867.7
Qwen3.5 4B SFT at 2 fps51.749.566.6
4 × 4 mean pooling at 16 fps51.249.165.2

The results do not surpass the original model at 2 fps. Spatial pooling operates independently within each frame, without temporal information from preceding frames, which may cause substantial information loss.

  1. Learned length allocation

The predictor produces varying token counts depending on video content, so we can only approximately align token budgets.

The limited downstream benefit of learned length allocation may arise because the lightweight predictor approximates the oracle policy imperfectly. Since reconstruction of later frames depends on representations retained from earlier frames, early length prediction errors, especially excessive compression, may reduce subsequent reconstruction quality. These errors can accumulate over time and limit the predictor's ability to realize the gains available under the oracle policy.

To demonstrate the advantage of variable token lengths, we train a downstream LLM using oracle search allocations. When the reconstruction oracle selects each frame's ratio, performance is slightly better than with fixed retention, indicating that content-adaptive allocation has the potential to improve performance further.

MethodANCHQV
Fixed 1/1654.553.867.7
Predictor52.751.466.2
Oracle55.354.668.2

The oracle determines the allocated length by searching reconstruction decoding results for multiple candidate ratios. This incurs substantial overhead in training label construction and at test time, making it unsuitable for practical inference. We move this search offline to construct supervision and train a lightweight predictor to predict ratios directly during inference, avoiding online reconstruction search.

We report oracle retention label and predictor output distributions at varied frame rates.

Oracle

FPS11/21/41/81/161/321/64Retention
16000.020.070.240.340.338.5
800.080.140.190.210.230.1521.7
40.050.070.210.330.180.090.0732.5
20.100.120.430.110.090.060.0945.0

Predictor

FPS11/21/41/81/161/321/64Retention
1600.010.010.030.180.300.477.6
80.030.040.100.140.190.240.2620.8
40.080.050.130.220.180.160.1831.1
20.120.070.300.120.110.120.1641.0

The predictor favors smaller retention ratios than the oracle. At 16 fps, it selects 1/64 for 47% of frames, compared with 33% for the oracle. Its average P-frame retention is 3.92%, compared with the oracle's 4.45%. Average retention is also lower at 8, 4, and 2 fps.

This bias may reflect the predictor's limited capacity and input information. We plan to explore richer contextual inputs and alternative predictor designs to improve its approximation of the oracle policy.

  1. Generalization

NativeCodec learns new incremental representations for P-frames, and downstream SFT aligns these representations with the LLM. This interface adaptation cost should be considered separately from generalization. Requiring integration without training would exclude learned visual interfaces from the outset and leave their performance and efficiency gains after training unevaluated.

The presence of similar task types in training and evaluation alone does not establish a lack of generalization. Assessing whether additional supervision explains the gains requires a same-data baseline. To address this task-type overlap, we include Qwen3.5-4B-SFT as a control trained on the same data. Under this matched training setting, our method achieves better results on Video-MME-v2 than the SFT baseline, providing sufficient evidence that it preserves the original model's generalization ability.

MethodBallBenchANCHQVAllMSVHMME-v2
Qwen3.5-4B12.249.446.064.650.718.539.524.1
Qwen3.5-4B-SFT15.351.749.566.653.218.844.023.3
NativeCodec-4B26.154.553.867.756.425.047.124.6

If the above responses address your concerns, we kindly ask you to consider raising your score.

AuthorsOpenReview ↗

Follow-up on our rebuttal to Reviewer BL9y

Dear Reviewer BL9y,

Thank you again for your time and feedback. We posted our rebuttal on November 11, including the 16-fps pooling control, the same-data SFT comparison, and additional oracle/predictor allocation results addressing your questions.

Could you please revisit our response and let us know whether it addresses your concerns, or which points need further clarification? Even a brief reply before the November 18 deadline would be very helpful, and we would welcome the opportunity to respond to any remaining questions.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal Before the November 18 Deadline

Dear Reviewers BL9y, GTcU, and uAhk,

Thank you again for your time and thoughtful feedback on our paper.

We posted our point-by-point responses on November 11, including additional matched-training baselines, ablation experiments, and analyses of the oracle and predicted token allocations.

With the November 18 deadline approaching, we would appreciate it if each of you could revisit our responses and share your follow-up feedback. We have not yet received any replies, and would welcome the opportunity to clarify any remaining concerns before the deadline.

Could you please let us know whether our clarifications and additional results address your concerns, or which issues remain unresolved? Even a brief response would help us focus any further discussion.

Thank you for your time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of rebuttal updates for the Area Chair

Dear Area Chair,

We thank you and the reviewers for the careful reviews.

We propose NativeCodec, a data-driven approach inspired by traditional video codecs. It learns a causal, variable-length visual-token representation that preserves dense temporal evidence within a limited token budget. It combines complete I-frame anchors with compact P-frame representations conditioned on earlier frames. Multi-ratio pixel and feature reconstruction trains these representations to remain useful at different lengths. A reconstruction-supervised predictor allocates tokens to each frame, and a cumulative-token rule determines anchor placement. We also introduce BallBench, a controlled benchmark for tracking identities through collisions between visually identical objects. Together, these contributions provide a way to reduce redundancy while retaining the observations needed for video understanding.

Below, we summarize the reviews, the experiments and analyses provided in rebuttal, and the conclusions supported by these results. Details are in our responses.

  1. Frame rate and pooling (BL9y, uAhk).

    The reviewers asked whether higher input frame rates could explain the improvements. They also asked whether simple spatial pooling could achieve the same effect.

    We trained a native-token control on the same data at the same input frame rate. We also added a spatial-pooling baseline under the same training conditions.

    NativeCodec outperforms both controls. These comparisons support the contribution of learned causal incremental representations.

  2. Matched downstream supervision (BL9y, uAhk).

    BL9y asked whether overlap in training task types explains the results. uAhk requested a BallBench baseline fine-tuned on the same data.

    We evaluated a baseline trained on the same downstream data on BallBench across multiple frame rates. We also compared this baseline with NativeCodec on Video-MME-v2.

    NativeCodec retains its BallBench advantage at every tested frame rate. It also performs better on Video-MME-v2. These results show that its advantages persist under matched downstream supervision.

  3. Variable-length allocation and token efficiency (BL9y, uAhk).

    The reviewers asked whether variable token lengths offer benefits over fixed retention.

    We added a comparison between oracle allocation and fixed retention.

    Oracle allocation improves temporal grounding performance. It also retains fewer visual tokens. These results demonstrate that variable-length token allocation can improve both accuracy and token efficiency.

  4. Reconstruction criterion and ablations (GTcU).

    GTcU asked whether reconstruction quality is a useful criterion for retaining information needed for downstream reasoning. The reviewer also requested ablations of the reconstruction tolerance and losses.

    We analyzed reconstruction quality and downstream performance across retention ratios. We compared joint pixel-and-feature training with feature-only training. We also examined sensitivity to the reconstruction tolerance.

    Reconstruction quality and downstream performance show broadly corresponding trends. Adding pixel reconstruction improves downstream performance under strong compression. The tolerance provides a way to control the retained token budget. These results support the reconstruction-based training and allocation design.

  5. Source of computational savings (GTcU).

    GTcU asked whether the savings mainly come from token reduction or from the encoder and parallel length prediction.

    We analyzed the roles of the inference components. We also referred to the paper's existing evaluation across visual-token budgets.

    Token reduction is the main source of computational savings in the language model. Parallel length prediction enables adaptive allocation without online reconstruction search. The budget evaluation shows that NativeCodec can achieve better temporal grounding performance with fewer visual tokens.

  6. Training, anchors, and frame-rate clarification (uAhk, GTcU).

    The reviewers asked why mixed-ratio training is used. They also asked about the rationale for anchor placement. uAhk asked how “encoding every video frame” relates to the experimental frame rate.

    We explained how mixed-ratio training supports different representation lengths within a sequence. We clarified how anchor placement follows the cumulative allocated token count. We also explained that the title describes encoding each frame supplied to the codec.

    These clarifications show how the training and anchor rules support the variable-length interface. The reported frame rate is an experimental operating point rather than a limit imposed by the coding mechanism.

We respectfully ask you to support acceptance of NativeCodec in light of the paper’s contributions and the additional evidence presented in our rebuttal.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.