acceptodds
Withdrawn from ICLR 2027

LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long video understanding, multimodal language models, test time training

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_PYgKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LongVU-TTT, which inserts a convolutional test-time-training layer between the vision encoder and the language model. Grouped convolutional fast weights adapt within each video using a self-supervised alignment objective, contextualizing frame features before compression. Experiments cover five video benchmarks, controlled temporal-operator comparisons, mechanism and selection ablations.

Critical strengths
  1. Useful controlled architecture comparisons. The study compares several temporal operators under a shared training subset and input budget.

  2. An informative discussion of memory limitations. The evidence-withholding diagnostic supports interpreting fast weights as temporal aggregation rather than dependable episodic storage, motivating explicit evidence retention. Evaluation spans five established video benchmarks.

Critical weaknesses
  1. The video-model baselines are dated. Comparisons mainly cover 2024–2025 methods and omit relevant 2026 approaches.

  2. The benefit of longer pre-aggregation is unclear. The 512→128 setting gains only 0.30 MLVU points over direct 128→128 processing and loses 0.62 LongVideoBench points. The frame-independent 512→128 baseline also degrades substantially. A matched 128-frame baseline and end-to-end cost comparisons are needed to justify processing four times as many frames.

  3. The parallel update rule is underspecified. Equation (3) does not identify the weights at which gradients are evaluated. Chunk-start gradients and sequential online gradients generally define different updates. Explicit pseudocode for gradient evaluation, momentum, stabilization, and chunk boundaries is needed to assess the claimed update behavior.

Questions for authors
  1. How does the method compare with relevant 2026 long-video compression approaches?

  2. What is the result of a frame-independent 128→128 baseline under matched training and token budgets? Why does the 512→128 baseline degrade, and what accuracy–latency trade-off justifies 512-frame pre-aggregation over direct 128-frame TTT?

Reviewer_8eGsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes LongVU-TTT, a convolutional test-time training module that contextualizes video features before compression using causal fast-weight updates. It combines gradient-based frame selection with uniform sampling to reduce up to 512 frames to 128 LLM input frames. Experiments cover five video-understanding benchmarks, with operator ablations and a diagnostic of temporal information retention.

Critical strengths

The controlled operator comparisons and ablations provide useful evidence that adaptive state propagation and spatial locality improve performance in the evaluated setting. The evidence-withholding diagnostic also usefully distinguishes temporal aggregation from reliable long-range episodic retention.

Critical weaknesses
  1. Table 1 does not establish whether LongVU was trained using the same SFT data and protocol as LongVU-TTT, so their performance gap cannot be attributed specifically to the proposed method.

  2. Table 1 compares LongVU-TTT’s 512→128 frames with LLaVA-Video’s 64 frames, while Table 3(a)’s 128→128 and 512→128 settings change both input length and selection policy, leaving these factors confounded with the claimed gains.

  3. The paper provides no comparison of time to first token (TTFT) or end-to-end inference latency against other models, and the training speedups in Figure 2 do not establish whether the additional visual processing and fast-weight updates offer a favorable inference accuracy–latency trade-off.

Questions for authors
  1. Was LongVU retrained using the same SFT data, training protocol, and training budget as LongVU-TTT, and if not, what evidence separates the proposed method’s contribution from training differences?

  2. Can the authors provide comparisons with matched input-frame counts, LLM visual-token budgets, and selection policies to isolate the contribution of the TTT module?

  3. How does LongVU-TTT compare with LongVU and LLaVA-Video in TTFT and end-to-end inference latency under the same hardware and evaluation conditions, including visual encoding, fast-weight updates, frame selection, and answer generation?

Reviewer_EXVHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces LongVU-TTT, which inserts a causal convolutional Test-Time Training module between a vision encoder and an LLM to contextualize video features before compression. Its recurrent state consists of grouped 2D convolutional fast weights that adapt to each video. This design aims to complement the ViT's representations with local spatial processing and temporal information propagation. A hybrid frame-selection strategy then retains a subset of frames for the LLM. The paper evaluates video understanding performance and compares temporal operators and components of the fast-weight mechanism.

Critical strengths
  1. An interesting application of fast-weight adaptation to video processing. The use of video-adaptive convolutional parameters as recurrent memory is a potentially useful application of TTT. Combining this adaptation with spatial processing before token compression is an interesting aspect of the submission.

  2. A relevant controlled-ablation design. Table 2 compares several temporal operators and investigates adaptation, state carry, and spatial locality. Its stated controls over the base model, training data, frame input, and LLM token budget address relevant questions about the method's contribution, although the interpretation of these results requires further clarification below.

Critical weaknesses
  1. The learning mechanism and its justification are insufficiently specified. Equation (2) gives a negative inner-product objective, but the description does not make clear how the inner product is reduced over spatial positions and channels, whether its arguments are normalized, or what constraints control their magnitudes. As written, without additional constraints, the objective is unbounded below when an aligned output is scaled up. This does not by itself establish instability under a finite number of updates, but the paper needs to explain the implemented objective and the mechanisms that make adaptation well behaved. Since the key and value projections are learned from the same frame, I also find the explanation insufficient to understand how outer training discourages trivial mappings and makes the updates useful for temporal reasoning. My concern is about the justification and specification of this learning process, rather than a demonstrated identity-collapse result.

    The motivation for local convolution also needs clarification. The ViT already includes spatial positional information and within-frame interactions. Although these do not make local convolution redundant, preserving a 2D grid alone does not explain its expected benefit. A clearer account of the complementary information or inductive bias, connected to the locality ablations, would strengthen the method's motivation.

  2. The experiments do not yet clearly isolate the contribution at the scale of the main results. Table 1 compares LLaVA-Video using 64 frames with LongVU-TTT processing up to 512 frames and retaining 128. The exact initialization, additional training history, and correspondence between the two models' training mixtures are not sufficiently documented. Consequently, the improvements cannot be attributed solely to the proposed temporal operator: changes in training and available visual evidence may also contribute.

    Table 2 is more directly relevant because it states that training and input conditions are controlled. However, it uses only 30% of the Stage 2 data, and its LLaVA-Video baseline is substantially below the Table 1 result, for example 60.35 versus 70.80 on MLVU. The paper does not adequately explain this discrepancy or establish whether the operator rankings persist with sufficient training. Lower scores alone do not demonstrate non-convergence, but the unclear relationship between these settings limits how confidently I can extrapolate the ablation results to the complete system.

Questions for authors

The following clarification requests could materially affect my assessment:

  1. What is the exact implemented inner objective, including reduction, normalization, and any constraints or stabilization of the fast weights and outputs? How does outer training make the learned key/value projections and their induced updates useful for temporal reasoning rather than trivial mappings?
  2. What initialization checkpoint and training schedule are used for each baseline in Tables 1 and 2? Does the full-scale comparison include a no-TTT model given matched additional training and input budgets? Please explain the drop in the Table 2 baseline and provide evidence that the reported operator rankings are not an artifact of the reduced training budget.
  3. What specific benefit is expected from local 2D processing beyond the ViT's existing spatial representations, and how do the 1-by-1 and shuffled-grid ablations support that explanation?
Reviewer_aT8BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) layer between the vision encoder and the LLM of a video MLLM. The TTT-Conv layer parameterizes fast weights as grouped convolutions operating on the 2D patch grid, updating them causally at each frame via a self-supervised alignment objective. The inner-loop gradient signals double as a frame-importance measure (gradient-direction shift + alignment-loss delta), driving a hybrid saliency-uniform frame selector that compresses 512 frames to 128. System-level optimizations enable 512-frame training on 40GB GPUs. Controlled comparisons show TTT-Conv is a strong temporal aggregator.

Critical strengths
  1. The grouped-convolution fast-weight parameterization is a meaningful departure from TTT-MLP. Preserving the 2D spatial structure of ViT patches during fast-weight propagation is intuitive.
  2. The gradient-signal frame selection is interesting. Using the inner-loop gradient shift and alignment-loss delta as free importance scores is a natural byproduct of TTT and avoids extra parameters.
  3. Practical system contributions. Batch-tiled ViT processing and asynchronous CPU activation offloading extend trainable frame length from 64 to 256 on 40GB GPUs, which matters for reproducibility on commodity hardware.
Critical weaknesses
  1. The "causal" claim is weakly supported by ablation. Table 2a shows TTT-Conv (causal)=65.50 MLVU vs. TTT-Conv (non-causal)=65.62 MLVU. If causal ordering barely matters, the emphasis on "Causal Test-Time Training" is misleading.
  2. 512 frames is not "long" by current standards. Several modern video MLLMs (InternVL2, LongVU, Video-XL) process 1024+ frames. The paper's claim of "long video understanding" should be contextualized, or the system optimizations should be pushed further to reach 1024+ frames.
  3. Inference latency and throughput are not reported. The paper only reports training-side optimizations. At inference, processing 512 frames through a recurrent TTT-Conv layer is itself non-trivial. What is the end-to-end inference time per video compared to a baseline that simply samples 128 frames and processes them independently?
  4. Comparison to prior work in Table 1 is not apples-to-apples. The authors acknowledge this as a "contextual comparison," but several listed methods (InternVL2-1B, VideoChat, Qwen2-VL) use different training data amounts, frame policies, and model sizes. The strongest claim should rest on Table 2 (controlled study) rather than Table 1.
  5. Missing direct comparison with LongVU. LongVU is the most directly related prior work (structured frame compression via DINOv2 features+text-guided selection), but it is cited only in related work without an empirical comparison. This is a notable omission.
  6. The mixing weight α=0.4 is set "empirically" without sensitivity analysis. Given that G_t and R_t capture different temporal scales (scene-level vs. event-level), a sensitivity sweep over α would be useful.
  7. No analysis of what the fast weights actually learn. The paper shows that the TTT-Conv improves representations, but there is no qualitative analysis of the learned fast-weight maps, what spatial regions they emphasize, or whether they correspond to semantically meaningful regions (e.g., moving objects, talking faces).
Questions for authors
  1. Causality: In Table 2a, non-causal TTT-Conv (65.62) slightly outperforms causal TTT-Conv (65.50). Why does the paper emphasize causality in the title? Are there tasks or settings where causality is essential (e.g., streaming / online video)?
  2. Inference cost: What is the end-to-end inference latency per video (512 frames through TTT-Conv + 128 frames through LLM) compared to a uniform-128 baseline? Is the recurrent TTT-Conv the bottleneck?
  3. Frame budget: The system supports up to 512 training frames. With the batch-tiled ViT + async CPU offloading, what prevents pushing to 1024 or 2048 frames? What is the next bottleneck?
  4. α sensitivity: How does performance vary as α ranges from 0 to 1? Is α=0.4 optimal on all five benchmarks or just MLVU?
  5. LongVU comparison: Why is there no direct empirical comparison with LongVU, which also performs structured frame compression? Does LongVU's approach outperform or underperform your saliency-based selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.