acceptodds
Withdrawn from ICLR 2027

Memory Internalization via Test-Time Training for Long-Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video understanding often requires answering many questions about a single video whose evidence is distributed across extended temporal contexts. Conventional video-language models repeatedly encode frames or retrieve visual evidence for every question. Parameterized video memory offers a write-once alternative, but existing methods rely primarily on synthesized text and fixed adaptation configurations, leaving open where video knowledge should be written and how visual evidence should shape the memory. We introduce Memory Internalization via Test-Time Training (MIT), a per-video adaptation framework that writes each long video into compact, video-specific adaptive parameters for video-free, text-only question answering. Under compact multi-scale supervision, causal tracing measures the causal memory utility of each model layer for video-grounded answering and selects video-specific adaptation locations. Visual memory distillation then trains the text-only memory to reproduce the representation change induced by the relevant video clip, preserving the internal effect of visual evidence beyond textual supervision. Across LVBench, LongVideoBench, and Video-MME, MIT achieves data-efficient video-free question answering with substantially fewer synthesized samples than prior parameterized video-memory methods, demonstrating that effective video internalization depends on both where knowledge is written and how visual evidence is distilled into the model.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long-video understanding, Test-Time Training, Causal Tracing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cMEROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for long video understanding in a way that does not require the video at query-time. They do so by a test-time optimization process. For each video, they generate a large set of synthetic QA and next-token prediction samples, which are used to select a subset of informative layers of the video QA model, based on how much they increase performance when using the visual information versus a text-only question. Then, these layers are fine-tuned with LoRA in the query-only setting to distill their behavior when they are exposed to visual information. This way, the LoRA parameters represent the "memory" of the video at hand. The method is validated on the Qwen3 backbone on a few standard long video QA datasets.

Critical strengths
  • The paper and proposed method are well motivated.
  • The idea of distilling the representation change induced by visual context into the query-only setting is interesting.
Critical weaknesses

A key weakness is the performance and associated computational cost. The initial computation cost is very high, yet the model does not outperform the base model on average on any of the three datasets. If the goal is efficiency, the method should be compared against other methods for more efficient long video QA, rather than only NKR. Still, since the cost for optimization on one video is 70 minutes, as listed, one needs over a hundred questions for a single video for the method to make sense. Moreover, several of the presentation choices in the paper are somewhat misleading. For instance, the main paper lists Video-MME as a main experiment several times, yet the results are buried in the appendix. The computational cost of training for a single video is also in the appendix, despite the cost of 70 minutes per video for the smallest model being a crucial limitation. Similarly, baselines are not bolded when they outperform the proposed method, etc.

Questions for authors
  • What are the detailed results on Video-MME (i.e., how do other baselines perform, and how is the performance on short/medium/long videos)?
  • If the main goal of the method is lowering computational cost, how does the method compare to other efficient QA methods, such as adaptive token methods or even zero-shot ones like video panels? How many questions does one need to ask before the current method becomes worth it for different video lengths, and is this realistic?
  • What is the reason for the budget-averaged memory selection, i.e., why is it important that lower lambdas already produce better performance? What is the difference in performance to simply using lambda=1? Some other ablations are also missing, for instance, the value of alpha.
  • Does the method work for other models than Qwen3?
  • Why are only the local QA samples used for layer selection?
Reviewer_GUGgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Memory Internalization via Test-Time Training (MIT3), a framework for offline long-video question answering that internalizes video information into video-specific LoRA parameters. Unlike conventional video-conditioned QA or retrieval-based approaches such as Video-RAG, MIT3 enables text-only inference without accessing the original video at query time. The method automatically generates multi-scale descriptions and pseudo-labeled QA pairs from each video, allowing video-specific LoRA fine-tuning without human-annotated ground-truth labels.

Building upon Neural Knowledge Representation (NKR), MIT3 introduces two main components: (1) causal layer selection, which identifies Transformer layers suitable for storing video-specific knowledge, and (2) visual memory distillation, which transfers video-induced hidden-state changes into the LoRA adapter. The authors evaluate MIT3 on LVBench, LongVideoBench, and Video-MME using Qwen3-VL-8B and 32B. MIT3 improves LVBench accuracy over a controlled NKR-style baseline by 6.8 and 5.7 percentage points, respectively, while substantially reducing query-time latency compared with direct video-conditioned inference. Ablation studies further support the contributions of the proposed components.

In my understanding, the primary contribution lies in the mechanism for internalizing video knowledge into model parameters rather than in the test-time training formulation itself. While the results demonstrate improvements in aggregate QA accuracy and inference efficiency, they do not fully establish the limits of long-term memory retention, particularly for sparse events in very long videos, or the practical scalability of training a separate LoRA adapter for each video compared with retrieval-based memory approaches.

Critical strengths

Video memory internalization without human-annotated ground-truth labels.

MIT3 enables video-specific memory construction without requiring manually annotated ground-truth labels. By automatically generating pseudo-labeled descriptions and question-answer pairs, the method can adapt to new videos without costly human annotation. This is a practically valuable property for long-video understanding, particularly in scenarios involving large collections of recorded videos.

Furthermore, the proposed causal layer selection and visual memory distillation improve accuracy over the controlled NKR-style baseline under matched supervision, suggesting that the proposed memory-writing mechanisms contribute beyond conventional pseudo-label-based fine-tuning.

Critical weaknesses

1. Insufficient evaluation of long-term memory retention and its limitations.

Although MIT3 reports promising accuracy on long-video benchmarks, the experiments do not systematically investigate how memory retention degrades as video duration increases. In particular, Needle-in-a-Haystack (NIAH) evaluation is missing. Since MIT3 internalizes video information into fixed-capacity LoRA parameters, it is essential to determine whether sparse but important events can be reliably preserved as video length increases.The reported aggregate accuracy on LVBench and LongVideoBench, including their long-video subsets, is insufficient to characterize the capacity and limitations of the proposed parameterized memory. This is a critical concern because reliable retention of information distributed across long temporal contexts is central to the paper's main claim.

2. Limited evidence of practical scalability and advantages over retrieval-based video memory.

MIT3 requires training a separate LoRA adapter for each video. According to Appendix C, internalizing a one-hour video takes approximately 70 minutes with Qwen3-VL-8B and 145 minutes with Qwen3-VL-32B, using four NVIDIA A100 GPUs. Although the paper demonstrates substantial query-time acceleration and reduced persistent memory consumption, it does not establish whether these benefits justify the additional per-video training cost compared with offline indexing approaches such as Video-RAG (NeurIPS 2025). A comprehensive end-to-end comparison accounting for memory construction, query latency, accuracy, and storage cost is missing. This limitation is particularly important for the practical use cases highlighted in the paper, such as surveillance archives and recorded meetings, where large numbers of videos are continuously accumulated.

Questions for authors

Q1. Category-wise evaluation of video understanding capabilities

Could the authors provide category-wise accuracy on LVBench and Video-MME, particularly for fine-grained information retrieval, temporal ordering, counting, and multi-event reasoning?

The current evaluation primarily reports aggregate accuracy, making it difficult to determine which types of video information are effectively preserved by MIT3 and which are lost during memory internalization. Such results would help clarify the strengths and limitations of the proposed method.

Q2. Reliability of automatically generated pseudo-labels

How accurate are the automatically generated descriptions and QA pairs used for LoRA fine-tuning, and how sensitive is MIT3 to incorrect pseudo-labels?

Since the proposed method relies on automatically generated supervision, errors in the pseudo-labels may be internalized into the video-specific LoRA parameters. Could the authors evaluate pseudo-label quality through factual verification against the original videos, potentially using ground-truth-annotated Video QA datasets, and analyze how label noise affects downstream QA accuracy?

This would help establish the reliability of the proposed memory construction process.

Q3. Knowledge integration across multiple videos and domains

Can MIT3 integrate knowledge from multiple videos, particularly when the videos belong to different domains?

The current formulation trains a separate LoRA adapter for each video. However, practical offline Video QA often involves querying collections of videos rather than a single predefined video. It is unclear how the proposed framework would handle questions requiring information from multiple videos or how video-specific adapters could be combined without interference. Could the authors clarify whether such scenarios are supported by the current framework and, if so, provide evidence of their effectiveness?

Reviewer_UnHQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces MIT3, a test-time training framework for long-video understanding that writes each video into a video-specific LoRA adapter, so questions are later answered from text only, without the video. It builds on NKR and addresses 2 main concerns from that method: 1) where in the model video knowledge should be written and 2) how visual evidence, not only synthesized text, should shape the memory. The backbone is a frozen Qwen3-VL (8B and 32B). The method includes three main components: 1 - Video-specific data generation where a VLM produces multi-scale descriptions and local/global QA pairs for each video, following NKR's pipeline. 2 - Causal layer selection where the video-induced hidden-state change is injected at each layer to measure its effect on the answer, and the top-K layers are selected to host the LoRA. 3 - Visual memory distillation where the adapter is trained to reproduce, without the video, the hidden-state change that the video causes in the frozen model. The method improves over an NKR-style baseline trained on the same data and comes close to direct video inference on LVBench, LongVideoBench and Video-MME.

Critical strengths

1 - Large storage savings: a 0.59 GB adapter per video versus 26.9 GB for a stored KV cache. 2 - Controlled comparison: the NKR-style baseline is trained on the same generated data, so the gain comes from the method, not the data. 3 - Consistent gains over that baseline: across both backbones, all benchmarks and all data budgets. 4 - Visual distillation helps clearly (+4.5 to +5.3 points). It is a simple, sensible way to use the video beyond text. 5 - Data efficient: about 1/8 of NKR's training examples. 6 - Near-direct-inference accuracy without video: it even surpasses direct inference on the long subsets with the 32B backbone. 7 - Interpretable layer analysis: a shared core of layers across videos, with some per-video variation. This is important also about the behavious of VLMs in general so it is very interesting and insightful.

Critical weaknesses
  1. The practical value of the main use case is not demonstrated. The paper's motivation is "write once, query many", but the cost of writing is high. Under the reported numbers, adaptation only pays off after roughly 135 (8B) or 84 (32B) queries per video in wall-clock time, and several times more in GPU-time.

  2. Lower accuracy than simply using the video, at a much higher upfront cost. MIT3 stays below direct inference on LVBench with 8B (43.5 vs. 45.6) and on Video-MME (66.7 vs. 69.1). It surpasses direct inference only on the long-video subsets with 32B, by a small margin (+0.8 to +2.0). In most settings, then, the user pays a large one-time cost to get answers that are worse.

  3. Missing simple baseline, i.e., the same generated text without training to a VLM or an LLM (is vision needed once you have the text samples generated from the teacher). All knowledge in the adapter comes from teacher-generated descriptions and QA pairs, which already exist as text. The obvious baseline is to give this text to the same LLM/VLM through retrieval or in-context, with no layer selection, distillation or per-video training.

  4. The memory cannot be updated or re-grounded. Once the adapter is trained, nothing that the teacher's captions missed can be recovered, because the method never goes back to the frames. New footage requires running the full pipeline again. The conclusion claims "scalable access to growing videos", but no incremental update mechanism is proposed or evaluated.

  5. Per-video layer selection adds cost for limited, mostly video-independent benefit.

Final remarks: To be fair, I fail to see a clear use case of this method. The problem of multiple questions per video for long-videos is real, but I see no clear solution here. The method is fancy, well evaluated, great technical details and proves a point that we can encode knowledge into weights (not the first time though), but the overcomplication of it in test-time is almost assured that very few (if any) systems are going to use it.

Questions for authors

Look above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.