acceptodds
Withdrawn from ICLR 2027

Beyond Uniform Pooling: Typed Object and Relation Evidence for Frozen Video LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video language models spend a fixed visual-token budget on uniformly pooled patches or learned latents, so the tokens a model reads carry no record of which object, moment, or relation they describe. We present StateWeaver, which replaces part of the uniformly pooled grid with typed evidence that a frozen language model reads within the same token budget. Each video is parsed once, with frozen trackers and a frozen image encoder, into a reusable EvidenceBank of scene features, tracked objects with state timelines, and pairwise relations. For each question, an answer-free retriever takes the dense connector’s own 384-token sequence, keeps 256 positions as question-independent context, and writes native scene, object, and relation-selected object features onto the other 128; only the connector is trained. On the official NExT-QA test, StateWeaver reaches 58.1%, 15.0 points above the 384-token dense connector and 8.7 points above a resampler that reads the same memory and question, and it also improves three further video benchmarks. A factorial ablation finds positive main effects for object evidence, relation-selected object appearance, and the posterior teacher; beyond that teacher, conditioning retrieval on the question adds no separable benefit. All comparisons use one frozen 8B backbone at one 384-token budget, and the resampler contrast changes the layout, the write, and the training objective together.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLMs, Video Understanding, Video Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L8h3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes StateWeaver, a structured visual connector for frozen video language models. Frozen segmentation, tracking, and image-encoding models first construct a reusable EvidenceBank containing scene features, object appearances and states, and ordered object relations. At question time, a hierarchical retriever selects native visual features and writes them through a gated interpolation into 128 positions of a 384-token dense representation, preserving the remaining 256 positions as question-independent context.

Critical strengths
  1. The connector design provides a concrete way to incorporate structured evidence under a fixed token budget.
  2. The statistical reporting and evaluation protocol are carefully documented.
  3. The paper provides useful evidence of visual dependence and reports limitations candidly.
Critical weaknesses
  1. The distinctive contribution of typed retrieval remains incompletely isolated. The stronger controls substantially reduce the original gap, but the remaining 3.4-point advantage still combines typed quotas, native-value selection, anchored interpolation, and retrieval distillation. The strongest resampler control does not receive the KL term and produces learned latent outputs rather than the same native-valued writes.
  2. The posterior teacher’s ability to improve retrieval quality is not directly demonstrated. Section 3.3 and Table 8 explicitly state that hard candidate selection blocks language-loss gradients to the retrieval scores. The retrieval metric receives KL and diversity gradients, while the posterior teacher uses the same parameters and receives language supervision only through the gate and shared query pathway.
  3. The strongest attribution results have limited demonstrated generality. The study uses one frozen Llama-3.1-8B/SigLIP2 combination, and StateWeaver is evaluated at one total token budget. Although the final method improves all three transfer benchmarks, the strongest world-preserving, auxiliary-objective-matched resampler comparison is reported only on NExT-QA.
Questions for authors
  1. Can the authors compare against an untyped native-value selector that preserves the same world positions, anchored gate, candidate access, and distillation objective?
  2. Could a question-blind student still be distilled from a question-and-answer-conditioned teacher?
  3. Do the gains over the strongest world-preserving, auxiliary-objective-matched resampler persist on MotionBench, LongVideoBench, and TVBench? How does the comparison change at a different total visual-token budget?
Reviewer_EmpTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes StateWeaver that uses structured video information to improve video qa under a fixed visual token budget. It first builds EvidenceBank to contain scene features, object tracking, object states, and relations. For each question, this paper trains a connector to select relevant evidence and writes it into 128 tokens. The method also uses a posterior teacher in training to guide the retrieval. On NExT-QA, StateWeaver outperforms the dense-384 connector by 15.0 points and a resampler using the same EvidenceBank by 8.7 points. It also improves performance on three additional benchmarks.

Critical strengths
  1. The design of gated writing is interesting since the write is in place and gated. Closing the gate can transfer the entire algorithm to a standard dense-384. This design allows an "anchors-only" to serve as a controlled ablation, and the evidence write can be evaluated at matched training.
  2. The improvement of 15 points over dense-384 and 8.7 points over resampler+EvidenceBank is kind of significant. The paper also provides a more rigorous matched objective control. It shows even with the same identical memory, world positions, and the majority of training objectives, StateWeaver (without QEC) still has an improvement of 3.4 points.
  3. It is good that the authors claim the limitations and negative findings honestly. For example, the authors explicitly claim that the current experiments cover only one frozen backbone and one primary token budget.
Critical weaknesses
  1. The largest concern for me is the contribution of the posterior teacher is not clear. The paper uses a posterior teacher to supervise an answer-free retriever, but there is no good explanation for why the teacher's retrieval distribution can provide better answer-relevant evidence. Based on Section 3.3 and Table 8, the LM loss does not directly update the retrieval scores, and the retrieval metric is mainly trained using KL distillation and a diversity penalty. Since the teacher and student use the same planner, minimizing the KL loss does not necessarily mean the teacher selects more evidence that is helpful to answer the questions. Also, the +1.56 points teacher effect in Figure 3b combines KL distillation and the additional teacher LM loss. The contribution of each is not isolated now.
  2. Another major concern is the contribution of the typed retrieval is not isolated either for now. If I understand the paper correctly, the typed retrieval is one of the major contributions of StateWeaver, but there is no ablation to show the difference between typed retrieval and untyped retrieval, and why we have to allocate fixed token numbers for each type. The flat retriever in Table 4 still uses typed branches and fixed token quotas, while Memory + resampler also changes the writing mechanism and training objective.
  3. The output of the relation branch does not directly contain relation states. Although the relation state in StateWeaver contains information such as distance, direction, and speed, these features are only used to select object pairs, while the final output to LLM is the object appearance features. Thus, the +2.75 points improvement might come from providing additional object appearance features rather than the relation-based selection itself. The current ablation compares the relation branch with dense anchors, but not with additional object features under the same token budget.
Questions for authors
  1. Could you separately ablate the KL distillation and teacher LM loss to show the contribution of each? Also, it would be good to explain how the posterior teacher learns to select more answer-relevant evidence when the LM loss does not directly update the retrieval scores.

  2. Could you compare typed retrieval with an untyped retriever under the same token budget and training settings?

  3. Could you explain why fixed token quotas (32/64/32) are necessary, or compare them with question-adaptive token allocation?

  4. Could you compare the relation branch with using the same 32 token positions for additional object features?

  5. Could you evaluate StateWeaver under different visual-token budgets (e.g., 128, 256, 384, and 512) and show how the accuracy and inference cost scale with the token budget? I think it is interesting to see the scaling trend of StateWeaver.

Reviewer_9iJ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about how video language models squeeze a whole video into a small number of tokens before the language model reads it. Normally every frame just gets shrunk to the same small grid, no matter what the question is, so the model never really knows which token is about which object or what’s happening between objects. The authors’ method, StateWeaver, first runs each video once through some off-the-shelf tools (SAM 2 for tracking objects, CoTracker3 for motion, SigLIP2 for image features) and saves the result as a reusable memory they call the EvidenceBank. It holds background scenes, the tracked objects over time, and pairs of objects that interact. Then, for each question, a small retriever picks the most useful scenes, objects, and object pairs from that memory. It blends them into 128 of the 384 tokens the model would normally see and leaves the other 256 alone. The language model, Llama-3.1-8B-Instruct, is kept frozen the whole time, and only the connecting part in the middle gets trained. During training they also run the retriever with the correct answer visible and teach the normal version to copy its choices.

On NExT-QA the method gets 58.1%, which is 15 points better than the simple shrink-every-frame approach with the same number of tokens, and 8.7 points better than a learned resampler that reads the same memory. It also helps a bit on MotionBench, LongVideoBench, and TVBench. The experiments are done really carefully, with lots of controls, three seeds, and proper statistics, and the authors are pretty honest about what works. Most of the gain seems to come from feeding in the object information. Actually tailoring the picks to the question adds very little on its own. The big catch is that everything runs on one frozen, fairly old text-only LLM, so all the accuracies are way below what real video models get today, and it’s not clear the gains would hold up in a normal setup where the language model is trained too.

Critical strengths

The paper presents a well-motivated idea for spending a video LLM’s token budget on object-level evidence instead of uniform pooling. It backs that idea with an unusually careful experimental study. The main strengths are:

  1. The evidence is written on top of the tokens the dense connector already produces, through a learned gate. When the gate is closed, the model reduces exactly to the dense baseline. This makes the comparison clean: the 12.7-point gain over the closed-gate version can be attributed to the written evidence rather than to other changes in the architecture.
  2. The experimental design is rigorous. The authors compare against a resampler that reads the same EvidenceBank, and they turn the object branch, relation branch, and teacher on and off in all eight combinations. Results are reported over three seeds with paired confidence intervals and a correction for testing multiple effects. They also excluded NExT-QA’s training videos from training and removed duplicate videos, which reduces the risk of data leakage.
  3. The claims are carefully calibrated. The authors state openly that conditioning the retrieval on the question adds little beyond the teacher. They also acknowledge that the relation branch writes object appearance rather than the relation itself, and that absolute accuracies are not comparable to fully trained video models. This transparency makes the reported findings more credible.
Critical weaknesses

The paper is careful within its own setting, but several issues limit how far the conclusions reach. The main weaknesses are:

  1. Everything is tested on a single, fairly old, frozen text-only LLM (Llama-3.1-8B-Instruct), with only the connector trained. In this setting even the best model reaches 58.1% on NExT-QA, far below current video LLMs that fine-tune the language model. It’s unclear whether the gains would survive in a more standard setup, for example with LoRA tuning or a pretrained video LLM. A second, more recent backbone would make the results much more convincing.
  2. The dense baseline is weak. It’s a single linear layer over a 4×4 grid per frame, only about 12 points above the question-only accuracy of 31.5%, so a large gain over it is not very surprising. A higher-capacity dense connector, or one trained with the same auxiliary losses, would be a fairer comparison.
  3. Much of the gain may come from extra visual information rather than from the retrieval itself. StateWeaver gets object features from SAM 2 and CoTracker3 pooled over the full 27×27 patch grid, which the dense baseline never sees. The question-blind planner is only 0.9 points below its matched arm. A simple baseline that writes features of, say, the largest tracked objects, with no retriever or teacher, is needed to show the typed retrieval actually matters.
  4. The method is mostly a combination of existing ideas. It draws on object tokens from trackers (Slot-VLM, VideoOrion, SlotNarrative), a reusable object memory (VideoAgent), answer-aware teacher distillation (Hindsight, SeViLA), and gated blending (Flamingo). The pipeline is also engineering-heavy: four training stages, about seven loss terms, and many hand-designed choices. Some of these add little, such as the final question-episode stage, which adds only 1.0 point with an interval touching zero.
  5. There is no comparison to any published method. Even SlotNarrative, which the authors themselves call the closest work, is not reimplemented in the same setting. This makes it hard to know where the method stands relative to the field.
  6. The evaluation is fairly narrow. NExT-QA is an older benchmark that is partly solvable from single frames, and newer benchmarks like Video-MME or HERBench are missing.
Questions for authors

The query includes all five answer options, and the question-blind planner is only 0.9 points behind. How different are its picks from the full model’s? And how much of the selection do you think comes from the options rather than the question itself?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.