- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes StateWeaver, a structured visual connector for frozen video language models. Frozen segmentation, tracking, and image-encoding models first construct a reusable EvidenceBank containing scene features, object appearances and states, and ordered object relations. At question time, a hierarchical retriever selects native visual features and writes them through a gated interpolation into 128 positions of a 384-token dense representation, preserving the remaining 256 positions as question-independent context.
Critical strengths
- The connector design provides a concrete way to incorporate structured evidence under a fixed token budget.
- The statistical reporting and evaluation protocol are carefully documented.
- The paper provides useful evidence of visual dependence and reports limitations candidly.
Critical weaknesses
- The distinctive contribution of typed retrieval remains incompletely isolated. The stronger controls substantially reduce the original gap, but the remaining 3.4-point advantage still combines typed quotas, native-value selection, anchored interpolation, and retrieval distillation. The strongest resampler control does not receive the KL term and produces learned latent outputs rather than the same native-valued writes.
- The posterior teacher’s ability to improve retrieval quality is not directly demonstrated. Section 3.3 and Table 8 explicitly state that hard candidate selection blocks language-loss gradients to the retrieval scores. The retrieval metric receives KL and diversity gradients, while the posterior teacher uses the same parameters and receives language supervision only through the gate and shared query pathway.
- The strongest attribution results have limited demonstrated generality. The study uses one frozen Llama-3.1-8B/SigLIP2 combination, and StateWeaver is evaluated at one total token budget. Although the final method improves all three transfer benchmarks, the strongest world-preserving, auxiliary-objective-matched resampler comparison is reported only on NExT-QA.
Questions for authors
- Can the authors compare against an untyped native-value selector that preserves the same world positions, anchored gate, candidate access, and distillation objective?
- Could a question-blind student still be distilled from a question-and-answer-conditioned teacher?
- Do the gains over the strongest world-preserving, auxiliary-objective-matched resampler persist on MotionBench, LongVideoBench, and TVBench? How does the comparison change at a different total visual-token budget?