acceptodds
Under review as a conference paper at ICLR 2027

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

Abstract

Long-video embedding requires capturing sparse query-relevant evidence under a limited visual-token budget. Uniform sampling can miss brief events in videos spanning minutes or hours, whereas encoding more frames in a single context increases memory and computation. We introduce Query-Aware Streaming Latent Reasoning (QASLR), a post-training framework that accumulates evidence across clips while keeping the embedding size fixed. QASLR selects a bounded set of frames, restores their temporal order, and processes them clip by clip with a vision-language backbone. A compact set of persistent think tokens cross-attends to each clip's features, while an embed token reads out a normalized representation after every update. This design integrates evidence across multiple backbone calls without requiring all selected frames to share a single context. Training combines final contrastive learning, step-wise contrastive supervision, and final-embedding self-distillation. Intermediate supervision trains partial-video readouts for retrieval, while self-distillation regularizes them toward the final representation. Query-aware selection produces query-conditioned representations for candidate-set scoring and reranking, whereas query-independent selection enables reusable corpus indexing. Under the full training recipe, HourVideo retrieval Hit@1 increases from 54.2 to 70.7 and from 57.4 to 72.8 for 2B and 8B Qwen3-VL-Embedding backbones, respectively. Gains extend to the evaluated moment-retrieval and video-QA tasks, and the streaming head transfers to a second Qwen-family embedding backbone. These results support streaming latent aggregation as an effective approach to integrating long-video evidence into fixed-dimensional representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM Embedding, long video understanding, video retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_i1YAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Query-Aware Streaming Latent Reasoning (QASLR), a post-training framework for long-video embeddings. Selected frames are divided into clips in temporal order and encoded by a frozen vision-language backbone. The resulting features are compressed into evidence tokens that update a fixed-size persistent memory of learnable think tokens, from which an embedding is read out after each clip. Training combines final contrastive learning, step-wise contrastive supervision, and self-distillation from the final embedding. The framework supports both query-independent indexing and query-conditioned candidate scoring. Experiments on nine tasks using Qwen3-VL-Embedding 2B and 8B, as well as RzenEmbed 7B, report improved retrieval performance, while gains over sub-batched single-round post-training are less consistent for video QA.

Critical strengths
  1. The paper addresses a useful problem: combining evidence from multiple video clips into a fixed-size embedding. The persistent think-token memory provides a clear way to retain information across clips without placing all selected frames in a single backbone context.
  2. The experiments provide evidence that cross-clip aggregation is useful for retrieval. QASLR improves retrieval performance on both the 2B and 8B backbones, and the appendix compares alternative aggregators and tests memory-reset controls. These results support the value of the proposed memory mechanism.
Critical weaknesses
  1. The novelty of QASLR is not sufficiently clear. The paper should better explain what is new in its memory design and training objectives.
  2. The evaluation lacks comparisons with strong existing methods. The architecture and training settings of single-round PT also need more detail. In addition, the role of sub-batching should be clarified so that the gains from streaming latent reasoning can be distinguished from those of the training strategy.
  3. The experiments do not fully address the challenges of sparse sampling and costly frame encoding. Most results use at most 128 selected frames, so it remains unclear whether QASLR can effectively use more frames at a reasonable cost. Experiments with larger frame budgets and ablations of the think-token count and clip size would help clarify this trade-off.
Questions for authors
  1. Comparisons with strong existing methods are missing. Could you compare QASLR with relevant long-video retrieval or aggregation methods under the same evaluation protocol?
  2. Is sub-batching a core contribution or a training strategy supporting streaming latent reasoning? If it is the latter, the main paper should place more emphasis on ablations of the streaming mechanism itself.
  3. Could you provide more details on single-round PT, including its architecture, trainable modules, frame budget, batch size, negative pool, and training schedule?
  4. The paper motivates QASLR by missed events under sparse sampling and the cost of encoding more frames. However, it mainly improves aggregation across selected clips, while using at most 128 frames and adding a streaming head. Could you clarify how this design addresses these limitations? Experiments with larger frame budgets and ablations of the think-token count and clip size would help assess the trade-off between performance, memory, and computation.
Reviewer_QXDiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Query-Aware Streaming Latent Reasoning (QASLR), a parameter-efficient post-training framework to extend vision-language embedding models to long videos. Existing methods typically place all selected frames into a single context window. In contrast, QASLR uses query-aware or uniform frame selection to extract up to 128 frames, partitions them chronologically into clips, compresses per-clip hidden states from a frozen VLM backbone into fixed-size evidence banks, and recurrently updates a compact set of learnable "think tokens" via cross-attention before reading out normalized step-level embeddings. The streaming head is trained with a unified query–positive interface combining a terminal InfoNCE loss, step-wise intermediate contrastive supervision, and online cosine self-distillation from the final-step embedding. This demonstrates empirical improvements across retrieval, moment retrieval, classification, and video QA benchmarks on Qwen3-VL-Embedding (2B and 8B) and RzenEmbed (7B).

Critical strengths
  • This paper aims to solve a highly practical problem in real-world long video retrieval. It decouples temporal video coverage from VLM context memory limits using a lightweight, parameter-efficient streaming head over a frozen backbone.
  • The performance gain is substantial. +11.8 Hit@1 points at the 25% video observation checkpoint on HourVideo.
Critical weaknesses

My main concern is regarding the novelty of the proposed method. The main contribution of this paper is streaming head and three part loss function.

  • Streaming head: The streaming head directly combines Perceiver IO cross-attention readouts (Jaegle et al., 2022) with recurrent memory tokens (Bulatov et al., 2022) and cosine-similarity token compression from prior memory-augmented video MLLMs such as MA-LMM (He et al., 2024).
  • Three-Part loss function: The training objective directly adapts online depth-wise self-distillation (Zhang et al., 2019) and progressive partial-video teacher–student supervision (Progressive Teacher-Student Learning for Early Action Prediction, Wang et al., 2019) to the clip-streaming axis.
Questions for authors

My main question to the authors is regarding the novelty of the proposed method. And it would be helpful to further clarify.

Reviewer_B8KLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Query-Aware Streaming Latent Reasoning (QASLR), a method designed to aggregate scattered evidence from long videos into fixed-dimensional embeddings, given a constrained budget for visual inputs. The approach begins by selecting up to 128 frames, restoring their temporal order, and partitioning them into segments; features are then extracted using a frozen vision-language backbone. A persistent latent memory of fixed size is updated segment-by-segment via cross-attention, while dedicated embedding tokens read from this memory at each step to produce a normalized representation. The training objective incorporates a final contrastive loss, intermediate-step contrastive supervision, and self-distillation aligned toward the final representation.

Critical strengths
  1. There are consistent retrieval gains over an internal baseline that employs the same sub-batching strategy.
  2. Ablation studies concerning the aggregation structure and auxiliary objectives provide some justification for the design.
Critical weaknesses
  1. The current experiments primarily compare raw embedding checkpoints, a single-turn post-training baseline constructed by the authors, and a generic aggregator. Relying on a stronger pre-trained backbone is no substitute for direct comparisons against methods specifically designed for long-video memory, compression, or query-conditioned re-ranking; in the absence of such benchmarks, the paper has yet to establish its competitive advantage over existing approaches.
  2. Appendix A.3 indicates that classification, retrieval, and moment retrieval tasks utilize subsets constructed through sampling, filtering, and partial assistance from GPT-4o. However, the paper fails to provide task-specific details such as exact sample sizes, candidate generation rules, the difficulty level of negative examples, filtering criteria, or quality control measures for annotations. Since video retrieval typically involves only 4 or 8 candidates—and moment retrieval only 8 or 16 candidate segments—these metrics primarily reflect the ranking of a limited candidate set; they cannot be directly equated to full-database retrieval or continuous temporal localization capabilities. In the four-candidate scenario, a Hit@5 score is guaranteed, which means the interpretation of this metric is contingent upon the distribution of the number of candidates. Although Table 4 introduces a two-stage retrieval process, it lacks specific information regarding the size of the candidate pool, the model used in the first stage, and the dimensions of the shortlist. Furthermore, the results are reported only as averages across three random seeds, with no estimates of uncertainty. These omissions directly affect the assessment of the credibility and significance of the study’s primary experimental conclusions.
Questions for authors
  1. Could you provide the exact number of evaluation samples, the number of unique videos, the distribution of the number of candidates, the method for selecting negative examples, and the filtering criteria for each task?
  2. In the context of temporal retrieval, how are the boundaries of candidate segments generated, and is a "hit" determined by segment identity or a temporal overlap threshold?
  3. What quality control measures were implemented for the annotations assisted by GPT-4o? Additionally, please provide uncertainty estimates for the performance differences between the primary methods.
  4. Which parameters are updated during a single round of post-training? What are the structures and parameter counts of the recurrent and Transformer aggregators shown in Table 5? Are the input frames, resolutions, training sample exposure, effective batch sizes, and the number of negative samples consistent across the different methods? When parameter counts or computational budgets are matched, do the primary retrieval advantages still hold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.