- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Chimera, a distillation framework that trains a Latent Bottleneck History Transformer (LBHT) as a teacher to explicitly compress observation histories into fixed-size latent representations, and then distills these representations into a recurrent transformer student. The proposed approach is evaluated mainly on memory-based relative pose estimation (Mem-RPE), where the student recurrent model is shown to narrow the gap with the full-history teacher under long sequence settings. The authors additionally validate the approach on streaming VQA to demonstrate broader applicability of memory distillation.
Critical strengths
-
The proposed distillation framework is effective. The use of a latent bottleneck history transformer as a teacher and direct memory representation alignment between teacher and student is a clean approach. The experiments show that this strategy can significantly improve recurrent transformer performance compared with existing recurrent baselines while maintaining fixed-size memory.
-
The experimental evaluation targets a meaningful long-horizon setting. The Mem-RPE task is appropriate for studying persistent memory, where retaining information from distant observations is essential. The evaluation on longer sequences provides evidence that the proposed method improves recurrent models in scenarios where standard recurrent approaches struggle.
Critical weaknesses
-
The evaluation scope is relatively narrow compared with the claimed general applicability. Although the paper includes Mem-RPE and streaming VQA experiments, the validation is still limited to a small number of tasks. More diverse long-context tasks would be necessary to demonstrate that the proposed approach is broadly useful for recurrent transformers.
-
Limited generality of the proposed approach. The learned compression mechanism is optimized with task-specific objectives, making it unclear whether the method can generalize across different tasks. If a separate teacher and distillation process are needed for each task, the training cost may limit practical applicability.
-
Insufficient analysis of memory size and compression trade-offs. The proposed method relies on a fixed-size bottleneck representation, but the paper does not provide comprehensive ablations on different memory sizes, numbers of bottleneck tokens, or compression ratios.
Questions for authors
-
Can the authors provide evaluations on more diverse long-context tasks to demonstrate that the proposed approach generalizes beyond Mem-RPE and streaming VQA?
-
Can the authors demonstrate that the learned memory compression mechanism transfers across different tasks without retraining a separate teacher?
-
How does performance change with different bottleneck sizes or memory budgets?
3 comments
Thank you for the clarification. I apologize for my earlier concern regarding generality. My research background is primarily in long-context and recurrent LLMs, where a common expectation is that a single model or method should generalize across a wide range of downstream tasks. This prior may have influenced my assessment of the paper's scope.
Regarding the memory-size analysis, the additional clarification about Table 7 and the rationale for adopting Kinaema's memory configuration for Mem-RPE also addresses my concern. I appreciate the authors' responses and will raise my score to 3 accordingly.
Thank you for having kept an open mind about our work, engaged in the discussion and for having raised the score! This is very much appreciated.
We are surprised that the scope was perceived negatively. This is a paper about streaming computer vision and our work goes beyond the scope of a large body of computer vision work, most of which target single tasks.
Generality. The compression is query-independent by design: neither the teacher bottleneck nor the student memory sees the query. The same recipe (N=4 segments, L1 matching, Eq. 4) was applied unchanged to both applications.
However, a teacher is trained per task, as in most distillation work. This is a one-time training cost, and the teacher is never deployed. Given the gains we obtain, we are surprised that the advantage of this procedure is not acknowledged. Focusing on a unified model for computer vision was not the claim of this work and we are not aware of work that targets this.
Memory size. Table 7 ablates the number of bottleneck tokens and the teacher depth for streaming VQA: 8, 16 and 32 tokens give 71.9, 72.2 and 72.4% on LLaVA-Video val, and larger bottlenecks bring no further gain. For Mem-RPE we deliberately kept the memory configuration of Kinaema (20 × 3072), whose paper ablates the number and dimension of memory embeddings for the same architecture. This isolates the effect of distillation at an identical memory budget.