- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether a pretrained language model can benefit from mid-training on raw, unannotated videos without relying on captions or text supervision. The authors propose a simple self-supervised training framework that converts video frames into continuous visual tokens and trains a causal language model to predict the next visual token through feature regression. After identical image-text instruction tuning, the video mid-trained model achieves improvements of 2.9 and 5.1 percentage points on video and image benchmarks, respectively, while maintaining comparable text performance to the model without mid-training.
Critical strengths
- The paper explores an important direction for scaling multimodal foundation models that leverages large amounts of raw video without requiring expensive annotation or synthetic caption generation. This work focuses on extending an existing pretrained language model through an additional video mid-training stage, making the proposed setting particularly relevant to practical multimodal model development.
- The proposed approach is conceptually straightforward and compatible with standard causal language model architectures. By directly predicting continuous visual representations, it avoids discrete visual tokenization and caption-based supervision.
- The paper evaluates the proposed approach across video, image, and text benchmarks. The results demonstrate consistent improvements across all four video benchmarks and all ten image benchmarks
Critical weaknesses
- The quality of the learned visual representations is not directly evaluated. The experiments primarily evaluate its effectiveness through downstream vision-language tasks. It remains unclear whether the proposed training objective actually improves the quality of the visual representations themselves.
- Insufficient analysis of language capability preservation during video mid-training. While the paper reports comparable text performance after multimodal instruction tuning, it remains unclear how the raw-video mid-training stage itself affects the pretrained language model's original capabilities.
Questions for authors
Please refer to the weakness part.
3 comments
Thank the author for the rebuttal. I would keep my score as weak acceptance.
We thank the reviewer for reading our response and for continuing to support the paper.
Response to Reviewer mQw5
Thank you for the thoughtful review. We address each comment below. Please let us know if anything remains unclear.
Re: The quality of the learned visual representations is not directly evaluated. The experiments primarily evaluate its effectiveness through downstream vision-language tasks. It remains unclear whether the proposed training objective actually improves the quality of the visual representations themselves.
Our goal is to mid-train a language model on raw video rather than to learn a stronger vision encoder, so we evaluate the full model on downstream tasks, as is standard for multimodal LLMs, and do not evaluate the visual representations directly. The downstream results still give indirect evidence on the visual side. After instruction tuning, the mid-trained model improves all ten image benchmarks, with the largest gains on visually dense tasks such as ChartQA (+8.2), DocVQA (+6.4), and AI2D (+6.1). We do not claim that the vision encoder's representations improve on their own.
Re: Insufficient analysis of language capability preservation during video mid-training. While the paper reports comparable text performance after multimodal instruction tuning, it remains unclear how the raw-video mid-training stage itself affects the pretrained language model's original capabilities.
We agree that the paper evaluates text performance only after instruction tuning, and we do not claim that mid-training alone preserves the language model's original capabilities. Our claim concerns the final model.
In Section 4.4, we observed that all mid-trained models, regardless of whether they used a visual next-token or next-caption objective, suffered from catastrophic forgetting of question-answering ability before instruction tuning, similar to Lu et al. (2024).
Because all models receive the same instruction tuning, our comparisons isolate the effect of mid-training. After instruction tuning, the text average of the visual model is comparable to that of the model without mid-training (48.9 vs. 48.0) and stays within 0.9 points of it across all evaluated checkpoints. It is also 1.5 points above the next-caption model, which uses text targets.