acceptodds
Under review as a conference paper at ICLR 2027

Mid-Training Language Models on Raw Video

Abstract

Multimodal large language models learn mostly from paired image-text data or annotated video, and raw web video is rarely used to further train an existing language model. We study whether raw video, with no captions and no text loss, can serve as mid-training data for a pretrained language model. Frames are encoded into continuous visual tokens, and the language model learns to predict the next visual token. We mid-train Qwen3-1.7B on raw clips from YT-Temporal-1B and then apply the same image-text instruction tuning to it and to the model without mid-training, so that the two differ only in mid-training. The mid-trained model scores 2.9 points higher on average across four video benchmarks and 5.1 points higher across ten image benchmarks, spanning perception, document, and chart tasks. Although mid-training uses no text data, after the same instruction tuning, the mid-trained model matches the text average of the model without mid-training. Analyses across training show that the image and video gains emerge within 30 % of training and plateau thereafter, varying by less than 0.5 points. Predicting captions fails to outperform next-visual-token prediction, demonstrating that video mid-training can remain purely self-supervised without the computational overhead or labeling noise of automated captioning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mid-training, vlm, video, llm, data scarcity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mQw5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether a pretrained language model can benefit from mid-training on raw, unannotated videos without relying on captions or text supervision. The authors propose a simple self-supervised training framework that converts video frames into continuous visual tokens and trains a causal language model to predict the next visual token through feature regression. After identical image-text instruction tuning, the video mid-trained model achieves improvements of 2.9 and 5.1 percentage points on video and image benchmarks, respectively, while maintaining comparable text performance to the model without mid-training.

Critical strengths
  1. The paper explores an important direction for scaling multimodal foundation models that leverages large amounts of raw video without requiring expensive annotation or synthetic caption generation. This work focuses on extending an existing pretrained language model through an additional video mid-training stage, making the proposed setting particularly relevant to practical multimodal model development.
  2. The proposed approach is conceptually straightforward and compatible with standard causal language model architectures. By directly predicting continuous visual representations, it avoids discrete visual tokenization and caption-based supervision.
  3. The paper evaluates the proposed approach across video, image, and text benchmarks. The results demonstrate consistent improvements across all four video benchmarks and all ten image benchmarks
Critical weaknesses
  1. The quality of the learned visual representations is not directly evaluated. The experiments primarily evaluate its effectiveness through downstream vision-language tasks. It remains unclear whether the proposed training objective actually improves the quality of the visual representations themselves.
  2. Insufficient analysis of language capability preservation during video mid-training. While the paper reports comparable text performance after multimodal instruction tuning, it remains unclear how the raw-video mid-training stage itself affects the pretrained language model's original capabilities.
Questions for authors

Please refer to the weakness part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mQw5

Thank you for the thoughtful review. We address each comment below. Please let us know if anything remains unclear.

Re: The quality of the learned visual representations is not directly evaluated. The experiments primarily evaluate its effectiveness through downstream vision-language tasks. It remains unclear whether the proposed training objective actually improves the quality of the visual representations themselves.

Our goal is to mid-train a language model on raw video rather than to learn a stronger vision encoder, so we evaluate the full model on downstream tasks, as is standard for multimodal LLMs, and do not evaluate the visual representations directly. The downstream results still give indirect evidence on the visual side. After instruction tuning, the mid-trained model improves all ten image benchmarks, with the largest gains on visually dense tasks such as ChartQA (+8.2), DocVQA (+6.4), and AI2D (+6.1). We do not claim that the vision encoder's representations improve on their own.

Re: Insufficient analysis of language capability preservation during video mid-training. While the paper reports comparable text performance after multimodal instruction tuning, it remains unclear how the raw-video mid-training stage itself affects the pretrained language model's original capabilities.

We agree that the paper evaluates text performance only after instruction tuning, and we do not claim that mid-training alone preserves the language model's original capabilities. Our claim concerns the final model.

In Section 4.4, we observed that all mid-trained models, regardless of whether they used a visual next-token or next-caption objective, suffered from catastrophic forgetting of question-answering ability before instruction tuning, similar to Lu et al. (2024).

Because all models receive the same instruction tuning, our comparisons isolate the effect of mid-training. After instruction tuning, the text average of the visual model is comparable to that of the model without mid-training (48.9 vs. 48.0) and stays within 0.9 points of it across all evaluated checkpoints. It is also 1.5 points above the next-caption model, which uses text targets.

Thank the author for the rebuttal. I would keep my score as weak acceptance.

We thank the reviewer for reading our response and for continuing to support the paper.

Reviewer_nzeDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes mid-training for MLLMs. It introduces a self-supervised approach, causal next visual token regression using the LLM, without using any text loss. Mid-training demonstrates an improvement on both video as well as image benchmarks compared to non mid-training baselines.

Critical strengths

The paper demonstrates that in MLLMs mid-training with a simple next-frame prediction objective leads to improvements in both video and image understanding.

Critical weaknesses
  1. The paper lacks novelty. Cambrian-S [1] already establish that next frame prediction enhances the performance of MLLMs.
  2. The paper lacks substantial evidence to claim that the improvements are attributed to the video structure and not simply additional data. While the baseline is trained with SFT, the introduced approach experiences additional training steps with a trainable vision encoder making the comparison unfair.

[1] Yang, Shusheng, et al. "Cambrian-s: Towards spatial supersensing in video." ICLR, 2026

Questions for authors
  1. Please provide fair comparisons with the baselines including: (a) similar token budget, (b) similar training steps, (c) frozen vision encoder during mid-training.
  2. Compare the model with mid-training with Cambrian-S style LFP (during SFT).
3 comments
AuthorsOpenReview ↗

Response to Reviewer nzeD

We appreciate the reviewer for their constructive feedback. We address the reviewer’s concerns accordingly. Please do not hesitate to respond if you have any further comments.

Re: The paper lacks novelty. Cambrian-S [1] already establish that next frame prediction enhances the performance of MLLMs. / Compare the model with mid-training with Cambrian-S style LFP (during SFT).

Cambrian-S (Yang et al., 2026) and our work use next-frame prediction for different purposes. As noted in L100–102, Cambrian-S adds a separate head that predicts the latent of the next frame and trains it jointly with instruction tuning. Its main purpose is to compute surprise during inference, which drives memory management and event segmentation (Section 4 in Yang et al. (2026)). The prediction loss itself does not improve benchmark performance. In Table 16 of Yang et al. (2026), adding it at the default weight changes VideoMME, EgoSchema, and Perception Test by at most 0.5 points and lowers VSI-Bench by 1.4 points.

In contrast, we use next-visual-token prediction on raw video as a separate mid-training stage before instruction tuning, with no captions and no text loss.

For convenience, the relevant figure and table from Yang et al. (2026) are collected at this anonymous link. They are Figure 9 in the arXiv version (Figure 7 in the ICLR version) and Table 16 in both versions.

Re: The paper lacks substantial evidence to claim that the improvements are attributed to the video structure and not simply additional data. While the baseline is trained with SFT, the introduced approach experiences additional training steps with a trainable vision encoder, making the comparison unfair.

Section 4.4 compares visual next-token prediction with caption next-token prediction on the same video source (~64.4B). Except for this objective, all training details are the same, including the number of training steps and trainable parameters.

As shown in Table 4 (copied below), visual next-token prediction matches next-caption prediction on both the video and image averages (within 0.3 points, slightly higher in both cases), showing that mid-training does not require captions, which is our motivation for this work. The unnecessariness of captioning removes its additional computation/storage cost and the risk of captioner errors.

Table 4. Comparison of mid-training objectives on YT-1B. All models are instruction-tuned on LLaVA-OneVision-Data, and No mid-training is the same model instruction-tuned without mid-training. The best value in each column is in bold.

ObjectiveVideoImageText
No Mid-Training49.6748.7248.03
Current Caption51.2848.7747.44
Next Caption52.5153.5447.40
Visual Next Token (ours)52.5953.8048.89

Re: Please provide fair comparisons with the baselines including frozen vision encoder during mid-training.

We conducted experiments with a frozen vision encoder. Due to limited time and resources, we evaluated it on all benchmarks at 0.5 epochs. Table R2 compares it with our model with a trainable vision encoder at the same epoch. After the same instruction tuning, the model with a frozen encoder still scores 2.3 and 3.5 points above the model without mid-training on the video and image averages, so the gains do not depend on training the vision encoder. Our model with a trainable encoder shows better performance on video and image benchmarks, while the one with a frozen encoder is higher on text benchmarks.

Table R2. Mid-training with a frozen or trainable vision encoder, at 0.5 epochs.

ObjectiveVideoImageText
No Mid-Training49.6748.7248.03
Video Mid-Training (Frozen Encoder)51.9652.1850.63
Video Mid-Training (Trainable Encoder, ours)52.3753.7648.32

I thank the authors for their response. The performance comparison with the frozen encoder clarifies my concern (Q1). However, my concern Q2 still remains. The paper lacks comparison with state of the art methods on the benchmarks. Furthermore, it provides limited analysis on the robustness of the approach to different Language models and to large scale models (>2B). Thus, I am maintaining my original score.

We really appreciate the reviewer’s follow-up, and we are glad the frozen-encoder comparison resolved the first concern.

Re: The paper lacks comparison with state-of-the-art methods on the benchmarks.

We would like to emphasize that our goal is not to achieve state-of-the-art performance. Instead, as we mentioned in the abstract (L11-12) and the introduction (L45-47), we study whether raw video without captions can serve as mid-training data for a pretrained language model.

To answer this, we compare two models that differ only in mid-training, with the same architecture, instruction-tuning data, and training settings. State-of-the-art methods are trained with different data and training paradigms, so comparing with them would not show what mid-training contributes. In our comparison, mid-training improves all four video and all ten image benchmarks.

Re: Furthermore, it provides limited analysis on the robustness of the approach to different Language models and to large scale models (>2B).

We agree with the reviewer's point, and we state this as a limitation in the Discussion (L484-485). Each mid-training run uses 128 H100 GPUs (L238-239), so we could not repeat the study with a larger language model within the rebuttal period.

Regarding robustness, reporting single runs is standard practice for multimodal LLMs at this scale due to the cost. LLaVA-OneVision (Li et al., 2025), DeepSeek-VL (Lu et al., 2024), and Cambrian-S (Yang et al., 2026) all report their results without multi-run uncertainty. Even so, the gains do not rest on one instruction-tuning run. The 0.3-, 0.5-, and 1.0-epoch checkpoints are instruction-tuned separately (see Section 4.3), and each improves all four video and all ten image benchmarks over the model without mid-training.

Reviewer_v5kkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether an existing language model can benefit from continued training on uncaptioned video before multimodal instruction tuning. SigLIP features are projected into Qwen3-1.7B, serialized across patches and frames, and predicted autoregressively using a cosine feature-regression loss with stop-gradient targets. The vision encoder, projector, language model, and prediction head are trained jointly on 6.99M clips, totaling 64.4B visual tokens. After the same image-text instruction tuning as the baseline, the model improves the average video score by 2.92 points and the average image score by 5.08 points. Caption-objective comparisons and intermediate-checkpoint evaluations suggest that visual prediction is competitive with generated next-frame captions and that most downstream gains appear early.

Critical strengths
  • The practical question is worthwhile, and the main comparison controls the subsequent instruction-tuning recipe. Improvements occur on all four video and all ten image benchmarks, including ChartQA and EgoSchema, rather than only one selected task. This supports the usefulness of additional uncaptioned visual training in the tested setup.
  • The comparison with current- and next-frame caption prediction makes the study more informative. Next-caption and visual prediction have very similar video/image averages, while visual prediction avoids an additional caption-generation pipeline. The checkpoint analysis also gives a useful indication that most gains are obtained.
Critical weaknesses
  • The experiments do not establish which part of raw-video mid-training produces the gains. The loss predicts adjacent spatial patches as well as frame-boundary tokens, and all visual and language components are updated. There is no frame-shuffled, repeated/static-frame, or matched image-only training control, nor a control that freezes the language model during mid-training. The discussion acknowledges the missing frame-shuffled ablation. Consequently, the results support additional self-supervised visual exposure, but do not yet isolate a benefit from temporal video structure or adapting the pretrained language model itself.
  • The reliability and compute interpretation of the comparisons need stronger support. Results use one 1.7B backbone without multi-run uncertainty for the main video/image gains, and the discussion reports a BBH discrepancy between instruction-tuning runs from the same initial state. Table 4 mentions a budget-controlled 0.5-epoch visual model, but does not report enough per-objective token/FLOP or wall-clock accounting to establish precisely what is matched. This limits conclusions about the relative efficiency of visual and caption objectives; the final text average also measures performance after instruction tuning, rather than preservation immediately after mid-training.
Questions for authors
  1. What does temporal ordering contribute beyond additional exposure to images? The prediction objective covers spatially adjacent patches within a frame as well as transitions between frames, and most target positions are spatial rather than frame-boundary predictions. The downstream improvements therefore do not by themselves establish that learning from temporal video structure is responsible.

  2. Which part of the pretrained system is learning the capability that later transfers? The vision encoder, projector, and language model all change during mid-training, but the motivation emphasizes continuing the training of an existing language model.

3 comments
AuthorsOpenReview ↗

Response to Reviewer v5kk

We appreciate the reviewer’s feedback. We address the reviewer’s concerns accordingly. Please do not hesitate to respond if you have any further comments.

Re: What does temporal ordering contribute beyond additional exposure to images?

We ran mid-training with the shuffled frames the reviewer requested and evaluated on video and image benchmarks in Table R1. Frames are permuted within each clip, and all other settings are identical. We compare at 0.3 epochs because we observed that the frame-shuffled model overfits with longer training.
At 0.3 epochs, ordered video improves over no mid-training on the video average (+3.1) and the image average (+4.9), whereas frame-shuffled video falls below it on both (−3.3 and −16.3). Because the two runs see the same frames and differ only in their order, additional visual exposure alone does not explain the video and image gains.

Table R1. Average performance by modality without mid-training and with mid-training on frame-shuffled or ordered video for 0.3 epochs.

AverageNo Mid-TrainingVideo Mid-Training (Frame-Shuffled)Video Mid-Training (Ordered)
Video49.6746.4152.75
Image48.7232.4253.63

Re: Results use one 1.7B backbone without multi-run uncertainty for the main video/image gains

Reporting single runs is standard practice for multimodal LLMs at this scale due to the cost. LLaVA-OneVision (Li et al., 2025), DeepSeek-VL (Lu et al., 2024), and Cambrian-S (Yang et al., 2026) all report their results without multi-run uncertainty. Our mid-training run alone uses 128 H100 GPUs. Even so, the gains do not rest on one instruction-tuning run. The 0.3-, 0.5-, and 1.0-epoch checkpoints are instruction-tuned separately (see Section 4.3), and each improves all four video and all ten image benchmarks over the model without mid-training. We agree that results on larger backbones are beneficial and left it as future work in the Discussion, as the reviewer acknowledged.

Re: the discussion reports a BBH discrepancy between instruction-tuning runs from the same initial state

We apologize for the confusion. The same initial state denotes the Qwen3-1.7B pretrained model. For clarity, we removed the sentence in the revision.

Re: Table 4 mentions a budget-controlled 0.5-epoch visual model, but does not report enough per-objective token/FLOP or wall-clock accounting to establish precisely what is matched.

Thank you for pointing this out. The mention of a budget-controlled 0.5-epoch visual model was inadvertently retained from an earlier draft and does not describe the reported experiments. We have removed it in the revision. All models in Table 4 are trained on the same number of frames. The additional text caption tokens amount to less than 1% of the visual tokens, so the models are compared under similar token counts and FLOPs.

Re: Which part of the pretrained system is learning the capability that later transfers? The vision encoder, projector, and language model all change during mid-training, but the motivation emphasizes continuing the training of an existing language model.

We mid-train a pretrained language model on raw video, motivated by text data scarcity. The vision encoder and projector are updated during mid-training to map frames into the LLM's embedding space. Predicting each visual token from the preceding context, within and across frames, is done by the language model, which is why we describe mid-training as continuing its training.

Since the three components are trained jointly, the video and image gains reflect the system as a whole. In the text benchmarks (Table 3), however, only the language model is used, and after identical instruction tuning, its text average is comparable to that of the model without mid-training.

Thanks for the detailed feedback! My concerns have been addressed and I lean to accept.

We appreciate the reviewer for the kind response. We are glad that our reply addressed the concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.