- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes JoTS, a self-supervised framework for multimodal dense retrieval from naturally interleaved image-text documents without annotated or synthetic query-document pairs. JoTS jointly optimizes retrieval-conditioned next-token prediction (NTP) and segment-continuation prediction (SCP), supported by Retriever-weighted cross-chunk attention and semantic-difficulty grouping. The method is pretrained on OBELICS and evaluated with LFM2.5-VL-450M and Qwen2.5-VL-3B on MMEB and MM-BRIGHT, where it improves over the corresponding unadapted backbones and self-supervised baselines, with additional ablations examining the contributions of NTP, SCP, entropy regularization, and visual inputs.
Critical strengths
-
The paper addresses an important problem: learning multimodal retrieval representations without manually annotated or synthetically constructed query-document pairs.
-
The proposed framework combines token-level and segment-level predictive objectives, providing a reasonable coarse-to-fine training design for interleaved image-text documents.
-
The experiments cover two multimodal retrieval benchmarks and include several ablation studies, including analyses of NTP, SCP, entropy regularization, and visual query inputs.
Critical weaknesses
-
The paper treats the text-image order in HTML/PDF/DOCX documents as “annotation-free supervision.” However, document order can also be determined by layout, advertisements, web templates, or citation structures, and does not necessarily reflect retrieval semantics. Therefore, the current experiments do not establish that the reported gains truly come from the claimed “natural document continuity.”
-
NTP-based retriever training has already been explored in Revela: Dense Retriever Learning via Language Modeling [1]. Moreover, the proposed SCP is closely related to future-representation prediction methods such as CPC [2] and JEPA [3]. The paper therefore needs to clarify its substantive novelty relative to these existing approaches.
[1] Cai, F., Chen, T., Zhao, X., et al. Revela: Dense Retriever Learning via Language Modeling. In: International Conference on Learning Representations (ICLR), 2026, pp. 1372–1395.
[2] van den Oord, A., Li, Y., and Vinyals, O. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748, 2018.
[3] Assran, M., Duval, Q., Misra, I., et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15619–15629.
Questions for authors
-
In Tables 2 and 4, the
w/ovariants do not specify which module has been removed. This makes the ablation settings ambiguous. In particular, doesw/odenotew/o SCP? -
The appendix still contains unresolved placeholders such as
[TODO: dataset revision]and[TODO: paper or legacy]. Please clarify the corresponding dataset version, evaluation protocol, and citation. -
Table 1 appears to swap the labels of the two JoTS variants. The LFM2.5-VL-450M row reports
JoTS (Qwen), whereas the Qwen2.5-VL-3B row reportsJoTS (LFM). Please clarify whether this is a labeling error or whether the experiments actually involve cross-backbone transfer.