acceptodds
Withdrawn from ICLR 2027

JoTS: Joint Token- and Segment-Level Prediction for Self-Supervised Multimodal Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal dense retrieval learns representations for semantic matching across textual and visual content. However, existing multimodal retrievers largely rely on manually annotated or synthetically generated relevance pairs, overlooking the supervision naturally present in naturally occurring multimodal documents, where text and images form coherent sequences. This natural document order can be regarded as annotation-free supervision. We introduce JoTS (Joint Token- and Segment-Level Prediction), a self-supervised framework that jointly optimizes retrieval-conditioned next-token prediction (NTP) and segment-continuation prediction (SCP). NTP uses Retriever-weighted context to guide textual token prediction, while SCP identifies the succeeding text or image segment from a dynamic candidate set. These parallel objectives capture fine-grained linguistic dependencies and higher-level multimodal continuity. We pretrain JoTS on OBELICS using vision-language backbones of different scales. On general-purpose MMEB and reasoning-intensive MM-BRIGHT, JoTS consistently improves its base backbones and outperforms representative supervised and contrastively trained retrievers, demonstrating the effectiveness of predictive self-supervision for multimodal retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal retrieval, Unsupervised Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GA58OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes JoTS, a self-supervised framework for multimodal dense retrieval from naturally interleaved image-text documents without annotated or synthetic query-document pairs. JoTS jointly optimizes retrieval-conditioned next-token prediction (NTP) and segment-continuation prediction (SCP), supported by Retriever-weighted cross-chunk attention and semantic-difficulty grouping. The method is pretrained on OBELICS and evaluated with LFM2.5-VL-450M and Qwen2.5-VL-3B on MMEB and MM-BRIGHT, where it improves over the corresponding unadapted backbones and self-supervised baselines, with additional ablations examining the contributions of NTP, SCP, entropy regularization, and visual inputs.

Critical strengths
  1. The paper addresses an important problem: learning multimodal retrieval representations without manually annotated or synthetically constructed query-document pairs.

  2. The proposed framework combines token-level and segment-level predictive objectives, providing a reasonable coarse-to-fine training design for interleaved image-text documents.

  3. The experiments cover two multimodal retrieval benchmarks and include several ablation studies, including analyses of NTP, SCP, entropy regularization, and visual query inputs.

Critical weaknesses
  1. The paper treats the text-image order in HTML/PDF/DOCX documents as “annotation-free supervision.” However, document order can also be determined by layout, advertisements, web templates, or citation structures, and does not necessarily reflect retrieval semantics. Therefore, the current experiments do not establish that the reported gains truly come from the claimed “natural document continuity.”

  2. NTP-based retriever training has already been explored in Revela: Dense Retriever Learning via Language Modeling [1]. Moreover, the proposed SCP is closely related to future-representation prediction methods such as CPC [2] and JEPA [3]. The paper therefore needs to clarify its substantive novelty relative to these existing approaches.

[1] Cai, F., Chen, T., Zhao, X., et al. Revela: Dense Retriever Learning via Language Modeling. In: International Conference on Learning Representations (ICLR), 2026, pp. 1372–1395.

[2] van den Oord, A., Li, Y., and Vinyals, O. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748, 2018.

[3] Assran, M., Duval, Q., Misra, I., et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15619–15629.

Questions for authors
  1. In Tables 2 and 4, the w/o variants do not specify which module has been removed. This makes the ablation settings ambiguous. In particular, does w/o denote w/o SCP?

  2. The appendix still contains unresolved placeholders such as [TODO: dataset revision] and [TODO: paper or legacy]. Please clarify the corresponding dataset version, evaluation protocol, and citation.

  3. Table 1 appears to swap the labels of the two JoTS variants. The LFM2.5-VL-450M row reports JoTS (Qwen), whereas the Qwen2.5-VL-3B row reports JoTS (LFM). Please clarify whether this is a labeling error or whether the experiments actually involve cross-backbone transfer.

Reviewer_Q7T4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains a multimodal dense retriever from interleaved image-text documents without any relevance labels. Following Revela, the Retriever and a Reference language model are copies of the same vision-language backbone. The Retriever encodes each chunk of a 16-chunk group under a query and a passage prompt, the resulting similarity matrix weights the in-batch attention inside the Reference model. The next-token prediction loss of the Reference model trains the Retriever through these weights. On top of this, the authors add segment-continuation prediction (SCP) where the Reference state at a segment boundary has to pick the next text or image segment among all segments of the group. Groups are built from a 26K-document subset of OBELICS by mixing semantically near, intermediate and far documents (SDG). The method is compared on MMEB-36 and MM-BRIGHT Tasks 2-4 to the unadapted backbone, a SimCSE-style baseline trained on the same data, CLIP and GME.

Critical strengths

S1) Using the natural order of text and images in web documents as training signal for a retriever is a relevant direction, and extending Revela to vision-language backbones is a natural step.

S2) The gains over the SimCSE-style baseline trained on the same data are large on MM-BRIGHT for both backbones.

S3) Two backbones of different size and two complementary benchmarks are used. The ablations of the entropy term and of the query images are useful.

Critical weaknesses

W1) The checkpoints are selected on the test benchmarks. Table 3 reports, for every \lambda, the best MMEB-36 P@1 among the step-200, step-400 and step-600 checkpoints, and the main model is just the best entry of this sweep. In Table 2, the full model is taken at step 200 while the ablations are taken at steps 400, 600 and 800, and it is not explained how these steps were chosen. No validation set is mentioned anywhere. Step 200 is also only a fifth of the 1,000-step schedule (800 of the 4,112 groups), which might imply that the full model gets worse on MMEB with more training, but no learning curves are shown. All numbers come from a single seed without any variance, while conclusions are drawn from differences of 0.2 points, such as for the entropy regularization (32.56 vs 32.34).

W2) The paper's own ablation does not support the claimed retrieval benefit of SCP. In Table 2, the NTP-only variant is better than the full model on MMEB retrieval (34.18 vs 32.15) and on all three MM-BRIGHT tasks, so is the variant without entropy regularization. The full model wins the MMEB average only through classification, VQA and grounding. Section 4.3 admits this trade-off, but Section 4.2 still claims that segment prediction "is especially beneficial for structurally demanding cross-modal retrieval", which Table 2 does not support. In Table 4 the full model gains 0.72 points from the query images, but the NTP-only model without query images (15.66) is still above the full model with them (12.74). The ablation is only done for LFM, there is no objective ablation for Qwen.

W3) The controls needed to attribute the gains are missing. SDG is listed as the first contribution, but it is never compared to random grouping with the same bundles, resource constraints and training budget.

W4) The comparison to other retrievers does not support the claims of the abstract. With Qwen, the SimCSE baseline is better than JoTS on every MMEB column (37.75 vs 34.09 on average) and JoTS only wins on MM-BRIGHT, yet Section 4.2 compares the Qwen result to the unadapted backbone (3.03) instead. GME-Qwen2-VL-2B is above both JoTS models on the MMEB average and on all three MM-BRIGHT tasks. TThe external numbers also need explanation. CLIP ViT-L/14 obtains 29.23 on MMEB and 26.35 on its retrieval part here, while the VLM2Vec paper reports 37.8 and 53.0 for the same model, and still 26.7 and 41.3 when instructions are prepended to CLIP [1]. The authors should state which checkpoint, preprocessing, query fusion and instruction handling were used before ranking JoTS above CLIP.

[1] VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks, Jiang et al., ICLR 2025 (Tables 2 and 4)

W5) The evaluation protocol is not fully specified. Appendix B.1 still contains "[TODO: dataset revision]" and "[TODO: paper or legacy] Task-4 protocol" for MM-BRIGHT, in the same subsection that describes the graded relevance of Task 4.

Minor weaknesses:

MW1) Qwen3-Embedding is cited as Bai et al. 2025, which is the Qwen2.5-VL report.

Questions for authors

Q1) On which data and by which criterion were the checkpoints and \lambda selected? Was MM-BRIGHT used for the selection? Please show MMEB and MM-BRIGHT as a function of the training step for the full model and the NTP-only variant, and results over several seeds.

Q2) Does SCP improve retrieval at matched steps and across seeds? Does the LFM ablation pattern hold for Qwen?

Q3) Why do the CLIP numbers differ from the official MMEB numbers? Do the gains over the external models remain when all models receive the same query images?

Reviewer_uFxZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work presents JoTS, a self-supervised multimodal dense retriever trained on raw interleaved OBELICS data without annotated relevance pairs. It integrates retriever-weighted in-batch attention with two predictive objectives, next-token and segment-continuation prediction, alongside a semantic difficulty grouping scheme. Evaluated under LoRA tuning on two VL backbones, JoTS outperforms unadapted backbones on MMEB-36 and obtains reasonable dense retrieval results on MM-BRIGHT.

Critical strengths
  1. The proposed method extends Revela’s retriever-weighted attention to interleaved image–text inputs and introduces joint optimization of next-token prediction and successor-segment discrimination. The successor-segment discrimination injects visual supervision to overcome the limitation of text-only token prediction, with only the retriever deployed at inference.
  2. Consistent gains against same-data SimCSE-style baselines are achieved on most MM-BRIGHT tasks using both Qwen and LFM backbones, providing positive evidence for the overall framework on reasoning-intensive multimodal retrieval.
Critical weaknesses
  1. The SCP retrieval benefit cannot be reliably attributed. The ablation cannot tease apart gains from segment ordering, same-document co-occurrence, or segment-level supervision. Table 2 compares ablation variants using checkpoints selected at different training steps, ranging from 200 to 800. While the full model is better on classification, VQA and visual grounding, removing SCP improves all four retrieval metrics. This weakens the attribution of the reasoning-intensive retrieval gains to SCP in Section 4.2. Although Section 4.3 briefly acknowledges the retrieval strength of the NTP-only variant, it does not adequately discuss or explain these consistent retrieval-side improvements.
  2. Severe reporting issues impede verification and reproducibility. The appendix retains unresolved TODO placeholders, Table 1 has incorrect model labels, the caption of Table 2 mismatches its content, several loss symbols are missing, some terms remain undefined, and the claimed prompt/readout comparisons are not presented.
Questions for authors
  1. Removing SCP improves all four retrieval metrics in Table 2. What evidence establishes a retrieval benefit from SCP beyond the inherited NTP framework, and how do the authors explain this trade-off?
  2. What evidence shows that learned cross-chunk routing contributes beyond segment-level supervision alone? A uniform-routing control retaining SCP and the same cross-chunk context would help isolate this contribution.
  3. Please complete the missing evaluation specifications and provide the claimed prompt/readout comparisons, including confirmation that the full and no-SCP variants use identical input and readout settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.