acceptodds
Withdrawn from ICLR 2027

Scene-Graph Molmo: Adding Structure to Pretraining Yields Strong Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretraining on large, diverse collections of image-caption pairs is the standard method for training strong Vision-Language Models (VLMs). However, captions have several drawbacks due to their lack of consistent structure: they can be brief, ambiguous, and leave out information that annotators think is “obvious”, leading to failures on downstream grounding and reasoning tasks. Motivated by human visual processing, we instead use a structured scene description: scene graphs. We curate PixMo-SceneGraph, a dataset of 704K image-scene-graph pairs derived solely from captions in the PixMo-Cap dataset, enabling a controlled comparison with unstructured caption data. We pretrain Molmo2 on PixMo-SceneGraph and finetune the resulting models on 8 standard academic datasets, finding that scene- graph pretraining improves performance over caption pretraining, particularly in generalizing to the grounding and reasoning tasks where caption-based models struggle. Combining scene graphs with captions yields further gains. Moreover, the structure of scene graphs also provides an interface for targeted data augmentation: we augment PixMo-SceneGraph with information typically omitted from captions—temporal statements, negations, affordances, and external knowledge—further improving performance. Our suite of models, Molmo-SceneGraph, outperform Qwen3-VL at 2B, 4B, and 8B scales across a broad range of academic, grounding, and reasoning benchmarks, despite being trained on 200–500× fewer tokens. Our results highlight the value of structure in pretraining data, and encourage researchers to rethink standard methods for VLM pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scene graph, vision-language models, pretraining, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_wwzMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study how to increase the information content of image captions by designing a scene graph for the corresponding images. They build a dataset of 704K samples from PixMo-Cap and use GPT-5.4-nano as a tool to automatically generate new structured captions. The results are stored in JSON format and are highly beneficial for downstream tasks. To demonstrate the effectiveness of their scene graph (SG) model, they conduct extensive experiments on different fine-tuning tasks, and the results validate their method. They also discuss and summarize the main findings to further enhance the applications of their model.

Critical strengths

They build a new model with a scene graph to further enhance the information content of the captions with the structure information omitted in the existing data sets. The method is neat and the result of experiments are convincing. The model only relied on the captions and SG pairs without the need of images. Their method need much less tokens than the original QWen-VL model.

Critical weaknesses
  1. The novelty of the idea is relatively low as the idea of scene graphs have been widely used and studied in the previous works even in the VLMs. Though they designed SG from a different view, the significance is not very substantial.
  2. Lack of extensive experiments, especially comparing with existing models in VLMs adopting the idea of SGs, e.g., Yu 2021, Park 2025. Especially the DAC paper.
  3. The main technical issue in the paper is the construction of their data set. The authors didn't discuss it in details in the main part of the paper, but leaving to the appendix.
Questions for authors
  1. Why the size of data set is 740K, as in Radford2023 paper, the size of 400 millions. What is the your performance limit if the size data set is as large as that of Radford 2023.
  2. Fig. 1 & 2 didn't provide any help to further improve the readability of the paper because they didn't contain too much details.
  3. Why the size of model is only 7B? Is is necessary to provide experiments on large models?
  4. In the paper, the authors emphasize that they only need 1/500 -- 1/300 tokens of the model in QWen-VL. Is it fair to compare them on this metric?
Reviewer_3AjYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies structured supervision for vision-language pretraining. It converts PixMo-Cap captions into 704K serialized scene graphs using a text-only LLM, and optionally augments them with knowledge, temporal statements, negations, and affordances. Using the Molmo2 architecture and identical downstream fine-tuning, it compares caption, scene-graph, and combined supervision at multiple model scales. The main reported gains occur in out-of-domain evaluation, particularly for the augmented training recipe.

Critical strengths

The same-image comparison with caption-only pretraining and identical downstream fine-tuning is a valuable experimental design. Generating graphs without giving the teacher images avoids an additional image-conditioned teacher and makes the internal comparison more informative than external model rankings. The focus on the form of supervision is worthwhile without requiring a new architecture.

The augmented recipe shows encouraging transfer results:​ SGAug+Cap improves the reported out-of-domain average over Cap at all three scales while largely preserving in-domain performance. The paper also includes meaningful caption/graph, augmentation, and image-level mixture comparisons, and the proposed dataset and model release could provide a useful resource.

Critical weaknesses

The central attribution to structure is not sufficiently isolated. Graph targets are much longer than captions, and combined training introduces additional responses; loss reweighting does not equalize token or compute budgets. Synthetic rewriting and augmentation also change the supervision content. Pure SG improves the out-of-domain average over Cap by only 0.2 and 0.6 points at 2B and 4B, without reported uncertainty, while larger gains involve augmentation. Content-matched dense-prose controls and budget-controlled comparisons are needed to establish a graph-specific benefit. The external 200–500× token comparison is not a controlled estimate of this benefit, particularly since the internal Cap baseline is already strong.

The reliability of the synthetic supervision and adapted evaluations needs stronger quantitative support. A caption-only teacher cannot directly verify absent objects or temporal events, so the proposed negations and inferences require an audit of caption fidelity and image-grounded correctness. The authors report sampled quality checks, but their extent and error rates are not established in the supplied material. VisualCOMET is also converted to multiple choice and RB Temporal expanded, with construction details deferred to an appendix absent from this PDF. These details, together with uncertainty and aggregation-sensitivity analysis, matter for interpreting the small and heterogeneous generalization gains.

Questions for authors

Can you compare graph targets with dense prose expressing the same facts under controlled token/compute budgets, and report repeated-run uncertainty for SG versus Cap and the combined recipes?

What do quantitative audits show about content preservation and unsupported claims, particularly negations and temporal inferences? Can you provide the referenced appendix and clarify how visually unverifiable inferences are represented or filtered?

How are VisualCOMET distractors and the additional RB Temporal examples constructed and validated? Do the conclusions hold with benchmark-family aggregation and when considering unmodified evaluations separately?

Reviewer_WHwUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PixMo-SceneGraph, comprising 704K caption-derived scene graphs, and uses them to pretrain Molmo-based VLMs. It compares captions, graphs, and combinations, with optional synthetic knowledge and reasoning annotations. Results suggest modest benefits from restructuring alone and larger gains from augmentation.

Critical strengths
  1. Evaluation across three model scales and diverse tasks provides meaningful breadth.
  2. The dataset-generation pipeline requires no additional image annotation. Augmented graphs plus captions improve the reported out-of-domain average over captions by 2.5 points at 2B and 1.4 points at 4B, supporting the usefulness of enriched supervision.
Critical weaknesses
  1. Graphs average approximately 1,200 target tokens versus 200 for captions; augmented graphs reach 2,200. Equal epochs and loss reweighting do not establish equal training budgets. No comparison with prose containing the same augmented information is reported. Consequently, gains cannot be confidently attributed to graph structure rather than longer supervision or added knowledge.
  2. Unaugmented graphs improve the out-of-domain average by only 0.2/0.6 points at 2B/4B, without reported variability; graph-only results are absent at 8B (Table 2).
Questions for authors
  1. Does the advantage persist with matched training budgets and prose targets containing the same information, including augmentation?
  2. Although language models do not receive new visual observations, they can still leverage existing knowledge to fill in attributes, infer relationships, or make educated guesses about ambiguous references. What quantitative checks establish that unaugmented graphs preserve caption information without introducing unsupported inferences?
  3. Could you ablate temporal augmentation to assess whether gains on VisualCOMET and RB-Temporal+ primarily reflect task-specific supervision?
Reviewer_MQkaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that captions are ambiguous and omit “obvious” information (spatial, temporal, negation, counts). The motivation is valid but not new; it follows directly from [1]. The authors convert 704K PixMo-Cap captions into JSON scene graphs with a text-only LLM. This is a clean, controlled design, though the LLM still adds inferences and the targets are 6–11× longer. Scene graphs as VLM supervision already exist [2–5], so the novelty is limited to using them as the pretraining target, and unlike [3, 4] the graphs have no grounding. The schema is also used to add knowledge, temporal, negation and affordance fields. This is a nice idea, but the content is generated without the image and is unverified. In the results, structure alone is within the noise of captions, and the gains come mostly from augmentation on benchmarks that match those fields. The claim of beating Qwen3-VL [6] with far fewer tokens is overstated, since the caption-only baseline also beats it in-domain.

References [1] A. Kamath et al. Scale Can’t Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning. TACL, 2026.  [2] R. Herzig et al. Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs. EMNLP, 2023.  [3] J. S. Park et al. Synthetic Visual Genome. CVPR, 2025.  [4] W. Wang et al. The All-Seeing Project V2: Towards General Relation Comprehension of the Open World. ECCV, 2024.  [5] Z. Yang et al. BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs. arXiv, 2024.  [6] S. Bai et al. Qwen3-VL Technical Report. arXiv:2511.21631, 2025.

Critical strengths
  • Clean controlled design. Generating graphs from the caption alone, without the image, isolates the form of supervision. The released 704K dataset is a useful resource.
  • Schema as an augmentation interface. Fields that can be switched on and off allow targeted injection of omitted knowledge, with measurable gains in temporal and spatial reasoning at small scale.
Critical weaknesses
  • Structure is not shown to help. Scene-graph-only pretraining is within ±0.6 of caption-only, with no seeds or error bars. At 8B, SG+Cap ties or trails caption-only. The effect is confounded with target length and LLM rewriting.
  • The Qwen3-VL comparison is misleading. The caption-only baseline also beats Qwen3-VL in-domain at all scales, so the margin comes from fine-tuning on the benchmarks, not from scene graphs. The two models also don’t share the same vision encoder at 2B/4B [6].
  • Augmentation gains look benchmark-aligned. The augmented fields mirror the Reporting-Bias categories [1], two of which the authors modified. Gains concentrate there (VisualCOMET alone gives about half of the 2B gain), and negation augmentation hurts RB-Negation at 4B. The augmented content is unverified.
  • Missing baselines and ablations. There is no comparison to [2–4], no per-field ablation, and no localization evaluation despite the “grounding” claims.

References [1] A. Kamath et al. Scale Can’t Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning. TACL, 2026.  [2] R. Herzig et al. Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs. EMNLP, 2023.  [3] J. S. Park et al. Synthetic Visual Genome. CVPR, 2025.  [4] W. Wang et al. The All-Seeing Project V2: Towards General Relation Comprehension of the Open World. ECCV, 2024.  [5] Z. Yang et al. BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs. arXiv, 2024.  [6] S. Bai et al. Qwen3-VL Technical Report. arXiv:2511.21631, 2025.

Questions for authors
  • Compare JSON scene graphs against controls with the same information and length: an LLM-rewritten dense caption, and the same graph written as prose. Report results over at least 3 seeds.
  • Finetune Qwen3-VL on the same 8 datasets for a fair comparison. Also report the original VisualCOMET and RB-Temporal protocols and one localization benchmark (e.g., RefCOCO).
  • Ablate each augmentation field separately, and audit the accuracy of the generated negations and temporal statements against the images. Why does negation augmentation hurt RB-Negation at 4B?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.