acceptodds
Under review as a conference paper at ICLR 2027

ReSynth: Verified Synthetic Data for Multi-Image Understanding

Abstract

People often understand the world by comparing different views of it. Vision-language models (VLMs), however, receive these views as separate images and still struggle to relate them. One problem is data. Training on these tasks requires groups of images with a known relationship: the same object from different viewpoints, or the same scene before and after a change. Such examples are hard to collect at scale from existing photo collections. We introduce ReSynth, which creates them entirely from generated images. Localized editing produces before-and-after pairs, viewpoint generation produces multiple views of the same instance, and verified scene contents support comparisons of counts and attributes. Generated images do not always match their prompts, so ReSynth does not assume that a requested relation was successfully rendered. Instead, it verifies facts such as object counts and whether an edit really happened, then builds questions only when the answer is supported by those verified facts. Using GRPO on 13K ReSynth questions, with no real images during training, Qwen2.5-VL-7B improves by 3.8 to 5.1 points on BLINK, MUIRBench, and VLM2-Bench, while single-image performance remains essentially unchanged. In a controlled comparison, using the object counts actually visible in generated images rather than the counts requested in their prompts improves counting accuracy by 9.75points. ReSynth suggests that generated images can provide useful multi-image supervision. The key is to verify what the generated images actually contain before using them for training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-image understanding, synthetic data generation, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q384OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReSynth, a framework for constructing multi-image supervision from generated images. ReSynth plans and generates candidate relations first, then verifies the resulting visual evidence and reuses it to construct questions across five multi-image abilities. The authors train Qwen2.5-VL-7B using GRPO and report improvements on BLINK, MUIRBench, and VLM2-Bench. Results show that RL on these generated images alone can improve multi-image understanding, even better than reported score obtained with real-world image data.

Critical strengths

The paper addresses a meaningful data bottleneck in multi-image understanding. Instead of relying only on existing image collections, ReSynth actively constructs cross-image relations such as viewpoint changes and localized edits.

Critical weaknesses
  1. The "verification" mechanism is not validated sufficiently. The authors claim the pipeline relies on "verified" synthetic supervision, but the verification is performed by a MLLM rather than by human annotation or review. The paper doesn't report human agreement or verification accuracy for counts, attributes, edits or same-instance relations. Since verification is central to ReSynth, this is a major missing validation.

  2. View-point generation is limited to simple and complete objects. The identity or cross-view supervision of constructed data are mainly from 'large, complete subjects in simple scenarios', which may limit generalization to real multi-image scenarios or real world cases.

  3. The method shows limited scaling to stronger models. Results with more data, and the gains on Qwen3-VL-8B are much smaller. It shows that he current data may only compensate for weaker models.

Questions for authors

1.How reliable is the automatic verification process? Could the authors provide human-audited accuracy or error rates for counts, attributes and same-instance verification?

  1. The scene-plan dimensions appear manually designed. The authors specify scene plan with main object, setting, activity, physcial relation, appearance, lighting, viewpoint and placement. Why these schema? Could the authors provide the reasons why these factors are sufficient or representative for multi-image understanding?

  2. The generated QA is checked for correctness, but not sufficiently for difficulty or representativeness. The paper does not establish whether the generated questions are sufficient or representative of multi-image reasoning.

  3. The five ability categories are not clearly justified as a comprehensive taxonomy. Identity, change, quantity, composition and correspondence are not mutually exclusive. For example, asking how many objects disappeared between two images belongs to "change" or "quantity"? Other dimensions of multi-image reasoning, such temporal ordering, motion, occlusion-based object permanence are not covered.

6 comments

Thank you for your suggestions on verification reliability and scene coverage. We add a human audit and supporting experiments, and respond point by point below.

Before addressing individual comments, we summarize the main updates. We switch to Qwen-Image-2.1 for generation and editing to address slow editing and inconsistent instruction following. We also use Qwen3-VL-8B for single-image annotation and local checks. This tests whether a model can help construct data that improves its own multi-image understanding. We expand the data from approximately 13K to 440K examples, with harder distractors and new viewpoint and temporal tasks. After training, 8B outperforms Qwen3-VL-235B on both BLINK and MUIRBench. We address your comments in light of these updates below.

W1 / Q1. Human Audit and Verification Reliability

  1. Z-Image did not consistently follow count and attribute instructions, while HunyuanImage editing was slow. We switch to Qwen-Image-2.1 for both generation and editing to improve instruction following and construction efficiency.

    We add a human audit of 800 samples to assess automatic verification. For each of counting, attributes, edits, and instances, we inspect 100 pass-group and 100 fail-group samples. We check each image against its annotation.

    Verification-pass group (100 samples per check):

    Human audit checkCorrectIncorrectInsufficient Evidence
    Object count is correct8758
    Specified attribute is correct10000
    Requested edit is achieved9550
    Object appearance is consistent across images9181
    Total373189

    Verification-fail group (100 samples per check):

    Human audit checkCorrectIncorrectInsufficient Evidence
    Object count is correct374617
    Specified attribute is correct70246
    Requested edit is achieved651817
    Object appearance is consistent across images70264
    Total24211444

    Annotation error rates are 4.5% in the pass group and 28.5% in the fail group. Insufficient Evidence covers occlusion, blurry details, or ambiguous counting units that prevent a reliable judgment.

  2. With better generation from Qwen-Image-2.1, the fail group also contains many correct or uncertain samples. We compare training on edit-check and instance-check pass/fail groups to test whether filtering still helps.

    VerificationEvaluation taskFail groupPass group
    Edit verificationMUIR spot-the-difference43.8245.29
    Edit verificationVLM2 change tracking23.7425.11
    Instance verificationVLM2 object comparison57.2260.56
    Instance verificationVLM2 person comparison66.5068.00

    The pass groups perform better on the tasks shown.

  3. As generation improves, clear generation errors become less frequent. Verification increasingly serves to filter ambiguous samples that are difficult to judge consistently. We further ask: how can single-image perception and generation be combined to construct effective multi-image supervision at scale? We expand the data from approximately 13K to 440K examples and introduce harder questions and distractors. We test whether stronger models continue to benefit and whether gains transfer to other multi-image and single-image tasks.


W2. Simple Scenes and Complete Subjects

  1. The original pipeline favors large, complete subjects in simple scenes because of limits in image generation and viewpoint editing. Preserving object appearance was harder with clutter, occlusion, or large viewpoint changes. We therefore prioritized images with clear relations. With Qwen-Image-2.1, we expand to different backgrounds, viewpoints, handheld settings, and partial occlusion. We also add clothing changes, similar-instance distractors, and local-detail variants. Semantic-part and geometric-point correspondence require matching local features rather than overall appearance alone.

  2. The following results use the 440K training corpus.

    ModelBLINK Multi-view ReasoningBLINK Visual Correspondence
    Qwen2.5-VL-7B Base44.3661.63
    ReSynth 7B93.9897.09
    Qwen3-VL-8B Base45.8687.21
    ReSynth 8B80.4597.09

W3. Scaling to Stronger Models

  1. Slow HunyuanImage editing limited the original data scale. Adding examples also did not increase difficulty. We switch to Qwen-Image-2.1, expand to 440K examples, and add similar-instance distractors, reversed change descriptions, and spatial and temporal tasks.

  2. Qwen3-VL-8B gains 6.10 percentage points across three main benchmarks. It exceeds the 235B model on BLINK and MUIRBench and also improves on additional benchmarks:

    ModelBLINKMUIRBenchVLM2-BenchMMIUDiCoBench
    Qwen2.5-VL-7B Base53.3456.2751.9852.9435.4
    ReSynth 7B68.6070.5475.6257.3343.4
    Qwen3-VL-8B Base65.4973.6578.7857.840.8
    ReSynth 8B74.7580.0881.3862.947.2
    Qwen3-VL-235B70.772.8———
  3. We do not construct training QA specifically for DiCoBench, yet both models improve on it. The 8B model's four-benchmark single-image average also rises from 66.60 to 69.26, extending the gains beyond multi-image tasks.


Q2. Choice of Scene-Planning Factors

  1. We choose these factors to control cross-image relations and scene diversity. Keeping an object fixed while changing its color creates an attribute-change pair. Keeping the same instance while changing the viewpoint creates a cross-view pair. Background, lighting, subject size, and scene complexity place these relations in varied visual conditions.

  2. Planning also reduces reliance on free-form caption generation, which tends to repeat a few subjects. In our analysis, the five most frequent subject categories account for 29.4% of freely generated descriptions. We therefore sample subjects and compatible scene factors before expanding them into captions. The planning ranges are:

    DimensionRange
    Objects491
    Scenes76 location types and 6,185 specific scene descriptions
    Physical relations27 templates
    Appearance4–11 choices per object type
    Lighting27 conditions
    Human activities20
    Viewpoint/pose5–6 choices per type
    Subject size5 levels
    Scene complexity3 levels

    The nominal Cartesian product gives approximately 1.33×10¹³ to 4.38×10¹³ combinations. During construction, we sample compatible combinations and specify which factors remain fixed and which change. These factors support controllable construction of perception-based multi-image tasks. We assess the resulting data through task coverage and transfer to external benchmarks.


Q3. Question Difficulty and Representativeness

  1. Question difficulty. Correct answers alone do not ensure that a question requires image comparison. We revise questions and distractors to reduce shortcuts. Change questions include reversed descriptions, requiring the model to identify what changed and in which direction. Counting questions include cross-image sums, quantity comparisons, and zero counts. Ordering questions use revised candidate permutations to reduce structural clues. Patch-completion distractors come from other images or outside the visible field of view, requiring the model to match the missing region to its context.

  2. Task representativeness. W3 above reports transfer results on MMIU, DiCoBench, and single-image tasks. We do not construct training questions specifically for DiCoBench, yet both models improve on it. The 8B model also improves on all four single-image benchmarks. These results show that the training gains extend to other tasks beyond the question types directly constructed.

  3. Improving generalization with RL. Beyond improving data construction, we also use RL to improve generalization to external tasks. On the same corpus, RL outperforms SFT on all three multi-image benchmarks:

    ModelTrainingBLINKMUIRBenchVLM2-Bench
    Qwen2.5-VL-7BSFT67.158.00.0
    Qwen2.5-VL-7BRL68.670.575.6
    Qwen3-VL-8BSFT62.158.40.0
    Qwen3-VL-8BRL74.880.181.4

    SFT scores 0.0 on VLM2-Bench because its outputs do not follow the required evaluation formats.

Q4. Capability Groups and Uncovered Dimensions

  1. Our groups draw on tasks in BLINK, MUIRBench, and related benchmarks. They are organized by image relations and construction methods, rather than mutually exclusive abilities.

    For example, asking how many objects disappeared involves both change and counting. If editing changes only object count within the same scene, we assign the question to change. If different scenes are combined for counting or comparison, we assign it to quantity. The grouping reflects construction, while a task can involve multiple abilities.

  2. We add viewpoint understanding and temporal understanding. Panorama rendering and geometric projection support surrounding-view and camera-turn questions. Video generation and program state sequences support action and state ordering.

    We primarily construct perception-based multi-image tasks. Although some dimensions remain uncovered, training on these tasks still generalizes across tasks. Gains transfer to DiCoBench and extend to abstract pattern induction in BLINK IQ Test:

    ModelDiCoBench gainBLINK IQ Test gain
    Qwen2.5-VL-7B8.006.00
    Qwen3-VL-8B6.4011.33

    The 7B and 8B models gain 6.00 and 11.33 points on BLINK IQ Test, respectively. Both models also improve on DiCoBench, for which we likewise construct no dedicated training questions.


Reviewer_ndV7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ReSynth, a framework for generating multi-image training data using only synthetic images. The method first generates images of different scenes, then uses image editing and viewpoint generation to create relationships between them. It verifies the image content and the generated relationships before constructing question-answer pairs. The verified images can be reused to create questions covering five abilities: identity, change, quantity, composition, and correspondence. The authors generate 13K training examples and use GRPO to train Qwen2.5-VL-7B, achieving an average improvement of ~4.2 points across three multi-image benchmarks.

Critical strengths

The paper shows that training on only 13K synthetic examples improves performance across three multi-image benchmarks, with an average gain of 4.25 percentage points on Qwen2.5-VL-7B.

The method verifies the content of generated images instead of relying on the original generation prompts. The ablation on object counting shows a 9.75 improvement when using verified data, providing evidence that verification can improve the quality of synthetic training supervision.

Critical weaknesses

The proposed framework combines several existing techniques for image generation, editing, verification, and QA construction. While the paper shows that the complete pipeline improves downstream performance, the experiments do not sufficiently isolate the contribution of its main components. In particular, verification is only evaluated for object counting, while the benefits of verifying cross-image relationships and reusing images across different tasks remain unclear.

While ReSynth improves Qwen2.5-VL-7B, the gains are considerably smaller for Qwen3-VL-8B, even when the training data is doubled. The improvements also vary across the five targeted abilities, with almost no improvement on composition (+0.3 points). These results raise questions about whether the generated examples are sufficiently challenging and whether the method can consistently improve multi-image understanding in stronger models.

Questions for authors

Can the authors provide ablation studies to better isolate the contributions of the main components? In particular, how much do the verification of edits and instance relationships, structured scene planning, and reuse of images across different tasks contribute to the final performance?

The improvements on Qwen3-VL-8B are considerably smaller than those on Qwen2.5-VL-7B. Do the authors have any analysis of why this happens and whether increasing the difficulty of the generated examples could improve performance on stronger models?

5 comments

Thank you for your suggestions on component contributions and data difficulty. We add controlled experiments and respond point by point below.

Before addressing individual comments, we summarize the main updates. We switch to Qwen-Image-2.1 for generation and editing to address slow editing and inconsistent instruction following. We also use Qwen3-VL-8B for single-image annotation and local checks. This tests whether a model can help construct data that improves its own multi-image understanding. We expand the data from approximately 13K to 440K examples, with harder distractors and new viewpoint and temporal tasks. After training, 8B outperforms Qwen3-VL-235B on both BLINK and MUIRBench. We address your comments in light of these updates below.

W1 / Q1. Contributions of the Main Components

  1. The original experiment mainly tests count verification. We add training comparisons for edit and instance verification, comparing data that pass or fail the corresponding checks.

    Qwen-Image-2.1 follows instructions more reliably. Human review found that some rejected samples are correct, while others are difficult to judge because of occlusion or unclear details. We therefore compare downstream training results to test whether filtering still improves supervision. For edit and instance verification separately, we use 1,000 examples in each pass and fail group.

    VerificationEvaluation taskFail groupPass groupGain
    Edit verificationMUIR spot-the-difference43.8245.29+1.47
    Edit verificationVLM2 change tracking23.7425.11+1.37
    Instance verificationVLM2 object comparison57.2260.56+3.34
    Instance verificationVLM2 person comparison66.5068.00+1.50

    Training on the pass groups improves editing-task scores by 1.37–1.47 points and instance-comparison scores by 1.50–3.34 points. These results extend the verification comparisons beyond counting.

  2. Image reuse addressed limited samples in the original pipeline. The same verified images supported identity judgments, grouping, and counting, improving image utilization. Faster generation and editing with Qwen-Image-2.1 now let us expand image and relation coverage directly. We rely less on repeatedly using a small image pool to increase the number of questions.


W2 / Q2. Stronger Models, Composition, and Data Difficulty

  1. Gains for the stronger model. Slow HunyuanImage editing limited the data scale. More examples alone did not make questions harder. We switch to Qwen-Image-2.1 and expand the dataset to 440K examples. To increase difficulty, we add similar-looking instances as distractors, requiring the model to distinguish fine details. We also add reversed change descriptions, requiring it to identify both what changed and in which direction. New spatial and temporal tasks require reasoning about viewpoints and event order. Qwen3-VL-8B gains 6.10 percentage points on average across the three main benchmarks. It exceeds Qwen3-VL-235B on both BLINK and MUIRBench.

    ModelBLINKMUIRBenchVLM2-Bench
    Qwen3-VL-8B Base65.4973.6578.78
    ReSynth 8B74.7580.0881.38
    Qwen3-VL-235B70.772.8—
  2. Improving Composition. Original patch-completion distractors often came from visible regions, so the model could eliminate content already shown. We instead use patches from other images or outside the visible field of view. This requires matching the missing region to its context. Training on the 440K corpus improves BLINK Jigsaw on both models:

    ModelBaseReSynth
    Qwen2.5-VL-7B53.3384.67
    Qwen3-VL-8B66.0088.00
  3. Transfer to other tasks. Both models improve on MMIU and DiCoBench. We do not construct training QA specifically for DiCoBench.

    ModelMMIUDiCoBench
    Qwen2.5-VL-7B Base52.9435.4
    Qwen2.5-VL-7B ReSynth57.3343.4
    Qwen3-VL-8B Base57.840.8
    Qwen3-VL-8B ReSynth62.947.2

    Gains also extend to single-image tasks. Qwen3-VL-8B improves on all four evaluated benchmarks.

    ModelMMStarMMMUMathVistaHallusion
    Qwen3-VL-8B Base70.962.275.557.8
    Qwen3-VL-8B ReSynth72.366.476.961.4

Thank you again for your feedback. We hope the added ablations and stronger-model results address your concerns. Please see our Revision Summary and revised manuscript.

With the deadline approaching, could you share any remaining concerns so we can address them in time?

If your concerns are resolved, we would appreciate your consideration of a higher score.

Reviewer_ndV7OpenReview ↗

reply to authors

Thanks to the authors for the detailed response and additional experiments. The new results on qwen and composition tasks are encouraging and address my second concern.

My main concern is still about what is driving the improvements. The dataset has grown from 13K to 440K examples, and the generation model and QA construction have also changed. While the additional verification experiments are useful, it is still difficult to separate the benefits of the proposed approach from those of the larger dataset and other changes so contribution of structured planning and image reuse remains unclear. I appreciate the additional work and the stronger results, but I think this is an important gap in evaluating the main contribution. I will therefore keep my original score.

Reviewer_KwnmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework for improving vision-language models using entirely generated training images. It addresses the difficulty of collecting image groups with meaningful relationships, such as different viewpoints of the same object or scenes before and after a change. The pipeline first generates source images from structured scene plans that control object categories, settings, activities, and visual relationships. It then verifies the rendered content and uses localized editing and viewpoint generation to create checked change and same-instance associations. These verified facts and associations form a reusable image pool from which questions are constructed across identity, change, quantity, composition, and correspondence. The resulting dataset contains approximately 13,000 examples, each involving two to eight images, and supports GRPO training with rule-based answer and format rewards. Extensive experiments demonstrate the effectiveness of the training pipeline

Critical strengths
  1. Many quantitative experiments show that training Qwen2.5-VL-7B on the generated dataset reallly improve the performance on principal benchmarks, reaching 58.4 on BLINK, 63.5 on MUIRBench, and 64.6 on VLM2-Bench
  2. The paper addresses an important problem with a clear and insightful formulation. The paper identifies the shortage of images with known relationships, such as shared identity, changed states, or complementary viewpoints
  3. The proposed framework is conceptually coherent from beginning to end and it also provides a meaningful demonstration that generated images can improve understanding beyond the synthetic training setting.
Critical weaknesses
  1. Section 4.3 changes intended counts to visible counts and replaces images whose counts cannot be verified.
  2. Table 2 primarily compares with results imported from other papers. Data volume, image tokens, rollout budgets, response prompts, and evaluation versions may differ, so the 1.9-point average advantage over MiCo/CGC does not establish superior data efficiency.
  3. Sections 3.2–3.3 use multimodal annotation, voting, and QA checks, with Qwen3.6-27B performing important construction roles. Repeated judgments from a related model can share systematic errors, especially for identity across synthesized views. This is sometimes referred as self-reference bias.
  4. ReSynth improves the weaker backbone, but simply expanding the dataset does not consistently improve performance: Qwen2.5-VL-7B peaks around 13K examples, and the larger datasets do not surpass that result Fig 5a. The stronger Qwen3-VL-8B gains much less from the original dataset and improves only modestly when the data are doubled in Fig 5b.
Questions for authors

Some more additional technical detail questions:

  1. What do 1× and 2× mean for data versus optimization steps, and how are scaling corpora sampled and checkpoints selected?
  2. Which model performs each verification stage, how many judgments are voted, and what exact thresholds reject images, associations, or QA?
2 comments

Thank you for your constructive review. We respond to your comments below.

Before addressing individual comments, we summarize the main updates. We switch to Qwen-Image-2.1 for generation and editing to address slow editing and inconsistent instruction following. We also use Qwen3-VL-8B for single-image annotation and local checks. This tests whether a model can help construct data that improves its own multi-image understanding. We expand the data from approximately 13K to 440K examples, with harder distractors and new viewpoint and temporal tasks. After training, 8B outperforms Qwen3-VL-235B on both BLINK and MUIRBench. We address your comments in light of these updates below.

W1. Design of the Counting Verification Experiment

  1. The original pipeline used Z-Image, which often generated counts that differed from the requested counts. We therefore corrected answers using the visible counts. After switching to Qwen-Image-2.1, count instructions are followed more reliably and erroneous samples are less frequent.

  2. To isolate the effect of count correction, we add a fixed-image comparison. Both groups use the same 1,000 multi-image QA examples. Only the answers differ: one group uses planned counts, and the other uses verified visible counts. Correction improves BLINK Counting from 49.17 to 50.00 and MUIR Counting from 40.60 to 43.59. These gains come from answer correction without replacing images.

  3. In subsequent data construction, we discard counting samples whose repeated judgments do not reach agreement. Human review found that these disagreements mainly involve occlusion, overlapping objects, or ambiguous counting units. A reliable count is difficult to establish in these cases. We filter them rather than retain them with corrected answers, keeping uncertain labels out of training.


W2. Fairness of the Comparison with MiCo/CGC

  1. MiCo and CGC do not provide open-source implementations that we can use to reproduce their training. We therefore cannot retrain and evaluate them under a unified setting.

  2. MiCo and CGC improve both data construction and training algorithms. We focus on improving the data and train the target model with standard GRPO. We scale the dataset to 440K examples and increase question difficulty through similar-instance distractors, reversed change descriptions, and other revisions. ReSynth then reaches an average score of 71.6 across the three main benchmarks, exceeding the reported averages of MiCo and CGC by approximately 11.3 percentage points.


W3. Related-Model Judgments and Systematic Bias

  1. We understand that related models can repeat the same errors. Qwen3-VL-8B mainly performs single-image localization, counting, and attribute annotation. For edit checks, we crop corresponding before-and-after regions and ask whether the specified change occurred. We combine these basic facts with generation relations to construct multi-image QA.

  2. Voting filters samples that are difficult to judge consistently, such as occluded objects or unclear changes. We also manually audited 400 samples that passed verification. We found 18 incorrect annotations, giving an annotation error rate of approximately 4.5% in the pass group.

  3. The construction model Qwen3-VL-8B gains 6.10 percentage points across three multi-image benchmarks. Its five-benchmark single-image average rises from 66.60 to 69.26. In the original pipeline, Z-Image followed instructions less reliably, so verification had to address more generation errors. With more reliable generation from Qwen-Image-2.1, we further investigate how single-image perception and generation can produce multi-image supervision. We study how greater scale and harder questions improve multi-image performance and transfer to other tasks.

W4. Data Scaling and Gains for Stronger Models

  1. We expand both data scale and relation coverage, and make the QA tasks more discriminative. Instance tasks include same-category distractors with similar appearances but different details. Change tasks include descriptions of the same change in the reverse direction. Correspondence tasks cover larger viewpoint and pose differences. Ordering and patch-completion tasks reduce shortcuts in candidate structure.

  2. Qwen3-VL-8B improves further despite serving as the perception model during construction. Gains also transfer to additional multi-image benchmarks.

    ModelBLINKMUIRBenchVLM2-BenchMMIUDiCoBench
    Qwen2.5-VL-7B Base53.3456.2751.9852.9435.4
    ReSynth 7B68.6070.5475.6257.3343.4
    Qwen3-VL-8B Base65.4973.6578.7857.840.8
    ReSynth 8B74.7580.0881.3862.947.2
    Qwen3-VL-235B70.772.8———

    The 8B model exceeds the 235B model on BLINK and MUIRBench. We do not construct training QA specifically for DiCoBench, yet both models improve on it.

  3. Training reward saturates earlier for 8B, while it continues to grow for 7B. We therefore use a halfway checkpoint for 8B and a full-training checkpoint for 7B. The stronger initial performance of 8B may leave less room for improvement on the current data. These results show that broader relation coverage and improved question construction can further improve stronger models.


Q1. Meaning of 1×/2× and Checkpoint Selection

  1. In the original paper, 1× and 2× mean approximately 13K and 26K distinct training examples, not repeated training on the same data.

  2. We expand the dataset to 440K examples and evaluate six data fractions, from 1/6 to the full corpus. Using only 1/6 improves the three-benchmark average by 14.91 points for 7B and 5.50 points for 8B.

  3. We select checkpoints based on training reward curves. The 8B reward saturates around halfway, while 7B continues to improve. We therefore use halfway and full-training checkpoints, respectively.


Q2. Verification Details

We use Qwen3-VL-8B for visual checks. Repeated sampling improves consistency and filters uncertain cases, such as occluded objects or unclear changes.

  1. Localization: sample 8 predictions and cluster same-label boxes at IoU ≥ 0.5. Keep stable boxes supported by at least 5 predictions; exclude other localization results.
  2. Counting: require at least 5 of 8 judgments to give the same count. Otherwise, do not use the count label to construct counting QA.
  3. Local editing: crop corresponding before-and-after regions to highlight small changes. Require at least 3 of 4 judgments to confirm the requested change. Otherwise, do not use the edit relation to construct change QA.

The appendix will provide the remaining thresholds, branch-specific acceptance rules, and QA construction and filtering rules.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

We thank the reviewers for their constructive feedback. Our main revision increases both data scale and question difficulty. We expand ReSynth from approximately 13K to 440K examples, introduce harder distractors, and add viewpoint and temporal tasks. We also update generation and editing to Qwen-Image-2.1 and add a human audit and controlled verification experiments.


Multi-image benchmarks

ModelBLINKMUIRBenchVLM2-Bench
Qwen2.5-VL-7B Base53.3456.2751.98
ReSynth 7B68.6070.5475.62
Qwen3-VL-8B Base65.4973.6578.78
ReSynth 8B74.7580.0881.38
Qwen3-VL-235B70.7072.80—

Base and trained models use the same Think evaluation protocol. ReSynth improves 7B and 8B by 17.73 and 6.10 average points across these three benchmarks.


Generalization to additional multi-image benchmarks

ModelMMIUDiCoBench
Qwen2.5-VL-7B Base52.9435.40
ReSynth 7B57.3343.40
Qwen3-VL-8B Base57.8040.80
ReSynth 8B62.9047.20

Both models improve on additional benchmarks, including DiCoBench, for which we construct no dedicated training questions.


Single-image benchmarks

ModelMMStarMMMUMathVistaHallusion
Qwen3-VL-8B Base70.9062.2075.5057.80
ReSynth 8B72.3066.4076.9061.40

Despite training on synthetic multi-image supervision, ReSynth 8B also improves on all four evaluated single-image benchmarks.

Additional experimental details, clarifications, and corrections have been incorporated into the revised manuscript PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.