- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The generation quality of open models degrades substantially as the number of reference images increases. This paper attributes the problem to a lack of training data containing many reference images and makes two contributions. First, it constructs MacroData, a dataset of 400K samples with up to 10 reference images each, covering four tasks: Customization, Illustration, Spatial, and Temporal. Second, it proposes MacroBench, a benchmark that evaluates performance as a function of the number of reference images. The authors fine-tune Bagel, OmniGen2, and Qwen-Image-Edit on MacroData and report improved performance on MacroBench and OmniContext.
Critical strengths
- The dataset is large-scale and rich in samples with many reference images. Beyond simple image composition, it unifies illustration generation, novel view synthesis, and next-frame prediction within a single framework, making it a valuable resource for the research community.
- Performance gains are observed consistently across all three base models. The paper also verifies agreement among multiple VLM judges and the correlation between the judge and human ratings. The ablations on data scale, the proportion of reference counts, and task combinations are also informative.
Critical weaknesses
The critical ones are Weaknesses 1 and 4.
-
The improvements with many reference images are not validated on an external benchmark independent of the training data. MacroBench uses the same pipeline and prompt format as the training data (MacroData). Although the samples do not overlap, the two come from the same distribution, so models trained on MacroData may have an advantage on MacroBench. Meanwhile, OmniContext, used as the external benchmark, contains at most three reference images, as the paper itself acknowledges. This is insufficient to objectively support the claim that performance improves with more reference images. Results on a multi-reference benchmark such as MultiBanana [2] would be desirable.
-
Differences from prior work and the novelty of the contribution are not sufficiently discussed. The paper claims that existing benchmarks are limited to customization and editing and include only a few reference images. However, several multi-reference benchmarks already exist [1, 2, 3, 4, 5].
-
For some tasks, it is questionable whether the intended capabilities are actually being measured.
- Spatial: The fine-tuned Bagel is reported to outperform Nano Banana Pro. However, the prompts use a dataset-specific spatial convention (e.g.,
From above, the sequence Front -> Left -> Back -> Right follows a clockwise order), so the score could improve simply by memorizing this convention. - Temporal: The task is framed as next-frame prediction, but the actual evaluation criteria are fidelity to the instruction (
ISCS) and natural continuity with the previous frames (CCS). If the instruction itself contains specific content of the ground-truth frame, the task does not measure the model's predictive ability. - Illustration: The paper itself describes this as a text-centric task (
Illustration-only wins its task because it is more text-heavy and semantically distinct), and it is unclear to what extent the reference images are actually used during generation. - Furthermore, an overall score computed as a simple average over tasks of such different natures is difficult to interpret.
- Spatial: The fine-tuned Bagel is reported to outperform Nano Banana Pro. However, the prompts use a dataset-specific spatial convention (e.g.,
-
Verification of training data quality is insufficient. Nano Banana Pro is used to generate supervision for the Customization task, but its output quality is questionable. Even closed models have been reported to degrade as the number of reference images increases [2], and in Table 14 of this paper, Nano Banana Pro's performance likewise declines as the number of references grows. It has also been pointed out that even closed models can introduce unintended modifications and attribute inconsistencies [1]. Nevertheless, the paper reports neither the pass rate of data filtering nor the results of any human quality inspection. If the authors claim that data quality is adequately ensured, they should state the verification procedure and results explicitly; if no such verification was performed, they should conduct one.
References
[1] Xia et al. DreamOmni2: Multimodal Instruction-based Generation and Editing, CVPR 2026.
[2] Oshima et al. MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation, CVPR 2026.
[3] Peng et al. DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation, ICLR 2025.
[4] Jang et al. Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models, NeurIPS 2024.
[5] Zhou et al. OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation, CVPR 2025.
Questions for authors
- Can you provide human evaluation results for the fine-tuned models (e.g., pairwise comparisons of outputs before and after fine-tuning)?
- How was the quality of the training data verified (e.g., filtering pass rates, whether human visual inspection was conducted, and the agreement rate)?
- Is the performance improvement maintained on multi-reference image generation benchmarks whose format differs from the one used in the paper (e.g., MultiBanana [2])?
If the authors address these questions and the weaknesses shown above in their manuscript, then I will consider raising the score.
9 comments
W3a — Does Spatial only reflect a memorized direction convention?
The fine-tuned model places new views relative to the references rather than recalling named directions. On real CO3D photographs with continuous camera specifications unseen in training, fine-tuning reduces the mean viewpoint error by 16.0° and raises the share of outputs within 30° of the target from 28% to 46%.
Each case gives three reference views of an object and a target camera written as a continuous orbit from the first view (e.g., "29.6° to the right, 9.7° above, at 0.90 × the distance"), without the ten direction names or the clockwise order. We selected 94 objects from 20 categories with fixed geometric criteria before generation and ran both models with identical inputs and seeds. We measure the rotation error to the true target view with VGGT poses, which are accurate to within 2° on the real photographs.
| 94 objects, 20 categories | Mean error ↓ | Median error ↓ | ≤ 15° | ≤ 30° | ≤ 45° | Better than BAGEL |
|---|---|---|---|---|---|---|
| BAGEL | 55.9° | 43.4° | 6% | 28% | 54% | — |
| BAGEL + MACRO | 39.9° | 31.6° | 12% | 46% | 64% | 60 / 94 |
| Δ [95% CI] | −16.0° [−24.1, −8.8] | −11.8° |
W3b — What the Temporal task and metrics measure
The review describes the two metrics with their roles swapped. ISCS (Image Sequence Consistency) scores whether identity, clothing, style and background are kept from the preceding frames. CCS (Context & Action Logic) scores whether the instructed event happens and the motion follows from the last frame. A model that ignores the frames fails on both.
The instruction is written from the full clip and can describe the target event, so the task is text-guided next-keyframe generation. A clip can continue in many valid ways, and the text specifies the intended one. The task remains hard, as base BAGEL scores only 4.27/10. To see how much of the gain depends on the description revealing the target, we rewrote every instruction from the input frames only and reran 100 cases:
| N = 100 | ISCS (input-only) | CCS (input-only) | Score (input-only) | ISCS (full) | CCS (full) | Score (full) |
|---|---|---|---|---|---|---|
| BAGEL | 3.79 | 4.23 | 3.95 | 4.02 | 4.63 | 4.27 |
| BAGEL + MACRO | 6.44 | 7.08 | 6.71 | 7.11 | 7.72 | 7.36 |
| Δ [95% CI] | +2.65 [1.97, 3.30] | +2.85 [2.14, 3.58] | +2.76 [2.12, 3.42] | +3.09 [2.43, 3.75] | +3.09 [2.37, 3.79] | +3.09 [2.44, 3.76] |
Without target information, fine-tuning still gains +2.76, close to the +3.09 with full descriptions, and the target-aware description adds about 0.65 for the fine-tuned model. We will rename the task to text-guided next-keyframe generation and report both settings.
W3c — Are the references used in Illustration?
The model relies on the reference images. Removing them lowers the score by 1.49, and replacing them with images from another article lowers it by 2.47.
The input is an article passage with its preceding images, and the target is the next image. The references carry details the text leaves open, such as the look of products or people and the page style. On 200 MacroBench cases (50 per bin, sampled deterministically) we ran BAGEL + MACRO with the correct references, the same number of references from a different article, and no references. All outputs are scored against the true article and references with the paper's TCS (text) and ICS (reference) rubric.
| BAGEL + MACRO, N = 200 | 1–3 | 4–5 | 6–7 | 8–10 | TCS | ICS | Score, all |
|---|---|---|---|---|---|---|---|
| Correct references | 6.46 | 5.57 | 6.57 | 6.38 | 6.37 | 6.17 | 6.24 |
| Other article's references | 4.60 | 2.62 | 3.62 | 4.25 | 3.98 | 3.65 | 3.77 |
| No references | 4.84 | 4.16 | 5.37 | 4.63 | 4.95 | 4.61 | 4.75 |
| Δ correct − none | +1.62 | +1.40 | +1.20 | +1.75 | +1.43 | +1.56 | +1.49 [1.04, 1.95] |
| Δ correct − mismatched | +1.86 | +2.94 | +2.95 | +2.13 | +2.38 | +2.52 | +2.47 [1.96, 2.97] |
Mismatched references score below no references, so wrong references mislead the model. The drop holds in all four bins and in both TCS and ICS, and excluding the 10 cases whose target duplicates a reference gives +1.56 and +2.48.
W3d — Averaging across tasks of different nature
All MacroBench tasks share the same size and sampling rule (250 cases per task and reference count), so the plain average is an equal-weight summary. It agrees with Tables 14–17, where fine-tuning improves every backbone on every task:
| Avg, Tables 14–17 | Customization | Illustration | Spatial | Temporal |
|---|---|---|---|---|
| BAGEL | 3.95 → 8.02 (+4.07) | 4.42 → 5.62 (+1.20) | 0.60 → 3.40 (+2.80) | 3.14 → 5.81 (+2.67) |
| OmniGen2 | 3.74 → 5.87 (+2.13) | 4.26 → 4.46 (+0.20) | 0.84 → 1.60 (+0.76) | 2.71 → 4.51 (+1.80) |
| Qwen-Image-Edit | 3.67 → 6.50 (+2.83) | 3.43 → 4.26 (+0.83) | 1.10 → 2.68 (+1.58) | 2.58 → 4.00 (+1.42) |
Dropping any one task leaves the Table 1 ranking essentially unchanged, with Bagel + MACRO third behind the two closed-source models. For comparing general capability, weighting tasks by difficulty or score variance may suit better, and we will discuss these alternatives in the paper.
W4a / Q2a — Construction pipeline, filtering and pass rates
Each task has a filter matched to its ground truth, and every reference must be needed to predict the target. Customization, the only task whose targets are generated, loses the most candidates as references increase (pass rate 95.7% at 1–3 references and 53.5% at 8–10). Stage counts, using candidates that reached the final filter as the denominator:
| Task | Source pool | Candidates | Pass quality rule | Pass rate | Selected | Kept of candidates |
|---|---|---|---|---|---|---|
| Customization | 189,692 images | 151,180 | 107,546 | 71.1% | 100,000 | 66.1% |
| Illustration | 48.1M image–text samples | 136,885 | 125,529 | 91.7% | 100,000 | 73.1% |
| Spatial | 127,454 assets | 120,000 | rule-based only | — | 100,000 | 83.3% |
| Temporal | 1M sampled videos | 138,244 | 113,672 | 82.2% | 100,000 | 72.3% |
- Customization. Human, object, scene, clothing and style images (OpenSubject, MVImgNet, DL3DV, Vibrant Clothes, WikiArt) are cleaned and aesthetically filtered. Sources are mixed at 8:6:3:2:1, an LLM rejects implausible combinations and writes the instruction, and Nano Banana Pro generates the target. Gemini-3-Flash scores the output against each reference and the instruction, and a candidate passes only if every score is ≥ 6.
- Illustration. A VLM selects anchor images in OmniCorpus-CC as targets, with the preceding text and images as context. Gemini-3-Pro rewrites the context, removes earlier images that do not contribute to the target, and scores training suitability and guidance. A candidate passes if both scores are ≥ 6 and 1–10 contributing references remain.
- Spatial. Untextured or low-saturation Objaverse assets and non-standard panoramas from DIT360, Pano360 and Polyhaven are removed. Input and target views are drawn from ten canonical directions, keeping only combinations with visual overlap.
- Temporal. OmniCorpus-YT videos are split into shots with TransNetV2, keyframes of the same scene are grouped by DINOv2 similarity, and Gemini-3-Flash scores temporal coherence (≥ 6 passes). The last frame becomes the target.
Passing candidates are then sampled evenly across reference bins. Pass rates by bin:
| References | Customization | Illustration | Temporal |
|---|---|---|---|
| 1–3 | 95.7% | 97.3% | 60.2% |
| 4–5 | 86.6% | 95.1% | 89.0% |
| 6–7 | 72.3% | 93.1% | 92.5% |
| 8–10 | 53.5% | 85.0% | 94.7% |
Most Customization rejections at high counts fail per-reference consistency, mirroring the drop of Nano Banana Pro in Table 14. Temporal shows the opposite trend because short clips often change little between shots and are filtered out for weak temporal progression.
W4b / Q2b — Human inspection of training-data quality
A blind human audit agrees with the automatic filter. Only 3 of 120 accepted samples (2.5%, 95% Wilson interval 0.9–7.1%) were judged bad, and overall agreement is 92.9% (Cohen κ = 0.86). We sampled 240 candidates deterministically from fixed shards, 10 accepted and 10 rejected for each of Customization, Illustration and Temporal in each reference bin. The annotator saw the references, instruction and target, but not the filter's decision. When scaling up, the error that matters is a bad sample the filter keeps (LLM pass, human fail), since it enters training. A good sample the filter rejects only reduces data volume.
| Task | LLM pass · human pass | LLM pass · human fail | LLM fail · human pass | LLM fail · human fail | Accepted but bad | Agreement | κ |
|---|---|---|---|---|---|---|---|
| Customization | 37 | 3 | 3 | 37 | 7.5% | 92.5% | 0.85 |
| Illustration | 40 | 0 | 2 | 38 | 0.0% | 97.5% | 0.95 |
| Temporal | 40 | 0 | 9 | 31 | 0.0% | 88.8% | 0.77 |
| All | 117 | 3 | 14 | 106 | 2.5% | 92.9% | 0.86 |
Of the 17 disagreements, 14 are samples the filter rejected and the annotator accepted, 9 of them Temporal sequences with limited motion. Spatial is not audited because each target is rendered from a 3D asset or projected from a panorama under known camera parameters, so it is the exact ground truth and the unique solution for the given pose. Spatial data is still filtered at the asset and view level (W4a). We will add the audit to the appendix.
Q1 — Pairwise human comparison before and after fine-tuning
Annotators prefer the fine-tuned output for every backbone, in 76–78% of non-tied judgments, and the preference grows with the number of references. Three annotators compared base and fine-tuned outputs, shown in random left/right order without model names, on 40 MacroBench cases per backbone (10 per task, 2 / 2 / 3 / 3 across reference bins, sampled deterministically) and chose the better one or a tie (360 judgments).
| Model | Fine-tuned better | Tie | Base better | Win rate w/o ties |
|---|---|---|---|---|
| Bagel | 67 | 32 | 21 | 76.1% |
| OmniGen2 | 60 | 43 | 17 | 77.9% |
| Qwen | 70 | 30 | 20 | 77.8% |
| All | 197 | 105 | 58 | 77.3% |
| References | Votes | Fine-tuned better | Tie | Base better | Win rate w/o ties |
|---|---|---|---|---|---|
| 1–3 | 72 | 29 | 31 | 12 | 70.7% |
| 4–5 | 72 | 41 | 18 | 13 | 75.9% |
| 6–7 | 108 | 60 | 32 | 16 | 78.9% |
| 8–10 | 108 | 67 | 24 | 17 | 79.8% |
We thank the reviewer for the thorough assessment and constructive suggestions. We agree that an external multi-reference evaluation and a direct audit of data quality are essential, and that our characterization of prior benchmarks was too broad. Below we address each point in order.
W1 / Q3 — External evaluation on MultiBanana
Fine-tuning on MacroData improves both backbones on MultiBanana, and the gain is larger with more references: +1.99 overall and +2.37 at 6–8 references on BAGEL, and +0.75 overall and +0.85 at 6–8 references on OmniGen2.
We evaluated base and fine-tuned BAGEL and OmniGen2 on all 1,247 tasks with 3–8 references in the public MultiBanana release. One task with an empty prompt is excluded for all models. Scoring follows the official protocol with the Qwen3-VL-8B-Instruct judge, five 10-point criteria and the 3:3:1:1:1 weighted total. Models in the same family share inputs, seeds and settings. Official OmniGen2 accepts five references, so for 6–8 references we extend its image-index embedding to ten rows as in Appendix C.2. The tables give the weighted total by task type and reference count in the layout of the MultiBanana paper (Local, Global and Back. are X−1 objects with a local edit, a global edit or a background, and Object is X objects).
BAGEL improves in all 24 cells and OmniGen2 in 22 of 24, with small dips on 3·Local (−0.08) and 7·Local (−0.05). All five criteria improve for both models, by +1.05 to +2.82 on BAGEL and +0.44 to +0.93 on OmniGen2, and every 95% bootstrap interval lies above zero. Our base-model scores closely match those reported by the MultiBanana authors with the same judge [2, supplementary Table 6] and by AutoRef (arXiv:2609.35530).
W2 — Relation to prior multi-reference work
We will revise the statement that existing benchmarks are limited to customization. Prior works either evaluate multi-reference generation without training data or target one sub-problem such as few-subject personalization. MACRO studies scaling to ten references across four capabilities (identity binding, interleaved illustration, 3D spatial reasoning and temporal continuation), and makes this regime trainable and measurable with a scalable 400K training set (52.5% with six or more references) and a benchmark stratified by task and reference count. We will add the following comparison to the related work section: