acceptodds
Under review as a conference paper at ICLR 2027

RefTree: Scaling Reference-to-Image Generation via Hierarchical Image Decomposition

Abstract

Although open-weight image generation models increasingly support reference images as inputs, they lag substantially behind proprietary counterparts, particularly as references grow in number and form complex hierarchical dependencies. To address this bottleneck, we introduce RefTree, a scalable and extensible data engine for multi-reference image generation. RefTree recursively decomposes target images into trees of visual concepts, which guide collaborative agents to extract the concepts as reference images, verify their consistency, and compose reference-grounded prompts via bottom-up merging. Operating entirely without closed-source models, the engine yields RefTree-480K, a dataset of 480K verified samples with 3.72M reference images and up to 20 references per sample, spanning characters, objects, scenes, and abstract controls such as poses, attributes, and styles. In addition, we present RefTree-Bench, which evaluates multi-reference generation using tree-derived, fine-grained visual rubrics. Models fine-tuned on RefTree-480K substantially improve reference consistency and prompt following, achieving state-of-the-art open-weight performance across four public benchmarks and RefTree-Bench and even matching Nano Banana Pro on two public benchmarks. Code, datasets, and models will be released.

Primary area:
datasets and benchmarks
Keywords:
Image Editing, Diffusion Models, Controllable Image Generation, Multimodal Data Curation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_c8mROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RefTree, a data construction pipeline for multi-reference image generation. Starting from a complete target image, it decomposes people, objects, and visual controls into a concept tree, then uses this structure to construct reference images and composition instructions. The authors build approximately 480K training samples and introduce RefTree-Bench. Models fine-tuned on the resulting data show substantial improvements over existing open-weight methods on several public benchmarks and RefTree-Bench.

Critical strengths
  1. The structured data construction approach is valuable. The concept tree represents attachment relations between entities and modification relations between attributes and entities. This turns complex reference combinations into more specific generation and verification tasks.
  2. The dataset is large and covers varied reference types. Each sample contains 7.75 references on average and up to 20 references, spanning characters, objects, scenes, and other visual controls.
Critical weaknesses
  1. Facial diversity in the synthetic data may affect reference-conditioned face generation. Table 4 shows higher face similarity when real images are added to synthetic training data than when synthetic data are used alone. I would like to discuss whether faces produced by the image generator may have certain “average” features and insufficient identity diversity, causing the trained model to reproduce these common features when generating faces. The paper does not currently examine this possibility.
  2. The construction and supervision role of style references are insufficiently clear. The paper shows examples of style references but does not provide a complete case linking a target image, its style reference, and the final instruction. This makes it difficult to understand how these references are obtained and how they differ from sketch or layout guidance.
Questions for authors
  1. Could the authors compare facial diversity in the real and synthetic data using an approach they consider appropriate, and discuss whether “average” facial features in generated images might help explain the improvement in face reference performance after adding real data?

  2. Could the authors show one or several complete style-reference training cases, including the target image, reference images, and final instruction, and explain how the style references are obtained? For example, if the target image is a black-and-white line drawing, would a character reference preserve that style or be converted into a photorealistic image? How is this case distinguished from sketch or layout guidance?

3 comments
AuthorsOpenReview ↗

Construction of style references and distinction from sketch / layout

Thank you for the questions. We added the construction of style references and complete training cases to App. A.4 and A.5, and summarize them below.

1. Construction and supervision of style references. The style reference is not extracted from the target image, since most of our targets are natural images without a distinct artistic style. Following MACRO and OmniRef, we sample it from an existing pool of style exemplars, each profiled by its attributes, such as color usage and brushwork. We then give the image editing model the target image, the style exemplar, and its profile, and the editing model transfers the target into this style while keeping the content and the identity of every person and object. The stylized image becomes the new target, and the exemplar becomes the style reference. Only the target is replaced: the references of people and objects stay as extracted from the original target, so the model learns to take the identity from them and the rendering style from the style reference. Finally, the instruction of such a sample states explicitly that the image should follow the style of this reference, e.g., "the overall artistic style should follow <img-6>". App. A.5 (Fig. 13) shows a complete case, and App. H (P23) gives the prompt of the editing model.

2. Non-photorealistic target images. Our pipeline does not require the extracted references to be photorealistic. If the target itself has an artistic style, e.g., a line drawing or a cartoon, the Extractor extracts the characters in this original style, and the prompt also states the style of the target. App. G.5 (Fig. 49) shows the output of our data engine for a frame of a 2D cartoon, in which both characters keep the cartoon style in their references. We find this question very constructive: since the majority of our targets are photorealistic, our current handling and analysis of non-photorealistic targets are limited, and we leave a dedicated treatment of different types of target images to future work.

At inference, we find that our model handles mixtures of photorealistic and non-photorealistic references well. App. G.5 (Fig. 50) shows three OmniContext cases with anime characters: the model places each character in a new scene in its anime style, and when an anime character is combined with photographs of a clip and a living room, it keeps the character in its anime style and renders the room and the clip photorealistically.

3. Distinguishing style from sketch and layout guidance. The instruction (input prompt) states the role of each control reference. All references in our training data are tagged with plain numbers (<img-1>, <img-2>, …), so the role is given by the text alone. For example:

  • a sketch: "laid out according to my sketch <img-5>";
  • a pose layout: "with the figures arranged as in <img-5>";
  • a style reference: "rendered in the visual style of <img-6>".

In our inference results, the model is not confused by references in different styles: it does not mix up which reference gives the style, or whether a reference is a layout sketch or a style reference. App. G.4 (Figs. 46–48) shows four cases that combine a line-drawing sketch with a separate style reference, which are easy to confuse. In all of them, the model does not render the image as a line drawing: it follows the sketch only for the placement of the objects and the scene, and takes the rendering style from the style reference, e.g., ink wash or decorative gouache. In addition, for one sketch we generate three images that share the sketch and all content references and differ only in the style reference: none, an ink-wash painting, or a clay figure. All three keep the layout of the sketch. Without a style reference, the image is photorealistic; with the ink-wash reference, it is rendered as an ink-wash painting. The sketch thus determines only the layout, and the style reference only the rendering.

AuthorsOpenReview ↗

Facial diversity of real and synthetic data

We thank the reviewer for this insightful question. We have now measured the face distribution and similarity directly, and the results confirm the reviewer's intuition and our motivation to add the real image data:

  1. Real faces are more diverse than synthetic faces, and synthetic faces partly share a common "average" component.
  2. This helps explain Table 4. Real faces are highly diverse, so they reward transferring the reference identity rather than reproducing the generator's common features. This is also a main motivation for including real images.
  3. Extracting references with the editing model does not remove the diversity advantage of real data.

Setup. We compare four face sets from the RefTree real and synthetic branches, excluding public data: faces in real targets, in the references extracted from them, in synthetic targets, and in the references extracted from those. We embed faces with ArcFace, the face-recognition model our pipeline uses to verify human references. We detected about 54K faces in 38K real targets and 56K faces in 26K synthetic targets (and about 55K faces in the references of each branch). After removing duplicate faces, we randomly sample 42,277 faces per set, the size of the smallest set, so that all four sets are compared at the same size. We report four metrics:

  • mean pairwise cosine ↓ between two random faces (≈ 0 for unrelated identities);
  • similarity to the average face ↓, the cosine between each face and the mean embedding of its set;
  • Vendi score ↑ (Friedman & Dieng, 2023), a diversity score computed from all pairwise similarities. It equals 1 if all faces are identical and grows as they become more distinct (here at most 512, the embedding dimension). It is a relative measure for comparing sets, not a count of the people in a set;
  • same-identity pairs ↓, the fraction of pairs with cosine > 0.4, which ArcFace would treat as the same person.
Faces (42,277 per set)Mean pairwise cos. ↓Sim. to average face ↓Vendi ↑Same-identity pairs ↓
Real targets0.0060.0773250.035%
Synthetic targets0.1150.3401960.637%
Real references0.0280.1692960.018%
Synthetic references0.0850.2922250.148%

Real faces are more diverse. Real targets are more diverse on all four metrics: two random real faces are nearly unrelated (mean cosine 0.006), and their Vendi score is higher (325 vs. 196). Synthetic faces still lie within the distribution of real faces, as they are as similar to the average real face as real faces are (0.073 vs. 0.076). However, they share a common component: their similarity to the average synthetic face is 0.340, while real faces are nearly orthogonal to it (0.016). This matches the reviewer's intuition that the generator tends to render faces with some common "average" features.

Effect of reference extraction. After editing:

  • Real faces become slightly more similar to each other (Vendi 325 → 296), but same-identity pairs also become slightly rarer (0.035% → 0.018%).
  • Synthetic faces become more diverse (Vendi 196 → 225).
  • Real references still remain clearly more diverse than synthetic references on all four metrics.

Why adding real data helps (Table 4). With synthetic data only (C), reproducing the generator's common features is rewarded almost as much as copying the reference identity. Real targets are more diverse, so only a model that transfers the reference identity fits them well. We believe this explains why adding the real branch (C → D) raises face similarity by 21.1% (0.19 → 0.23), although the synthetic branch provides more complex compositions.

We added this analysis to App. F.

Reviewer_c8mROpenReview ↗

Response to Authors

Thank you for the extensive additional experiments and thoughtful discussion. The responses have addressed my concerns, and I have no further concerns. I look forward to the public release of the models and datasets.

Reviewer_LF7POpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RefTree, a data generation pipeline. It decomposes a finished image into a "visual concept tree" consisting of entities (e.g., people and objects), controls (e.g., material, pose, and style), and the scene. Based on this tree, multiple agents crop and verify reference images, then write prompts. Using only open models, the authors build RefTree-480K, a dataset of 480K samples with up to 20 references each. They also propose RefTree-Bench, a benchmark that derives per-reference scoring criteria from the same tree. They fine-tune FLUX.2 [klein] and Cosmos3-Nano and report the best results among open models on four public benchmarks and on RefTree-Bench.

Critical strengths
  • The evaluation method, which lets you trace where each reference failed, effectively addresses the weakness of holistic scoring when the number of references is large.
  • Because the pipeline does not rely on closed models, it is less constrained by the performance ceiling or terms of use of a teacher model and is easier to scale.
  • The method achieves the best results among open models on all four external public benchmarks, and an ablation on data sources with the same base model is also provided.
Critical weaknesses

The critical one is Weakness 1.

  1. The contributions of the tree structure and of each agent are not disentangled. The core of the proposed method lies in (a) representing images as trees and (b) leakage-preventing cropping, verification, and bottom-up prompt composition. However, the only ablation in the data generation pipeline compares data-source combinations. What happens if the same procedure is applied to a flat decomposition that simply lists objects without a tree? It remains unclear whether the performance gains stem from the tree structure or simply from larger data volume, more references, or greater diversity.

  2. The quality of the training data has not been verified by humans. Training data quality is ensured only by automatic checks, such as judgments by Qwen3-VL-32B and face matching. No human inspection results show whether the trees are correct, whether information about other entities leaks into the references, or whether the correspondence between references and entities in the prompts is correct. The authors should also show how well the Verifier's judgments agree with human judgments and with other judges (e.g., a stronger VLM). Note that I understand that Table 5 shows agreement with human ratings for benchmarks.

  3. Performance as the number of references increases is shown only on RefTree-Bench. For evaluation with an increasing number of references, results on MultiBanana [1], which was constructed in a format different from the training data, would be desirable.

References.
[1] Oshima et al. MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation, CVPR 2026.

Questions for authors
  1. Can you provide comparisons with a flat decomposition that does not use the tree, and with variants in which each agent is removed?
  2. How well do the Verifier's judgments for the training data agree with humans and with other judges?
  3. Can you report results broken down by the number of references on external benchmarks such as MultiBanana?
4 comments
AuthorsOpenReview ↗

Ablations of the tree structure and of each agent

We thank the reviewer for raising this key question. We disentangle the tree structure and each agent with the ablations below. The tree structure changes the form of the samples, so we evaluate it by training on matched data; the agents act on the quality of each sample, so we evaluate the data they produce.

1. Tree structure: training on tree-structured vs. flat data. To separate the tree structure from data volume and reference count, we train the same Cosmos3-Nano backbone with the same optimization budget on two datasets of 144K samples each.

  • Tree (ours): 144K samples from RefTree-480K, subsampled toward fewer references to narrow the gap in reference count with the flat data (5.58 vs. 5.39 per sample).
  • Flat: 144K samples without any tree structure: the ~29K training samples that have no nested structure, plus samples that keep only the top-level objects, with their subtrees dropped and their references regenerated by extracting each object directly from the target; control, scene, style, and global layout references are kept.
Training data#SamplesRefs / sampleMACROOmniContextRCPFPQFacePose
Public263K4.338.228.445.920.677.440.080.41
Flat144K5.398.318.586.020.657.790.190.39
Tree (ours)144K5.588.468.617.020.717.810.210.42

RC, PF, PQ, Face, and Pose are the RefTree-Bench metrics of Tab. 2; Public is public-only training (setting A of Tab. 4). At equal size and a similar reference count, tree-structured data outperforms flat data on every metric, most of all in reference consistency on RefTree-Bench (7.02 vs. 6.02), whose cases contain nested references; on OmniContext, whose test cases are flat, the two are close. Tree-structured data also outperforms public-only training on every metric with about half as many samples (144K vs. 263K). The gains therefore come from the tree structure, not from data volume or reference count alone.

2. Extractor: leakage prevention. For each image node, our Extractor compiles the editing instruction from the tree: the profile to preserve, the non-folded descendants to remove or replace, and the nodes that overlap its region (Sec. 3.3). We compare it with a variant in which the VLM directly proposes the editing instruction without this subtree context, with the same VLM, editor, crop, and diversity augmentation. We evaluate 150 synthetic references whose subtree context is non-empty, e.g., a person whose garment is a separate reference. GPT-5.5 checks each reference for leakage, i.e., visible content that belongs to another node, and rates identity preservation (0–3).

Instruction generationLeakage ↓Identity (0–3) ↑
VLM proposes directly (no subtree context)29.3%2.91
Compiled from the subtree context (ours)2.0%2.89

The subtree context reduces leakage from 29.3% to 2.0% of references while preserving identity equally well. Without it, separately referenced garments and overlapping objects often remain in the reference, giving the trained model conflicting sources for the same content. This ablation also measures the leakage into references raised in Weakness 2.

3. Prompt Composer: bottom-up vs. direct composition. We compare our bottom-up Composer with direct composition, in which the same VLM (Qwen3-VL-32B) writes the prompt from the full tree in a single pass, with the same inputs and rules. We use 100 base samples and 100 samples with controls. A prompt is valid if it uses every reference tag; GPT-5.5 identifies which concept of the tree each tag is bound to, and a tag missing from the prompt counts as a wrong binding.

MetricBottom-up (ours)Direct
Valid prompts, base samples ↑100%92%
Valid prompts, samples with controls ↑81%25%
Correctly bound tags, base samples ↑88%76%
Correctly bound tags, samples with controls ↑94%25%

The bottom-up Composer binds 91% of all tags correctly (direct composition: 48%), and the gap is largest on samples with controls (94% vs. 25%): as references and control rules accumulate, direct composition loses track of the tags, dropping them or binding them to the wrong entities, and yields a valid prompt for only 25% of these samples. The bottom-up Composer largely avoids this because each call handles a single subtree, and controls are added to an already valid prompt. These rates are measured without retries; in our pipeline, a string check regenerates any prompt that misses a tag, so all training prompts are valid (App. D.3). Figs. 21–22 show both prompts for the same targets.

4. Verifier. The next thread compares the Verifier with human annotators and stronger VLMs and checks the correctness of our prompts (Task 3).

We added these ablations to App. D.

AuthorsOpenReview ↗

Human verification of the training data

Thanks for the constructive suggestions. We agree that human verification is very helpful, both for debugging the data pipeline and for making design decisions. We therefore added a human verification study: 5 annotators completed the three tasks below. Following your suggestion, we also compare the Verifier with two stronger judges on the references of Task 1: GPT-5.5, with the Verifier's prompt and inputs, and Claude Opus 5.5, with the annotators' guidelines. Please see App. E (Tabs. 14–15, Fig. 23) for all ratings, more details, and screenshots of the annotation interface.

Task 1: Reference quality. Annotators check whether a reference matches its entity in the target image (good, minor mismatch, or clear mismatch / missing). We use 200 references from final training samples and 100 references rejected by the Verifier, 150 synthetic and 150 real, each split evenly over people, objects, parts, and scenes. The table counts only the 200 references from final training samples, i.e., those that passed the Verifier.

SourceGood ↑Minor mismatchClear mismatch / missing ↓
Ours (synthetic)93%4%3%
Ours (real)81%14%5%

Only 3% (synthetic) and 5% (real) of the references that pass the Verifier are a clear mismatch. The others are rated good (93% and 81%) or a minor mismatch (4% and 14%), which passes the Verifier's threshold (a score of at least 2 of 3; see Q2).

Task 2: Image quality. 100 target images each from our synthetic targets, our real targets, MICo, and MACRO (400 in total), shown blind and in random order (good, medium, or poor).

SourceGood ↑MediumPoor ↓
MICo53%24%23%
MACRO85%11%4%
Ours (synthetic)86%12%2%
Ours (real)98%2%0%

Our targets are rated at least as natural as those of MACRO and MICo: 86% (synthetic) and 98% (real) are rated good, compared with 85% for MACRO and 53% for MICo.

Task 3: Prompt correctness. Annotators check whether a prompt is factually consistent with the target and all references, including whether every tag is bound to the right entity (correct, some factual errors, or severe errors). 100 prompts, 50 synthetic and 50 real.

SourceCorrect ↑Some factual errorsSevere errors ↓
Ours (synthetic)82%15%3%
Ours (real)91%6%3%

Only 3% of the prompts have severe errors. The Composer ablation in the previous thread reports an 81% valid rate on samples with controls; it is measured without retries. In our pipeline, a string check regenerates any prompt that misses a tag, so the other 19% are retried and every training prompt is valid. Wrong bindings can pass this check and remain as noise, which this task measures; given the current results, this noise does not justify a further binding check, which we leave to future work (App. D.3).

Q2: Agreement with the Verifier. On the Task 1 references, we measure how auto verifier and other rater make the same accept / reject decision on both passed and failed samples. Since the Verifier keeps references with a score of at least 2 of 3, we count good and minor mismatch as acceptance.

RatersVerifier-accepted ↑Verifier-rejected ↑
Verifier vs. humans96%65%
Verifier vs. GPT-5.594%52%
Verifier vs. Claude Opus 5.596%58%
  • The references the Verifier accepts are reliable. These are the references that enter training as images, and almost all of them are also accepted by the annotators (96%) and by both frontier models (94% and 96%).
  • On the references it rejects, the Verifier is clearly stricter than the other raters, and being strict is not wrong. The other raters confirm 52–65% of its rejections. A rejected reference is never used as an image: the node falls back to text and is described in the prompt (Sec. 3.3). A false rejection therefore only turns an image condition into a text condition, whereas a false acceptance would bind a wrong image to an entity.
AuthorsOpenReview ↗

MultiBanana and performance vs. number of references

Thank you for the suggestion. MultiBanana is an interesting benchmark: it contains many rare concepts and challenging cases. It is constructed in a format different from our training data, which makes it a good out-of-distribution test for our models. We have included it in the revised version. We followed the official evaluation code. The official protocol averages the scores of two judges, GPT-5 and Gemini 2.5 Flash. Since Gemini 2.5 Flash has been retired, we replaced it with the closest available model, Gemini 3 Flash, so our scores are not directly comparable with those reported in the MultiBanana paper. We therefore re-evaluated the models of Tab. 2 under the same protocol.

Results by task category (1–10, following Table 3 of MultiBanana; best open-weight result in bold):

ModelSingleTwoX Obj.X−1 Obj. + LocalX−1 Obj. + GlobalX−1 Obj. + Backgr.
GPT-Image-27.867.066.576.237.386.13
Nano Banana Pro7.686.495.555.166.074.93
OmniGen23.953.082.512.632.522.39
Qwen-Image-Edit-25116.804.042.432.292.492.41
DreamOmni25.173.072.382.482.482.27
FLUX.2 [klein] Base 9B6.563.973.783.353.073.05
FLUX.2-dev6.494.163.783.343.433.16
HiDream-O15.783.283.442.862.722.58
FireRed-Image-Edit-1.16.584.143.793.043.413.43
BAGEL + MICo5.172.972.382.482.252.14
Qwen-Image + MICo6.463.303.182.883.062.58
BAGEL + MACRO5.463.063.092.523.082.76
Qwen-Image-Edit-2511 + MACRO6.513.923.212.903.673.39
Qwen-Image-Edit-2511 + DyRef5.703.813.433.013.503.20
Cosmos3-Nano + RefTree6.315.424.664.175.514.49
FLUX.2 [klein] Base 9B + RefTree7.136.315.424.985.994.98

Score by number of references (weighted total on the multi-reference tasks, as in Fig. 5 of MultiBanana; Drop: change from 3 to 8 references; all models in App. B.3):

Model345678Drop
GPT-Image-26.846.626.796.536.366.31−8%
Nano Banana Pro6.015.735.535.235.164.91−18%
FLUX.2 [klein] Base 9B4.443.853.423.042.712.41−46%
FireRed-Image-Edit-1.14.543.943.493.142.852.54−44%
Qwen-Image-Edit-2511 + MACRO4.824.113.242.892.522.17−55%
Cosmos3-Nano + RefTree5.475.064.734.594.394.01−27%
FLUX.2 [klein] Base 9B + RefTree5.855.605.515.145.064.90−16%

Both RefTree-trained models outperform all open-weight baselines on the two-reference and all multi-reference tasks. They also degrade much more slowly as references are added: from 3 to 8 references, the open-weight baselines lose 39–64% of their weighted total, while our models lose 16% and 27%. The gain of FLUX.2 [klein] Base 9B + RefTree over its base model grows from 1.41 at 3 references to 2.49 at 8, where it matches Nano Banana Pro (4.90 vs. 4.91); GPT-Image-2 remains ahead. We added the weighted total versus the number of references to the main text (Fig. 4b) and the full results, including all five criteria, to App. B.3.

Thank you for the thorough response and the additional experiments. My concerns have been addressed, and the new results are convincing. I will raise my score.

As the other reviewer, I also look forward to the public release of the models and datasets.

Reviewer_L1STOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets image generation from many input references, aiming to close the gap to closed-source models. It names two obstacles, data and evaluation, and addresses both with RefTree, an agentic, tree-based, top-down approach. Unlike bottom-up pipelines, targets are not rendered by a proprietary teacher. The tree also allows flexible reference relationships and types. The outputs are RefTree-480K, which has more samples and references per sample than existing datasets, and RefTree-Bench. The benchmark has 1,006 tree-annotated cases and an evaluation protocol that agrees better with humans. Two base models are trained on the data. The gap to closed-source models is not closed, but both improve over the evaluated open-source baselines. Additional analyses cover performance vs reference count, comparison with training on public data, and a human study of the evaluation protocol. The main issues are:

  • thin method details on the agentic pipeline
  • no results for part of the advertised reference types
  • no ablation of the tree design I recommend weak accept: a valuable, well-motivated contribution with extensive evaluation, and gaps that a rebuttal could close.
Critical strengths

Motivated and thoughtful design.

  • Top-down extraction from real targets scales without a proprietary teacher.
  • Handles edge cases (verifier, text fallbacks) and models nested references of mixed types. Well-positioned against related work.

Positive results.

  • Detailed data statistics of RefTree-480K (Tab. 1, Fig. 2). A tree-based evaluation protocol in RefTree-Bench with higher agreement with human ratings (Tab. 5).
  • Multiple base models, one with and one without multi-reference ability.
  • Strong qualitative results (Figs. 8–19). Beats open-source models and approaches closed ones (Tab. 2), on public benchmarks as well as the proposed one.
  • Informative extended analysis. Performance vs. reference count (Tab. 3, Fig. 4) and comparison with public-data training (Tab. 4, Fig. 5).
Critical weaknesses

Missing details around the method. Sec. 3 relies heavily on agents, but the appendix gives no system prompts. Objectives are described only at a high level, leaving the intended heuristics ambiguous. Appendix tree examples would help: one from RefTree-480K and one from RefTree-Bench, marking node types, states, edge types, design-time leaves, and fallbacks.

Missing fundamental results and ablations.

  • There are no qualitative results for Style, Pose, Sketch, or Layout, which the teaser advertises. Figs. 8–19 show only characters, objects, attributes and scenes.
  • Only Face and Pose have dedicated metrics. Scene, Style, Attribute, Sketch, and Layout lack one.
  • No ablation backs the tree design (node/edge/state types, bottom-up Prompt Composer, dedicated Verifier). Nothing isolates these from simply training with more references gathered in a flatter, simpler manner.
Questions for authors

Method details.

  • In Tree Construction. Why are edge types (attachment vs modification) and node types (Entity, Control, Scene) needed, and what constrains each?
  • In Profiling and validation. What is “spatial support”? How are category, visual cues, and visibility used? What are text-only nodes at validation time, before the Extractor runs? Why is falling back to text valid (here and after verification), and how does it affect subtrees?
  • In Composition plan. Why is a “folded” state needed rather than image or text?
  • In Context-aware instruction compilation. How are plain and aggressive modes sampled, and what is their effect? Is scene inpainting done by an inpainting model?
  • The Prompt Composer “shortcut” claim isn’t convincing. A concrete example would help.
  • Stage names differ between Fig. 3 (panels 4-6) and Sec. 3 (Extractor, Verifier, Prompt Composer), please align them.

Benchmark.

  • Why is Pose&Layout only ~1.3% of references (Fig. 2), when poses and layouts can be extracted from any person or scene? A larger share would strengthen the claimed type diversity.
  • Unclear benchmark metrics. Which metrics use off-the-shelf scoring models vs. a VLM? How is each entity localized in the output?

Results.

  • Please add plain Cosmos3-Nano to Tabs. 2-3 and Fig. 4, as for FLUX.2 [klein] Base 9B, and data statistics (samples, average references) to Tab. 4 for a fairer comparison.
  • In Tab. 4, why is the synthetic subset (C) so surprisingly better than the real one (B)?
  • Minor: Fig. 1’s middle rows show neither boxes nor references. Fig. 5 is small and hard to understand. State the bold convention (best open-weight) in the Tab. 2 and 3 captions.
5 comments
AuthorsOpenReview ↗

Details of the data engine

We thank Reviewer L1ST for the detailed review. Due to the page limit, Sec. 3 focused on the high-level design. Sec. 3 now includes more details, including the control references, e.g., attribute, and sketch. The revised appendix now documents the full pipeline, with a step-by-step flowchart (App. A, Fig. 6) and the prompt of every agent (App. H); the data engine will also be released. Answers to the specific questions follow.

Tree examples. App. A.5 (Figs. 7–15) now shows annotated trees of synthetic, real, and RefTree-Bench samples, marking node types, edge types, and plan states, and points out the design-time leaves (control nodes) and the nodes that fell back to text after verification.

Edge and node types. The types tell each agent which procedure applies. An attachment edge links two things that both exist in the target (a garment on a person, a cup on a table); the child is a region inside its parent, so it can be extracted separately and removed from the parent's reference. A modification edge links a node to a property without a region of its own (the material of a watch, the texture of a garment). It cannot be extracted directly, so the Extractor transfers it onto a carrier such as a plain swatch (a gold watch yields a gold material ball) and flips it on the parent (a silver version of the same watch), keeping it out of the parent's reference. Entity nodes are grounded regions whose references are checked against the target region with a rubric. Control nodes (attributes, pose, style, lighting, sketch) are leaves that add no content of their own; they control how an entity or the whole image is rendered. Their references therefore specify a visual property or the overall layout rather than content, e.g., an unrelated exemplar for a style, a skeleton for a pose, or a sketch of the target for its layout. The single scene node is a child of the root; its reference is the target without the foreground and is also checked for leaked content.

Profiling and validation. Spatial support is the node's region in the target (bounding box and area fraction). It is used to crop the node for extraction and verification, to find overlapping nodes that must be excluded, and to give the Prompt Composer the anchor layout. Category routes a node to the matching extraction and verification procedure (e.g., ArcFace for people) and drives the dataset statistics. Visual cues are identity-bearing details (a logo, a print); the Extractor preserves them, and the Verifier turns them into rubric items. Visibility (occlusion, blur, prominence, quality) decides whether a node may become an image reference; for example, a heavily occluded desk cannot become an image reference because it cannot be extracted faithfully.

Text-only nodes and fallback. Text-only nodes are visual concepts that stay in the tree but are described directly with text in the prompt. They are never turned into a reference image. Falling back to text keeps the sample valid: the concept no longer has a reference image, but the prompt still describes it in words, so the model learns to generate it from the text. For subtrees, the fallback of a parent node is independent of its children: children with their own references stay valid. For example, if the man in Fig. 3 falls back, "the man <img-1> in the suit <img-2>" becomes "a dark-haired man in the suit <img-2>".

Folded state. The three states differ in what the parent's reference contains, and each mirrors a user input:

  • image: the child has its own reference and is removed from the parent's (a person and a separate photo of his coat);
  • text: the child is removed and described in words ("... wearing a coat");
  • folded: the child stays in the parent's reference without its own tag (a photo of a man already wearing his coat).

Plain/aggressive modes and scene. During training, we sample the plain and aggressive versions of each reference uniformly. Plain versions show the concept on a neutral background; aggressive versions also include larger changes in setting, viewpoint, and lighting. No separate inpainting model is used for the scene; our wording in Sec. 3.3 was imprecise and we have revised this part accordingly. The scene reference is produced in a single step: the editing model is instructed to remove all foreground anchors in one shot. A leakage check follows to identify whether all the foreground objects are removed.

Prompt Composer example. App. D.3 (Figs. 21–22) shows, for the same targets, that a prompt written from the full tree at once contains errors, such as tags bound to the wrong entity, that the bottom-up prompt avoids. We have also revised the wording of "shortcut" in Sec. 3.4 to make it clearer. See also our response on ablations for a quantitative comparison of both.

Stage names. Fixed: the panels of Fig. 3 now read Tree Constructor, Tree of Visual Concepts, Extractor, Verifier, and Prompt Composer.

AuthorsOpenReview ↗

Ablations of the tree design and the pipeline components

To isolate the contributions of the tree design and of the individual components, we ran three ablations. The Composer and the Verifier act on data quality, so we evaluate the data they produce; the tree design changes the form of the samples, so we evaluate it by training on matched data.

1. Prompt Composer: bottom-up vs. direct composition. We compare our bottom-up Composer with direct composition, in which the same VLM (Qwen3-VL-32B) writes the prompt from the full tree in a single pass, with the same inputs. A prompt is valid if it uses every reference tag; GPT-5.5 identifies which concept of the tree each tag is bound to, and a tag missing from the prompt counts as a wrong binding.

MetricBottom-up (ours)Direct
Valid prompts, base samples ↑100%92%
Valid prompts, samples with controls ↑81%25%
Correctly bound tags, base samples ↑88%76%
Correctly bound tags, samples with controls ↑94%25%

The bottom-up Composer binds 91% of all tags correctly, compared with 48% for direct composition. The gap is largest on samples with controls (94% vs. 25%): as references and control rules accumulate, direct composition loses track of the tags, dropping them or binding them to the wrong entities, and yields a valid prompt for only 25% of these samples. The bottom-up Composer largely avoids this because each call handles a single subtree, and controls are added to an already valid prompt. Figs. 21–22 show both prompts for the same targets.

Our Composer's 81% valid rate on samples with controls is measured without retries: in our pipeline, a string check regenerates any prompt that misses a tag, so the other 19% are retried and every training prompt is valid. Wrong bindings can pass this check and remain as noise, measured by the human evaluation below; given the current results, this noise does not justify a further binding check, which we leave to future work (App. D.3).

2. Verifier and prompt quality (human evaluation). Five annotators rated 300 reference images and 100 prompts; the full study is in our response to Reviewer LF7P.

  • References. Annotators compare each reference with its target region and rate it as good, minor mismatch, or clear mismatch / missing. They rate references from final training samples and references that the Verifier rejected, to measure how often its decisions agree with theirs.
  • Prompts. Annotators check whether each prompt from our pipeline is factually consistent with the target and the references, including whether every tag is bound to the right entity, and rate it as correct, some factual errors, or severe errors.
CheckSourceGood / correct ↑Minor / some errorsClear mismatch / severe errors ↓
References that passed the VerifierOurs (synthetic)93%4%3%
Ours (real)81%14%5%
Prompts from the bottom-up ComposerOurs (synthetic)82%15%3%
Ours (real)91%6%3%

Only 3% (synthetic) and 5% (real) of the references that pass the Verifier are a clear mismatch, and only 3% of the prompts have severe errors. The annotators confirm 96% of the Verifier's acceptances and 65% of its rejections; GPT-5.5 and Claude Opus 5.5 confirm 94% and 96% of its acceptances.

3. Tree design: training on tree-structured vs. flat data. To separate the tree design from "more references gathered in a flatter, simpler manner", we train the same Cosmos3-Nano backbone with the same optimization budget on two datasets of 144K samples each.

  • Tree (ours): 144K samples from RefTree-480K, subsampled toward fewer references to narrow the gap in reference count with the flat data (5.58 vs. 5.39 per sample).
  • Flat: 144K samples without any tree structure: the ~29K training samples that have no nested structure, plus samples that keep only the top-level objects, with their subtrees dropped and their references regenerated by extracting each object directly from the target; control, scene, style, and global layout references are kept.
Training data#SamplesRefs / sampleMACROOmniContextRCPFPQFacePose
Public263K4.338.228.445.920.677.440.080.41
Flat144K5.398.318.586.020.657.790.190.39
Tree (ours)144K5.588.468.617.020.717.810.210.42

RC, PF, PQ, Face, and Pose are the RefTree-Bench metrics of Tab. 2; Public is setting A of Tab. 4. The results show that 1) with about half as many samples (144K vs. 263K), our data still outperforms the public data combination on every metric, so its advantage does not come from quantity alone; 2) with the flat version, reference consistency on RefTree-Bench drops markedly (7.02 → 6.02), while the gain on OmniContext is small because its test cases are flat as well.

AuthorsOpenReview ↗

Style, pose, and sketch results and metrics

We thank the reviewer for the questions on control references and their metrics. This thread covers style, pose, and sketch; layout is discussed in a separate thread.

Qualitative results. App. G.4 (Figs. 39–48) now includes examples with sketch, style, and pose references, as well as depth and lighting controls, generated by FLUX.2 [klein] Base 9B fine-tuned on our data. The model follows each type of control. The examples also include cases that combine a sketch with a style reference, where the model correctly treats the line drawing as layout guidance rather than as a style to copy: the output follows the layout of the sketch but is not rendered as a line drawing.

Metrics. Pose and bounding-box layout can be measured deterministically: for pose, we re-detect keypoints in the output; for layout, we re-ground each generated entity and compare its position with the layout condition. For style, sketch, depth, attribute, and scene controls, it's difficult to find a uniform and reliable deterministic metric, so we score them with the VLM, which judges whether the specified visual property has been transferred to the output image or entity. App. C.3 (Fig. 19) adds examples of these VLM judgments on those control signal. In each, the judge names a concrete, visible discrepancy, such as a wrong number of people relative to the sketch, a background object missing from the depth map, or a photographic rendering where the style reference is a watercolor. In the paper, these global controls were averaged into reference consistency (RC) together with all other references; App. C.4 (Tab. 9) now reports the style, sketch, depth, and lighting scores separately for the compared models. Fine-tuning on our data raises FLUX.2 [klein] Base 9B on all four controls (style 0.26 → 0.78, sketch 1.60 → 2.35, depth 1.60 → 2.92, lighting 1.08 → 1.95 on the 0–3 scale). If the reviewer would like additional deterministic metrics for specific controls, we are happy to report them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal and the revision

We thank all reviewers and the AC for their time and constructive feedback. We are glad that the reviewers found the tree-based data construction well motivated and valuable (L1ST, c8mR), noted that the engine scales without a closed-source teacher model (L1ST, LF7P), appreciated that the tree-derived evaluation of RefTree-Bench traces where each reference fails and agrees better with human ratings (L1ST, LF7P), and recognized the large and diverse dataset (c8mR) and the best open-weight results on four public benchmarks and RefTree-Bench (L1ST, LF7P).

We have addressed every concern and question raised in the reviews with new experiments, analyses, and clarifications, and we revised the paper accordingly; all changes are marked in red in the revised PDF. The table below maps each concern to our response; the details are in our replies to each reviewer.

Concern (reviewer)What we didWhere in the paper
Missing details of the agentic pipeline; tree examples (L1ST)A step-by-step flowchart with the model of every step, the prompts of all agents, annotated trees of synthetic, real, and benchmark samples, and answers to all method questions; the data engine will be releasedSec. 3.3, App. A, A.5, H
No ablation of the tree structure and of each agent (L1ST, LF7P)Tree vs. flat data at equal size (144K samples each): tree data is better on every metric, e.g., reference consistency 7.02 vs. 6.02, so the gains do not come from data volume or reference count alone. Extractor: leakage into references drops from 29.3% to 2.0%. Prompt Composer: 91% vs. 48% correctly bound tags. Verifier: see the human study belowApp. D
No human verification of the training data; agreement of the Verifier with humans and other judges (LF7P, L1ST)A human study with five annotators on references, target images, and prompts; e.g., 93% (synthetic) and 81% (real) of the references that pass the Verifier are rated good. Almost all references that pass the Verifier are also accepted by the annotators (96%), GPT-5.5 (94%), and Claude Opus 5.5 (96%)App. E
No results or metrics for style, pose, sketch, and layout (L1ST)Qualitative results for style, pose, sketch, depth, lighting, and layout; separate scores for style, sketch, depth, and lighting, where fine-tuning on our data lifts FLUX.2 [klein] above GPT-Image-2 and Nano Banana Pro; a new bounding-box layout metricTab. 2, App. C.3–C.5, G.4, G.6
Many references evaluated only on our own benchmark (LF7P)MultiBanana by task and by number of references: best open-weight results on all multi-reference tasks; from 3 to 8 references, our models lose 16% and 27% of their score, the open-weight baselines 39–64%Fig. 4b, App. B.3
"Average" faces in synthetic data (c8mR)ArcFace analysis of 42K faces per set: real faces are more diverse (Vendi 325 vs. 196), and synthetic faces share a common component, which helps explain the gain in face similarity from real dataApp. F
Construction of style references; line-drawing targets; style vs. sketch (c8mR)The construction from a profiled exemplar pool with a complete training case; references of targets with an artistic style keep this style; sketch + style examples show that the model takes the layout from the sketch and the rendering from the style referenceSec. 3.3, App. A.4–A.5, G.4–G.5
Benchmark details and presentation (L1ST)Which metrics use which model and how entities are localized; share of pose and layout references (more control samples at release); plain Cosmos3-Nano; data sizes in Tab. 4; synthetic vs. real data; Fig. 3 stage names, Fig. 5 annotations, and bold conventionsSecs. 3 and 5, Tabs. 2–4, Figs. 3 and 5

We believe these revisions address all questions and concerns raised by the reviewers, and we are happy to provide further details during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.