- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces RefTree, a data construction pipeline for multi-reference image generation. Starting from a complete target image, it decomposes people, objects, and visual controls into a concept tree, then uses this structure to construct reference images and composition instructions. The authors build approximately 480K training samples and introduce RefTree-Bench. Models fine-tuned on the resulting data show substantial improvements over existing open-weight methods on several public benchmarks and RefTree-Bench.
Critical strengths
- The structured data construction approach is valuable. The concept tree represents attachment relations between entities and modification relations between attributes and entities. This turns complex reference combinations into more specific generation and verification tasks.
- The dataset is large and covers varied reference types. Each sample contains 7.75 references on average and up to 20 references, spanning characters, objects, scenes, and other visual controls.
Critical weaknesses
- Facial diversity in the synthetic data may affect reference-conditioned face generation. Table 4 shows higher face similarity when real images are added to synthetic training data than when synthetic data are used alone. I would like to discuss whether faces produced by the image generator may have certain “average” features and insufficient identity diversity, causing the trained model to reproduce these common features when generating faces. The paper does not currently examine this possibility.
- The construction and supervision role of style references are insufficiently clear. The paper shows examples of style references but does not provide a complete case linking a target image, its style reference, and the final instruction. This makes it difficult to understand how these references are obtained and how they differ from sketch or layout guidance.
Questions for authors
-
Could the authors compare facial diversity in the real and synthetic data using an approach they consider appropriate, and discuss whether “average” facial features in generated images might help explain the improvement in face reference performance after adding real data?
-
Could the authors show one or several complete style-reference training cases, including the target image, reference images, and final instruction, and explain how the style references are obtained? For example, if the target image is a black-and-white line drawing, would a character reference preserve that style or be converted into a photorealistic image? How is this case distinguished from sketch or layout guidance?
3 comments
Facial diversity of real and synthetic data
We thank the reviewer for this insightful question. We have now measured the face distribution and similarity directly, and the results confirm the reviewer's intuition and our motivation to add the real image data:
- Real faces are more diverse than synthetic faces, and synthetic faces partly share a common "average" component.
- This helps explain Table 4. Real faces are highly diverse, so they reward transferring the reference identity rather than reproducing the generator's common features. This is also a main motivation for including real images.
- Extracting references with the editing model does not remove the diversity advantage of real data.
Setup. We compare four face sets from the RefTree real and synthetic branches, excluding public data: faces in real targets, in the references extracted from them, in synthetic targets, and in the references extracted from those. We embed faces with ArcFace, the face-recognition model our pipeline uses to verify human references. We detected about 54K faces in 38K real targets and 56K faces in 26K synthetic targets (and about 55K faces in the references of each branch). After removing duplicate faces, we randomly sample 42,277 faces per set, the size of the smallest set, so that all four sets are compared at the same size. We report four metrics:
- mean pairwise cosine ↓ between two random faces (≈ 0 for unrelated identities);
- similarity to the average face ↓, the cosine between each face and the mean embedding of its set;
- Vendi score ↑ (Friedman & Dieng, 2023), a diversity score computed from all pairwise similarities. It equals 1 if all faces are identical and grows as they become more distinct (here at most 512, the embedding dimension). It is a relative measure for comparing sets, not a count of the people in a set;
- same-identity pairs ↓, the fraction of pairs with cosine > 0.4, which ArcFace would treat as the same person.
| Faces (42,277 per set) | Mean pairwise cos. ↓ | Sim. to average face ↓ | Vendi ↑ | Same-identity pairs ↓ |
|---|---|---|---|---|
| Real targets | 0.006 | 0.077 | 325 | 0.035% |
| Synthetic targets | 0.115 | 0.340 | 196 | 0.637% |
| Real references | 0.028 | 0.169 | 296 | 0.018% |
| Synthetic references | 0.085 | 0.292 | 225 | 0.148% |
Real faces are more diverse. Real targets are more diverse on all four metrics: two random real faces are nearly unrelated (mean cosine 0.006), and their Vendi score is higher (325 vs. 196). Synthetic faces still lie within the distribution of real faces, as they are as similar to the average real face as real faces are (0.073 vs. 0.076). However, they share a common component: their similarity to the average synthetic face is 0.340, while real faces are nearly orthogonal to it (0.016). This matches the reviewer's intuition that the generator tends to render faces with some common "average" features.
Effect of reference extraction. After editing:
- Real faces become slightly more similar to each other (Vendi 325 → 296), but same-identity pairs also become slightly rarer (0.035% → 0.018%).
- Synthetic faces become more diverse (Vendi 196 → 225).
- Real references still remain clearly more diverse than synthetic references on all four metrics.
Why adding real data helps (Table 4). With synthetic data only (C), reproducing the generator's common features is rewarded almost as much as copying the reference identity. Real targets are more diverse, so only a model that transfers the reference identity fits them well. We believe this explains why adding the real branch (C → D) raises face similarity by 21.1% (0.19 → 0.23), although the synthetic branch provides more complex compositions.
We added this analysis to App. F.
Response to Authors
Thank you for the extensive additional experiments and thoughtful discussion. The responses have addressed my concerns, and I have no further concerns. I look forward to the public release of the models and datasets.
Construction of style references and distinction from sketch / layout
Thank you for the questions. We added the construction of style references and complete training cases to App. A.4 and A.5, and summarize them below.
1. Construction and supervision of style references. The style reference is not extracted from the target image, since most of our targets are natural images without a distinct artistic style. Following MACRO and OmniRef, we sample it from an existing pool of style exemplars, each profiled by its attributes, such as color usage and brushwork. We then give the image editing model the target image, the style exemplar, and its profile, and the editing model transfers the target into this style while keeping the content and the identity of every person and object. The stylized image becomes the new target, and the exemplar becomes the style reference. Only the target is replaced: the references of people and objects stay as extracted from the original target, so the model learns to take the identity from them and the rendering style from the style reference. Finally, the instruction of such a sample states explicitly that the image should follow the style of this reference, e.g., "the overall artistic style should follow <img-6>". App. A.5 (Fig. 13) shows a complete case, and App. H (P23) gives the prompt of the editing model.
2. Non-photorealistic target images. Our pipeline does not require the extracted references to be photorealistic. If the target itself has an artistic style, e.g., a line drawing or a cartoon, the Extractor extracts the characters in this original style, and the prompt also states the style of the target. App. G.5 (Fig. 49) shows the output of our data engine for a frame of a 2D cartoon, in which both characters keep the cartoon style in their references. We find this question very constructive: since the majority of our targets are photorealistic, our current handling and analysis of non-photorealistic targets are limited, and we leave a dedicated treatment of different types of target images to future work.
At inference, we find that our model handles mixtures of photorealistic and non-photorealistic references well. App. G.5 (Fig. 50) shows three OmniContext cases with anime characters: the model places each character in a new scene in its anime style, and when an anime character is combined with photographs of a clip and a living room, it keeps the character in its anime style and renders the room and the clip photorealistically.
3. Distinguishing style from sketch and layout guidance. The instruction (input prompt) states the role of each control reference. All references in our training data are tagged with plain numbers (<img-1>, <img-2>, …), so the role is given by the text alone. For example:
In our inference results, the model is not confused by references in different styles: it does not mix up which reference gives the style, or whether a reference is a layout sketch or a style reference. App. G.4 (Figs. 46–48) shows four cases that combine a line-drawing sketch with a separate style reference, which are easy to confuse. In all of them, the model does not render the image as a line drawing: it follows the sketch only for the placement of the objects and the scene, and takes the rendering style from the style reference, e.g., ink wash or decorative gouache. In addition, for one sketch we generate three images that share the sketch and all content references and differ only in the style reference: none, an ink-wash painting, or a clay figure. All three keep the layout of the sketch. Without a style reference, the image is photorealistic; with the ink-wash reference, it is rendered as an ink-wash painting. The sketch thus determines only the layout, and the style reference only the rendering.