- Rating
- 3
- Confidence
- 3(−1)
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
YANAMI presents a large-scale multimodal dataset and benchmark for stateful, multi-turn visual interaction. The dataset contains 402,639 dialogues, 2,047,236 images, and 2,643,291 turns across image fusion, editing, world modeling, and visual reasoning. Raw images are drawn from four existing public datasets (Zebra-CoT, WEAVE, MiCo-150K, and MACRO); models filter the materials and construct new dialogues, while GPT-Image-2 and Nano Banana 2 support image generation and repair. The paper reports no count of unique source images, per-source contribution, or breakdown of reused versus generated/edited images. YANAMI-Bench contains 400 manually checked dialogues with 1,883 images, 3,173 turns, 1,600 dialogue-state-understanding questions, and 8,467 atomic visual checks. The authors evaluate 15 models and fine-tune BAGEL on YANAMI data. They report improvements on the benchmark and on external editing/understanding evaluations. The primary evaluation uses GT-History, replacing earlier assistant-generated images with ground-truth images; a narrower Self-History evaluation on BAGEL and Qwen-Edit shows lower performance.
Critical strengths
- YANAMI-Bench is substantial for a manually checked multi-turn benchmark: the authors report 400 dialogues and thousands of atomic visual checks. Three annotators verify retained benchmark samples, and a human-rating study reports Pearson correlations above 0.80 for the automatic judge on the tested model outputs.
- The experiments include both task-specific and external evaluations. The paper also reports a Self-History condition rather than presenting GT-History as the only protocol, which makes the deployment gap at least partially visible.
Critical weaknesses
- The novelty claim is not sufficiently differentiated from WEAVE (Chow et al., CVPR 2026), which already studies interleaved multi-turn comprehension/generation and evaluates visual memory, multi-turn generation, and world-knowledge reasoning. YANAMI does cite Chow et al. (2026b) in Related Work and lists WEAVE-100K in Appendix A.1, but gives little substantive analysis of the conceptual overlap; it then broadly states that prior datasets can be handled turn by turn without showing why that applies to WEAVE. YANAMI may contribute greater scale (its table lists 402,639 dialogues versus 100,750 for WEAVE-100K), six explicit capabilities, and a larger benchmark, but these are better framed as incremental extensions than as establishing a new research gap. A direct comparison and more careful novelty claim are needed.
- Overall scores use different weightings for models with and without visual-understanding capability, so cross-category ranking is not directly comparable. The automatic judge is checked on 300 ratings from three systems; this is useful but limited evidence for all criteria and all evaluated models. The paper should emphasize component metrics when comparing unlike model classes.
- The paper reports overall corpus statistics, four scenario shares, and dialogue-length statistics, but source-level composition remains unclear. Raw images come from four existing public datasets (Zebra-CoT, WEAVE, MiCo-150K, and MACRO) and are pooled into scenario-specific materials without a one-to-one source-to-scenario mapping. The paper does not report per-source retained counts, source-by-scenario composition, the number of unique raw source images, or the proportions of reused, generated, and edited image instances. Quality control for YANAMI-400K relies on permissive model-assisted filtering and automatic dialogue/image-text verification, whereas exhaustive human verification is described for the 400-dialogue benchmark. Without source- and scenario-stratified quality evidence, it is difficult to assess the training corpus’s representativeness, source-specific bias, and overall quality, or to determine how much the dataset contributes beyond new dialogue/state supervision and generated or edited turns. Thus, 2,047,236 is a count of image instances, not evidence of 2M unique new source images. The reported SHA-256 check rules out exact overlap between YANAMI-Bench and YANAMI-400K, but does not establish within-corpus deduplication or quantify reuse from the source datasets.
Questions for authors
- Can you report Self-History or full autoregressive rollouts across more evaluated models, including long dialogues and several consecutive image-generation turns, with failure accumulation measured over time?
- Can you compare YANAMI fine-tuning against a matched-volume baseline with the same image/turn count and optimization budget but without stateful supervision, and report the exact update/epoch schedule?
- Although the paper reports scenario shares and dialogue-length statistics, can you provide per-source retained counts, a source-by-scenario breakdown, unique raw-source image counts, and the proportions of reused, generated, and edited image instances? What within-corpus and cross-source duplicate or near-duplicate filtering was applied, and can you report source- and scenario-stratified quality audits?
I will consider updating the score if the author solves my concerns and gives more details.
21 comments
Q2: Performance of BAGEL + YANAMI under Self-History and as Dialogue Length Increases
Thank you for your valuable suggestions. We have added results for BAGEL + YANAMI under Self-History. These results show that the fine-tuned model retains its performance advantage and mitigates performance degradation caused by self-generated visual history. We have also added the relevant content to Appendix D.8 of the revised version.
| Model | GT-History | Self-History | Relative Drop |
|---|---|---|---|
| Ours | 0.69 | 0.65 | 5.80% |
| BAGEL | 0.53 | 0.48 | 9.43% |
| Qwen-Edit | 0.47 | 0.43 | 8.51% |
To examine failure accumulation over time, we also report scores at the first and last image-generation turns under Self-History for the original BAGEL, our fine-tuned BAGEL, and Qwen-Edit:
| Model | First Generation Turn | Last Generation Turn | Relative Drop |
|---|---|---|---|
| Ours | 0.7048 | 0.5786 | 17.90% |
| BAGEL | 0.5424 | 0.4286 | 20.97% |
| Qwen-Edit | 0.6468 | 0.3352 | 48.17% |
These results show that our model maintains its performance advantage at both the first and last generation turns and exhibits robustness under Self-History.
Q3: Comparison with a Baseline Matched in Image Count and Optimization Budget but without Stateful Supervision
Thank you for this valuable suggestion. We have added the following experiment to examine the contribution of our state-management design beyond the benefits of training on additional data. We have also added the relevant content to Appendix C and D.7 of the revised version.
Specifically, we introduce a control experiment using the same visual materials but their original supervision. The control model uses the images employed in YANAMI training, while its supervision comes from the original questions and answers in the corresponding source datasets. It does not use the cross-turn state-management dialogues and targeted supervision constructed by YANAMI.
YANAMI’s images primarily originate from four existing datasets, with some low-quality images regenerated or edited. The control model uses the same replacement images, ensuring that both groups share the same visual materials and image-quality improvements. Thus, this experiment compares the effects of source-dataset supervision and YANAMI’s state-management supervision, while controlling for the addition of image data and improvements in image quality.
Both groups use the same training configuration: 16 H100 GPUs, BF16 precision, and FSDP FULL_SHARD. Each GPU processes one packed sequence per step, with gradient accumulation set to 1. The target length of each packed sequence is 12,288 tokens, while both individual segments and packed sequences are capped at 16,384 tokens. Data are sampled at a 4G:1U ratio, and both models are trained for 13,500 optimizer updates. Here, batch size = 1 refers to one packed sequence per GPU rather than one original training sample. Each sequence may contain multiple independent samples, isolated through attention masks to reduce padding and improve computational efficiency. The results are as follows:
| Model | DSU | SC | VQ | Overall |
|---|---|---|---|---|
| Ours | 95.69 | 69.26 | 61.37 | 78.25 |
| BAGEL (Control Fine-Tuning) | 80.56 | 59.27 | 60.27 | 67.99 |
| BAGEL Base | 81.44 | 52.03 | 58.20 | 65.03 |
Fine-tuning BAGEL on the control data yields an Overall score of 67.99 on YANAMI-Bench, an improvement of 2.96 points over the original model. This remains substantially below the 78.25 achieved by fine-tuning on YANAMI with its state-management design. These results support the additional value of our dialogue-state supervision design.
Q1: Differences between YANAMI and WEAVE
Thank you for your valuable comments. We agree that our discussion of the differences from WEAVE was too general. We provide a more specific comparison below and will incorporate these clarifications into the revised manuscript. We have also added the relevant content to Appendix A.1 and D.5 of the revised version.
WEAVE has explored history-dependent multi-turn generation, visual memory, and consistency. YANAMI’s primary distinction is that we explicitly construct currently valid requirements that are established, maintained, and revised through dialogue history—namely, dialogue-state modeling.
First, we inspected all WEAVEBench examples and sampled examples from its dataset. Both inspections indicate that WEAVE tasks can be completed without relying on historical dialogue text. In addition, Section 4.1 of WEAVE describes a partial in-context setting for some models: “using only self-generated images with explicitly mentioned visual context, excluding other historical interactions.” This setting further supports the observation that WEAVE tasks can be performed without historical dialogue text.
In contrast, in YANAMI, removing valid constraints or user corrections from earlier dialogue affects performance even when the historical images and current instruction remain unchanged. To examine this dependency, we remove the preceding textual dialogue history while keeping the visual inputs fixed. The Overall score drops by 23.35 points, demonstrating the need for dialogue-state modeling.
The distinction therefore concerns whether dialogue history carries a currently valid state that cannot be recovered from visual context alone, rather than merely whether a task involves multiple historical images or editing turns. YANAMI explicitly incorporates persistent constraints, user corrections, and constraint updates, distinguishing its effective dialogue-state management design from WEAVE.
Regarding evaluation, YANAMI additionally introduces four categories of dedicated state-understanding questions and examines the understanding–generation gap in unified multimodal models (UMMs). Specifically, in complex multi-turn dialogues spanning understanding, generation, and editing, can existing UMMs faithfully translate their understanding of the current dialogue state into image generation or editing?
Our evaluation shows that existing UMMs still have substantial room for improvement in closing this understanding–generation gap. This provides a finer-grained diagnosis of failures in state-dependent interactions.