acceptodds
Under review as a conference paper at ICLR 2027

YANAMI: A Large-Scale Multimodal Dataset and Benchmark For Multi-turn Stateful Interaction

39 views

Abstract

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal *stateful interaction*, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce **YANAMI** (**Y**our **A**ssistant **N**eeds to **A**ssimilate **M**ultiturn **I**nteraction), the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising **YANAMI-400K** and **YANAMI-Bench**. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios—Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning—and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates **Dialogue State Understanding** and **Visual Realization**, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states. Together, YANAMI provides a unified foundation for training and evaluating UMMs toward robust multimodal stateful interaction.

Primary area:
datasets and benchmarks
Keywords:
Unified Multimodal Models, Multi-Turn Multimodal Interaction, Multimodal Dialogue-State Management

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_W9tROpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

YANAMI presents a large-scale multimodal dataset and benchmark for stateful, multi-turn visual interaction. The dataset contains 402,639 dialogues, 2,047,236 images, and 2,643,291 turns across image fusion, editing, world modeling, and visual reasoning. Raw images are drawn from four existing public datasets (Zebra-CoT, WEAVE, MiCo-150K, and MACRO); models filter the materials and construct new dialogues, while GPT-Image-2 and Nano Banana 2 support image generation and repair. The paper reports no count of unique source images, per-source contribution, or breakdown of reused versus generated/edited images. YANAMI-Bench contains 400 manually checked dialogues with 1,883 images, 3,173 turns, 1,600 dialogue-state-understanding questions, and 8,467 atomic visual checks. The authors evaluate 15 models and fine-tune BAGEL on YANAMI data. They report improvements on the benchmark and on external editing/understanding evaluations. The primary evaluation uses GT-History, replacing earlier assistant-generated images with ground-truth images; a narrower Self-History evaluation on BAGEL and Qwen-Edit shows lower performance.

Critical strengths
  • YANAMI-Bench is substantial for a manually checked multi-turn benchmark: the authors report 400 dialogues and thousands of atomic visual checks. Three annotators verify retained benchmark samples, and a human-rating study reports Pearson correlations above 0.80 for the automatic judge on the tested model outputs.
  • The experiments include both task-specific and external evaluations. The paper also reports a Self-History condition rather than presenting GT-History as the only protocol, which makes the deployment gap at least partially visible.
Critical weaknesses
  • The novelty claim is not sufficiently differentiated from WEAVE (Chow et al., CVPR 2026), which already studies interleaved multi-turn comprehension/generation and evaluates visual memory, multi-turn generation, and world-knowledge reasoning. YANAMI does cite Chow et al. (2026b) in Related Work and lists WEAVE-100K in Appendix A.1, but gives little substantive analysis of the conceptual overlap; it then broadly states that prior datasets can be handled turn by turn without showing why that applies to WEAVE. YANAMI may contribute greater scale (its table lists 402,639 dialogues versus 100,750 for WEAVE-100K), six explicit capabilities, and a larger benchmark, but these are better framed as incremental extensions than as establishing a new research gap. A direct comparison and more careful novelty claim are needed.
  • Overall scores use different weightings for models with and without visual-understanding capability, so cross-category ranking is not directly comparable. The automatic judge is checked on 300 ratings from three systems; this is useful but limited evidence for all criteria and all evaluated models. The paper should emphasize component metrics when comparing unlike model classes.
  • The paper reports overall corpus statistics, four scenario shares, and dialogue-length statistics, but source-level composition remains unclear. Raw images come from four existing public datasets (Zebra-CoT, WEAVE, MiCo-150K, and MACRO) and are pooled into scenario-specific materials without a one-to-one source-to-scenario mapping. The paper does not report per-source retained counts, source-by-scenario composition, the number of unique raw source images, or the proportions of reused, generated, and edited image instances. Quality control for YANAMI-400K relies on permissive model-assisted filtering and automatic dialogue/image-text verification, whereas exhaustive human verification is described for the 400-dialogue benchmark. Without source- and scenario-stratified quality evidence, it is difficult to assess the training corpus’s representativeness, source-specific bias, and overall quality, or to determine how much the dataset contributes beyond new dialogue/state supervision and generated or edited turns. Thus, 2,047,236 is a count of image instances, not evidence of 2M unique new source images. The reported SHA-256 check rules out exact overlap between YANAMI-Bench and YANAMI-400K, but does not establish within-corpus deduplication or quantify reuse from the source datasets.
Questions for authors
  1. Can you report Self-History or full autoregressive rollouts across more evaluated models, including long dialogues and several consecutive image-generation turns, with failure accumulation measured over time?
  2. Can you compare YANAMI fine-tuning against a matched-volume baseline with the same image/turn count and optimization budget but without stateful supervision, and report the exact update/epoch schedule?
  3. Although the paper reports scenario shares and dialogue-length statistics, can you provide per-source retained counts, a source-by-scenario breakdown, unique raw-source image counts, and the proportions of reused, generated, and edited image instances? What within-corpus and cross-source duplicate or near-duplicate filtering was applied, and can you report source- and scenario-stratified quality audits?

I will consider updating the score if the author solves my concerns and gives more details.

21 comments

Q1: Differences between YANAMI and WEAVE

Thank you for your valuable comments. We agree that our discussion of the differences from WEAVE was too general. We provide a more specific comparison below and will incorporate these clarifications into the revised manuscript. We have also added the relevant content to Appendix A.1 and D.5 of the revised version.

WEAVE has explored history-dependent multi-turn generation, visual memory, and consistency. YANAMI’s primary distinction is that we explicitly construct currently valid requirements that are established, maintained, and revised through dialogue history—namely, dialogue-state modeling.

First, we inspected all WEAVEBench examples and sampled examples from its dataset. Both inspections indicate that WEAVE tasks can be completed without relying on historical dialogue text. In addition, Section 4.1 of WEAVE describes a partial in-context setting for some models: “using only self-generated images with explicitly mentioned visual context, excluding other historical interactions.” This setting further supports the observation that WEAVE tasks can be performed without historical dialogue text.

In contrast, in YANAMI, removing valid constraints or user corrections from earlier dialogue affects performance even when the historical images and current instruction remain unchanged. To examine this dependency, we remove the preceding textual dialogue history while keeping the visual inputs fixed. The Overall score drops by 23.35 points, demonstrating the need for dialogue-state modeling.

ModelHistoryOverall
OursFull GT-History78.25
OursHistorical Text Removed54.90

The distinction therefore concerns whether dialogue history carries a currently valid state that cannot be recovered from visual context alone, rather than merely whether a task involves multiple historical images or editing turns. YANAMI explicitly incorporates persistent constraints, user corrections, and constraint updates, distinguishing its effective dialogue-state management design from WEAVE.

Regarding evaluation, YANAMI additionally introduces four categories of dedicated state-understanding questions and examines the understanding–generation gap in unified multimodal models (UMMs). Specifically, in complex multi-turn dialogues spanning understanding, generation, and editing, can existing UMMs faithfully translate their understanding of the current dialogue state into image generation or editing?

Our evaluation shows that existing UMMs still have substantial room for improvement in closing this understanding–generation gap. This provides a finer-grained diagnosis of failures in state-dependent interactions.

Q2: Performance of BAGEL + YANAMI under Self-History and as Dialogue Length Increases

Thank you for your valuable suggestions. We have added results for BAGEL + YANAMI under Self-History. These results show that the fine-tuned model retains its performance advantage and mitigates performance degradation caused by self-generated visual history. We have also added the relevant content to Appendix D.8 of the revised version.

ModelGT-HistorySelf-HistoryRelative Drop
Ours0.690.655.80%
BAGEL0.530.489.43%
Qwen-Edit0.470.438.51%

To examine failure accumulation over time, we also report scores at the first and last image-generation turns under Self-History for the original BAGEL, our fine-tuned BAGEL, and Qwen-Edit:

ModelFirst Generation TurnLast Generation TurnRelative Drop
Ours0.70480.578617.90%
BAGEL0.54240.428620.97%
Qwen-Edit0.64680.335248.17%

These results show that our model maintains its performance advantage at both the first and last generation turns and exhibits robustness under Self-History.

Q3: Comparison with a Baseline Matched in Image Count and Optimization Budget but without Stateful Supervision

Thank you for this valuable suggestion. We have added the following experiment to examine the contribution of our state-management design beyond the benefits of training on additional data. We have also added the relevant content to Appendix C and D.7 of the revised version.

Specifically, we introduce a control experiment using the same visual materials but their original supervision. The control model uses the images employed in YANAMI training, while its supervision comes from the original questions and answers in the corresponding source datasets. It does not use the cross-turn state-management dialogues and targeted supervision constructed by YANAMI.

YANAMI’s images primarily originate from four existing datasets, with some low-quality images regenerated or edited. The control model uses the same replacement images, ensuring that both groups share the same visual materials and image-quality improvements. Thus, this experiment compares the effects of source-dataset supervision and YANAMI’s state-management supervision, while controlling for the addition of image data and improvements in image quality.

Both groups use the same training configuration: 16 H100 GPUs, BF16 precision, and FSDP FULL_SHARD. Each GPU processes one packed sequence per step, with gradient accumulation set to 1. The target length of each packed sequence is 12,288 tokens, while both individual segments and packed sequences are capped at 16,384 tokens. Data are sampled at a 4G:1U ratio, and both models are trained for 13,500 optimizer updates. Here, batch size = 1 refers to one packed sequence per GPU rather than one original training sample. Each sequence may contain multiple independent samples, isolated through attention masks to reduce padding and improve computational efficiency. The results are as follows:

ModelDSUSCVQOverall
Ours95.6969.2661.3778.25
BAGEL (Control Fine-Tuning)80.5659.2760.2767.99
BAGEL Base81.4452.0358.2065.03

Fine-tuning BAGEL on the control data yields an Overall score of 67.99 on YANAMI-Bench, an improvement of 2.96 points over the original model. This remains substantially below the 78.25 achieved by fine-tuning on YANAMI with its state-management design. These results support the additional value of our dialogue-state supervision design.

Reviewer_QGTEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces YANAMI, a dataset and benchmark for stateful multi-turn multimodal interaction. YANAMI-400K contains 400k dialogues across image fusion, image editing, world modeling, and visual reasoning tasks. YANAMI-Bench contains 400 human-verified dialogues and separately evaluates Dialogue State Understanding (DSU) and Visual Realization (VR). The authors evaluate open-source and closed-source models and demonstrate the dataset’s training utility by fine-tuning BAGEL, with improvements on both YANAMI-Bench and external generation and editing benchmarks.

Critical strengths
  1. The dataset and benchmark addresses a practically important problem for multi-turn multimodal interaction. With approximately 400K dialogues, 2.05 million images, and 2.64 million interaction turns across four scenarios, YANAMI has substantial potential as a training and research resource.
  2. The training benefits have been validated well. Fine-tuning BAGEL improves its YANAMI-Bench score from 65.02 to 78.25, while external GEdit-Bench-EN performance increases from 6.52 to 7.19 and GenEval from 0.73 to 0.75. The external improvements support the practical utility of the dataset beyond adaptation to the authors’ own benchmark.
Critical weaknesses
  1. The overall scores for YANAMI-Bench are not directly comparable across different model categories. Appendix B.3 uses 0.4DSU + 0.4SC + 0.2VQ for unified multimodal models and 0.6SC + 0.4VQ for image-only models. As shown in table 2, the score for different type of models is not comparable. The usage of YANAMI-Bench needs to be further clarified well.
  2. The benchmark’s dependence on earlier dialogue history is not quantitatively established. Although the examples and Table 6 motivate the need for multi-turn state tracking, no history-ablation experiment is reported. In particular, it remains unclear how much full dialogue history improves performance over the latest instruction with the same relevant visual inputs.
Questions for authors
  1. With visual inputs controlled, how much does performance decline when earlier textual history is removed or shuffled? Do these results establish that cross-turn state tracking is necessary?
  2. Evaluation clarification. Which cross-model conclusions remain valid when DSU is reported separately and all models are compared using a common SC/VQ metric?
4 comments

Q1: Removing Prior Textual History with Visual Inputs Held Fixed

Thank you for this valuable comment. We remove the preceding textual dialogue history while keeping the visual inputs unchanged. The Overall score drops by 23.35 points, from 78.25 to 54.90, demonstrating the need for cross-turn state tracking. We have also added the relevant content to Appendix D.5 of the revised version.

ModelHistoryOverall
OursFull GT-History78.25
OursHistorical Text Removed54.90

Q2: Results with DSU Reported Separately and Common SC/VQ Metrics Used Across All Models

Thank you for your valuable suggestions. The results below report DSU separately and compare all models using common SC/VQ metrics. We present the top six models here; the complete leaderboard will be included in the appendix of the revised manuscript. We have also added the relevant content to Appendix D.1 of the revised version.

BAGEL fine-tuned on YANAMI remains the highest-ranked open-source UMM and ranks second among all open-source models. It also achieves the highest SC score among all open-source models, demonstrating substantially improved dialogue-state consistency after fine-tuning.

FLUX.2 [dev], the only open-source model ranked above ours, has an advantage in Visual Quality (VQ). We believe this reflects the difference in model scale: FLUX.2 [dev] has 32B parameters, whereas BAGEL has approximately 7B active parameters and 14B total parameters. Despite having approximately 4.57 times as many active parameters, FLUX.2 [dev] achieves a lower SC score than our model, highlighting our model’s dialogue-state management capabilities.

Our original cross-model conclusions therefore remain valid when DSU is reported separately and all models are compared using common SC/VQ metrics. The First vs. Last Generation Turn and GT-History vs. Self-History analyses do not involve DSU and are therefore unaffected.

ModelSizeDSUSCVQOverall in SC and VQ score
GPT-Image-2UndisclosedN/A84.5880.7083.03
Nano Banana 2UndisclosedN/A83.2877.4080.93
FLUX.2 [dev]32BN/A64.5071.5067.30
BAGEL + YANAMI-400K7B active95.6969.2561.4066.11
Emu3.534B71.0063.4868.0865.32
UniPic 3.020BN/A62.5556.1059.97

Thank you again for reviewing our work. We would be happy to discuss any further questions you may have.

Thank you again for your thoughtful and constructive feedback. We look forward to continuing the discussion and welcome any further questions or suggestions you may have.

Dear Reviewer,

Thank you again for taking the time to review our submission. As the discussion period draws to a close, we wanted to follow up briefly on our rebuttal. If time permits, we would be grateful to hear whether our responses address your main concerns and whether you have any further questions about our submission or rebuttal. If any questions remain, we will do our best to address them in the time remaining.

Thank you again for your time and consideration. We wish you all the best in your future research.✨✨

Best regards,

The Authors

Reviewer_Wk7VOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces YANAMI-400K and YANAMI-Bench for training and evaluating stateful multimodal interaction. The work covers four interaction scenarios and emphasizes tracking, preserving, and updating dialogue states. Its evaluation distinguishes dialogue-state understanding from visual realization. Experiments demonstrate that fine-tuning BAGEL on the proposed dataset improves multimodal interaction and general image generation/editing performance.

Critical strengths
  1. The paper addresses a practically important problem: maintaining and updating user requirements throughout multimodal conversations. The large-scale training dataset and human-verified benchmark are potentially valuable resources for this research direction.
  2. Separating dialogue-state understanding from visual realization provides a useful diagnostic perspective. The improvements reported on both YANAMI-Bench and external image editing benchmarks support the practical utility of the dataset.
Critical weaknesses
  1. The positioning relative to prior visual dialogue and multi-turn multimodal understanding work is incomplete. The paper does not discuss Visual Dialog, MMDU, or MMCR. Visual Dialog studies answering image-grounded questions using dialogue history; MMDU provides multi-turn, multi-image understanding data and evaluation; and MMCR explicitly emphasizes contextual dependencies, referential consistency, and logical progression across turns. These works are relevant to the motivation and dialogue-state understanding component of YANAMI. Although YANAMI additionally addresses image generation/editing and explicit state updates, the manuscript should more clearly distinguish these contributions from established contextual understanding capabilities.
  2. The experimental support for the specific benefits of stateful supervision remains limited. The BAGEL improvements demonstrate the utility of the overall dataset, but do not by themselves separate the contribution of state-management design from additional training and data quality. Furthermore, the main evaluation uses GT-History, while the Self-History comparison in Figure 6(c) includes the original BAGEL and Qwen-Edit but not BAGEL fine-tuned on YANAMI. Consequently, the presented results do not directly establish whether the proposed training improves robustness when errors from earlier generated images affect subsequent turns.
Questions for authors
  1. Why are Visual Dialog, MMDU, and MMCR not discussed in the related work? Please clarify how YANAMI’s dialogue-state understanding tasks and state-management capabilities differ from the history-conditioned understanding and contextual reasoning studied in these works.
  2. Could the authors evaluate BAGEL before and after fine-tuning on MMDU and MMCR-Bench, or comparable established multi-turn understanding benchmarks, to assess whether the improvements generalize beyond YANAMI-Bench?
  3. What evidence isolates the contribution of the state-management design from additional training and data quality? A matched-budget comparison using the same visual content with and without cross-turn state dependencies would help clarify this contribution.
  4. Figure 6(c) reports performance degradation under Self-History for the original BAGEL and Qwen-Edit. Could the authors provide the corresponding results for BAGEL+YANAMI under the same evaluation conditions, and examine whether fine-tuning preserves its gains and reduces degradation caused by self-generated visual history?
9 comments

Q1 (Part 1): Differences between YANAMI and Visual Dialog, MMDU, and MMCR

Thank you for your valuable comments.

We apologize for omitting these works from the Related Work section due to space limitations. We have now added a discussion of Visual Dialog, MMDU, and MMCR, together with a clarification of their differences from YANAMI, to Appendix A.1 of the revised manuscript.

The differences between these works and YANAMI are as follows.

History-conditioned question answering, cross-turn reference resolution, and basic contextual reasoning—the three capabilities addressed by these datasets—are foundational to multi-turn dialogue. YANAMI also builds on these foundations.

However, in many dialogue tasks, users do more than ask questions or request comparisons and summaries of provided images. Interactions often also involve: (1) user corrections, which revise the model's earlier interpretation or information previously supplied by the user; and (2) persistent constraints, which are introduced in earlier turns and must remain satisfied in subsequent turns. Although these requirements also fall within contextual reasoning, they are not covered by the relatively simple contextual reasoning tasks addressed in the aforementioned works. Our experiments provide direct evidence of these challenges. In the Image Editing scenario, the original BAGEL achieves 82.0 on Constraint Tracking and 59.0 on Correction Updating, compared with 98.0 and 99.0 after fine-tuning on YANAMI (Appendix D.2, Table 17). These results show that retaining active constraints and incorporating user corrections remain challenging for the original model, and that YANAMI supervision improves these capabilities. This motivates both additional training supervision and corresponding benchmarks for evaluating models in these more complex dialogue settings. These constitute YANAMI's main distinctions from prior work at the multimodal understanding level.

Furthermore, as you noted, YANAMI is a unified multimodal dataset covering understanding, image generation, and image editing. We therefore incorporate these capabilities and constraints across all three task types, while additionally addressing consistency during generation and editing. Our work thus goes beyond extending existing multimodal multi-turn understanding tasks to image generation and editing. It addresses a more complex and realistic setting in which models must understand and faithfully realize the dialogue state established through user–model interactions, corrections, and constraints.

Q1 (Part 2): Dialogue-State Understanding and Visual Realization

In such dialogues spanning understanding, generation, and editing, two challenges arise: whether existing models correctly understand what should be done at the current turn, namely the current dialogue state, and whether they can faithfully translate that understanding into image generation or editing. Our benchmark evaluates both capabilities.

For state understanding, each dialogue includes four diagnostic questions covering Reference Resolution, Context Retrieval, Constraint Tracking, and Correction Updating. Model responses are assessed against reference answers for semantic correctness (Appendix B.1). For visual realization, generated or edited images are evaluated using predefined atomic visual checks against the ground-truth target. Each check receives a binary score according to whether the required state property is visibly satisfied, and State Consistency is calculated as the proportion of satisfied checks. Visual Quality is assessed separately to distinguish perceptual image quality from adherence to the valid dialogue state (Appendix B.2).

Building on these evaluations, we further examine whether correctly understood requirements are faithfully realized in visual outputs by aligning state-understanding questions with visual checks targeting the same requirements. On samples where models correctly answer the corresponding state-understanding questions, we assess their generated or edited images using binary visual checks. We define the understanding–realization gap as one minus the normalized realization score. Figure 6(a) shows that the overall understanding–realization gap decreases from 0.58 for the original BAGEL to 0.37 after YANAMI fine-tuning. These results indicate that correctly answering state-understanding questions does not guarantee faithful visual realization, and that YANAMI fine-tuning narrows, but does not eliminate, this gap.

Appendix E, Figure 8 provides a concrete example: when the user corrects a headphone design from wireless to wired, subsequent visual outputs must incorporate this correction while preserving an earlier requirement that all products remain unbranded. The corresponding visual checks assess whether the output reflects both the corrected connection type and the persistent no-branding constraint, rather than merely satisfying the latest composition instruction.

By comparison, the three aforementioned works focus their evaluations on history-conditioned question answering, cross-turn reference resolution, and basic contextual reasoning. These evaluations are insufficient for assessing complex interactive dialogues involving persistent constraints and user corrections. They also do not cover image generation and editing or measure the gap between understanding and visual realization in unified multimodal models (UMMs).

Q2: Performance of BAGEL fine-tuned on YANAMI on MMDU

Thank you for your valuable suggestion.

Since MMCR-Bench has not been publicly released, we evaluate our model on MMDU only. The results are as follows, we have also added the relevant content to Appendix D.6 of the revised version.:

Metric (Mean ×10)BAGEL BaseOurs
Creativity25.72923.185
Richness24.52322.653
Visual Perception42.62045.271
Logical Coherence60.70265.775
Answer Accuracy43.35347.298
Image Relationship Understanding41.75146.836
Overall Score32.53538.201
Average Score39.78041.836

Fine-tuning BAGEL on YANAMI improves its MMDU Overall Score by 5.666 points and its average across the first six metrics by 2.056 points over the original BAGEL. In particular, we observe substantial improvements in four metrics relevant to multi-turn dialogue: Visual Perception, Logical Coherence, Answer Accuracy, and Image Relationship Understanding. These results indicate that the benefits generalize beyond YANAMI-Bench and improve the model’s multi-turn understanding capabilities.

During MMDU evaluation, the LLM judge directly assigns an Overall Score rather than deriving it arithmetically from the individual metrics. We therefore report both this LLM-assigned Overall Score and the Average Score across the first six metrics.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely thank you and all three reviewers for evaluating our work and providing feedback.

We are encouraged that the reviewers appreciated the following aspects of YANAMI:

  • Practical importance of the problem. In multi-turn multimodal interactions, models must track users’ evolving requirements and reflect the currently valid requirements in subsequent outputs (Wk7V, QGTE).
  • Research value of the dataset and benchmark. YANAMI provides large-scale training data and an evaluation benchmark with 400 human-verified dialogues and 8,467 atomic visual checks (Wk7V, QGTE, W9tR).
  • Diagnostic value of the evaluation. Evaluating dialogue-state understanding and visual realization separately helps identify where a model fails. The Self-History evaluation further shows how models perform when their own outputs become part of the dialogue history (Wk7V, W9tR).
  • Utility of the training data. Fine-tuning BAGEL on YANAMI improves performance on both YANAMI-Bench and external evaluations (Wk7V, QGTE, W9tR).

The discussion centered on two questions. Can earlier dialogue change the correct output for the current turn? Does dialogue-state modeling supervision provide additional gains under the same training budget? We added two controlled experiments. Holding the visual inputs and current instruction fixed, removing earlier text history reduces the overall score from 78.25 to 54.90. Under the same training budget, the control model trained on a dataset without dialogue-state modeling supervision scores 67.99, while the model trained on YANAMI scores 78.25.

Summary of Rebuttal and Discussion

Reviewer(s)Key issueHow we addressed itResolved?
Wk7VDifferences from Visual Dialog, MMDU, and MMCR, and generalization to external benchmarksWe expanded the related work discussion to explain the role of persistent constraints and user corrections in dialogue-state understanding and visual realization. We added an MMDU evaluation, where the Overall Score improves from 32.54 to 38.20.Yes
Wk7V, W9tRWhether dialogue-state supervision provides additional gainsUnder the same training budget, the control model trained on a dataset without dialogue-state modeling supervision scores 67.99, while the model trained on YANAMI scores 78.25.Yes
Wk7V, W9tRPerformance and robustness under Self-HistoryWe added an evaluation of BAGEL + YANAMI under Self-History. The results show that the model is robust under Self-History and has the smallest performance drop relative to GT-History.Yes
QGTE, W9tRWhether earlier text history provides independent informationHolding the visual inputs and current instruction fixed, removing earlier text history reduces the overall score from 78.25 to 54.90. The example of swapping the positions of the bottle and wine glass further illustrates how an earlier correction changes the correct output for the current turn.Yes
QGTE, W9tRFair comparison across model categoriesWe report Dialogue State Understanding (DSU) separately and compare models using common State Consistency (SC) and Visual Quality (VQ) metrics. BAGEL + YANAMI ranks second overall among open-source models, behind only FLUX.2 [dev], which has 32B parameters, approximately 4.57 times BAGEL’s 7B active parameter count. BAGEL + YANAMI ranks first in State Consistency among open-source models.Yes
W9tRDifferences from WEAVE and dialogue constructionWe clarified how YANAMI supervises cross-turn constraints and user corrections, and how its dialogues were independently constructed using shared visual materials.Yes
W9tRData provenance, deduplication, and quality auditingWe reported statistics by data source and scenario, the proportion of images regenerated or edited (2.46%), and the SHA-256 checks for exact duplicates. We also conducted a human audit of 400 dialogues sampled across scenarios, with an average score of 4.72 out of 5.Yes

By the end of the discussion, Wk7V had raised their rating from 2 to 3. All three reviewers gave a rating of 3.

The additional experiments support the paper’s central contribution. YANAMI provides large-scale supervision for training and evaluating cross-turn dialogue-state management. Under the same training budget, fine-tuning BAGEL on YANAMI improves dialogue-state understanding and visual state consistency, and the model retains its performance advantage under Self-History.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.