acceptodds
Withdrawn from ICLR 2027

Gestalt: Large Multimodal Interplay Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In this paper, we propose Gestalt, a new paradigm of large multimodal model built around multimodal interplay. Despite rapid advances, large multimodal models are reaching a bottleneck: existing approaches focus primarily on accommodating additional modalities while overlooking the distinct characteristics of each modality and the relations among them. Motivated by the multistage property of human multisensory perception, we propose a multimodal interplay pyramid that organizes multimodal modeling as a progression from modality-specific processing, through cross-modal alignment, to deeper multimodal integration. Guided by this pyramid, Gestalt adopts a unified discrete diffusion framework and an interplay-partitioned architecture, with learnable interplay tokens mediating cross-modal exchange and integration. The pyramid also structures its data organization and progressive training strategy. Strong performance across image generation, multimodal understanding, and text-only evaluation shows that Gestalt significantly improves cross-modal integration while preserving modality-specific information, effectively harnessing the strengths of diffusion-based multimodal models and offering a promising path toward unified multimodal intelligence. Code, data, and model will be publicly released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language model, multimodal interplay, unified model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7Qp4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Gestalt, a unified multimodal model built on the masked-diffusion LM LLaDA-8B-Instruct that does image understanding, text-to-image generation and text-only tasks in one network. The authors argue that existing unified models just put modalities together without modeling which information is shared, modality-specific or synergistic, and propose an "interplay pyramid" to fix this. In practice: the first four layers process image and text tokens with separate FFNs and no direct attention between them, with 32 learnable interplay tokens as the only channel; the remaining layers use full attention. SFT data is sorted into four interaction types and fed in a three-stage curriculum. Gestalt is reported to be best on UniGenBench and TIIF-Bench and the strongest diffusion-based unified model on most understanding and text-only benchmarks.

Critical strengths

The experimental results are solid: best overall on UniGenBench and TIIF-Bench, and the strongest diffusion-based unified model on most understanding and text-only benchmarks (I checked the reported margins against the tables).

Critical weaknesses
  1. There is no ablation of the architecture: no run without Zone I, without interplay tokens, with a shared FFN in Zone I, or with a different number of layers / interplay tokens. All baselines are external models trained on different data, and most also use different backbones and tokenizers, so the gains cannot be attributed to the design rather than to LLaDA-8B and the 78M + 13.7M + 2.5M training examples.

  2. Table 5 does not include LLaDA-8B-Instruct, which Gestalt is initialized from. The LLaDA paper reports about 65 / 75 / 88 on MMLU / HellaSwag / ARC-C for that model, while Gestalt gets 49.5 / 53.4 / 43.7, so I am confused by the claim that language capability is retained. Some baseline numbers in Table 5 are also confusing: BAGEL gets 28.0 on MMLU and 23.6 on ARC-C, and LLaDA-o gets 25.3 on MMLU, which is chance level for four-way multiple choice.

  3. Related work is one paragraph. MBT (bottleneck tokens), VLMo / BEiT-3 (modality-specific FFNs), Mixture-of-Transformers / BAGEL's MoT and LMFusion are all close to Zone I and are not discussed. The redundancy / uniqueness / synergy taxonomy comes from PID (Williams & Beer 2010; Liang et al. 2023) and has already been used by MMoE; none of these is cited either.

  4. The reproducibility statement says hyperparameters are given; they are not. Learning rate, batch size, steps, compute, λv / λt, inference steps / CFG and the SFT sampling ratios are all missing.

Questions for authors
  1. Can you provide ablations on the same backbone and data (no Zone I, no interplay tokens, shared FFN in Zone I), even at the reduced scale of Table 8?
  2. Please add LLaDA-8B-Instruct to Table 5 under the same protocol, and explain the chance-level baseline numbers.
  3. What is new relative to MBT, VLMo / BEiT-3 and BAGEL's MoT?
Reviewer_McaAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Gestalt is a unified discrete diffusion model for image understanding, text-to-image generation, and text-only tasks. It is initialized from LLaDA-8B-Instruct and uses IBQ visual tokens.

The main architectural change is in the first 4 of 32 layers ("Zone I"). There, image and text tokens use separate FFNs and cannot attend to each other directly; they exchange information only through 32 learnable interplay tokens. The other 28 layers are standard layers with full attention and a shared FFN.

Training has three stages:

  • joint masked pretraining on 70M image–caption pairs;
  • conditional continual pretraining on 8M pairs;
  • SFT on 13.7M instruction examples plus 2.5M T2I examples.

The SFT data is labeled as redundant, text-unique, visual-unique, or synergistic. A three-phase curriculum shifts the sampling ratio toward visual-unique and then synergistic data. The whole design is motivated by an "interplay pyramid": modality-specific processing → cross-modal alignment → synergy.

Validation covers:

  • best overall scores on UniGenBench and TIIF-Bench;
  • best among diffusion-based unified models on 5 of 8 understanding benchmarks and 5 of 6 text-only benchmarks;
  • representation analyses: embedding similarity to LLaDA and 2D visualizations of hidden states;
  • a curriculum-vs-uniform ablation on 10% of the SFT data.
Critical strengths
  1. Strong fine-grained T2I results. Gestalt has the best overall score on UniGenBench (73.30) and TIIF-Bench (74.57 short / 78.86 long). This puts it above both generation-only models (FLUX.1 Pro, SD 3.5 Large, OmniGen2) and unified models including BAGEL, and the margin grows with longer prompts. It shows that a discrete diffusion model can be competitive on compositional generation while also handling understanding.
  2. A strong diffusion-based unified model. Among the compared diffusion-based unified models, it ranks first or second on 7 of 8 understanding benchmarks and leads on 5 of 6 text-only benchmarks. If the code, data, and model are released as promised, it is a useful baseline.
Critical weaknesses
  1. The central design claim is not tested.

The paper's main claim (Sec. 5, L479–481) is that modality-specific processing and controlled exchange before full fusion works better than sharing all computation from the first layer. No experiment tests this.

Zone I changes only 4 of 32 layers. Almost everything else differs from the baselines:

  • data scale and data sources;
  • filtering with Qwen3-VL-32B;
  • GPT-4o-generated T2I data (ShareGPT-4o-Image, BLIP3o-60k).

MMaDA and Lumina-DiMOO are also LLaDA-based unified discrete diffusion models, yet they too are trained on different data. So the gains in Tables 1–6 cannot be attributed to Zone I or to the interplay tokens; they may come entirely from data and training.

I ask for two controlled runs with the same data and recipe as Gestalt. A reduced scale is fine, e.g., a subset of the pretraining data plus the 10% SFT protocol from Appendix B.1.

  • (a) No Zone I. All 32 layers use full attention and a shared FFN, with no interplay tokens. This tests whether the preserve-then-integrate design helps at all.
  • (b) Zone I without the bottleneck. Keep the modality-specific FFNs in the first 4 layers, but remove the interplay tokens and allow direct image–text attention. Comparing (a), (b), and the full model separates two effects: that of the modality-specific FFNs, and that of routing cross-modal exchange through interplay tokens.

Both runs should be evaluated at least on the T2I benchmarks, the vision-centric benchmarks, and the synergy benchmarks. Without these runs, the architecture that the paper presents as its main contribution remains unvalidated.

  1. The evidence for the interplay mechanism and the curriculum is weak.
  • Figure 6. It visualizes interplay tokens at layer 16. That layer is already in Zone II, where the interplay tokens are ordinary tokens, and there is no comparison model.
  • Table 4. It shows high embedding similarity to LLaDA, but MMLU drops to 49.50, against 65.5 reported for LLaDA-8B-Instruct. High similarity does not mean capability is preserved.
  • Synergy benchmarks. MM-IMDb and SRBench have no unimodal controls.
  • Curriculum ablation (Table 8). It is a single run on 10% of the data, and the differences are small and mixed. Phase 2 is meant to strengthen visual-unique ability, yet it is worse than uniform mixing on MMVP (−6.67).
  1. The central design claim is not tested.

The paper's main claim (Sec. 5, L479–481) is that modality-specific processing and controlled exchange before full fusion works better than sharing all computation from the first layer. No experiment tests this.

Zone I changes only 4 of 32 layers. Almost everything else differs from the baselines:

  • data scale and data sources;
  • filtering with Qwen3-VL-32B;
  • GPT-4o-generated T2I data (ShareGPT-4o-Image, BLIP3o-60k).

MMaDA and Lumina-DiMOO are also LLaDA-based unified discrete diffusion models, yet they too are trained on different data. So the gains in Tables 1–6 cannot be attributed to Zone I or to the interplay tokens; they may come entirely from data and training.

I ask for two controlled runs with the same data and recipe as Gestalt. A reduced scale is fine, e.g., a subset of the pretraining data plus the 10% SFT protocol from Appendix B.1.

  • (a) No Zone I. All 32 layers use full attention and a shared FFN, with no interplay tokens. This tests whether the preserve-then-integrate design helps at all.
  • (b) Zone I without the bottleneck. Keep the modality-specific FFNs in the first 4 layers, but remove the interplay tokens and allow direct image–text attention. Comparing (a), (b), and the full model separates two effects: that of the modality-specific FFNs, and that of routing cross-modal exchange through interplay tokens.

Both runs should be evaluated at least on the T2I benchmarks, the vision-centric benchmarks, and the synergy benchmarks. Without these runs, the architecture that the paper presents as its main contribution remains unvalidated.

  1. The evidence for the interplay mechanism and the curriculum is weak.
  • Figure 6. It visualizes interplay tokens at layer 16. That layer is already in Zone II, where the interplay tokens are ordinary tokens, and there is no comparison model.
  • Table 4. It shows high embedding similarity to LLaDA, but MMLU drops to 49.50, against 65.5 reported for LLaDA-8B-Instruct. High similarity does not mean capability is preserved.
  • Synergy benchmarks. MM-IMDb and SRBench have no unimodal controls.
  • Curriculum ablation (Table 8). It is a single run on 10% of the data, and the differences are small and mixed. Phase 2 is meant to strengthen visual-unique ability, yet it is worse than uniform mixing on MMVP (−6.67).
  • "Text-unique" category. 3.957M of the 4.2M "text-unique" examples are text-only data, so the curriculum mostly controls when text-only data is added.

These issues are secondary to Weakness 1. Still, apart from the benchmark numbers, the paper offers little evidence for the "interplay" story.

Questions for authors

On Weakness 1 (clarification):

  1. Do you have any results, even small-scale or preliminary, comparing Gestalt under matched data and training with (a) a fully shared variant and (b) Zone I without interplay tokens? How were N1 = 4 and 32 interplay tokens chosen? If the gains largely disappear under matched data, the paper's main claim does not hold, and this would change my assessment.
  2. T2I is the strongest result. However, according to Appendix A.3 the T2I data is not part of the interplay curriculum, so only the architecture could explain an interplay-related gain there. What evidence do you have that Zone I contributes to the T2I gains, rather than the T2I training data, which includes GPT-4o-generated images?

On Weakness 2 (clarification):

  1. Figure 6 uses layer 16, which is already in Zone II. What do the interplay tokens look like in Zone I, where they are the only cross-modal path? Would mean-pooled text tokens, or a model without interplay tokens, show similar clustering? If so, the figure says little about the interplay tokens themselves.
  2. What does LLaDA-8B-Instruct score under your text-only protocol? Gestalt's MMLU of 49.50 is well below the 65.5 reported in the LLaDA paper. How does this fit with the claim that language representations and capability are preserved?
  3. 3.957M of the 4.2M text-unique examples are text-only. Is Phase 1 then effectively "more text-only data"? Phase 2 is said to strengthen visual perception, yet it underperforms uniform mixing on MMVP (−6.67). How is this consistent with that claim?

Minor:

  • Missing implementation details:
    • which FFN the interplay tokens use in Zone I, and how its outputs are combined;
    • where the interplay tokens sit in the sequence;
    • the loss weights λ_v and λ_t;
    • training hyperparameters (learning rate, batch size, steps), image resolution, and decoding settings.
  • Citations: LLaDA-o and LaViDa-O are cited to t
Reviewer_UVuaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gestalt, a unified multimodal model designed around the concept of multimodal interplay, which distinguishes modality-specific information, cross-modal redundancy, and multimodal synergy. Inspired by this conceptual framework, the authors introduce an interplay-partitioned Transformer architecture within a unified discrete diffusion framework. Specifically, the model consists of two sequential zones: (1) a bottleneck interplay zone that employs modality-specific FFNs and restricts direct cross-modal attention, using learnable interplay tokens to mediate information exchange; and (2) a fully connected zone that enables unrestricted multimodal attention and shared processing. The framework is complemented by an interplay-oriented data organization strategy and a progressive training curriculum. Experiments demonstrate competitive performance across text-to-image generation, multimodal understanding, and text-only benchmarks. In particular, Gestalt achieves strong results on UniGenBench and TIIF-Bench, as well as vision-centric benchmarks such as CV-Bench.

Critical strengths

1. Strong empirical performance across multimodal understanding and generation.

The paper demonstrates competitive performance across a broad set of benchmarks, covering fine-grained text-to-image generation, multimodal understanding, and language-only capabilities. Particularly noteworthy are the results on TIIF-Bench (74.57/78.86 for short/long prompts) and UniGenBench (73.30), where Gestalt outperforms the reported unified multimodal baselines and several specialized generation models. The strong performance on CV-Bench, including improvements over BAGEL on both 2D and 3D evaluations, further suggests that the proposed model achieves a useful balance between visual understanding and generation. These results constitute meaningful empirical evidence for the effectiveness of the overall framework.

2. A coherent approach to integrating modality-specific processing and cross-modal information exchange.

The proposed interplay pyramid provides an intuitive motivation for organizing multimodal computation. In particular, separating modality-specific processing from unrestricted multimodal integration, while introducing dedicated interplay tokens for information exchange, is a reasonable design choice. The paper also extends this principle to data organization and training, resulting in a coherent framework rather than an isolated architectural modification. This perspective could be valuable for future research on unified multimodal models.

Critical weaknesses

1. Insufficient ablation studies to establish the effectiveness and necessity of the proposed architectural components.

The interplay-partitioned architecture is the central technical contribution of this paper. However, the experimental section lacks controlled architectural ablations to demonstrate that its individual components are responsible for the reported improvements.

In particular, the paper does not compare the proposed architecture against a standard fully connected multimodal Transformer trained under the same conditions. Consequently, it remains unclear whether restricting cross-modal interactions in early layers is more effective than simply allowing unrestricted attention throughout the network.

Similarly, the necessity of the learnable interplay tokens is not established. The paper does not examine whether comparable performance could be achieved without these tokens or with alternative cross-modal interaction mechanisms. The effects of the number of interplay tokens, the depth of Zone I, and the modality-specific FFNs are also unexplored.

These omissions are particularly consequential because the proposed design introduces additional architectural complexity, including modality-specific parameters and specialized attention patterns. Without controlled comparisons under matched training conditions, it is difficult to determine whether the improvements originate from the proposed interplay mechanism, the additional parameters, or the extensive multimodal training recipe.

2. The empirical evidence does not sufficiently substantiate the proposed multimodal interplay theory or its connection to the model's improvements.

The paper's central motivation is that modality-specific information, cross-modal redundancy, and multimodal synergy should be progressively modeled through distinct architectural and training stages. However, the experiments provide limited evidence that the proposed architecture actually realizes this hierarchy.

First, the representation visualizations in Figures 5 and 6 are primarily qualitative. The apparent overlap between image and text representations does not necessarily establish effective cross-modal information integration, and the clustering of interplay-token representations by information type does not demonstrate that these tokens causally facilitate multimodal synergy. More direct measurements or controlled interventions would be necessary to support these interpretations.

Second, the interplay-oriented curriculum has only been evaluated on a 10% subset of the supervised fine-tuning data (Table 8). The improvements are inconsistent across benchmarks: at the final stage, the curriculum improves MMStar-P and CVBench-2D but reduces MME-P relative to uniform mixing. Moreover, the results do not isolate the benefits of the proposed information-based curriculum from those of alternative data-ordering or sampling strategies.

Finally, there is no direct comparison between an architecture without interplay partitioning and the proposed architecture on synergy-oriented benchmarks under an otherwise identical training recipe. Consequently, the reported improvements on MM-IMDb and SRBench demonstrate the capability of Gestalt as a whole but do not establish that the proposed interplay mechanism is responsible for stronger multimodal synergy.

This weakness affects the paper's central scientific claim: while the framework is motivated by a compelling conceptual interpretation of multimodal learning, the current evidence does not sufficiently distinguish its underlying mechanism from conventional architectural modifications and training improvements.

Questions for authors

Q1. How much does each architectural component contribute to the final performance?

Q2. How sensitive is the model to the configuration of interplay tokens and architectural partitioning?

Q3. Can the authors provide more direct evidence that the interplay mechanism facilitates multimodal synergy?

Q4. Does the interplay-oriented curriculum consistently outperform simpler training strategies at full scale?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.