- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Gestalt, a unified multimodal model built on the masked-diffusion LM LLaDA-8B-Instruct that does image understanding, text-to-image generation and text-only tasks in one network. The authors argue that existing unified models just put modalities together without modeling which information is shared, modality-specific or synergistic, and propose an "interplay pyramid" to fix this. In practice: the first four layers process image and text tokens with separate FFNs and no direct attention between them, with 32 learnable interplay tokens as the only channel; the remaining layers use full attention. SFT data is sorted into four interaction types and fed in a three-stage curriculum. Gestalt is reported to be best on UniGenBench and TIIF-Bench and the strongest diffusion-based unified model on most understanding and text-only benchmarks.
Critical strengths
The experimental results are solid: best overall on UniGenBench and TIIF-Bench, and the strongest diffusion-based unified model on most understanding and text-only benchmarks (I checked the reported margins against the tables).
Critical weaknesses
-
There is no ablation of the architecture: no run without Zone I, without interplay tokens, with a shared FFN in Zone I, or with a different number of layers / interplay tokens. All baselines are external models trained on different data, and most also use different backbones and tokenizers, so the gains cannot be attributed to the design rather than to LLaDA-8B and the 78M + 13.7M + 2.5M training examples.
-
Table 5 does not include LLaDA-8B-Instruct, which Gestalt is initialized from. The LLaDA paper reports about 65 / 75 / 88 on MMLU / HellaSwag / ARC-C for that model, while Gestalt gets 49.5 / 53.4 / 43.7, so I am confused by the claim that language capability is retained. Some baseline numbers in Table 5 are also confusing: BAGEL gets 28.0 on MMLU and 23.6 on ARC-C, and LLaDA-o gets 25.3 on MMLU, which is chance level for four-way multiple choice.
-
Related work is one paragraph. MBT (bottleneck tokens), VLMo / BEiT-3 (modality-specific FFNs), Mixture-of-Transformers / BAGEL's MoT and LMFusion are all close to Zone I and are not discussed. The redundancy / uniqueness / synergy taxonomy comes from PID (Williams & Beer 2010; Liang et al. 2023) and has already been used by MMoE; none of these is cited either.
-
The reproducibility statement says hyperparameters are given; they are not. Learning rate, batch size, steps, compute, λv / λt, inference steps / CFG and the SFT sampling ratios are all missing.
Questions for authors
- Can you provide ablations on the same backbone and data (no Zone I, no interplay tokens, shared FFN in Zone I), even at the reduced scale of Table 8?
- Please add LLaDA-8B-Instruct to Table 5 under the same protocol, and explain the chance-level baseline numbers.
- What is new relative to MBT, VLMo / BEiT-3 and BAGEL's MoT?