- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to assess understanding-to-generation transfer in unified multimodal models. Rather than treating changes in understanding and generation scores as direct evidence of capability transfer, the authors decompose the claim into three components: source validity, i.e., whether the observed understanding gain reflects the intended visual attribute; target influence, i.e., whether the understanding-trained update causally changes generation; and attribute correspondence, i.e., whether the generation change follows the specific attribute value associated with the update.
To audit these components, the paper introduces removable square-core updates trained only on understanding tasks, fixed-checkpoint image and instruction controls, paired full-versus-frozen generation comparisons with matched randomness, and donor-replacement interventions that swap norm-matched conditioning changes between prompts. The study evaluates counting and spatial-relation tasks across six unified multimodal architectures, with an additional standard LoRA reference. The experiments show that source-side answer gains can be highly instruction-dependent, understanding and generation changes exhibit heterogeneous and sometimes opposite-sign relationships across models, and donor replacement does not consistently induce donor-value following. The paper also analyzes limitations of norm-matched swaps and uses prompt-contrast calibration to qualify the interpretation of negative intervention results.
Critical strengths
- The paper addresses an important and timely question for unified multimodal models. A central motivation for unified architectures is that understanding and generation may share representations or capabilities, but it is nontrivial to determine whether improvements on one side genuinely transfer to the other. The paper usefully separates architectural sharing from stronger claims about capability transfer, and I found this framing conceptually valuable.
- The audit framework is carefully designed and empirically broad. The decomposition into source validity, target influence, and attribute correspondence provides a clear progression of increasingly stronger claims, and the corresponding controls—fixed-checkpoint instruction/image interventions, paired full-vs.-frozen generation, and donor replacement with dual readouts—are thoughtful attempts to rule out alternative explanations. The evaluation also covers six unified architectures, two attribute families (counting and spatial relations), and an additional LoRA reference, which makes the empirical study substantially more informative than a single-model case study.
Critical weaknesses
- The audit framework is stronger at ruling out overly strong transfer claims than at positively establishing transfer. Many results remain “unresolved” because of prompt sensitivity, metric ambiguity, norm-matching blind spots, or carrier limitations. The paper would be substantially stronger with a convincing positive-control setting showing that the full framework can successfully detect known attribute-level transfer.
- The scope of the evidence is somewhat narrow relative to the paper’s broad framing. The instruction-control experiment establishes that the measured source gain is prompt-dependent, but not necessarily that the original gain is non-visual, and the paper does not show how unusual this sensitivity is compared with standard VQA/adaptation settings. More broadly, the substantive analysis is limited to counting and spatial relations with only a few independent training runs, which weakens broad generalization to understanding-to-generation transfer as a whole.
Questions for authors
- Broader task coverage. The current empirical study mainly focuses on counting and spatial relations. Could the authors provide evidence on at least one qualitatively different type of capability (e.g., semantic recognition, factual/visual knowledge, or reasoning-related attributes) to support that the proposed audit framework captures a broader phenomenon rather than failure modes specific to these two relatively low-level compositional tasks? This would materially affect how broadly I interpret the paper’s conclusions.
- What can the framework establish positively? Could the authors clarify what empirical outcome would constitute a positive conclusion of successful understanding-to-generation transfer under their framework? At present, many analyses are effective at showing that a stronger interpretation is not justified, but it is less clear what the framework can positively establish beyond target influence. A concrete decision criterion or worked positive example would help clarify the scientific claim of the framework.
- Positive control. Could the authors provide a convincing positive-control setting in which attribute-level transfer is known to exist by construction, and then demonstrate that the proposed source-validity / target-influence / attribute-correspondence pipeline successfully recovers it? Given the acknowledged blind spots of norm matching and carrier choice, such a control would help distinguish genuine absence of evidence from limited sensitivity of the audit.
If the authors can satisfactorily address these concerns in the rebuttal, I would consider increasing my rating.