- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Knowledge Transfer Probing (KTP), a controlled protocol that uses fictional entity-attribute associations across seven visual knowledge categories to test whether knowledge supplied only through text-to-image (T2I) training becomes accessible in text-only question answering (T2T). Its main contribution is showing that visual generation can transfer knowledge to language, but the transfer is direct, limited, and uneven across models and categories. Using BAGEL, SenseNova-U1, and GLM-Omni, the paper further finds that auxiliary alignment—especially interleaved image-text alignment—and Visual CoT can improve textual access to visually acquired knowledge, though much of what models generate correctly remains inaccessible through language.
Critical strengths
The paper’s main strength is that it introduces KTP, a carefully controlled protocol that uses fictional entity-attribute associations across seven visual knowledge categories to isolate whether knowledge supplied only through text-to-image training becomes accessible in text-only question answering. This is consequential because it directly tests a central but underexplored question for unified multimodal models: whether generation and understanding genuinely share knowledge, rather than merely coexisting in one model. The empirical study is also broad, covering three architecturally distinct UMMs—BAGEL, SenseNova-U1, and GLM-Omni—and comparing multiple supervision strategies, including direct transfer, textual alignment, interleaved image-text alignment, and Visual CoT. These comparisons show that visual generation can transfer knowledge to language, but that transfer is limited, uneven, and strongly dependent on supervision design. This combination of a clean controlled protocol and cross-model evidence makes the paper a useful step toward understanding and improving knowledge sharing in UMMs.
Critical weaknesses
However, I have several concerns:
- Incomplete ablation
Interleaved alignment differs from textual alignment in at least four ways: the interleaved sequence structure, the explicit image-generation loss, higher task diversity, and higher supervision density with longer sequences. The paper’s ablation only masks the image loss while keeping the interleaved structure and text supervision (Figure 5, GLM-Omni). This shows that the image loss matters, but it does not isolate the contribution of the interleaved structure itself, nor does it control for supervision density, sequence length, or task diversity. A key missing condition is non-interleaved training with an equivalent image-generation loss. Without it, we cannot tell whether the gain comes from interleaving, from simply adding more image supervision, or from adding more gradient signal in general. The data sampling ratios also differ across models (Table 6) and are not held fixed across conditions, so data mixture is another uncontrolled variable. In short, the paper shows that image loss inside interleaved sequences helps, but it does not yet prove that interleaved organization is the reason.
- Architecture motivation
The paper tests BAGEL, SenseNova-U1, and GLM-Omni, but the intervention is not derived from the design of any specific UMM. It is therefore unclear why GLM-Omni benefits so much more from interleaved alignment than the other two models. For example, GLM-Omni improves from 12% to 64% on shape and from 0% to 40% on spatial, while BAGEL and SenseNova-U1 show much smaller gains. This asymmetry is likely tied to architecture. The paper does not provide representation-level or attention-level analyses to test these hypotheses. Probing whether text and visual hidden states become more aligned after interleaved training, whether text tokens attend more to image tokens, or whether target attributes are linearly decodable from text representations would substantially strengthen the architectural insight. Without such analysis, the cross-model differences remain descriptive rather than explanatory. Moreover, the ablation in Section 4.3 is conducted only on GLM-Omni. Given its distinct architecture, it is unclear whether the same mechanism generalizes to other UMM designs. Ablations on BAGEL and SenseNova-U1 would clarify whether the observed benefit of image loss within interleaved sequences is architecture-specific or more broadly applicable.
- Retrieval vs. Knowledge
The KTP protocol measures text-only QA after T2I training. Low textual accessibility is interpreted as limited transfer, but this aggregate metric mixes three different failure modes: the target knowledge was never acquired; the knowledge is visually accessible but cannot be retrieved through language; or the knowledge is retrievable, but the model lacks an explicit protocol to bind the visual concept to the textual question. A two-stage diagnostic can separate these. First, provide the test image and ask the model to describe it (I2T). Second, without providing the image, prepend that description to the original question and ask the model to answer (T2T). If the second stage succeeds while the original KTP fails, it would show that the knowledge is present and textually usable, but the model cannot spontaneously retrieve it when only the text question is given. This would recast the paper’s central finding from “knowledge transfer is weak” to “the cross-modal retrieval or binding protocol is missing,” which has direct implications for UMM training design. If the second stage also fails, then the problem is more fundamental: either the knowledge was not acquired, or the visual extraction itself is unreliable. The current Visual CoT experiment partially addresses this, but it relies on the model generating its own image and does not isolate retrieval from generation quality. A diagnostic that uses the real test image would be cleaner.
- Insufficient mechanistic account of the transfer gap
I think the proposed solutions are relatively direct: adding text-only QA supervision or interleaving image generation with visual observation and textual reasoning improves textual accessibility. However, these interventions do not explain why knowledge learned through generation is not already accessible through language, nor where the gap between generation and question answering arises. The paper does not identify whether the bottleneck lies in shared representations, modality-specific pathways, cross-modal binding, knowledge retrieval, or the decoding objective. As a result, the proposed methods are effective empirical training recipes, but they do not yet constitute a principled account of the transfer failure or explain why the interventions work. More broadly, it remains unclear whether the observed gap is specific to the seven synthetic tasks or reflects a general limitation of unified multimodal models across different generation tasks. The paper should either provide analyses that connect each intervention to a specific bottleneck and discuss its expected generality, or substantially narrow its claims about improving knowledge sharing in UMMs.
Questions for authors
There is a minor problem: VLM evaluation reliability.
Visual accessibility relies on a VLM judge, which may introduce evaluator bias. I think it may be good to add a human agreement study on a subset of generated images to help validate the reported visual accessibility scores. But most importantly, the authors should address the weaknesses above.
2 comments
Response to Reviewer MGH5 (2/2)
W4: Insufficient mechanistic account of the transfer gap.
We sincerely thank the reviewer for the insightful comment. Before conducting mechanistic studies, a more fundamental question must first be answered: does a gap exist between visual generation and language access, how large is it, and which training designs can narrow it? Our paper is built around this question, and our contributions consist of two progressive levels: (1) A fine-grained, comprehensive, and knowledge-isolated measurement protocol, KTP. (2) General strategies for improving knowledge acquisition from visual generation.
Although our paper does not provide a complete mechanistic explanation, these experiments already lead to several conclusions:
- Knowledge acquisition is not the main bottleneck: Visual accessibility is substantially higher than textual accessibility, indicating that the models have learned the target knowledge from T2I training, and the problem lies mainly in accessing it through language (see W3).
- Explicit visual supervision matters: On GLM-Omni, masking the image loss in interleaved sequences reduces the average QA accuracy from 26.86% to 13.71%.
- The organization of supervision matters: Under matched visual supervision and training budgets, the interleaved structure still brings additional gains (+1.90 percentage points on average; see W1).
Taking these clues as a starting point, an in-depth mechanistic study is our next step. The directions suggested by the reviewer are all highly valuable, and the controlled setting and public data provided by KTP are well suited to support such analyses.
Q1: VLM evaluation reliability.
We sincerely thank the reviewer for this suggestion. We randomly sampled 100 records from all T2I evaluation records, covering all three models (BAGEL 43 / GLM-Omni 30 / SenseNova-U1 27) and all seven categories (color 18, spatial 20, size 16, shape 13, count 12, number 12, text 9). Among them, 46 were judged as correct by the VLM and 54 as incorrect, giving a roughly balanced set of positive and negative samples. Two annotators independently judged the evaluation results. After cross-checking, the agreement between the VLM judgments and the human judgments is 99%, validating the reliability of the VLM-based evaluation.
Response to Reviewer MGH5 (1/2)
Thank you for the review and constructive suggestions. We address each point below.
W1: Ablation of interleaved structure.
We sincerely thank the reviewer for this suggestion. We have added an ablation study on the interleaved structure for all three models:
Textual accessibility (%) on the test entities:
W2: Architecture motivation.
We thank the reviewer for this comment. We selected these three models so that they complement each other along the main architectural design dimensions of UMMs, allowing us to test whether the knowledge transfer phenomenon is consistent across architectures. Specifically, we characterize UMM architectures along three dimensions:
Among them, BAGEL and SenseNova-U1 represent two open-source, state-of-the-art paradigms. For the paradigm of "discrete tokens + unified NTP objective + fully shared parameters", we could not find an open-source model comparable to the other two in scale and performance. We therefore trained GLM-Omni in-house from the pretrain base checkpoint of GLM-4.1V-Thinking to cover this paradigm.
W3: Analysis of failure modes.
We thank the reviewer for the in-depth discussion of failure modes. Our experimental results in fact indicate that low textual accessibility arises because "the knowledge is retrievable, but the model lacks an explicit protocol to bind the visual concept to the textual question."