acceptodds
Under review as a conference paper at ICLR 2027

Can Visual Generation Expand What Unified Multimodal Models Know?

11 views

Abstract

Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what these models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation targets and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image-Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible. We will release the dataset and evaluation protocol.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
unified multimodal models, alignment, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MGH5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Knowledge Transfer Probing (KTP), a controlled protocol that uses fictional entity-attribute associations across seven visual knowledge categories to test whether knowledge supplied only through text-to-image (T2I) training becomes accessible in text-only question answering (T2T). Its main contribution is showing that visual generation can transfer knowledge to language, but the transfer is direct, limited, and uneven across models and categories. Using BAGEL, SenseNova-U1, and GLM-Omni, the paper further finds that auxiliary alignment—especially interleaved image-text alignment—and Visual CoT can improve textual access to visually acquired knowledge, though much of what models generate correctly remains inaccessible through language.

Critical strengths

The paper’s main strength is that it introduces KTP, a carefully controlled protocol that uses fictional entity-attribute associations across seven visual knowledge categories to isolate whether knowledge supplied only through text-to-image training becomes accessible in text-only question answering. This is consequential because it directly tests a central but underexplored question for unified multimodal models: whether generation and understanding genuinely share knowledge, rather than merely coexisting in one model. The empirical study is also broad, covering three architecturally distinct UMMs—BAGEL, SenseNova-U1, and GLM-Omni—and comparing multiple supervision strategies, including direct transfer, textual alignment, interleaved image-text alignment, and Visual CoT. These comparisons show that visual generation can transfer knowledge to language, but that transfer is limited, uneven, and strongly dependent on supervision design. This combination of a clean controlled protocol and cross-model evidence makes the paper a useful step toward understanding and improving knowledge sharing in UMMs.

Critical weaknesses

However, I have several concerns:

  1. Incomplete ablation

Interleaved alignment differs from textual alignment in at least four ways: the interleaved sequence structure, the explicit image-generation loss, higher task diversity, and higher supervision density with longer sequences. The paper’s ablation only masks the image loss while keeping the interleaved structure and text supervision (Figure 5, GLM-Omni). This shows that the image loss matters, but it does not isolate the contribution of the interleaved structure itself, nor does it control for supervision density, sequence length, or task diversity. A key missing condition is non-interleaved training with an equivalent image-generation loss. Without it, we cannot tell whether the gain comes from interleaving, from simply adding more image supervision, or from adding more gradient signal in general. The data sampling ratios also differ across models (Table 6) and are not held fixed across conditions, so data mixture is another uncontrolled variable. In short, the paper shows that image loss inside interleaved sequences helps, but it does not yet prove that interleaved organization is the reason.

  1. Architecture motivation

The paper tests BAGEL, SenseNova-U1, and GLM-Omni, but the intervention is not derived from the design of any specific UMM. It is therefore unclear why GLM-Omni benefits so much more from interleaved alignment than the other two models. For example, GLM-Omni improves from 12% to 64% on shape and from 0% to 40% on spatial, while BAGEL and SenseNova-U1 show much smaller gains. This asymmetry is likely tied to architecture. The paper does not provide representation-level or attention-level analyses to test these hypotheses. Probing whether text and visual hidden states become more aligned after interleaved training, whether text tokens attend more to image tokens, or whether target attributes are linearly decodable from text representations would substantially strengthen the architectural insight. Without such analysis, the cross-model differences remain descriptive rather than explanatory. Moreover, the ablation in Section 4.3 is conducted only on GLM-Omni. Given its distinct architecture, it is unclear whether the same mechanism generalizes to other UMM designs. Ablations on BAGEL and SenseNova-U1 would clarify whether the observed benefit of image loss within interleaved sequences is architecture-specific or more broadly applicable.

  1. Retrieval vs. Knowledge

The KTP protocol measures text-only QA after T2I training. Low textual accessibility is interpreted as limited transfer, but this aggregate metric mixes three different failure modes: the target knowledge was never acquired; the knowledge is visually accessible but cannot be retrieved through language; or the knowledge is retrievable, but the model lacks an explicit protocol to bind the visual concept to the textual question. A two-stage diagnostic can separate these. First, provide the test image and ask the model to describe it (I2T). Second, without providing the image, prepend that description to the original question and ask the model to answer (T2T). If the second stage succeeds while the original KTP fails, it would show that the knowledge is present and textually usable, but the model cannot spontaneously retrieve it when only the text question is given. This would recast the paper’s central finding from “knowledge transfer is weak” to “the cross-modal retrieval or binding protocol is missing,” which has direct implications for UMM training design. If the second stage also fails, then the problem is more fundamental: either the knowledge was not acquired, or the visual extraction itself is unreliable. The current Visual CoT experiment partially addresses this, but it relies on the model generating its own image and does not isolate retrieval from generation quality. A diagnostic that uses the real test image would be cleaner.

  1. Insufficient mechanistic account of the transfer gap

I think the proposed solutions are relatively direct: adding text-only QA supervision or interleaving image generation with visual observation and textual reasoning improves textual accessibility. However, these interventions do not explain why knowledge learned through generation is not already accessible through language, nor where the gap between generation and question answering arises. The paper does not identify whether the bottleneck lies in shared representations, modality-specific pathways, cross-modal binding, knowledge retrieval, or the decoding objective. As a result, the proposed methods are effective empirical training recipes, but they do not yet constitute a principled account of the transfer failure or explain why the interventions work. More broadly, it remains unclear whether the observed gap is specific to the seven synthetic tasks or reflects a general limitation of unified multimodal models across different generation tasks. The paper should either provide analyses that connect each intervention to a specific bottleneck and discuss its expected generality, or substantially narrow its claims about improving knowledge sharing in UMMs.

Questions for authors

There is a minor problem: VLM evaluation reliability.

Visual accessibility relies on a VLM judge, which may introduce evaluator bias. I think it may be good to add a human agreement study on a subset of generated images to help validate the reported visual accessibility scores. But most importantly, the authors should address the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MGH5 (1/2)

Thank you for the review and constructive suggestions. We address each point below.

W1: Ablation of interleaved structure.

We sincerely thank the reviewer for this suggestion. We have added an ablation study on the interleaved structure for all three models:

  • Training data: Each interleaved example is split into one standalone T2I example and one standalone T2T example.
  • Training setup: The training budget and the number of epochs are kept identical.
  • Results: Under matched visual supervision and training budgets, the interleaved structure brings additional gains (+1.90 percentage points on average). Given the generally low accuracy levels, this indicates that the interleaved structure indeed helps the model learn the connections between the two modalities.

Textual accessibility (%) on the test entities:

Training conditionBAGELGLM-OmniSenseNova-U1Avg
Textual alignment15.4312.5728.0018.67
Ablation (interleaved → T2I + T2T)16.0024.0026.8622.29
Interleaved alignment17.1426.8628.5724.19

W2: Architecture motivation.

We thank the reviewer for this comment. We selected these three models so that they complement each other along the main architectural design dimensions of UMMs, allowing us to test whether the knowledge transfer phenomenon is consistent across architectures. Specifically, we characterize UMM architectures along three dimensions:

  1. Image modeling form: continuous latents, raw pixels, or discrete tokens;
  2. Visual representation acquisition: relying on separate pretrained visual encoders (e.g., ViT, VAE) or using no separate encoder;
  3. Backbone parameter organization: a Mixture-of-Transformer-Experts (MoT) structure with separate understanding and generation branches, or a single Transformer with fully shared parameters.
ModelImage modeling formVisual encoderBackboneGeneration objective
BAGELContinuous latentsSeparate ViT (understanding) + VAE (generation)MoTDiffusion / Flow Matching
SenseNova-U1Raw pixelsNo separate encoderMoT (NEO-unify)Diffusion / Flow Matching
GLM-OmniDiscrete tokensDiscrete image tokenizer (based on X-Omni)Single Transformer, fully shared parametersNTP

Among them, BAGEL and SenseNova-U1 represent two open-source, state-of-the-art paradigms. For the paradigm of "discrete tokens + unified NTP objective + fully shared parameters", we could not find an open-source model comparable to the other two in scale and performance. We therefore trained GLM-Omni in-house from the pretrain base checkpoint of GLM-4.1V-Thinking to cover this paradigm.

W3: Analysis of failure modes.

We thank the reviewer for the in-depth discussion of failure modes. Our experimental results in fact indicate that low textual accessibility arises because "the knowledge is retrievable, but the model lacks an explicit protocol to bind the visual concept to the textual question."

  1. Failure mode 1: the target knowledge was never acquired. In Section 4.1, we measure visual accessibility. The relatively high scores show that the models can learn knowledge from image generation and access it visually.
  2. Failure modes 2 & 3: is the knowledge retrievable? In Sections 4.2 and 4.3, adding alignment data improves textual accessibility on the test entities. This shows that alignment data helps the model establish a mechanism that binds visual concepts to textual questions. Although the test entities receive only T2I training, the model can still retrieve their knowledge with the help of this binding mechanism.
AuthorsOpenReview ↗

Response to Reviewer MGH5 (2/2)

W4: Insufficient mechanistic account of the transfer gap.

We sincerely thank the reviewer for the insightful comment. Before conducting mechanistic studies, a more fundamental question must first be answered: does a gap exist between visual generation and language access, how large is it, and which training designs can narrow it? Our paper is built around this question, and our contributions consist of two progressive levels: (1) A fine-grained, comprehensive, and knowledge-isolated measurement protocol, KTP. (2) General strategies for improving knowledge acquisition from visual generation.

Although our paper does not provide a complete mechanistic explanation, these experiments already lead to several conclusions:

  • Knowledge acquisition is not the main bottleneck: Visual accessibility is substantially higher than textual accessibility, indicating that the models have learned the target knowledge from T2I training, and the problem lies mainly in accessing it through language (see W3).
  • Explicit visual supervision matters: On GLM-Omni, masking the image loss in interleaved sequences reduces the average QA accuracy from 26.86% to 13.71%.
  • The organization of supervision matters: Under matched visual supervision and training budgets, the interleaved structure still brings additional gains (+1.90 percentage points on average; see W1).

Taking these clues as a starting point, an in-depth mechanistic study is our next step. The directions suggested by the reviewer are all highly valuable, and the controlled setting and public data provided by KTP are well suited to support such analyses.

Q1: VLM evaluation reliability.

We sincerely thank the reviewer for this suggestion. We randomly sampled 100 records from all T2I evaluation records, covering all three models (BAGEL 43 / GLM-Omni 30 / SenseNova-U1 27) and all seven categories (color 18, spatial 20, size 16, shape 13, count 12, number 12, text 9). Among them, 46 were judged as correct by the VLM and 54 as incorrect, giving a roughly balanced set of positive and negative samples. Two annotators independently judged the evaluation results. After cross-checking, the agreement between the VLM judgments and the human judgments is 99%, validating the reliability of the VLM-based evaluation.

Reviewer_7BUMOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether unified multimodal models can acquire new knowledge through image generation training and subsequently access that knowledge through text-only question answering.

The authors introduce KTP, a controlled evaluation protocol that constructs fictional entities across seven visual knowledge categories. Target knowledge is provided exclusively through image generation supervision, while knowledge transfer is evaluated through text-only questions. The paper further investigates textual alignment and interleaved image-text alignment to improve the accessibility of visually acquired knowledge.

Experiments on BAGEL, GLM-Omni, and SenseNova-U1 reveal a substantial gap between visual and textual knowledge accessibility. Additional alignment supervision partially bridges this gap, with interleaved alignment achieving higher average accuracy across all three models.

Critical strengths
  1. Well-designed experimental protocol. KTP uses fictional entities and information isolation to introduce target knowledge exclusively through image generation supervision. This controlled setting provides a useful approach to investigating whether visually acquired knowledge can become accessible through language.

  2. Insightful cross-model analysis. The paper systematically examines the gap between visual and textual knowledge accessibility across three different architectures. The alignment comparisons, image-loss ablation, and Visual CoT experiments provide valuable evidence for understanding how different supervision strategies influence cross-modal knowledge transfer.

Critical weaknesses
  1. Limited evidence for direct knowledge transfer. In Table 1, the average text-only QA accuracy of BAGEL and GLM-Omni falls below the random-guessing baseline, while SenseNova-U1 only slightly exceeds it. Although positive transfer is observed in certain categories, the results do not yet establish robust knowledge transfer across models.

  2. Limited novelty and insufficient isolation of the alignment effect. The discrepancy between visual knowledge and verbal recall has already been investigated in related studies [1, 2]. While KTP provides a useful controlled evaluation protocol, the improvements from interleaved alignment are uneven and primarily concentrated in GLM-Omni. Furthermore, the image-loss ablation demonstrates the importance of visual supervision but does not fully establish whether the gains arise specifically from interleaved training rather than additional image generation supervision.

[1] Modal Aphasia: Can Unified Multimodal Models Describe Images from Memory?

[2] Quantifying Cross-Modality Memorization in Vision-Language Models

Questions for authors
  1. Could the authors include a control group without target-specific image generation supervision, while maintaining the same text-only QA formatting supervision? Reporting results across multiple random seeds or entity splits would also help establish the reliability of direct knowledge transfer.

  2. Could the authors compare interleaved image-text alignment against additional standalone T2I training under matched visual supervision and training budgets? This would help clarify whether the improvements arise from the interleaved structure itself rather than simply from additional visual supervision.

If the authors make a convincing effort to address this concern, I would consider raising my rating.

2 comments

Thank you for the review and constructive suggestions. We address each point below.

W1 / Q1: Clarification on direct knowledge transfer.

We sincerely thank the reviewer for this careful observation, which is precisely the point we intend to make: the direct transfer ability of UMMs under pure T2I training is quite limited. In our experiments, only weak transfer is observed in text-only QA, while substantially more target knowledge is expressed in generated images. For example, SenseNova-U1 achieves 88% visual accessibility on the number category but 0% textual accessibility.

Therefore, the conclusion Table 1 intends to convey is that, under the current setting, models can reproduce part of the target facts in generation, yet these facts remain difficult to access through language. This gap motivates our subsequent alignment experiments: can additional cross-modal connections improve textual accessibility? Building on this, we propose Textual Alignment and Interleaved Image–Text Alignment, both of which yield improvements on this problem.

W2: Novelty.

Many existing works attempt to integrate generation and understanding in Unified Multimodal Models (UMMs) so that the two capabilities reinforce each other, yet there is still no general and fair way to measure this effect. To address this problem, our contributions consist of two progressive levels:

  1. A fine-grained, comprehensive, and knowledge-isolated measurement protocol. As the reviewer noted in the Critical Strengths, we propose Knowledge Transfer Probing (KTP) to evaluate how visual generation training facilitates knowledge acquisition in UMMs with different architectures and training paradigms. Our protocol directly measures knowledge transfer at the micro level of individual knowledge units, rather than inferring it from improvements in aggregate benchmark scores. Compared with Modal Aphasia, our data design is more fine-grained and comprehensive (seven categories of visual knowledge), and the isolation from pre-trained knowledge is much cleaner (fictional entity names plus LLM-based checks for information isolation), which ensures the rigor of our experiments.
  2. General strategies for improving knowledge acquisition from visual generation. Building on KTP, we design a series of comparative experiments and ablations, and propose several general approaches to improve UMMs' ability to acquire knowledge from visual generation. Beyond Modal Aphasia, we explore multiple alignment strategies to narrow the gap (Textual Alignment and Interleaved Image–Text Alignment), and examine how performance differs across model architectures and tasks. These strategies yield an average improvement of 9.71 percentage points in textual accessibility across the three models.

Q2: Ablation of interleaved structure.

We sincerely thank the reviewer for this suggestion. We have added an ablation study on the interleaved structure for all three models:

  • Training data: Each interleaved example is split into one standalone T2I example and one standalone T2T example.
  • Training setup: The training budget and the number of epochs are kept identical.
  • Results: Under matched visual supervision and training budgets, the interleaved structure brings additional gains (+1.90 percentage points on average). Given the generally low accuracy levels, this indicates that the interleaved structure indeed helps the model learn the connections between the two modalities.

Textual accessibility (%) on the test entities:

Training conditionBAGELGLM-OmniSenseNova-U1Avg
Textual alignment15.4312.5728.0018.67
Ablation (interleaved → T2I + T2T)16.0024.0026.8622.29
Interleaved alignment17.1426.8628.5724.19

Thanks for author's response, which have addressed my main concerns. I have raised my scores.

Reviewer_tL3vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether learning to generate images can expand what unified multimodal models (UMMs) know that whether knowledge picked up through image-generation supervision becomes accessible in text-only QA. They introduce Knowledge Transfer Probing (KTP), a controlled protocol built on fictional entities across 7 knowledge categories, and evaluate three UMMs . The headline finding is direct but limited and uneven transfer: visual accessibility substantially exceeds textual accessibility. They then add Textual Alignment and Interleaved Image–Text Alignment, plus a Visual CoT inference step, to try to close the gap.

Critical strengths

The research question is cleanly posed — whether generation and understanding actually reinforce each other in UMMs is a central open question, and the paper gives it a crisp, falsifiable form. KTP is the strongest part: fictional entities, information isolation, and balanced attributes make a genuinely controlled measurement tool, and the promised dataset/protocol release is a real contribution. The quantification is informative — visual accessibility far exceeds textual accessibility (gaps ~22–41pp; e.g., SenseNova-U1 88% visual vs. 0% text on number), and even after interleaved alignment Pr(T|I) stays under 50%. This puts concrete numbers on a "modal aphasia"-type dissociation. Some causal evidence: masking the image loss in interleaved sequences drops average QA from 26.86% to 13.71%, suggesting the explicit image-generation supervision does real work. The effect is consistent across three architecturally different UMMs, supporting generality. Reporting is honest about the limits and points to a plausible direction (interleaved > textual; Visual CoT helps).

Critical weaknesses

Novelty needs clarification against an existing, closely related work. The core phenomenon — visual memory far exceeding textual access — was already named and characterized as Modal Aphasia (Aerni et al., ICLR 2026)​, which the authors themselves cite. This paper studies a related but different setting: knowledge newly injected via generation training (isolated through fictional entities) rather than already-memorized concepts. That scope is reasonable and arguably a real extension. However, the paper never clearly states what is genuinely new beyond Modal Aphasia — e.g., whether the dissociation behaves differently for injected vs. memorized knowledge, and whether the controlled protocol itself is the contribution. As written, the novelty rests mostly on re-measuring a known dissociation under a cleaner, synthetic setup. No method innovation. Textual Alignment, Interleaved Alignment, and Visual CoT are combinations of existing techniques; there is no new algorithmic contribution. Conclusions are mostly negative and improvements limited. Transfer stays limited and uneven; even after interleaved alignment + Visual CoT, accuracies remain low (e.g., GLM-Omni 41.14%). The paper characterizes a problem more than it solves one. Low statistical power. 7 categories × ~50 units, test entities a subset (~25 each), many cells at 0% or single digits, and no significance testing — hard to tell which differences are real. Evaluation reliability assumed, not validated. Visual accessibility is judged by a VLM and information isolation relies on an LLM reviewing captions; neither is validated against human judgment, so systematic error is possible.

Questions for authors

What is genuinely new beyond Modal Aphasia (Aerni et al., ICLR 2026)? Please articulate the increment — is it the controlled protocol, the focus on newly injected knowledge, or the mitigation strategies — and whether injected vs. memorized knowledge dissociate differently. Given ~25 test entities per category and many cells at 0%/single digits, are the headline gaps and alignment gains statistically significant? Please report CIs or a paired test. How accurate is Judge_VLM for visual accessibility, and how reliable is the LLM used for information isolation? Any human validation on a subset? Any mechanistic insight into why injected visual knowledge stays hard to access in text (e.g., a generation-specific branch/subspace, relevant to BAGEL's MoT split)? Future-work suggestions (not required): extend to natural visual data and real knowledge; isolate architectural effects; explore stronger alignment/inference strategies.

1 comment

Thank you for the review and constructive suggestions. We address each point below.

W1 / Q1: Novelty.

Many existing works attempt to integrate generation and understanding in Unified Multimodal Models (UMMs) so that the two capabilities reinforce each other, yet there is still no general and fair way to measure this effect. To address this problem, our contributions consist of two progressive levels:

  1. A fine-grained, comprehensive, and knowledge-isolated measurement protocol. We propose Knowledge Transfer Probing (KTP) to evaluate how visual generation training facilitates knowledge acquisition in UMMs with different architectures and training paradigms. Our protocol directly measures knowledge transfer at the micro level of individual knowledge units, rather than inferring it from improvements in aggregate benchmark scores. Compared with Modal Aphasia, our data design is more fine-grained and comprehensive (seven categories of visual knowledge), and the isolation from pre-trained knowledge is much cleaner (fictional entity names plus LLM-based checks for information isolation), which ensures the rigor of our experiments.
  2. General strategies for improving knowledge acquisition from visual generation. Building on KTP, we design a series of comparative experiments and ablations, and propose several general approaches to improve UMMs' ability to acquire knowledge from visual generation. Beyond Modal Aphasia, we explore multiple alignment strategies to narrow the gap (Textual Alignment and Interleaved Image–Text Alignment), and examine how performance differs across model architectures and tasks. These strategies yield an average improvement of 9.71 percentage points in textual accessibility across the three models.

W2: Clarification on conclusions and improvements.

The observations from our experiments are two-sided:

  1. Generation can benefit understanding, and the benefit is further enhanced with alignment data. Textual accessibility improves by 8.28, 12.29, and 8.57 percentage points for BAGEL, GLM-Omni, and SenseNova-U1, respectively.
  2. However, the effects are inconsistent across models and tasks, which merits further investigation in future work.

W3 / Q2: Statistical analysis.

We sincerely thank the reviewer for this suggestion. We have added a paired statistical analysis using a two-sided McNemar test, with 95% confidence intervals estimated via paired bootstrap. Compared with direct transfer, interleaved alignment improves accuracy by 8.00, 12.57, and 8.57 percentage points for BAGEL, GLM-Omni, and SenseNova-U1, respectively, all with p<0.05p < 0.05, indicating that the gains brought by alignment data are statistically significant.

ModelDirect acc. (%)Interleaved acc. (%)Δ acc. (pp)95% CI for Δ (pp)McNemar p-value
BAGEL9.1417.14+8.00[1.71, 14.29]0.028817
GLM-Omni14.2926.86+12.57[4.57, 20.57]0.005355
SenseNova-U120.0028.57+8.57[2.86, 14.29]0.027533

Note: The direct-transfer accuracies in Table 1 of the paper are evaluated on all 350 entities, whereas this supplementary analysis uses only the 175 entities in the test set; hence the numbers differ slightly.

W4 / Q3: VLM evaluation reliability.

We sincerely thank the reviewer for this suggestion.

  1. VLM (visual accessibility). We randomly sampled 100 records from all T2I evaluation records, covering all three models (BAGEL 43 / GLM-Omni 30 / SenseNova-U1 27) and all seven categories (color 18, spatial 20, size 16, shape 13, count 12, number 12, text 9). Among them, 46 were judged as correct by the VLM and 54 as incorrect, giving a roughly balanced set of positive and negative samples. Two annotators independently judged the evaluation results. After cross-checking, the agreement between the VLM judgments and the human judgments is 99%, validating the reliability of the VLM-based evaluation.
  2. LLM (information isolation). We randomly sampled 100 records from all LLM leakage judgments, of which 95 were judged by the LLM as non-leaking and 5 as leaking. Two annotators independently judged the results. After cross-checking, the agreement between the LLM judgments and the human judgments is 100%, validating the reliability of information isolation.

Q4: Mechanistic insight.

  1. We attribute the difficulty of textual access mainly to the gap between modalities, e.g., diffusion for images vs. next-token prediction (NTP) for text in MoT architectures.
  2. In Section 4.3, interleaved data benefits GLM-Omni notably more than BAGEL and SenseNova-U1. We conjecture that GLM-Omni's unified NTP architecture, which shares all parameters between generation and understanding, better connects the two modalities during interleaved training. A detailed mechanistic analysis is left for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.