acceptodds
Withdrawn from ICLR 2027

Omni-Modal LLMs Know More Than They Say

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (*e.g.*, vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks. Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly. We term this disconnect *knows-but-cannot-say*. To investigate this phenomenon, we introduce Token-level Modality Alignment (TMA), which measures the alignment between individual modality representations and response representations. Our analysis shows that non-textual representations are less aligned with response tokens and receive less attention than textual representations, suggesting a potential explanation for uneven modality preferences. Motivated by these findings, we propose Omni-Catalyst, a simple *perceive* *infer* strategy that complements the original modality inputs with self-generated text to make non-textual knowledge more accessible. Across seven Omni-LLMs, Omni-Catalyst consistently improves performance (+7.29%) and yields more balanced modality preferences, with further gains from diverse training strategies. Further analysis demonstrates that the knowledge extraction phases increase TMA, suggesting that improved alignment makes non-textual information more accessible during response generation. In summary, our findings uncover **how Omni-LLMs can better express modality knowledge through language**.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omni-modal LLM, Modality Representation, Multimodal

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XHioOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why omni-modal LLMs can extract modality information yet fail to use it in answering. It proposes TMA to measure input–response representation alignment and OMNI-CATALYST to supplement original inputs with self-generated descriptions. Multi-model benchmark evaluations show performance gains, while attention and masking analyses investigate the proposed mechanism.

Critical strengths

The self-generated descriptions improve performance across several model families and benchmarks, suggesting practical utility beyond a single model or task.

Critical weaknesses

Limited novelty. Caption-then-answer was already established by PICa (AAAI 2022). The paper does not sufficiently establish a substantial methodological advance over this paradigm. Insufficient support for the central claim. TMA measures similarity to the generated response. Low TMA may reflect a mismatch with an incorrect answer rather than inaccessible knowledge. The attention and masking analyses do not adequately resolve this ambiguity.

Questions for authors

What is the substantive methodological contribution beyond PICa and subsequent caption-based reasoning approaches? What evidence distinguishes low generation accessibility from low similarity caused by an incorrect response, and establishes that the relevant information was already encoded during direct answering?

2 comments

We thank the reviewer for the insightful review. We have highlighted revisions made in response to this review in blue and gray (common to all reviewers) in the updated version of the paper. We address the major ones below.

W1 & Q1 - Novelty of the method

We appreciate the reviewer’s concern and acknowledge that prior works, including PICa (which we have additionally cited in the Related Works section) and OmniXR, also employ modality descriptions. However, our novelty lies not merely in prompting, but in systematically investigating how Omni-LLMs access modality knowledge during generation and introducing overlooked perception-centered inference and training strategies. Concretely, we introduce Token-level Modality Alignment (TMA) to characterize how text and non-textual modalities show different accessibility during response generation, even when the text is irrelevant. Therefore, our analyses show that non-textual knowledge can remain recoverable yet underutilized because it is less accessible than text. We further internalize this strategy into model parameters by explicitly optimizing modality description generation rather than solely answer prediction. Perception-oriented training outperforms direct inference-oriented training and is compatible with existing self-distillation strategies. Our contribution is thus a unified analysis and methodology for making already-perceived modality knowledge more accessible during generation. We have revised the Introduction section to better emphasize the contribution of this paper.

W1 - Differences with the prior method

Unlike prior caption-then-answer approaches, OMNI-CATALYST preserves original modality representations while augmenting them with textual descriptions. This matters because generated descriptions may be incomplete or incorrect; our main (MX’s in Fig. 9) and ablation results (Tab. S4) show that augmentation outperforms replacement.

W1 & Q1 - TMA measure

Importantly, TMA measures generation accessibility, not correctness: even incorrect responses can exhibit high text alignment. Thus, low TMA is not simply a consequence of incorrect answers. Rather, our findings suggest that textual descriptions provide a more accessible pathway to modality knowledge during language generation, even though textualization may preserve less information than the original sensory inputs. Retaining both modality and textual representations improves overall performance.

References

[1] Omnixr: Evaluating omni-modality language models on reasoning across modalities (ICLR, 2025).

[2] An empirical study of gpt-3 for few-shot knowledge-based vqa (AAAI, 2022).

Thank you for the response and the clarifications.

While I appreciate the added discussion distinguishing augmentation from replacement and the training-level formulation, my core concern regarding the TMA analysis remains unresolved. The rebuttal states that TMA measures generation accessibility rather than correctness, but does not provide controlled evidence to rule out confounding from incorrect responses or to establish that the relevant modality information is already encoded during direct answering.

Given that the central claim is still not convincingly supported and the methodological novelty remains incremental, I will maintain my current rating of 1.

Reviewer_iag6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a proposed gap between the modality information omni-modal LLMs can describe and the information they use when answering multimodal questions. It introduces Token-level Modality Alignment (TMA), which computes, for each modality token, its maximum cosine similarity to generated-response tokens using last-decoder-layer hidden states. Motivated by this analysis, Omni-Catalyst generates textual modality descriptions and uses them alongside the original sensory inputs, with both training-free and trainable variants. Experiments evaluate reasoning performance and modality preference across several model families and benchmarks.

Critical strengths
  • Omni-Catalyst offers an accessible way to improve multimodal answering by generating modality descriptions while retaining the original inputs. The results show improvements across several models and benchmarks, with particularly useful gains from audio descriptions.
Critical weaknesses
  • One of my main concern is that the novelty of this paper limited. The proposed method is relatively straightforward and seems closely related to existing prompting strategies. While the experimental results are promising, the paper does not clearly demonstrate a substantial conceptual or methodological advance.

  • Appendix E (p. 25) reports the fraction of examples with non-zero keyword recall, allowing semantic matches to key reference-description elements for the provided modalities. Under this criterion, recovering one such element is sufficient to count toward the reported fraction, even if that element alone does not provide sufficient evidence to answer the question. The gap between this measure and answer accuracy therefore does not, by itself, establish that the bottleneck lies in read-out rather than encoding. Checking whether the facts needed to answer each question are recovered, and whether reasoning still fails in those cases, would strengthen this interpretation.

  • The masking experiment in Figure 4a and response-resampling results in Table S5 provide useful support for TMA as a diagnostic. However, in the modality-count analysis (Figures 1b and 4b; Appendix E, p. 27), the three-modality condition consists of two real modalities (video/image and audio) plus task-irrelevant text, so the two-to-three comparison also changes relevance and context rather than isolating modality count. Appendix E further describes Figure 4b as taking the strongest modality’s alignment for each example before aggregating across examples and models. This does not directly support Section 3.2’s interpretation of increased variance as greater differences among modality tokens. The broader claim that adding modalities exacerbates the accessibility problem (Section 3.2) therefore needs clarification or narrower wording.

Questions for authors
  1. Could the knowledge analysis verify recovery of the facts needed to answer each question, beyond non-zero keyword recall based on verbatim or semantic matches to key elements of the ground-truth description? Please report the proportion of reasoning failures among examples where those necessary facts are verified as present. For Figure 4a, please clarify whether the same sensory-token positions were masked in both tasks.

  2. Please clarify the statistic and source of variability plotted in Figure 4b, and how the maximum-over-modalities aggregation supports the token-level interpretation in Section 3.2. What do the plotted points and error bars represent? Could the authors distinguish effects of adding relevant evidence from adding irrelevant text in the modality-count analysis? Please specify the prompts used for the knowledge and reasoning conditions, including how the irrelevant text was framed.

  3. Section 5.2 already notes that adding STaR after perceive-only training does not further improve performance, while Table S10 reports improved out-of-distribution audio-captioning scores. In Table S8, Perceive + STaR has the numerically highest reported average across the four models without inference-time Omni-Catalyst augmentation (37.6). With this augmentation, its average is 42.0, compared with 43.4 for perceive-only training and 43.6 for the baseline models without additional training. Could the authors clarify which evaluation setting and comparison support the conclusion’s claim of “the strongest descriptions and performance”?

3 comments
AuthorsOpenReview ↗

Response to Reviewer iag6 (Part 1)

We thank the reviewer for the thorough review. We have highlighted revisions made in response to this review in red and gray (common to all reviewers) in the updated version of the paper. We address the major ones below.

W1 - Novelty of the method

Our novelty lies not merely in prompting, but in systematically investigating how Omni-LLMs access modality knowledge during generation and introducing overlooked perception-centered inference and training strategies. Concretely, we introduce Token-level Modality Alignment (TMA), revealing that non-textual information can remain recoverable yet underutilized during generation. Importantly, our improvements are not simply attributable to prompt design. OMNI-CATALYST demonstrates robustness across prompt formulations (App. A), while our comparisons with Self-CoT (§6.2) and external description models (App. F) further examine the effectiveness of modality-specific descriptions beyond conventional prompting.

Furthermore, we explicitly optimize modality description generation rather than solely answer prediction. This perception-oriented training outperforms direct inference-oriented training and is compatible with diverse self-distillation strategies. Our contribution is a unified analysis and methodology for improving the utilization of modality knowledge. We have revised the Introduction to clarify these contributions. We have revised the Introduction section to better emphasize the contribution of this paper.

W2 - Metric justification & Q2 - Proportion of failure examples

Thank you for your insightful question and suggestion. We agree that non-zero keyword recall may not fully recover answer-relevant evidence. We therefore conducted an additional audit of 2,866 baseline-incorrect model–sample pairs from the main Omni-Catalyst results. An outcome-blind Gemma-4-12B-it judge separately assessed whether the generated audio description was faithful to the reference and whether it recovered sufficient question-relevant audio evidence, considering the reference image information. The judge did not see the tested model’s final answer or correctness.

Among the 581 cases corrected by Omni-Catalyst, 63.3% contained faithful descriptions, and 27.2% additionally satisfied the question-relevant knowledge criterion:

ModelIncorrect → correctFaithful descriptionFaithful description with sufficient question-relevant audio knowledge
Gemma-2B195108 (55.4%)46 (23.6%)
Gemma-4B15698 (62.8%)45 (28.8%)
Qwen-3B11371 (62.8%)30 (26.5%)
Qwen-7B11791 (77.8%)37 (31.6%)
Total581368 (63.3%)158 (27.2%)

Importantly, we also evaluated cases that remained incorrect, rather than examining only successful corrections:

Description criterionCasesBecame correctRemained incorrect
Faithful audio description1,681368 (21.9%)1,313 (78.1%)
Faithful description with sufficient question-relevant audio evidence550158 (28.7%)392 (71.3%)

Importantly, 28.7% of cases with sufficient question-relevant audio evidence were successfully corrected by Omni-Catalyst, compared with 21.9% among cases with faithful descriptions alone. Moreover, the remaining 71.3% demonstrate that even when question-relevant evidence is explicitly recovered, successful reasoning is not guaranteed. These findings strengthen our central motivation: recovering modality knowledge and effectively using it during answer generation are distinct challenges. Omni-Catalyst directly addresses this gap by making modality information more accessible through explicit textual descriptions.

In response to your second question, Fig. 4a uses task-specific TMA rankings, so we did not enforce identical sensory-token positions across tasks. However, as illustrated in App. E (p. 26) and Fig. S9, we observe that high-TMA tokens in both tasks correspond to semantically relevant modality content, whereas low-TMA tokens correspond to less meaningful regions or frames regardless of the type of task.

AuthorsOpenReview ↗

Response to Reviewer iag6 (Part 2)

W3 - Experiment interpretation

Thank you for the recognition. Our experiment considers a specific setting in which Omni-LLMs are built upon text-based foundational models, with additional sensory modalities (\eg, video, audio) incorporated into the language-centric architecture. Thus, we investigate modality expansion under this setting rather than exhaustively evaluating all possible modality combinations. Different foundational-model architectures may require examining additional modality combinations.

We also acknowledge that the semantic relevance of the added text could confound the comparison. To address this, we replace the irrelevant text with relevant text in the updated manuscript (new Fig. S18, App. F). We observe results similar to those in Fig. 4b, suggesting that the observed alignment pattern is not influenced by the semantic relevance of the added modality.

Our original intention was to investigate how generation-level modality alignment changes as additional modalities become available. Since Fig. 4b takes the maximum modality-level TMA for each sample, the increased variance reflects greater variability in the strongest modality’s alignment across samples, rather than differences among modality tokens (Fig. 3). As suggested, we have clarified this direction in Section 3.2.

Q2 - Figure clarification

Sorry for the lack of explanation. Here are the responses to your questions, one by one:

1. Statistics and sources of variability & plotted points and error bars

For example ss, condition cc, and input modality mm, we compute

As,c,m=1∣Is,c,m∣∑i∈Is,c,mmax⁡j∈Rs,ccos⁡(hi,hj),As,c=max⁡m∈cAs,c,m.A_{s,c,m}=\frac{1}{\lvert I_{s,c,m}\rvert}\sum_{i \in I_{s,c,m}} \max_{j \in R_{s,c}} \cos\left(h_i, h_j\right), \qquad A_{s,c}=\max_{m \in c} A_{s,c,m}.

Here, Is,c,mI_{s,c,m} contains the input-token positions for modality mm, Rs,cR_{s,c} contains the generated-response positions, and hi,hjh_i, h_j are the last-decoder-layer hidden states. The variability in Figure 4b reflects differences across input modality samples. Specifically, each bar represents As,cA_{s,c} averaged across 300 instances sampled from the OmniBench benchmark. Each dot illustrates the mean TMA of a group of samples, formed by dividing the observed TMA range into 60 equal-width bins and merging adjacent bins until each group contains at least 10 examples; dot area is proportional to the number of examples in the group. The error bar is the minimum-to-maximum range of raw per-example TMA scores.

2. Maximum-over-modalities aggregation

We first compute TMA at the token level by measuring each input token’s maximum cosine similarity to the generated-response representations. For Figure 4b, we average these token-level scores within each modality and then take the maximum across available modalities. This aggregation characterizes the modality with the highest average alignment to response generation. It does not imply that all modalities or tokens become more accessible. Hence, we want to highlight that as more modalities are available, which is what omni-modal models are originally intended to be built for, there is a higher likelihood that the available modalities are accessible to the model during answer generation. Subsequently, accessibility is more likely to become uneven across modalities, as illustrated by the increased variance in TMA.

3. Adding relevant evidence vs. irrelevant text

We added new Fig. S19 (App. F), replacing the irrelevant text with relevant evidence. We note that the trend is similar to Fig. 4b (irrelevant text), as expected. Hence, modality type matters, not semantics, for TMA score effects.

AuthorsOpenReview ↗

Response to Reviewer iag6 (Part 3)

4. Prompts and irrelevant text

We used “You are a helpful assistant. Please analyze {input_label} carefully and describe it.” as the system prompt for the knowledge condition. The user instruction is as follows:

Available sensory inputsInput labelUser instruction
Imagethe given image (visual only, no audio)Describe what you see.
Audiothe given audio onlyDescribe what you hear.
Image + audiothe given image and audioDescribe what you see and hear.

For the reasoning condition, the system instruction replaces describe it with answer the question, and the user prompt is: Please answer the following question based on {input_label}: {question}. Please choose only one answer from the following options: {options}.

For the irrelevant text framing when the modality count is 3, the prompts are:

  • System prompt: “A separate sentence of unrelated filler text is also shown to you -- it is NOT a caption or label present in the image, and NOT part of the audio; it has nothing to do with the scene. Ignore it completely and describe only what you actually see and hear.”
  • User content: “The sentence below is unrelated filler text picked at random for this experiment. It is NOT a caption, subtitle, or label shown in the image, and it is NOT part of the audio -- it has no connection to the image or audio at all: "{sentence}"”

For the irrelevant sentence, we chose short, meaningful sentences about everyday topics unrelated to the image, audio, or task, such as weather, finance, transport, and food. Examples include:

  • “The weather is expected to be rainy tomorrow.”
  • “The stock market closed slightly higher today.”
  • “The recipe calls for two cups of flour and a pinch of salt.”

We added these details in the Appendix.

Q3 - Writing consistency

Sorry for the confusion. Yes, you are correct on all the details. First, we want to clarify that our concluding statement on “StaR-based self-refinement producing the strongest descriptions and performance” refers to Perceive+StaR in Figure 6, with Omni-Catalyst applied only during training (not inference). While Perceive+StaR achieves better out-of-distribution audio captioning (Table S10), it does not further improve reasoning performance when an additional training-free Omni-Catalyst is applied during inference time.

Second, regarding your statement: “With this augmentation, its average is 42.0, compared with 43.4 for perceive-only training and 43.6 for the baseline models without additional training,” we notice that training-free Omni-Catalyst at inference time on the vanilla model achieves the best results. However, we also wanted to show that we could internalize training-free Omni-Catalyst into model parameters using different training /self-training strategies, and that they could improve over the baselines across models.

Because the original conclusion statement could mislead readers into thinking that Perceive+StaR always achieves the best results, we revised it to emphasize our method's adaptability to different self-training strategies.

Reviewer_Epz7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper reports a "knows-but-cannot-say" phenomenon in Omni-LLMs: the model can encode information from non-text modalities (audio, video, and so on) correctly, yet fails to use it effectively when generating the answer, and once the same information is rewritten as text, the answer becomes correct. On this basis the authors propose Token-level Modality Alignment (TMA). For the last-layer hidden state of each modality token, TMA takes the maximum of its cosine similarities to the hidden states of all response tokens (Eq. 1), and it keeps the full within-sample and cross-sample distributions to characterize how well modality tokens align with the generated representations. The authors observe that non-text tokens have lower TMA and receive less attention than text tokens, and that the gap grows as more modalities are added. From this they propose OMNI-CATALYST: the model first generates its own text description dmd_m of each modality, and the descriptions are then fed in together with the original modality representations to produce the answer ("perceive → infer"). The method comes in a training-free form (single-pass / two-pass) and a trainable form (LUL_U, LPL_P, plugged into STaR / R3V / SD-Zero). The experiments cover 5 main models (gemma-4-e2b/e4b-it, Qwen2.5-Omni-3B/7B, OneLLM) plus 3 models in the appendix, four benchmarks (OmniBench / WorldSense / AVHBench / OmniPreference), and several single-modality tasks. The abstract reports an average improvement of +7.29%.

Critical strengths
  1. The problem is well chosen and matters in practice. Omni-LLMs doing worse on multimodal input than single-modality experts is a real problem. The paper recasts it from "insufficient perception" to "perceived information does not reach the generation path". This change of viewpoint has value in itself, and it gives later work an actionable problem definition.

  2. One key comparison is clean and convincing. Replacing the original modality with its description (the MA / MV / ME variants) lowers performance, while adding the description raises it. This directly supports the claim that the gain comes from an added textual pathway rather than from substituting text for the modality, and it is the strongest link in the paper's evidence chain.

  3. The coverage aims to be broad. The experiments span model families and benchmarks and extend to single-modality settings such as fMRI / IMU / point clouds. The appendix gives the prompts, LoRA hyperparameters, a model architecture table, and the training configurations, so the work is set up to be reproducible.

  4. The attempt to validate TMA deserves credit. The appendix masks tokens by TMA tier, analyzes stability under response resampling with ICC / Fleiss κ, and compares TMA against CD / CKNNA / SEA. This shows the authors are aware that the metric needs additional support.

Critical weaknesses

W1 (decisive) The evidence for the core diagnosis does not hold

The paper claims that the model perceives correctly but fails to express what it perceives, yet the only measurement supporting "perceives correctly" is the text description itself. Appendix E states:

We evaluate perceptual knowledge using the corresponding open-ended modality-description task.

So "know" is measured by having the model do a description task and having an LLM judge score the result, and description generation is exactly what OMNI-CATALYST trains and improves. A poor description can come from a failure to perceive or from weak description ability, and this measurement cannot tell the two apart. The paper also runs no experiment that is independent of the output channel (e.g., linear probes on hidden states, or causal interventions on attention). More concretely:

  • The §1 statement "Downstream accuracy collapses with each added modality while perception accuracy holds" and the §3.2 statement "knowledge performance remains high regardless of the TMA of masked tokens, whereas answering performance is substantially lower" are not backed by any numbers. Figure 1b and Figure 4a are purely schematic, with no paired perception accuracy / downstream accuracy values and no sample sizes.
  • The construct validity of TMA, which the diagnosis relies on, is also questionable. Eq. 1 takes the max over response tokens, so the quantity is structurally non-decreasing as the response grows, and the paper has no length-matched or length-normalized control at all. The correlation between TMA and accuracy is never reported, and the §3.2 main text itself says "the association is modest".

Please provide: the actual perception accuracy and downstream accuracy values and sample sizes in Figure 1b and Figure 4a, the judge prompt and human spot-check rate on the perception side, a length-matched TMA control, and the correlation between TMA and accuracy.

W2 (decisive) The gain disappears as base models get stronger, and the conclusion has no stated shelf life

Appendix Table S7 reports:

ModelBaseline+OMNI-CATALYSTGain
Qwen3-Omni-30B-A3B-Instruct51.652.6+1.0 (Music +0.0, Sound Event −1.1)
MiniCPM-o-2.639.142.0+2.9 (Music −1.9)
OmniVinci36.848.3+11.5

In the main table, by contrast, the gains for gemma-4-e2b/e4b are +11.2 / +13.4. So on the newer Qwen3-Omni-30B, the method's gain collapses to 1.0 percentage point, and some subcategories are negative.

  • The abstract's claim that the method "consistently improves" does not hold. +1.0 and +13.4 are averaged into the same number. What supports "consistently" is the count of heterogeneous models; the gains themselves are not stable.
  • The paper does not discuss the shelf life of its conclusion. No experiment or discussion touches on whether this phenomenon will disappear by itself as base-model post-training improves. If it is a transient product of insufficient training alignment, then presenting it as an architecture-level bottleneck that needs a dedicated method is an overgeneralization.

The LPL_P objective the authors already have (using modality descriptions as supervision to train the model to "perceive") is the direction with more weight, yet it gets a single incremental experiment of 2–3% and is then set aside, so the paper undervalues its own most valuable part.

Please provide: whether the phenomenon still holds on a strong base model tuned as well as possible (e.g., the Qwen3-Omni series), and why the gain collapses.

W3 (decisive) The main results are not solid

  • The +7.29% cannot be reproduced from Table 1. The number does not appear in any table or row of the paper, and averaging the per-row Avg. deltas of the four models in Table 1 (11.2 / 12.9 / 7.5 / 7.6) gives 9.8. The abstract says "Across seven Omni-LLMs", but the other 3 models appear only in appendix Table S7, which covers OmniBench alone, so the 7 models are pieced together. The paper also never says whether the number is an absolute or a relative improvement.
  • The description stage can see the question and options, and there is no control for it. The single-pass prompt includes the line "Do not mention any answer options in the transcription." The need to write this prohibition shows that the options are already in the context when the description is generated, yet the paper has no question-blind or option-hidden control experiment.
  • Baselines and statistics are underreported. The paper never states whether CoM and ThinkOmni were rerun with the same model, prompt, and inference configuration. Self-CoT has no entry in the references, and the paper does not say whether the authors reimplemented it themselves. No random-seed counts or error bars are reported, and the asterisks for p-values in Figure 5 come with no test name.

Please provide: the exact calculation behind +7.29% (models, benchmarks, subsets, absolute or relative improvement), the gain under a question-blind control, and the reproduction protocol for CoM / ThinkOmni / Self-CoT, with the number of random seeds and error bars.

Questions for authors
  1. What are the actual perception accuracy / downstream accuracy values and sample sizes in Figure 1b and Figure 4a? How is perception measured (hidden-state probes, model self-description + judge, or human annotation)? What is the judge prompt, and what is the human spot-check rate?
  2. Please give the full calculation behind +7.29%: the models, benchmarks, and subsets included, and whether the number is an absolute or relative improvement.
  3. On Qwen3-Omni-30B-A3B-Instruct the gain is only +1.0 (with Music +0.0 and Sound Event −1.1), while on Qwen2.5-Omni-3B it is +5.8. How do the authors explain this decline as base-model capability increases? Could the phenomenon disappear with stronger post-training?
  4. Can the description stage see the question and options? If that visibility is cut off, how much of the gain remains?
  5. TMA uses the max similarity over response tokens. Does this definition grow with response length? Is there a length-matched control? What is the correlation coefficient between TMA and downstream accuracy?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Epz7 (Part 1)

We thank the reviewer for the detailed review. We have highlighted revisions made in response to this review in yellow and gray (common to all reviewers) in the updated version of the paper. We address the major ones below.

W1 - The evidence for the core diagnosis & Q5 - Length-matched control experiment

We thank the reviewer for raising these important concerns regarding the distinction between perceptual knowledge and generation accessibility, as well as the validity of TMA. We clarify the following points.

1. Distinguishing perceptual knowledge from generation accessibility

We acknowledge that open-ended descriptions cannot fully disentangle perceptual failures from limitations in description generation. However, our study focuses on cases where modality information is recoverable from the input, rather than cases where the modality encoders inherently fail to capture the relevant information. Specifically, our knowledge performance evaluation serves as a proxy for identifying recoverable perceptual information, allowing us to investigate whether such information is effectively utilized during downstream inference. Diagnosing failures originating within the modality encoders themselves is beyond the scope of the current work.

Importantly, both direct answering and OMNI-CATALYST operate on the same original modality inputs, with the latter complementarily expressing modality information as text. Although it introduces no new external perceptual information (beyond what modality encoders already encode from the original modalities), it substantially improves downstream accuracy, suggesting the observed failure cannot be attributed solely to insufficient perception. Instead, explicitly rendering modality information into text can provide a more accessible pathway for subsequent language generation.

We agree that this does not conclusively establish that all relevant information was already encoded during direct answering. We therefore clarified the scope of our claim, distinguishing recoverable modality knowledge from independently verified latent knowledge and focusing on generation accessibility rather than encoder-level perceptual failures in the updated manuscript.

2. Clarification of Figures 1b and 4a and perception evaluation

We respectfully clarify that Figures 1b and 4a are based on empirical evaluations, rather than purely schematic illustrations. As detailed in Appendix E, we evaluate downstream performance using actual benchmark accuracy, while assessing perceptual knowledge through open-ended modality descriptions. An independent text-only judge extracts key content elements from ground-truth descriptions and evaluates their semantic presence in model-generated descriptions. Perceptual knowledge accuracy is computed as the fraction of examples with non-zero keyword recall.

Figure 1b compares these two measurements as the number of available modalities increases, while Figure 4a examines their sensitivity to masking modality tokens grouped by TMA. In particular, Figure 4a shows that masking higher-TMA tokens produces greater degradation in downstream answering performance, whereas description-based knowledge performance remains comparatively stable. We provided the underlying numerical results, sample sizes, evaluation prompts, and further implementation details to make these analyses more transparent and reproducible in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Epz7 (Part 2)

3. TMA length sensitivity and its relationship with accuracy

We acknowledge that the maximum operation in Eq. (1) introduces potential response-length dependence. To examine this, we compute TMA statistics across three response-length groups (minimum 47 samples per group). As shown below, reasoning TMA increases with response length, whereas knowledge TMA exhibits a non-monotonic pattern, suggesting that response length alone does not determine TMA.

Response length (tokens)Reasoning TMA meanReasoning varianceKnowledge TMA meanKnowledge variance
1–640.28940.01550.36340.0045
65–1280.38730.01640.33660.0065
≥1290.41780.01820.38140.0188

We additionally provide a length-matched control of Fig. 4a in Fig. S18, where the gap between knowledge extraction and reasoning performance persists across TMA levels, particularly for shorter responses.

Regarding the correlation between TMA and accuracy, TMA characterizes token-level alignment distributions rather than sample-level scores. Aggregating these distributions for correlation analysis would discard fine-grained information. Instead, Fig. 4a directly evaluates TMA's relationship with downstream performance through token masking: masking higher-TMA tokens causes greater degradation in answering accuracy, while description-based knowledge extraction remains comparatively stable. This supports TMA as a diagnostic of tokens' relative contribution to response generation, rather than a direct measure of encoded perceptual knowledge. We have clarified this experimental design and its interpretation in the revised manuscript.

W2 - Writing consistency regarding the performance & Q3 - Scalability

We thank the reviewer for raising this important concern regarding the diminishing gains on stronger models and the long-term relevance of our approach. We will answer your questions one-by-one:

1. Why do gains diminish on stronger models, and could the phenomenon disappear with improved post-training?

We hypothesize that the smaller gains on Qwen3-Omni-30B-A3B-Instruct stem from its stronger multimodal capabilities and more extensive post-training. As larger models approach performance saturation on the evaluated benchmarks, the potential for further improvement becomes increasingly limited. Nevertheless, the gain on Speech (+1.9) suggests that modality-specific limitations remain even in stronger models. Further advances in post-training may continue to reduce the benefits of OMNI-CATALYST. However, our findings demonstrate that modality knowledge utilization remains an important challenge in current Omni-LLMs, even as model capabilities improve. In the revision, we have clarified that the improvement depends on the base model's capabilities and remaining potential for improvement, rather than assuming a persistent architectural bottleneck.

2. Clarifying the claim of “consistent” improvements

Our original use of “consistently improves” was intended to indicate that OMNI-CATALYST achieves positive average gains across all evaluated model families, not uniform improvement magnitudes across models or individual tasks. We agree that this wording could overstate the stability of the improvements, particularly given the smaller gains and occasional negative subcategory results on stronger models. We weakened this claim and explicitly highlight the substantial variation in gains across model capabilities in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Epz7 (Part 3)

3. Strengthening the emphasis on modality-description-based training

We appreciate the reviewer’s recognition of our modality-description-supervised training objective as a promising direction. We agree that this component deserves greater emphasis, particularly because inference-time gains may naturally diminish as base models become stronger.

Our current training experiments show that modality descriptions can serve not only as inference-time guidance but also as supervision to improve the model’s direct multimodal perception, yielding additional improvements of about 2–3 percentage points. Although these gains are modest, they provide initial evidence that we can incorporate the underlying insight into model training rather than relying exclusively on inference-time augmentation.

Our experiments represent our best effort within the availability and limitations of existing open-source training datasets. We believe more diverse and targeted multimodal training data could further strengthen this direction, although this remains to be validated experimentally.

In the revision, we substantially shifted the paper’s emphasis toward modality-description-based supervision as a generalizable training direction, positioning inference-time OMNI-CATALYST as a diagnostic and practical demonstration of the underlying phenomenon rather than a necessarily permanent intervention. We believe this revised framing better captures the significance and potential long-term relevance of our findings while acknowledging the limitations observed on stronger models.

W3 - Main result explanation & Q2 - Full calculation

Sorry for the confusion. We calculated as follows on the OmniBench benchmark:

ModelBaselineOmniCatalyst
gemma-4-e2b-it21.432.6+11.2
gemma-4-e4b-it27.140.5+13.4
Qwen2.5-Omni-3B43.849.6+5.8
Qwen2.5-Omni-7B46.651.7+5.2
Qwen3-Omni-30B-A3B-Instruct51.652.6+1.0
omnivinci36.848.3+11.5
MiniCPM-o-2 639.142.0+2.9
Avg.39.145.3+7.3

Note that the improvements are absolute since we do not divide the improvement values by the original baseline values.

Also, we clarify that while the single-pass control shows questions/options in the context while generating descriptions, the question-blind control is our original two-pass OmniCatalyst (Sec. 4.1). Here, the description generation stage does not mention any questions/options in the prompts (please refer to the first Prompts 1-3 in Fig. S2). The modality preference distribution and downstream performance results for the two-pass and single-pass are in Tables S6 and Tab. 2/S7, respectively. The reason we selected the single-pass OmniCatalyst as our main training-free method is because 1) it reduces the inference time compared to the question-blind controlled two-pass OmniCatalyst, and 2) while it shows similar downstream performance to the two-pass OmniCatalyst (Tab. 2/S7), it significantly achieves better modality preference balance (Tab. S6).

ThinkOmni and CoM experimental results match those in the paper because ThinkOmni depends on an external model (e.g., DeepSeek, Qwen3), and CoM does not release its code. We follow the same protocol as in the ThinkOmni without using the external model. We run our results across 5 random seeds, and all the standard deviations fall below 0.1. We use a one-sided exact NcNemar test [1] to visualize Fig. 5 because our data are paired binary outcomes: both the base and the variant answer every question, and each answer is either right or wrong.

Reference

[1] Approximate statistical tests for comparing supervised classification learning algorithms (Neural computation, 1998).

Q1 - Perception and downstream performance

Please refer to App. E - Knowledge and reasoning performance (p. 25).

Q4 - Question/options visibility cutoff

We want to clarify that the description stage does not see the question and options for the Omni-Catalyst (two-call). For the single-call version, the model is first asked to generate modality descriptions, where the prompt mentions "Do not mention any answer options in the transcription." (Fig. S2), and then to infer. Hence, results with visibility cutoff are the same as the baseline results. Our Omni-Catalyst improved across benchmarks by better extracting embedded modality knowledge through textual modality descriptions, based on our analyses linking modality alignment and generation accessibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.