W1 (decisive) The evidence for the core diagnosis does not hold
The paper claims that the model perceives correctly but fails to express what it perceives, yet the only measurement supporting "perceives correctly" is the text description itself. Appendix E states:
We evaluate perceptual knowledge using the corresponding open-ended modality-description task.
So "know" is measured by having the model do a description task and having an LLM judge score the result, and description generation is exactly what OMNI-CATALYST trains and improves. A poor description can come from a failure to perceive or from weak description ability, and this measurement cannot tell the two apart. The paper also runs no experiment that is independent of the output channel (e.g., linear probes on hidden states, or causal interventions on attention). More concretely:
- The §1 statement "Downstream accuracy collapses with each added modality while perception accuracy holds" and the §3.2 statement "knowledge performance remains high regardless of the TMA of masked tokens, whereas answering performance is substantially lower" are not backed by any numbers. Figure 1b and Figure 4a are purely schematic, with no paired perception accuracy / downstream accuracy values and no sample sizes.
- The construct validity of TMA, which the diagnosis relies on, is also questionable. Eq. 1 takes the max over response tokens, so the quantity is structurally non-decreasing as the response grows, and the paper has no length-matched or length-normalized control at all. The correlation between TMA and accuracy is never reported, and the §3.2 main text itself says "the association is modest".
Please provide: the actual perception accuracy and downstream accuracy values and sample sizes in Figure 1b and Figure 4a, the judge prompt and human spot-check rate on the perception side, a length-matched TMA control, and the correlation between TMA and accuracy.
W2 (decisive) The gain disappears as base models get stronger, and the conclusion has no stated shelf life
Appendix Table S7 reports:
| Model | Baseline | +OMNI-CATALYST | Gain |
|---|
| Qwen3-Omni-30B-A3B-Instruct | 51.6 | 52.6 | +1.0 (Music +0.0, Sound Event −1.1) |
| MiniCPM-o-2.6 | 39.1 | 42.0 | +2.9 (Music −1.9) |
| OmniVinci | 36.8 | 48.3 | +11.5 |
In the main table, by contrast, the gains for gemma-4-e2b/e4b are +11.2 / +13.4. So on the newer Qwen3-Omni-30B, the method's gain collapses to 1.0 percentage point, and some subcategories are negative.
- The abstract's claim that the method "consistently improves" does not hold. +1.0 and +13.4 are averaged into the same number. What supports "consistently" is the count of heterogeneous models; the gains themselves are not stable.
- The paper does not discuss the shelf life of its conclusion. No experiment or discussion touches on whether this phenomenon will disappear by itself as base-model post-training improves. If it is a transient product of insufficient training alignment, then presenting it as an architecture-level bottleneck that needs a dedicated method is an overgeneralization.
The LP objective the authors already have (using modality descriptions as supervision to train the model to "perceive") is the direction with more weight, yet it gets a single incremental experiment of 2–3% and is then set aside, so the paper undervalues its own most valuable part.
Please provide: whether the phenomenon still holds on a strong base model tuned as well as possible (e.g., the Qwen3-Omni series), and why the gain collapses.
W3 (decisive) The main results are not solid
- The +7.29% cannot be reproduced from Table 1. The number does not appear in any table or row of the paper, and averaging the per-row Avg. deltas of the four models in Table 1 (11.2 / 12.9 / 7.5 / 7.6) gives 9.8. The abstract says "Across seven Omni-LLMs", but the other 3 models appear only in appendix Table S7, which covers OmniBench alone, so the 7 models are pieced together. The paper also never says whether the number is an absolute or a relative improvement.
- The description stage can see the question and options, and there is no control for it. The single-pass prompt includes the line "Do not mention any answer options in the transcription." The need to write this prohibition shows that the options are already in the context when the description is generated, yet the paper has no question-blind or option-hidden control experiment.
- Baselines and statistics are underreported. The paper never states whether CoM and ThinkOmni were rerun with the same model, prompt, and inference configuration. Self-CoT has no entry in the references, and the paper does not say whether the authors reimplemented it themselves. No random-seed counts or error bars are reported, and the asterisks for p-values in Figure 5 come with no test name.
Please provide: the exact calculation behind +7.29% (models, benchmarks, subsets, absolute or relative improvement), the gain under a question-blind control, and the reproduction protocol for CoM / ThinkOmni / Self-CoT, with the number of random seeds and error bars.
We thank the reviewer for the insightful review. We have highlighted revisions made in response to this review in blue and gray (common to all reviewers) in the updated version of the paper. We address the major ones below.
W1 & Q1 - Novelty of the method
We appreciate the reviewer’s concern and acknowledge that prior works, including PICa (which we have additionally cited in the Related Works section) and OmniXR, also employ modality descriptions. However, our novelty lies not merely in prompting, but in systematically investigating how Omni-LLMs access modality knowledge during generation and introducing overlooked perception-centered inference and training strategies. Concretely, we introduce Token-level Modality Alignment (TMA) to characterize how text and non-textual modalities show different accessibility during response generation, even when the text is irrelevant. Therefore, our analyses show that non-textual knowledge can remain recoverable yet underutilized because it is less accessible than text. We further internalize this strategy into model parameters by explicitly optimizing modality description generation rather than solely answer prediction. Perception-oriented training outperforms direct inference-oriented training and is compatible with existing self-distillation strategies. Our contribution is thus a unified analysis and methodology for making already-perceived modality knowledge more accessible during generation. We have revised the Introduction section to better emphasize the contribution of this paper.
W1 - Differences with the prior method
Unlike prior caption-then-answer approaches, OMNI-CATALYST preserves original modality representations while augmenting them with textual descriptions. This matters because generated descriptions may be incomplete or incorrect; our main (MX’s in Fig. 9) and ablation results (Tab. S4) show that augmentation outperforms replacement.
W1 & Q1 - TMA measure
Importantly, TMA measures generation accessibility, not correctness: even incorrect responses can exhibit high text alignment. Thus, low TMA is not simply a consequence of incorrect answers. Rather, our findings suggest that textual descriptions provide a more accessible pathway to modality knowledge during language generation, even though textualization may preserve less information than the original sensory inputs. Retaining both modality and textual representations improves overall performance.
References
[1] Omnixr: Evaluating omni-modality language models on reasoning across modalities (ICLR, 2025).
[2] An empirical study of gpt-3 for few-shot knowledge-based vqa (AAAI, 2022).