- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper frames text-centric post-training for Omni-LLMs as strengthening the model’s reasoning core on purely textual chain-of-thought and verifiable multiple-choice data, then transferring that capacity to audio-visual reasoning. A diagnostic study reveals a marked perception–reasoning gap, and gradient-projection evidence suggests that perception and reasoning objectives are only partially coupled, motivating separate training emphases. The recipe applies SFT with script-based CoT and GRPO with script-based MCQs, updates only the language/Thinker decoder via LoRA while freezing audio-visual encoders and projection modules, and can rely on synthetic fictional scripts, questions, and rationales generated entirely by a text-only LLM to train entity tracking, event-state updates, cross-segment evidence integration, and multi-hop composition. This text-only stage improves omni-modal reasoning with lower training compute but can degrade perception, so the authors propose a staged text-centric paradigm: first strengthen reasoning in text, then use reduced native audio-visual data to restore perceptual ability while retaining the reasoning gains. Validation across multiple Omni-LLM families, reasoning and perception benchmarks, synthetic-data settings, and an agentic case study supports the transfer claim and the compute–perception trade-off.
Critical strengths
The paper’s strongest acceptance case is its diagnostic-to-intervention design: it first establishes a perception–reasoning gap across multiple Omni-LLMs and supports partial decoupling with a local gradient-projection analysis, then uses those findings to motivate and structure a systematic five-RQ study. This is consequential because it makes the text-only training proposal feel evidence-driven rather than merely pragmatic, and it gives the reader a clear causal story for why text-centric post-training should transfer. A second key strength is the breadth and practicality of the validation: gains are shown across model scales/families and multiple reasoning benchmarks, with compute accounting, synthetic text-only data, and a staged recipe that explicitly addresses perceptual degradation. This matters for acceptance because it turns a narrow training trick into a potentially scalable post-training paradigm with a credible compute–perception trade-off.
Critical weaknesses
The most consequential weakness is that the central promise of the staged text-centric paradigm—retaining the main reasoning gains while restoring perception—is not supported by a sufficiently clean, budget-matched comparison and ablation of stage order, reduced-data size, initialization, and compute. Without that, it is hard to rule out that the final gains come simply from additional training or native audio-visual data rather than from the proposed text-centric arrangement, which weakens the paper’s main practical claim. A second weakness concerns mechanism and external validity: the decoupling evidence is a local first-order analysis, the synthetic data depend on an opaque text-only generator, and the benchmark suite may favor text-transferable reasoning while perceptual degradation is acknowledged but not deeply explained. These issues are consequential because they limit confidence that the observed transfer is robust, causally understood, and likely to generalize beyond the tested Qwen/Gemma settings.
Questions for authors
-
In the RQ5 text-centric staged recipe, how do you use compute- and data-matched baselines to isolate the respective contributions of text-only reasoning training and the subsequent reduced-data native audio-visual refinement? Could you add ablations over stage order, refinement-data size, and total training budget to rule out the possibility that the final gains arise merely from additional training or from native audio-visual data rather than from the proposed text-centric arrangement?
-
Your evidence for partial decoupling mainly comes from local first-order gradient projections at the initial checkpoint. To what extent does this local analysis predict the dynamics of full SFT+GRPO training? Can you provide training-time measures of gradient, representation, or loss coupling—or layer-/module-level analyses—to explain why text-only training improves reasoning while causing perceptual degradation?
-
Since the synthetic data are generated entirely by a text-only LLM, how do you ensure that their task distribution, difficulty, and evidence structure match those of real audio-visual reasoning? How robust is this text-only transfer in unseen domains, longer videos, and other model families, and can the perceptual degradation be reliably recovered through reduced-data native audio-visual refinement?