acceptodds
Under review as a conference paper at ICLR 2027

Text-Centric Post-Training for Omni-Modal Reasoning

Abstract

Improving joint audio-visual reasoning in Omni Large Language Models typically incurs substantial data construction and training costs. Our diagnostics reveal multi-hop reasoning difficulties despite correct answers to all corresponding single-hop questions and suggest partial decoupling in the local optimization of perception and reasoning objectives. This motivates post-training with different emphases on these capabilities. Text-only reasoning training yields gains across data sources, model scales, and families. With the best-performing text-only configuration, supervised fine-tuning followed by reinforcement learning (RL) raises Qwen2.5-Omni-7B's geometric mean of nine reasoning scores by 25.83% over the base model, outperforming the complete native audio-visual route with 56.6% fewer GPU-hours. Training on data synthesized entirely by a text-only LLM raises this geometric mean by 21.01% without audio-visual data in construction or training. However, text-only training degrades perception. We therefore propose a text-centric post-training paradigm: text-only training provides the main reasoning optimization, and reduced-data native audio-visual RL then refines perception. Refinement uses about 90% fewer input tokens than full-data audio-visual RL, restores perception above the base level, and retains 93.5% of the best-performing text-only pipeline's reasoning gain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Omni-modal large language models, audio-visual reasoning, cross-modal transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sDegOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper frames text-centric post-training for Omni-LLMs as strengthening the model’s reasoning core on purely textual chain-of-thought and verifiable multiple-choice data, then transferring that capacity to audio-visual reasoning. A diagnostic study reveals a marked perception–reasoning gap, and gradient-projection evidence suggests that perception and reasoning objectives are only partially coupled, motivating separate training emphases. The recipe applies SFT with script-based CoT and GRPO with script-based MCQs, updates only the language/Thinker decoder via LoRA while freezing audio-visual encoders and projection modules, and can rely on synthetic fictional scripts, questions, and rationales generated entirely by a text-only LLM to train entity tracking, event-state updates, cross-segment evidence integration, and multi-hop composition. This text-only stage improves omni-modal reasoning with lower training compute but can degrade perception, so the authors propose a staged text-centric paradigm: first strengthen reasoning in text, then use reduced native audio-visual data to restore perceptual ability while retaining the reasoning gains. Validation across multiple Omni-LLM families, reasoning and perception benchmarks, synthetic-data settings, and an agentic case study supports the transfer claim and the compute–perception trade-off.

Critical strengths

The paper’s strongest acceptance case is its diagnostic-to-intervention design: it first establishes a perception–reasoning gap across multiple Omni-LLMs and supports partial decoupling with a local gradient-projection analysis, then uses those findings to motivate and structure a systematic five-RQ study. This is consequential because it makes the text-only training proposal feel evidence-driven rather than merely pragmatic, and it gives the reader a clear causal story for why text-centric post-training should transfer. A second key strength is the breadth and practicality of the validation: gains are shown across model scales/families and multiple reasoning benchmarks, with compute accounting, synthetic text-only data, and a staged recipe that explicitly addresses perceptual degradation. This matters for acceptance because it turns a narrow training trick into a potentially scalable post-training paradigm with a credible compute–perception trade-off.

Critical weaknesses

The most consequential weakness is that the central promise of the staged text-centric paradigm—retaining the main reasoning gains while restoring perception—is not supported by a sufficiently clean, budget-matched comparison and ablation of stage order, reduced-data size, initialization, and compute. Without that, it is hard to rule out that the final gains come simply from additional training or native audio-visual data rather than from the proposed text-centric arrangement, which weakens the paper’s main practical claim. A second weakness concerns mechanism and external validity: the decoupling evidence is a local first-order analysis, the synthetic data depend on an opaque text-only generator, and the benchmark suite may favor text-transferable reasoning while perceptual degradation is acknowledged but not deeply explained. These issues are consequential because they limit confidence that the observed transfer is robust, causally understood, and likely to generalize beyond the tested Qwen/Gemma settings.

Questions for authors
  1. In the RQ5 text-centric staged recipe, how do you use compute- and data-matched baselines to isolate the respective contributions of text-only reasoning training and the subsequent reduced-data native audio-visual refinement? Could you add ablations over stage order, refinement-data size, and total training budget to rule out the possibility that the final gains arise merely from additional training or from native audio-visual data rather than from the proposed text-centric arrangement?

  2. Your evidence for partial decoupling mainly comes from local first-order gradient projections at the initial checkpoint. To what extent does this local analysis predict the dynamics of full SFT+GRPO training? Can you provide training-time measures of gradient, representation, or loss coupling—or layer-/module-level analyses—to explain why text-only training improves reasoning while causing perceptual degradation?

  3. Since the synthetic data are generated entirely by a text-only LLM, how do you ensure that their task distribution, difficulty, and evidence structure match those of real audio-visual reasoning? How robust is this text-only transfer in unseen domains, longer videos, and other model families, and can the perceptual degradation be reliably recovered through reduced-data native audio-visual refinement?

Reviewer_tgwgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main contribution of this paper is to demonstrate that text-only post-training can improve downstream audio-visual reasoning in Omni models that already possess audio-visual capabilities, while reducing training costs. The authors compare text-only and audio-visual SFT–GRPO pipelines and further demonstrate transfer using entirely synthetic text data. An orthogonal gradient projection study suggests partial separability between perception and reasoning objectives in local optimization, motivating a staged training approach. Since text-only training degrades perception, the proposed pipeline follows text-only SFT and GRPO with a small amount of audio-visual GRPO to restore perception while retaining most of the reasoning gains. However, the experiments establish improvements on audio-visual benchmarks without directly demonstrating that text-only training resolves the specific multi-hop reasoning bottleneck identified in the initial diagnostic study.

Critical strengths

A key strength is the diagnostic study contrasting single-hop perception with multi-hop reasoning. By constructing multi-hop questions from facts whose corresponding single-hop questions all evaluated models answer correctly, the authors provide evidence of a gap between extracting audio-visual information and integrating it across reasoning steps. Although related limitations have been observed in text-only settings, this controlled diagnostic offers a compelling motivation for investigating reasoning transfer from text to audio-visual tasks.

Critical weaknesses

The connection between the multi-hop diagnosis and the main results remains incomplete. The initial study identifies failures on multi-hop questions despite correct answers to the corresponding single-hop questions. However, the subsequent evaluation does not isolate improvements on this specific capability. Evaluating the models before and after text-only training on held-out, paired single-hop and multi-hop questions would help establish whether the proposed training actually addresses the bottleneck that motivates the paper. In particular, improvements on examples whose prerequisite single-hop questions remain correctly answered would provide stronger evidence of improved reasoning.

The source of the transfer gains remains insufficiently understood. The reported benchmark improvements do not clearly distinguish better integration of correctly perceived evidence from changes in perception or other aspects of response generation. An error analysis separating failures to extract audio-visual evidence from failures to reason over that evidence would clarify when text-only training is sufficient and when native audio-visual training remains necessary. The gradient projection study provides evidence of local partial separability, but does not directly explain how text-only training produces the observed transfer.

A joint text–audio-visual training baseline is missing. The paper compares separate training routes and sequential refinement, but does not establish whether the proposed staging is preferable to mixing text and audio-visual examples during training. A comparison under matched data and compute budgets would help determine whether the benefits arise from the training order or simply from access to complementary supervision.

The evaluation is broad, but lacks a capability-level breakdown. Although the paper evaluates multiple reasoning and perception benchmarks, the aggregate results do not show how gains vary with reasoning depth, dependence on audio versus vision, or the need to combine both modalities. Such analyses would better establish the scope and limitations of text-to-audio-visual transfer.

Questions for authors
  1. Are the text-only and audio-visual SFT datasets comparable in their reasoning supervision, particularly the quality and multi-hop complexity of their reasoning traces? A comparison using the same questions and reference reasoning traces, with only the input modality changed, would help isolate the effect of text-only training.

  2. The relationship between the gradient projection study and the proposed training pipeline remains unclear. Could the gradient projection method be used for training?

  3. Moreover, Table 5 shows that text-only GRPO substantially recovers the perceptual performance lost during text-only SFT, despite receiving no audio-visual inputs. What accounts for this recovery? Distinguishing changes in evidence extraction, evidence use, and response behavior would help clarify the proposed separation between perception and reasoning.

Reviewer_FVGNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper demonstrates that Omni-LLMs struggle with multi-hop reasoning, even when they can correctly answer the underlying single-hop perception questions.
  • Their analysis reveals that perception and reasoning objectives are partially decoupled in local parameter optimization.
  • Motivated by this finding, the paper explores a two-stage training pipeline combining SFT and GRPO in both text-only and audio-visual setups. While the text-only setup generally improves reasoning performance, it degrades perceptual abilities. To address this, the authors perform the second-stage training combining text-only and audio-visual data to recover perceptual performance.
  • This work is validated on several Omni-LLMs and evaluated on several popular perception and reasoning audio and video benchmarks.
Critical strengths
  • This is a very well-written paper that analyses several important aspects of the proposed approach.
  • The training data cover a wide variety of sources, and the authors conduct a comprehensive evaluation, exploring several baseline models.
  • The experiments on the partial decoupling of perception and reasoning optimization are particularly interesting and align well with the paper's central motivation.
  • Exploring alternative and more efficient training pipelines for improved omni-modal reasoning is an important direction that would be of interest to the broader research community.
  • The study reports detailed compute and token usage for each training stage, which is helpful for understanding the overall training costs.
  • I also appreciate the authors' investigation of perceptual degradation.
Critical weaknesses
  • In the diagnostic evaluation, the analysis jumps from single-hop perception directly to multi-hop reasoning. It would be interesting to also consider two intermediate potential failure modes: (1) single-hop reasoning and (2) multi-hop perception, i.e., perception tasks that require multiple grounding steps across spatio-temporal inputs. This distinction is important because single-hop perception ability does not necessarily translate to single-hop reasoning, other failure modes may stem from multi-grounding and modality bias.

  • The terms text-only and text-centric are used several times in the introduction, but their definitions and the distinction between them are not clear. Please clarify what each term means and how they differ.

  • Table 1: It would be helpful to include additional key details, such as the total number of samples at different stages of training and the minimum, maximum, and average video durations. There are also a few points that need clarification regarding the experimental setup and the fairness of the comparison:

    • The text-only setup uses datasets that are also video-based. Are the videos not provided during training? If so, this needs to be clarified, as otherwise it is unclear how the setup qualifies as text-only. If the videos are used during training, then how is this setup considered text-only?

    • The data sources and training data volumes used for audio-visual training appear to be considerably narrower than those used in the text-only setup. As per table 12, for example, during SFT, the text-only setup uses 263K samples compared to 42K for the audio-visual setup. Similarly, during GRPO, the text-only setup uses 35K samples compared to only 1K for the audio-visual setup. Given this substantial difference in training data volume, I do not think it is fair to claim that "Text-only reasoning training yields broad omni-modal reasoning gains with less training compute than the complete native audio-visual route." A more controlled comparison is needed to support this claim.

  • It is not clear why script-based MCQ yields stronger omni-modal reasoning transfer. Could the authors elaborate on why this setup works best? There are several possible explanations: (1) the training data mixture contains a substantially larger proportion of MCQs, so the gains may primarily stem from data scale; (2) the reward signal during GRPO is much cleaner for MCQs compared to open-ended QA; or (3) the MCQ format allows the model to exploit text biases or spurious correlations in the answer choices, potentially taking shortcuts without genuinely improving its reasoning capabilities. You can potentially evaluate the reasoning quality as well beyond the final answer based evaluation to rule out the option 3. I think it would be valuable to investigate this aspect more deeply. Given that diagnostic evaluation is one of the key focuses of this paper, it would be interesting to conduct a controlled experiment using the same set of samples in both MCQ and open-ended QA formats. This could help disentangle the contributions of data scale, reward quality, and potential shortcuts, and provide a clearer understanding of where the observed gains originate.

  • Given the drop in perceptual performance in the text-only setup, could the authors analyse whether the model is actually arriving at the correct answer by reasoning over the relevant audio-visual evidence? Beyond answer accuracy, it would be interesting to examine the quality of the reasoning traces and whether the model is grounding its reasoning in the correct evidence. Could you also investigate generation and inference efficiency? For instance, if the model can effectively ground its reasoning in the audio-visual evidence, it may be able to arrive at the correct answer much faster, using a shorter reasoning chain, rather than generating excessively long reasoning traces as might be the case in the text-only setup.

Questions for authors

Please see weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.