acceptodds
Under review as a conference paper at ICLR 2027

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

17 views

Abstract

Recent advances have enabled unified omni-modal models in understanding audio, vision, and language. However, existing benchmarks, training data, and learning methods largely treat the modalities independently, leaving the capability of audio-visual joint reasoning poorly evaluated and insufficiently elicited. We address this gap with a benchmark, data engine, and learning method. First, we introduce OmniReasoningBench, a benchmark where both audio and visual evidence are indispensable. It comprises 1,150 multiple-choice and open-ended questions across two tasks, reasoning over video and reasoning beyond video. Second, we develop a data engine OmniQA. It automatically constructs evidence-grounded QA pairs that explicitly necessitate audio-visual joint reasoning, together with time-stamped clue chains that guide the annotation of thinking process. Besides our benchmark, this engine produces training data OmniReasoning-SFT-112K and OmniReasoning-RL-19K. Finally, we propose an on-policy self-distillation method Modality-Factored Self-Distillation (MFSD). It evaluates each sampled response under modality-specific clue contexts, disentangling the contributions of individual clues and their cross-modal interactions for token-level credit assignment. With our training data and learning method, our model OmniReasoning-30B-A3B achieves 50.0% on OmniVideoBench and 42.5% on OmniReasoningBench, improving the base model Qwen3-Omni-30B-A3B-Thinking by 12.8 and 9.3 percentage points, respectively. Moreover, it delivers substantial gains on general and long-video benchmarks, including Video-MME-v2. We hope our work offers a solid step for facilitating future research in omni-modal joint reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Reinforcement Learning, Omni Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EPhWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OmniReasoning, a framework for evaluating and improving audio-visual joint reasoning in omni-modal large language models (Omni-LLMs). Three main contributions:

  1. OmniReasoningBench: 1,150 questions across 19 task types, built so both audio and visual evidence are genuinely required;
  2. OmniQA: a data engine for evidence-grounded QA pairs with timestamped audio-visual clue chains (112K SFT, 19K human-validated RL);
  3. Modality-Factored Self-Distillation (MFSD): an RL method that assigns credit by modality, explicitly rewarding non-additive cross-modal interactions.

The authors show OmniReasoningBench actually requires joint reasoning. Visual-only accuracy falls to 17.2% MCQ / 8.7% OE versus 42.7% / 41.9% with joint input. Their model (OmniReasoning-30B-A3B) gains +9.3 points overall on the benchmark and +6.1–12.8 points across four public video benchmarks over the base Qwen3-Omni model.

Critical strengths

The biggest contribution here is just proving the benchmark does what it claims. The multi-gate verification pipeline (clue validity + shortcut removal under four modality conditions) plus the Table 6 analysis gives real evidence that OmniReasoningBench isn't gameable on a single modality. The contrast with prior benchmarks is stark. If the rest of the paper were weaker, this alone would probably be worth publishing.

Table 3 is the most important result table in the paper. GRPO and RLSD both gain on OmniReasoningBench but lose ground on general video benchmarks (OmniVideoBench drops from 47.0 to 44.8/45.4). MFSD is the only method that moves both up at the same time. The formulation is also solid.

Critical weaknesses

The paper's central argument is about improving joint reasoning. But the gains on open-ended questions are basically flat (+0.7 pts on RbV OE). OmniReasoning-30B-A3B (42.5%) also sits ~15 points below Gemini-3.7-Flash (57.6%). If the method primarily helps with MCQ discrimination rather than generative reasoning, the "joint reasoning" framing is doing more work than the results support.

OmniQA uses Gemini-3.1-Pro for event segmentation and Qwen3.8 for question generation. The SFT-112K dataset has synthetically generated reasoning chains the paper acknowledges may be noisy or hallucinated. The RL-19K set needs human validation. It's not clear how much of the improvement is from the MFSD method versus knowledge implicitly distilled from stronger proprietary models. Reproducibility also takes a hit.

Questions for authors

Q1: Table 6 breaks down modality ablations for Qwen3.5-Omni-Plus and Gemini but not for OmniReasoning-30B-A3B. What happens when you run the proposed model on audio only? If audio-only performance jumped substantially after MFSD training, that would suggest the method is improving audio processing rather than cross-modal integration.

Q2: Keeping encoders frozen throughout SFT and RL is a significant constraint. Has any experimentation been done with partially unfreezing them, especially during RL? Given MFSD explicitly targets token-level cross-modal interaction, it seems like encoder adaptation could either amplify or expose what's actually going on.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer EPhW (1/2)

We sincerely appreciate your constructive feedback and valuable suggestions.

Weakness 1: The gains on open-ended questions are basically flat (+0.7 pts on RBV OE*), and OmniReasoning-30B-A3B (42.5%) sits ~15 points below Gemini-3.7-Flash (57.6%).***

We acknowledge that the gain on open-ended questions is limited, especially on RBV-OE (+0.7). However, this reflects the scarcity of open-ended training data, instead of a lack of joint reasoning. Open-ended RBV data is hard to construct, so it is much scarcer than MCQ data in our training set, and we will make scaling up open-ended data a key direction for future work.

ModelRoV-MCQRoV-OERBV-MCQRBV-OEOverall
Qwen3-Omni-30B-A3B-Thinking31.925.946.43233.2
OmniReasoning-30B-A3B (ours)48.0 (+16.1)30.1 (+4.2)58.8 (+12.4)32.7 (+0.7)42.5 (+9.3)

This does not contradict our central claim, for three reasons:

  1. MCQ requires joint reasoning. Each ROV question has 8 options (chance: 12.5%) and is built on a multi-hop evidence chain. The visual-only Qwen3.5-Plus reaches only 17.2% on RoV-MCQ, and frontier Omni-LLMs given a single modality reach only 14.9–26.1% (Table 6), so the +16.1-point gain on RoV-MCQ cannot come from eliminating options with one modality or with text priors.
  2. The gains are not from single-modality understanding**.** Audio-only and visual-only accuracy rise by at most +2.4 points after MFSD, whereas joint-input accuracy rises by +10.1 (MCQ) and +7.7 (OE).
  3. The gap to Gemini-3.7-Flash is expected. Our model is an open 30B-A3B MoE with about 3B active parameters, and we do not claim to close the gap to frontier proprietary models. Among models of similar scale, it leads clearly (42.5, versus 33.2 for its base model and 25.4 for Nemotron-3-Nano-Omni).

Weakness 2: It is not clear how much of the improvement comes from the MFSD method versus knowledge implicitly distilled from stronger proprietary models.

Both the high-quality OmniReasoning data and the MFSD method contribute, and the existing ablation in Table 4 of the main paper already separates the two:

MethodOmniReasoningBenchOmniVideoBenchLVOmniBenchVideo-MME-v2Video-MMMU
Qwen3-Omni-30B-A3B-Thinking33.237.237.736.568.3
+ SFT (SFT-112K)36.14740.240.373.7
+ GRPO (RL-19K)38.344.839.839.172.9
+ RLSD (RL-19K)39.245.440.139.872
+ MFSD (RL-19K, ours)42.55044.74474.4

We have three observations:

  1. Data effect. SFT on OmniReasoning-SFT-112K is the only stage through which knowledge from proprietary models could enter, and it improves the base model on all five benchmarks (+2.9 / +9.8 / +2.5 / +3.8 / +5.4). We attribute this gain to the OmniQA data.
  2. Method effect. GRPO, RLSD, and MFSD start from the same SFT checkpoint with the same RL-19K data and budget, so their differences come from the learning signal alone. MFSD improves over SFT by a further +6.4 / +3.0 / +4.5 / +3.7 / +0.7, and exceeds RLSD by +3.3 / +4.6 / +4.6 / +4.2 / +2.4 and GRPO by +4.2 / +5.2 / +4.9 / +4.9 / +1.5.
  3. Same data, opposite outcomes. GRPO and RLSD, trained on exactly the same data, fall below SFT on all four public benchmarks (e.g., OmniVideoBench: 47.0 → 44.8 / 45.4), while MFSD is the only method that improves on all five. Better data alone therefore does not explain MFSD's gains.
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer EPhW (2/2)

Question 1: What happens when you run the proposed model on audio only?

MFSD mainly improves the combination of the two modalities rather than audio processing, and the additional experiment you suggested confirms this. Following the protocol of Table 6, we evaluated the SFT checkpoint and OmniReasoning-30B-A3B (+MFSD) on reasoning over video under joint (A+V), audio-only, and visual-only inputs:

ModelTypeA+VAudio-onlyVisual-only
SFT checkpointMCQ37.8724.2719.73
OmniReasoning-30B-A3B (+MFSD)MCQ48.00 (+10.13)26.67 (+2.40)20.80 (+1.07)
SFT checkpointOE22.49.63.47
OmniReasoning-30B-A3B (+MFSD)OE30.13 (+7.73)10.67 (+1.07)3.73 (+0.26)

We have two observations:

  1. Audio-only accuracy does not decrease**.** After MFSD, audio-only accuracy rises by only +2.40 (MCQ) and +1.07 (OE), and visual-only accuracy by only +1.07 and +0.26. Our audio-only MCQ accuracy (26.67) is at the same level as frontier Omni-LLMs in Table 6 (20.53–26.13).
  2. Joint-input accuracy rises far more. It rises by +10.13 (MCQ) and +7.73 (OE), about 4× and 7× the largest single-modality gain, so the improvement lies in combining the two modalities rather than in processing either one alone.

Question 2: Has any experimentation been done with partially unfreezing the encoders, especially during RL?

Freezing or unfreezing the encoders has little effect on our results, because our training targets at cross-modal, semantic, joint reasoning and thus provides the encoders no additional visual-level & audio-level supervision. Visual supervision would arise only if joint reasoning relied on low-level visual details such as clothing texture or depth, which our task does not involve, and the ViT and AuT of Qwen3-Omni have already been well trained on large amounts of data. To verify this, we unfroze the ViT and AuT and re-ran MFSD with everything else identical (same SFT checkpoint, data, and budget):

SettingOmniReasoningBenchOmniVideoBenchLVOmniBenchVideo-MME-v2Video-MMMU
MFSD, encoders frozen42.55044.74474.4
MFSD, encoders unfrozen41.948.944.343.774.8

We have two observations:

  1. Unfreezing brings no further gain. The two settings differ by at most 1.1 points on any benchmark, and the frozen setting is better on four of the five.
  2. The gains come from cross-modal integration in the language model. Unfreezing the encoders neither amplifies nor removes the improvement of MFSD, which suggests that the bottleneck lies in how the language model integrates the audio and visual evidence, exactly what MFSD targets.

If our responses already address your concerns, we hope that you could consider raising the score. Thank you again for your positive assessment and helpful suggestions!

Dear Reviewer,

The discussion period will close in less than 24 hours. We would greatly appreciate any remaining thoughts or follow-up on our responses before the deadline. We remain fully open to discussion and are happy to clarify any points or provide additional details if helpful. Thank you again for your time and thoughtful review.

Reviewer_4wbAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The authors introduce a benchmark for audio-visual reasoning that evaluates reasoning over video by ensuring there is a tangible benefit to integrating both modalities, rather than relying on either one alone. It also evaluates reasoning beyond video, which requires contextualizing the information acquired from a video to answer a separate question that may involve an additional figure. Additionally, they create a synthetically generated dataset containing both SFT and RL data to train an audio-visual model. Finally, they extend RLSD to a cross-modal setting to train a model built on top of Qwen3-Omni-30B, yielding an improvement of about 10 points on their proposed benchmark, as well as an average improvement of ~8.4 points on standard audio-visual benchmarks.

Critical strengths
  • The paper identifies a known limitation in existing multimodal benchmarks, where a single modality is sufficient for solving the query. Their benchmark addresses this issue by chaining evidence across modalities and screening samples for shortcuts in either modality.
  • They also target this limitation in training by proposing MFSD, which ensures that the model gets rewarded for connecting the different modalities. MFSD shows consistent and significant gains over RLSD, motivating the method (Table 3).
  • Their model shows significant improvements over the baseline and similarly sized models, and competitive performance compared to proprietary or much larger models.
Critical weaknesses
  • The reasoning beyond video benchmark is interesting, even though it does not seem to be as challenging as reasoning over video for most models (Table 1).
  • The human validation procedure is only mentioned briefly. It would be worth discussing it in more detail.
  • There is limited discussion of the computational overhead of MFSD. If I am understanding correctly, it should require additional forward passes for each of the clue contexts.
Questions for authors
  • Why is MiMo a proprietary model in Table 2 but an open-source model in Table 1?
  • Why is the beyond video benchmark not as challenging for models? It seems a little counterintuitive to me.
  • What exactly did the human validation entail for the RL data? Was there human validation for the benchmark?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer 4wbA (1/2)

We sincerely appreciate your constructive feedback and valuable suggestions.

Weakness 1 & Question 2: The reasoning beyond video benchmark does not seem to be as challenging as reasoning over video for most models (Table 1). Why is it not as challenging?

These two settings are challenging in different ways, and RBV scores higher mainly because it does not depend on the long-range, multi-hop dependencies that ROV requires. In ROV, the model must locate and bind several pieces of evidence across modalities and events in long videos without any textual help, and the answer is a fine-grained detail hidden in the video. In RBV, the model only needs to acquire one piece of knowledge (a rule, value, or procedure) from the audio-visual stream. And then the new scenario is given and the remaining reasoning is domain reasoning, at which strong LLMs excel. So part of the difficulty shifts from long-range, fine-grained grounding to reasoning.

Weakness 2 & Question 3: What exactly did the human validation entail for the RL data? Was there human validation for the benchmark?

Both OmniReasoningBench and OmniReasoning-RL-19K were fully validated by humans under the same protocol, and we report the cost as follows: 40+ annotators worked for about two months at a cost of tens of thousands of US dollars. The annotators were split into two independent groups, and the data was split into two batches. The procedure has two rounds:

  1. Round 1: validation, verification, and revision. Each group took one batch and, for every item, watched the full video with audio and checked four criteria:
    1. answer correctness: the gold answer is correct according to the video;
    2. answer uniqueness: exactly one option is correct;
    3. clue grounding: every timestamped audio and visual clue is present in the video at the stated time;
    4. cross-modal link: the link between the audio and visual evidence in the reasoning chain holds.
  2. Defective items were revised or discarded.
  3. Round 2: cross-validation. The two groups swapped batches and re-checked the other group's items against the same criteria, so every item was examined by two independent groups. Only items that passed cross-validation entered the final benchmark and RL data.

We will add this information in the future version.

AuthorsOpenReview ↗

Official Comment by Authors for Reviewer 4wbA (2/2)

Weakness 3: Limited discussion of the computational overhead of MFSD.

Your understanding is correct that MFSD scores each sampled response under three additional clue contexts (audio, visual, and both), but the overhead is moderate:

  1. Prefill only, no extra rollouts. The extra passes are teacher-forced forward passes over responses that have already been sampled, so no extra tokens are generated and no gradients flow through the scoring views.
  2. Only two more passes than RLSD. The no-clue view is already computed by every policy-gradient method, and RLSD adds one view (joint clues), while MFSD adds three.
  3. No separate teacher model. All views share the actor's weights, so no extra model needs to be loaded.
  4. Generation dominates the step time. As a result, most of each RL step is still spent on autoregressively generating long thinking trajectories, and the extra prefill passes add only a small fraction.

Measured on our setup:

MethodExtra scoring passes
GRPO0
RLSD1
MFSD3

Question 1: Why is MiMo a proprietary model in Table 2 but an open-source model in Table 1?

MiMo-V2.5 is an open-source model, and listing it as proprietary in Table 2 was a typo that we will correct in the revised version.

If our responses already address your concerns, we hope that you could consider raising the score. Thank you again for your positive assessment and helpful suggestions!

Dear Reviewer,

The discussion period will close in less than 24 hours. We would greatly appreciate any remaining thoughts or follow-up on our responses before the deadline. We remain fully open to discussion and are happy to clarify any points or provide additional details if helpful. Thank you again for your time and thoughtful review.

Thank you for your response and clarifications. I will maintain my positive rating and update my soundness score.

Reviewer_3L48OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates audio-visual joint reasoning and introduces OmniReasoningBench, a benchmark covering reasoning over and beyond videos. The authors also develop OmniQA, an automated data-generation pipeline with evidence verification, and propose Modality-Factored Self-Distillation (MFSD), which uses modality-conditioned token likelihoods to guide RL training. Experiments show improvements over the base model and standard RL baselines across several multimodal benchmarks.

Critical strengths
  1. The paper highlights the limitations of existing multimodal benchmarks and attempts to evaluate reasoning that requires complementary audio and visual evidence.
  2. The paper proposes a scalable data-generation pipeline.
  3. The RL algorithm MFSD consistently outperforms GRPO and RLSD across the reported benchmarks.
Critical weaknesses
  1. Reasoning beyond video appears to conflate audio-visual joint reasoning with retrieving knowledge from one video modality and applying it to an external image or scenario. Its inclusion weakens the construct validity of the proposed benchmark.
  2. The pipeline relies heavily on Gemini-generated captions, which may contain fine-grained perceptual or temporal errors. The paper provides limited quantitative evidence of annotation accuracy, particularly for cross-modal relationships and temporal grounding.
  3. The implementation and reliability of the media verifier are unclear. In particular, verifying individual clues does not necessarily establish that their cross-modal relationships are correct, and single-modality model failures do not prove that the corresponding information is insufficient.
  4. Non-additive token likelihood gains do not necessarily indicate correct cross-modal evidence integration. Without direct validation of this assumption, it remains unclear whether MFSD improves joint reasoning or simply exploits statistical interactions between textual clues.
Questions for authors
  1. The event segmentation procedure, a foundational component of OmniQA, is largely unspecified.
  2. Why do GRPO and RLSD degrade performance relative to SFT on four general benchmarks in Table 3? What evidence supports the proposed explanation?
  3. Given the heavy reliance on text-generated reasoning traces and textual privileged clues, how can the authors distinguish improvements in audio-visual grounding from improvements in text-based reasoning? Would a caption-only baseline help clarify this?
6 comments
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer 3L48 (1/4)

We sincerely appreciate your constructive feedback and valuable suggestions.

Weakness 1: Reasoning beyond video appears to conflate audio-visual joint reasoning with retrieving knowledge from one video modality and applying it to an external image or scenario.

Reasoning beyond video (RBV) is not a retrieval task but a dimension orthogonal to reasoning over video (RoV): RoV asks whether the model can understand the video, while RBV asks what it can do with that understanding in a new scenario.

An RBV question therefore requires audio-visual joint reasoning on the current video in the first place. For example, the rules and values in lectures, tutorials, and demonstrations are conveyed jointly by speech, demonstration, and on-screen content. The extracted knowledge must then be combined with information that exists only in the new figure or text. This argument is supported by our experiments (Table 1): within the same model family, adding audio raises RBV accuracy from 46.0 to 60.4 on MCQ and from 30.0 to 40.7 on OE (Qwen3.5-Plus vs. Qwen3.5-Omni-Plus), and our omni-modal training raises RBV-MCQ from 46.4 to 58.8. If RBV could be solved from a single modality, access to audio would not make such a difference on RBV.

Weakness 2: Reliance on Gemini captions; limited quantitative evidence of annotation accuracy.

Yes, the captions contain hallucinations; however, they were not included into the final labels. This is because in OmniQA, captions serve only as a proposal source, and every piece of caption content that a question depends on is re-verified against the raw media before the question is accepted. Specifically, each decisive clue is cut from the video around its own timestamp (±2 s, at most 45 s) and checked on the muted clip for visual clues and on the audio clip for audio clues, so a clue with a wrong timestamp fails verification, and every quote must additionally appear verbatim in the captions of its claimed modality. Each reasoning step and gold answer is then checked for correctness and uniqueness, and defective items are repaired or rejected.

On top of this verification, every item in RL-19K and in the benchmark was reviewed by 40+ annotators who watched the full video with audio, which took about two months and cost tens of thousands of US dollars.

Consequently, even though the captions are noisy, the final training data is already high-quality, which the experiments have validated: In Table 3, SFT on this data improves the base model on all five benchmarks (+2.9 / +9.8 / +2.5 / +3.8 / +5.4). As for a direct quantitative evaluation of caption accuracy, one could run our open-source pipeline on captioning benchmarks such as Video-MME, or conduct a human study of caption quality. But both are costly and beyond our current resources. So we could include them in a future version.

Weakness 3: Implementation of the media verifier; per-clue checks vs. cross-modal relations; single-modality failure vs. insufficiency.

We partially agree with the reviewer: converting audio and visual content into text cannot guarantee that every cross-modal relation is 100% correct. Nevertheless, with the surrounding context, the correspondence between the two modalities is usually clear, and our procedure thus can cover most cases. Concretely, clips are cut with ffmpeg (visual clips sampled at 2 fps) and judged by frontier models with a strict prompt. This prompt tells them the claim may come from a hallucinated caption and must be judged from the clip only. A single failed clue will reject the question. Regarding your two specific questions:

  1. Per-clue checks vs. cross-modal relations. A cross-modal link is allowed in only three programmatically checked forms:
    1. (a) a shared identifier that appears in both quotes;
    2. (b) co-temporal clues;
    3. (c) a caption entry that states the relation explicitly.
  2. For (a) and (b), once the audio clue is verified on the audio and the visual clue on the visual stream, the link is fully determined by the content and timestamps of the two clues, and for (c) the relation is verified directly on the clip with audio. The modality-bias gate then checks the link functionally: a solver given both modalities must derive the gold answer, while solvers given either modality alone must fail. Table 6 confirms this, where joint inputs exceed the single-modality oracle by +9.9 to +26.7 points.
  3. Single-modality failure reflects insufficiency, not perception errors. First, by construction, the anchor fact must be exclusive to one modality and the answer fact to the other (e.g., an audio-only fact may not also appear as on-screen text). Second, in Table 6 the same model answers correctly with joint input but reaches only 14.9–26.1% on 8-way MCQ (chance: 12.5%) and 6.9–14.7% on open-ended questions with a single modality, so the failure largely comes from missing modality information.
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer 3L48 (2/4)

Weakness 4: Non-additive likelihood gains may not indicate correct cross-modal integration.

We first define audio-visual joint reasoning: reasoning that uses the information of both modalities simultaneously, where such information cannot be obtained from individual modality. Under this definition, the results on both our benchmark and the public benchmarks show that MFSD indeed promotes audio-visual joint reasoning. Under the same SFT checkpoint, data, and budget, adding the interaction term to RLSD yields +3.3 / +4.6 / +4.6 / +4.2 / +2.4 on OmniReasoningBench, OmniVideoBench, LVOmniBench, Video-MME-v2, and Video-MMMU. Note that MFSD is the only method that improves on all of them.

Among the correct rollouts, the share that uses audio-visual joint reasoning rises from 46.2% to 63.1% under MFSD. Under GRPO it falls from 47.4% to 44.6%, and under RLSD it falls from 48.8% to 46.7%. MFSD therefore improves joint reasoning itself rather than exploiting statistics of textual clues.

Step0255075100125150
GRPO47.40%47.00%46.60%45.10%45.80%43.90%44.60%
RLSD48.80%50.60%50.50%49.50%49.00%47.50%46.70%
MFSD46.20%55.60%55.40%59.10%64.00%62.50%63.10%

As for the statistical interactions you mention, the interaction is between verified audio and verified visual evidence from the same video (rAr_A holds only audio evidence, rVr_V only visual evidence), and non-additivity is exactly the signature of complementary evidence: StS_t is close to 0 if either clue alone explains a token, and positive only when both support it jointly. Moreover, correctness and credit are decoupled, because the outcome verifier decides correctness while StS_t only redistributes credit within a response, so it cannot cause an incorrect response to be rewarded.

We acknowledge that text cannot fully cover some modality information, such as timbre and part of the tone, but this is out of our current scope and we will address it in future work.

AuthorsOpenReview ↗

Official Comment by Authors for Reviewer 3L48 (3/4)

Question 1: The event segmentation procedure is largely unspecified.

Thank you for pointing this out. Segmentation and captioning in OmniQA proceed in two stages. Both use Gemini-3.1-Pro with high media resolution and high thinking effort. We now describe the details as follows.

Stage 1: Whole-video event planning.

  1. Input. Videos without an audio stream are discarded. The full video is re-encoded at 2 fps, with a timestamp burned into every frame
  2. Output. A video summary, a specific video type (e.g., "NBA game highlight analysis" rather than "sports"), and a list of events with global start and end times.
  3. Segmentation rules in the prompt. (i) Boundaries are placed at semantic event changes, not at fixed windows. (ii) Each event summary must state its key visual and audio evidence, so that both streams inform where a boundary lies. (iii) Each event lasts at most 3 minutes; a longer semantic event is split at natural sub-boundaries.
  4. Post-processing. Timestamps are parsed and clamped to the video duration. An invalid end time falls back to an even split. Any segment longer than 180 s is split deterministically into consecutive chunks of at most 180 s. If no valid event is returned, the first 180 s form one segment.
  5. RBV videos. The plan additionally lists the problems, examples, or cases solved in the video, with their statements and final answers.

Stage 2: Event-level audio-visual captioning. Each event is captioned separately with three inputs: the video, the event's global time range, and the captions of the preceding events. The preceding captions serve only to resolve references, so that recurring people, objects, and sound sources keep the same identifiers across events. Three outputs are produced:

  1. A global caption that integrates what is seen and heard in temporal order.
  2. Audio-only captions, split whenever the speech, speaker, music, sound, or silence changes. Each item is typed (speech / music / sound / background / silence / unclear) and starts with a specific source (e.g., "the host in the blue shirt says ..."). Speech is transcribed verbatim and split into 3–6 s phrase chunks, with no item longer than 10 s.
  3. Visual-only captions, split whenever the shot, scene, subject, action, on-screen text, or layout changes. Each item records exact counts, precise colors, spatial layout, and verbatim on-screen text.

All items use global timestamps. An event whose output lacks timestamped audio or visual items is re-captioned.

Question 2: Why do GRPO and RLSD degrade relative to SFT on the four general benchmarks? What evidence supports the explanation?

This is a good observation. GRPO and RLSD fall below SFT on the four general benchmarks (OmniVideoBench, LVOmniBench, Video-MME-v2, and Video-MMMU) because our SFT data primarily covers the pattern of audio-visual joint reasoning, whereas neither algorithm encourages this pattern in RL. So these RL methods drift away from what SFT stage has learned. During rollouts, both often reach correct answers through shortcuts such as answering from the single modality, eliminating the options by priors, or guessing a plausible value, and the outcome reward cannot tell these apart from genuine joint reasoning. The three algorithms differ in how they treat such shortcut tokens:

  1. GRPO gives every token of a correct rollout the same credit, so shortcut tokens are reinforced as strongly as integrative ones.
  2. RLSD weights tokens by the joint-clue gain, which is also high for tokens supported by a single clue, so it cannot discourage the shortcut either.
  3. MFSD gives extra credit only to tokens whose support exceeds what either modality provides alone, and relatively down-weights single-modality tokens.

To verify this quantitatively, we track the correct rollouts in each training batch (global batch size 256) every 25 steps over the first 150 steps, and an LLM judge (e.g. DeepSeek-V4.1-Flash) reads each thinking trace to decide whether the answer is reached through audio-visual joint reasoning. The table reports the share of joint-reasoning rollouts among correct rollouts:

Step0255075100125150
GRPO47.40%47.00%46.60%45.10%45.80%43.90%44.60%
RLSD48.80%50.60%50.50%49.50%49.00%47.50%46.70%
MFSD46.20%55.60%55.40%59.10%64.00%62.50%63.10%

The share fluctuates downward with GRPO (47.4% → 44.6%) and RLSD (48.8% → 46.7%) but rises markedly with MFSD (46.2% → 63.1%). These dynamics match the benchmark results: GRPO and RLSD fall below SFT (e.g., OmniVideoBench: 47.0 → 44.8 / 45.4), whereas MFSD improves over SFT on all of them (OmniVideoBench 50.0, LVOmniBench 44.7, Video-MME-v2 44.0, Video-MMMU 74.4).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (2/2)

(III) Key Improvements Made During Rebuttal

  1. Evidence that MFSD Improves Joint Reasoning (Reviewer 3L48)

We tracked the share of audio-visual joint reasoning rollouts during RL training, showing that it rises markedly under MFSD but declines under GRPO and RLSD.

  1. Analysis of RL Baselines Falling Below SFT (Reviewer 3L48)

We explained why GRPO and RLSD degrade relative to SFT on general benchmarks and provided quantitative evidence for this explanation.

  1. Caption-Only and Question-Only Baselines (Reviewer 3L48)

We added these baselines to separate the gains from text-based reasoning and from audio-visual grounding.

  1. Single-Modality Evaluation of Our Model (Reviewer EPhW)

We evaluated the SFT and MFSD checkpoints under audio-only and visual-only inputs, showing that the gains come from combining the modalities rather than processing either one alone.

  1. AV-Oracle Analysis for Our Model (Reviewer EPhW)

We extended the Table 6 oracle analysis to the SFT and MFSD checkpoints, showing that MFSD substantially enlarges the margin of joint input over the single-modality oracle.

  1. Disentangling Data and Method Contributions (Reviewer EPhW)

We clarified, with controlled comparisons under the same SFT checkpoint, data, and budget, how much of the improvement comes from the data and how much from MFSD.

  1. Ablation of Encoder Unfreezing (Reviewer EPhW)

We conducted experiments that unfreeze the audio and visual encoders during MFSD, confirming that the gains come from cross-modal integration in the language model.

  1. Discussion of Open-Ended Gains and the Gap to Frontier Models (Reviewer EPhW)

We analyzed the source of the smaller open-ended gains and positioned our model against models of similar scale.

  1. Validity and Difficulty of Reasoning Beyond Video (Reviewers 3L48, 4wbA)

We clarified why reasoning beyond video requires audio-visual joint reasoning and why it shows different difficulty from reasoning over video.

  1. Annotation Quality and Media Verifier (Reviewer 3L48)

We detailed how captions are re-verified against raw video, how cross-modal links are checked, and why single-modality failure reflects missing information.

  1. Human Validation Protocol (Reviewers 3L48, 4wbA)

We described the two-round human validation and cross-validation procedure for both the benchmark and the RL data.

  1. Computational Overhead of MFSD (Reviewer 4wbA)

We analyzed the additional cost of MFSD compared with GRPO and RLSD.

  1. Details of Event Segmentation and Captioning (Reviewer 3L48)

We provided a detailed walkthrough of the two-stage segmentation and captioning procedure in OmniQA.

  1. Correction of Model Labeling (Reviewer 4wbA)

We clarified the open-source status of MiMo-V2.5 and corrected the typo in Table 2.

(IV) Reviewer Endorsement

All three reviewers keep their positive ratings (Weak Accept) after the discussion. Reviewer 4wbA further updated their soundness score, and Reviewer EPhW confirmed that our new single-modality experiments addressed their question, after which we provided the follow-up AV-oracle analysis they requested.

We will incorporate all the new experiments, analyses, and implementation details into the revised version, mainly in the experiments section and the Appendix. We hope this summary facilitates your evaluation.

Best regards,

The Authors

AuthorsOpenReview ↗

General Response (1/2)

Dear AC, PCs, and Reviewers,

Thank you for your time and effort in handling our submission and for the constructive discussion. To help you assess our work efficiently, we provide a concise summary of our contributions, the reviewers' evaluation, and the key improvements made during the rebuttal.

(I) Main Contributions

This paper studies audio-visual joint reasoning in omni-modal LLMs. This capability is poorly evaluated and insufficiently elicited because existing benchmarks, training data, and learning methods largely treat the modalities independently. We address the gap with three components:

  1. OmniReasoningBench. 1,150 multiple-choice and open-ended questions across two tasks: reasoning over video and reasoning beyond video. Both audio and visual evidence are indispensable, and multi-gate verification (clue validity plus shortcut removal under four modality conditions) and human cross-validation ensure this.
  2. OmniQA data engine. It automatically constructs evidence-grounded QA pairs with time-stamped audio-visual clue chains. It produces OmniReasoning-SFT-112K and the fully human-validated OmniReasoning-RL-19K.
  3. Modality-Factored Self-Distillation (MFSD). An on-policy self-distillation method. It scores each sampled response under modality-specific clue contexts and gives token-level credit to the non-additive cross-modal interaction, so it rewards exactly the tokens that need both modalities.

Our model, OmniReasoning-30B-A3B, improves its base model Qwen3-Omni-30B-A3B-Thinking by +9.3 on OmniReasoningBench and by +12.8 on OmniVideoBench, with consistent gains on LVOmniBench, Video-MME-v2, and Video-MMMU. MFSD is the only RL method that improves over the SFT checkpoint on all five benchmarks.

(II) Strengths Recognized by Reviewers

We are glad that reviewers have recognized the strengths of our work:

  • Rigorous benchmark that genuinely requires both audio and visual evidence (@3L48, 4wbA, EPhW)
  • Effective and well-motivated MFSD, consistently outperforming GRPO and RLSD (@3L48, 4wbA, EPhW)
  • Scalable data pipeline OmniQA (@3L48, 4wbA)
  • Strong empirical results over the base model and similarly sized models (@4wbA, EPhW)
  • Clear and easy-to-follow presentation (@3L48, 4wbA, EPhW)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.