acceptodds
Under review as a conference paper at ICLR 2027

CogniRoute: Learning to Route Social Evidence in Omni-Modal Models

Abstract

Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation, reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social video QA resource with 118 K structured training examples, grounded reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Modal Language Model, Reinforcement Learning, Audio-Visual Learning, Mix-of-Experts, Temporal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9YsEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CogniRoute, a cognitive schema-guided Mixture-of-Experts (MoE) training framework for multimodal social understanding. The proposed method aims to address the limitation that multimodal models, despite their ability to process visual, audio, and textual information, may fail to identify and utilize the evidence most relevant to a given question.

Critical strengths

The paper proposes an evidence-guided MoE routing framework with meaningful methodological novelty.

The experimental evaluation is comprehensive, and the proposed method achieves substantial improvements in social understanding tasks.

Critical weaknesses

The substantial performance improvements on the proposed benchmark contrast with the relatively modest gains on public benchmarks, raising concerns about the generalization capability of the method.

Questions for authors

CogniRoute achieves substantial performance improvements on OmniSocialBench, whereas the gains on several public audio-video understanding benchmarks are relatively limited. Could the authors further explain this performance discrepancy and provide additional evidence that the proposed evidence-guided routing strategy generalizes beyond OmniSocialBench?

Reviewer_fFWGOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CogniRoute, a two-stage framework for improving how MoE-based omni-modal models select visual, audio, and temporal evidence. Schema-Aligned Predictive Routing aligns expert-routing signatures with training-only labels describing evidence source, reasoning demand, and temporal scope. Route-Aware MoE Reinforcement Learning jointly optimizes token generation and expert selection using rewards for answer correctness, modality consistency, and temporal grounding. The authors also introduce OmniSocialBench, containing 118K training examples and 800 manually verified evaluation examples spanning four social reasoning categories. With Qwen3-Omni-30B-A3B, CogniRoute achieves 59.38% accuracy, compared with 40.13% for standard SFT.

Critical strengths
  1. Meaningful problem. The paper addresses an important limitation of omni-modal models: accessing multiple modalities does not guarantee using the right evidence. SAPR introduces structured, training-only supervision to shape expert routing without additional labels, offering a potentially generalizable approach.
  2. Substantial in-domain gains and informative ablations. CogniRoute substantially improves accuracy on OmniSocialBench. Random or shuffled tag controls and token-only versus joint token–gate RL comparisons suggest that meaningful schema labels and gate optimization both contribute to the improvements.
Critical weaknesses
  1. Insufficient novelty relative to prior routing RL. MoE-GRPO [1] already jointly optimizes token generation and expert routing using GRPO-style objectives. Although CogniRoute introduces task-specific rewards, the methodological advance beyond existing gate-level RL remains unclear. This may weakens the novelty claim.

[1] MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models

Questions for authors
  • Q1. How does RMRL differ methodologically from MoE-GRPO (CVPR 2026), beyond the reward design? Can the authors provide a controlled comparison using the same backbone and training setup?
  • Q2. Are the identical results for OmniVinci and Qwen3-Omni in Tables 8 and 1 correct?
Reviewer_sg5yOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new mechanism to train mixture-of-experts model to better make use of the correct evidence from the appropriate modality signals towards improving the Social Video QA. The authors propose two mechanisms one via supervised fine-tuning stage and another during the re-inforcement learning stage via GRPO.

The fine tuning stage, termed SAPR, uses an augmented cosine loss (in addition to typical next-token prediction + expert router load-balancing loss) which take into account expert probabilities pooled separately for each modality (text, video and audio) and averaged across layers. The cosine distance between the predicted tags embedding (derived from prediction head for tags from routing) and the target tag embedding.

The RL stage (RMRL) treats the expert choice as policy actions in addition to the typical token outputs. The reward is a combination of answer correctness, LLM judge which scores on the modality relevant reasoning and temporal grounding.

The proposed techniques require datasets annotated with schema tags, reasoning traces using Gemini 3.1 Pro. The dataset (OmniSocialBench) is constructed using a combination of publicly available QA datasets: OmniMMSI, SocialIQ, MTMEUR, VideoChat, NeXTQA and LLaVA-Video. For evaluation, 800 manually verified examples.

Experiments demonstrate step improvements with SAPR and another step with RMRL to reach an accuracy of 59.38%.

Critical strengths

This is an important area for research in Omni models, i.e., extracting right information from appropriate modality over the temporal dimension at right moments. Most models lack such ability and to tackle this from MOE routing layer is a good and principled area that should have a positive impact not restricting to the domain of SocialQA. The multi-pronged approach by attacking the problem from both SFT and RL stages is a good addition.

The technique proposed are technically sound in handling pooling separately first across the modalities (to remove biases imposed by modality due to frame-rate, duration) and across layers.

Dataset is potentially valuable for further future research in this area (authors expected to release this to the community). Gemini annotation methodology seems reasonable. Manually verified evaluation split is valuable.

Experiments go beyond just accuracy. Evaluations on other tasks beyond SocialQA ensures sanity and guards against regressions.

Critical weaknesses

The paper is missing RL related citations of prior work, especially the ones dealing with application of RL To MoE routing, and application of RL to video/multimodal temporal alignment.

Missing Baselines:

  • SFT followed by vanilla GRPO (maybe with correctness reward)
  • RMRL without SAPR
  • Established methods to ingest the data with tags
  • Baseline that has been fine-tuned on in-domain data The above baselines will add clarity and a sense of impact of the proposed method.

On reproducibility, several hyper parameters are missing in the paper - especially GRPO related (group size, clipping, KL, rollout temperature, steps, learning rate), weighting of the losses, reward parameters, LoRA.

Questions for authors

Given the evaluation dataset has a sample size of only 800, it is difficult to assess improvements versus noise. Are the improvements statistically significant?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.