acceptodds
Withdrawn from ICLR 2027

Chain of Modality: Understanding and Controlling Modality Organization Bias in Omni-MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Omni-modal Large Language Models (Omni-MLLMs) are designed to reason over diverse sensory streams within a unified model. However, we find that multimodal reasoning is not solely determined by sensory evidence, but also by how modality information is organized. Through systematic analysis across diverse reasoning scenarios, we show that different modality organizations exhibit distinct advantages and limitations, and no single topology is universally optimal. Rather than proposing a general-purpose improvement to Omni-MLLM accuracy, we ask a narrower question: can these topology-induced failures be systematically identified and corrected? Motivated by this, we propose Chain of Modality (CoM), a framework that dynamically reorganizes multimodal topology during inference and learns adaptive organization strategies. Experiments across five benchmarks, diverse architectures, and model scales show that CoM reliably recovers topology-sensitive failures, through both training-free planning and lightweight Planner-SFT, while preserving performance on the full benchmark.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Omni-modal Large Language Models, Modality Organization, Adaptive Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ER6hOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how organizing audio and visual inputs affects omni-modal reasoning, rather than claiming general benchmark superiority. It distinguishes four reasoning requirements and compares Single, Parallel, Sequential, and Interleaved organizations. Chain of Modality uses a Planner to choose an organization, a frozen Reasoner to answer, and distribution fusion for parallel branches. Training-free planning and planner-only LoRA supervision are evaluated on topology-sensitive subsets and full benchmarks, with additional cross-model transfer. The central contribution is a diagnostic account of organization-dependent failures and an attempt to learn corrective routing.

Critical strengths
  1. The paper frames a focused, testable problem. Table 1 contrasts evidence requirements, while Tables 5–6 show higher local attention accompanying improved accuracy on temporal reasoning but lower accuracy on temporal hallucination. This is useful evidence against a universally beneficial interleaving policy, without establishing attention as the sole cause of the accuracy changes.
  2. The intervention retains the Reasoner and tests both training-free and learned planning. Random-selection controls (Tables 12–13), cross-backbone transfer (Table 8), and full-benchmark results (Table 7) give the proposal substance beyond selected qualitative demonstrations. Full-benchmark improvements are modest and WorldSense is essentially unchanged, consistent with the paper's narrow scope.
Critical weaknesses
  1. The baseline and fixed-organization comparisons need reconciliation. Section 5.1 calls the vanilla baseline the 'A-V organization', rather than unambiguously specifying its executable topology. If this denotes A→V in Table 10, the DailyOmni/WorldSense baseline scores of 29.0/27.8 in Table 7 differ from 38.7/28.4 in Table 10; the former pair instead matches that table's Interleave scores.
  2. The displayed strict RoPE implication is not established by the stated bound. On p. 5, B depends on query/key coefficients and oscillatory partial sums; the long-term-envelope interpretation alone does not imply |B_int(1)| > |B_seq(T)| on p. 6.
Questions for authors
  1. For Tables 7, 10, 12, and 14, please reconcile the sample sets, prompt variants, and executable modality organizations.
  2. What additional assumptions justify the strict distance inequality for the evaluated models, beyond a long-term-envelope interpretation, and how are chunk positions mapped to actual position IDs?
Reviewer_cCUTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that, in Omni-MLLMs, how audio and visual inputs are organized changes predictions, and that no organization is best across reasoning scenarios. Evidence comes from ~200-sample scenario subsets and attention/RoPE-distance analyses. The proposed Chain of Modality (CoM) uses a Planner that picks modalities and an organization per sample, the frozen backbone as Reasoner, and a Decider that fuses parallel branches. Evaluation focuses on “topology-sensitive” subsets of five benchmarks, plus full-benchmark results and planner transfer to Qwen-Omni-3B and Ola-7B.

Critical strengths
  • A well-posed, clearly scoped question. Treating input organization as an inference-time variable is a fresh angle for audio-visual models, and the authors state plainly that general accuracy is not their goal. This framing, together with Sec. 3’s attempt at a mechanistic explanation (attention allocation, RoPE distance).

  • A lightweight, backbone-agnostic design. The Reasoner stays frozen, only a LoRA Planner is trained, and full-benchmark accuracy is preserved. A Planner trained on Qwen-Omni-7B also improves Qwen-Omni-3B and Ola-7B on the subsets, so if the gains hold up, the method would be cheap to train and easy to reuse across backbones.

Critical weaknesses
  • The main evaluation favors CoM by construction. The topology-sensitive subsets are mined from Qwen-Omni-7B’s own predictions under the candidate organizations, so a correct organization always exists and even random selection beats the baseline. On the WorldSense, DailyOmni, and OmniBench subsets, fixed audio-only outperforms CoM(SFT). Full-benchmark accuracy changes by −0.1 to +1.4 points outside AVHBench, in single greedy runs without confidence intervals. The results support partial recovery relative to the chosen baseline, but do not isolate the additional value of adaptive topology control over simple modality selection.

  • Some scenario-specific preferences are weakly resolved.. Two of the five Qwen-Omni-7B preferences in Table 1 rest on margins of ≤1 point over ~200 samples, and on Ola-7B, which cannot run interleaving, audio-only is best for temporal reasoning. After SFT, the Planner never picks Parallel and uses a single modality for 75% of WorldSense and 60% of OmniBench samples. This weakens the paper’s framing around organization topology.

Questions for authors
  1. Qwen-Omni-7B (Qwen2.5-Omni) uses time-aligned TMRoPE. Which position IDs do audio and video tokens receive under each organization, and does the distance argument in Sec. 3 still hold?

  2. Are the subsets for Qwen-Omni-3B and Ola-7B (Table 8) built from each model’s own predictions or from Qwen-Omni-7B’s?

Reviewer_mhvoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how audio and visual input organization affects Omni-MLLM reasoning. It compares single, parallel, sequential, and interleaved inputs, then proposes CoM, where a Planner selects an organization before a frozen Reasoner answers. The idea is evaluated on five benchmarks and several model backbones.

Critical strengths

The central observation is useful: the same audio and visual evidence can produce different answers under different input organizations. The controlled examples and cross-model results suggest this is a real behavior worth studying.

Critical weaknesses
  1. The topology-sensitive subsets are defined using model outcomes and retain samples where at least one topology succeeds. This makes oracle accuracy 100% by construction and favors any method that changes the default topology.

  2. The released code does not match the paper closely enough. The SFT action space omits explicit Parallel and Interleaved choices.

Questions for authors
  1. Did the unreset batch in test_dailypomni_com-sft.py affect the reported DailyOmni results?

  2. Why does the code force every Temporal-centric prediction to use the interleaved result, regardless of the predicted modality route?

  3. Why does Table 10 report 0.0% for two OmniBench organizations when the released data gives 35.48% and 30.65%?

  4. Can the authors provide complete training and evaluation code, plus an equal-compute comparison against simple two-pass topology ensembling?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.