acceptodds
Withdrawn from ICLR 2027

Omni Group Fusion: Groupoid-Compatible Correspondence and Equivariant Transport Across Multimodal Sensor Frames

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Omni and any-to-any models increasingly ingest text, images, video, audio, depth, and 3D into unified sequence backbones, achieving architectural interface uniformity through unconstrained attention. However, this convenience overlooks a fundamental physical reality: physical sensors and spatial language naturally operate in distinct, uncalibrated, and uncertain local reference frames. Treating directional and geometric quantities as flat, interchangeable scalars forces backbones to memorize pose-dependent statistical co-occurrences, precipitating sample inefficiency, cross-modal gradient starvation, and fragility under novel viewpoints or clock drift—while heuristic point-pose canonicalization breaks discontinuously under physical ambiguity. We introduce Omni Group Fusion (OGF), which grounds multimodal fusion on an observation groupoid respecting independent local coordinate charts. OGF operationalizes four transformation-consistent primitives: (i) routing tokens to shared world anchors via group-invariant descriptors, preventing correspondence drift under sensor rotations; (ii) maintaining multi-hypothesis spatial and temporal posteriors to represent frame ambiguity and clock drift without discontinuous collapse; (iii) equivariantly transporting typed representations ( vectors, tensors, coordinates) into a common world frame before pooling; and (iv) adjudicating consensus to attenuate conflicting sensors, updating anchors via Clebsch–Gordan couplings and reading back into frozen backbones through zero-initialized residuals. We theoretically establish global equivariance, local-gauge independence, stabilizer well-definedness, posterior composition consistency, and streaming stability. Across a controlled physical world, real continuous sensors, and five frozen generative backbones (3B–8B), factorial ablations isolate different claims: removing the group returns chance; typed transport, not orbit coverage, is what continuous sensors require; and, on Qwen2.5-Omni-7B, breaking sensor pairing removes routed synergy on four of the six audio–visual benches.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal fusion, equivariant representation, groupoid, geometric deep learning, omni‑modal model, sensor frame, world modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_s1oSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that Omni/any-to-any models fuse modalities as flat, untyped tokens and do not explicitly account for the distinct and uncertain reference frames of physical sensors and spatial language. It proposes Omni Group Fusion (OGF), which models multimodal fusion on an observation groupoid of local frames. Tokens are routed to shared world anchors using invariant descriptors; spatial and temporal alignment are represented with multi-hypothesis posteriors; typed features such as ℓ=1\ell=1 vectors, ℓ=2\ell=2 tensors, and coordinates are transported into a common frame before pooling; and anchor representations interact through Clebsch--Gordan couplings. The method is evaluated in a controlled SO(3) world, small heads on frozen real sensor features, and residual sidecars inserted into five frozen 3B--8B multimodal trunks. The paper also provides theoretical results for equivariance, gauge independence, ambiguity handling, posterior composition, and stability

Critical strengths
  • The Stage II experiments provide clear evidence for explicit transformation structure on geometric tasks. Under held-out rotations, OGF achieves exact or near-exact invariance on TUM and eliminates the rotation gap on ScanRefer, while rotation augmentation and pose concatenation do not. These results support the claim that typed transport can generalize across transformations not covered during training.

  • The evaluation separates controlled mechanism tests, real frozen-feature experiments, and large-model integration. It also includes modality-subset comparisons, pairing/frame/clock interventions, matched ablations, multiple seeds in Stages I--II, and reported negative results. This makes it possible to examine several components of the method separately rather than relying only on aggregate benchmark gains.

  • The fusion criterion in Section 3.2 is useful. It requires the full model to outperform independently trained proper modality subsets and additionally checks whether disrupting pairing, frame, or temporal relationships causes the expected degradation.

Critical weaknesses
  • The Stage III experiments do not clearly establish that the geometric/groupoid components are responsible for the gains on Omni models. Appendix H.1 lists ℓ=1\ell=1 and ℓ=2\ell=2 channels, token-layout coordinates, timestamps, and frame posteriors, but does not clearly specify how the typed channels are constructed or demonstrate that they satisfy the transformation laws assumed by the theory. This makes it difficult to determine which theoretical guarantees apply to the Stage III implementation.

  • The Stage III ablations suggest that correspondence routing contributes more than geometric transport on the six audio--visual benchmarks. The wrong-frame intervention has little effect, and retraining without transport or with a uniform frame posterior remains close to full OGF and above LoRA. Removing correspondence causes substantially larger drops. The audio--visual results therefore provide stronger evidence for the routing mechanism than for the full transport mechanism.

  • Synroute\mathrm{Syn}_{\mathrm{route}} is weaker than the fusion criterion introduced in Section 3.2. In Stage III, the frozen trunk still receives the complete multimodal input; V-route and A-route only restrict which streams enter the sidecar. Thus, the result shows a benefit from jointly routing modalities through the sidecar, but does not demonstrate synergy over independently trained modality subsets in the stronger sense used earlier in the paper.

  • Statistical support for the Stage III results is limited. Omni-7B at 96K is evaluated with two training seeds and no confidence interval, while the other Core96K trunks and scales use one seed. Since many OGF--LoRA differences are below one percentage point, it is difficult to assess how consistently the reported improvements would reproduce across training runs.

Questions for authors
  • How are the Stage III ℓ=1\ell=1 and ℓ=2\ell=2 channels and frame posteriors constructed? Under what transformations are these quantities equivariant, and which assumptions of Theorems 1--5 therefore apply to the Stage III implementation?

  • Can the authors compare OGF with a parameter-matched anchor-routing sidecar that preserves correspondence/routing but removes typed channels, frame posteriors, equivariant transport, and CG coupling? The current ablations suggest that correspondence may explain much of the Stage III audio--visual gain.

  • Can the authors report Stage III synergy against independently trained V-only and A-only sidecars? Since the frozen trunk receives all modalities in the V-route and A-route conditions, how should Synroute\mathrm{Syn}_{\mathrm{route}} be interpreted relative to the stronger Syn\mathrm{Syn} criterion in Section 3.2?

Reviewer_uHbNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes Omni Group Fusion (OGF), a multimodal fusion module combining invariant correspondence, uncertain frame alignment, and equivariant feature transport through shared anchors. It provides theoretical guarantees and integrates the module into frozen generative backbones through residual connections. The evaluation is broad with extensive ablations and perturbation tests. However, unclear implementation and definitions limit reproducibility.

Critical strengths

OGF improves performance on spatial reasoning and localization tasks under viewpoint changes. It also achieves gains in temporal alignment and video moment retrieval. Additionally, the work provides rigorous mathematical guarantees under its stated assumptions.

Critical weaknesses

The main concern is on the organization of this paper. The paper is overly dense and frequently assumes that readers have a strong mathematical background and substantial familiarity with prior work. The paper introduces sophisticated mathematical concepts without clearly mapping them to the actual architecture. For example, it is unclear how the typed features are constructed from Qwen embeddings and how the equivariance assumptions are satisfied. Although Eq. (6) formally defines a discrete frame posterior, the description does not make clear how its hypotheses are instantiated and updated in the generative-backbone implementation. A concrete computational example is needed. The training objective is insufficiently specified. Eq. (10) lists seven losses without explicit mathematical definitions of the individual terms.

Questions for authors

Please see the questions in Weaknesses.

Reviewer_TDNyOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I believe the topic of this paper falls outside the scope of ICLR. I encourage the authors to submit it to mathematics journal. Therefore, I recommend a desk rejection without proceeding to a full review.

Critical strengths

Not Applicable

Critical weaknesses

I could not understand the core problems treated in this paper.

Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.