- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that Omni/any-to-any models fuse modalities as flat, untyped tokens and do not explicitly account for the distinct and uncertain reference frames of physical sensors and spatial language. It proposes Omni Group Fusion (OGF), which models multimodal fusion on an observation groupoid of local frames. Tokens are routed to shared world anchors using invariant descriptors; spatial and temporal alignment are represented with multi-hypothesis posteriors; typed features such as vectors, tensors, and coordinates are transported into a common frame before pooling; and anchor representations interact through Clebsch--Gordan couplings. The method is evaluated in a controlled SO(3) world, small heads on frozen real sensor features, and residual sidecars inserted into five frozen 3B--8B multimodal trunks. The paper also provides theoretical results for equivariance, gauge independence, ambiguity handling, posterior composition, and stability
Critical strengths
-
The Stage II experiments provide clear evidence for explicit transformation structure on geometric tasks. Under held-out rotations, OGF achieves exact or near-exact invariance on TUM and eliminates the rotation gap on ScanRefer, while rotation augmentation and pose concatenation do not. These results support the claim that typed transport can generalize across transformations not covered during training.
-
The evaluation separates controlled mechanism tests, real frozen-feature experiments, and large-model integration. It also includes modality-subset comparisons, pairing/frame/clock interventions, matched ablations, multiple seeds in Stages I--II, and reported negative results. This makes it possible to examine several components of the method separately rather than relying only on aggregate benchmark gains.
-
The fusion criterion in Section 3.2 is useful. It requires the full model to outperform independently trained proper modality subsets and additionally checks whether disrupting pairing, frame, or temporal relationships causes the expected degradation.
Critical weaknesses
-
The Stage III experiments do not clearly establish that the geometric/groupoid components are responsible for the gains on Omni models. Appendix H.1 lists and channels, token-layout coordinates, timestamps, and frame posteriors, but does not clearly specify how the typed channels are constructed or demonstrate that they satisfy the transformation laws assumed by the theory. This makes it difficult to determine which theoretical guarantees apply to the Stage III implementation.
-
The Stage III ablations suggest that correspondence routing contributes more than geometric transport on the six audio--visual benchmarks. The wrong-frame intervention has little effect, and retraining without transport or with a uniform frame posterior remains close to full OGF and above LoRA. Removing correspondence causes substantially larger drops. The audio--visual results therefore provide stronger evidence for the routing mechanism than for the full transport mechanism.
-
is weaker than the fusion criterion introduced in Section 3.2. In Stage III, the frozen trunk still receives the complete multimodal input; V-route and A-route only restrict which streams enter the sidecar. Thus, the result shows a benefit from jointly routing modalities through the sidecar, but does not demonstrate synergy over independently trained modality subsets in the stronger sense used earlier in the paper.
-
Statistical support for the Stage III results is limited. Omni-7B at 96K is evaluated with two training seeds and no confidence interval, while the other Core96K trunks and scales use one seed. Since many OGF--LoRA differences are below one percentage point, it is difficult to assess how consistently the reported improvements would reproduce across training runs.
Questions for authors
-
How are the Stage III and channels and frame posteriors constructed? Under what transformations are these quantities equivariant, and which assumptions of Theorems 1--5 therefore apply to the Stage III implementation?
-
Can the authors compare OGF with a parameter-matched anchor-routing sidecar that preserves correspondence/routing but removes typed channels, frame posteriors, equivariant transport, and CG coupling? The current ablations suggest that correspondence may explain much of the Stage III audio--visual gain.
-
Can the authors report Stage III synergy against independently trained V-only and A-only sidecars? Since the frozen trunk receives all modalities in the V-route and A-route conditions, how should be interpreted relative to the stronger criterion in Section 3.2?