acceptodds
Under review as a conference paper at ICLR 2027

When Debiasing Relocates Shortcuts: Orthogonal Interaction Fine-Tuning for 3D Large Multimodal Models

Abstract

Despite continued progress in 3D scene understanding, 3D large multimodal models (3D-LMMs) remain susceptible to language priors: predictions that should depend on the current scene can still be shaped by question patterns, co-occurrence statistics, and answer preferences. Existing prior-aware methods seek to suppress such shortcuts, but our analysis shows that gains on targeted debiasing objectives do not necessarily imply stronger scene dependence. Correction-induced changes can persist after explicit scene evidence is removed while systematically reshaping the answer policy. We term this phenomenon *Shortcut Relocation*: suppressing one shortcut can shift adaptation toward another shortcut-compatible policy rather than stronger scene-specific dependence. This finding motivates *Corrective Function Allocation*, a view of debiasing that explicitly restricts the functional components available to newly learned correction. We instantiate this principle with *Orthogonal Interaction Fine-Tuning* (OIF). OIF uses language-prior diagnostics only for risk localization, and restricts newly trainable corrective capacity to a centered bilinear interaction between linguistic context and the 3D scene; a policy-preservation objective limits unnecessary changes elsewhere. Under the specified reference measure, the added residual contains no first-order unimodal main effects. Matched-control experiments on Real-3DQA and its viewpoint-rotation variants, together with 3D-POPE, show that OIF reduces answer-policy drift while better preserving cross-benchmark robustness. These results show that prior-aware debiasing depends not only on identifying shortcut-prone predictions, but also on constraining where newly introduced corrective capacity can reside in function space. Code is included in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Large Multimodal Models, Language Priors, Multimodal Debiasing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_j4p3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates Shortcut Relocation in prior-aware debiasing for 3D large multimodal models (3D-LMMs). To address this issue, the authors introduce Corrective Function Allocation and propose Orthogonal Interaction Fine-Tuning (OIF). Experiments on Real-3DQA, viewpoint-rotation variants, and 3D-POPE, using Chat-Scene and 3D-LLaVA backbones, demonstrate improvements in several question-answering metrics and reduced answer-policy drift, although some robustness metrics exhibit limited improvements or degradation.

Critical strengths
  1. A meaningful conceptual perspective supported by a principled formulation. The paper identifies an important distinction between reducing language-prior dependence and ensuring that corrective adaptation relies on appropriate scene evidence. Corrective Function Allocation provides a principled framework for examining this distinction, and OIF offers a mathematically well-defined implementation through centered bilinear interactions.
  2. Comprehensive controlled comparisons and encouraging cross-backbone results. The experimental evaluation covers debiased question answering, viewpoint consistency, and object hallucination, with structural ablations, matched risk-selection controls, multiple training seeds, and efficiency measurements.
Critical weaknesses
  1. The central mechanism evidence does not sufficiently establish improved scene grounding. The zero scene-neutralized retention reported for OIF in Figure 4 follows directly from its multiplicative architecture and the intervention defined in Appendix C.3, which explicitly sets the centered scene summary to zero. Consequently, this result verifies a structural identity rather than independently demonstrating that the learned corrections rely on the intended 3D evidence. Furthermore, the orthogonality guarantee holds under an artificial product reference measure, not necessarily under the observed joint distribution or distribution shifts. The linguistic input also contains situation descriptions that may convey scene-specific information. Although the paper acknowledges some of these limitations, its central claim would benefit from more direct counterfactual evaluations that distinguish genuine scene grounding from alternative shortcut-compatible interactions.
  2. The empirical improvements are mixed, and the main baseline behavior raises concerns about the strength of the conclusions. On Chat-Scene, OIF improves Real-3DQA EM and EM-R, but overall VRS decreases from 11.90 to 11.57, while 3D-POPE accuracy improves only marginally from 72.01% to 72.19% and F1 decreases from 77.38% to 77.20% (Table 1). Similar F1 trade-offs occur on 3D-LLaVA (Table 4). Moreover, the central Shortcut Relocation diagnosis relies heavily on 3DR-FT, whose 3D-POPE affirmative rate reaches 97.78% and whose ground-truth likelihood shifts are uniformly negative in Appendix D.5. This extreme degradation warrants additional checks to determine whether the observed behavior represents a general debiasing failure mode or is particularly associated with this baseline.
Questions for authors
  1. Can the authors provide counterfactual scene-intervention experiments that do not explicitly force the OIF residual to zero, such as scene swapping, object-level perturbations, or spatial-relation changes? It would also be valuable to separate the information contributed by situation descriptions from that supplied by explicit 3D scene inputs. These results would help establish whether OIF improves dependence on the intended scene evidence.
  2. Can the authors further validate the 3DR-FT implementation and explain its extreme affirmative prediction bias and uniformly negative ground-truth likelihood shifts? Does Shortcut Relocation remain evident under alternative debiasing baselines or less severely degraded checkpoints?
1 comment

Thank you for the detailed response and additional experiments. The newly introduced counterfactual scene-intervention experiments substantially address my primary concern regarding whether OIF improves dependence on explicit 3D evidence beyond its structural orthogonality guarantee. The additional results on MainEffect and Full-BM also alleviate my concern that Shortcut Relocation is merely an artifact of the severely degraded 3DR-FT baseline. Furthermore, the multi-seed and cross-backbone evaluations strengthen the empirical evidence for the proposed method.

Nevertheless, some concerns remain. The absolute spatial-relation counterfactual accuracy is still relatively low, and the contribution of situation descriptions has not been fully isolated. Moreover, the additional structural controls do not completely establish the generality of Shortcut Relocation across independent debiasing approaches, and some robustness trade-offs persist.

Overall, I believe the authors have addressed most of my major concerns. Considering the additional evidence and the paper's conceptual and technical contributions, I have decided to increase my rating from 2 (Weak Rejection) to 3 (Weak Acceptance).

Reviewer_XLYDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that prior-aware fine-tuning of 3D-LMMs can "relocate" shortcuts instead of removing them. The authors compare 3DR-FT with its Chat-Scene parent, with and without object tokens. On Real-3DQA, 70.8% of the changed predictions stay changed even after the scene is removed, and 3D-POPE answers collapse toward "yes."

The proposed fix is OIF. The backbone stays frozen. OIF adds to the final hidden state a bias-free, rank-512 Hadamard product of two centered summaries, one of the language input and one of the scene. Under a product reference measure, this residual lies in the interaction subspace of a Hoeffding decomposition. Training uses cross-entropy on a "prior-persistent" cohort (19.4% of SQA3D) and KL to the parent on all other rows. On Chat-Scene, OIF improves Real-3DQA by +1.68 EM and +1.95 EM-R and lowers the 3D-POPE affirmative rate. Gains on 3D-LLaVA are larger.

Critical strengths
  • A real question. An outcome-level debiasing loss does not constrain which input pathway carries the correction. The paired full-input / scene-removed change diagnostic is a useful tool that others can reuse.

  • Careful controls. The paper includes budget-matched MainEffect and Full-BM arms, random masks matched on cohort composition, and training-seed replicates. It is also frank about what the orthogonality result does not imply.

  • Cheap, with one robust result. OIF adds 5.77M parameters and keeps a single decoding path. The Real-3DQA EM gain survives seed averaging.

Critical weaknesses
  • The relocation evidence is thin and possibly confounded.
  • Table 3 reports ablations on 3D-POPE only, where all gaps are under 1 point. None are reported on Real-3DQA, which is where the robust gain is.
Questions for authors

Can you provide multi-seed Real-3DQA results for MainEffect, Full-BM, matched random, All rows, and a parent-error-only selector? Given Table D.4, do you still claim above-parent 3D-POPE accuracy?

1 comment
Reviewer_XLYDOpenReview ↗

Final Recommendation

Thank you for the detailed rebuttal. My concerns have been addressed. I confirm my original score and support this paper for acceptance.

Reviewer_W1CrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this draft, the authors showed that prior-aware debiasing of 3D large multimodal models can shift the model to another shortcut instead of making it rely more on the scene, which they called Shortcut Relocation. They proposed Orthogonal Interaction Fine-Tuning (OIF), which freezes the backbone and only learns a small bilinear interaction between language and scene features. Experiments on Real-3DQA and 3D-POPE showed less policy drift and modest QA gains.

Critical strengths
  1. Checking whether debiasing really increases scene dependence (full input vs. scene removed) is a simple and useful idea.
  2. OIF is lightweight, and the authors include budget-matched controls, multiple seeds and confidence intervals.
Critical weaknesses
  1. OIF's zero retention without the scene holds by construction, so it does not show that the corrections actually use the scene. A scene-swap test would be more convincing.
  2. The key ablations (MainEffect, Full-BM, random masks) are only reported on 3D-POPE, not on Real-3DQA, the main target.
  3. The gains are small and partly within seed noise. For example, the three-seed mean on 3D-POPE is below the original model.
  4. Minor: Shortcut Relocation is only shown for one method; several key terms are only defined in the appendix.
Questions for authors

Please refer to the Critical Weaknesses section for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.