When Debiasing Relocates Shortcuts: Orthogonal Interaction Fine-Tuning for 3D Large Multimodal Models

Despite continued progress in 3D scene understanding, 3D large multimodal models (3D-LMMs) remain susceptible to language priors: predictions that should depend on the current scene can still be shaped by question patterns, co-occurrence statistics, and answer preferences. Existing prior-aware methods seek to suppress such shortcuts, but our analysis shows that gains on targeted debiasing objectives do not necessarily imply stronger scene dependence. Correction-induced changes can persist after explicit scene evidence is removed while systematically reshaping the answer policy. We term this phenomenon *Shortcut Relocation*: suppressing one shortcut can shift adaptation toward another shortcut-compatible policy rather than stronger scene-specific dependence. This finding motivates *Corrective Function Allocation*, a view of debiasing that explicitly restricts the functional components available to newly learned correction. We instantiate this principle with *Orthogonal Interaction Fine-Tuning* (OIF). OIF uses language-prior diagnostics only for risk localization, and restricts newly trainable corrective capacity to a centered bilinear interaction between linguistic context and the 3D scene; a policy-preservation objective limits unnecessary changes elsewhere. Under the specified reference measure, the added residual contains no first-order unimodal main effects. Matched-control experiments on Real-3DQA and its viewpoint-rotation variants, together with 3D-POPE, show that OIF reduces answer-policy drift while better preserving cross-benchmark robustness. These results show that prior-aware debiasing depends not only on identifying shortcut-prone predictions, but also on constraining where newly introduced corrective capacity can reside in function space. Code is included in the supplementary material.