Beyond Disentanglement: Adaptive Partial Sharing for Spatial Reasoning Control in Vision-Language Models

How should the feature subspaces used to control spatial reasoning in vision– language models be organized across relations: fully shared, relation-selective, or partially shared? We investigate this question through controlled spatial coun- terfactuals that connect changes in visual geometry to internal causal sensitiv- ity and learned low-rank control. Our TriAxis-CF dataset comprises 300 fac- tual/counterfactual image pairs, each reversing one spatial relation while pre- serving the other two, with geometry-derived labels and role-reversed questions. These paired contrasts provide controlled donors for activation interventions. Across three models evaluated under a shared protocol and an independent Qwen3 pilot, horizontal reasoning exhibits a repeated distribution of causal sensitivity across early visual regions, intermediate relation-text positions, and later final- input positions. Increasing visual state differences coexist with diminishing out- put effects, establishing a distinction between representation change and its causal influence on spatial judgments. A comparison of fully shared, relation-selective, and shared–private control in Qwen3 identifies adaptive partial sharing as an effec- tive structure for balancing corrective intervention and answer preservation. Our SP-LoReFT combines a shared branch with relation-specific branches that adapt into partially overlapping subspaces. On held-out internal scenes, it attains the highest accuracy point estimate among the evaluated configurations, improving from 88.33% to 90.56% over shared rank-8 control. Break decreases from 6.18% to 3.89%, while Repair increases from 65.05% to 66.99%. These findings support adaptive partial sharing for spatial reasoning control, with its principal observed benefit arising from improved preservation of already-correct judgments.