acceptodds
Under review as a conference paper at ICLR 2027

Authorization-Conditioned Capability Isolation for Collaborative MLLM Inference

Abstract

Collaborative MLLM inference creates a mismatch between what a remote model can infer from a transmitted representation and what is authorized under the specific request. Existing privacy-oriented approaches primarily aim to reduce input reconstruction or semantic leakage, but do not explicitly control which capabilities a deployed model may exercise under request-level authorization. This work formulates authorization-conditioned capability isolation as transforming a shared visual representation so that a denied target contributes no additional capability beyond the model's No-Image behavior, while non-target capabilities remain close to their Raw behavior under the same transmitted state. We introduce an Authorization-Conditioned Bottleneck (ACB) that realizes this objective through policy-conditioned representation modulation, including token scaling, channel reweighting, rank-limited correction, and fixed-representation off-target preservation. In the primary evaluation on our constructed AuthCap, a 13-capability authorization benchmark, ACB achieves authorized utility of while reducing denied target capability to within of the No-Image reference. With off-target preservation, unrelated capability retention improves from to . These results demonstrate the feasibility of request-level capability isolation: suppressing unauthorized capabilities while substantially retaining unrelated visual capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Large Language Models, Representation Privacy, Collaborative Inference, Authorization-Conditioned Capability Isolation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rgzDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces authorization-conditioned capability isolation for collaborative MLLM inference, aiming to suppress an explicitly denied target capability toward a No-Image baseline while preserving unrelated capabilities under the same transmitted representation. The proposed Authorization-Conditioned Bottleneck (ACB) performs a lightweight, policy-conditioned transformation of the client-side visual representation using token scaling, channel reweighting, and a low-rank subtractive correction, with training that jointly optimizes target suppression and fixed-representation off-target preservation.

Critical strengths
  1. The topic is highly interesting, highlighting the emerging problem of authorization-conditioned capability isolation, which differs from prior works that primarily focus on reconstructing from shared intermediate activations.

  2. The proposed ACB is effective without requiring any modification to the cloud-based MLLM.

Critical weaknesses
  1. The benchmark relies substantially on synthetic document tasks and screened non-text tasks; therefore, its generalization to broader natural-image domains and real-world usage scenarios remains to be demonstrated.

  2. The policy space is limited to single-capability authorization. Combinatorial or multi-capability authorization (e.g., allowing capability A while simultaneously denying capabilities B and C) is not evaluated, limiting the demonstrated real-world applicability of the proposed method.

  3. ACB involves multiple stages and loss functions; however, the individual contribution of each component is not clearly evaluated. A comprehensive ablation study is needed to quantify the contribution of each stage and loss term to the overall performance.

  4. The evaluation is primarily conducted on a single Qwen model, without experiments across a broader range of MLLMs, making it unclear whether the proposed method generalizes to different model architectures and families.

Questions for authors
  1. How does ACB perform on real-world examples and more diverse natural-image domains?

  2. How does ACB perform under multi-capability policies (e.g., simultaneously denying access to phone numbers and names while allowing access to amounts)? Can the current design support compositional authorization involving multiple policy embeddings simultaneously?

  3. How does each component of ACB contribute to its overall effectiveness?

  4. How does ACB perform across different MLLMs and model architectures?

Reviewer_5jXYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates request-level capability authorization in collaborative multimodal large language model inference, aiming to suppress unauthorized visual capabilities while preserving unrelated capabilities. The authors propose Authorization-Conditioned Bottleneck, which combines policy-conditioned token scaling, channel reweighting, and rank-limited correction. The training objective jointly optimizes authorized behavior preservation, denied-target suppression, and off-target capability retention. To evaluate this framework, the authors construct AuthCap, a benchmark covering 13 visual capabilities, and conduct experiments on Qwen2.5-VL-7B and Molmo2-O-7B. Results demonstrate effective behavioral suppression on the primary backbone, with off-target retention improving from 0.1400 to 0.5479.

Critical strengths
  1. The paper introduces a meaningful distinction between capability authorization and conventional representation privacy. Rather than indiscriminately suppressing visual information, the proposed formulation aims to control which capabilities remain accessible under specific authorization policies. This provides a useful perspective on privacy-aware collaborative inference.
  2. The proposed framework jointly considers unauthorized capability suppression and unrelated capability preservation. By evaluating different capabilities on the same protected representation, the method avoids trivially satisfying authorization constraints by removing all visual information.
  3. The method is lightweight and supported by several informative experiments. ACB modifies intermediate visual representations without fine-tuning the cloud MLLM. The experiments include component comparisons, preservation-loss ablations, cross-backbone evaluation, paraphrasing tests, and adaptive probing, providing useful evidence about the method's effectiveness and limitations.
Critical weaknesses
  1. The suppression–preservation trade-off remains substantial and backbone-dependent. Off-target retention reaches only 0.5479 on Qwen2.5-VL-7B. On Molmo2-O-7B, target UQG increases to 0.0712, with considerably larger residual leakage for certain capabilities, weakening the generalization claims.
  2. The component-level ablation study is incomplete. Although token-only and token-plus-channel variants are evaluated, the results suggest that rank-limited correction is responsible for most of the suppression improvement. The absence of a rank-correction-only baseline and more systematic component combinations makes it difficult to establish the independent contributions of token scaling, channel reweighting, and low-rank correction.
  3. The presentation quality requires substantial improvement. Figure 2 uses several small bar charts with limited comparative information.
Questions for authors
  1. Authorization policy specification. How are the capability identity and authorization state determined in practical deployment? Are they explicitly provided by users or administrators, or automatically inferred from the query? If automatic inference is required, how would incorrect capability classification or authorization decisions affect the security guarantees?
  2. Independent contributions of ACB components. Given the dramatic improvement after introducing rank-limited correction, can the authors provide additional ablations, particularly rank-correction-only and channel-only variants, to clarify whether token scaling and channel reweighting are necessary beyond the low-rank intervention?
  3. Training-loss contributions and interactions. Can the authors provide a more complete ablation of L_{\mathrm{allow}}, L_{\mathrm{deny}}, L_{\mathrm{off}}, and L_{\mathrm{reg}}? How do the gradients from denied-target suppression and off-target preservation interact when the corresponding capabilities share visual information or semantic features?
  4. Choice of suppression reference. Why is No-Image behavior used as the denied-target reference rather than explicit refusal or an alternative privacy-preserving response? Since No-Image predictions may still exploit language priors, how does this choice affect the interpretation of near-zero UQG and the practical authorization guarantees?
Reviewer_NNtYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates authorization-conditioned capability isolation in collaborative multimodal large language model (MLLM) inference, aiming to suppress unauthorized visual capabilities while preserving authorized and unrelated capabilities. To address this problem, the authors propose an Authorization-Conditioned Bottleneck (ACB), which transforms client-side visual representations before transmission to a frozen cloud MLLM. The proposed framework integrates policy-conditioned token scaling, channel reweighting, and rank-limited feature correction. A joint training objective is designed to preserve authorized capabilities, suppress denied capabilities toward the No-Image baseline, and maintain unrelated capabilities using the same protected representation. For validation, the authors construct AuthCap, a benchmark covering 13 visual capabilities with matched authorization settings. Experiments on Qwen2.5-VL-7B demonstrate strong suppression of denied capabilities while retaining a portion of unrelated capabilities. Additional evaluations include ablation studies, robustness tests, adaptive probing, and cross-backbone experiments on Molmo2-O-7B to assess the effectiveness and limitations of the proposed framework.

Critical strengths
  1. The paper addresses an important problem in collaborative MLLM inference by introducing authorization-conditioned capability isolation. Unlike conventional privacy-preserving approaches that primarily focus on preventing information reconstruction, this work aims to suppress unauthorized capabilities while preserving authorized and unrelated capabilities. This problem formulation is practically relevant and provides a useful perspective on fine-grained authorization control in collaborative MLLM systems.

  2. Relatively Comprehensive Experimental Evaluation. The authors construct the AuthCap benchmark covering 13 visual capabilities and conduct experiments on two MLLM backbones. The evaluation includes comparisons with existing baselines, component ablations, cross-backbone experiments, paraphrase robustness tests, and adaptive probing. These experiments provide useful empirical evidence for the feasibility of the proposed approach and offer insights into the trade-off between capability isolation and utility preservation, although important limitations remain.

Critical weaknesses
  1. The proposed ACB mainly combines established techniques, including token scaling, channel reweighting, and low-rank feature correction. The paper does not sufficiently clarify the methodological novelty beyond their combination. Furthermore, comparisons with closely related concept erasure methods, such as INLP, LEACE, and SPLINCE, are missing. The component ablations are also insufficient to establish the necessity of each module, particularly since low-rank correction appears to account for most of the capability suppression. These limitations make it difficult to assess whether the proposed framework represents a substantial methodological advancement over existing approaches.

  2. The experimental results raise concerns about the reliability of the proposed authorization mechanism. The paraphrase robustness evaluation reports a robust-denial rate of only 0.7685, while several capabilities exhibit substantial residual unauthorized capability on Molmo2-O-7B. More importantly, introducing the off-target preservation objective increases adaptive probe accuracy from 0.5301 to 0.8087, suggesting that considerable sensitive information remains recoverable from the protected representations. Although adaptive probing falls outside the primary frozen-model threat model, these results reveal important limitations in the practical protection offered by ACB. Consequently, the current experiments do not sufficiently demonstrate robust capability isolation across different models, query formulations, and evaluation conditions.

Questions for authors
  1. The proposed Authorization-Conditioned Bottleneck (ACB) primarily combines established techniques, including token scaling, channel reweighting, and low-rank feature correction. The paper should clearly demonstrate what distinctive advantages emerge from their combination. In particular, it remains unclear why these components are individually insufficient and whether their integration is necessary for effective capability isolation and preservation.
  2. The experimental baselines mainly include query-guided token pruning, reconstruction-oriented representation protection, and simplified variants of ACB. However, these approaches do not directly address the same selective capability removal objective. More relevant comparisons with concept erasure and task-preserving representation methods are missing. Moreover, Table 2 reports only authorized utility and target UQG, without off-target retention, which is central to the claimed contribution.
  3. Introducing low-rank correction reduces target UQG from 0.6122 to 0.0007, suggesting that this component may account for most of the capability suppression. However, the paper does not evaluate low-rank correction alone or alternative combinations such as token-plus-rank and channel-plus-rank correction. Additional ablations removing query conditioning and policy conditioning are also needed to clarify the contributions of the proposed architectural components.
  4. Although AuthCap covers 13 capabilities, a substantial portion of the benchmark consists of synthetic documents and relatively simple information extraction or object-presence tasks. These settings may not adequately reflect realistic MLLM applications involving compositional visual reasoning, multiple sensitive attributes, and strongly correlated semantic concepts.
  5. Several methodological details remain insufficiently specified. In particular, the regularization term L_{\text{reg}} in Eq. (17) is not explicitly formulated, although it contributes to the overall optimization objective. Additionally, the notation for token scaling and channel reweighting in Figure 1 is not fully consistent with Eqs. (9)–(10). The operational implementation of the No-Image reference (Z_0) also requires further clarification.
  6. Figure 1 contains excessive decorative elements, colorful panels, and small text, making it difficult for readers to clearly understand the overall architecture of the proposed method. The training and inference pipelines are not clearly distinguished.
  7. Although the paper claims that ACB preserves authorized capabilities, the cross-backbone results reveal a substantial performance degradation on Molmo2-O-7B. Specifically, Table 11 shows that authorized utility decreases from 0.7606 under Raw representations to 0.6519 with ACB, representing an absolute reduction of 0.1087. This contrasts sharply with the results on Qwen2.5-VL-7B, where authorized utility remains close to the Raw baseline. Moreover, the Molmo experiments directly reuse hyperparameters selected for Qwen without backbone-specific tuning. These results raise concerns about whether the proposed method can reliably preserve authorized capabilities across different MLLM architectures.
  8. Several results on Molmo2-O-7B exhibit substantial variation across random seeds. For example, Table 12 reports a target UQG of 0.2708 ± 0.2320 for document presence and an authorized utility of 0.5042 ± 0.3178 for template ID. These large standard deviations indicate that the effectiveness of the proposed method may be highly sensitive to training randomness for certain capabilities.
  9. The reported off-target retention is evaluated on a restricted subset of capability pairs rather than the complete held-out test set. The authors should report the proportion and distribution of excluded samples, provide absolute off-target performance on the complete evaluation set, and examine whether the claimed preservation advantages remain consistent without this filtering.
  10. Missing Sensitivity Analysis of the Off-Target Supervision Count.The proposed off-target preservation mechanism relies on additional supervision from other capabilities, yet the number of supervised off-target queries is fixed at K = 2 without a systematic sensitivity analysis. Since the method aims to preserve multiple unrelated capabilities, it is important to understand how preservation performance changes as K increases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.