acceptodds
Withdrawn from ICLR 2027

Opaque Multimodal-LLM Coordinated Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) enhance LLMs' capabilities but raise significant privacy and compliance concerns, creating a need for machine unlearning (MU). Existing MU methods for MLLMs predominantly rely on white-box parameter updates, which are impractical in Large-Model-as-a-Service (LMaaS) settings with restricted model access. Current API-level alternatives adopt unimodal interventions that fail to address the entangled multimodal representations in MLLMs, leading to ineffective cross-modal forgetting. To overcome this, we propose OpaqueMU, a multimodal unlearning framework under a split-access LMaaS setting, where the Transformer backbone remains opaque while input-side representations and output scores are accessible. OpaqueMU introduces a coordinated two-stage architecture: a Manifold Sanitization (MS) module that selectively disentangles sensitive information within modality-specific representations prior to fusion, and a Distribution Alignment (DA) module that suppresses residual forget-related signals at the output level while preserving retained knowledge. Experiments across two multimodal unlearning benchmarks demonstrate that OpaqueMU achieves a favorable forgetting–utility trade-off while robustly preserving retained knowledge, alongside a 66% reduction in inference latency compared with the strongest baseline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal LLM, Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AX7tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OpaqueMU, a multimodal machine unlearning framework under a split-access LMaaS setting, where the Transformer backbone remains frozen while input representations and output logits are accessible to the client. It combines Manifold Sanitization (MS), which applies visual embedding offsets and textual subspace projections, with Distribution Alignment (DA), which reweights output distributions to suppress target knowledge while preserving retained utility. Experiments on MLLMU-Bench and CLEAR report favorable forgetting–utility trade-offs, improved privacy robustness, and reduced inference latency.

Critical strengths

S1. Interesting problem setting. Exploring multimodal unlearning without modifying the Transformer backbone is a potentially useful direction for model services with restricted access.

S2. Modality-aware design. The framework separately handles visual and textual representations before fusion and combines representation-level intervention with output-level correction.

S3. Comprehensive evaluation. The paper covers two benchmarks, multiple forgetting ratios, two model architectures, ablation studies, efficiency analysis, and robustness evaluations.

Critical weaknesses

W1. Unclear practical motivation and unlearning guarantees. The split-access setting assumes client access to input representations and full logits but not the backbone, which is uncommon in commercial APIs and lacks a concrete deployment motivation. Since the backbone remains unchanged, the method provides no evidence of model-level knowledge removal, and target information may remain accessible by bypassing the client-side modules. The authors should clarify the threat model and distinguish behavioral suppression from model-level unlearning.

W2. Questionable result reliability and efficiency comparisons. In Table 2, all four Grad Diff Rouge scores exactly equal the corresponding accuracies divided by 100, while MANU's GEN accuracies duplicate its QA accuracies, suggesting possible reporting errors. Figure 5 reports over 2 seconds of inference latency for Grad Diff and MANU, despite their lack of additional inference-time modules, compared with 0.61 seconds for OpaqueMU. The approximately 0.2-second training times per epoch and unclear inclusion of multi-megabyte logit transmission (Table 4) further require clarification.

W3. Limited novelty and insufficient evidence of component synergy. The method primarily combines established techniques, including representation offsets, subspace projection, adversarial alignment, and logit reweighting, without sufficiently discussing related concept erasure approaches such as INLP and LEACE. In Table 3, the full model and DA-only variant achieve identical F-QA (31.20%) and F-GEN accuracies (6.00%), while MS-only achieves higher R-QA accuracy (44.21% vs. 36.97%). Stronger comparisons with simpler methods under identical access assumptions are needed to justify the proposed combination.

W4. Insufficient evidence of selective forgetting. Aligning forget and retain representations does not necessarily remove sensitive information, and pushing output distributions toward uniformity may merely reduce confidence. On CLEAR, F-VQA drops from 95.21% to 92.55%, while R-VQA also drops from 98.23% to 95.74%, suggesting limited selectivity. Moreover, Table 13 shows that OpaqueMU still reveals sensitive attributes in a forgotten sample. Stronger evaluations of information recoverability are needed.

W5. Incomplete evaluation protocol and potentially unfair baselines. The compared methods have different model-access privileges, making it difficult to isolate the proposed method's benefits. Baseline tuning is also unclear: Grad Diff increases F-VQA accuracy from 36.80% to 44.80% in Table 2 and exhibits severe degradation on the 2B model in Table 10. The paper should clarify whether retain training and evaluation samples overlap and evaluate forgetting generalization on unseen queries and broader multimodal capabilities.

Questions for authors

Q1: In Table 2, all four Rouge values reported for Grad Diff on MLLMU-Bench equal the corresponding accuracies divided by 100, and MANU’s GEN accuracies are identical to its QA accuracies. Are these reporting errors? If so, what are the corrected numbers, and how exactly is Rouge computed? Relatedly, since Grad Diff and MANU yield standard models after training, why is their inference latency in Figure 5 above 2 seconds?

Q2: If a client bypasses MS and DA and queries the frozen backbone with unmodified embeddings, can the forget-set information still be recovered? If so, what unlearning guarantee does OpaqueMU provide, and how does it differ in essence from a client-side guardrail such as a refusal classifier?

Reviewer_dKYDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OpaqueMU for multimodal unlearning under the condition that the transformer backbone is not accessible and remain unchanged, while input representation and output logits are accessible. The proposed MS modifies visual representations by learned offsets and textual representations by subspace projection. The proposed DA then reweights the output probabilities. Experiments on MLLMU-Bench and CLEAR show forgetting effectiveness and utility preservation.

Critical strengths
  1. The access setting is interesting and clearly stated. Section 3 studies the setting where clients can modify input representations but don't have access to parameters or gradients. The setting supports unlearning in senarios where full model editing is unavailable. The limitation section also acknowledge the ordinary commercial APIs may not provided the required interfaces.

  2. The ablations provide evidence that combining input and output unlearning techniques can help. In Table 3, combine MS and DA reduce forget set vqa accuracy while increase retrain set accuracy. These results support the practical value of combining input and output interventions.

Critical weaknesses
  1. The claimed limitations of existing works are insufficiently supported. Figure 1 and table 1 claim prompt-based and logits-based methods as lacking multimodal capabilities, but the reason that their original focus on text does not support this claim. Table 2 evaluate the baselines but does not demonstrate cross-modal entanglement causes the limitations. Thus, the motivation of the paper is insufficiently validated.

  2. Limited method novelty. The paper mainly combines existing intervention ideas for a setting where the parameter is not accessible but input and output representations are available. For instance, the output probability reweighting in Section 4.2 already appears in Hiranandani et al. (2025). The main contribution is the combination of visual offsets, text projection, and output correction under the access constraints. Table 3 demonstrates the effectiveness of this combination. However, limited technique insight is provided except for the empirical results. I thus consider this is a useful but incremental contribution.

  3. The geometric analysis does not support the removal of sensitive info. Figure 4 measures the distribution overlap after projecting the representations into a fixed direction. But greater overlap along the direction does not imply the sensitive info is not included in other directions or cannot be recovered from other modalites. The analysis provide limited support for the disentanglement mechanism.

  4. The efficiency advantage is not clearly detailed. Figure 5 shows lower latency but the details such as hardware, model configurations, sequence length, etc are not detailed. Without these details, it is difficult to determine whether the gains are due to the proposed method or diffirences in implementations or generation length. The results cannot sufficiently justify the latency judgement in Table 1, which gives the latency judgement about the entire method classes.

Questions for authors
  1. What evidence supports the claim in Figure 1 and Table 1 that prompt-based and logit-based methods lack multimodal capabilities?

  2. What is the main technical contribution beyond combining existing interventions under the proposed access constraints? Which analysis or comparison explains the benefits of the specific design choices, beyond showing that the combined pipeline is effective?

  3. How does the increased distribution overlap in Figure 4 support the removal of sensitive info?

  4. Please clarify the hardware, model configurations, input and output sequence lengths, and the latency calculation procedure used in Figure 5. Can the results support the general latency claims in Table 1?

Reviewer_SNafOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies multimodal unlearning under a split-access LMaaS setting. The client uploads input embeddings, and the server returns output logits. The authors propose OpaqueMU, which contains two modules: Manifold Sanitization (MS) and Distribution Alignment (DA). These modules modify input embeddings and output distributions to suppress sensitive information.

Critical strengths
  1. The setting is clearly defined, and the backbone remains unchanged. The split-access setting lies between white-box access and strictly API-only access. All unlearning operations run on the client side. Table 1 compares this approach with white-box, context-based, and offset-based methods.

  2. The experiments cover several settings. The paper evaluates two benchmarks, multiple forget ratios, and two models. It also reports Rewrite and Induce attacks, membership inference attacks, adaptive membership inference attacks, costs, parameter sensitivity, and case studies.

Critical weaknesses
  1. Autoregressive generation requires a client–server round trip for each token, with substantial communication and latency costs. The server must wait for the client to run DA, select a token, and upload its modified embedding before generating the next token. The paper does not state whether the reported 0.61 s latency includes network round trips. It also does not discuss the number of requests. Communication costs are reported only for short outputs. Using the per-token logit size inferred from Table 4, about 297 KiB, the reported downloads correspond to roughly 9.5 output tokens for MLLMU-Bench (2.7 MiB) and 31 for CLEAR (9.0 MiB). Under the same assumption, a 500-token output would require about 145 MiB of downloaded logits.

  2. Apparent errors in the main table raise serious concerns about the reliability of the results. In Table 2, MANU’s F-GEN and R-GEN accuracies (41.60% and 40.13%) exactly match its F-QA and R-QA accuracies, while Vanilla’s F-GEN accuracy is only 14.00%. The F-VQA and R-VQA accuracies of δ-unlearning (36.80% and 40.46%) also exactly match Vanilla. This raises the possibility that the baseline had little effect.

Questions for authors
  1. What are the deployment costs over a real network? Under a specified RTT and bandwidth, how many requests and how much logit download traffic does a 500-token output require? What is the end-to-end latency? Given these costs, does this setting have practical deployment value?

  2. How do the authors explain the unusual numbers in Table 2? Are these table-entry errors? Please verify the MANU results and provide corrections where needed. Was the δ-unlearning offset applied correctly? Why do its F-VQA and R-VQA accuracies exactly match Vanilla?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.