- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OpaqueMU, a multimodal machine unlearning framework under a split-access LMaaS setting, where the Transformer backbone remains frozen while input representations and output logits are accessible to the client. It combines Manifold Sanitization (MS), which applies visual embedding offsets and textual subspace projections, with Distribution Alignment (DA), which reweights output distributions to suppress target knowledge while preserving retained utility. Experiments on MLLMU-Bench and CLEAR report favorable forgetting–utility trade-offs, improved privacy robustness, and reduced inference latency.
Critical strengths
S1. Interesting problem setting. Exploring multimodal unlearning without modifying the Transformer backbone is a potentially useful direction for model services with restricted access.
S2. Modality-aware design. The framework separately handles visual and textual representations before fusion and combines representation-level intervention with output-level correction.
S3. Comprehensive evaluation. The paper covers two benchmarks, multiple forgetting ratios, two model architectures, ablation studies, efficiency analysis, and robustness evaluations.
Critical weaknesses
W1. Unclear practical motivation and unlearning guarantees. The split-access setting assumes client access to input representations and full logits but not the backbone, which is uncommon in commercial APIs and lacks a concrete deployment motivation. Since the backbone remains unchanged, the method provides no evidence of model-level knowledge removal, and target information may remain accessible by bypassing the client-side modules. The authors should clarify the threat model and distinguish behavioral suppression from model-level unlearning.
W2. Questionable result reliability and efficiency comparisons. In Table 2, all four Grad Diff Rouge scores exactly equal the corresponding accuracies divided by 100, while MANU's GEN accuracies duplicate its QA accuracies, suggesting possible reporting errors. Figure 5 reports over 2 seconds of inference latency for Grad Diff and MANU, despite their lack of additional inference-time modules, compared with 0.61 seconds for OpaqueMU. The approximately 0.2-second training times per epoch and unclear inclusion of multi-megabyte logit transmission (Table 4) further require clarification.
W3. Limited novelty and insufficient evidence of component synergy. The method primarily combines established techniques, including representation offsets, subspace projection, adversarial alignment, and logit reweighting, without sufficiently discussing related concept erasure approaches such as INLP and LEACE. In Table 3, the full model and DA-only variant achieve identical F-QA (31.20%) and F-GEN accuracies (6.00%), while MS-only achieves higher R-QA accuracy (44.21% vs. 36.97%). Stronger comparisons with simpler methods under identical access assumptions are needed to justify the proposed combination.
W4. Insufficient evidence of selective forgetting. Aligning forget and retain representations does not necessarily remove sensitive information, and pushing output distributions toward uniformity may merely reduce confidence. On CLEAR, F-VQA drops from 95.21% to 92.55%, while R-VQA also drops from 98.23% to 95.74%, suggesting limited selectivity. Moreover, Table 13 shows that OpaqueMU still reveals sensitive attributes in a forgotten sample. Stronger evaluations of information recoverability are needed.
W5. Incomplete evaluation protocol and potentially unfair baselines. The compared methods have different model-access privileges, making it difficult to isolate the proposed method's benefits. Baseline tuning is also unclear: Grad Diff increases F-VQA accuracy from 36.80% to 44.80% in Table 2 and exhibits severe degradation on the 2B model in Table 10. The paper should clarify whether retain training and evaluation samples overlap and evaluate forgetting generalization on unseen queries and broader multimodal capabilities.
Questions for authors
Q1: In Table 2, all four Rouge values reported for Grad Diff on MLLMU-Bench equal the corresponding accuracies divided by 100, and MANU’s GEN accuracies are identical to its QA accuracies. Are these reporting errors? If so, what are the corrected numbers, and how exactly is Rouge computed? Relatedly, since Grad Diff and MANU yield standard models after training, why is their inference latency in Figure 5 above 2 seconds?
Q2: If a client bypasses MS and DA and queries the frozen backbone with unmodified embeddings, can the forget-set information still be recovered? If so, what unlearning guarantee does OpaqueMU provide, and how does it differ in essence from a client-side guardrail such as a refusal classifier?