- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes LTU, a zero-dependency framework for removing both cross-modal and uni-modal knowledge from MLLMs. LTU uses linear-delta to efficiently identify critical tokens and optimize model unlearning. Experimental results show superior performance over existing methods.
Critical strengths
- The paper addresses a critical gap in MLLM unlearning by removing both cross-modal and uni-modal knowledge without relying on external resources like retain sets or auxiliary models, which is a significant innovation.
- The linear-delta method introduces a first-order approximation to reduce computational overhead and perturbation bias compared to brute-force scoring, enhancing efficiency and accuracy.
- Extensive experiments on CLEAR and MLLMU benchmarks demonstrate LTU’s superior performance in forget quality and model utility, with robustness against relearning and quantization attacks.
- The framework is scalable to larger models, as shown in the experiments, and the theoretical analysis supports the validity of the linear-delta approach.
Critical weaknesses
- The paper lacks detailed discussion on the limitations of linear-delta in scenarios with highly non-convex loss landscapes, as highlighted in Appendix G. The empirical Fisher approximation used in second-order methods might introduce noise, but this is not fully addressed.
- The experimental results for uni-modal unlearning (e.g., Table 13 and Table 15) show lower forget quality compared to cross-modal results, suggesting potential weaknesses in handling uni-modal data. The reasons for this discrepancy are not thoroughly analyzed.
- The theoretical proofs in Appendix F and G assume local smoothness and convexity, which may not hold in practice. While the paper cites these assumptions, it does not validate them empirically or discuss their implications for real-world applications.
Questions for authors
- The paper does not adequately discuss linear-delta’s limitations under highly non-convex loss landscapes and fails to fully examine noise from the empirical Fisher approximation adopted by second-order methods.
- The uni-modal unlearning results are worse than cross-modal ones, yet the paper has no in-depth analysis explaining this performance gap.
- The theoretical proofs rely on local smoothness and convexity assumptions that may not hold in practice, with no empirical validation or discussion of real-world impacts.
2 comments
Dear Reviewer 1KBD,
Thank you very much for your time and helpful comments. We hope our response addresses your concerns. As we are approaching the rebuttal deadline, please let us know if you have any further questions or concerns.
Sincerely, Authors of Linear Token Unlearning.
We appreciate the reviewer for thoughtful comments. We would like to offer clarifications for each point.
Weakness 1 and Question 1
Linear-delta predicts the effect of replacing a token from the gradient at the original token, so its error grows with the curvature of the loss and with the distance to the replacement. We conduct an experiment to show that curvature is less affected by the choice of the replacement. LTU uses a set of different tokens for replacements (‘#’ or ‘[UNK]’) and chose them randomly for swapping the tokens. If the loss curvature is highly non-linear then each replacement would result in a different output distribution and selected tokens will vary. We ran an experiment to verify if the same set of tokens are selected when different replacements are used. We score the vision and text tokens with Linear-delta and brute-force when the prefix was swapped with ‘[UNK]’ exclusively. Then we report the overlap of top-20 tokens.
The result shows that the loss curvature, although non-convex, has minor contribution to error. While this cannot show the entire loss landscape, the result demonstrates that the loss is more linear to the perturbation and hence offers a solid ground for approximation.
Weakness 2 and Question 2
LTU’s uni-modal forget quality is not systematically lower than its cross-modal forget quality. It is lower in 5 of 12 settings and higher in 7 settings. Refer to the table below. In Table 15 (MLLMU 10%), uni-modal forget quality is more than twice the cross-modal value for both backbones (LLaVA 0.357 vs. 0.165; Qwen 0.428 vs. 0.134). Where it is lower, the gap is mostly small (e.g., MLLMU 1% with Qwen: 0.987 vs. 0.998). Moreover, the gaps are also small in terms of the underlying test statistic. Forget quality is the p-value of a KS test, which takes discrete and steeply changing values: on CLEAR 1%, LLaVA’s cross-modal 0.874 and uni-modal 0.766 correspond to KS distances of 0.114 and 0.125 from the retrained model, which is a difference of about one sample. In all settings, LTU’s uni-modal forget quality remains the highest among the compared methods.
Weakness 3 and Question 3
We first clarify the assumptions.
Appendix F relies only on local smoothness along the perturbation path of each token (Assumption 1); it does not assume convexity. Convexity enters only through the empirical-Fisher approximation in Appendix G, which we discuss under weakness 1.
Assumption 1 requires that, as a token moves along the path to its replacement, the gradient changes at a bounded rate β. Remark 1 left β unestimated. We now estimate it directly. For each tested token we move it 25%, 50%, 75% and 100% of the way to the # replacement, we recompute the gradient, and take the largest change in the gradient per unit distance.
The median over tokens is β = 0.130 for visual tokens and β = 0.070 for text tokens. Assumption 1 therefore holds with finite, measured constants along the actual perturbation paths. The values also show that per unit distance, the loss curves less for text tokens than for visual tokens, which is consistent with linear-delta being less reliable for text tokens (Weakness 1). The result shows that LTU produces small βs, which means smaller error bounds.