- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes -Unlearning, which freezes the LLM and learns a bias on the attention logits between groups of similar tokens, weakening the attention paths that produce the forget data while preserving those used by the retained data. Experiments show that it outperforms existing unlearning methods on standard benchmarks.
Critical strengths
Strengths
- The paper tackles an important and practical problem. Removing targeted knowledge from LLMs while preserving their general capabilities is an interesting problem closely related to AI safety.
- The proposed method consistently outperforms existing unlearning baselines across multiple models and forget ratios.
Critical weaknesses
Weaknesses
-
The theory renames known quantities without adding anything new to the method. "Token rendering" and "renderability" are just the output distribution and the log-probability, and Prop.1 is a first-order Taylor expansion that holds for any loss on log-probabilities, as used in most prior analyses of fine-tuning dynamics. Likewise, the transmittance , said to be borrowed from volume rendering, is just a learnable bias subtracted from the attention logits. The volume rendering and NeRF background is unnecessary and distracting.
-
The claimed efficiency is unclear. Training still requires backpropagation through the full model, as with LoRA, and the table has entries, far more than the rank-8 LoRA used by the baselines.
Questions for authors
Questions
- The issue raised by Proposition 1 seems to apply equally to the proposed method, since the relation depth is trained with the same gradient-ascent loss, and its effect on the retained data is still not explicitly controlled. Why should updating be better than updating the model weights with LoRA?
- Since the authors claim that their method is efficient, could they compare training time, peak GPU memory, and inference overhead with the LoRA baselines? This comparison should also account for whether the method can use standard fused attention kernels such as FlashAttention.
Dear Reviewer GWJz,
Reviewing many assigned papers is a time-consuming task and so we would like to sincerely thank you again for your time and effort to review the paper.
As the rebuttal and discussion period is approaching its end, we would be grateful if you could kindly take a moment to review our responses and additional experimental results. We have carefully addressed your concerns and hope that our clarifications have helped resolve the issues raised in your review.
If any concerns remain, we would be more than happy to provide further clarification or additional evidence. We would greatly appreciate your updated assessment in light of our responses.
Thank you again for your valuable time and consideration.
Best regards,
The Authors