- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CoLU performs gradient ascent on the forget set, weighting each target token by a power of its current probability, while adding a cross-entropy regularizer on the retain set. Each run updates one complete Transformer block. BLS briefly trains every layer on a data subset, shortlists three candidates, then retrains each candidate from the original checkpoint on the full data and selects the final model by NFR. Experiments cover Phi-1.5 and Llama-2-7B on TOFU and the Books and News domains of MUSE, with results for effectiveness, screening time, and memory use.
Critical strengths
- The parameter constraint has practical value. Updating one layer substantially reduces peak memory. The paper also reports end-to-end screening cost rather than only the cost of a single training run.
- The layerwise analysis is informative. Full sweeps show that effective layers cluster near the beginning and end of the models. BLS includes at least one of the top three full-sweep layers in all six TOFU settings, making low-cost layer screening a promising direction.
Critical weaknesses
- Model selection and final evaluation are not independent. FS and BLS use NFR to select layers and models, while NFR is computed from the same forget and retain metrics later reported as final results. MUSE additionally selects the best checkpoint across epochs. Without independent validation and test splits, the selected best values are difficult to interpret.
- The method's contributions are not isolated. The token probabilities are not validated as calibrated confidence estimates, and the experiments omit WGA, uniform weighting, calibrated weighting, and controls that apply the same BLS procedure and retain regularization to baselines. The current evidence cannot determine whether the gains come from probability weighting, retain regularization, or the capacity constraint of updating one layer.
Questions for authors
- If layers, hyperparameters, and epochs are selected on a fully independent validation split and evaluation is performed only once on a held-out test split, do CoLU's gains over SatImp, SimNPO, and Retrain remain significant, with confidence intervals?
- Do the BLS screening subset, NFR evaluation set, official test set, and hyperparameter-tuning data overlap? For MUSE, which split and metric are used to select the best checkpoint?
- Under the same search budget and across multiple random seeds, does CWA consistently outperform WGA, uniform, rank-based, and calibrated weighting, as well as baseline objectives equipped with the same BLS procedure and retain regularization?
5 comments
Rebuttal to reviewer tvji Cont.
Q3. Comparisons with alternative objectives and weighting schemes
Uniform weighting corresponds to GA in our formulation. Our additional three-seed results on Phi-1.5 compare CoLU with GA, NPO, SatImp, WGA, rank-based weighting, and calibrated CWA, all using BLS. CoLU achieves the highest mean NFR among the evaluated methods in all three forget splits.
| Method | Forget-1% NFR ↑ | Forget-5% NFR ↑ | Forget-10% NFR ↑ |
|---|---|---|---|
| WGA + BLS | 92.6228 ± 0.4845 | 86.9112 ± 1.2748 | 89.6009 ± 0.8902 |
| GA + BLS (uniform weighting) | 70.5212 ± 4.6109 | 70.5613 ± 3.2018 | 67.0881 ± 1.2646 |
| NPO + BLS | 83.9244 ± 2.0538 | 82.5425 ± 0.7814 | 82.0721 ± 2.2633 |
| SatImp + BLS | 89.0712 ± 3.1300 | 88.9464 ± 2.5044 | 81.6678 ± 0.3125 |
| Rank-based + BLS | 84.1644 ± 5.6051 | 76.8766 ± 3.4608 | 75.6105 ± 1.7871 |
| Calibrated CWA + BLS | 88.0661 ± 2.0012 | 76.2384 ± 1.8541 | 81.0271 ± 0.4637 |
| CoLU (BLS) | 95.0355 ± 0.4693 | 90.0832 ± 0.9183 | 91.6610 ± 0.1553 |
CoLU achieves the highest mean NFR among all compared BLS-based methods across the three Phi-1.5 forget splits: 95.04, 90.08, and 91.66. These results support its improved forgetting-retention trade-off relative to the evaluated objectives and weighting schemes.
Conclusion
The component ablations support the roles of layerwise updates and retain regularization, and the additional retain validation/test split supports validation-based layer selection. The three-seed BLS comparisons show the highest mean NFR for CoLU among the evaluated methods across all three Phi-1.5 splits, supporting its combination of CWA, retain regularization, and single-layer updates with BLS for layer selection. We respectfully ask you to reconsider your assessment and raise the score in light of these clarifications and additional evidence.
Sources
[1] Wang et al. Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond. ICLR 2025. Paper.
[2] Yang et al. Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning. ICML 2025. Paper.
[3] Fan et al. Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning. NeurIPS 2025. Paper.
Dear Reviewer tvji, as the discussion period draws to a close, we would appreciate your consideration of our additional validation-based layer-selection study and three-seed BLS comparisons addressing your evaluation and component-analysis concerns. We hope this evidence supports a more favorable assessment and an increase in your score. Thank you for your time.
Rebuttal to reviewer tvji
Thank you for your detailed questions about model selection and the roles of the individual components. We clarify the evaluation protocol and provide an additional validation-based layer-selection study below.
W2. Evidence for the individual components
We respectfully disagree with this characterization of the experimental evidence. Table 1 includes full-model CWA, which uses CWA and retain regularization while updating the full model. CoLU (BLS) improves both forget and retain ES over full-model CWA in five of the six TOFU settings and over full-model SatImp in all six settings. This comparison supports the benefit of combining CWA with layerwise updates. Table 3 further compares layerwise unlearning with full-model unlearning under GA, NPO, SatImp, and CWA on Phi-1.5 Forget-5%. With each layerwise result selected by NFR from a full sweep, layerwise unlearning achieves higher NFR for all four objectives. Finally, Table 11 in Appendix C shows that removing retain regularization substantially reduces retention. Together, these comparisons support the roles of each component in CoLU.
Q1(W1). Validation-based layer selection
On Phi-1.5, we partitioned the original 400-example retain evaluation set into 200 validation examples and 200 test examples. For layer selection, NFR uses retain ES from the validation subset; the test subset is excluded from this selection and used to evaluate retained performance. The forget evaluation set remains unchanged. The table reports forget ES on this set, retain ES on the 200 test examples, and NFR computed from these reported scores.
CoLU (BLS) achieves the lowest forget ES and highest NFR among the compared methods in all three splits. Its NFR exceeds SatImp by 4.80, 1.95, and 9.96 points, and SimNPO by 12.85, 7.61, and 13.12 points, respectively. On Forget-1% and Forget-10%, its retain ES is within 0.006 of SimNPO, while its forget ES is substantially lower. These results support the effectiveness of validation-based layer selection when retained performance is evaluated on separate test data.
Q2. Evaluation data and MUSE checkpoint selection
Unlearning targets information in the forget set, so we evaluate forgetting on the data designated for removal (train=test). The retain evaluation set is separate from the retain data used for optimization. In the original experiments, BLS uses retain evaluation data when scoring candidate layers. Performance-based tuning on hyperparameters also appears in previous methods such as WGA [1], which uses UWC and reports forget and retain ES across hyperparameter sweeps. Our additional study in Q1 separates the retain data used for layer selection from those used for final evaluation which further supports the effectiveness of CoLu.
For MUSE, layer and checkpoint selection use NFR from Eq. (3), computed on the final evaluation sets. NFR equally weights the mean relative reduction in forget-set VerbMem and KnowMem and normalized retain-set KnowMem. PrivLeak is reported but not used for selection. We use this NFR to rank BLS candidates and select the best checkpoint across the eight full-data unlearning epochs. We follow prior MUSE studies in selecting checkpoints based on evaluation performance. SatImp [2] selects among ten epoch checkpoints, prioritizing PrivLeak, while SimNPO [3] evaluates after each epoch and selects the best-performing model.