- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how long-tailed learning strategies affect subsequent machine unlearning. The authors conduct a systematic empirical study across vision and language tasks, examining resampling, data augmentation, and loss reweighting under different forgetting settings. The evaluation covers CIFAR-10, CIFAR-100, Tiny-ImageNet, and long-tailed variants of TOFU and ELUDe, involving multiple unlearning methods and strategy-matched retraining references. The experimental results suggest that improving tail performance often leads to larger deviations from retraining, while strategies with similar training exposure can exhibit different forgetting behaviors. The paper further identifies strategy-dependent effects under severe data imbalance, variations across target classes, and interactions between learning strategies and unlearning algorithms.
Critical strengths
-
The paper investigates an important and relatively underexplored aspect of machine unlearning by examining how training-time long-tailed learning strategies influence subsequent forgetting. The findings highlight the importance of considering training configurations when evaluating unlearning methods.
-
The authors evaluate multiple learning strategies and unlearning methods across vision and language benchmarks, with additional backbone robustness experiments. The use of strategy-matched retraining references and the analysis of strategy–method interactions provide useful empirical insights.
Critical weaknesses
-
The novelty of the paper could be better clarified, particularly in relation to existing studies on how training strategies affect machine unlearning.
-
The paper mainly presents empirical observations, while the underlying mechanisms explaining the observed differences remain insufficiently explored.
-
The experimental settings may introduce certain confounding factors, making it difficult to fully attribute the observed differences to long-tailed learning strategies.
-
The evaluation relies on a relatively limited set of metrics, which may not fully capture the effectiveness and reliability of machine unlearning.
-
Although the experiments cover multiple datasets and models, further validation on more diverse and realistic long-tailed scenarios would strengthen the generalizability of the conclusions.
Questions for authors
-
Could the authors further clarify the main contributions of this work compared with existing studies on training strategies and machine unlearning?
-
To what extent might the observed differences be influenced by the hyperparameter settings of the unlearning methods?
-
Could the authors provide further insights into the underlying factors that explain why different long-tailed learning strategies lead to different unlearning behaviors?
-
How sensitive are the main conclusions to the choice of evaluation metrics? Would alternative metrics lead to similar observations?
-
To what extent can the findings be generalized to more diverse datasets, model architectures, and real-world long-tailed distributions?
9 comments
References
-
Buda, M., Maki, A., & Mazurowski, M. A. (2018). A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106, 249-259.
-
Ahn, S., Ko, J., & Yun, S. Y. (2023). Cuda: Curriculum of data augmentation for long-tailed recognition. arXiv preprint arXiv:2302.05499.
-
Kaya, Y., & Dumitras, T. (2021, July). When does data augmentation help with membership inference attacks?. In International conference on machine learning (pp. 5345-5355). PMLR.
-
Falcao, A. M., & Cordeiro, F. R. (2025, September). Data Augmentation Improves Machine Unlearning. In 2025 38th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) (pp. 1-2). IEEE.
-
Wu, R., Garov, K., & Chaudhuri, K. (2026, April). Learning-time encoding shapes unlearning in llms. In International Conference on Learning Representations (Vol. 2026, pp. 87040-87067).
-
Chen, M., Gao, W., Wang, C., Liu, G., Abdelmoniem, A. M., & Peng, K. (2025, August). From Expansion to Retraction: Long-tailed Machine Unlearning via Boundary Manipulation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 (pp. 215-226).
[Q2/W3]
We agree with the reviewer that HP selection and potential confounding factors warrant further investigation. We selected unlearning HPs under Standard and fixed them across learning strategies to examine the effect of preceding training history under a common unlearning procedure, rather than independently optimized configurations. Similar fixed-protocol comparisons appear in Wu et al. (ICLR 2026) and Falcao and Cordeiro (SIBGRAPI 2025). Nevertheless, we agree that strategy-specific HP selection and potential confounding factors warrant further investigation.
We therefore conducted additional strategy-specific HP tuning on CIFAR-100-LT, Tiny-ImageNet-LT, TOFU-LT, and ELUDe-LT, covering four vision and five LLM unlearning methods. The selected configurations were evaluated on held-out conditions without retuning.
The two central cross-modality findings are reproduced after strategy-specific HP selection: improving Tail performance does not necessarily make subsequent unlearning easier, and matched Tail exposure does not guarantee similar forgetting behavior.
Representative results show that stronger Tail performance continues to be accompanied by larger forgetting discrepancies after tuning:
| Setting | Strategies | Pre-unlearning Tail performance ↑ | Tuned forgetting gap ↓ |
|---|---|---|---|
| CIFAR-100-LT, IF=50 | Standard / CB Loss | Tail TA: 10.84 / 11.78% | ΔT: 27.00 / 39.25 pp |
| Tiny-ImageNet-LT, IF=10 | Standard / ROS | Tail TA: 24.78 / 26.16% | ΔT: 17.95 / 28.28 pp |
| TOFU-LT | Standard / ROS | Seen P: 0.452 / 0.958 | Seen Gap: 0.069 / 0.259 |
| ELUDe-LT | Standard / Paraphrase | Seen P: 0.962 / 0.998 | Seen Gap: 0.107 / 0.252 |
The exposure-controlled comparisons also reproduce the second finding:
-
Vision: Tail ColorJitter and CB Loss use the same original training examples at the same sampling frequency, yet retain different Tail forgetting gaps after tuning (18.75 vs. 39.25 pp).
-
LLMs: Exposure-matched ROS and Paraphrase exhibit different Seen Tail Probability Gaps on TOFU-LT (0.259 vs. 0.126) and ELUDe-LT (0.199 vs. 0.252), preserving the original benchmark-dependent reversal.
The findings also persist under both Fixed and Tuned HP protocols in the additional financial and medical experiments. As detailed in our response to [Q5/W5], Credit Default reproduces the two central findings, while BloodMNIST-LT also reproduces the imbalance-dependent differences observed in vision (Section 5; Appendix B.7).
We additionally examined two potential confounding factors in the LLM experiments: unequal optimizer-update counts and differences in forget-input composition. Under the original epoch-based protocol, ROS and Paraphrase involve five times as many unlearning optimizer updates as Standard and Loss Weighting. We therefore conducted 120 additional runs across two benchmarks, three seeds, and five unlearning methods using two controls:
-
Step-matched: Preserves augmented forget inputs while matching optimizer-update counts to Standard.
-
Common-source: Uses identical original forget inputs and matched update budgets across strategies while retaining their distinct pre-unlearning checkpoints.
The central strategy-dependent differences persist under the stronger Common-source control, despite identical forget inputs and optimizer-update budgets.
| Benchmark | Standard | ROS | Loss Weighting | Paraphrase |
|---|---|---|---|---|
| TOFU-LT | 0.049 | 0.391 | 0.306 | 0.209 |
| ELUDe-LT | 0.121 | 0.267 | 0.288 | 0.204 |
Seen Tail Probability Gap under Common-source, averaged over five unlearning methods and three seeds. Lower values indicate closer agreement with matched retraining.
Step matching reduces the Held-out Tail Probability Gaps of ROS/Paraphrase from 0.088/0.066 to 0.022/0.028 on TOFU-LT, and from 0.087/0.127 to 0.034/0.043 on ELUDe-LT. Thus, unequal optimizer-update budgets explain part of the original Held-out discrepancies, but do not account for the persistent strategy-dependent Seen Tail differences under matched inputs and budgets.
These experiments show that the main findings are not explained solely by Standard-selected HP transfer, unequal optimizer-update counts, or forget-input composition. Strategy-dependent unlearning differences persist after HP adaptation and under the stronger LLM controls.
We thank the reviewer for the thoughtful assessment and constructive feedback. Since the questions closely correspond to the critical weaknesses, we address each weakness [W#] together with its corresponding question [Q#] below.
[W1/Q1]
We appreciate the reviewer's request to clarify how our contribution differs from existing studies on training strategies and machine unlearning. Our novelty lies in systematically characterizing how preceding long-tailed learning strategies shape subsequent machine unlearning, rather than simply showing that training-time choices affect unlearning. Prior studies have investigated long-tailed recognition, privacy leakage, and the effects of training-time choices on unlearning. However, their primary research questions differ from ours:
Our study identifies two central findings across vision and language, together with domain-specific observations:
Cross-modality findings: Improving Tail performance does not necessarily make subsequent unlearning easier, and the same Tail exposure does not guarantee similar forgetting behavior (Table 1; Figures 2–3).
Domain-specific findings: In vision, severe imbalance amplifies strategy-dependent differences in Tail forgetting, and improved Tail forgetting under CB Loss can coincide with larger Head discrepancies (Figure 6; Table 2). In LLMs, learning strategies redistribute forgetting discrepancies across Seen and Held-out Tail QA and change the relative behavior of unlearning methods (Figures 7–8).
These findings establish that unlearning behavior depends not only on the unlearning method, but also on how the deleted data were learned. Unlike prior studies focused on individual training interventions or new unlearning algorithms, our work systematically examines these interactions under long-tailed distributions across vision and language.
We further reproduce the main findings under strategy-specific HP selection (Appendix B.5) and extend the evaluation to financial and medical data (Section 5; Appendix B.7).