- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates interaction-level machine unlearning in Semantic ID-based generative recommendation. The authors identify a previously underexplored failure mode termed SID-induced unlearning spillover, where removing the influence of forgotten interactions can unintentionally affect neighboring items that share Semantic ID tokens. Importantly, these local effects may remain largely invisible under conventional global recommendation metrics. To address this issue, the authors propose Neighborhood-Aware Unlearning (NAU), which combines item-level forgetting, retain-set optimization, target separation, and neighborhood probability mass control. By adjusting the direction of neighborhood mass regularization, NAU aims to preserve legitimate neighboring items during spam removal or suppress semantically related items during unwanted-item removal. Experiments on three Amazon recommendation datasets evaluate NAU against general-purpose and recommendation-specific unlearning baselines. The evaluation covers multiple SID tokenization settings, two deletion scenarios, and both global and neighborhood-level recommendation behavior. The results demonstrate promising forgetting-utility trade-offs and improvements in local neighborhood utility, with substantially lower computational costs than retraining.
Critical strengths
-
The paper highlights an important limitation in existing recommendation unlearning evaluations: maintaining overall recommendation utility does not necessarily mean that the performance on items related to forgotten interactions is preserved. This concern is especially relevant to Semantic ID-based generative recommendation, where multiple items may share autoregressive token prefixes. The authors provide empirical evidence showing that recommendation quality within these local neighborhoods can change considerably, even when the overall NDCG remains largely stable.
-
NAU offers a straightforward way to control how probability mass is distributed among neighboring items during unlearning. The proposed objective is relatively simple, does not depend on a specific model architecture, and avoids second-order optimization or changes to the underlying Semantic IDs. The experimental evaluation is also fairly extensive. Beyond comparisons with several unlearning baselines, the authors examine two deletion scenarios, different SID tokenizers, the contributions of individual loss components, neighborhood utility, probability redistribution, and computational efficiency.
Critical weaknesses
-
The proposed objectives and evaluation metrics do not convincingly demonstrate interaction-level unlearning. My main concern is that the paper does not clearly distinguish removing the influence of forgotten training interactions from simply suppressing certain items or adjusting their recommendation exposure.
- First, the forgetting objective applies gradient ascent to the negative log-likelihood of forgotten interactions, thereby reducing the probability of the corresponding target items under their original histories. However, lower prediction probabilities alone do not necessarily indicate that the influence of these interactions has been removed from the model.
- Second, the spam-removal evaluation focuses on the difference in target exposure rates between the unlearned and retrained models. While this is a reasonable measure of whether the promotion effect introduced by poisoning has been reduced, similar aggregate exposure rates do not guarantee that the models produce comparable predictions for individual users.
- Third, the unwanted-item removal setting appears to adopt a different definition of forgetting, which is closer to behavioral suppression. Specifically, the goal is to prevent requesting users from receiving recommendations of unwanted items, even when retraining without the deleted interactions would not necessarily lower their exposure. The paper should more clearly distinguish this objective from interaction-level unlearning.
-
The experiments do not adequately establish whether the observed neighborhood effects arise specifically from shared Semantic IDs or from more general neighborhood regularization. This raises questions about whether NAU directly addresses the structural dependencies introduced by shared SID tokens.
- First, the neighborhoods used for training and evaluation are defined differently. Section 4 selects the top-8 most similar items based on cosine similarity in the pre-quantization embedding space, whereas the neighborhood evaluation considers items sharing the first SID token with the forgotten target. This inconsistency makes it difficult to assess whether the proposed objective effectively protects the neighborhoods most affected by SID sharing.
- Second, the neighborhood mass objective does not explicitly preserve the probability distribution of the retrained model. Eq. (3) only regulates the total probability mass assigned to the selected neighborhood. With a positive objective weight, minimizing this term encourages the model to allocate more probability mass to neighboring items, but does not control how that mass is distributed among them.
- Third, the current comparisons do not clearly isolate the benefits of the proposed neighborhood construction. Although Appendix H includes useful analyses of shared SID prefixes and restricted parameter updates, the paper lacks direct comparisons with simpler neighborhood regularization methods under comparable neighborhood sizes and training budgets. Such experiments would help determine whether the gains come from addressing SID-specific dependencies or from the broader effect of encouraging neighborhood probability mass.
Questions for authors
-
Q1. How do the authors distinguish the actual removal of forgotten training information from simply suppressing items associated with forgotten interactions? Could the authors provide additional comparisons between the unlearned and retrained models at the individual prediction or probability-distribution level, rather than focusing mainly on aggregate target exposure rates?
-
Q2. In the unwanted-item removal setting, are the deleted interactions also excluded from the historical contexts of subsequent retained training samples? If they remain in these histories, how can the retrained reference be considered free from the influence of the deleted interactions?
-
Q3. Why are the training neighborhoods defined using the top-8 most similar items in embedding space, while spillover is evaluated based on first-token SID sharing? Could the authors examine how much these two neighborhood definitions overlap and compare NAU with alternatives based on shared SID prefixes, random item selection, and embedding similarity, while keeping neighborhood sizes consistent?
6 comments
Author response: Part 2/3
W1 / Q1 (iii). Per prediction
Response. In short: In the evaluated Beauty mid-popularity case, NAU matches retraining on whether the promoted item is shown for every query. Per-query utility differences, however, are dominated by training randomness and do not clearly distinguish the methods.
New experiment: per-query agreement (Beauty, mid-popularity target): the poisoned model shows the promoted item where retraining does not on 0.8% of queries, NAU on none. For per-query NDCG (Beauty seed 2, three retrained models), a second retrained model differs from the reference by 2.95 to 3.17 points per query, and the poisoned model (3.15), Forget+Repair (3.10), and NAU (3.11) all lie at that level. At the probability level, Table 19 compares, per held-out history, the log-probability of the target and its SID neighbors with the retrained model; at our selected setting the target ends about 4 nats below its retrained value (some over-suppression, see Reviewer M2BR, W3).
W1, third point. Sensitive item deletion as behavioral suppression
Response. We agree and make this explicit in Section 2, which also renames the scenario "sensitive item deletion" (unwanted-item removal in the submission). It combines an interaction deletion, whose reference is retraining (Schelter et al., 2023), with a behavioral requirement from the request, that the user no longer be recommended the category. We state suppression for the requesting users as the goal and report the retrained exposure as a reference. With the refined evaluation (Q2), retraining lowers this exposure on Beauty and Sports but not on Toys, so the second part does not follow from the first.
New experiment: a scenario where deletion and suppression coincide. A brand takedown, in which all interactions with one Beauty brand (95 items, 933 users) are deleted for every user, so the retrained model never recommends it and any remaining exposure is distance to retraining. NAU, with its spam weights, comes closest to retraining at full utility (mean over 5 seeds; baselines with their sensitive item deletion settings):
| Brand takedown (Beauty) | Brand exposure, its buyers (UHF@10) | Brand exposure, other users (UHR@10) | |
|---|---|---|---|
| Retrained | 0 | 0 | 100 |
| Before deletion | 34.6% | 2.82% | 101.7% |
| SCIF | 34.5% | 2.81% | 101.7% |
| Finetune | 23.3% | 1.15% | 100.6% |
| Kookmin | 3.96% | 0.04% | 77.9% |
| SEIF / Forget only / Fanchuan | 0 | 0 | 26.9 / 0.0 / 14.8% |
| Forget+Repair | 1.12% | 0.04% | 96.4% |
| NAU | 0.14% | 0.01% | 96.4% |
NAU leaves eight times less of the brand's exposure than Forget+Repair at identical utility (lower on 4 of 5 seeds, equal on the fifth), while the methods that remove the brand completely destroy the model.
Q2. Deleted interactions in later histories
Response. Yes, deleted interactions are excluded from the retained training data in both scenarios. For spam removal, all interactions of the fake users are forgotten. For sensitive item deletion, retraining uses the requesting users' sequences after removing the deleted interactions. On the unlearning side, all sensitive training examples of requesting users are in the forget set.
Correction to the evaluation. During our experimental audit, we identified that some sensitive-item evaluation runs still included deleted interactions in the requesting users' test histories. We corrected this consistently for all methods, including retraining, and updated the affected tables. Training, unlearning, and model selection are unchanged. The spam-removal experiments are unaffected.
Author response: Part 3/3
W2 / Q3. Are the effects SID-specific?
Response. Yes, see our response to Reviewer M2BR (W1): the SASRec contrast, the random-SID control on all three datasets, the 70 to 75% overlap of the two neighborhood definitions, and SID-prefix vs. random vs. embedding neighbors at equal size and budget (random: no gain; SID-prefix +10.2; embedding +23.5).
Eq. 3 controls only the total mass. You are right. Eq. 3 sets how much probability the neighborhood receives in total, not how this probability is spread over its items. Our new analysis shows what this means in practice: the added probability goes mostly to the eight items the term is trained on, which at rise above their retrained level, while no statistically resolved additional effect is observed for the remaining neighbors (Reviewer M2BR, W1). The term thus counteracts the damage that the forget and separation terms cause through the prefix tokens the neighbors share with the target, for the nearest items, and does not reach bigger neighborhoods. A term that matches the distribution within the neighborhood is a natural extension, which we state as future work.
New experiment: neighborhood size. Larger neighborhoods do not help: on Beauty validation, the neighborhood gain peaks at and falls for larger as the region becomes less local, which is why we use (Beauty, 3 targets x 5 seeds; whole-set values all within the retraining noise floor):
| Whole set / (test) | Neighborhood gain over Forget+Repair (validation, points) | |
|---|---|---|
| 2 | 0.075 / 96.1 | +10.4 [+3.2, +17.9] |
| 4 | 0.071 / 96.0 | +9.9 [+3.1, +17.0] |
| 8 | 0.081 / 96.2 | +17.1 [+10.4, +24.5] |
| 16 | 0.076 / 96.4 | +15.5 [+9.0, +22.0] |
| 32 | 0.076 / 96.3 | +12.3 [+6.8, +18.3] |
| 64 | 0.092 / 96.2 | +8.6 [+4.6, +12.9] |
New experiment: a simpler neighborhood regularizer at equal size and budget (Beauty, 15 runs). In short: up-weighting the same neighbors through the retain data, the simplest alternative, cannot match NAU, because it costs global utility. We tested rehearsal, i.e., Forget+Repair whose retain batches are enriched with retained examples whose next item is one of the same 8 neighbors:
| Whole set | ||
|---|---|---|
| Rehearsal, 25% of each retain batch | 88.4% | 0.100% |
| Rehearsal, 50% | 82.1% | 0.158% |
| NAU | 96.2% | 0.081% |
Rehearsal costs 8 to 14 points of global utility and ends only about as close to retraining as Forget+Repair (0.108%), while NAU's mass term acts without a global cost. Together with the random-neighbor variant (Reviewer M2BR, W1), this separates the contribution of the neighborhood construction from that of the regularizer.
Changes to the paper
- Section 2: sensitive item deletion defined as an interaction deletion plus a behavioral requirement; scenario renamed.
- Section 4 and Appendix C: clarified the forget/retain construction; corrected sensitive-item evaluation histories and updated affected results.
- Section 7 and Appendix: Eq. 3 controls only the total mass, with the k sweep; distribution matching as future work.
- Appendix: membership inference, per-query agreement, brand takedown, SASRec and random-SID controls, neighbor source, rehearsal.
Author response: Part 1/3
We thank the reviewer for the precise framing. We agree that a lower probability alone does not show removal.
Summary
Table, figure, and appendix numbers refer to the originally submitted manuscript unless explicitly stated otherwise.
W1 / Q1 (i). Are the deleted interactions still detectable?
Response. Under our loss-based membership inference attack, NAU is the only method retaining usable recommendation utility whose AUC difference from retraining has a 95% confidence interval containing zero on all three datasets. In contrast, Forget+Repair retains a significantly positive difference on Beauty and Toys.
New experiment: membership inference. A loss-based membership inference attack on sensitive item deletion. Members are the requesting users' training interactions with the category (62 / 94 / 56 interactions over five seeds), non-members the 174 / 272 / 156 held-out cases with the category as next item. Members and non-members come from different users and positions, so even a model that never saw the members separates the two groups to some degree. The right reference is therefore not 0.5 but the AUC of the retrained model, which never saw the deleted interactions; we report each model's AUC minus the retrained model's AUC (paired bootstrap 95% intervals):
Finetune, SCIF, and TRACER barely change the model for the requesting users (they reach about as many of them as before deletion, see Reviewer FL1d, W1b) and keep the deleted interactions as detectable as before; Forget only overshoots, which makes the deletion itself detectable; Forget+Repair remains detectable on Beauty and Toys. Among the methods that keep the model's utility, NAU is the only one whose interval includes 0 on all three datasets. A failed attack does not prove removal, so we report it alongside the behavioral metrics.
W1 / Q1 (ii). Is the attack's influence removed beyond the target?
Response. Yes, it is removed to an extent that output-side suppression cannot reach. Blocking the promoted item at decoding time removes its exposure as completely as NAU but leaves utility at the poisoned model's level; NAU recovers 1.4 points of τP more (33 of 45 runs). Part of this comes from the update on the retained data itself (Finetune, which does not forget, also gains utility), so we read it as evidence that unlearning changes the model's behavior beyond the target, which blocking cannot, rather than as proof of removal.