- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies conflicts between user requests and interaction histories in LLM-based generative recommendation. The authors propose AIMS, which uses a frozen model to derive target–competitor margin targets from single-event deletions for initially correct training requests. The student trains on complete histories with cross-entropy and an asymmetric auxiliary loss that routes gradients through competitor scores only. Experiments on an industrial dataset and two public benchmarks show ranking improvements over the evaluated baselines without changing inference.
Critical strengths
- This paper uncovers that influential history events need not support the target and that target-score gains need not improve ranking.
- The experiments show ranking improvements over the evaluated baselines without additional inference-time operations.
Critical weaknesses
- The motivation for retaining complete histories is not validated against key history-editing baselines.
Questions for authors
I have several questions and requests for clarification:
-
Motivation. The limitation attributed to key history-editing baselines is not experimentally validated. The paper argues that these methods may discard useful personalization, but the deleted-input and history-dropout ablations do not test those methods. Please design experiments to verify whether these baselines remove useful preference information and substantiate the motivation for retaining complete histories.
-
Single-event Supervision. Table 9 reports small, inconsistent Recall@10 gains from greedy deletion of up to three events. Single-event deletions can nevertheless miss improvements requiring joint removals. Please clarify this limitation and whether the modest multi-event gains reflect limited interaction effects or limitations of the greedy search.
-
Top-K Optimization. AIMS uses a competitor near the recommendation cutoff to define a ranking margin. This technique is quite similar to the top-K optimization methods, such as SL@K [R1]. Please discuss its relationship to top-K optimization, which will further improve the paper's contribution and novelty.
Reference
- [R1] Breaking the Top-K Barrier: Advancing Top-K Ranking Metrics Optimization in Recommender Systems. KDD 2025.
7 comments
Response to Reviewer Zsn3 (2/3): what the editing baselines show, and single-event supervision
Q1, continued: what the comparison shows. In what follows, a result written as a/b is for Llama-3.1-8B and Gemma-3-12B, the two backbones used for the ablations.
- The editors that select events remove useful history. ReLLa-style selection deletes 28%/25% of request-consistent events and the learned filter 7%/9%. Where the history fully agrees with the request, both fall below Continued CE (−0.23/−0.30 and −0.09/−0.12), while AIMS does not (+0.08/+0.12). This group is small, but on the 75% of requests without an explicit constraint both editors also gain far less than AIMS (+0.03/+0.09 and +0.10/+0.21, versus +0.27/+0.64).
- Removing every violating event is not enough. The rule-based filter does so, yet it gains about half as much recall as AIMS on conflict requests (+0.30/+0.43 vs. +0.56/+0.87). This suggests that violating events still carry preferences that transfer across categories.
- Whether a deletion helps is hard to predict without the target. The learned filter is trained on the same labels that define our targets, yet reaches a validation AUC of only 0.66/0.68 (cf. Table 14). This is why AIMS uses deletions offline, as targets.
- Editing and AIMS can be combined: adding the rule-based filter to AIMS gives a further +0.07/+0.08 on conflict requests, at the cost of a parsed constraint and item metadata at serving time.
(Sec. 4.3, Table R2, App. C.8.)
Q2. Single-event supervision: limited interactions, or a limitation of greedy search? Mainly limited interactions. We exhaustively evaluated all event pairs (up to 435 views per request) on 1,000 random protected requests per backbone (App. C.13, Table R8):
- Search is not the bottleneck at K = 2: the greedy pair is the exhaustive optimum in 71%/69% of requests, and on average it attains 97%/96% of the optimal pair's gain.
- Interactions are small: the median interaction |Δm({a, b}) − Δm({a}) − Δm({b})| is 11%/13% of the sum of the two single gains, so the optimal pair is mostly the best single deletion plus a nearly additive second one. It contains none of the top-3 single deletions in only 6.2%/7.5% of requests.
- Larger targets are harder to reach: the share of protected requests whose final margin reaches mᶜᶠ − ε falls from 84.1/80.7% at K = 1 to 73.6/69.2% at K = 3, consistent with Conservative selection matching or exceeding greedy K = 3 (7.291/8.609 vs. 7.270/8.578).
As you suggested, Sec. 6 now states that single-event targets can miss improvements requiring joint removals.
Response to Reviewer Zsn3 (3/3): relation to top-K optimization (SL@K)
Q3. Relation to top-K optimization such as SL@K. Thank you for the pointer. SL@K (Yang et al., KDD 2025) is now discussed in Sec. 5 and App. E and compared in App. C.14. It estimates each user's top-K score quantile and weights the softmax loss of each positive by a sigmoid of its score minus that quantile, so that positives scoring inside the top K dominate the loss. AIMS shares this boundary view: its competitor, the reference model's rank-(R+1) item, plays the role of the top-K threshold. The differences:
- SL@K re-weights positives toward the metric. AIMS adds a request-specific margin target derived from history deletions, which has no counterpart in SL@K. Without it AIMS reduces to R-comp, which yields 29%/51% of its gain.
- The softmax loss raises positives and lowers negatives jointly. ACS lowers only the boundary competitor, and symmetric routing is significantly worse in our setting (Table 3C).
- SL@K estimates the quantile from about 1,000 sampled items per user on embedding-based backbones. Here each item score is a teacher-forced LLM pass over a SID sequence, so AIMS fixes one competitor offline. A quantile-like boundary inside AIMS (Set-max, Table 9B) does not help.
We adapted SL@K to SID scores on KuaiSearch-Lite, where candidate pools make the quantile estimate natural (16 sampled pool candidates per instance), and combined it with ACS (R@10 / N@10):
| KuaiSearch-Lite, candidate pool | Llama-3.1-8B | Gemma-3-12B |
|---|---|---|
| Continued CE | 26.36 / 17.38 | 28.04 / 18.74 |
| Strongest baseline | 27.21 / 17.75 (LTRGR) | 28.80 / 18.71 (LETTER (RG)) |
| SL@K, adapted | 27.38 / 17.86 | 29.06 / 19.02 |
| AIMS | 27.70 / 18.12 | 29.97 / 19.82 |
| SL@K + ACS | 28.04 / 18.37 | 30.42 / 20.13 |
SL@K improves over the existing baselines. Adding ACS to it adds +0.66/+1.36 R@10 points and exceeds AIMS alone, so request-specific margin targets and top-K-aware weighting are complementary. (Table R9.)
We hope the history-editing baselines you asked for (Table R2) and the answers above resolve your concerns. If a specific issue remains, please point us to it and we will address it during the discussion.
Response to Reviewer Zsn3 (1/3): history-editing baselines
Thank you for the constructive questions. Q1 led to the main new experiment of the revision. In what follows, a result written as a/b is for Llama-3.1-8B and Gemma-3-12B, the two backbones used for the ablations.
Weakness / Q1. Validate the motivation for complete histories against history-editing baselines, and verify whether they remove useful preference information. In short: all three editors trail AIMS overall and in every request group, including conflict requests, where editing should help most, and the two that select events also delete request-consistent history. We agree the original comparisons did not test this: every deletion variant used the complete history at inference. We implemented three editors that remove history events at training and at inference (same θ0 and schedule as Continued CE, with hyperparameters chosen on validation):
To measure what editing removes, we report the share of request-consistent (PASS) history events each editor deletes, and ΔR@10 over Continued CE (6.902 / 7.945) in three groups: requests without an explicit constraint (75% of the test set), requests whose history fully agrees with the request (κ = 0, about 1%), and conflict requests (κ ≥ 0.5). Industrial dataset:
Table R4 (App. C.8) gives the full table, with History dropout, Request-only CE, and violation rates.