acceptodds
Under review as a conference paper at ICLR 2027

When History Misleads: Asymmetric Margin Supervision for Instruction-Guided LLM Generative Recommendation

Abstract

In instruction-guided generative recommendation, LLM-based recommenders need to balance two goals: responding to the user's current request and aligning with the preferences in their interaction history. When the two conflict, history events can override the request. We show that turning the effect of individual history events into supervision faces two obstacles. First, the events that most influence a recommendation are not necessarily the ones that support the target item. Second, removing a misleading event can raise the target's score but a competing item's score even more, so a higher target score alone does not guarantee a better ranking. We propose Asymmetric Intervention-Guided Margin Supervision (AIMS), which converts the effect of removing individual history events into ranking supervision. For training requests already ranked correctly, a frozen reference model identifies request-specific deletions that improve both the target's score and its margin over a competitor near the recommendation cutoff. These margins serve as training targets, while the complete history is retained as input. Training combines cross-entropy with an asymmetric auxiliary loss that penalizes margin shortfalls and routes its gradient only through the competitor score. Inference is unchanged, requiring no history editing or deletion search. Across six LLM backbones on an industrial dataset and two public benchmarks, AIMS improves Recall and NDCG over strong baselines. Ablations support request-specific margins and asymmetric supervision, and the selected deletions preferentially remove constraint-violating history.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
generative recommendation, LLM-based recommendation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zsn3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conflicts between user requests and interaction histories in LLM-based generative recommendation. The authors propose AIMS, which uses a frozen model to derive target–competitor margin targets from single-event deletions for initially correct training requests. The student trains on complete histories with cross-entropy and an asymmetric auxiliary loss that routes gradients through competitor scores only. Experiments on an industrial dataset and two public benchmarks show ranking improvements over the evaluated baselines without changing inference.

Critical strengths
  • This paper uncovers that influential history events need not support the target and that target-score gains need not improve ranking.
  • The experiments show ranking improvements over the evaluated baselines without additional inference-time operations.
Critical weaknesses
  • The motivation for retaining complete histories is not validated against key history-editing baselines.
Questions for authors

I have several questions and requests for clarification:

  • Motivation. The limitation attributed to key history-editing baselines is not experimentally validated. The paper argues that these methods may discard useful personalization, but the deleted-input and history-dropout ablations do not test those methods. Please design experiments to verify whether these baselines remove useful preference information and substantiate the motivation for retaining complete histories.

  • Single-event Supervision. Table 9 reports small, inconsistent Recall@10 gains from greedy deletion of up to three events. Single-event deletions can nevertheless miss improvements requiring joint removals. Please clarify this limitation and whether the modest multi-event gains reflect limited interaction effects or limitations of the greedy search.

  • Top-K Optimization. AIMS uses a competitor near the recommendation cutoff to define a ranking margin. This technique is quite similar to the top-K optimization methods, such as SL@K [R1]. Please discuss its relationship to top-K optimization, which will further improve the paper's contribution and novelty.

Reference

  • [R1] Breaking the Top-K Barrier: Advancing Top-K Ranking Metrics Optimization in Recommender Systems. KDD 2025.
7 comments
AuthorsOpenReview ↗

Response to Reviewer Zsn3 (1/3): history-editing baselines

Thank you for the constructive questions. Q1 led to the main new experiment of the revision. In what follows, a result written as a/b is for Llama-3.1-8B and Gemma-3-12B, the two backbones used for the ablations.

Weakness / Q1. Validate the motivation for complete histories against history-editing baselines, and verify whether they remove useful preference information. In short: all three editors trail AIMS overall and in every request group, including conflict requests, where editing should help most, and the two that select events also delete request-consistent history. We agree the original comparisons did not test this: every deletion variant used the complete history at inference. We implemented three editors that remove history events at training and at inference (same θ0 and schedule as Continued CE, with hyperparameters chosen on validation):

  • ReLLa-style selection. Keep the k ∈ {5, 10, 20} history events most similar to the request. ReLLa retrieves by the candidate item, which does not exist at inference in full-catalog generative retrieval, so we retrieve by the request.
  • Learned deletion filter (STEAM/FILCO-style). A head on frozen θ0 states predicts from (request, event) whether deleting the event is eligible under Eq. (7), trained on the AIMS labels with 2-fold cross-fitting. It uses the information that defines the AIMS targets, but to edit the input, and so receives more task-specific supervision than STEAM's corrector, which is trained on randomly inserted and deleted items.
  • Rule-based constraint filter. Delete every history event that violates the request's explicit category constraint (parser of App. D, no targets).

To measure what editing removes, we report the share of request-consistent (PASS) history events each editor deletes, and ΔR@10 over Continued CE (6.902 / 7.945) in three groups: requests without an explicit constraint (75% of the test set), requests whose history fully agrees with the request (κ = 0, about 1%), and conflict requests (κ ≥ 0.5). Industrial dataset:

MethodR@10Δ, no constraintΔ, κ = 0Δ, κ ≥ 0.5PASS events removedAdded serving cost
ReLLa-style selection6.977 / 8.078+0.032 / +0.087−0.231 / −0.296+0.268 / +0.35227.6 / 25.3%+4% latency
Learned deletion filter7.028 / 8.177+0.101 / +0.208−0.088 / −0.121+0.247 / +0.3717.4 / 8.9%+31% latency
Rule-based filter6.974 / 8.048+0.012 / +0.019+0.004 / +0.009+0.302 / +0.4280%parser + item metadata
AIMS7.223 / 8.609+0.271 / +0.641+0.080 / +0.120+0.558 / +0.8740%none
AIMS + rule-based filter7.234 / 8.629+0.268 / +0.645+0.077 / +0.124+0.627 / +0.9510%parser + item metadata

Table R4 (App. C.8) gives the full table, with History dropout, Request-only CE, and violation rates.

AuthorsOpenReview ↗

Response to Reviewer Zsn3 (2/3): what the editing baselines show, and single-event supervision

Q1, continued: what the comparison shows. In what follows, a result written as a/b is for Llama-3.1-8B and Gemma-3-12B, the two backbones used for the ablations.

  1. The editors that select events remove useful history. ReLLa-style selection deletes 28%/25% of request-consistent events and the learned filter 7%/9%. Where the history fully agrees with the request, both fall below Continued CE (−0.23/−0.30 and −0.09/−0.12), while AIMS does not (+0.08/+0.12). This group is small, but on the 75% of requests without an explicit constraint both editors also gain far less than AIMS (+0.03/+0.09 and +0.10/+0.21, versus +0.27/+0.64).
  2. Removing every violating event is not enough. The rule-based filter does so, yet it gains about half as much recall as AIMS on conflict requests (+0.30/+0.43 vs. +0.56/+0.87). This suggests that violating events still carry preferences that transfer across categories.
  3. Whether a deletion helps is hard to predict without the target. The learned filter is trained on the same labels that define our targets, yet reaches a validation AUC of only 0.66/0.68 (cf. Table 14). This is why AIMS uses deletions offline, as targets.
  4. Editing and AIMS can be combined: adding the rule-based filter to AIMS gives a further +0.07/+0.08 on conflict requests, at the cost of a parsed constraint and item metadata at serving time.

(Sec. 4.3, Table R2, App. C.8.)

Q2. Single-event supervision: limited interactions, or a limitation of greedy search? Mainly limited interactions. We exhaustively evaluated all event pairs (up to 435 views per request) on 1,000 random protected requests per backbone (App. C.13, Table R8):

  • Search is not the bottleneck at K = 2: the greedy pair is the exhaustive optimum in 71%/69% of requests, and on average it attains 97%/96% of the optimal pair's gain.
  • Interactions are small: the median interaction |Δm({a, b}) − Δm({a}) − Δm({b})| is 11%/13% of the sum of the two single gains, so the optimal pair is mostly the best single deletion plus a nearly additive second one. It contains none of the top-3 single deletions in only 6.2%/7.5% of requests.
  • Larger targets are harder to reach: the share of protected requests whose final margin reaches mᶜᶠ − ε falls from 84.1/80.7% at K = 1 to 73.6/69.2% at K = 3, consistent with Conservative selection matching or exceeding greedy K = 3 (7.291/8.609 vs. 7.270/8.578).

As you suggested, Sec. 6 now states that single-event targets can miss improvements requiring joint removals.

AuthorsOpenReview ↗

Response to Reviewer Zsn3 (3/3): relation to top-K optimization (SL@K)

Q3. Relation to top-K optimization such as SL@K. Thank you for the pointer. SL@K (Yang et al., KDD 2025) is now discussed in Sec. 5 and App. E and compared in App. C.14. It estimates each user's top-K score quantile and weights the softmax loss of each positive by a sigmoid of its score minus that quantile, so that positives scoring inside the top K dominate the loss. AIMS shares this boundary view: its competitor, the reference model's rank-(R+1) item, plays the role of the top-K threshold. The differences:

  • SL@K re-weights positives toward the metric. AIMS adds a request-specific margin target derived from history deletions, which has no counterpart in SL@K. Without it AIMS reduces to R-comp, which yields 29%/51% of its gain.
  • The softmax loss raises positives and lowers negatives jointly. ACS lowers only the boundary competitor, and symmetric routing is significantly worse in our setting (Table 3C).
  • SL@K estimates the quantile from about 1,000 sampled items per user on embedding-based backbones. Here each item score is a teacher-forced LLM pass over a SID sequence, so AIMS fixes one competitor offline. A quantile-like boundary inside AIMS (Set-max, Table 9B) does not help.

We adapted SL@K to SID scores on KuaiSearch-Lite, where candidate pools make the quantile estimate natural (16 sampled pool candidates per instance), and combined it with ACS (R@10 / N@10):

KuaiSearch-Lite, candidate poolLlama-3.1-8BGemma-3-12B
Continued CE26.36 / 17.3828.04 / 18.74
Strongest baseline27.21 / 17.75 (LTRGR)28.80 / 18.71 (LETTER (RG))
SL@K, adapted27.38 / 17.8629.06 / 19.02
AIMS27.70 / 18.1229.97 / 19.82
SL@K + ACS28.04 / 18.3730.42 / 20.13

SL@K improves over the existing baselines. Adding ACS to it adds +0.66/+1.36 R@10 points and exceeds AIMS alone, so request-specific margin targets and top-K-aware weighting are complementary. (Table R9.)

We hope the history-editing baselines you asked for (Table R2) and the answers above resolve your concerns. If a specific issue remains, please point us to it and we will address it during the discussion.

Reviewer_pKAgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses conflicts between user history and current instructions in LLM-based generative recommendation. It proposes AIMS, which uses history deletions to construct counterfactual margin targets and an asymmetric loss to suppress competing items. Experiments on three datasets and six LLM backbones show consistent improvements in Recall and NDCG, supported by ablation studies and conflict analysis.

Critical strengths
  1. The paper identifies two meaningful issues in history-aware recommendation: prediction influence does not necessarily reflect target utility, and improving the target score does not guarantee better ranking. These observations provide a clear motivation for the proposed counterfactual margin supervision.

2.AIMS consistently improves accuracy across three datasets and backbones. The ablation studies carefully examine the contributions of margin construction, deletion selection, and asymmetric gradient routing, providing convincing evidence for the effectiveness of the overall approach.

Critical weaknesses
  1. The Magnitude Shuffle baseline retains a substantial portion of AIMS's performance gains, suggesting that much of the improvement may come from margin regularization rather than identifying misleading history events.
  2. The conflict-specific analysis is conducted only on the proprietary industrial dataset, while the public benchmarks mainly evaluate ranking performance. This makes it difficult to assess whether the claimed benefits in handling conflicting user instructions generalize beyond the industrial setting.
Questions for authors
  1. Figure 3 shows that Magnitude Shuffle retains a large portion of AIMS's performance gains. Could the authors compare AIMS with Shuffle across different levels of instruction history conflict, particularly when the conflict is severe? This would help clarify whether request-specific deletion gains provide benefits beyond general margin regularization.

2.The instruction history conflict analysis is only conducted on the proprietary industrial dataset, while the public benchmarks mainly evaluate ranking performance. Could the authors provide a similar conflict-focused evaluation on a public dataset, such as KuaiSearch-Lite, to verify whether the findings generalize?

  1. AIMS requires evaluating multiple history deletion before training, which may introduce substantial computational overhead. Could the authors report the total training cost compared to the baselines?
4 comments
AuthorsOpenReview ↗

Response to Reviewer pKAg (1/2): what the Shuffle control shows

Thank you for the positive assessment and the precise questions. In what follows, a result written as a/b is for Llama-3.1-8B and Gemma-3-12B, the two backbones used for the ablations.

W1 / Q1. Does Shuffle show that the gain is margin regularization, not identification of misleading events? How do AIMS and Shuffle compare under severe conflict? Margin regularization alone does not explain the gain. Margin protection without deletion gains (R-comp) gives 29%/51% of the improvement, and the deletion gains give the other 71%/49%. Their request-specific part is about zero without conflict and about half of AIMS's gain under severe conflict.

Overall (ΔR@10 over Continued CE, now explicit in Sec. 4.3):

ComponentLlama-3.1-8BGemma-3-12B
Margin protection without deletion gains (R-comp)+0.093 (29%)+0.336 (51%)
Deletion-gain values, shuffled across requests (Shuffle − R-comp)+0.135 (42%)+0.209 (31%)
Each request's own gain (AIMS − Shuffle, p < 0.05)+0.093 (29%)+0.119 (18%)

By conflict severity κ, the share of evaluable history that violates the request's constraint (App. C.12). The strata cover the test requests with an explicit, evaluable constraint, about a quarter of the test set:

ΔR@10 over Continued CEκ = 0(0, 0.5)[0.5, 0.8)[0.8, 1]
R-comp+0.078 / +0.097+0.085 / +0.143+0.091 / +0.178+0.097 / +0.205
Shuffle+0.083 / +0.109+0.124 / +0.231+0.205 / +0.352+0.318 / +0.541
AIMS+0.080 / +0.120+0.212 / +0.343+0.397 / +0.623+0.653 / +1.022
AIMS − Shuffle−0.003 / +0.011+0.088 / +0.112+0.192 / +0.271+0.335 / +0.481
AIMS − Shuffle, share of AIMS's gainabout 042% / 33%48% / 43%51% / 47%
  • At κ ≥ 0.8, AIMS − Shuffle is +0.34/+0.48 (95% CIs [0.25, 0.42] and [0.38, 0.58]). R-comp grows far less with κ than AIMS (+0.02/+0.11 vs. +0.57/+0.90 from the first to the last stratum).
  • Shuffle is not a deletion-free regularizer: it reuses the deletion gains and the set of requests with an eligible deletion. Targets that ignore which requests have a helpful deletion fall below it: 7.051/8.224 for a difficulty-based target of tuned scale (D-free) and 7.097/8.321 for the AIMS gain distribution reassigned by difficulty (D-cal), versus 7.130/8.490 for Shuffle and 7.223/8.609 for AIMS.
  • Identification is also tested without training (Table 14): the selected deletion removes a constraint-violating event 16.8/16.6 points more often than chance. The excess falls to 2.9/4.1 points under a generic request and reverses (−12.3/−11.1) when the target itself violates the constraint.
AuthorsOpenReview ↗

Response to Reviewer pKAg (2/2): conflict analysis on public data, and compute

W2 / Q2. Conflict analysis on a public dataset. We replicated the protocol of App. D on KuaiSearch-Lite with a deterministic parser that matches the query text against the released three-level category hierarchy (no targets, no model outputs). 38.4% of test requests carry an explicit category constraint, and 91.2% of these meet the coverage condition of Eq. (10). ΔR@10 over Continued CE under the candidate-pool protocol of Table 2 (strongest baseline: LTRGR / LETTER (RG)):

κ stratum (share)Strongest baselineAIMSAIMS − strongest
κ = 0 (9.7%)+0.79 / +0.70+0.83 / +1.04+0.04 / +0.34
(0, 0.5) (21.3%)+0.88 / +0.74+1.17 / +1.55+0.29 / +0.81
[0.5, 0.8) (30.2%)+0.97 / +0.79+1.58 / +2.24+0.61 / +1.45
[0.8, 1] (38.8%)+1.02 / +0.83+2.11 / +2.96+1.09 / +2.13

All three industrial findings replicate:

  • AIMS's advantage over the strongest baseline grows with conflict, from +0.04/+0.34 to +1.09/+2.13, while the baseline's own gain rises only from +0.79/+0.70 to +1.02/+0.83.
  • On conflict requests (κ ≥ 0.5), AIMS lowers the observed-violation rate from 34.6/31.9% under Continued CE to 27.1/24.2%.
  • The selected deletions remove a violating event 77.3/79.6% of the time, versus 61.8/63.2% expected from history composition (+15.5/+16.4 points, against +16.8/+16.6 on the industrial data). The excess reverses (−9.8/−10.6) when the target itself violates the constraint.

(New App. D.1, Table R10.) The parser will be released with the code.

Q3. Total training cost. Industrial dataset, Llama-3.1-8B, A100-80GB GPU-hours:

MethodOffline, one-timeTrainingTotal (× Continued CE)
Continued CE—21.51.00×
CFT—37.91.76×
LETTER (RG)—24.31.13×
LTRGR14.2 (candidate decoding)30.62.08×
S-DPO6.1 (reference scoring)33.81.86×
AIMS14.2 (initial decoding) + 9.8 (deletion scoring)25.72.31×

Like LTRGR (Li et al., AAAI 2024) and S-DPO (Chen et al., NeurIPS 2024), AIMS adds cost only at training time. The offline pass runs once and is reused across seeds and hyperparameter runs, and training itself costs 20% more than Continued CE, less than LTRGR (+42%), S-DPO (+57%), and CFT (+76%). At inference, AIMS runs exactly like Continued CE on the complete history, with no editing or deletion search, so its serving cost is unchanged. By contrast, the baselines requested by Reviewer Zsn3, which edit the history at inference, add up to 31% latency (App. C.8). (New App. B.6, Table R3.)

Thank you for the detailed response and additional experiments. The conflict-stratified comparison with Shuffle addresses my main concern about the value of request-specific supervision. The results on KuaiSearch-Lite and the compute breakdown also address my remaining questions. Overall, my concerns are resolved, and I will maintain my positive recommendation.

Thank you very much for your positive assessment and helpful questions! We have also added all the new analyses and results you suggested to the revised version.

Reviewer_cEjsOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AIMS, a method for instruction-guided generative recommendation that uses single-event history deletions to construct request-specific target–competitor margin targets. A frozen reference model identifies useful deletions, and an asymmetric auxiliary loss encourages the student to suppress a boundary competitor while retaining the complete history. Experiments cover six LLM backbones, one industrial dataset, and two public benchmarks.

Critical strengths

The paper addresses a relevant problem and presents a clearly specified training procedure. The experiments cover multiple model families, use matched baselines, and include useful ablations. Avoiding additional inference-time computation is also a practical advantage.

Critical weaknesses
  1. The methodological novelty is limited. AIMS combines deletion-based counterfactual supervision, pairwise margin learning, and a hinge penalty against a fixed boundary competitor. The resulting objective is only indirectly connected to the actual top-10 decision, which depends on all competing candidates rather than a single fixed item. Although the ablations support the proposed design choices, they do not establish a sufficiently principled or general methodological advance over these existing ingredients.

  2. The evaluation also leaves a substantial gap between the claims and the evidence. Both public benchmarks use teacher-forced ranking over candidate pools rather than full-catalog SID decoding, while the only full-catalog results come from a proprietary dataset that cannot be independently reproduced. Moreover, the absolute Recall@10 improvement over the strongest baseline on the industrial dataset is only 0.036–0.136 percentage points, despite being reported as statistically significant. These results provide limited evidence that AIMS offers a substantial, independently verifiable improvement in the intended full-catalog recommendation setting.

Questions for authors

How much of the reported improvement would remain under full-catalog decoding on a public benchmark, and how does AIMS compare with a more direct pairwise margin-distillation baseline under the same tuning protocol?

6 comments
AuthorsOpenReview ↗

Response to Reviewer cEjs (1/3): your two questions

Thank you for the review. In what follows, a result written as a/b is for Llama-3.1-8B and Gemma-3-12B, the two backbones used for the ablations.

In short. Under full-catalog decoding on public KuaiSearch-Lite, AIMS is best on both backbones: +3.6%/+3.8% R@10 over the strongest baseline, with 95% CIs excluding zero. Margin-MSE gains +0.03/+0.12 R@10 over Continued CE, against +0.32/+0.66 for AIMS. Details are below. The size of the gains and novelty follow in (2/3) and (3/3).

Q1 / W2a. How much of the improvement remains under full-catalog decoding on a public benchmark? It holds, with relative gains comparable to or larger than under candidate pools. Every method was re-evaluated with SID-constrained beam search (beam 100) over the full KuaiSearch-Lite catalog, on the two backbones used for all ablations, with the Table 2 checkpoints unchanged.

KuaiSearch-Lite, full catalogLlama R@10Llama N@10Gemma R@10Gemma N@10
Continued CE8.2145.1029.0355.684
CFT8.2515.1179.0115.663
LETTER (RG)8.5365.3119.4725.958
LTRGR8.6025.3389.3185.861
S-DPO8.3875.2149.3565.889
AIMS8.9125.5539.8326.198
AIMS-FC8.9865.5979.9056.241
  • Over the strongest baseline: +0.31/+0.36 R@10 points, +3.6%/+3.8% relative (candidate pools: +1.8%/+4.1%). The 95% paired bootstrap CIs are [0.07, 0.55] and [0.11, 0.61].
  • Over Continued CE: +8.5%/+8.8% relative (candidate pools: +5.1%/+6.9%).
  • The baselines keep their R@10 order from Table 2. Re-deriving the protection set and competitors under full-catalog decoding (AIMS-FC) adds +0.07.

The public benchmarks release each request's impression pool with click labels, so the submission ranked every method on identical pools. Because sampled-candidate metrics can deviate from full-ranking ones (Krichene & Rendle, 2020), the revision adds full-catalog decoding. The experiment uses only public data, and the code, processed data, and full-catalog evaluation scripts will be released upon acceptance (Reproducibility Statement). (Sec. 4.2, Table R1, App. C.9.)

Q2. Comparison with a direct pairwise margin-distillation baseline under the same tuning protocol. We added Margin-MSE (Hofstätter et al., 2020) with the θ0, schedule, protection set, and competitors of AIMS, and λ tuned on the validation grid used for all baselines. The table varies the margin target (rows) and the loss (columns), where symmetric means that the gradient flows through both scores (industrial R@10, with Continued CE at 6.902 / 7.945):

TargetMSE, symmetricHinge, symmetricHinge, competitor-only (ACS)
Reference margin m⁰6.931 / 8.068 (Margin-MSE)—6.995 / 8.281 (R-comp)
Counterfactual margin mᶜᶠ6.981 / 8.2466.955 / 8.253 (Symmetric)7.223 / 8.609 (AIMS)

With either symmetric loss, the counterfactual targets yield at most +0.08/+0.31 over Continued CE. Routing the gradient only through the competitor turns the same targets into the full gain of +0.32/+0.66. (Sec. 4.3, Table R5, App. C.10.)

AuthorsOpenReview ↗

Response to Reviewer cEjs (2/3): size of the gains, and the link to the top-10 decision

W2b. "The absolute Recall@10 improvement over the strongest baseline on the industrial dataset is only 0.036–0.136 percentage points." In relative terms, AIMS improves on the strongest baseline by 0.9–1.8% in R@10 and 1.2–2.0% in N@10 on this dataset (Table 1), a size that industrial search and recommendation teams have reported as meaningful. Walmart describes a +0.26% offline gain in EM Recall@20 as a large real-world impact at its scale (Yang et al., "Scaling and Stabilizing Large-Scale Embedding-Based Retrieval", SIGIR 2026). Airbnb considers a +0.3% gain in offline NDCG promising for a statistically significant booking gain online, and its +0.48% NDCG gain came with +0.61% more bookers (Tan et al., "Optimizing Airbnb Search Journey with Multi-task Learning", KDD 2023). Per backbone:

Industrial dataset, AIMS vs. strongest baselineLlama-8BLlama-70BGemma-4BGemma-12BQwen3-8BQwen3-14B
R@10 gain (points)+0.087+0.113+0.036+0.113+0.060+0.136
95% CI[0.031, 0.143][0.049, 0.178][0.004, 0.069][0.052, 0.175][0.012, 0.109][0.071, 0.202]
R@10 gain (relative)+1.22%+1.04%+0.93%+1.33%+0.90%+1.76%
N@10 gain (relative)+1.76%+1.34%+1.20%+1.63%+1.29%+1.95%
  • R@10 on this dataset is only 3.7–11.0%, which is why these gains amount to 0.036–0.136 points. Every 95% CI excludes zero, and AIMS beats the strongest baseline in every training seed (Table 8).
  • The gain is concentrated where the history conflicts with the request, which is the case AIMS targets. Where at least 80% of the evaluable history violates the request's constraint, the gain over the strongest baseline is +0.41/+0.58 points for Llama-3.1-8B/Gemma-3-12B, about five times the full-test-set figure (App. C.11).
  • On public data the gain is larger: +0.47 to +1.17 points (1.2–4.6%) over the strongest baseline across the 12 candidate-pool settings, and +3.6%/+3.8% under full-catalog decoding.

AIMS has the best R@10 and N@10 in all 18 dataset–backbone settings, at unchanged serving cost.

W1b. "The resulting objective is only indirectly connected to the actual top-10 decision, which depends on all competing candidates rather than a single fixed item." For a fixed candidate set, the decision depends on the other candidates only through the one at the boundary: the target is in the top R if and only if it outscores the R-th best other item, the comparison behind the top-k hinge loss (Lapin et al., "Top-k Multiclass SVM", NeurIPS 2015). For every protected request, that item in θ0's decoded list is exactly our competitor c, so at initialization the margin is the decision variable itself. The boundary item can change during training (Sec. 2.3), so we measured how well the margin tracks the decision:

  • After training, 95.8/94.1% of protected training requests remain top-10 hits under AIMS, versus 91.2/88.7% under Continued CE.
  • On test requests, AIMS raises retention (initial top-10 hits that stay hits) from 94.73/92.84% under Continued CE to 96.28/94.12%, and recovery (initial misses that become hits) from 0.595/0.863% to 0.828/1.475% (all p < 0.05, Fig. 4).
  • Richer boundaries do not help: averaging ACS over up to five boundary candidates changes R@10 by at most 0.032 points, and comparing against the highest-scoring of them lowers it by up to 0.13/0.12 (Table 9B).
AuthorsOpenReview ↗

Response to Reviewer cEjs (3/3): novelty

W1a. Novelty: is AIMS a principled and general advance over its ingredients? The combination you describe is our Symmetric ablation: deletion-derived targets, a pairwise margin, and a hinge against the fixed competitor, with gradients through both scores as in standard margin losses. It recovers only 17%/46% of AIMS's gain (Table 3).

What is new is where the supervision comes from. Which deletion helps depends on the target (Table 14) and is hard to predict without it: a filter trained to predict it from the request and the event reaches a validation AUC of only 0.66/0.68 (App. C.8). AIMS therefore uses the label-selected deletion as a training target and never as an input, which makes it privileged information in the sense of generalized distillation (Lopez-Paz et al., 2016). Prior methods either edit the input (CauseRec, STEAM, ReLLa, FILCO-style filters) or match a teacher fixed in advance (Margin-MSE, context distillation). To our knowledge, none of them uses a label-selected deletion as a training target while keeping the input unchanged (Table R11).

On "principled": the other two design choices, the selection rule and the one-sided loss, follow from the same analysis, and their alternatives are worse.

  • A deletion can raise the target's score while raising the competitor's more, so it is accepted only if both the target score and the margin improve (Eq. 7). Selecting by target uplift or by influence lowers R@10 from 7.223/8.609 to 7.091/8.426 and 7.063/8.312.
  • The counterfactual margin is one the model already attains with one event removed, so it is a floor, and CE already supplies the target gradient. The loss is therefore one-sided and acts only on the competitor. With the same targets, symmetric gradients recover less than half of the gain (Table R5).

On "general": AIMS is an auxiliary loss that changes neither the architecture, the decoder, nor the input. It applies unchanged to three LLM families from 4B to 70B on three datasets, with the best R@10 and N@10 in all 18 settings. It also combines with other objectives: on top of SL@K, a top-K ranking loss (Yang et al., KDD 2025), the AIMS auxiliary loss adds +0.66/+1.36 R@10 points (Table R9).

We hope the two experiments you asked for (Tables R1 and R5) and the answers above resolve your concerns. If a specific issue remains, please point us to it and we will address it during the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: new results and revisions

We thank the reviewers. They asked for (i) full-catalog evidence on public data, (ii) comparisons with more direct alternatives, and (iii) a finer account of where the gain comes from. We ran every requested experiment. The results are in the revised PDF, with changes in blue and new tables R1–R12. The contribution list is updated. In the table below, a result written as a/b is for Llama-3.1-8B and Gemma-3-12B, the two backbones used for the ablations.

#New experimentAsked byResult
1Full-catalog SID decoding on public KuaiSearch-LitecEjsAIMS is best on both backbones: +3.6%/+3.8% R@10 over the strongest baseline, +8.5%/+8.8% over Continued CE (Table R1)
2Margin-MSE with reference and with counterfactual targetscEjs+0.03/+0.12 R@10 over Continued CE, vs. +0.32/+0.66 for AIMS (Table R5)
3Three inference-time history editors: ReLLa-style selection, a learned deletion filter, a rule-based constraint filterZsn3All trail AIMS overall and in every request group. The rule-based filter, which removes every violating event, gains about half as much as AIMS on conflict requests (Tables R2, R4)
4AIMS vs. Shuffle and R-comp by conflict severitypKAgRequest-specific part of the gain: about zero without conflict, and +0.34/+0.48 (about half of AIMS's gain) when at least 80% of the evaluable history violates the request's constraint (Table R7)
5Conflict analysis on public KuaiSearch-LitepKAgAs on the industrial data, AIMS's advantage over the strongest baseline grows with conflict severity, from +0.04/+0.34 to +1.09/+2.13 R@10 points (Table R10)
6End-to-end computepKAgTraining costs 20% more than Continued CE, less than LTRGR, S-DPO, and CFT. The offline pass runs once and is reused. Serving cost is unchanged (Table R3)
7Exhaustive pair deletions on 1,000 requestsZsn3Greedy finds the optimal pair in 71%/69% of requests. Pairwise interactions are small (Table R8)
8SL@K baseline, and SL@K + ACSZsn3Complementary: SL@K + ACS > AIMS > SL@K (Table R9)

The main text (now 10 pages) includes the full-catalog results (Sec. 4.2) and the editing baselines (Sec. 4.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.