acceptodds
Under review as a conference paper at ICLR 2027

SAIL: SAILING BEYOND TEXT VIA BEHAVIOR- GROUNDED LLM SUMMARIES

10 views

Abstract

Ads recommendation systems must transfer signals from the organic content side to the ads side, but content items and ads sit in separate modeling spaces with no directly alignable ID representation and text that rarely shares words, so rich behavioral signals on the organic side are hard to reuse commercially. Existing approaches use either discrete Semantic IDs or text-derived representations such as pretrained embeddings and prompt-generated summaries. In our evaluated cross-domain setting, raw-text baselines and prompt-only LLM summaries exhibit a similar empirical surface-representation ceiling where performance saturates around a shared reference level. We propose Sail, a two-stage post-training recipe that keeps the LLM's contribution in explicit natural language rather than opaque Semantic IDs: SFT distills a structured summary from a supply-grounded teacher, denoising raw text and initializing the schema with weak behavioral context; RL with a listwise retrieval reward then performs retrieval alignment, selecting and shaping the collaborative signals that make the summary useful for matching items to ads. Co-occurrence context is used only as training-time privileged information; at inference the summary is written from item text alone. On I2A-Bench, Sail raises AUC by points over a strong raw-text baseline, remains effective under an independent judge and transfers to a different evaluation protocol, and its largest performance drops occur when fields capturing commercial and audience-bridging signals are removed. The learned summary also transfers, without retraining, to a downstream LLM-based CTR task and improves macro-average AUC over seven objectives.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Ads, Recommendation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UdvzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAIL tackles the cross-domain item-to-ad retrieval gap by replacing opaque, model-bound Semantic IDs with explicit, human-readable structured summaries. Its core contribution is a two-stage post-training framework that first distills a commercial summary schema from a supply-grounded 397B teacher into a 9B student model (SFT as denoising), and then applies GRPO with a listwise retrieval reward over a frozen ad pool to align summaries directly for matching (RL as retrieval alignment). The authors validate SAIL on 12A-BENCH, demonstrating a +2.53 AUC improvement over strong raw-text baselines under both training (gpt-oss-120b) and held-out (Gemini 3.6 Flash) LLM judges aligned with human preference. Further validation shows that these learned summaries generalize zero-shot across four Amazon subdomains and boost macro-average AUC by +2.56 points when transferred to a downstream CTR prediction task without retraining. AI Documentation: This assessment represents my original, unedited thoughts written directly without any assistance from an LLM or AI writing tool.

Critical strengths

The paper’s primary strength is replacing opaque, model-bound Semantic IDs with explicit, structured natural language summaries that can be inspected and reused downstream. This is highly consequential because it bypasses the black-box nature of generative recommendation while proving that text representations can directly transfer to CTR prediction without retraining (+2.56 macro-AUC). Furthermore, the two-stage post-training recipe (SFT distillation and GRPO with listwise retrieval rewards) cleverly injects behavioral co-occurrence as privileged training-time information, successfully breaking the empirical surface-representation ceiling at test time using item text alone.

Critical weaknesses

The paper’s main weakness is its heavy reliance on an LLM listwise judge to both optimize the RL reward and evaluate candidate pools restricted to only 30 ads. In production ad systems with millions of candidates, optimizing against a synthetic LLM judge on tiny candidate sets risks rewarding judge-aligned textual tropes rather than true user click behavior or multi-stage retrieval ranking. Additionally, generating a 10-field summary on the fly using a 9B model per incoming item introduces severe inference latency and compute overhead, making real-time deployment in high-throughput ad serving engines practically unfeasible without concrete latency SLAs or further distillation.

Questions for authors

(1) How does SAIL perform when evaluated directly against actual online ground-truth user click/conversion logs rather than the synthetic LLM listwise judge over small (K=30K=30) candidate sets? (2) What is the measured p99 end-to-end inference latency and GPU compute overhead for generating 10-field summaries with the 9B model during real-time online item ingestion?

Reviewer_LHAsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies item-to-ad (I2A) retrieval, where the organic-content side and the ads side reside in separate modeling spaces with no alignable ID representation and little lexical overlap. The authors identify an empirical phenomenon: raw-text representations and prompt-only LLM summaries saturate at a common empirical AUC upper bound, which they term the surface-representation ceiling. They then propose SAIL to break this ceiling.

SAIL keeps the LLM's contribution in explicit natural language rather than opaque Semantic IDs, via a two-stage post-training recipe. In Stage 1 (SFT as denoising), a supply-grounded teacher (Qwen3.5-397B) reads the item text together with training-time privileged information (historically co-clicked ads) and writes a ten-field structured summary that denoises the raw text and initializes the summary schema with weak behavioral context; the teacher is then distilled into a Qwen3.5-9B student. In Stage 2 (RL as retrieval alignment), the policy is aligned with GRPO under a listwise retrieval reward, in which a gpt-oss-120b judge ranks 30 candidates in a single pass and the reward is 0.5·AUC + 0.5·NDCG@10; this selects and shapes the collaborative signals that actually benefit ad matching. Co-occurrence context serves only as training-time privileged information: at inference, item summaries are generated online from item text alone while ad-side summaries are fixed offline, so the representation remains inspectable and reusable.

On the internal I2A-Bench, SAIL shows substantial offline gains over the raw-text baseline, and ablations validate the contribution of the informational inputs (Target Audience, Adjacent Content Topics), the training paradigm (SFT then RL), and transfer to a downstream LLM-based CTR task.

Critical strengths
  1. The proposed approach provides an effective, ad-aware item summarization scheme, and empirically demonstrates its advantage over purely text-based methods. SAIL writes collaborative signals that benefit ad matching into readable, structured summaries via supply-grounded SFT and listwise-retrieval RL alignment; on I2A-Bench it yields substantial offline gains over the raw-text baseline, and ablations confirm the contribution of the informational inputs and of the two training stages.
Critical weaknesses
  1. Limited novelty of the overall approach. The limitations of purely LLM-based methods for recommendation tasks are already well known, and the task-specific components, mining advertising-oriented item information, and the SFT and RL training recipe, are relatively conventional in design. The paper also fails to clearly articulate the generality and the necessity of using textual summaries as the item/ad representation.

  2. The I2A experiments are not reproducible, and the main task lacks the strongest supervised baselines under the same setting, so the effectiveness of the method is not established. On I2A-Bench, the authors compare neither a two-tower dense retriever trained on the same click pairs nor a state-of-the-art Semantic-ID model trained on the same data. They acknowledge in Section 4.1 that a dense retriever is a natural alternative but do not run it, and the trained TIGER in Table 3 is evaluated on a different task. Consequently, the gains cannot be attributed to "writing collaborative signals into readable text" as opposed to "any click-supervised training," and the distinctiveness of the contribution relative to supervised dense retrieval is not isolated. Moreover, the rank-level validity of the evaluation protocol itself is unverified: the judge is validated only through score correlation (Figure 3c), with no verification of ranking consistency or of stability across reruns and candidate reorderings. Because I2A-Bench is internal and non-releasable and the main results are mediated by an LLM judge, even a rebuttal that supplies all of the above baselines would leave the central claim externally unverifiable.

Questions for authors
  1. The paper insists on textual summaries as the intermediate medium; what is the necessity of this premise? Could the retrieval and CTR tasks instead transfer information through ID- or SID-based representations rather than text?

  2. For the experiments, I would like to see two additional baselines on I2A-Bench: a two-tower dense retriever trained on the same click pairs (e.g., HSTU, KuaiFormer, TransRetrieval) and SID-based generative-recommendation baselines (e.g., TIGER, LC-Rec, OneRec-v2), to validate the effectiveness of the proposed approach.

  3. On serving cost, a comparison between the proposed modeling scheme and the main baselines is needed, including the latency/FLOPs of online per-item JSON summary generation with the 9B policy, the RL training cost, and a comparison against dense-retrieval serving cost.

  4. The listwise judge is used in both training and evaluation; is there a risk of overoptimization or self-preference? Moreover, the judge is currently validated only through score correlation; verification of ranking consistency and of stability across judge reruns is lacking.

  5. In the downstream CTR task (Table 4), purchase (−2.34) and app activation (−3.68) show negative transfer: is there a field-level attribution or would adding conversion-intent fields mitigate this?

Reviewer_2q1mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SAIL, a method that generates structured summaries for matching organic content with ads. It first trains a student model on teacher summaries informed by historical interactions between content and ads, then applies reinforcement learning using a ranking reward based on click labels. At inference, the model requires only text. Experiments on an internal benchmark report ranking improvements over raw text and prompt-generated summaries under two LLM rankers. Additional experiments evaluate transfer to Amazon recommendation tasks and downstream click-through rate prediction.

Critical strengths

The paper addresses the practical problem of matching organic content with ads. SAIL uses behavioral information during training but requires only text at inference, allowing it to handle new items without interaction histories. Its structured summaries are readable and reusable in downstream tasks. Improvements over raw text and prompt-only summaries are reported under two LLM judges, including one not used for training, supporting the usefulness of the summaries beyond the training judge.

Critical weaknesses
  1. The experiments do not isolate the benefit of using historical ad context to generate the teacher summaries. The stage ablation shows that SFT and RL improve performance, but there is no matched comparison with teacher summaries generated from text alone. This leaves the role of behavioral grounding, a central part of the proposed method, insufficiently supported.

  2. The main evaluation uses 200 test items, each with 30 candidate ads and uniformly sampled negatives. It lacks a retrieval baseline trained on the same click data, and the reported three-seed averages have no uncertainty estimates. The results therefore leave unclear whether SAIL offers a reliable advantage over trained retrieval methods beyond the sampled LLM ranking setup.

Questions for authors
  1. How much does the historical ad context in the teacher prompt affect the final ranking results? Could you compare against teacher summaries generated from text alone, keeping the student model and the remaining training settings unchanged?

  2. How does SAIL compare with a dense retriever trained on the same click data and evaluated on the same test items and candidate sets?

  3. Could you provide per-seed AUC results and paired confidence intervals for the gains over raw text and the strongest prompt-only baseline? These would help assess the reliability of the improvements on the 200-item test set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.