acceptodds
Under review as a conference paper at ICLR 2027

PREFERENCE OPTIMIZATION IS NOT ENOUGH: HARNESSING CATALOG SUPPORT FOR GENERATIVE RECOMMENDATION

Abstract

Direct Preference Optimization (DPO) and its multi-negative extension, Softmax-DPO, align generative recommenders through relative item comparisons. However, unconstrained generation operates over the full autoregressive sequence space, where better preference margins do not guarantee that probability mass remains on valid catalog items. We term this failure mode catalog-support drift: a model can become better at preference discrimination while generating item strings that cannot be resolved against the catalog, undermining the practical usefulness of its recommendations. Addressing this mismatch requires learning not only which items users prefer, but also how to preserve support for valid outputs. To this end, we introduce , a dehallucination-oriented extension of Softmax-DPO that integrates sequence- and prefix-level support control with preference learning. At the sequence level, a full-slate ranking objective improves within-slate ordering, while a primal–dual constraint regulates the slate's total probability mass against an SFT-calibrated budget. At the prefix level, an on-policy trie loss penalizes probability assigned to catalog-invalid continuations along the evolving policy's generation paths. Together, these objectives shape the policy itself without imposing a catalog mask at inference. Under full-policy ancestral sampling, our analysis relates slate probability mass to exact outside-slate risk and bounds finite-horizon catalog-exit probability by the expected prefix-support loss. Experiments across five real-world recommendation datasets reveal the practical severity of catalog-support drift. Extensive results show that DeHall achieves superior performance on both catalog validity and policy-ranking accuracy, reducing nucleus-sampling catalog-string violation rates. Component analyses further support the complementary roles of prefix- and sequence-level support control. Code is available at https://anonymous.4open.science/r/DEHALL-SDPO-6776/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Recommender Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mRxoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies catalog-support drift in LLM-based generative recommendation: DPO and Softmax-DPO can improve relative preferences among candidate items without preserving the probability mass assigned to valid catalog items. Consequently, better candidate ranking can coexist with substantially worse free-generation validity.

The proposed DEHALL-SDPO augments Softmax-DPO with full-slate listwise policy ranking (LPR), primal–dual sequence-mass control (Dual Mass) using an SFT-calibrated budget, and on-policy prefix-support regularization (Prefix) based on a catalog trie. These objectives modify training while leaving inference unmasked. Experiments use Mistral-7B-Instruct-v0.3 on five recommendation datasets and evaluate candidate ranking and generation under multiple decoding protocols.

Critical strengths
  • The paper identifies a meaningful evaluation blind spot. Candidate ranking, relative preference margins, and free-generation validity measure different properties. The checkpoint-level measurements illustrate failures that ranking metrics alone can conceal, providing a useful perspective on training and evaluating generative recommenders.

  • The recovery in generation validity over the evaluated Softmax-DPO baseline is substantial. On Amazon CD, for example, greedy catalog violation decreases from 59.19% to 1.56%, while greedy Hit increases from 22.19% to 38.69%. These results demonstrate the practical importance of support preservation under the evaluated setup.

Critical weaknesses
  • Improved per-output correctness does not establish better recommendation-list quality. On Amazon CD, sample-10 mean Hit improves from 22.66% to 37.52% relative to SFT, but Any Hit decreases from 59.50% to 47.56%, and Unique decreases from 53.48% to 17.09%. MovieLens also exhibits lower Any Hit. These results indicate that improved per-output correctness accompanies a more concentrated output distribution.

  • The connection between the theoretical interpretation and achieved optimization behavior remains incomplete. The selected models do not satisfy the validation mass budgets. For example, the selected MovieLens model has mean negative-log slate mass of 0.02729 versus an SFT budget of 0.00840; Amazon CD has 0.09462 versus 0.05825. It is therefore unclear whether adaptive dual weighting provides more effective support control than an ordinary penalty.

Questions for authors
  • Which rows in Tables 1–2 use validation versus held-out test evaluation? Can all main methods be evaluated on the same independent test set under a unified selection protocol?

  • At the selected λr, does the complete method outperform Softmax-DPO + chosen-NLL, Prefix + chosen-NLL, and fixed-weight mass regularization? Can these comparisons control both candidate exposure and computation?

  • Why does Dual Mass fail to meet the SFT-calibrated budget? Is this due to incomplete optimization, a training–validation distribution difference, or another factor?

Reviewer_3CuROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the application of DPO and Softmax-DPO on generative recommendation tasks. The authors show through theoretical and empirical analysis that using while using DPO increases the margin between preferred and rejected items, it has the propensity to shift the probability mass towards invalid catalog items. To address this problem, the authors propose a joint loss function comprising of three signals: full-slate listwise optimization for ranking loss, slate supervised loss to preserve probability mass on preferred items and catalog-prefix loss to shift the probability mass away from invalid continuations at training time.

Empirical studies shows improvements in recommendation accuracy and catalog validity on top of SFT, DPO, Softmax-DPO on five recommendation datasets.

Critical strengths
  1. This paper shows a clear distinction between the application of DPO for ranking tasks versus it's application for generation tasks. The theoretical analysis clearly isolates the limitations of using relative preference objectives. Furthermore the checkpoint analysis which tracks preference margin, ranking accuracy, slate mass and invalid generation simultaneously shows a failure mode of preference optimization,
  2. The empirical results shows clear improvement in validity across give datasets along with gains in ranking performance.
Critical weaknesses
  1. It is not clear whether there is an established belief that DPO by itself is sufficient for generative recommendations task. While the authors have cited the Softmax-DPO paper and the theoretical analysis seems robust, the authors should clearly identify which prior conclusions in the existing literature needs revisions based on this work.
  2. Furthermore, the original Softmax-DPO paper reported high valid ratio of the GoodReads, LastFM and MovieLens task but the experiments in this paper show severe degradation in validity. While the metrics computation may not be equivalent, the authors should clarify the difference in valid ratio across the two papers.
  3. Full-slate scoring, fresh autoregressive rollouts, differentiable prefix re-scoring introduces additional computational overhead which have not been quantified. It is not clear whether using the same batch size or epochs leads to an equitable computational budget.
  4. The strongest weakness of this paper are missing baselines. There are several possible baselines for this paper:
  • Softmax-DPO + chosen-item NLL loss
  • weighted KTO
  • SFT loss + DPO loss
  • SFT/Softmax-DPO + Constrained decoding at inference time
Questions for authors

Please see the weaknesses

Reviewer_2UpFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify catalog-support drift in direct preference optimization for generative recommendation: improved discrimination between target and negative items does not necessarily prevent the model from generating invalid items. To address this issue, they theoretically analyze the generation process and propose DEHALL-SDPO, which augments the S-DPO objective with three components: a listwise policy ranking loss that promotes the target item over the full candidate set using raw policy scores, a mass-control objective that limits the probability assigned outside the candidate set, and an on-policy catalog-prefix support loss that encourages valid continuations along model-generated trajectories. Experiments demonstrate that DEHALL-SDPO reduces catalog-invalid generation under the evaluated greedy and nucleus-sampling settings.

Critical strengths
  • Intuitive design. The additional objectives are straightforward and encourage the model to assign more probability mass to valid items and continuations. The experiments also demonstrate reductions in catalog-invalid generation rates.
Critical weaknesses
  • Efficiency compared with constrained decoding. Generative recommenders can produce invalid outputs, particularly when generating items from the full catalog. However, constrained decoding can enforce catalog validity, raising questions about the practical advantages of DEHALL-SDPO. The proposed method introduces additional training costs through on-policy rollouts, auxiliary objectives, and full-candidate scoring. The authors should therefore compare DEHALL-SDPO with constrained decoding, in terms of recommendation quality, training overhead, and inference latency to establish its practical value. The authors are also encouraged to report results in a full-catalog generation setting, where the model generates recommendations without being given a candidate set in the prompt.

  • Accuracy gains and limited backbone. The recommendation accuracy gains from DEHALL-SDPO are modest across the five datasets. Given the importance of recommendation accuracy, further evidence is needed to establish the magnitude and robustness of these gains. Besides, all experiments use Mistral-7B-Instruct-v0.3 as the backbone. Evaluating more recent backbone models would help assess the generalizability of the findings.

Questions for authors

Please refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.