acceptodds
Desk rejected at ICLR 2027

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern recommender systems must adapt to dynamic, need-specific objectives, yet most are optimized for one static target. Reinforcement-learning post-training gives LLMs strong instruction-following abilities, suggesting a route to aligning them with complex recommendation goals. We study closed-set autoregressive ranking, where an LLM permutes a fixed candidate set given user context and a need instruction. Applying RL here raises two challenges: (i) an item's value changes across objectives, but sequence-level rewards collapse a ranking to one scalar and cannot identify which placements satisfy the current need; and (ii) interaction feedback is sparse and noisy, with reward reliability varying across users, items, and needs, causing inefficient and unstable updates. We propose FlexRec, a post-training RL framework combining (1) a sequential-dependence-aware item-level reward based on swaps within the remaining candidate pool and (2) critic-guided, uncertainty-aware scaling that models reward uncertainty and down-weights low-confidence signals. Across recommendation scenarios and objectives, FlexRec outperms strong baselines and improves NDCG@5 by up to 59%, Recall@5 by up to 109.4% in need-specific ranking, and Recall@5 by up to 24.1% in generalization over the base model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recommendation system, large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MpB8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FlexRec, a reinforcement learning post-training framework for adapting LLM-based recommenders to different recommendation needs. It studies closed-set autoregressive reranking, where the model generates a permutation of candidate items conditioned on user context and a need-specific instruction.

The method introduces two main components. First, a swap-based item-level reward estimates the contribution of individual ranking decisions by comparing each placement against counterfactual swaps with subsequent positions. Second, an uncertainty-aware GRPO mechanism employs a critic to estimate interaction rewards and predictive variances, using the latter to down-weight potentially unreliable policy updates.

The authors construct three recommendation objectives, namely Maximizing Interest, Explore New Topics, and Trend Promotion, on KuaiRec and MovieLens-1M. They additionally evaluate product search on ESCI. Experimental results show substantial improvements over several traditional and LLM-based ranking baselines, particularly for interest maximization.

Critical strengths

1. A relevant problem with a clearly motivated training framework.

Adapting a recommender to different objectives without maintaining separately optimized ranking models is an important research direction. The proposed framework addresses two meaningful challenges in LLM-based ranking: fine-grained credit assignment and unreliable reward supervision under sparse interactions. The method is coherent and provides a concrete way to incorporate both factors into GRPO-based training.

2. Encouraging ranking performance and useful ablation studies.

The results on the interest-maximization task are substantial. For example, FlexRec achieves NDCG@5 values of 0.597 on KuaiRec and 0.615 on MovieLens-1M, outperforming the strongest alternative post-trained baseline in Table 1. The paper also evaluates established and recent ranking methods, provides results on a third dataset, and examines reward normalization and uncertainty weighting. These results suggest that the proposed training recipe can be effective.

Critical weaknesses

1. The evaluation for flexible, need-conditioned recommendation could be further strengthened.

The central claim concerns adaptation to varying recommendation needs, but the new-topic and trending objectives are constructed using deterministic combinations of existing semantic, categorical, and popularity signals. In particular, the new-topic objective applies a multiplicative bonus to embedding-based relevance, while the trending objective combines normalized semantic similarity and popularity with fixed weights.

Although these constructions provide controlled evaluation targets, it is unclear how well they represent real, potentially conflicting user or platform objectives.

More importantly, the generalization results are mixed. In Table 6, FlexRec reduces Recall@5 and MRR@5 on KuaiRec Trend Promotion relative to the base LLM. The controlled introduction-switch experiment also reports relatively small differences between matching and non-matching instructions for some objectives.

Questions for authors
  1. Objective construction and practical validity: Why were the fixed coefficients for niche discovery and trend promotion selected? How sensitive are the results to these choices, and can the authors evaluate the method on more realistic or independently annotated need-specific objectives?
  2. Simple need-aware baselines: Since the constructed objectives depend on semantic similarity, item categories, and recent popularity, how would a deterministic objective-aware reranker, or a lightweight learning-to-rank model using the same information, perform?
  3. Reward ablation: Can the authors provide a factorial ablation separating future-only versus prefix-agnostic swaps from pooled versus rank-wise normalization? How much of the improvement is attributable to each design choice?
1 comment
AuthorsOpenReview ↗

Rebuttal for Reviewer MpB8

Q1. Objective construction and practical validity

Our construction follows the intuition that users can have several needs at once. A user may want to discover unfamiliar topics or follow current trends while still receiving relevant recommendations. Combining relevance with novelty or popularity gives us an objective that captures these preferences together.

X Trends uses a 60/40 split between similarity to the user’s interests and the trend score in its light-ranking stage [1]. Pandora’s Discovery mode lets listeners keep the same station while hearing more artists who rarely play there [2]. Our objectives capture these two product choices. For Trend Promotion, we likewise give relevance more weight than recent popularity (70/30). For Explore New Topics, we boost an item’s relevance score by 50% when its topic is absent from the user’s history, so a relevant but unfamiliar item can rise in the ranking. Therefore, we believe the multi-need label construction is realistic.

[1] X. Trends Recommendations. X Help Center.

[2] Pandora. Pandora: Personalized Music App. App Store, “Pandora Modes.”


Q2. Simple need-aware baselines

We already report two relevant baselines. STAR ranks candidates deterministically by cosine similarity to the user’s recent history (Appendix D, lines 835–842). BERT4Rec is a learned sequential ranker trained on interaction sequences (lines 320–322). Both are compared with FlexRec in Tables 1, 2, and 6, and STAR is also evaluated on ESCI in Table 9. The semantic retriever and the LLM rerankers receive the same item metadata (Appendix E, lines 853–854).


Q3. Reward ablation

The completed factorial comparison is shown below. Values are NDCG@5 on dense KuaiRec under GRPO.

Swap rulePooled normalizationRank-wise normalization
Future-only0.6070.460
Prefix-agnostic0.4610.492

With pooled normalization, future-only swaps improve NDCG@5 by 0.146. With rank-wise normalization, prefix-agnostic swaps score 0.032 higher. Pooled normalization improves future-only swaps by 0.147, while rank-wise normalization improves prefix-agnostic swaps by 0.031.


We hope these clarifications and the additional ablation address your concerns. In light of this evidence, we would appreciate your considering an upward revision of your rating.

Reviewer_t3YLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Closed-set autoregressive ranking: given a context, a candidate set, and a one-sentence "need" instruction, an LLM emits the candidates as an ordered sequence. The authors argue that sequence-level RL rewards cannot tell which placement served the active need, and propose two changes. First, an item-level reward built from counterfactual swaps against later positions in the remaining pool only. Second, a critic that predicts both mean and variance, with the predicted variance used to down-weight GRPO advantages from low-confidence rollouts. Results are reported for single-need training, joint training over three constructed needs, and zero-shot transfer, on KuaiRec, MovieLens-1M, and ESCI.

I'd welcome a response to the questions above. Realign the rollout budget and rerun every baseline, add seed variance, and show the model on a need whose objective wasn't constructed by the retriever — the paper would read very differently, and the score would move with it.

Critical strengths

The swap construction is right, and it targets a real problem. In autoregressive ranking the value of position k is genuinely only defined against the remaining pool; swapping with a fixed prefix item tells you nothing. The authors say this clearly and then turn it into something computable. Table 7 backs it up with separate rows for independent contribution, prefix-agnostic swaps, and rank-wise normalization, so the reader can see where the gain comes from instead of taking "our method is better" on faith.

Appendix I deserves credit too. The authors state the swap-effect identity in terms of gain differences, relate it to a local action advantage under exchangeable suffix completion, and then say outright that this does not give unbiased policy gradients or global objective equivalence. In a paper whose main text is written with a lot of confidence, that kind of hedge in the appendix is a good sign — they know where the boundary is.

Critical weaknesses
  1. The reward comparison is confounded by rollout budget, and the authors admit it. Appendix N says ReasonRank and BLADE-N were adapted without their external SFT or teacher pipelines, and that all four methods used four rollouts, one epoch, 833 updates — while FlexRec uses G=8. Their own sentence: "these results are not an isolation of the objective at identical sampled-token budget." That sentence cancels the interpretive value of every RL comparison in Table 1. Group size matters a lot for GRPO advantage estimation, and 4 versus 8 is easily enough to produce the whole gap in that table.

  2. All the baselines collapsing is itself the warning sign. On KuaiRec, Qwen2.5-3B-Instruct zero-shot gets N@5 = 0.375. After RL post-training Rec-R1 gets 0.391, ConvRec-R1 0.389, ReasonRank 0.385, BLADE-N 0.383. Four RL methods, all at or below the zero-shot line, three well under TALLRec's 0.507. Either these methods do nothing in this setting, or the adaptations weren't trained properly. Either way they can't serve as "strong baselines" against FlexRec's 0.597.

  3. The headline numbers are much bigger than the real gain. The abstract advertises +59% NDCG@5 and +109.4% Recall@5, both against the zero-shot backbone. Against the strongest non-FlexRec baseline, TALLRec at 0.507, the gain is 17.8%; on ML-1M against TALLRec's 0.569 it's 8.1%. And the 24.1% Recall@5 transfer figure comes from one cell, ML-1M under Explore New Topics. The same Table 6 shows KuaiRec Trend Promotion at Recall@5 −27.4% and MRR@5 −29.5%. Picking the best cell for the abstract while the negatives sit in the appendix is not an honest way to report.

  4. The "needs" are hand-written scoring functions, not user needs. Appendix B is explicit. Explore New Topics takes an embedding-retrieval base score and multiplies niche candidates by 1.5. Trend Promotion is 0.7 × normalized similarity + 0.3 × normalized activity. And the candidate sets themselves come from the same embedding retriever (text-embedding-004, following STAR). So the model is trained to reproduce "retriever score plus a constant bonus," then evaluated on that same map. That's circular. Any model that learns to check whether a topic label appeared in the history wins, and that is a different thing from "learning reusable ranking principles" — which is the entire basis of the transfer claim.

  5. Everything is a single-run point estimate. Appendix N says it outright: "All scores are single-run point estimates." No seed count, no variance, no intervals anywhere in the paper. Table 7 shows why that matters. The same prefix-agnostic swap reward scores 0.461 under GRPO and 0.383 under PPO — a spread of 0.078. The authors' claimed core effect, pooled versus rank-wise normalization, is 0.607 versus 0.460, a gap of 0.147. Those two are within a factor of two. RL on a 3B model routinely moves several NDCG points between seeds. Without seeds there is no way to separate the effect from the noise.

Questions for authors
  1. What happens to Rec-R1 and ConvRec-R1 N@5 if you set group size to 8 and match the update count? If they land above 0.55, how much of Table 1 survives? Without that control, how do you attribute the gain to the reward design rather than the sampling budget?

  2. In Table 8, constant and shuffled weighting produce identical numbers on all five metrics. What is the actual observed range of c_i within a single prompt group? If it is near-constant, has the uncertainty mechanism ever been active?

  3. The Explore New Topics and Trend Promotion targets are produced by the same retriever that builds the candidate sets, plus hand-set coefficients. How do you separate "learned ranking principles" from "reproducing the retriever score times 1.5"? Is there any evidence the model works on a need whose target function it was not trained against?

  4. The terminal item always draws a negative advantage and is not masked. Have you measured what happens if you drop it, or normalize it separately?

  5. Please report mean and standard deviation over at least three seeds, especially for the pooled versus rank-wise comparison in Table 7. With a single run, how much of that 0.147 is seed noise?

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer t3YL

Q1. Matching the training budget

Thank you for flagging this. We incorrectly reported four rollouts per prompt in Appendix N. All RL methods in Table 1 used eight, including ReasonRank and BLADE-N. Rec-R1, ConvRec-R1, and FlexRec also completed the same number of optimizer updates on KuaiRec. The NDCG@5 scores under those settings are the ones in Table 1: 0.391 for Rec-R1, 0.389 for ConvRec-R1, and 0.597 for FlexRec. Table 10 gives a more direct comparison of the reward designs on dense KuaiRec, where the training budget and feedback source were held fixed: FlexRec scored 0.607, Rec-R1 0.572, and ConvRec-R1 0.381. We will correct Appendix N.


Q2. Variation in uncertainty weights

For each prompt group, we measure the coefficient range as max(c_i) − min(c_i). In FlexRec, this range has a mean of 0.413 and a standard deviation of 0.214 across groups, showing substantial variation among responses to the same prompt. The identical scores in Table 8 summarize the test performance of the constant and shuffled controls. Constant weighting removes the variation within each group, while shuffling breaks the association between a response and its uncertainty weight. FlexRec achieves NDCG@5 of 0.595, compared with 0.565 for either control.


Q3. Transfer to unseen needs

Tables 2 and 6 already report transfer to two needs held out during training. We trained FlexRec only on Maximizing Interest, using watch ratios on KuaiRec and ratings on MovieLens-1M. The discovery bonus and the popularity weight entered only when we scored the rankings at test time. FlexRec improved NDCG@5 over its original backbone on both needs on both datasets. On MovieLens-1M, for example, FlexRec scored 0.748 on Explore New Topics and 0.630 on Trend Promotion, compared with 0.729 and 0.581 for the backbone.

The constructed objectives reflect choices that recommendation platforms face in practice. X Trends combines relevance to a user’s history with a trend score [1]. Pandora’s Discovery mode plays artists a listener would rarely hear on the same station [2]. Our coefficients give us concrete versions of relevance–popularity and relevance–novelty trade-offs to evaluate.

[1] X. Trends Recommendations.

[2] Pandora. Pandora Modes, in the Pandora app description.


Q4. The terminal item

Our swap reward measures what the model gains by placing one remaining candidate ahead of another. At the last rank, only one candidate remains, so there is no ranking decision left to reward. Zero is natural here: an earlier rank also receives zero raw reward if every available swap leaves NDCG unchanged. We apply the same pooled normalization to every rank, so the final item’s advantage is negative only when the pooled mean is positive (Appendix H.1, lines 1020–1025).

Table 7 includes a control that normalizes each rank separately and gives the final item zero advantage in every rollout (Appendix I, lines 1156–1162). We also report repeated runs of this control in our response to Q5 below.


Q5. Results across three seeds

We repeated the GRPO ablations with three seeds. The table reports mean ± sample standard deviation across three seeds:

MethodNDCG@5NDCG@10NDCG@30Recall@5MRR@5
Future (pooled)0.5634 ± 0.08110.6112 ± 0.06110.7738 ± 0.03380.3114 ± 0.04890.7980 ± 0.0805
Future (rank-wise)0.4376 ± 0.02760.4978 ± 0.01930.7188 ± 0.01210.2123 ± 0.02180.6716 ± 0.0612
Independent0.4744 ± 0.03610.5439 ± 0.02320.7378 ± 0.06140.2394 ± 0.02600.6503 ± 0.080
Prefix-agnostic0.4272 ± 0.02920.4881 ± 0.02280.7138 ± 0.01490.1982 ± 0.01370.6289 ± 0.0761

Pooled normalization retains a mean NDCG@5 gain of 0.1258 over rank-wise normalization and achieves higher mean performance on all five metrics. The improvement therefore persists when averaged across three seeds.


We hope these clarifications and the additional experiments address your concerns. In light of this evidence, we would appreciate your considering an upward revision of your rating.

Thanks for your response. After carefully reading the authors' rebuttal, I have decided to maintain my current score.

Reviewer_YFcgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies closed set autoregressive ranking and proposes FlexRec, a post training RL framework with two technical components to address two key challenges when applying RL for LLM based recommendation. Experiments verify the performance improvements brought by the proposed framework.

Critical strengths
  1. The research motivation is clearly stated.
  2. The paper includes relevant theoretical analysis for the swap based reward design.
  3. Empirical experiments demonstrate the effectiveness of FlexRec across multiple datasets and multi need scenarios.
Critical weaknesses
  1. Statistical significance tests are absent. It is unclear whether the performance gains come from random seed variation.
  2. Empirical results are presented, yet the underlying mechanism behind the observed improvements is insufficiently analyzed.
  3. The overhead of swap based item level reward is only discussed from a theoretical perspective with a claimed <0.1% extra training time. Empirical wall clock efficiency measurements are required to back this claim.
  4. There are several grammar errors, such as “Table 1 and 9 reports performance”;
Questions for authors

1.The swap based item level reward relies on counter factual swaps with future positions under a fixed prefix. However, it remains unclear how sensitive this reward is to bad rollout samples. When the LLM generates very poor ranking sequences, would this item level reward provide misleading training signals?

  1. The critic predicts both reward mean and predictive variance for uncertainty aware weighting. But the paper does not evaluate whether the predicted uncertainty is well calibrated.

  2. Table 7 presents ablation studies, but there is no ablation that removes both item level reward and uncertainty aware update simultaneously, to show the joint effect of the two core components.

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer YFcg

Q1. Effect of poor rollouts

A poor rollout helps FlexRec learn which placements to avoid. Suppose the model samples [low, medium, high] and [high, medium, low] for the same prompt, where the labels denote relevance. Swapping the first item with either later item improves the first ranking and worsens the second. Our swap reward (Eq. 1) therefore gives negative credit to placing low first and positive credit to placing high first. This guides the model toward rankings like the second in future rollouts.


Q2. Evaluation of predicted uncertainty

Table 3 in Appendix H (lines 925–932) evaluates how predicted uncertainty relates to prediction error. We compute the Pearson correlation between each predicted variance and the squared error of the corresponding mean prediction. The correlation is positive and statistically significant on both KuaiRec (rho = 0.300) and MovieLens-1M (rho = 0.132), with p < 0.05 on each dataset. These results show that higher predicted variance tends to accompany larger errors. For the downstream use of this signal, Table 8 shows that uncertainty weighting raises NDCG@5 on KuaiRec from 0.566 to 0.595. Absolute calibration of the predicted variances remains to be evaluated.


Q3. Removing both components

Removing both components gives standard GRPO with a single reward for the whole ranking and no uncertainty weighting. This is the Rec-R1 baseline we already report in Tables 1 and 10.


We hope these clarifications make the method and the existing experimental evidence clearer. If they address your main concerns, we would appreciate your considering an upward revision of your rating.

Reviewer_YFcgOpenReview ↗

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

After carefully reading the responses, I note that some of the concerns have been addressed in the relevant experiments, yet not all raised concerns are fully resolved. Therefore, my reviewer decision remains unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.