acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Sampling Imbalance in Generative Listwise Recommendation via Discrepancy-Guided Exploration

Abstract

Generative listwise recommender systems search over a combinatorial space of item lists, yet sparse interaction feedback often leads to highly imbalanced exploration: training increasingly concentrates on well-covered lists, while potentially high-utility lists remain rarely sampled and insufficiently evaluated. We identify this sampling imbalance as a key bottleneck and show that the discrepancy between a list's empirical reward and its model-assigned selection probability provides a direct signal of under-exploration. Building on this insight, we propose a Teacher–Student collaborative framework (Tesco) for discrepancy-guided exploration. An auxiliary Teacher combines the Student's discrepancy signal with list reward to preferentially sample under-explored yet promising lists, while the Student learns from both self-generated and Teacher-generated experiences through alternating sampling with a shared replay buffer. This mechanism enables targeted exploration beyond the Student's current sampling distribution without modifying the original reward definition or observed interaction data. Experiments on three real-world recommendation datasets under both simulated online and offline settings show that Tesco consistently improves list-level utility and ranking quality while maintaining competitive diversity and coverage, with average-reward gains of up to 4.5% over the strongest baseline. Extensive ablation and sensitivity analyses further validate the effectiveness and robustness of the proposed discrepancy-guided exploration mechanism. The code is available at https://anonymous.4open.science/r/Tesco-77DD/.

Primary area:
reinforcement learning
Keywords:
Listwise Recommendation, Sampling Imbalance, Teacher–Student Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_h68eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TESCO, a Teacher-Student method that uses the student's trajectory-balance discrepancy to sample lists in generative recommendation. TESCO-TB improves the average reward over GFN4Rec-TB on three datasets, with a modest memory cost. The motivation is not well justified, and further improvements are needed in terms of logical rigor.

Critical strengths
  1. Using the trajectory-balance discrepancy to guide sampling fits the GFlowNet objective directly. The signal is already computed during Student training, so no extra reward model is required.
  2. The Teacher reward mixes this discrepancy with the list reward. On KuaiRand-1K, this full setting beats both the reward-only and loss-only variants in average and maximum reward.
  3. TESCO-TB improves average reward over GFN4Rec-TB on all three datasets, in both the simulated online and offline settings. The extra cost is modest: about 12–15% more GPU memory, since the Teacher shares the backbone.
Critical weaknesses
  1. The motivational claims in the third paragraph of the Introduction are largely subjective and lack supporting evidence. The following points need further justification: a) "existing generative approaches still suffer from insufficient exploration of lists that have never interacted with users" needs to be supported by empirical data analysis; b) "This self-reinforcing training dynamic traps the model in a local optimum defined by historical data, pre-venting it from perceiving or learning from unobserved yet informative high-loss regions, and ultimately exacerbating exposure bias." This claim lacks supporting evidence.
  2. The correctness of the motivation "prioritizing under-explored lists that may match the user’s preferences" itself should be justified first, but the authors do not do so.
  3. The references include only one paper from 2025 and none from 2026. Is this because the research topic is already old?
  4. In Table 8, on KuaiRand-27K, TS-TB (n=3) has an average reward of 2.365 ± 0.032 and a maximum reward of 2.065 ± 0.022. If the maximum reward is the maximum over the same set of lists, it cannot be lower than the average reward of those lists.
  5. Figure 4 first discards configurations whose average reward is not above 2.40, and then concludes from the remaining successful configurations that the method is insensitive to C and α. Whether the method is sensitive depends precisely on the configurations that were discarded.
  6. Sections 2.2 and 2.3 mostly restate GFlowNets and GFN4Rec, so the new Teacher reward is hard to separate from prior work. It is recommended to move this background into a Preliminary section and state what is new relative to GFN4Rec and Kim et al. (2025).
Questions for authors
  1. In the second paragraph of the Introduction, it is still unclear why the probability mass over generated lists concentrates on already discovered high-reward lists. What does this have to do with cross-entropy? Cross-entropy is itself a proper scoring rule for probabilities; if the data distribution is well specified, it should in principle recover the appropriate probabilities.
11 comments
AuthorsOpenReview ↗

Response to Reviewer h68e (1/9)

Dear Reviewer h68e,

We sincerely thank you for your careful review and constructive feedback. We are encouraged by your recognition of TESCO’s sampling strategy, reward improvements, and modest memory overhead. Your comments provide valuable guidance for strengthening the motivation, clarity, and logical rigor of our paper. Below, we respond to your concerns point by point and describe the corresponding revisions.

W1. The motivational claims in the third paragraph of the Introduction are largely subjective and lack supporting evidence. The following points need further justification: a) "existing generative approaches still suffer from insufficient exploration of lists that have never interacted with users" needs to be supported by empirical data analysis; b) "This self-reinforcing training dynamic traps the model in a local optimum defined by historical data, pre-venting it from perceiving or learning from unobserved yet informative high-loss regions, and ultimately exacerbating exposure bias." This claim lacks supporting evidence.

Thank you for your suggestions, and we provide detailed clarifications below. We agree that these two claims require empirical support, and that we should distinguish coverage in list generation, the potential value of underexposed content, and the effects of feedback loops on training and exposure distributions. We address questions (a) and (b) separately below.

1. Regarding question (a), existing list-generation experiments show that training directly on offline historical trajectories can limit the coverage of generated results. The ICML 2026 paper Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Training conducted experiments generating six-item lists on MovieLens-1M. Appendix E.1, Table 4 reports that Dataset-GFN, trained directly on offline trajectories, covers 7.9±0.9 distinct items in its generated samples and achieves an average reward of 1.681±0.202. With an improved training method, TD-GFN achieves coverage of 186.8±6.3 and an average reward of 1.988±0.006. These results are means and standard deviations over three random seeds. They show that, in this experimental setting, direct training on historical trajectories does limit coverage, and that broader coverage can be achieved alongside higher rewards. Here, Coverage measures the number of distinct items appearing in generated lists; it does not directly measure the discovery rate of lists with which the user has not previously interacted.

We further analyzed real randomized-exposure logs from KuaiRand-1K. The results are as follows:

ObservationCount
Randomized exposures of videos never previously shown to the corresponding user41,953 exposures
Exposures that resulted in long views3,543 exposures (8.45%)
Distinct users responsible for these long views677 users
Likes received by previously unreached candidates238 likes
Long-view pairs for which the video had previously been shown to other users2,755 pairs (77.76%)

The most direct finding is that 3,543 user–video pairs had no exposure records in the regular logs for the entire month, yet received long views after randomized exposure. Our analysis of the real KuaiRand-1K logs shows that randomized exposure can discover valuable interactions that regular recommendations have not reached. Among users with prior viewing histories, 3,543 previously unreached user–video pairs received long views after randomized exposure. This indicates that relying on existing exposure records can overlook user interests with recommendation value, providing empirical motivation for further exploration of candidates with which the user has not previously interacted.

AuthorsOpenReview ↗

Response to Reviewer h68e (2/9)

Existing studies using randomized-exposure test data also support the possibility that historical logs omit some learnable user interests. The SIGIR 2025 paper Improving Sequential Recommenders through Counterfactual Augmentation of System Exposure trains on regular exposure data and validates and tests on KuaiRand randomized-exposure data. In its Table 2, SASRec achieves a Recall@20 of 0.0052, whereas CaseRec-R, which introduces counterfactual exposure augmentation, achieves 0.0229; NDCG@20 also increases from 0.0017 to 0.0099. This shows that supplementing training information from regions insufficiently covered by the historical exposure policy can improve recommendation performance on randomly exposed content. Although this study concerns sequential recommendation, it provides complementary evidence on the relationship between exposure and preference learning. Thus, existing generative recommendation methods still face insufficient exploration of recommendation lists with which the user has never interacted.

2. Regarding question (b), closed-loop experiments on generative recommendation have observed exposure distributions becoming more concentrated through feedback loops. The 2026 preprint Do Generative Recommenders Deepen the Information Cocoon? uses an LLM-based user simulator to conduct 15 successive rounds of recommendation, user selection, and model retraining. On Toys and Games, the exposure Gini coefficient increases from 0.933 to 0.970 for TIGER and from 0.903 to 0.916 for OneRec. A higher Gini coefficient indicates more concentrated exposure. This provides experimental evidence that repeatedly using feedback generated by the model’s own recommendations can sustain or reinforce exposure concentration.

Research on high-loss-guided sampling further shows that changing the training sampling distribution can alleviate the problem of a model’s own policy missing valuable modes. The ICLR 2025 paper Adaptive Teachers for Amortized Samplers reports experiments on a four-dimensional deceptive grid with side length 32. Trajectory Balance using samples from its own policy discovers 16.6±4.8 modes on average, prioritized experience replay discovers 46.6±14.6, and a Teacher guided by the Student’s loss discovers 246.6±14.7. These results support the role of the training sampling distribution in mode discovery and show that loss-based guidance can help discover regions poorly covered by the original policy. Because this is a synthetic sampling task, its support for the recommendation training mechanism should be understood as evidence at the mechanism level.

Long-term controlled comparisons in production systems also show that increased exploration can obtain real interactions with content that had historically received limited exposure. The KDD 2026 paper PinEqualizer: Full Funnel Content Exploration and Debiasing System at Pinterest reports that, in long-term controlled comparisons starting in the second half of 2024, its exploration and debiasing system achieved cumulative increases in interactions with underexplored content of 37%, 13%, and 27% on Homefeed, Search, and Related, respectively. This supports the role of exposure opportunities in feedback accumulation and demonstrates the practical value of selective exploration of underexposed content.

Together, these findings support the motivation for valuable exploration under limited feedback. For question (a), list-generation experiments and the analysis of real randomized-exposure logs support the existence of limited coverage and the potential value of poorly covered candidates. For question (b), closed-loop recommendation experiments, mode-discovery experiments, and production comparisons support feedback reinforcement and the need to adjust the sampling distribution. We have revised the third paragraph of the Introduction to add evidence from TD-GFN, our KuaiRand-1K data analysis, and closed-loop simulations of generative recommendation, providing stronger support for our research motivation.

AuthorsOpenReview ↗

Response to Reviewer h68e (3/9)

W2. The correctness of the motivation "prioritizing under-explored lists that may match the user’s preferences" itself should be justified first, but the authors do not do so.

Thank you for your suggestion. We have updated the third paragraph of the Introduction to include analyses of TD-GFN and KuaiRand-1K, as detailed below.

1. Real randomized-exposure data show that candidates not yet reached by regular recommendations can contain content that matches user preferences. Our KuaiRand-1K analysis identifies 41,953 randomized exposures of videos never previously shown to the corresponding users. Of these, 3,543 exposures result in long views, accounting for 8.45% and involving 677 users. This indicates that candidates outside historical exposure coverage can still receive positive feedback, providing direct empirical motivation for exploring insufficiently reached content.

2. Existing list-generation experiments further show that broader generation coverage can be achieved alongside higher recommendation rewards. In the aforementioned experiment generating six-item lists on MovieLens-1M, Dataset-GFN covers 7.9±0.9 distinct items and achieves an average reward of 1.681±0.202, whereas TD-GFN achieves coverage of 186.8±6.3 and an average reward of 1.988±0.006. This result supports the research direction of improving both coverage and preference-related utility in list generation. Production results provide complementary evidence: Pinterest’s exploration and debiasing system achieved cumulative increases of 37%, 13%, and 27% in interactions with underexplored content across three product surfaces, demonstrating the practical value of selective exploration of underexposed content.

Together, these findings support the motivation for exploring insufficiently covered recommendation spaces under the guidance of user preferences. The randomized-exposure analysis supports the potential value of unreached candidates, while list-generation experiments and production results support the benefits of broader exploration. Overall, this evidence sufficiently establishes the validity of our research motivation.

Reviewer_2iM5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes TESCO, a Teacher-Student framework for improving exploration in generative listwise recommendation. The Student is a GFlowNet trained to match list-generation probabilities to rewards. An auxiliary Teacher targets trajectories with positive Student discrepancies, combining discrepancy-based weighting with list rewards. Both models learn through alternating sampling and a shared replay buffer. Experiments on MovieLens-1M and two versions of KuaiRand report improved average rewards over GFN4Rec in simulated online and offline settings, with additional ablations and computational analyses.

Critical strengths
  • Searching the combinatorial space of recommendation lists can concentrate training on a restricted set of outcomes. Learning an auxiliary exploration policy is a reasonable way to broaden the Student’s training distribution.

  • Sharing the backbone while maintaining separate decision heads limits additional computation. Stop-gradient through the Teacher’s reward also provides a sensible separation between the two optimization objectives.

  • TESCO-TB improves simulated online average reward over GFN4Rec-TB by approximately 3.5% on MovieLens-1M and 4.4-4.5% on KuaiRand. Results average five runs and include uncertainty estimates. The appendix also reports training-time increases of approximately 12-13%, providing useful context for the accuracy gains.

Critical weaknesses
  • Adaptive Teachers for Amortized Samplers already introduces an auxiliary Teacher trained on Student discrepancy, additional weighting for positive discrepancies, reward mixing, and joint Teacher-Student training with replay. These constitute the central mechanism of TESCO. TESCO’s main changes appear to be its recommendation application, backbone sharing, and an asymmetric discrepancy transformation that uses a linear positive branch and a squared negative branch. These changes may be useful, but the paper does not establish why they provide a substantive advance over directly applying the existing method to GFN4Rec. In particular, comparing the proposed reward against the original adaptive-teacher reward under identical conditions is essential.

  • A positive discrepancy indicates insufficient model-assigned probability relative to the estimated reward target. It does not necessarily establish that a list has received insufficient historical exploration. The same discrepancy can arise from model misspecification, an inaccurate normalizer, or stochastic reward fluctuations.

  • The manuscript should explain whether offline training restricts sampling to logged lists, labels generated lists using a learned simulator, or uses another mechanism.

  • The closest missing comparisons are the original adaptive-teacher method and loss-prioritized replay. Both directly address the proposed explanation for improvement. A tuned exploration baseline would also help distinguish the benefit of learned discrepancy guidance from a broader behavior distribution.

  • Comparisons should match environment interactions and reward-query budgets, alongside reporting computation. Equal training-step counts alone may conceal differences in the experiences collected or optimization performed. The reported overhead analysis is useful, but does not establish improved sample efficiency under a matched interaction budget.

Questions for authors

Please see weakness above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2iM5 (1/5)

Dear Reviewer 2iM5,

We thank the reviewer for the time and effort devoted to evaluating our work and for the constructive feedback. We are encouraged by your recognition of TESCO’s exploration strategy, efficient design, and empirical improvements. Your comments provide valuable guidance for strengthening the manuscript. Below, we address your concerns point by point and explain the corresponding revisions.

W1. Adaptive Teachers for Amortized Samplers already introduces an auxiliary Teacher trained on Student discrepancy, additional weighting for positive discrepancies, reward mixing, and joint Teacher-Student training with replay. These constitute the central mechanism of TESCO. TESCO’s main changes appear to be its recommendation application, backbone sharing, and an asymmetric discrepancy transformation that uses a linear positive branch and a squared negative branch. These changes may be useful, but the paper does not establish why they provide a substantive advance over directly applying the existing method to GFN4Rec. In particular, comparing the proposed reward against the original adaptive-teacher reward under identical conditions is essential.

Thank you for this suggestion. We have moved this background material to the Preliminaries section in the revised manuscript. Below, we explain TESCO’s specific innovations relative to GFN4Rec and Adaptive Teacher.

  1. TESCO replaces the positive-residual branch of the Teacher reward with a linear form, preserving the learning signal from small positive discrepancies while slowing the growth of weights for large positive residuals. GFN4Rec uses a single generator trained on list rewards; Adaptive Teacher uses a quadratic positive-residual term, (1+C)δ2(1+C)\delta^2, mixed with the task reward. In Table 5, TESCO improves average reward from 2.394 to 2.430 compared with the control that transfers the original Teacher reward to the recommendation setting.
  2. TESCO shares user preference representations while retaining independent sampling decisions. GFN4Rec uses one user encoder and one generation policy, whereas the Teacher and Student in Adaptive Teacher learn their respective policies. TESCO shares a user-encoding backbone updated by the Student while retaining independent heads for the forward policy and initial flow estimator, allowing the models to reuse preference knowledge and learn different proposal distributions.
  3. TESCO allocates training opportunities according to user-specific list utility and the Student’s current fitting needs. GFN4Rec already supports user-conditioned list generation, and Adaptive Teacher already adjusts sampling using terminal rewards and Student residuals. TESCO applies these two signals to user–list pairs, assigning different priorities to lists with equal rewards but different degrees of fit under the current model.
  4. TESCO’s recommendation ablations demonstrate the complementary roles of reward and residual signals. GFN4Rec already evaluates reward and diversity, and Adaptive Teacher already combines task rewards with residuals. TESCO’s full model improves both reward and coverage over Reward-Only, while achieving higher reward with lower coverage than Loss-Only, supporting the value of the joint signal in this recommendation setting.
  5. TESCO achieves gains in both reward and coverage on a larger recommendation catalog. The original GFN4Rec paper evaluates ML1M and KuaiRand-1K, whereas the original Adaptive Teacher paper does not report KuaiRand recommendation experiments. On KuaiRand-27K, which contains 507,493 items, TESCO improves average reward by 4.48% and the number of distinct items per batch by 5.10% over GFN4Rec-TB in the simulated online Greedy setting, while increasing ILD from 0.529 to 0.544.

These comparisons summarize TESCO’s reward modification, shared representations with independent decisions, and allocation of training opportunities at the user–list level. The ablations and KuaiRand-27K results support the value of these designs in the evaluated recommendation settings.

AuthorsOpenReview ↗

Response to Reviewer 2iM5 (2/5)

1. The motivation and central theme of our work remain valid: allocating list-sampling opportunities according to user utility and the model’s learning state under limited training resources. Reward matching provides a learning target for list generation, while the extent to which rewarding lists influence training also depends on their sampling opportunities and how well the current policy fits them. In personalized recommendation, this issue further manifests in different users responding differently to the same list and different lists for the same user presenting different learning needs. We study how to use existing feedback and the Student’s current fitting discrepancy to continually adjust sampling priorities for user–list pairs, directing training opportunities toward candidate lists that merit further learning. This directly concerns how recommender systems use training feedback in a combinatorial space.

2. We translate this research question into a concrete recommendation training mechanism through the positive-residual transformation, shared preference representations, and independent policies. The linear positive branch retains more weight for small positive residuals while slowing weight growth for large positive residuals, allowing the Teacher to adjust competition for sampling opportunities between high-reward lists with small discrepancies and lists with larger errors. Sharing the user-encoding backbone allows the Teacher to reuse preference knowledge learned by the Student, while independent policy and flow-estimation heads allow the two models to learn list selection according to their respective objectives. The shared representation is updated by the Student, so its direct optimization follows the recommendation objective. Together with user-conditioned rewards and residuals, these designs form a collaborative training scheme that considers personalized recommendation value, current fitting needs, and knowledge reuse, specifying how adaptive sampling can be applied to personalized listwise recommendation.

3. The original-reward control, component ablations, and larger-catalog experiments jointly support the value of these designs in the evaluated recommendation settings. TESCO achieves a higher average reward than the control using the original Teacher reward, indicating a task-level benefit from the positive-residual transformation that merits investigation. The Reward-Only and Loss-Only ablations further reveal how the joint signal affects the trade-off between utility and coverage. On KuaiRand-27K, with its larger recommendation catalog, the full framework substantially improves average reward, maximum reward, and the number of covered items over GFN4Rec-TB. These results connect the method’s design to the practical objectives of recommendation quality and item coverage, providing a concrete approach and empirical evidence for organizing training samples around user utility and learning needs.

Finally, we have clarified our specific innovations in the second contribution listed in the Introduction of the revised manuscript. We introduce TESCO, which allocates limited list-sampling opportunities according to user-specific utility and the Student’s current fitting needs. The Teacher combines observed list rewards with asymmetric discrepancy weighting, adjusting sampling priorities through a linear positive-residual branch. The shared user encoder, updated by the Student, enables preference knowledge to be reused, while independent policy and initial flow heads allow the two models to learn different list-generation policies.

We have also added comparisons with the original Adaptive Teacher reward function and loss-prioritized replay, as detailed in W4.

AuthorsOpenReview ↗

Response to Reviewer 2iM5 (3/5)

W2. A positive discrepancy indicates insufficient model-assigned probability relative to the estimated reward target. It does not necessarily establish that a list has received insufficient historical exploration. The same discrepancy can arise from model misspecification, an inaccurate normalizer, or stochastic reward fluctuations.

Thank you for this valuable comment. We first clarify the meaning of discrepancy under the Trajectory Balance objective: the forward trajectory flow represents the unnormalized probability mass that the Student assigns to the trajectory through its generation policy, while the backward trajectory flow is determined by the terminal reward and backward policy. The discrepancy is the difference between their logarithms. A large positive discrepancy therefore indicates that, under the current reward and normalization estimates, the Student’s forward flow is lower than the backward target flow, leaving a large flow-matching discrepancy for that trajectory.

TESCO uses this discrepancy to adjust the training sampling distribution and combines it with list reward so that the Teacher considers both recommendation value and the Student’s current fitting needs. Here, discrepancy serves as a sampling-priority signal rather than an estimate of the number of historical exploration attempts.

Overall, we agree that this residual alone cannot identify the source of the mismatch. Insufficient sampling, limited model capacity, normalization estimation errors, and stochastic reward fluctuations can all produce positive discrepancies. We do not claim that insufficient sampling is the sole cause. We use the rewards of previously evaluated lists and the Student’s discrepancies in fitting those rewards to train the Teacher to propose subsequent lists for evaluation. The Student can then learn from feedback obtained through both its own samples and the Teacher’s samples. Our focus is whether this adaptive proposal mechanism improves the final list reward under the same total feedback-query budget. We have revised the relevant wording to “insufficient forward-flow allocation relative to the reward target under the current estimates” and clarified its role as a heuristic sampling signal in the fourth paragraph of the Introduction.

W3. The manuscript should explain whether offline training restricts sampling to logged lists, labels generated lists using a learned simulator, or uses another mechanism.

Thank you for this valuable comment. We provide a detailed explanation below.

Our offline training uses only the fixed training logs and their existing feedback, without querying the simulator to supply reward labels for newly generated lists. Specifically, training lists are constructed from user histories according to the preprocessing rules in Appendix C, with list length K=6K=6. List rewards are calculated from the multi-behavior feedback in the corresponding records. The user states, lists, and feedback used in offline training all come from the training set, and no new simulator-generated interaction data are added during training.

In this setting, both the Student’s training discrepancy and the Teacher’s auxiliary reward are computed from existing logged samples, so discrepancy guidance is restricted to training data supported by the logs. The offline experiments therefore examine learning from existing feedback data. In contrast, the simulated online experiments allow the Student and Teacher to generate lists, obtain simulator feedback, and use these experiences for subsequent training, examining their effects on exposure allocation and exploration. We have added this explanation to Section 3.1 of the revised manuscript.

Reviewer_wV77OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies insufficient exploration in generative listwise recommendation caused by imbalanced training-sample coverage. The authors propose TESCO, a Teacher–Student collaborative framework in which the Student's GFlowNet discrepancy — the residual, in log space, between a list's reward and the model-assigned probability mass — is used to gauge how under-explored a list is, while an auxiliary Teacher combines this discrepancy signal with the list reward to preferentially sample lists that are under-explored yet potentially high-utility. The two models are trained with alternating sampling over a shared replay buffer. Experiments are conducted on MovieLens-1M and on two scales of KuaiRand, under both simulated online and offline evaluation.

Critical strengths
  1. The problem is practically relevant. The paper targets the imbalance between exploration and exploitation in generative listwise recommendation, where a model may concentrate on list combinations that already received substantial training feedback while ignoring other combinations with potential value. This issue is closely related to exposure bias, popularity bias, and long-tail recommendation, and the motivation is well developed.

  2. The method design is sensible and the modification cost is modest. TESCO introduces an auxiliary Teacher on top of an existing GFlowNet-based list recommendation framework, combining the Student's training discrepancy with the list reward to form the Teacher reward and using a shared replay buffer to support alternating sampling. The Teacher and the Student share the user encoder and backbone and keep only separate decision heads, so the original recommender need not be rebuilt, and the design can be combined with existing generative listwise recommenders.

  3. The experiments examine both utility and exploratory capacity. The authors compare average and maximum reward, coverage, intra-list diversity (ILD), and ranking metrics (R-NDCG, R-MRR) on datasets of different scales, and further analyze the Teacher reward design, the Teacher–Student sampling ratio, and computational overhead. The experiments on KuaiRand-27K, the largest setting, provide some evidence for the method's applicability to larger combinatorial spaces.

Critical weaknesses
  1. The soundness of using training discrepancy to measure under-exploration still needs to be established. The paper interprets a large positive discrepancy as evidence that the list has not been sufficiently explored by the Student, but a large residual may equally arise from noise in the reward estimate (the stochasticity of the simulator's sampling), insufficient model capacity, or information loss in aggregating item-level rewards — none of which implies that the list is worth further exploration. More importantly, the discrepancy can only be computed for trajectories that have already been sampled, so using it to identify under-explored regions is circular: sampling must come first, and the signal follows.

  2. According to Eq. 12, the Teacher reward contains R(s_K) raised to the power alpha. However, the dataset and reward description in the appendix indicate that the item-level reward weights for KuaiRand include a negative value (the hate signal is assigned a negative weight), so the list-level reward can be negative; at the same time, the hyperparameter sensitivity grid contains several non-integer values of alpha. For a negative base and a non-integer exponent, the power is not defined over the reals. The paper does not explain whether the reward is shifted, clipped, or otherwise made non-negative.

  3. The simulated online environment provides limited support for real-world effectiveness. The online experiments rely on a pretrained neural user-response simulator rather than real user interactions. Although the authors report that the rankings induced by the simulator largely agree with offline rankings, this agreement may partly stem from the simulator itself being trained on offline data (the authors also acknowledge that it may converge to behavior resembling the offline distribution), and it is therefore insufficient to establish the exploration benefit in a real environment. In addition, the list reward is obtained as a weighted average of item-level feedback; the paper should further clarify whether the model can capture the combinatorial dependencies within a list, and whether the simulator reflects such dependencies.

  4. The analysis of the accuracy–coverage trade-off is not yet sufficient. TESCO attains higher average and maximum reward in greedy mode, whereas the exploration mode (Explore) yields higher coverage and ILD, and a clear gap remains between the two configurations. The current results do not yet establish that the method remains superior to the baselines under an equal exploration budget or an equal coverage constraint. The authors should report the reward–coverage trade-off curve and compare methods under a fixed sampling budget. In addition, the paper reports means and standard deviations over multiple random seeds but provides no paired significance test on the main tables, while the appendix nevertheless uses phrases such as "not significant"; the statistical protocol should be made explicit and testable.

Questions for authors

See Weaknesses

9 comments
AuthorsOpenReview ↗

Response to Reviewer wV77 (1/7)

Dear Reviewer wV77,

We thank the reviewer for the careful evaluation and constructive feedback. We are encouraged by your recognition of the practical relevance of our problem, the lightweight Teacher–Student design, and the evaluation of both recommendation quality and exploration across datasets of different scales. Your comments have guided our efforts to improve the clarity and completeness of the paper. Below, we provide point-by-point responses and describe the corresponding revisions.

W1. The soundness of using training discrepancy to measure under-exploration still needs to be established. The paper interprets a large positive discrepancy as evidence that the list has not been sufficiently explored by the Student, but a large residual may equally arise from noise in the reward estimate (the stochasticity of the simulator's sampling), insufficient model capacity, or information loss in aggregating item-level rewards — none of which implies that the list is worth further exploration. More importantly, the discrepancy can only be computed for trajectories that have already been sampled, so using it to identify under-explored regions is circular: sampling must come first, and the signal follows.

We thank the reviewer for this valuable comment. We provide the following clarifications.

1. The rationale for using discrepancy follows from its meaning in the Trajectory Balance objective. In GFlowNets, which we use to model the flow distribution, the forward trajectory flow represents the unnormalized probability mass assigned to a trajectory by the Student's generative policy, while the backward trajectory flow is determined by the terminal reward and the backward policy. The difference between the logarithms of the backward and forward flows constitutes the discrepancy. A large positive discrepancy therefore indicates that, under the current reward and normalization estimates, the forward flow assigned by the Student is lower than the backward target flow, leaving a large mismatch in the fitted trajectory flow. We use this mismatch as a signal for adjusting the training distribution, so that the Teacher focuses more on trajectories that the current model has not sufficiently fitted and that also have reward value. In summary, discrepancy is a justified signal for adjusting the training distribution because it reflects a fitting mismatch defined through GFlowNet trajectory flows.

2. The Teacher uses feedback from evaluated trajectories to learn a generative policy that can generate new lists. It generates lists according to its own generation probabilities, so subsequent Teacher sampling can produce lists that have never appeared in the buffer. The Teacher selects items one at a time based on the user state and the generated list prefix; its role is not restricted to repeatedly drawing existing trajectories. In summary, the Teacher uses discrepancy to continually adjust the exploration direction based on available feedback and to provide a signal for subsequent policy updates.

AuthorsOpenReview ↗

Response to Reviewer wV77 (2/7)

W2. According to Eq. 12, the Teacher reward contains R(sK)R(s_K) raised to the power alpha. However, the dataset and reward description in the appendix indicate that the item-level reward weights for KuaiRand include a negative value (the hate signal is assigned a negative weight), so the list-level reward can be negative; at the same time, the hyperparameter sensitivity grid contains several non-integer values of alpha. For a negative base and a non-integer exponent, the power is not defined over the reals. The paper does not explain whether the reward is shifted, clipped, or otherwise made non-negative.

Thank you. In our implementation, raw rewards are used for evaluation, while rewards transformed to be positive are used to construct the training objectives. We explain the transformation and its specific use in Eq. (12) below.

The training reward is kept strictly positive through a fixed shift. Specifically, the raw list reward RrawR_{\mathrm{raw}} is the average over the list of the weighted sums of each item's multi-behavior feedback. Its theoretical ranges on MovieLens and KuaiRand are [0,3] and [-1,6], respectively. Before constructing the training objectives, we apply the following transformation to make rewards positive:

R~=Rraw−Rmin⁡+0.001,\widetilde R = R_{\mathrm{raw}} - R_{\min} + 0.001,

The lower bounds are Rmin⁡=0R_{\min}=0 for MovieLens and Rmin⁡=−1R_{\min}=-1 for KuaiRand. The resulting training reward ranges are therefore [0.001,3.001] and [0.001,7.001], respectively, and are always strictly positive.

The Student's log-reward term uses log⁡R~\log\widetilde R, and the Teacher reward in Eq. (12) multiplies the weighted residual term by R~α\widetilde R^\alpha. Thus, the noninteger power is applied to the strictly positive training reward R~\widetilde R, rather than to the potentially negative raw reward. Consequently, this exponentiation is well defined over the real numbers for the positive real values of α\alpha used in our work. During evaluation, both Avg. R and Max. R are computed from the raw reward RrawR_{\mathrm{raw}}, without the training transformation. This treatment ensures that the training objectives are mathematically well defined while preserving the original meaning of the evaluation metrics.

In summary, the experimental training procedure and conclusions remain valid. Sections 2.1, 2.2, and 2.3 of the revised manuscript further clarify the reward transformation, parameter settings, and corresponding notation to distinguish the roles of the two rewards in training and evaluation.

AuthorsOpenReview ↗

Response to Reviewer wV77 (3/7)

W3. The simulated online environment provides limited support for real-world effectiveness. The online experiments rely on a pretrained neural user-response simulator rather than real user interactions. Although the authors report that the rankings induced by the simulator largely agree with offline rankings, this agreement may partly stem from the simulator itself being trained on offline data (the authors also acknowledge that it may converge to behavior resembling the offline distribution), and it is therefore insufficient to establish the exploration benefit in a real environment. In addition, the list reward is obtained as a weighted average of item-level feedback; the paper should further clarify whether the model can capture the combinatorial dependencies within a list, and whether the simulator reflects such dependencies.

We thank the reviewer for this suggestion. We use simulated online evaluation to address a specific question: when different policies generate recommendation lists absent from the historical logs, can Teacher-guided exploration improve recommendation outcomes under the same feedback mechanism and interaction budget? We elaborate on the following two points.

1. Recent studies also compare recommendation policies using a common user response simulator within their respective experimental environments. UOEP, proposed by Zhang et al. (SIGIR 2024), trains user response models separately on KuaiRand, MovieLens-1M, and RL4RS, samples click feedback for each item in each recommendation list, and then averages the item rewards. It compares policies such as UOEP, HAC, and TD3 within each simulated environment. [1] Lu et al. (AAAI 2026) use a pretrained DeepFM model in KuaiSim to generate feedback and compare LHRL with 11 baselines. [2] HRPO, proposed by Guo et al. (KDD 2026), also compares generative recommendation and reinforcement learning methods under a common KuaiSim session evaluation protocol, reporting average reward, cumulative reward, and item coverage. [3]

2. For a fair comparison, we evaluate the main baseline GFN4Rec and TESCO in the same KuaiSim environment and include all Teacher queries in the interaction budget. The original GFN4Rec paper uses a multi-behavior user response simulator trained on real logs to compare against CF, PRM, and ListCVAE on MovieLens-1M and KuaiRand-1K. [4] Following this simulated online evaluation approach, we use KuaiSim as the interaction environment and hold the user response model, candidate set settings, list length, and reward definition fixed across methods on each dataset. KuaiSim itself provides simulated environments and corresponding evaluation protocols for tasks including listwise recommendation and session recommendation. [5] When comparing TESCO with the Student-only GFN4Rec-TB, each submitted list counts as one feedback query, and the two methods use the same total number of queries. This tests whether having the Teacher propose a portion of the training lists yields benefits with the same number of feedback opportunities.

Overall, the simulated online experiments provide controlled comparative evidence that TESCO improves recommendation rewards under the same feedback budget. This conclusion comes from direct comparisons of different policies under a common feedback mechanism, while the consistency between offline and simulated online rankings serves as a supplementary observation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Kind Reminder: Only A Few Hours Left for Discussion

Dear reviewers,

We hope that everything goes well for you! During the rebuttal phase, we have done our utmost to respond to you in great detail and thoroughly revise the manuscript. We sincerely hope that our persistent efforts can satisfy you.

Although there are only a few hours left before the end of the discussion period, we kindly request a timely update on your evaluation of our work. Your feedback would mean a great deal to us, and we would deeply appreciate your consideration.

Thank you once again for your valuable time and continued support!

Best regards,

Authors of submission 17652

AuthorsOpenReview ↗

Final Follow-Up as Discussion Period Ends

Dear reviewers,

We hope this message finds you well. As the discussion period is about to conclude, we would like to take this opportunity to sincerely thank you once again for your valuable time and thoughtful feedback on our work.

During the rebuttal phase, we have worked diligently to address your comments and thoroughly revise the manuscript, aiming to meet your expectations and clarify any concerns. Your insights have been invaluable in improving the quality and clarity of our work, and we deeply appreciate the effort you have dedicated to reviewing our manuscript.

While we understand the constraints of time, we sincerely hope you might consider providing a final update to your evaluation. Your feedback and acknowledgment of our revisions would be greatly encouraging to us as we continue our efforts to advance this work.

Thank you again for your support and for the opportunity to learn and improve through this process. We wish you all the best in your future endeavors.

Best regards,

Authors of submission 17652

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.