- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a feedback-driven supervision evolution framework for generative recommendation. The framework consists of two components: the Supervision Evolution Cycle (SEC), which uses LLMs to identify and construct additional supervision based on model feedback, and the Feedback-Guided Learning Cycle (FGLC), which dynamically adjusts the learning order of the constructed supervision through short training probes. The framework is instantiated in an industrial recommendation system using intent-based supervision expansion. Offline experiments and online A/B testing demonstrate improvements in retrieval performance and business metrics.
Critical strengths
-
The paper addresses a relevant problem in generative recommendation, where observed user interactions provide incomplete supervision of user preferences. Adapting supervision construction and learning based on model feedback is a reasonable and practically meaningful direction.
-
The paper provides extensive offline experiments, including controlled comparisons that examine the contributions of SEC and FGLC separately. The results consistently demonstrate performance improvements over the considered baselines.
-
The framework is evaluated on a large-scale industrial recommendation system, with online A/B testing showing improvements in ad clicks and revenue. The reported deployment further supports its practical relevance.
Critical weaknesses
-
The proposed framework largely builds on existing ideas in feedback-driven data generation and adaptive curriculum learning. The authors themselves acknowledge these connections when introducing SEC and FGLC in Sections 4.1 and 4.2, but their technical distinctions from prior approaches could be better clarified. The intent-based supervision expansion in Section 4.3 also appears conceptually related to existing LLM-based recommendation augmentation methods. Overall, the methodological novelty seems somewhat limited despite the practical value of integrating these components.
-
It seems that LLM-based diagnosis plays a central role in SEC by determining which categories should receive additional supervision. However, the paper provides limited details on how these decisions are made in practice. Although Appendix A includes the diagnosis prompt, concrete examples of the inputs and resulting decisions would be helpful. Moreover, the comparison with the low-recall baseline in Table 3 does not fully establish the benefit of using an LLM over simpler heuristic-based approaches that consider the same feedback information.
-
At each stage, FGLC performs separate, relatively short training runs for all remaining supervision batches, followed by evaluations on the full decision set to select the next batch. This process is repeated after each model update, potentially introducing substantial computational overhead. While the authors acknowledge this overhead in Appendix C.2, the experiments use matched committed-update budgets rather than matched total computational budgets, making it difficult to assess the gains relative to the additional computational cost.
-
Sections 4.1 and 4.2 introduce a general framework, whereas Section 4.3 presents its recommendation-specific instantiation through intent-based supervision expansion. Although this distinction is briefly mentioned, the transition feels somewhat abrupt, particularly given the section titles.
Questions for authors
NA