acceptodds
Withdrawn from ICLR 2027

RelDiff: A Reinforcement Learning Guided Diffusion Framework for Recommendation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models have demonstrated promising potential in recommender systems. Existing methods typically initiate the generation process by injecting random Gaussian noise, as they have no idea about the authentic user preferences. Owing to the sparse nature of recommendation data, this uninformative paradigm often destroys the original interaction structure and leads to a severe generation collapse. To address the issue, we propose RelDiff, a novel reinforcement learning guided diffusion framework for recommendation. Rather than injecting randomly sampled Gaussian noise, we employ a Preference-Conditional Policy to generate preference noise that actively steers the diffusion process to recover authentic user preferences. Besides, based on a comprehensive analysis of the training dynamics, we design a Negative-aware Refinement strategy to transform the stagnant optimization into an efficient push-pull process. Additionally, we decouple the diffusion process from the inference stage by injecting generative preference into embeddings to maintain training efficiency. Extensive experiments on five real-world datasets demonstrate that RelDiff significantly outperforms state-of-the-art methods. Our work offers an effective solution for enhancing recommendation performance and suggests a novel paradigm for applying reinforcement learning in diffusion frameworks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Collaborative Filtering, Diffusion Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AFGtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the critical issue that existing diffusion-based recommendation models adopt user-agnostic fixed Gaussian noise, which easily leads to generation collapse during training, and proposes the RelDiff framework. The framework leverages the Actor-Critic network from reinforcement learning to generate user-conditioned preference-aware noise; it designs a NAR strategy, which constructs a push-pull loss based on a reference model to mitigate generation collapse. It also introduces a contrastive injection mechanism that integrates diffusion generation information into item embeddings, eliminating the need for iterative denoising during the inference stage. Extensive experiments are conducted on five real-world datasets, where the proposed method outperforms state-of-the-art baselines across graph-based models, contrastive learning methods and existing diffusion recommendation approaches, achieving better ranking metrics. The paper also performs comprehensive experiments including module ablation, convergence analysis, efficiency evaluation, signal-to-noise ratio testing and hyperparameter sensitivity analysis. Nevertheless, the work still has limitations: the code is not publicly available, some theoretical assumptions have inherent constraints, and statistical significance tests are not conducted.

Critical strengths
  • Existing diffusion recommendation models uniformly adopt user-agnostic standard Gaussian noise, which easily damages the sparse structure of user-item interactions. This paper utilizes Actor-Critic reinforcement learning to learn a customized noise distribution for different users based on reconstruction rewards, breaking the limitation of fixed noise.

  • NAR does not directly discard samples with poor reconstruction performance. Instead, it leverages a reference model to construct a push-pull loss to alleviate the performance stagnation in the late training stage. Then it distills the generation information into embeddings via contrastive learning, avoiding the high time cost caused by repeated denoising in diffusion inference.

Critical weaknesses
  • The appendix explicitly withdraws the original claims about guaranteed convergence acceleration and stable gradient updates. Counter‑examples show that the repulsion term in NAR may produce incorrect gradients even for genuine positive user‑item interactions. Since NAR is one of the two core innovations of RelDiff, the lack of valid theoretical guarantees weakens the interpretability and reliability of the whole method.

  • Multiple critical implementation details, reference‑checkpoint update rule, gradient‑stopping settings, exact hyper‑parameter selection workflow remain unspecified. Moreover, statistical significance tests are missing for all performance improvements.

Questions for authors
  • For the NAR strategy, when the reconstruction confidence rr is calculated from batch‑wise statistics, are gradients detached from these batch statistics during back‑propagation? Also, what exact update rule is adopted for selecting and refreshing the reference checkpoint fθ−f_{\theta}^{-}? Without these implementation details, it is hard to judge how NAR behaves in practical training.

  • The reported performance improvements lack statistical significance tests. Could you provide paired statistical test results across multiple repeated runs to confirm that the observed gains of RelDiff over baselines stem from your proposed modules instead of random fluctuation or hyper‑parameter luck?

  • The PNG module uses one‑step contextual‑bandit‑style actor‑critic optimization relying purely on immediate reconstruction reward. Is there any observed risk that the policy only optimizes reconstruction loss but fails to improve downstream ranking metrics? Have you tracked the correlation between reconstruction reward and final recommendation ranking performance in your experiments?

  • In the SNR analysis, higher SNR is obtained with learned preference noise. Does this elevated SNR truly encode user‑specific preference information, or is it merely an outcome of reducing overall noise magnitude? Could you supply control experiments such as norm‑matched random noise to rule out the magnitude‑only effect?

Reviewer_DvXUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RelDiff, a reinforcement-learning-guided diffusion framework for collaborative filtering. It targets two problems: user-independent Gaussian noise that disrupts sparse interactions, and a training bottleneck the authors call generation collapse. RelDiff replaces standard noise with a user-conditioned policy trained by reconstruction rewards. It injects the reconstruction signal into item embeddings through contrastive learning, so inference needs no iterative denoising. It also re-weights training around a reference checkpoint using confidence scores. In experiments on five datasets, RelDiff ranks items more accurately than all 12 baselines.

Critical strengths

1.The experimental coverage is broad. The paper evaluates on five datasets against 12 baselines from three families (graph, contrastive, and diffusion). It also includes ablations (Fig. 3), convergence curves (Fig. 4), a critic–reward comparison (Fig. 5), an efficiency analysis (Fig. 6), hyperparameter sensitivity (Fig. 7), and an SNR analysis (Fig. 8).

2.The related work is reasonably detailed. The introduction reviews many diffusion-based recommenders, including DiffRec, DiffuRec, DreamRec, DDRM, and GiffCF. Appendix D gives background on diffusion models and reinforcement learning.

3.The figures are clear and well presented. The ablation figure (Fig. 3) is clearly drawn and easy to read. The bar charts use a consistent color scheme across the five datasets, so the comparison between variants is easy to follow.

Critical weaknesses

W1. The paper changes the noise prior of diffusion, but does not redo the derivation. Standard DDPM assumes that the noise prior is N(0, I). All of its formulas are derived under this assumption, including the closed-form forward step, the reverse (denoising) step, and the training objective. In Eq. 1, the paper replaces this prior with N(μϕ(eu), diag(σϕ(eu)2)), whose mean and variance both depend on the user. The paper then directly substitutes this w into the standard closed form in Eq. 5, zt = √ᾱt ei + √(1 − ᾱt) w, and keeps the standard objective in Eq. 7. This direct substitution is not valid, as it breaks the assumption under which these DDPM formulas were derived. The paper does not give a separate, correct derivation for the new prior. Therefore, the derivation of the diffusion process with PNG is incorrect, and the claimed diffusion framework does not hold.

W2. The core motivation is not supported by evidence. The abstract claims that "owing to the sparse nature of recommendation data, such perturbations can disturb interaction structure." No experiment measures this disturbance. Section 2 only shows DiffRec training curves, and the authors admit that the SNR analysis in Appendix B.2 does not establish semantic preservation. In addition, Eq. 5 adds noise to the dense item embedding ei, not to the sparse interaction matrix A. Moreover, the paper does not conduct controlled experiments with strictly set sparsity levels to verify how the noise affects the interaction structure. Therefore, the claimed link between data sparsity and the disturbance of interaction structure remains unsupported.

W3. The experimental results are inconsistent and cannot be verified. In Fig. 1, DiffRec reaches about 0.045 Recall@20 on Electronics. Table 2 reports 0.0236 for the same model, dataset, and metric. With the Fig. 1 value, the reported gain on Electronics would largely disappear. The curves in Fig. 4 also do not reach the values in Table 2 (e.g., Office Recall@20 and Electronics NDCG@20). Moreover, the number of runs is unspecified, and no variability estimates are reported. The authors state that source code, configurations, logs, seeds, and data splits "were not supplied" and are "not independently verified" (§5.1.4, Appendix C). Therefore, the reported improvements cannot be trusted.

Questions for authors

1.Derivation and motivation. Please derive the forward process, the reverse step, and the training objective under the noise w ~ πϕ(·|eu). Please also explain how fθ(ei,t) handles the user-dependent shift. Finally, please provide a direct measurement showing that Gaussian noise disturbs the interaction structure and that PNG reduces this disturbance.

2.Result consistency. Please explain the gap between Fig. 1, Fig. 4, and Table 2. Please report the mean and standard deviation over multiple seeds, state whether model selection used validation data, and release the source code and configurations.

3.RL design and ablations. §4.1.2 states that the objective is a one-step contextual bandit with no Bellman bootstrap. The reward is also differentiable through the reparameterized w. Please explain why the method is framed as reinforcement learning and why two critics are needed. Please also provide BPR-only and BPR + contrastive backbone ablations, and explain why λ1 is best at the smallest tested value in Fig. 7.

4.Baselines (suggestion). The diffusion baselines stop at 2024. Please compare with more recent generative collaborative filtering methods, such as Collaborative Diffusion Models for Recommendation (Chen et al., SIGIR 2025), Flow Matching for Collaborative Filtering (FlowCF; Liu et al., KDD 2025), and Towards A Tri-View Diffusion Framework for Recommendation (TV-Diff; Chen et al., KDD 2026). Please also check the DDRM setup, whose results are about one order of magnitude lower than all other methods.

References

[1] Mengru Chen, Lianghao Xia, Yong Xu, and Ronghua Luo. Collaborative Diffusion Models for Recommendation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 1975–1984, 2025. DOI: 10.1145/3726302.3729929.

[2] Chengkai Liu, Yangtian Zhang, Jianling Wang, Rex Ying, and James Caverlee. Flow Matching for Collaborative Filtering. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2025. DOI: 10.1145/3711896.3736967.

[3] Ximing Chen, Pui Ieng Lei, Yijun Sheng, Yanyan Liu, and Zhiguo Gong. Towards A Tri-View Diffusion Framework for Recommendation. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 116–127, 2026. DOI: 10.1145/3770854.3780157.

Reviewer_gi4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RelDiff studies diffusion-assisted collaborative filtering using a user-conditioned Gaussian perturbation policy, contrastive injection of reconstructed item representations, and Negative-aware Refinement (NAR). The policy is trained using immediate reconstruction rewards, twin critics, and entropy regularization; the manuscript appropriately identifies this as a one-step contextual-bandit-style objective. NAR weights reference-anchored reconstruction terms according to relative reconstruction confidence. At inference, graph embeddings and dot-product scores replace iterative denoising. Experiments on five datasets report improvements over graph, contrastive, and diffusion baselines, together with PNG/NAR ablations and training diagnostics.

Critical strengths

1. Consistent and potentially meaningful reported ranking improvements. Table 2 reports the highest Recall and NDCG at both cutoffs on all five datasets. Relative Recall@20 gains over the strongest displayed baseline range from 5.21% on Office to 49.41% on ML-1M. These point estimates motivate further investigation of the combined training procedure, although their variability is not reported.

2. A practical inference design with initial component evidence. The contrastive injection mechanism permits embedding-based ranking without iterative denoising at inference. Figure 3 also compares removal of PNG, NAR, and both, providing initial evidence that these components help within the tested configuration. Avoiding inference denoising is an architectural advantage; a measured latency advantage remains to be established.

Critical weaknesses

1. The NAR analysis does not correspond to the objective defined in the main text. Let f denote the current prediction, a the reference output, and Delta = f - a. Equations (9)-(10) define f_pos = (1-beta)f + betaa = a + (1-beta)Delta and f_neg = (1+beta)f - betaa = a + (1+beta)Delta. However, Appendix A, Equation (16), substitutes a + betaDelta and a - betaDelta into Equation (11). These are different expressions, so the gradients, optimum, and counterexample in the appendix do not directly characterize the main-text NAR objective.

2. The evidence does not adequately identify the source or reliability of the reported gains. Figure 3 compares learned perturbations with standard Gaussian noise, but does not separate user conditioning from learned perturbation magnitude/covariance or establish the necessity of critic-based optimization. Higher SNR in Figure 8 can result from smaller perturbations alone. Norm-matched Gaussian, user-independent learnable noise, shuffled-user conditioning, and direct reparameterized optimization of the reconstruction reward are consequential missing controls. A matched graph+BPR baseline without the diffusion branch and a separate contrastive-injection ablation would further clarify the contribution. In addition, the training plateaus in Figure 1 do not distinguish the proposed 'generation collapse' from ordinary convergence or reconstruction/ranking objective mismatch. The manuscript explicitly leaves data splits, final configurations, seeds, repeat counts, and checkpoint-selection rules unspecified (Section 5.1.4 and Appendix C); therefore the large point-estimate gains cannot yet establish reproducible superiority under a verified common protocol. Figure 6 also contains Recall values differing from Table 2 and measures per-epoch time rather than matched-quality training time or online latency. Together, these issues leave a promising training recipe without sufficiently reliable evidence for its claimed mechanism or comparative advantage.

Questions for authors
  1. Which NAR expressions are implemented: Equations (9)-(10), or the combinations in Appendix Equation (16)? Please provide the matching corrected derivation and specify beta, whether confidence/batch statistics and target embeddings are detached, and how the reference checkpoint is selected and updated. If the implementation differs from the main text, which reported experiments use it?

  2. Under matched training budgets, how does PNG compare with norm-matched Gaussian noise, user-independent learnable Gaussian noise, shuffled-user conditioning, and direct backpropagation through the reparameterized reconstruction reward without critics? These comparisons would materially affect my assessment of user conditioning and actor-critic necessity.

  3. What are the results for the same graph encoder and BPR objective without the diffusion branch, and for removal of contrastive injection alone? What evidence distinguishes 'generation collapse' from an ordinary optimization plateau or reconstruction/ranking mismatch?

  4. Please provide dataset versions, filtering/splits, candidate exclusions, validation/checkpoint-selection rules, final per-dataset settings, and repeated-run uncertainty under a shared baseline protocol. Please also reconcile Figure 6 with Table 2 and the 0-1000-step SNR plot with the reported 100-500-step search range. These are requests to clarify and substantiate existing results, rather than suggestions for expanding the application scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.