- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes RGBT for learning recommender models from noisy implicit feedback. It combines Bayes-label transition matrix (BLTM) with GMM-based reliability weighting, where the GMM uses prediction scores and user-item co-occurrence features to estimate sample reliability. Experiments on three datasets and six backbones show improved recommendation performance in many settings, and lower average transition matrix estimation error under synthetic noise. The paper also claims consistency and variance reduction guarantees.
Critical strengths
-
The idea of using reliability estimation together with transition matrix correction is reasonable. Keeping all noisy interactions in the main training objective can be useful, especially when reliable interactions are limited.
-
The empirical results are quite good in some settings, particularly LightGCN and XSimGCL. Also good to see the authors directly evaluate transition matrix estimation under controlled noise, rather than only reporting downstream recommendation metrics.
Critical weaknesses
-
I have some concerns about the theoretical analysis. The Bayes optimal classifier is not necessarily preserved under instance-dependent noise. Also, training with hard pseudo-labels cannot generally recover the clean class posterior, which makes the consistency claims questionable. The variance reduction proof also seems to assume equal conditional variances for different estimators, without sufficient justification.
-
The contribution of GMM is not very clear to me. The reliability weights are applied to pseudo-label learning, but do not directly enter the transition matrix loss. From the ablation results, removing GMM only causes small performance drops, while the improvement over original BLTM is much larger. So how much improvement actually comes from GMM calibration? The current experiments do not quite explain this.
Questions for authors
-
How can Theorem 3 recover a non-one-hot clean posterior from hard Bayes labels? What additional assumptions make the transition matrix identifiable?
-
Where exactly do GMM weights enter the transition network update? Can authors report matrix estimation error for w/o GMM under the same training setup?
-
Does the variance inequality still hold with highly concentrated GMM weights? The derivation of Eq. (30) seems not valid to me.
6 comments
Response for Submission2305 by Reviewer JtFJ
Q4: How do GMM weights affect the transition update, and what is the error without GMM?
A4: In the revised (Sec. 3.1), the loss is computed using classifier predictions adjusted by the estimated transition matrix, so it directly depends on the transition network parameters. Multiplying each distilled sample's loss by its GMM weight therefore scales its gradient contribution to the transition network update, while samples outside the distilled set keep unit weight.
Table 2 (Sec. 4.3) compares Full and w/o GMM, which sets all sample weights to one, under the same training setup.
Mean entrywise error on MovieLens, averaged over four backbones and four noise rates. Lower is better.
| Noise | w/o GMM | Full |
|---|---|---|
| Symmetric | 1.1301 | 1.1050 |
| Pairflip | 2.2393 | 2.2240 |
We measure the entrywise distance between the estimated matrix averaged over validation interactions and the known transition matrix.
Q5: Does the variance bound hold for concentrated weights, and is Eq. (30) valid?
A5: The variance inequality is not guaranteed when the GMM weights are highly concentrated. In that regime shrinks and the bound on grows; once the domination condition fails, need not dominate . Theorem 5 therefore claims strict variance reduction only under the domination condition; when it fails, strict variance reduction is not claimed, and Corollary 1 degrades as (Sec. 3.2.4, {When the condition fails}; Appendix A.5, {Conclusion}; Appendix A.6).
Eq. (30) is rewritten: the two conditional variances are no longer assumed equal, and their difference is kept explicitly and bounded as . The derivation is in the {Conditional decomposition} paragraph of Sec. 3.2.4 and in Appendix A.5, which states explicitly that the two conditional variances are not assumed equal and that their difference is bounded.
Please refer to the latest version of the PDF for all specific modifications.
Liu Y, Cheng H, Zhang K. Identifiability of label noise transition matrix[C]//International Conference on Machine Learning. PMLR, 2023: 21475-21496.
Thanks for the detailed response and for fixing the theoretical issues.
I appreciate that the authors revised the theorems carefully, restricting Theorems 1, 3, and 4 to , adding the identifiability assumptions, and correcting Eq. (30) with the explicit bound and domination condition. These changes make the math much more rigorous.
However, after these revisions, the core contribution feels noticeably narrower. The consistency guarantees now only hold on where the clean posterior is already one-hot, and strict variance reduction requires a domination condition that can easily break when GMM weights become concentrated. More importantly, the new ablation numbers in Table 2 show that GMM weighting only brings a tiny reduction in matrix estimation error over w/o GMM (e.g., 2.2393 to 2.2240 under Pairflip), and the component analysis (in response to dQxs) shows that under Pairflip noise, the selected GMM component actually has a higher label error rate (23.74% vs. 16.44%) and lowers the clean label proportion (79.97% to 77.64%). Since most of the empirical gain seems to come from direct distilled-label supervision and alternating updates rather than GMM calibration, I will keep my rating at 2.
Response for Submission2305 by Reviewer JtFJ
Thank you for the valuable and insightful comments. We will revise the submission according to the comments. Our responses are as follows.
Q1: IDN breaks Bayes preservation; hard labels can't yield soft posteriors; variance proof assumes equal conditional variances.
A1: The Bayes-optimal classifier is not claimed to be preserved in general. Assumption 1 (Sec. 3.2.1) only requires Bayes-label preservation on the reliable set Xrel, and nothing is claimed off Xrel; Theorem 1 and Theorem 3 are both stated on Xrel.
Hard pseudo-labels cannot recover a non-one-hot clean posterior. Theorem 3 (Sec. 3.2.2) claims f→η only on Xrel. The new {Scope of Theorem 3} paragraph states that off Xrel, f converges only to the one-hot distilled-label distribution π∗(x), with no claim of recovering η(x); the (0.7,0.3) case is a limitation of the information content of hard labels. Theorem 4 is restricted to Xrel for the same reason and explicitly depends on Assumptions 2--4 and the basin condition.
The two conditional variances are no longer assumed equal. Their difference Δcond is kept explicitly and bounded as ∣Δcond∣≤C(K,ρmax)(wmax2/wmin2)neff−1, see the {Conditional decomposition} paragraph of Sec. 3.2.4 and the term-by-term derivation in Appendix A.5. Theorem 5 imposes the domination condition, which ensures that κij(x) dominates ∣Δcond∣; when this condition fails, strict variance reduction is not claimed, and Corollary 1 degrades as neff−1/2, see the {When the condition fails} paragraph of Sec. 3.2.4, the {Conclusion} paragraph of Appendix A.5, and Appendix A.6.
Q2: GMM’s contribution is unclear: its weights bypass the transition loss, and ablations show only marginal gains.
A2: In the revised Lclass (Sec. 3.1), GMM weights directly scale distilled samples' contributions to the transition network update, in addition to weighting distilled label supervision of the classifier (details and Table 2 errors are provided in A4).
Compared with BLTM, both Full and w/o GMM incorporate two designs described in Sec. 3.1 and Algorithm 1: direct supervision of the classifier using distilled labels, and alternating updates of the classifier and transition network. Full versus w/o GMM measures the additional effect of reliability weighting. Table 2 (Sec. 4.3) reports a lower average transition matrix estimation error with GMM under both noise types.
Q3: How does Theorem 3 recover soft posteriors from hard labels, and what ensures identifiability?
A3: Theorem 3 does not claim to recover a non-one-hot clean posterior from hard labels. It claims f→η only on the reliable set Xrel, where η is one-hot and hard labels carry the same information as the clean posterior; off Xrel, f converges only to the one-hot distilled-label distribution π∗(x), and no recovery of η(x) is claimed, with (0.7,0.3) being a limitation of the information content of hard labels. Theorem 3 relies on Assumption 1 (Bayes-preserving on Xrel) and Assumption 2 (uniformly reliable distillation).
Identifiability of the transition matrix is supplied by Assumption 3 (local identifiability) and Assumption 4 (Lipschitz transition matrix): Assumption 3 requires the Jacobian of ϕ↦g(x;ϕ)⊤η(x) to have full column rank at ϕ∗, giving pointwise uniqueness, and Assumption 4 requires T⋆(x) to be Lipschitz in x, controlling the cross-instance approximation error. The {Identifiability} paragraph of Sec. 2 notes that η~=T⋆⊤η provides only K−1 equations against K(K−1) degrees of freedom, so identification does not follow from this relation alone; this is consistent with Liu et al. (ICML 2023), who show that instance-dependent transition matrices are generally not identifiable from a single noisy label without additional assumptions. These two assumptions supply the missing structure.