- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CalibCF is a training-free spectral collaborative filtering method. The central idea is to let the user/item degrees tell you the shape of the noise spectrum, then use the singular values already computed in a short spectral head to calibrate its scale. The method uses that estimated noise edge to set a cutoff and apply a continuous shrinkage rule.
The paper backs this up with theoretical results under a degree-corrected random-graph model, and experiments on five datasets, including full-catalog recommendation metrics, baseline comparisons, ablations, transfer tests, and spectral-fit analyses. From the code I reviewed, the main calibration and scoring formulas appear to match the paper.
Critical strengths
-
I found the main idea clean and useful. Separating noise-spectrum shape from scale is a nice way to make spectral calibration depend on the graph, rather than relying on a rank or cutoff that may not transfer across datasets. The method also targets a practical setting: only a truncated spectral head is available.
-
The experiments test more than just the final recommendation score. Alongside five datasets and several strong baselines, the paper studies spectral fit, ablations, head size, and transfer. That makes the evaluation feel meaningfully tied to the method’s central claims. The results look promising overall, though I would not describe every win as decisive or statistically significant.
PS. Code is available.
Critical weaknesses
-
The evaluation protocol and the supplied code do not line up cleanly, and the full results are hard to reproduce from this ZIP. The main text describes a per-user 70/10/20 split, while the code uses the existing public test split for Gowalla and Yelp and holds validation data out of the training split. The appendix also refers to public splits, so this may be a reporting inconsistency rather than a flawed experiment—but it needs to be made explicit. The ZIP also lacks the baseline, significance-testing, and appendix-analysis scripts, so I could not independently reproduce the paper’s full set of comparisons.
-
Some of the headline evidence is less conclusive than the broad claims suggest. On ML-10M, CalibCF’s NDCG@20 is essentially tied with EASE and the reported difference is not statistically significant. More broadly, the theoretical guarantee rests on graph and spectral assumptions that may not hold for very sparse, heavy-tailed recommendation graphs, and the paper offers limited evidence about how well those assumptions describe its real datasets.
Questions for authors
-
Could you clarify the exact train/validation/test protocol used for each dataset—especially Gowalla and Yelp—and reconcile the main text’s 70/10/20 description with the public splits used in the code?
-
Could you provide the scripts and exact configurations used to generate the baseline results, significance analyses, and appendix experiments? In particular, what SVD implementations and settings produced the runtime numbers?
-
How well do the degree-conditioned template and edge estimate hold up on real, very sparse or heavy-tailed graphs? Do you have diagnostics that would let a practitioner tell when the separable-template assumptions are a poor fit?
-
What is the motivation for the fixed 1.02 margin in the exclusion update, and does the calibration or recommendation quality change materially for nearby values?
-
The paper calls CalibCF “training-free,” but it still computes a truncated SVD and selects \kappa and \gamma on a validation split. Could you clarify what “training-free” means here—specifically, whether it means no gradient-based parameter learning—and help isolate how much of the gain comes from the degree-conditioned edge calibration rather than the low-rank SVD itself?
Please Respond to the Authors’ Rebuttal
Dear Reviewer FyVA,
The discussion period ends in a few hours. Could you kindly check the authors’ responses and let us know whether your concerns have been addressed?
Thank you!
Best regards, The AC