- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper evaluates existing PFR methods using a common implementation and evaluation protocol. It systematically studies the behavior of existing methods under various conditions.
Critical strengths
- Benchmarking existing methods helps to better gauge the overall progress in the PFT from different perspectives.
Critical weaknesses
- Limited Novelty: The paper uses existing metrics and datasets for the evaluation. Also, the paper doesn't provide any concrete directions based on its findings. The conclusions drawn in this paper are well known in the FL literature.
- In Section 2.2, existing PFR methods are categorized along three aspects. However, these categories are ignored during the results discussion.
Questions for authors
N/A
3 comments
I thank the authors for the clarification. I agree with the authors that one of the roles of a benchmarking paper is to provide a reliable and comparable evaluation basis, while the research contribution lies in the new questions addressed and empirical findings obtained through systematic analysis.
-
However, several of the evaluation questions, such as personalization across heterogeneous users, dependence on local interaction availability, and so on, have already been explored in prior work, although they may not have been evaluated across all the methods included in this benchmark. Therefore, I believe the paper should more clearly state the novelty of the research questions it raised and explain how the resulting empirical findings provide new insights beyond those established in prior studies.
-
Section 4 presents cross-experiment insights regarding local data quantity and reliability, as well as collaborative benefits and propagation risk. However, it remains unclear to what extent these findings advance our understanding of PFR methods beyond existing studies.
-
As in Section 2.2, the paper summarizes three complementary aspects of personalization design, using representative methods to illustrate different implementations. Section 3 organizes the analysis around evaluation questions while considering relevant methodological differences when interpreting the results. I believe the authors should more explicitly and systematically explain how the different implementations of the three complementary aspects of personalization design relate to the empirical findings and support the conclusions drawn.
Therefore, I would like to keep my score unchanged.
Follow-up Clarification on the Contributions of PFR-Bench
Dear Reviewer 2q6u,
Thank you for your continued feedback. Our goal in PFR-Bench is to provide a comprehensive and fair evaluation of representative personalized federated recommendation methods under a unified protocol. Beyond overall recommendation performance, we systematically examine how different methods behave across user groups, local data conditions, and federated collaboration, aiming to better understand their strengths, limitations, and trade-offs.
We agree that several evaluation questions have been explored in prior work. For example, KE-FedRS (WWW 2026) focuses on alleviating interaction sparsity, while PipAttack (WSDM 2022) investigates targeted poisoning attacks. These studies provide valuable insights into specific challenges, whereas our multidimensional evaluation further reveals how different capabilities and limitations are related. Specifically, our cross-experiment analysis reveals that strong performance with limited local interactions does not necessarily imply robustness to interaction noise, and methods benefiting from collaboration can exhibit markedly different risks of malicious information propagation. Although existing benchmarks such as pFL-Bench (NeurIPS 2022) also provide multidimensional evaluations, they primarily focus on general personalized federated learning algorithms rather than systematically examining PFR methods and their recommendation-specific behaviors.
We hope these comparisons help clarify how PFR-Bench builds upon prior research: by jointly examining established evaluation questions, it provides new empirical insights into the relationships and trade-offs among PFR methods. We appreciate your consideration of this perspective.
Response to Reviewer 2q6u
Critical Weaknesses 1: Limited Novelty: The paper uses existing metrics and datasets for the evaluation. Also, the paper doesn't provide any concrete directions based on its findings. The conclusions drawn in this paper are well known in the FL literature.
Response:
We would like to clarify an important point regarding the contribution of PFR-Bench. We believe that using established datasets and metrics should not, in itself, be considered a limitation of a benchmark's novelty. Their role is to provide a reliable and comparable evaluation basis, while the research contribution lies in the new questions addressed and empirical findings obtained through systematic analysis. In PFR-Bench, we go beyond comparing overall recommendation accuracy by examining how different personalization approaches behave as the amount and reliability of local interaction data change, and how their use of collaborative knowledge relates to performance and robustness. These analyses allow us to identify differences in method behavior and trade-offs that are difficult to establish from individual studies conducted under different settings. Together, these findings provide a more systematic understanding of the strengths, limitations, and trade-offs of different PFR approaches under varying conditions.
Regarding the reviewer's concern that our conclusions are already well known in FL, we believe it is important to distinguish general insights from the systematic empirical evidence supporting them. While some high-level observations may have been discussed in the broader FL literature, how they manifest across different PFR methods and experimental conditions remains insufficiently understood. PFR-Bench helps fill this gap by providing comparable empirical evidence on where these observations hold, how they differ across methods, and under what conditions their limitations emerge. This empirical understanding, rather than the discovery of entirely new FL principles, is a central contribution of our benchmark.
Critical Weaknesses 2: In Section 2.2, existing PFR methods are categorized along three aspects. However, these categories are ignored during the results discussion.
Response:
We appreciate the reviewer's comment and would like to clarify the relationship between Section 2.2 and the experimental analysis. Section 2.2 summarizes three complementary aspects of personalization design, using representative methods to illustrate different implementations rather than grouping methods into distinct categories. Since each method involves multiple design choices, its experimental behavior cannot be simply attributed to any single aspect. Accordingly, Section 3 organizes the analysis around evaluation questions while considering relevant methodological differences when interpreting the results, as exemplified in Section 3.1. This keeps our conclusions grounded in the observed results, rather than assumptions based on method design alone.