acceptodds
Under review as a conference paper at ICLR 2027

PFR-Bench: Benchmarking Personalization in Federated Recommender Systems

Abstract

Recent studies on personalized federated recommendation have explored diverse personalization designs, but inconsistent implementations and evaluation protocols make existing results difficult to compare. More importantly, accuracy-centric evaluation provides limited insight into how these designs behave across users and under federated conditions. We therefore introduce PFR-Bench, a unified benchmark that standardizes six representative methods across ten datasets under a common experimental protocol. Beyond aggregate recommendation accuracy, PFR-Bench examines how personalization varies across heterogeneous users, depends on local interaction evidence, and behaves when client information becomes unreliable. Our evaluation reveals substantial differences that are obscured by overall performance alone. In particular, methods that use limited local evidence effectively are not necessarily those that remain stable when that evidence becomes unreliable, suggesting that local evidence should be modeled jointly through its quantity and reliability. We further find that stronger use of cross-client knowledge can compensate for insufficient local evidence while also increasing exposure to manipulated information from other clients, highlighting the need to balance collaborative benefit against propagation risk. PFR-Bench provides a reproducible basis for further evaluation and development of personalization mechanisms in federated recommendation. Code and benchmark resources are available at https://anonymous.4open.science/r/PFR-BENCH-AD63/.

Primary area:
datasets and benchmarks
Keywords:
Federated Recommendation, Personalization Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2q6uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper evaluates existing PFR methods using a common implementation and evaluation protocol. It systematically studies the behavior of existing methods under various conditions.

Critical strengths
  1. Benchmarking existing methods helps to better gauge the overall progress in the PFT from different perspectives.
Critical weaknesses
  1. Limited Novelty: The paper uses existing metrics and datasets for the evaluation. Also, the paper doesn't provide any concrete directions based on its findings. The conclusions drawn in this paper are well known in the FL literature.
  2. In Section 2.2, existing PFR methods are categorized along three aspects. However, these categories are ignored during the results discussion.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2q6u

Critical Weaknesses 1: Limited Novelty: The paper uses existing metrics and datasets for the evaluation. Also, the paper doesn't provide any concrete directions based on its findings. The conclusions drawn in this paper are well known in the FL literature.

Response:

We would like to clarify an important point regarding the contribution of PFR-Bench. We believe that using established datasets and metrics should not, in itself, be considered a limitation of a benchmark's novelty. Their role is to provide a reliable and comparable evaluation basis, while the research contribution lies in the new questions addressed and empirical findings obtained through systematic analysis. In PFR-Bench, we go beyond comparing overall recommendation accuracy by examining how different personalization approaches behave as the amount and reliability of local interaction data change, and how their use of collaborative knowledge relates to performance and robustness. These analyses allow us to identify differences in method behavior and trade-offs that are difficult to establish from individual studies conducted under different settings. Together, these findings provide a more systematic understanding of the strengths, limitations, and trade-offs of different PFR approaches under varying conditions.

Regarding the reviewer's concern that our conclusions are already well known in FL, we believe it is important to distinguish general insights from the systematic empirical evidence supporting them. While some high-level observations may have been discussed in the broader FL literature, how they manifest across different PFR methods and experimental conditions remains insufficiently understood. PFR-Bench helps fill this gap by providing comparable empirical evidence on where these observations hold, how they differ across methods, and under what conditions their limitations emerge. This empirical understanding, rather than the discovery of entirely new FL principles, is a central contribution of our benchmark.

Critical Weaknesses 2: In Section 2.2, existing PFR methods are categorized along three aspects. However, these categories are ignored during the results discussion.

Response:

We appreciate the reviewer's comment and would like to clarify the relationship between Section 2.2 and the experimental analysis. Section 2.2 summarizes three complementary aspects of personalization design, using representative methods to illustrate different implementations rather than grouping methods into distinct categories. Since each method involves multiple design choices, its experimental behavior cannot be simply attributed to any single aspect. Accordingly, Section 3 organizes the analysis around evaluation questions while considering relevant methodological differences when interpreting the results, as exemplified in Section 3.1. This keeps our conclusions grounded in the observed results, rather than assumptions based on method design alone.

I thank the authors for the clarification. I agree with the authors that one of the roles of a benchmarking paper is to provide a reliable and comparable evaluation basis, while the research contribution lies in the new questions addressed and empirical findings obtained through systematic analysis.

  • However, several of the evaluation questions, such as personalization across heterogeneous users, dependence on local interaction availability, and so on, have already been explored in prior work, although they may not have been evaluated across all the methods included in this benchmark. Therefore, I believe the paper should more clearly state the novelty of the research questions it raised and explain how the resulting empirical findings provide new insights beyond those established in prior studies.

  • Section 4 presents cross-experiment insights regarding local data quantity and reliability, as well as collaborative benefits and propagation risk. However, it remains unclear to what extent these findings advance our understanding of PFR methods beyond existing studies.

  • As in Section 2.2, the paper summarizes three complementary aspects of personalization design, using representative methods to illustrate different implementations. Section 3 organizes the analysis around evaluation questions while considering relevant methodological differences when interpreting the results. I believe the authors should more explicitly and systematically explain how the different implementations of the three complementary aspects of personalization design relate to the empirical findings and support the conclusions drawn.

Therefore, I would like to keep my score unchanged.

AuthorsOpenReview ↗

Follow-up Clarification on the Contributions of PFR-Bench

Dear Reviewer 2q6u,

Thank you for your continued feedback. Our goal in PFR-Bench is to provide a comprehensive and fair evaluation of representative personalized federated recommendation methods under a unified protocol. Beyond overall recommendation performance, we systematically examine how different methods behave across user groups, local data conditions, and federated collaboration, aiming to better understand their strengths, limitations, and trade-offs.

We agree that several evaluation questions have been explored in prior work. For example, KE-FedRS (WWW 2026) focuses on alleviating interaction sparsity, while PipAttack (WSDM 2022) investigates targeted poisoning attacks. These studies provide valuable insights into specific challenges, whereas our multidimensional evaluation further reveals how different capabilities and limitations are related. Specifically, our cross-experiment analysis reveals that strong performance with limited local interactions does not necessarily imply robustness to interaction noise, and methods benefiting from collaboration can exhibit markedly different risks of malicious information propagation. Although existing benchmarks such as pFL-Bench (NeurIPS 2022) also provide multidimensional evaluations, they primarily focus on general personalized federated learning algorithms rather than systematically examining PFR methods and their recommendation-specific behaviors.

We hope these comparisons help clarify how PFR-Bench builds upon prior research: by jointly examining established evaluation questions, it provides new empirical insights into the relationships and trade-offs among PFR methods. We appreciate your consideration of this perspective.

Reviewer_2ubxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents PFR-Bench, a benchmark for personalized federated recommendation. It standardizes six methods across ten datasets and evaluates them under a common experimental protocol. Based on these experiments, the paper highlights two main observations: (1) the usefulness of local evidence depends jointly on its quantity and reliability, and (2) stronger use of cross-client knowledge can compensate for insufficient local evidence but may also increase the propagation risk of manipulated information. The paper provide a reproducible empirical basis for presonalized federated recommendation.

Critical strengths
  1. The paper studies an important problem in personalized federated recommendation. Existing methods are often evaluated under different implementations and protocols, making direct comparison difficult. PFR-Bench provides a common empirical basis across six representative methods and ten datasets, which is useful for understanding how different personalization mechanisms behave under shared conditions.
  2. The evaluation goes meaningfully beyond aggregate recommendation accuracy. The paper examines user heterogeneity, local-data availability, interaction corruption, efficiency, and poisoning propagation.
Critical weaknesses
  1. The evaluation protocol raises concerns about data leakage and the necessity of federated collaboration. Although the authors have unified several PFR frameworks, I believe they have overlooked a fundamental issue. As a researcher working on PFR, I am aware of a long-standing concern regarding negative sampling, discussed in PFedRec Issue #3 (https://github.com/Zhangcx19/IJCAI-23-PFedRec/issues/3). The discussion reports that disabling federated learning can achieve substantially better performance, even reaching HR@10 = 1.0, partly due to the use of future positive interactions in training negative sampling. Unfortunately, the released PFR-Bench implementation (data/ncf_dataloader.py) also constructs its training negative pool using training, validation, and test interactions, introducing the same future-information leakage concern. More importantly, the authors do not report any local-only results. If purely local training already outperforms federated methods, what is the value of comparing different federated personalization mechanisms? Local-only training is itself a fully personalized approach. Without establishing that federated learning provides benefits over local training, many of the paper's conclusions about personalization and collaborative knowledge difficult to justify.
  2. The benchmark overlooks the flexibility of personalization mechanisms. Many PFR methods can adjust the balance between local and global embeddings through their hyperparameters. For example, FedCA uses an interpolation coefficient to control the contribution of local and aggregated representations, while FedRAP uses regularization weights to influence the learning of personalized and shared item representations. However, the benchmark appears to treat these methods as fixed configurations rather than exploring how their behavior changes with these parameters. In fact, simply adjusting these parameters may already produce many of the personalization behaviors discussed in the paper. For instance, a method's dependence on local interactions may change substantially under different hyperparameter configurations. Without such analysis, it is difficult to determine whether the reported findings reflect fundamental differences between personalization mechanisms or merely the selected hyperparameter settings.
Questions for authors
  1. Could the authors provide results and analysis under a normal negative sampling protocol that avoids future information leakage and allows the model benefits from federated learning?
  2. Could the authors report the performance of purely local training without federated learning?
  3. Could the authors provide results and analysis under different hyperparameter settings, particularly extreme configurations of parameters that control the degree of personalization versus global collaboration?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 2ubx (Part 1)

Thank you for your thoughtful and constructive feedback! We appreciate your attention to these important evaluation issues, which has encouraged us to revisit our experimental protocols and conduct additional analyses to strengthen PFR-Bench. We value your perspective and would welcome any further suggestions as we continue refining the benchmark.

Due to limited computational resources and time, we were only able to complete the additional experiments on 2 datasets (FilmTrust and LastFM-2K). We present the results below in response to your two related concerns.

Weakness 1 & Q1–Q2: Negative Sampling Protocol and Benefits of Federated Collaboration

Response:

We thank the reviewer for drawing attention to an important methodological choice that has been approached differently in recommendation research. One common practice is to exclude all observed positive interactions from negative sampling, as reflected in RecBole's RepeatableSampler [1] and Cao et al. (NAACL 2024) [2]. This aims to prevent false negatives: treating held-out positive interactions as negatives introduces conflicting supervision, potentially distorting learned user preferences [3]. Alternatively, studies such as LightGCN [4] and RecBole's standard Sampler exclude only training positives [1]. This maintains stricter data isolation, ensuring that validation and test interactions do not influence training, even indirectly through negative sample selection. Both practices are supported by prior work and reflect reasonable methodological choices. This also suggests that the effect of protocol selection on experimental findings deserves attention.

In designing PFR-Bench, we adopted the former protocol, which is commonly used in existing open-source federated recommendation implementations, and applied it consistently across all methods to provide a fair basis for comparison. At the same time, we agree with the reviewer that examining whether the findings hold across different sampling protocols would make the benchmark more comprehensive and reliable. We therefore extended our evaluation to the training-only negative sampling protocol on two representative datasets, keeping other experimental settings unchanged. This allows us to examine the effects of negative sampling choices on recommendation performance and the consistency of our personalization findings. The additional results are presented below.

MethodFilmTrust HR@10FilmTrust NDCG@10LastFM-2K HR@10LastFM-2K NDCG@10
PFedRec0.65040.44250.55750.5400
GPFedRec0.60700.41650.48250.4579
FedDAE0.40870.23890.42690.3437
FedCA0.57490.37290.51690.5150
FedRAP0.04790.02020.51810.4846
FedEM0.26120.16630.51690.5018

Results show that, compared with the original protocol, training-only negative sampling generally yields lower recommendation performance. To further examine the benefits of federated collaboration, we compare each method with its local-only counterpart under both the original and training-only negative sampling protocols. The results are presented below. Overall, federated collaboration provides substantial improvements over local-only training, particularly on FilmTrust under both negative sampling protocols. These benefits are more consistently observed under the original protocol, while becoming less pronounced on LastFM-2K under the training-only protocol.

Negative SamplingMethodTrainingFilmTrust HR@10FilmTrust NDCG@10LastFM-2K HR@10LastFM-2K NDCG@10
OriginalPFedRecLocal0.92660.82980.79690.7238
Federated0.91620.81570.81750.7191
GPFedRecLocal0.55610.37490.55440.5107
Federated0.91320.80930.78000.6800
FedDAELocal0.09360.03960.56690.5397
Federated0.91690.81800.83250.6790
FedCALocal0.72680.59150.62310.5820
Federated0.92660.83830.68120.6240
FedRAPLocal0.41240.23860.63750.5909
Federated0.76650.51360.64980.5912
FedEMLocal0.82040.67480.65750.6008
Federated0.89150.79550.73750.6596
Training-onlyPFedRecLocal0.12430.05220.55440.5353
Federated0.65040.44250.55750.5400
GPFedRecLocal0.12500.07190.56380.5179
Federated0.60700.41650.48250.4579
FedDAELocal0.09360.03960.56690.5397
Federated0.40870.23890.42690.3437
FedCALocal0.10180.05210.55310.5315
Federated0.57490.37290.51690.5150
FedRAPLocal0.16690.07920.57130.5404
Federated0.04790.02020.51810.4846
FedEMLocal0.12350.05720.57310.5427
Federated0.26120.16630.51690.5018
AuthorsOpenReview ↗

Response to Reviewer 2ubx (Part 2)

We further repeated three personalization analyses under the training-only negative sampling protocol, covering user activity, interest mainstreamness, and recommendation popularity distributions, and the figure reuslts are summarized in Figures 1-3. The main observations remain largely consistent with those under the original protocol. Users with richer interaction histories generally achieve better performance, while the effects of interest mainstreamness follow similar patterns across datasets. Moreover, different personalization methods retain distinct popularity preferences in their recommendations. These results provide additional support for the consistency of our personalization findings under different negative sampling protocols.

We have also extended the evaluation to dependence on local interaction availability and reliability under unreliable client information. Due to limited computational resources and the long runtime of these experiments, the evaluations are still ongoing, and we will provide the results once available.

[1] Zhao, W. X., Mu, S., Hou, Y., Lin, Z., Chen, Y., Pan, X., ... & Wen, J. R. (2021, October). Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In proceedings of the 30th acm international conference on information & knowledge management (pp. 4653-4664). [2] Cao, Y., Mehta, N., Yi, X., Keshavan, R. H., Heldt, L., Hong, L., ... & Sathiamoorthy, M. (2024, June). Aligning large language models with recommendation knowledge. In Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 1051-1066). [3] Ma, H., Xie, R., Meng, L., Feng, F., Du, X., Sun, X., ... & Meng, X. (2026). Negative sampling in recommendation: A survey and future directions. ACM Transactions on Information Systems, 44(4), 1-44. [4] He, X., Deng, K., Wang, X., Li, Y., Zhang, Y., & Wang, M. (2020, July). Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval (pp. 639-648).

AuthorsOpenReview ↗

Response to Reviewer 2ubx (Part 3)

Weakness 2 & Q3: Hyperparameter Sensitivity of Personalization Mechanisms

Response:

We thank the reviewer for raising this valuable point. In PFR-Bench, we compare methods under well-tuned configurations selected through validation, providing a consistent basis for evaluating their performance and personalization behavior across different dimensions. Beyond this comparison, we agree that examining the flexibility of personalization mechanisms across different configurations can further enrich the benchmark by providing a more complete understanding of their capabilities and limitations.

Motivated by this suggestion, we conducted additional experiments on FedCA and FedRAP under different local–global balance settings. For FedCA, in addition to the original interp. = 0.9 (local-dominant), we considered interp. = 0.1 (global-dominant) and interp. = 0.5 (balanced local–global contributions). For FedRAP, given its sensitivity to the regularization coefficient, we conducted preliminary tuning and selected α = 0 (global-only) and α = 0.1 (greater local contribution), alongside the original α = 0.01. To provide a comprehensive assessment, we first revisited dependence on local interaction availability and then extended the evaluation to personalization across heterogeneous users and reliability under unreliable client information.

Overall, the results show that the main findings remain largely consistent across different hyperparameter settings, suggesting that they are not merely driven by specific parameter choices. Detailed results are presented below.

  • Dependence on Local Interaction Availability. (Figure 1) For a direct comparison, we present the additional configurations of FedCA and FedRAP alongside their original settings and the other four methods in the same figure. For FedCA, increasing the global contribution provides little benefit under sparse interactions, indicating that greater reliance on collaborative information does not necessarily compensate for limited local evidence. For FedRAP, even the global-only configuration benefits considerably from additional local interactions, suggesting that using only global representations does not eliminate the need for sufficient local interaction data. These findings suggest that local-data dependence is not simply determined by the balance between local and global representations.

  • Personalization Across Heterogeneous Users. For a clearer comparison, we organize the results to show both the effects of different hyperparameter settings within each method and the performance of these configurations relative to other methods. Across different settings, the main patterns of personalization performance remain largely consistent. Regarding user activity (Figures 2-7), high-activity users generally achieve better performance across configurations. Regarding the popularity of users' historical interests (Figures 8-13), Mixed users on FilmTrust and Mainstream users on LastFM-2K tend to perform better, consistent with the patterns observed across other methods. Hyperparameter changes mainly affect the magnitude of these performance differences. In contrast, the popularity distribution of recommended items shows varying sensitivity to parameter choices (Figure 14). Increasing the global contribution shifts FedCA's recommendations toward Head items, whereas FedRAP maintains relatively similar Head/Mid/Tail distributions across its tested settings. These results suggest that hyperparameter choices can change recommendation behavior without necessarily altering the broader patterns observed across user groups.

Reviewer_cKW7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents PFR-Bench, a novel benchmarking study on personalized federated recommendation. Particularly, the paper collects a wide and heterogenous set of 6 personalized federated recommender systems and an extensive group of 10 recommendation datasets with different characteristics. As for the selected federated recommendation techniques, the authors classify the different approaches around three methodological dimensions, namely: personalized components, collaborative knowledge construction, and local-collaborative integration. Then, it analyzes the selected approaches over diverse evaluation dimensions: 1) an overall recommendation performance analysis on two accuracy metrics; 2) a study on local data dependence by understanding the impact of different user activity levels and user preferences (i.e., in terms of popularity of interacted items) and the influence of the number of local interactions; 3) an analysis on the impact of federated collaboration, by considering the communication and training efficiency and reliability under client information that might be noisy. The main insights from the benchmark suggest that PFR approaches should jointly model the amount of available local interactions and their reliability level; moreover, cross-client knowledge exchange can lead to both benefits in terms of collaboration among users, but it may also cause harmful external influence, so a trade-off between the two dimensions should be taken into account.

Critical strengths

I think the main strengths of the paper lay in the novelty of the proposed benchmark, which represents (to the best of my knowledge) an important empirical gap in the related PFR literature. Moreover, the benchmark is supported by an extensive selection of models, spanning well-established and recent approaches implementing different strategies, as well as the considered datasets which cover a great range of statistics and (also in this case) less/more recent instances. Then, I think the conducted experiments are able to cover complementary evaluation dimensions, from overall performance analysis to more careful empirical investigations considering the local data dependence and federated collaboration, both with specific definitions. Finally, the derived findings provide insightful points to pave the way for future approaches in the PFR field.

Critical weaknesses

My main concerns are mostly on some experimental design choices and presentation of model results.

Regarding the former, I think the authors should better clarify some aspects, such as the number of negative samples they use; in Section 2.3, they say they used 99 randomly sampled negatives, but then in Appendix A, they claim the negative sampling number is set to 4. Moreover, it is not clear how they ensured the reproducibility of results: did they set a fixed random seed? In this respect, did they also try to run experiments across different random seeds to ensure the generalizability of results? I acknowledge this is not a common practice in the recommendation field in general, but from a benchmarking analysis I would expect a more empirical depth. Finally, I think the work would benefit from additional accuracy results with different top-kk cut-offs, as k=10k=10 may not provide enough evidence of the generalizability of the presented outcomes.

Regarding the latter, I understand the model categorization the authors made in Section 2.2 and in Table 1. However, I think it would have been much clearer to report it with checkmarks on shared methodological dimensions, because in the current version it is not quite easy to distinguish models depending on their similar/different strategies. Related to this, I think results should have been presented following a similar direction, outlining how different model and their strategies provide different empirical results in all evaluated dimensions. If I am not mistaken, this was not clearly stated throughout the results discussion, even if I admit Section 4 was pretty clear in stating the main take-home messages.

Questions for authors

My questions are mainly related to first concern I raised.

  1. Can the authors clarify what is the negative sampling number they choose for the experiments?
  2. Did the authors repeat experiments across different random seeds? If not, I would suggest to run some additional experiments to prove results hold also across different seeds; I do not expect, for obvious reasons, that all experiments are repeated again on many random seeds, but I would suggest at least a notable selection of the main experiments would be conducted in this way.
  3. Did the authors conduct further experiments with additional top-kk cut-offs? If not, I would suggest adding more experiments by considering other cut-offs, for instance with k=20,50k=20,50.
4 comments
AuthorsOpenReview ↗

Response to Reviewer cKW7 (Part 1)

Q1: Can the authors clarify what is the negative sampling number they choose for the experiments?

Response:

We thank the reviewer for pointing out this ambiguity. The negative sampling settings differ between training and evaluation. During training, we sample 4 negative items for each positive interaction. During leave-one-out testing, each held-out positive item is ranked alongside 99 randomly sampled negative items, forming 100 candidates in total. These settings follow a commonly adopted recommendation protocol [1][2]. We will revise Section 2.3 and Appendix A to clearly distinguish negative sampling for training from candidate sampling for testing.

[1] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pp. 173–182, 2017.

[2] Chunxu Zhang, Guodong Long, Tianyi Zhou, Peng Yan, Zijian Zhang, Chengqi Zhang, and Bo Yang. Dual personalization on federated recommendation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI'23, 2023.

Q2: Did the authors repeat experiments across different random seeds? If not, I would suggest to run some additional experiments to prove results hold also across different seeds; I do not expect, for obvious reasons, that all experiments are repeated again on many random seeds, but I would suggest at least a notable selection of the main experiments would be conducted in this way.

Response:

We thank the reviewer for suggesting additional experiments to assess the stability of our results across random seeds. Following this suggestion, we repeated the overall performance evaluation of all six PFR methods on five of the ten benchmark datasets using two additional random seeds (2025 and 2027), while keeping the other experimental settings unchanged. The results across all three seeds are reported below.

Overall, most methods exhibit relatively small performance variations across different seeds, with the main comparative trends remaining largely consistent. Although occasional ranking changes occur among methods with similar performance, the relative advantages of different approaches are generally preserved. These results provide further evidence for the stability of our overall performance comparisons.

We also noticed minor transcription errors in three previously reported FedRAP values (FilmTrust NDCG and LastFM-2K HR/NDCG), which have been corrected in the table below.

MethodSeedML100K HRNDCGFilmTrust HRNDCGLastFM HRNDCGMusical HRNDCGMoviesTV HRNDCG
PFedRecOrig.0.72530.43400.91620.81570.81750.71910.29090.16220.39120.2116
20250.71560.41390.91560.78220.82470.72610.28360.16100.39010.2095
20270.71050.42140.91320.81070.81630.71910.29090.16130.38600.2067
GPFedRecOrig.0.71690.43320.91320.80930.78000.68000.26810.14900.36000.1981
20250.69980.40920.91390.80360.79060.69590.27360.15770.35280.1850
20270.70100.39820.91020.80740.81630.71430.27260.15660.36220.2004
FedDAEOrig.0.54400.30530.91690.81800.83250.67900.26530.14530.37360.1988
20250.53970.30080.91540.80740.81620.65570.27720.14780.36050.1933
20270.56200.31090.89750.73130.83560.68320.26990.15080.38130.2040
FedCAOrig.0.88340.76590.92660.83830.68120.62400.80600.63880.56270.3666
20250.89740.76290.90790.82450.67370.61660.83070.61790.53340.3491
20270.89290.77200.94010.84700.71060.64300.84260.69960.56520.3662
FedRAPOrig.0.87270.74130.76650.56360.59120.54810.82530.64790.66570.4655
20250.85720.72920.76610.56280.59060.55360.84720.67130.67440.4753
20270.85370.73080.77620.56370.59310.55210.84720.64670.69330.4890
FedEMOrig.0.96080.88650.89150.79550.73750.65960.89200.69010.77890.5279
20250.97030.87520.86900.77550.75750.67170.89920.69210.79820.5361
20270.97140.88450.84730.76200.73810.65730.90480.69890.75480.5034
AuthorsOpenReview ↗

Response to Reviewer cKW7 (Part 2)

Q3: Did the authors conduct further experiments with additional top-k-cut-offs? If not, I would suggest adding more experiments by considering other cut-offs, for instance with k=20, 50.

Response:

We thank the reviewer for suggesting additional evaluations with different top-k cutoffs. Following this suggestion, we evaluated all six PFR methods on five of the ten benchmark datasets at k = 10, 20, and 50, using random seed 2027. The results show that the relative performance of different methods remains largely consistent across these cutoffs, indicating limited sensitivity to the choice of k. The results are reported below.

Methodtop-k-cut-offsML100K HRNDCGFilmTrust HRNDCGLastFM HRNDCGMusical HRNDCGMoviesTV HRNDCG
PFedRec100.71050.42140.91320.81070.81630.71910.29090.16130.38600.2067
200.86960.46200.93110.81540.86000.73040.40710.19050.56240.2512
500.97770.48410.96560.82220.92000.74230.63310.23480.83520.3053
GPFedRec100.70100.39820.91020.80740.81630.71430.27260.15660.36220.2004
200.86210.43950.93260.81310.87130.72830.39250.18680.51550.2390
500.97990.46360.96030.81860.92690.73930.62760.23280.77830.2910
FedDAE100.56200.31090.89750.73130.83560.68320.26990.15080.38130.2040
200.75190.35890.90940.73430.87940.69420.38330.17920.54200.2443
500.94590.39820.94990.74230.93000.70430.59010.21990.81170.2975
FedCA100.89290.77200.94010.84700.71060.64300.84260.69960.56520.3662
200.93850.78350.95730.85140.77000.65800.90030.71420.72690.4071
500.98830.79370.98580.85700.89130.68170.97350.72880.90920.4435
FedRAP100.85370.73080.77620.56370.59310.55210.84720.64670.69330.4890
200.91300.74600.88700.59200.65060.56660.89660.65920.81500.5197
500.98410.76040.96560.60760.79690.59520.96890.67360.93790.5443
FedEM100.97140.88450.84730.76200.73810.65730.90480.69890.75480.5034
200.99360.89010.89070.77300.80190.67330.95520.71170.88400.5361
501.00000.89140.96560.78800.91380.69550.99180.71890.97790.5551

Dear Authors,

Thanks for the rebuttal, which I carefully assessed.

  • Q1. OK, this is clear now.

  • Q2 - Q3. Thanks for having conducted more experiments with multiple random seeds and cut-offs. I think they can ensure a better generalization of the presented results across different empirical settings.

I do not have further concerns regarding the paper.

AuthorsOpenReview ↗

Thank You for Your Feedback!

Thank you very much for your careful consideration and encouraging feedback. We truly appreciate your time and suggestions, which have helped us improve our work!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewer Discussion and Author Responses

We sincerely thank the Area Chair and all reviewers for their careful evaluations and constructive discussions. As the discussion period concludes, we would like to briefly summarize the key points and outcomes.

PFR-Bench provides a unified evaluation of six representative personalized federated recommendation (PFR) methods across ten datasets, examining recommendation performance, personalization behavior, and reliability under different conditions. Through multidimensional and cross-experiment analyses, we aim to better understand the strengths, limitations, and trade-offs of existing PFR methods.

  • Reviewer cKW7. The reviewer raised questions regarding negative sampling settings, experimental stability across random seeds, and different top-k cutoffs. We clarified the evaluation settings and conducted additional experiments across multiple random seeds and top-k values, which largely confirmed the stability of our comparative findings. In the subsequent discussion, the reviewer acknowledged these results and explicitly stated that no further concerns remained.
  • Reviewer 2ubx. The reviewer raised concerns about negative sampling, the benefits of federated collaboration, and hyperparameter sensitivity. We conducted additional evaluations under an alternative negative sampling protocol, local-only comparisons, and extensive hyperparameter analyses. The results revealed some protocol-dependent changes in method rankings, while several personalization patterns remained consistent across the tested settings. The remaining disagreement concerns how protocol sensitivity should be interpreted in assessing the benchmark. While the reviewer requests a complete re-evaluation under the alternative protocol, our perspective is that a more comprehensive assessment should consider different reasonable sampling choices and their empirical implications, rather than treating either protocol as definitive.
  • Reviewer 2q6u. The reviewer raised concerns about the novelty of our findings and their connection to personalization designs. We clarified how cross-experiment analyses reveal relationships not readily apparent from individual evaluations, such as the differences between sparsity adaptation and noise robustness, and between collaborative benefits and propagation risks. The remaining difference concerns the empirical contribution of these findings. While we recognize that individual evaluation questions have been studied previously, we believe their systematic examination across PFR methods reveals how the relative strengths of PFR methods can vary across evaluation conditions.

We recognize that some differences in perspective remain regarding the evaluation scope and empirical contributions of PFR-Bench. We sincerely hope that the additional evidence and clarifications will encourage a fresh consideration of the paper, particularly its contributions as a systematic benchmark for PFR research. We are grateful to the Area Chair and all reviewers for their time, thoughtful feedback, and constructive engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.