- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FLARE, a federated framework for generating reliable local explanations using collaboratively learned SCARF representations while keeping raw data and explanation generation client-local. It constructs query-specific neighborhoods from real source and target instances, incorporates support-aware neighborhood completion and adaptive weighting, and fits weighted Ridge surrogates. Experiments across seven classification tasks compare FLARE against federated explainability baselines, reporting improvements in fidelity, reliability, and robustness across different client configurations and black-box models.
Critical strengths
- The paper addresses an important problem in federated explainability, particularly the difficulty of generating faithful local explanations when client data are sparse or heterogeneous.
- The evaluation considers multiple datasets, client configurations, and black-box models, incorporating reliability measures beyond conventional fidelity, including neighborhood support, stability, and robustness.
Critical weaknesses
- The contribution of federated representation learning is not adequately isolated. Comparisons against locally trained SCARF or non-federated representation baselines are missing, making it unclear whether federation itself drives the improvements.
- Baseline comparability is questionable because FLARE uses client-local black-box predictors, while some baselines explain federated predictors. The ablation study also disables two proposed mechanisms simultaneously, preventing attribution of their individual contributions.
- Several reporting inconsistencies undermine confidence in the results, including contradictory FedProx coefficients, identical FastSHAP++ results across client counts, and claims of superiority on every metric that are not fully supported by Table 1.
Questions for authors
- Can the authors compare federated SCARF with locally trained SCARF and conventional feature-space representations under otherwise identical settings?
- Are all methods evaluated against identical black-box predictors and common evaluation neighborhoods? If not, how are the fidelity comparisons justified?
- Can the authors provide separate ablations for neighborhood completion and adaptive weighting, along with baseline comparisons using the same reliability metrics?
- Can the authors clarify the inconsistent FedProx coefficients, repeated FastSHAP++ results across client counts, and discrepancies between the claimed and reported performance?
5 comments
Final days
Dear Reviewer, as the reviewer discussion phase comes to an end tomorrow, we hope that we have adequately addressed your comments and concerns. Please let us know if you need any further clarification or information to help you reassess your evaluation. Thank you again for your time and valuable feedback.
Consistency of Black-Box Predictors and Common Evaluation Neighborhoods
To address this concern, we ensure that all baseline methods use the same G-LibSVM black-box model configuration as FLARE. We clarify that the original baseline implementations employ different black-box prediction models. During manuscript preparation, we extend these implementations to support G-LibSVM that allows all methods to explain the same client-specific prediction targets while preserving their respective explanation mechanisms. Furthermore, all methods explain identical target-test instances and are evaluated on common held-out neighborhoods. We compute fidelity F1, MCC and AUC using a consistent evaluation protocol.
To verify this setup, we conduct prediction consistency checks across multiple test instances for each client. Specifically, we provide identical inputs to the G-LibSVM predictors used by FLARE and the baseline implementations and compare their predicted class labels and class probabilities. We confirm agreement across the evaluated instances. We also verify that all methods explain the same queries and use identical held-out evaluation observations by matching client identifiers, query indices and evaluation sample indices. These checks confirm that all methods explain consistent black-box predictions and are evaluated on the same test instances and neighborhoods, thereby allowing a fair comparison of their explanation fidelity.
We'll further clarify this evaluation protocol in the revised manuscript.
We thank the reviewer for the feedback and detailed observations. We address the concerns below and provide additional experimental details.
1. Contribution of federated representation learning
We conduct additional experiments with independently trained client-local SCARF encoders (LocalEnc). We compare LocalEnc with FedAvg (FA) and FedProx (FP) across 3, 5, and 10 clients using SVM and DNN black-box models.
The following table presents representative results for the 5-client setting using SVM.
All datasets use five clients except ACS Income (50 clients).
The results show that federated encoder training, particularly with FedProx, generally provides stronger fidelity than local training, with more improvements on Adult, Bank and Diabetes datasets. LocalEnc remains competitive on Student datasets and ACS Income. These findings support the benefits of collaborative representation learning without sharing raw client data.
2. Component-wise ablation study
We agree that disabling both mechanisms simultaneously does not isolate their individual contributions. Therefore, we conduct additional experiments evaluating adaptive surrogate class weighting (CW) and support-aware neighborhood completion (NC) separately and jointly.
The following table presents Fidelity F1 and explanation reliability (Rel.) for the 5-client FedProx-SVM setting.
The complete configuration achieves the highest Fidelity F1 and the highest explanation reliability across all datasets. CW generally improves fidelity, whereas NC more consistently improves reliability. Their combination provides the strongest overall results. We will incorporate these separate ablations into the revised manuscript.
3. Baseline comparability and evaluation protocol
FLARE is evaluated using SVM and DNN predictors, while FedRF-Surrogate, Fed-LIME, and FastSHAP++ results in Table 1 correspond to SVM black-box configurations. We acknowledge that the current presentation does not sufficiently establish whether the predictors and evaluation neighborhoods are identical across methods. We are further working on analyzing the black-box predictor configurations and evaluation protocols and will provide additional clarifications and results in a follow-up response.
We thank the reviewer again for the valuable suggestions. The additional experiments strengthen the evaluation of federated representation learning and the individual components of FLARE. We are also working on the remaining concerns and would be happy to provide further results or clarifications to support your final assessment.