- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper study the performance degradation problem in VFL caused by increasingly feature-partitioning feature block. The author first derive the worst-case bound on the population risk, reduces the identified factors without unduly restricting the extraction model’s expressiveness. Experiments are conducted on CIFAR10, CIFAR100 and HAPT.
Critical strengths
- This paper identified an interesting issue in VFL that become relevant as feature partition become fine-grained.
- The proposed method is simple and yet yielding large improvement, particularly at large .
- I also appreciate the authors explicitly acknowledge that the population-risk bound is two-sided, and reducing the bound does not necessary improve the actual risk.
- The runtime analysis and aggregation ablation are useful.
Critical weaknesses
-
The implementations appear inconsistent, i.e. main text state Adam but CIFAR appendix state AdamW.
-
all experiments use full participation . This substantially limits the empirical validation of participant-mean aggregation, since its claimed advantage is most relevant when the number of participating clients varies. Under fixed full participation, mean versus sum differs largely by a fixed scaling, and the paper's own ablation consequently finds little difference.
-
the theoretical motivation for T3 appears incomplete. Proposition 1 uses the Jacobian-Lipschitz constant of an inaccessible reference extractor , whereas T3 penalizes finite-difference curvature of the local models . I do not see an argument connecting these quantities. The experimental extractors additionally use ReLU, while Assumption 1 assumes continuous differentiability and a Lipschitz Jacobian.
Questions for authors
- See weakness
5 comments
W3. Theoretical motivation of (T3) and concerns about using ReLU.
Thanks to the reviewer's insightful suggestion, we noticed a deficiency in our explanation. As the reviewer recognizes, the reference extractor is a conceptual function introduced for the theoretical analysis and is not accessible in practice. Therefore, in principle, we cannot establish a method that guarantees the reduction of .
However, we believe that our idea of reducing the curvature of the accessible local extractors , instead of that of the inaccessible reference extractor , is reasonable. As explained in (T2), under (T1) and (T2), is proportional to the variance of the participating clients' feature blocks scaled by the curvature . Hence, can be reduced only by reducing . In the limit , this direction coincides with reducing the curvature of the local extraction models . This can be proven rigorously under a restriction such as expressing as a convex combination of the . Our original idea, however, did not rely on such a restriction: as described above, we designed (T3) because, in the limit , the direction of reducing coincides with that of reducing the curvature of the local extraction models .
We have added this explanation to Section~3.2 of the revised version (blue-colored sentences, lines 266-274, 316-333), and we would be grateful if the reviewer could read it. If the concern has not been mitigated, we would appreciate it if the reviewer could point it out again.
On the other hand, we understand the reviewer's point regarding ReLU. However, Lipschitz smoothness is assumed in many theoretical analyses (e.g., in the optimization literature [1, 2] below), while deep neural networks containing ReLU are commonly used in the corresponding empirical evaluations. We do not think this should be a basis for a negative score.
[1] S. J. Reddi et al., ``Adaptive Federated Optimization,'' ICLR 2021.
[2] S. P. Karimireddy et al., ``SCAFFOLD: Stochastic Controlled Averaging for Federated Learning,'' ICML 2020.
We understand that the reviewer is busy. However, the discussion period is running short, and about four days have passed since we posted our rebuttal. We have tried to respond to the reviewer's constructive comments sincerely and carefully, and we sincerely hope for a constructive exchange of views with each other. We would greatly appreciate it if the reviewer could let us know whether our responses have mitigated the concerns or point out any remaining issues, while there is still time to address them.
Thank you for taking the time to review our paper and for the constructive comments. We address the three weaknesses below. In addition, the revised paper is available, in which the updated phrases are highlighted in blue. We would greatly appreciate it if the reviewer could share whether our responses mitigate the concerns. We are happy to continue the discussion until the concerns are resolved.
W1. Typo regarding Adam/AdamW.
Thank you for pointing this out. This is a typo in the main text. We used AdamW for CIFAR-10/100 and Adam for HAPT, as stated in Appendix C. See the revised version, lines 446--447 in Section 4.1.
W2. Limitation to full participation cases.
Thank you for pointing this out. First, we explain why experiments under partial participation are difficult (Point 1). Second, we report the results of a partial-participation experiment that we conducted during the rebuttal period in a limited scenario (Point 2).
(Point 1) Our source code follows the LASER-VFL implementation (https://github.com/Valdeira/LASER-VFL), which evaluates the training objective as an exact weighted sum over all subsets instead of a sampled estimate. It therefore requires enumerating all 2K−1 client-availability patterns. Consequently, in the large-K regime (e.g., K=8 for 255 patterns and K=16 for 65,535 patterns), the training time increases exponentially with K because of this enumeration. We tested this, but could not run it within a realistic amount of time. This limitation is already stated in the original version (see lines 406--412).
However, we agree with the reviewer's concern regarding the evaluation under partial participation. We therefore addressed it during the rebuttal period, following a suggestion from another reviewer (7B81). Instead of enumerating all 2K−1 client-availability patterns, we fix the number of missing clients to ν and enumerate only (νK) patterns, with a small ν (e.g., (K,ν)=(8,1) for 8 patterns, (K,ν)=(8,2) for 28 patterns). Under this limitation, the evaluation under partial participation becomes feasible.
(Point 2) Following the above policy, we additionally conducted a partial-participation experiment during the rebuttal period. As explained above, we made this feasible by restricting the client-availability patterns to those in which the number of missing clients is a small number ν, even in the large-K regime (e.g., (K,ν)=(8,1),(8,2)), on the two image classification benchmarks (CIFAR-10/100). Our method has four variants, depending on the presence or absence of the two optional regularizations; because of limited computational resources, we tested only the variant without either regularization (P1).
Test accuracy under partial participation in limited scenarios. Since each method is extended so that each client holds its own fusion model, K prediction results are obtained; their mean and standard deviation are reported.
Although this was conducted under limited partial-participation scenarios, the effectiveness of PIGR-VFL is maintained. We have added these experimental results to Appendix C.4 in the revised version. In a further revised version, we will extend this experiment, for example, with the other variants of our method. We hope that this addresses the reviewer's concern.