- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper improves gradient inversion attacks in federated learning using VLM-based semantic guidance. Their observations suggest that existing attack methods suffer from unreliable label inference under defenses, insufficient semantic guidance for global structure and color, and poor recovery of fine-grained details. To address these three issues the authors first use a magnitude-based score to infer the true labels, which are later used by the semantic guidance using a VLM. The VLM guidance basically is a natural language prompt based on the inferred label in the first stage. At the final stage, they use a stage-wise gradient matching from layer to layer to recover the fine-grained details. Their experimental results show improvement and outperforming baseline across several benchmarks.
Critical strengths
- The method is sound, and well motivated. The explanation of three observations, and their proposed method to address those three observations is strong.
- The experimental results including ablations support their claims and demonstrate strong reported results.
- The experimental section convers three different types of evaluation, which covers reconstruction under raw gradients, reconstruction under purturbed gradients (gradients that have defense mechanisms), and one medical evaluation. I found this to be strong empirical evaluation. The inclusion of medical evaluation is also to be much appreciated.
Critical weaknesses
- The major weakness is that the increased batch size experiment in appendix (table 4) contains fewer benchmarks compared to the main results reported in table 1, 2, and 3. If the other benchmarks can be reported for increased batch size experiments, the overall claim could be stronger.
Questions for authors
- For larger batch sizes, whether the other benchmarks reported in the main result tables (table 1 & 2) have been evaluated? If so, can you share those results?
- For reconstruction from perturbed gradients under defense, the defense parameters (noise, clipping, etc.) have been mentioned. My question is, whether those parameters define any particular defense privacy budget, and if so what are those?
I find the method introduced in the paper to be interesting and well motivated, and the evaluations are consistent. For that reason, I’m leaning toward a weak acceptance.
3 comments
Response to the authors
I thank the authors for providing the answers to my questions. One interesting addition to the paper would have been to see how the method performs under different formal privacy budgets. But I don't see it as an absolute requirement, and hence I will retain my score for weak acceptance.
Response to the remaining concern on formal privacy budgets.
We thank the reviewer for the positive feedback and for maintaining the positive score. We argue that the appropriate privacy budget depends on the attack scenario.
(i) Membership inference attacks identify whether a specific sample was used in training. They require a strict privacy budget (e.g., ε ≤ 1), since their success is bounded by ε and this bound is nearly reached in practice (Nasr et al., 2023; Annamalai & De Cristofaro, 2024). (ii) Gradient inversion attacks can be enhanced with auxiliary knowledge (such as generative models and public datasets) and label inference (Balle et al., 2022; Hayes et al., 2023; Guerra-Balboa et al., 2026). Therefore, ε alone is not an absolute measurement of the reconstruction risk, and an exact privacy budget should depend on application scenarios.
We will include these further discussions in the revised manuscript!
References
- Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramèr, F., Jagielski, M., Carlini, N., and Terzis, A. Tight Auditing of Differentially Private Machine Learning. USENIX Security, 2023.
- Annamalai, M. S. M. S., and De Cristofaro, E. Nearly Tight Black-Box Auditing of Differentially Private Machine Learning. NeurIPS, 2024.
- Balle, B., Cherubin, G., and Hayes, J. Reconstructing Training Data with Informed Adversaries. IEEE S&P, 2022.
- Hayes, J., Mahloujifar, S., and Balle, B. Bounding Training Data Reconstruction in DP-SGD. NeurIPS, 2023.
- Guerra-Balboa, P., Sauer, A., Arcolezi, H. H., and Strufe, T. Understanding Disclosure Risk in Differential Privacy with Applications to Noise Calibration and Auditing. PVLDB, 2026.
Responses to Reviewer ZFGv
Responses to Reviewer ZFGv
We thank the reviewer for finding our method sound and well motivated, and for appreciating our evaluation, including the medical domain. Below, we address each concern.
W1 & Q1. Larger batch sizes with the other benchmarks in Tables 1 and 2.
We further evaluate CelebA and LSUN with a batch size of B=16. For CelebA, we use 2 classes defined by the gender attribute, i.e., male and female. For LSUN, we use the 10 scene categories (e.g., bedroom, church, and kitchen) as classes. Since both datasets contain fewer classes than the batch size of 16, we report the duplicate-label setting.
We argue that larger and more practical batch sizes remain an open challenge for diffusion-based GIAs, which is an interesting direction for future research.
Q2. Do the defense parameters (noise, clipping, etc.) define a particular privacy budget?
Thank you for the question. We set the defense parameters by perturbation strength, so they do not define a formal privacy budget. Instead, we show the utility–privacy trade-off in Sec. A.4 of the Appendix.