acceptodds
Under review as a conference paper at ICLR 2027

Sem2Pix: Privacy Leakage from Gradients via Semantic-to-Pixel Fine-Grained Guidance Diffusion Models

Abstract

Federated learning (FL) enables collaborative model training among multiple clients without requiring them to expose their raw images. Several studies have introduced attacks known as gradient inversion attacks (GIAs), which exploit the gradients shared during training to reconstruct clients' raw images. Recent work has shown that training-free diffusion models can transform Gaussian noise into recovered private images under guidance from shared gradients without requiring knowledge of the clients' data distribution. Our observations suggest that (i) guidance should constrain semantic information, such as global shape and color distribution, and (ii) additional guidance is needed to recover fine-grained details, which are particularly important in sensitive domains such as healthcare. In this paper, we present Semantic-to-Pixel Fine-Grained Guidence (Sem2Pix), which leverages vision–language models (VLMs) to enhance guidance for diffusion-based GIAs. First, we introduce VLM guidance for reverse diffusion to constrain the reconstructed image to be semantically meaningful. Second, we enhance the recovery of fine-grained details through stage-wise gradient guidance, which dynamically matches the layer-wise gradients induced by the predicted clean image with the corresponding shared client gradients. With these two steps, Sem2Pix can recover high-resolution images with high fidelity in practical federated learning scenarios involving defense mechanisms, larger batch sizes, and multiple local updates.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, Federated Learning, Gradient Inversion Attack, Privacy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZFGvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper improves gradient inversion attacks in federated learning using VLM-based semantic guidance. Their observations suggest that existing attack methods suffer from unreliable label inference under defenses, insufficient semantic guidance for global structure and color, and poor recovery of fine-grained details. To address these three issues the authors first use a magnitude-based score to infer the true labels, which are later used by the semantic guidance using a VLM. The VLM guidance basically is a natural language prompt based on the inferred label in the first stage. At the final stage, they use a stage-wise gradient matching from layer to layer to recover the fine-grained details. Their experimental results show improvement and outperforming baseline across several benchmarks.

Critical strengths
  • The method is sound, and well motivated. The explanation of three observations, and their proposed method to address those three observations is strong.
  • The experimental results including ablations support their claims and demonstrate strong reported results.
  • The experimental section convers three different types of evaluation, which covers reconstruction under raw gradients, reconstruction under purturbed gradients (gradients that have defense mechanisms), and one medical evaluation. I found this to be strong empirical evaluation. The inclusion of medical evaluation is also to be much appreciated.
Critical weaknesses
  • The major weakness is that the increased batch size experiment in appendix (table 4) contains fewer benchmarks compared to the main results reported in table 1, 2, and 3. If the other benchmarks can be reported for increased batch size experiments, the overall claim could be stronger.
Questions for authors
  • For larger batch sizes, whether the other benchmarks reported in the main result tables (table 1 & 2) have been evaluated? If so, can you share those results?
  • For reconstruction from perturbed gradients under defense, the defense parameters (noise, clipping, etc.) have been mentioned. My question is, whether those parameters define any particular defense privacy budget, and if so what are those?

I find the method introduced in the paper to be interesting and well motivated, and the evaluations are consistent. For that reason, I’m leaning toward a weak acceptance.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer ZFGv

Responses to Reviewer ZFGv

We thank the reviewer for finding our method sound and well motivated, and for appreciating our evaluation, including the medical domain. Below, we address each concern.

W1 & Q1. Larger batch sizes with the other benchmarks in Tables 1 and 2.

We further evaluate CelebA and LSUN with a batch size of B=16B=16. For CelebA, we use 2 classes defined by the gender attribute, i.e., male and female. For LSUN, we use the 10 scene categories (e.g., bedroom, church, and kitchen) as classes. Since both datasets contain fewer classes than the batch size of 16, we report the duplicate-label setting.

DatasetMethodLPIPS ↓SSIM ↑PSNR ↑GMSD ↓CLIP ↑
CelebAGGDM0.72170.202911.71220.29500.2570
GGSS-R0.46090.334515.38590.17550.5248
Sem2Pix0.39830.398019.04030.14660.7151
LSUNGGDM0.77100.174710.37910.32400.2193
GGSS-R0.51390.292613.78540.19660.4773
Sem2Pix0.45720.347717.13410.17000.6614
ImageNetGGDM0.92100.08907.8200.41200.1050
GGSS-R0.65400.182011.2000.25200.3520
Sem2Pix0.62100.208014.1500.23500.5120

We argue that larger and more practical batch sizes remain an open challenge for diffusion-based GIAs, which is an interesting direction for future research.

Q2. Do the defense parameters (noise, clipping, etc.) define a particular privacy budget?

Thank you for the question. We set the defense parameters by perturbation strength, so they do not define a formal privacy budget. Instead, we show the utility–privacy trade-off in Sec. A.4 of the Appendix.

Reviewer_ZFGvOpenReview ↗

Response to the authors

I thank the authors for providing the answers to my questions. One interesting addition to the paper would have been to see how the method performs under different formal privacy budgets. But I don't see it as an absolute requirement, and hence I will retain my score for weak acceptance.

AuthorsOpenReview ↗

Response to the remaining concern on formal privacy budgets.

We thank the reviewer for the positive feedback and for maintaining the positive score. We argue that the appropriate privacy budget depends on the attack scenario.

(i) Membership inference attacks identify whether a specific sample was used in training. They require a strict privacy budget (e.g., ε ≤ 1), since their success is bounded by ε and this bound is nearly reached in practice (Nasr et al., 2023; Annamalai & De Cristofaro, 2024). (ii) Gradient inversion attacks can be enhanced with auxiliary knowledge (such as generative models and public datasets) and label inference (Balle et al., 2022; Hayes et al., 2023; Guerra-Balboa et al., 2026). Therefore, ε alone is not an absolute measurement of the reconstruction risk, and an exact privacy budget should depend on application scenarios.

We will include these further discussions in the revised manuscript!

References

  • Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramèr, F., Jagielski, M., Carlini, N., and Terzis, A. Tight Auditing of Differentially Private Machine Learning. USENIX Security, 2023.
  • Annamalai, M. S. M. S., and De Cristofaro, E. Nearly Tight Black-Box Auditing of Differentially Private Machine Learning. NeurIPS, 2024.
  • Balle, B., Cherubin, G., and Hayes, J. Reconstructing Training Data with Informed Adversaries. IEEE S&P, 2022.
  • Hayes, J., Mahloujifar, S., and Balle, B. Bounding Training Data Reconstruction in DP-SGD. NeurIPS, 2023.
  • Guerra-Balboa, P., Sauer, A., Arcolezi, H. H., and Strufe, T. Understanding Disclosure Risk in Differential Privacy with Applications to Noise Calibration and Auditing. PVLDB, 2026.
Reviewer_GMxuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work mainly studies the diffusion-based gradient inversion attacks. Based on their three observations, they proposed magnitude-aware label leakage, vision-language semantic guidance and stage-wise gradient guidance. The experimental results show the large improvement.

Critical strengths

This paper is easy to follow and well-written. The studied diffusion-based gradient inversion attacks is an interesting and important problem.

Critical weaknesses

This work mainly studies the diffusion-based gradient inversion attacks. Based on their three observations, they proposed magnitude-aware label leakage, vision-language semantic guidance and stage-wise gradient guidance. Yet the contributions and experimental comparisons might be insufficient for an acceptance of top-tier conferences with the following concerns. The main concern is the experiment part. The authors said ‘Our implementation was built on the GGSS-R codebase’, while there exist the following differences. (1) Compared attacked models, this work only used ResNet-18, while in GGSS-R, the used attacked models include MLPs with 3/4/5 layers (MLP-3/4/5), CNN, ResNet18, and VGGNet16. (2) The results in Table 1 might not be consist with the ones in GGSS-R paper. The authors need to explain it. The potential inconsistent results might weaken their contributions and the large improvement. Besides, the authors might did more comparisons with GGSS-R, to support the contributions and describe the differences and improvement.

Fig.1 is hard to understand the claimed observations, without detailed experimental setting, or key cues.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Responses to Reviewer GMxu

We thank the reviewer for finding our paper studied problem interesting and important. Below, we address each concern. All revisions are marked in blue in the revised paper.

W1. Attacked models: this work only used ResNet-18, while GGSS-R also used MLP-3/4/5, CNN, and VGG-16.

We thank the reviewer for this suggestion. Appendix A.5.4 (Table 7) already evaluates Sem2Pix on ResNet-18, DenseNet-121, and ViT-B/16. Shallow fully connected networks are easier to invert than deeper backbones (Carletti et al., 2025). For a direct comparison, we further evaluate GGSS-R and Sem2Pix on untrained MLP-3/4/5 and ResNet-18 without BatchNorm on ImageNet.

ModelMethodLPIPS ↓SSIM ↑PSNR ↑GMSD ↓CLIP ↑
MLP-3GGSS-R0.05170.845333.43900.00880.9809
MLP-3Sem2Pix0.01820.924136.84030.00430.9939
MLP-4GGSS-R0.06590.809831.14070.01050.9767
MLP-4Sem2Pix0.02420.909934.71080.00520.9923
MLP-5GGSS-R0.08450.762929.25920.01280.9713
MLP-5Sem2Pix0.03230.890832.89040.00640.9902
ResNet-18GGSS-R0.17870.561722.37600.01950.9561
ResNet-18Sem2Pix0.07760.745125.84700.00860.9864

W2. The results in Table 1 might not be consistent with those in the GGSS-R paper.

We thank the reviewer for this important point. The difference comes from the victim model. Following prior work on worst-case leakage (Zhu et al., 2019; Zhao et al., 2020), we attack an untrained ResNet-18 without BatchNorm. In contrast, the GGSS-R paper uses ResNet-18 with BatchNorm, which makes the gradients harder to invert (Huang et al., 2021). We therefore further evaluate both methods on ResNet-18 with BatchNorm.

DatasetBatchNormMethodLPIPS ↓SSIM ↑PSNR ↑GMSD ↓CLIP ↑
CelebA✗GGSS-R0.02720.926938.51300.01240.9634
✗Sem2Pix0.00610.936637.90400.00560.9835
✓GGSS-R0.23020.692725.40030.07610.8706
✓Sem2Pix0.09730.816629.58590.02620.9593
ImageNet✗GGSS-R0.17870.561722.37600.01950.9561
✗Sem2Pix0.07760.745125.84700.00860.9864
✓GGSS-R0.33980.428315.21390.07940.8695
✓Sem2Pix0.16780.643319.80220.03060.9606
LSUN✗GGSS-R0.07370.914835.29800.02850.9671
✗Sem2Pix0.01970.932233.66800.00430.9888
✓GGSS-R0.27020.680023.08540.09320.8724
✓Sem2Pix0.09710.829427.31250.02200.9679

W3. More comparisons with GGSS-R to support the contributions and describe the differences and improvement.

We thank the reviewer for this suggestion. In addition to W1 and W2, the paper already compares Sem2Pix with GGSS-R under defense mechanisms (Table 2) and in the healthcare domain (Table 3). The Appendix further compares them with larger batch sizes (Table 4), multiple local steps (Table 5), and more communication rounds (Table 6).

We remark that GGSS-R steers the whole reverse process with full gradient updates, which is sensitive to deeper networks and the usage of BatchNorm (W1 and W2). In contrast, Sem2Pix (i) provides VLM semantic steering as the initialization of the global structure and color, and (ii) applies stage-wise gradient matching to progressively recover the fine-grained details.

W4. Fig. 1 is hard to understand without detailed experimental settings or key cues.

Thank you for the suggestion. We have redesigned Fig. 1 to include the experimental setting in the caption and zoomed-in regions that highlight the loss of semantic and fine-grained details.

References

  • Zhu, L., Liu, Z., and Han, S. Deep Leakage from Gradients. NeurIPS, 2019.
  • Zhao, B., Mopuri, K. R., and Bilen, H. iDLG: Improved Deep Leakage from Gradients. arXiv, 2020.
  • Huang, Y., Gupta, S., Song, Z., Li, K., and Arora, S. Evaluating Gradient Inversion Attacks and Defenses in Federated Learning. NeurIPS, 2021.
  • Carletti, V., Foggia, P., Mazzocca, C., Parrella, G., and Vento, M. SoK: Gradient Inversion Attacks in Federated Learning. USENIX Security, 2025.

Reviewer_WqqeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Gradient inversion attack is a critical threat for federated learning, which exploit the gradients shared during training to reconstruct clients' raw private images. Exploring how does gradient inversion attack work is a necessary way to understand and address this kind of data leakage risk. This paper claims that existing attack methods somewhat fail to recover class-compatible global structure and fine-grained details. There are three challenges: label leakage, semantic guidance, and fine-grained refinement. To address these challenges, this paper proposes Sem2Pix, a training-free diffusion-based attack that leverages VLMs for semantic guidance. Specifically, Sem2Pix constructs target text prompts from the inferred private labels and uses VLM guidance to align the reverse trajectory with the semantic information encoded in these prompts. Then it use stage-wise gradient guidance to match the layer-wise gradients induced by the predicted clean image with the correspinding shared client gradients, thereby progressively recovering fine-grained details. Experimental results on CelebA, LSUN, and ImageNet show the effectiveness of the proposed Sem2Pix.

Critical strengths
  1. This paper starts with reconstruction results generated by existing methods, then summarizes some observations to identify the challenges in this field. These observations and summaries are clear and benefit to the development of the research field.
  2. In response to the proposed observations, the authors introduced magnitude-aware label leakage, vision-language semantic guidance, and stage-wise gradient guidance. Each contribution is proposed to address one challenge. They are make sense.
  3. The evaluations are based on three mainstream datasets and the paper covers three kinds of comparison methods (pixel-based, GAN-based, and Diffusion-based) with nine attack methods. The experiments are comprehensive.
Critical weaknesses
  1. The ablation study about the vision-language semantic guidance is missing. Additionally, the paper uses CLIP but no other comparison VLMs. The effect of different VLMs is unclear.
  2. The attack cost is higher than comparison methods. Please clarify the feasible solutions to deal with this limitation.
  3. The layout of many figures leave large-region of blank space, such as Figure 1, 2, 4. Please polish the presentation.
  4. The samples used for visualization are almost that one, an image with a black chicken. Why? Please show qualitative results with more diversity.
  5. If the inferred label k is wrong, will the vision-language semantic guidance have the opposite effect, leading to the wrong generation.
Questions for authors

Please clarify the wrong inferred label issue and the ablation study of vision-language semantic guidance.

2 comments
AuthorsOpenReview ↗

Responses to Reviewer Wqqe

We thank the reviewer for recognizing our motivation. We address each concern below.

W1 & Q2. Effect of different VLMs.

We compare CLIP with SigLIP 2 [1], a stronger general-purpose VLM trained on WebLI (10B images), on natural images. We also compare CLIP with CONCH [2], a histopathology VLM trained on 1.17M image–caption pairs, on Camelyon17.

DatasetVLMLPIPS ↓SSIM ↑PSNR ↑GMSD ↓CLIP ↑
CelebACLIP0.00610.936637.90400.00560.9835
CelebASigLIP 20.06080.851826.47640.03910.9791
ImageNetCLIP0.07760.745125.84700.00860.9864
ImageNetSigLIP 20.15230.610121.29880.09540.9422
LSUNCLIP0.01970.932233.66800.00430.9888
LSUNSigLIP 20.07790.808525.79110.05960.8696
Camelyon17CLIP0.18810.670816.11240.12890.8322
Camelyon17CONCH0.12520.805923.22020.06250.9736

(i) Sem2Pix reconstructs the private images with all VLMs, showing that it is robust to the choice of VLM. (ii) On natural images, CLIP performs best, indicating that a general-purpose VLM provides sufficient semantic guidance. (iii) On Camelyon17, CONCH outperforms CLIP on all metrics. This suggests that domain-specific VLMs can further amplify privacy leakage in healthcare.

W2. Attack cost.

As discussed in Sec. A.3, CLIP guidance is the main cost. To reduce it, we move the transition in Eq. 11 earlier so that semantic guidance covers fewer reverse steps. The results on ImageNet are summarized below.

TransitionLPIPS ↓SSIM ↑PSNR ↑GMSD ↓CLIP ↑Time / image
t=T/2t = T/2 (main)0.07760.745125.84700.00860.9864~20.0 min
t=3T/4t = 3T/40.10850.692024.52000.01240.9872~14.9 min
t=7T/8t = 7T/80.15800.598022.84000.01820.9610~12.4 min

An earlier transition reduces the attack cost but lowers fidelity. Semantic guidance first establishes the global structure. Stage-wise gradient guidance then refines the fine-grained details. A shorter semantic stage therefore gives a weaker starting point. Therefore, the transition timestep provides a trade-off between fidelity and cost.

W3. Figure layout.

We have redesigned Figs. 1, 2 and 4 to remove the blank space. We also enlarged the fonts and added zoomed-in regions to highlight fine-grained details.

W4. Diversity of qualitative results.

We have added more diverse qualitative results in Appendix A.7. They cover the unperturbed setting on ImageNet, CelebA, and LSUN (Figs. 8–10), defense mechanisms (Fig. 11), and Camelyon17 (Fig. 12). Sem2Pix consistently preserves both the global structure and the fine-grained details.

W5 & Q1. Effect of a wrongly inferred label.

We intentionally steer the semantic guidance toward a label that differs from the private image.

DatasetSettingLPIPS ↓SSIM ↑PSNR ↑GMSD ↓CLIP ↑
CelebAWrong label0.48780.534712.43410.05540.7148
CelebASem2Pix0.00610.936637.90400.00560.9835
ImageNetWrong label0.65080.522819.66340.13040.6062
ImageNetSem2Pix0.07760.745125.84700.00860.9864
LSUNWrong label0.46250.715720.19750.04260.4672
LSUNSem2Pix0.01970.932233.66800.00430.9888

A wrong label decreases both semantic and pixel-level fidelity, since semantic guidance determines the global structure and color early in the reverse process. This highlights the importance of accurate label inference, which motivates our magnitude-aware label leakage (Sec. 3.2). It is also consistent with prior work on label leakage in GIAs [3, 4] and defenses [5, 6].

References [1] Tschannen et al. SigLIP 2. arXiv:2502.14786, 2025.

[2] Lu et al. A Visual-Language Foundation Model for Computational Pathology. Nature Medicine, 2024.

[3] Ye et al. High-Fidelity Gradient Inversion in Distributed Learning. AAAI, 2024.

[4] Hu et al. Venom: Liquid Diffusion-Guided Gradient Inversion for Breaking Differential Privacy in Federated Learning. AAAI, 2026.

[5] Zhang et al. CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling. NDSS, 2025.

[6] Raswa et al. Diffusion to Obfuscation: Time-Adaptive Synthesized Generation Against Gradient Leakage Attacks in Federated Learning. ECCV, 2026.

Dear Reviewers Wqqe & GMxu,

Thank you once more for your thorough review. We hope the clarifications and additional evidence in our responses and in the revised manuscript have addressed your concerns. With the discussion period ending soon, we would welcome any follow-up questions while there is still time for us to reply. We would also appreciate your reconsideration of your assessment and rating in light of our responses.

Thank you for your time and consideration!

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses to Reviewers' Comments

The authors thank the reviewers for their careful reading and helpful feedback, which have made the paper more complete. All reviewers find the paper sound and the studied problem important. Herewith, we provide more evidences in response to their feedbacks, as summarized below.

  1. Effect of the VLM (Wqqe). Sem2Pix successfully reconstructs the private images with all VLMs. CLIP performs best on natural images, while CONCH raises PSNR on Camelyon17 from 16.11 to 23.22 dB. This suggests that domain-specific VLMs can further amplify privacy leakage in healthcare.
  2. Wrongly inferred label (Wqqe). A wrong label lowers fidelity on all datasets (e.g., CelebA PSNR drops from 37.90 to 12.43). This highlights the importance of accurate label inference and motivates our magnitude-aware label leakage (Sec. 3.2).
  3. Attack cost (Wqqe). Moving the transition earlier (T/2 → 3T/4 → 7T/8) reduces the time per image from ~20.0 to ~14.9 and ~12.4 min, at the cost of lower PSNR (25.85 → 24.52 → 22.84) on ImageNet. We thank the reviewer for this feedback, which reveals a practical trade-off between attack cost and fidelity.
  4. Consistency with GGSS-R (GMxu). The gap comes from the victim model. We attack an untrained ResNet-18 without BatchNorm, whereas GGSS-R uses BatchNorm. Even with BatchNorm, Sem2Pix still outperforms GGSS-R on all datasets.
  5. More architectures (GMxu). Sem2Pix also outperforms GGSS-R on MLP-3/4/5 and ResNet-18, which complements the results in Table 7 (Appendix).
  6. Larger batch sizes (ZFGv). With a batch size of 16, Sem2Pix achieves the best performance on CelebA, LSUN, and ImageNet, while we also agree that larger batch sizes remain an open challenge and an interesting direction for future research on diffusion-based GIAs.
  7. Privacy budget (ZFGv). Our defense parameters are set by perturbation strength rather than a formal ε, and Sec. A.4 reports the utility–privacy trade-off. While membership inference requires a strict budget, ε alone is not an absolute measurement of the reconstruction risk in GIAs, since auxiliary knowledge can enhance the attack.
  8. Presentation (Wqqe, GMxu). We redesigned Figs. 1, 2, and 4 with clearer captions, larger fonts, and zoomed-in regions, and added more diverse qualitative results in Appendix A.7 (Figs. 8–12).

To sum up, we bring the motivation that a gradient inversion attack should recover both the semantic and fine-grained details of private images. Sem2Pix achieves this with a two-stage guidance, i.e., vision–language semantic guidance followed by stage-wise gradient guidance, which recovers private images from semantics to pixels. Moreover, we provide more experimental results in practical settings, both in the main paper and in the Appendix, including defense mechanisms, the healthcare domain, larger batch sizes, multiple local steps, more communication rounds, and deeper architectures, to further support our contributions.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.