- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VisDelta studies visual reconstruction from publicly released VLM LoRA adapters. It compares target separation under the public base and adapted models to guide latent optimization in a public image generator. The procedure combines candidate ranking, hard-negative separation, differential guidance, constrained refinement, and selection across transformed views. Experiments across several VLMs and face/breed datasets report improved target-related reconstruction. The evidence primarily concerns representative identity or class appearance rather than exact recovery of individual training photographs.
Critical strengths
-
The paper presents a comprehensive and clearly explained inversion pipeline. Using the public base model as a reference is well motivated for studying released adapters, and the component ablations support the usefulness of differential guidance.
-
Evaluation covers multiple VLM architectures and visual domains. Improvements over SMI-AW are consistent on FaceScrub and CelebA; CelebA Acc@1 increases from 22.57% to 26.38%. Wrong-adapter and permuted-condition controls further support dependence on the matched adapter and target.
Critical weaknesses
-
Attribution to private-data-specific visual information remains incomplete. A base-versus-adapted response difference identifies an adaptation effect, but does not establish whether reconstructed features come specifically from private training photographs rather than improved identity recognition combined with the generator's prior. Paired metrics use held-out images of the same identity or class. This supports representative reconstruction, which can itself pose a privacy risk, but stronger claims about private training information require more direct evidence.
-
The incremental contribution needs clearer isolation. SMI-AW already uses latent optimization, initial candidate selection, and augmentation-based final selection; LoRA-Leak already uses the pretrained model as a reference for leakage calibration in language models. The main new element is applying differential target separation to VLM image reconstruction. A stronger controlled comparison is needed to show its value beyond these existing ideas, especially given the modest gains and missing uncertainty estimates.
Questions for authors
-
Can matched adapters trained on different image subsets of the same identities, or sample/identity exclusion controls, identify what private visual information is recovered? How is overlap between the public generator's data and the target identities assessed?
-
What happens when the same base-model differential signal is added to SMI-AW, with ranking, transformations, selection, and total compute matched? This would help isolate the proposed contribution.
-
Please report variation across attack runs and absolute runtime. How is the independent identity/breed evaluator trained and validated, and how many targets and reconstructions enter each reported metric?
4 comments
Response to Reviewer uyVh (2/3): Contribution of differential guidance (W2/Q2)
This comment addresses W2 and Q2, which ask us to isolate the contribution of differential guidance () by inserting it directly into SMI-AW under the same surrounding pipeline, while also controlling for total guidance weight and optimization budget. We added this controlled comparison in Table 5 (p. 10) of the revised manuscript, with three-seed verification in Appendix E.12 (p. 25).
1) Direct insertion into SMI-AW under a matched pipeline
Both SMI-AW and VisDelta perform model inversion against VLMs, but SMI-AW optimizes latent codes using scores from the adapted VLM alone, whereas VisDelta contrasts against the base VLM through . On CelebA with Qwen2.5-VL-7B, we held the surrounding pipeline fixed—sharing candidate pools (), transformation distributions, and selection views ()—and inserted directly into SMI-AW. The table below summarizes the single-run results (seed 42), three-seed Acc@1 (mean std), evaluator distance, and runtime from Table 5:
| Configuration | Acc@1 ↑ | Acc@5 ↑ | Three-seed Acc@1 | ↓ | Time (s/target) ↓ |
|---|---|---|---|---|---|
| SMI-AW () | 22.00% | 49.50% | 22.17% ± 0.76% | 157.864 | 68.4 |
| SMI-AW + VisDelta pipeline (w/o ) | 23.00% | 50.50% | 22.83% ± 0.76% | 155.620 | 69.5 |
| SMI-AW + (matched pipeline) | 25.00% | 51.50% | 24.83% ± 0.76% | 151.240 | 97.8 |
| VisDelta w/o (rescaled weights) | 21.50% | 48.50% | 21.50% ± 0.50% | 158.920 | 69.1 |
| VisDelta w/ score difference | 23.50% | 51.00% | 23.33% ± 0.76% | 154.360 | 84.2 |
| SMI-AW () | 22.50% | 50.00% | 22.50% ± 0.50% | 156.410 | 114.5 |
| VisDelta | 26.00% | 52.00% | 26.00% ± 1.00% | 149.385 | 98.6 |
Adding the surrounding pipeline to SMI-AW raises Acc@1 from 22.00% to 23.00%. Inserting within that same pipeline raises Acc@1 from 23.00% to 25.00%, yielding a 2.00 percentage point (pp) gain in each of the three paired seeds (Top-1 hits rise from (46, 47, 44) to (50, 51, 48) out of 200), while improving FaceNet distance from 0.921 to 0.896 and evaluator distance from 155.620 to 151.240. This directly demonstrates that improves both target recognition and feature similarity when the surrounding pipeline is held fixed.
2) Controls for guidance scale and differential score formulation
Two additional controls further clarify the source of the gain:
- Guidance-weight control: Removing and setting preserves the default coefficient sum of 1.50. Acc@1 increases from 18.50% in the unrescaled ablation (Table 3, p. 9) to 21.50%, while full VisDelta retains a 4.50 pp advantage (26.00%).
- Simple score difference: Replacing with under the same default weights () gives 23.50% Acc@1, compared with 26.00% for our hard-negative differential margin—a three-seed paired gain of pp (95% CI: [1.95, 3.38] pp). As shown in Appendix A.1, Eq. (22) (p. 15), subtracting the mean score gain of competing conditions cancels shared additive shifts and isolates the adapter's improvement in target separation.
3) Full-method gains and computational efficiency
Combining with target agreement and the hard-negative margin in full VisDelta reaches 26.00% Acc@1 and 52.00% Acc@5, achieving paired three-seed Acc@1 gains of pp (95% CI: [2.45, 3.88] pp) over matched-pipeline SMI-AW and pp (95% CI: [0.45, 1.88] pp) over SMI-AW + (Table 5 and Appendix E.12). Furthermore, doubling SMI-AW refinement to 100 steps requires 114.5 s/target and yields 22.50% Acc@1, whereas VisDelta reuses frozen vision features across and to reach 26.00% at 98.6 s/target (3.50 pp higher accuracy with lower runtime), confirming that differential guidance provides a repeatable improvement beyond pipeline design and expanded baseline optimization.
Response to Reviewer uyVh (3/3): Evaluator protocol and reliable accuracy reporting (Q3)
In response to Q3, we explain how the independent Inception-v3 evaluator was trained and validated, how many targets and reconstructions contribute to each accuracy estimate under Eq. (31), and how accuracy varies across attack seeds. Table 5 (p. 10) also reports absolute runtime on CelebA: 98.6 s per target for VisDelta and 68.4 s for SMI-AW at 50 refinement steps.
1) Explicit evaluator training and validation protocol
In the revised manuscript, Appendix C (pp. 18–19) specifies the complete Inception-v3 evaluator protocol. For each target identity or breed, 20 images are divided into 14 adaptation, 3 validation, and 3 held-out reference images. Each evaluator is independently fine-tuned from ImageNet-pretrained weights on the 14 adaptation images per target for 50 epochs using SGD, momentum 0.9, initial learning rate 0.01, and cosine annealing, selecting the checkpoint with the highest validation Top-1 accuracy. The table below summarizes the training/validation splits and validation accuracies across all three datasets:
| Dataset | Training images | Validation images | Validation Top-1 (hits) | Validation Top-5 (hits) |
|---|---|---|---|---|
| FaceScrub | 7,420 | 1,590 | 96.42% (1,533/1,590) | 99.25% (1,578/1,590) |
| CelebA | 2,800 | 600 | 94.83% (569/600) | 98.67% (592/600) |
| Stanford Dogs | 1,680 | 360 | 86.39% (311/360) | 97.22% (350/360) |
The three held-out reference images per target are excluded from both victim adaptation and evaluator training. Evaluators remain frozen and are used only after the attack; candidate ranking, latent optimization, and final selection use only VLM scores, making the evaluation protocol and data partitions fully transparent.
2) One final reconstruction and one prediction per target
Thank you for identifying the mismatch between the original percentages and Eq. (31), which we have corrected throughout the revised manuscript. The original script classified multiple transformed views of each selected reconstruction and pooled predictions over valid views (skipping failed face detections), resulting in a view-level denominator.
We retained the same selected reconstructions and frozen evaluator and recomputed Acc@1/5 using one final untransformed reconstruction per target, following Eq. (31). Denominators are fixed at 530 identities for FaceScrub, 200 for CelebA, and 120 breeds for Stanford Dogs, using a 90% central crop resized to when face detection fails so every target remains in the denominator (Appendix C). For seed 42, VisDelta's Top-1/Top-5 counts are 131/530 and 270/530 on FaceScrub (24.72%/50.94%) and 52/200 and 104/200 on CelebA (26.00%/52.00%). All single-run results in Tables 1–3 (pp. 7–9), Tables 7–9 (pp. 22–23), and Figures 5, 6, 8–10 now report seed 42 under this target-level protocol, so every accuracy value corresponds directly to integer target hits.
3) Consistent gains across paired attack seeds
We evaluated Qwen2.5-VL-7B across paired seeds 42, 43, 44, sharing initial latent codes and transformation draws across methods within each seed. The table below summarizes the multi-seed mean standard deviation from Table 10 (p. 23):
| Dataset | Method | Acc@1 ↑ | Acc@5 ↑ |
|---|---|---|---|
| FaceScrub | Base Only | 11.38% ± 0.29% | 33.96% ± 0.50% |
| FaceScrub | Final CE Only | 19.18% ± 0.76% | 44.59% ± 0.97% |
| FaceScrub | SMI-AW | 21.45% ± 0.58% | 48.30% ± 0.86% |
| FaceScrub | VisDelta | 24.97% ± 0.61% | 51.07% ± 0.76% |
| CelebA | Base Only | 13.00% ± 0.50% | 35.17% ± 0.76% |
| CelebA | Final CE Only | 20.50% ± 1.00% | 46.33% ± 1.26% |
| CelebA | SMI-AW | 22.17% ± 0.76% | 49.83% ± 1.04% |
| CelebA | VisDelta | 26.00% ± 1.00% | 52.17% ± 0.76% |
VisDelta's paired Acc@1 gains over SMI-AW are pp on FaceScrub (95% CI: [3.25, 3.79] pp) and pp on CelebA (95% CI: [3.12, 4.55] pp; two-sided Student's , ). Separately, 10,000 paired bootstrap resamples of seed-42 targets give identity-level Acc@1 CIs of [21.13%, 28.49%] and [20.00%, 32.50%] (Appendix E.10, pp. 22–23). Both paired seed intervals exclude zero, confirming that VisDelta's recovery advantage on both face datasets is statistically consistent and directly traceable to integer predictions.
Follow-up on our response
We appreciate the care you put into your review and the constructive feedback you shared. Our response is now posted, and we have uploaded a revised manuscript with the changes marked in blue. If you have a chance before the discussion closes later today, we would welcome your thoughts on whether the revisions address your concerns. We would be glad to answer any further questions. Thank you again for your time.
Response to Reviewer uyVh (1/3): Dependence on private adaptation data (W1/Q1)
We thank Reviewer uyVh for the constructive review. We have uploaded a revised manuscript with all substantive changes highlighted in blue. Below we address W1 and Q1, which ask whether VisDelta reconstructs visual information learned from the private adaptation images rather than pre-existing associations in the base VLM or public StyleGAN2 generator.
1) Recovery with synthetic identities and anonymized labels
In the revised manuscript, Table 4 (p. 10) and Appendix E.11 (p. 24) evaluate Qwen2.5-VL-7B on 100 synthetic StyleGAN2 identities assigned random two-word pseudonyms (20 images per identity via anchor-latent perturbations and FaceNet consistency checks: 14 adaptation, 3 validation, and 3 held-out images) and on Anonymized FaceScrub, where all 530 names are replaced by anonymous labels (
ID_001–ID_530) in both adaptation and attack queries. All methods share the same public generator. The table below summarizes the recovery and sample-level distance results from Table 4:Base Only is near chance (1.00% and 0.19%), whereas VisDelta reaches 28.00% and 21.89% Acc@1, exceeding SMI-AW by 6.00 and 3.96 percentage points (pp) (three-seed Acc@1: 27.67% ± 0.58% and 21.76% ± 0.58% Section 5.4, pp. 9–10; Table 4 also reports CS and LPIPS). This confirms that recovery remains effective with newly assigned labels instead of original celebrity names.
2) Reconstructions are closer to actual adaptation images than held-out images
To compare proximity to training and held-out images of the same identity using equal reference-set sizes, δtrain averages nearest-neighbor FaceNet distance over 20 random draws of 3 training images, while δheld uses the 3 held-out images. VisDelta's distance gap Δδface=δheld−δtrain is +0.070 on synthetic identities and +0.061 on anonymized FaceScrub (both p<0.0001 via two-sided paired t-tests across 100 and 530 identities), showing that reconstructions align more closely with the actual adaptation images after controlling for reference-set size.
3) Gains persist across distances to the public generator
We computed each CelebA target's minimum FaceNet distance to 10,000 generated StyleGAN2-FFHQ images and split the 200 targets at the median into near and far groups of 100 (Section 5.4, p. 10; Appendix E.11, p. 24). VisDelta outperforms SMI-AW by 4.00 pp in both the near group (29.00% vs. 25.00% Acc@1) and the far group (23.00% vs. 19.00%), showing consistent gains across variation in proximity to the generator manifold.
4) Changing training images reverses reconstruction alignment
In Table 11 (p. 25), each CelebA identity's 14 adaptation images are split into disjoint seven-image subsets DA and DB, and adapters M1(A) and M1(B) each train on 1,400 images with identical initialization, batch-order seed, and 10-epoch schedule. The table below reports the mean nearest-neighbor FaceNet distance from VisDelta's reconstructions to each seven-image subset:
Switching the training subset reverses alignment, with a 0.076 distance advantage for the trained subset in both directions (p<0.0005 across 200 identities). Moreover, training ten adapters that each exclude 20 target identities (replacing their 280 images to keep 2,800 training examples) reduces Acc@1 on excluded targets from 26.00% (Table 1, p. 7) to 12.50% (near Base Only's 13.00%). Subset MIA yields 68.40%/67.90% AUC for the two subset-trained adapters. When neither subset is used for training, the negative control gives 50.40% AUC, close to chance. Table 11 also reports 64.00%/63.50% R@1 for the two subset-trained adapters. Together, these controls show that recovery depends on target-identity inclusion and captures training-subset visual cues from the private adaptation data.