acceptodds
Under review as a conference paper at ICLR 2027

VisDelta: Model Inversion from Released LoRA Adapters for Vision-Language Models

Abstract

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for vision-language models (VLMs) that updates only a small fraction of their parameters. When trained on private images, LoRA adapters retain visual information from them. Adapters are commonly shared on public model platforms, creating a serious risk of model inversion. In this paper, we propose VisDelta, a model inversion attack against released VLM LoRA adapters. It compares the base and LoRA-adapted VLMs on the same image and query and uses their response difference to guide image reconstruction. Firstly, VisDelta samples latent codes at random and generates an initial image from each using a public image generator. It ranks these images by an adapter-aware score, retains the codes that produced the top-ranked images, and refines each code through adapter-aware differential guidance to reconstruct private images associated with the target. The guidance combines target agreement, separation from hard-negative conditions, and the adapter-induced gain in target separation, while keeping each code close to its initial value. Refinement uses random image transformations; final selection balances average and worst-view losses over fresh transformations to choose a reconstruction with stable target responses. Experiments across four VLMs and three benchmark datasets show that VisDelta recovers target-related visual content from LoRA adapters trained on private images. On CelebA with Qwen2.5-VL-7B, it achieves 26.38% Acc@1 and 52.47% Acc@5, exceeding the strongest baseline by 3.81 and 2.29 percentage points, respectively. These results show that released VLM LoRA adapters can expose private visual information.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model inversion, Low-Rank Adaptation, vision-language models, parameter-efficient fine-tuning, visual privacy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uyVhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VisDelta studies visual reconstruction from publicly released VLM LoRA adapters. It compares target separation under the public base and adapted models to guide latent optimization in a public image generator. The procedure combines candidate ranking, hard-negative separation, differential guidance, constrained refinement, and selection across transformed views. Experiments across several VLMs and face/breed datasets report improved target-related reconstruction. The evidence primarily concerns representative identity or class appearance rather than exact recovery of individual training photographs.

Critical strengths
  1. The paper presents a comprehensive and clearly explained inversion pipeline. Using the public base model as a reference is well motivated for studying released adapters, and the component ablations support the usefulness of differential guidance.

  2. Evaluation covers multiple VLM architectures and visual domains. Improvements over SMI-AW are consistent on FaceScrub and CelebA; CelebA Acc@1 increases from 22.57% to 26.38%. Wrong-adapter and permuted-condition controls further support dependence on the matched adapter and target.

Critical weaknesses
  1. Attribution to private-data-specific visual information remains incomplete. A base-versus-adapted response difference identifies an adaptation effect, but does not establish whether reconstructed features come specifically from private training photographs rather than improved identity recognition combined with the generator's prior. Paired metrics use held-out images of the same identity or class. This supports representative reconstruction, which can itself pose a privacy risk, but stronger claims about private training information require more direct evidence.

  2. The incremental contribution needs clearer isolation. SMI-AW already uses latent optimization, initial candidate selection, and augmentation-based final selection; LoRA-Leak already uses the pretrained model as a reference for leakage calibration in language models. The main new element is applying differential target separation to VLM image reconstruction. A stronger controlled comparison is needed to show its value beyond these existing ideas, especially given the modest gains and missing uncertainty estimates.

Questions for authors
  1. Can matched adapters trained on different image subsets of the same identities, or sample/identity exclusion controls, identify what private visual information is recovered? How is overlap between the public generator's data and the target identities assessed?

  2. What happens when the same base-model differential signal is added to SMI-AW, with ranking, transformations, selection, and total compute matched? This would help isolate the proposed contribution.

  3. Please report variation across attack runs and absolute runtime. How is the independent identity/breed evaluator trained and validated, and how many targets and reconstructions enter each reported metric?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uyVh (1/3): Dependence on private adaptation data (W1/Q1)

We thank Reviewer uyVh for the constructive review. We have uploaded a revised manuscript with all substantive changes highlighted in blue. Below we address W1 and Q1, which ask whether VisDelta reconstructs visual information learned from the private adaptation images rather than pre-existing associations in the base VLM or public StyleGAN2 generator.

1) Recovery with synthetic identities and anonymized labels

In the revised manuscript, Table 4 (p. 10) and Appendix E.11 (p. 24) evaluate Qwen2.5-VL-7B on 100 synthetic StyleGAN2 identities assigned random two-word pseudonyms (20 images per identity via anchor-latent perturbations and FaceNet consistency checks: 14 adaptation, 3 validation, and 3 held-out images) and on Anonymized FaceScrub, where all 530 names are replaced by anonymous labels (ID_001–ID_530) in both adaptation and attack queries. All methods share the same public generator. The table below summarizes the recovery and sample-level distance results from Table 4:

SettingMethodAcc@1 ↑Acc@5 ↑δtrain\delta_{\mathrm{train}} ↓δheld\delta_{\mathrm{held}} ↓
Synthetic IDsBase Only1.00%5.00%1.1921.194
Synthetic IDsFinal CE Only20.00%44.00%0.8950.931
Synthetic IDsSMI-AW22.00%48.00%0.8720.911
Synthetic IDsVisDelta28.00%55.00%0.8120.882
Anonymized FaceScrubBase Only0.19%0.94%1.2051.204
Anonymized FaceScrubFinal CE Only16.04%39.81%0.9120.938
Anonymized FaceScrubSMI-AW17.92%42.64%0.8850.918
Anonymized FaceScrubVisDelta21.89%46.79%0.7970.858

Base Only is near chance (1.00% and 0.19%), whereas VisDelta reaches 28.00% and 21.89% Acc@1, exceeding SMI-AW by 6.00 and 3.96 percentage points (pp) (three-seed Acc@1: 27.67% ± 0.58% and 21.76% ± 0.58% Section 5.4, pp. 9–10; Table 4 also reports CS and LPIPS). This confirms that recovery remains effective with newly assigned labels instead of original celebrity names.

2) Reconstructions are closer to actual adaptation images than held-out images

To compare proximity to training and held-out images of the same identity using equal reference-set sizes, δtrain\delta_{\mathrm{train}} averages nearest-neighbor FaceNet distance over 20 random draws of 3 training images, while δheld\delta_{\mathrm{held}} uses the 3 held-out images. VisDelta's distance gap Δδface=δheld−δtrain\Delta\delta_{\mathrm{face}}=\delta_{\mathrm{held}}-\delta_{\mathrm{train}} is +0.070 on synthetic identities and +0.061 on anonymized FaceScrub (both p<0.0001p<0.0001 via two-sided paired tt-tests across 100 and 530 identities), showing that reconstructions align more closely with the actual adaptation images after controlling for reference-set size.

3) Gains persist across distances to the public generator

We computed each CelebA target's minimum FaceNet distance to 10,000 generated StyleGAN2-FFHQ images and split the 200 targets at the median into near and far groups of 100 (Section 5.4, p. 10; Appendix E.11, p. 24). VisDelta outperforms SMI-AW by 4.00 pp in both the near group (29.00% vs. 25.00% Acc@1) and the far group (23.00% vs. 19.00%), showing consistent gains across variation in proximity to the generator manifold.

4) Changing training images reverses reconstruction alignment

In Table 11 (p. 25), each CelebA identity's 14 adaptation images are split into disjoint seven-image subsets DA\mathcal D_A and DB\mathcal D_B, and adapters M1(A)M_1^{(A)} and M1(B)M_1^{(B)} each train on 1,400 images with identical initialization, batch-order seed, and 10-epoch schedule. The table below reports the mean nearest-neighbor FaceNet distance from VisDelta's reconstructions to each seven-image subset:

Adapter used by VisDeltaDistance to DA\mathcal D_A ↓Distance to DB\mathcal D_B ↓
M1(A)M_1^{(A)}, trained on DA\mathcal D_A0.8380.914
M1(B)M_1^{(B)}, trained on DB\mathcal D_B0.9190.843

Switching the training subset reverses alignment, with a 0.076 distance advantage for the trained subset in both directions (p<0.0005p<0.0005 across 200 identities). Moreover, training ten adapters that each exclude 20 target identities (replacing their 280 images to keep 2,800 training examples) reduces Acc@1 on excluded targets from 26.00% (Table 1, p. 7) to 12.50% (near Base Only's 13.00%). Subset MIA yields 68.40%/67.90% AUC for the two subset-trained adapters. When neither subset is used for training, the negative control gives 50.40% AUC, close to chance. Table 11 also reports 64.00%/63.50% R@1 for the two subset-trained adapters. Together, these controls show that recovery depends on target-identity inclusion and captures training-subset visual cues from the private adaptation data.

AuthorsOpenReview ↗

Response to Reviewer uyVh (2/3): Contribution of differential guidance (W2/Q2)

This comment addresses W2 and Q2, which ask us to isolate the contribution of differential guidance (LΔ\mathcal L_\Delta) by inserting it directly into SMI-AW under the same surrounding pipeline, while also controlling for total guidance weight and optimization budget. We added this controlled comparison in Table 5 (p. 10) of the revised manuscript, with three-seed verification in Appendix E.12 (p. 25).

1) Direct insertion into SMI-AW under a matched pipeline

Both SMI-AW and VisDelta perform model inversion against VLMs, but SMI-AW optimizes latent codes using scores from the adapted VLM M1M_1 alone, whereas VisDelta contrasts M1M_1 against the base VLM M0M_0 through LΔ\mathcal L_\Delta. On CelebA with Qwen2.5-VL-7B, we held the surrounding pipeline fixed—sharing candidate pools (N=512,K=8N=512,K=8), transformation distributions, and selection views (L=10L=10)—and inserted LΔ\mathcal L_\Delta directly into SMI-AW. The table below summarizes the single-run results (seed 42), three-seed Acc@1 (mean ±\pm std), evaluator distance, and runtime from Table 5:

ConfigurationAcc@1 ↑Acc@5 ↑Three-seed Acc@1δeval\delta_{\mathrm{eval}} ↓Time (s/target) ↓
SMI-AW (T=50T=50)22.00%49.50%22.17% ± 0.76%157.86468.4
SMI-AW + VisDelta pipeline (w/o LΔ\mathcal L_\Delta)23.00%50.50%22.83% ± 0.76%155.62069.5
SMI-AW + LΔ\mathcal L_\Delta (matched pipeline)25.00%51.50%24.83% ± 0.76%151.24097.8
VisDelta w/o LΔ\mathcal L_\Delta (rescaled weights)21.50%48.50%21.50% ± 0.50%158.92069.1
VisDelta w/ score difference gΔg_\Delta23.50%51.00%23.33% ± 0.76%154.36084.2
SMI-AW (T=100T=100)22.50%50.00%22.50% ± 0.50%156.410114.5
VisDelta26.00%52.00%26.00% ± 1.00%149.38598.6

Adding the surrounding pipeline to SMI-AW raises Acc@1 from 22.00% to 23.00%. Inserting LΔ\mathcal L_\Delta within that same pipeline raises Acc@1 from 23.00% to 25.00%, yielding a 2.00 percentage point (pp) gain in each of the three paired seeds (Top-1 hits rise from (46, 47, 44) to (50, 51, 48) out of 200), while improving FaceNet distance from 0.921 to 0.896 and evaluator distance from 155.620 to 151.240. This directly demonstrates that LΔ\mathcal L_\Delta improves both target recognition and feature similarity when the surrounding pipeline is held fixed.

2) Controls for guidance scale and differential score formulation

Two additional controls further clarify the source of the gain:

  • Guidance-weight control: Removing LΔ\mathcal L_\Delta and setting (λans,λmar)=(0.75,0.75)(\lambda_{\mathrm{ans}},\lambda_{\mathrm{mar}})=(0.75,0.75) preserves the default coefficient sum of 1.50. Acc@1 increases from 18.50% in the unrescaled ablation (Table 3, p. 9) to 21.50%, while full VisDelta retains a 4.50 pp advantage (26.00%).
  • Simple score difference: Replacing Δm\Delta m with gΔ=s1−s0g_\Delta=s_1-s_0 under the same default weights (λΔ=1,τ=1\lambda_\Delta=1,\tau=1) gives 23.50% Acc@1, compared with 26.00% for our hard-negative differential margin—a three-seed paired gain of 2.67±0.292.67\pm0.29 pp (95% CI: [1.95, 3.38] pp). As shown in Appendix A.1, Eq. (22) (p. 15), subtracting the mean score gain of competing conditions cancels shared additive shifts and isolates the adapter's improvement in target separation.

3) Full-method gains and computational efficiency

Combining LΔ\mathcal L_\Delta with target agreement and the hard-negative margin in full VisDelta reaches 26.00% Acc@1 and 52.00% Acc@5, achieving paired three-seed Acc@1 gains of 3.17±0.293.17\pm0.29 pp (95% CI: [2.45, 3.88] pp) over matched-pipeline SMI-AW and 1.17±0.291.17\pm0.29 pp (95% CI: [0.45, 1.88] pp) over SMI-AW + LΔ\mathcal L_\Delta (Table 5 and Appendix E.12). Furthermore, doubling SMI-AW refinement to 100 steps requires 114.5 s/target and yields 22.50% Acc@1, whereas VisDelta reuses frozen vision features across M0M_0 and M1M_1 to reach 26.00% at 98.6 s/target (3.50 pp higher accuracy with lower runtime), confirming that differential guidance provides a repeatable improvement beyond pipeline design and expanded baseline optimization.

AuthorsOpenReview ↗

Response to Reviewer uyVh (3/3): Evaluator protocol and reliable accuracy reporting (Q3)

In response to Q3, we explain how the independent Inception-v3 evaluator was trained and validated, how many targets and reconstructions contribute to each accuracy estimate under Eq. (31), and how accuracy varies across attack seeds. Table 5 (p. 10) also reports absolute runtime on CelebA: 98.6 s per target for VisDelta and 68.4 s for SMI-AW at 50 refinement steps.

1) Explicit evaluator training and validation protocol

In the revised manuscript, Appendix C (pp. 18–19) specifies the complete Inception-v3 evaluator protocol. For each target identity or breed, 20 images are divided into 14 adaptation, 3 validation, and 3 held-out reference images. Each evaluator is independently fine-tuned from ImageNet-pretrained weights on the 14 adaptation images per target for 50 epochs using SGD, momentum 0.9, initial learning rate 0.01, and cosine annealing, selecting the checkpoint with the highest validation Top-1 accuracy. The table below summarizes the training/validation splits and validation accuracies across all three datasets:

DatasetTraining imagesValidation imagesValidation Top-1 (hits)Validation Top-5 (hits)
FaceScrub7,4201,59096.42% (1,533/1,590)99.25% (1,578/1,590)
CelebA2,80060094.83% (569/600)98.67% (592/600)
Stanford Dogs1,68036086.39% (311/360)97.22% (350/360)

The three held-out reference images per target are excluded from both victim adaptation and evaluator training. Evaluators remain frozen and are used only after the attack; candidate ranking, latent optimization, and final selection use only VLM scores, making the evaluation protocol and data partitions fully transparent.

2) One final reconstruction and one prediction per target

Thank you for identifying the mismatch between the original percentages and Eq. (31), which we have corrected throughout the revised manuscript. The original script classified multiple transformed views of each selected reconstruction and pooled predictions over valid views (skipping failed face detections), resulting in a view-level denominator.

We retained the same selected reconstructions and frozen evaluator and recomputed Acc@1/5 using one final untransformed reconstruction per target, following Eq. (31). Denominators are fixed at 530 identities for FaceScrub, 200 for CelebA, and 120 breeds for Stanford Dogs, using a 90% central crop resized to 224×224224\times224 when face detection fails so every target remains in the denominator (Appendix C). For seed 42, VisDelta's Top-1/Top-5 counts are 131/530 and 270/530 on FaceScrub (24.72%/50.94%) and 52/200 and 104/200 on CelebA (26.00%/52.00%). All single-run results in Tables 1–3 (pp. 7–9), Tables 7–9 (pp. 22–23), and Figures 5, 6, 8–10 now report seed 42 under this target-level protocol, so every accuracy value corresponds directly to integer target hits.

3) Consistent gains across paired attack seeds

We evaluated Qwen2.5-VL-7B across paired seeds 42, 43, 44, sharing initial latent codes and transformation draws across methods within each seed. The table below summarizes the multi-seed mean ±\pm standard deviation from Table 10 (p. 23):

DatasetMethodAcc@1 ↑Acc@5 ↑
FaceScrubBase Only11.38% ± 0.29%33.96% ± 0.50%
FaceScrubFinal CE Only19.18% ± 0.76%44.59% ± 0.97%
FaceScrubSMI-AW21.45% ± 0.58%48.30% ± 0.86%
FaceScrubVisDelta24.97% ± 0.61%51.07% ± 0.76%
CelebABase Only13.00% ± 0.50%35.17% ± 0.76%
CelebAFinal CE Only20.50% ± 1.00%46.33% ± 1.26%
CelebASMI-AW22.17% ± 0.76%49.83% ± 1.04%
CelebAVisDelta26.00% ± 1.00%52.17% ± 0.76%

VisDelta's paired Acc@1 gains over SMI-AW are 3.52±0.113.52\pm0.11 pp on FaceScrub (95% CI: [3.25, 3.79] pp) and 3.83±0.293.83\pm0.29 pp on CelebA (95% CI: [3.12, 4.55] pp; two-sided Student's tt, df=2df=2). Separately, 10,000 paired bootstrap resamples of seed-42 targets give identity-level Acc@1 CIs of [21.13%, 28.49%] and [20.00%, 32.50%] (Appendix E.10, pp. 22–23). Both paired seed intervals exclude zero, confirming that VisDelta's recovery advantage on both face datasets is statistically consistent and directly traceable to integer predictions.

AuthorsOpenReview ↗

Follow-up on our response

We appreciate the care you put into your review and the constructive feedback you shared. Our response is now posted, and we have uploaded a revised manuscript with the changes marked in blue. If you have a chance before the discussion closes later today, we would welcome your thoughts on whether the revisions address your concerns. We would be glad to answer any further questions. Thank you again for your time.

Reviewer_EdmJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces VisDelta, a model inversion attack on released LoRA adapters for vision–language models. It optimizes latent codes of a public image generator using differences in target separation between the base and adapted models. Candidate screening, transformation-aware refinement, and consistency-based selection are combined to recover target-related visual content without accessing private training images or training-time gradients.

Critical strengths

The paired comparison between the base and adapted models is well motivated for the adapter-release setting. Evaluating identical inputs under both models provides a direct measure of adapter-induced response changes. The experimental design includes useful wrong-adapter and permuted-condition controls, alongside component ablations. Using independent evaluators only after reconstruction also avoids directly optimizing the attack against its evaluation metrics. index index

Critical weaknesses
  1. The methodological novelty is limited and needs sharper positioning. Generator-based latent optimization, initial candidate screening, and augmentation-based final selection already appear in SMI-AW and its supplementary protocol. The main addition here is the base–adapter differential margin, rather than the three-stage pipeline itself. A matched-pipeline comparison against a simpler target-score difference would better establish the value of the hard-negative differential formulation.
  2. Attribution to private visual information remains incomplete. An adapter-induced response change does not, by itself, distinguish newly learned visual information from improved access to concepts already represented in the base model. Evaluation against one held-out image per identity or breed supports target-level reconstruction, but does not resolve this distinction. The wrong-adapter and permuted-condition controls establish dependence on the adapter and target, not on particular private visual examples. Matched counterfactual adapters, following the formulation in Appendix A.3, would provide stronger evidence.
  3. The reported accuracies are inconsistent with the stated evaluation protocol unless additional aggregation was performed. With 200 CelebA identities and one final reconstruction per target, Eq. (31) yields accuracy in increments of 0.5 percentage points. Values such as 26.38% Acc@1 and 52.47% Acc@5 therefore require an unreported averaging procedure or a different evaluation count. The exact denominator, number of independent runs, and aggregation method must be specified. Without uncertainty estimates, the reliability of the relatively small improvements over SMI-AW is also unclear.
  4. Essential implementation details are missing. StarGAN v2 requires a source image in addition to a style code. Algorithm 1 introduces a public context sampler, but does not explain its source-image distribution or how it is instantiated for the dog experiments. This matters because source-image content could contribute to apparent reconstruction quality. The source-image protocol should be specified and controlled experimentally.
  5. Several textual and reporting errors should be corrected. The paragraph at the bottom of page 3 ends abruptly with “across freshly.” Appendix D contains a sentence without a verb: “The pipeline of proposed VisDelta in Algorithm 1.” In addition, Appendix A.3 refers to membership and image–condition retrieval results, but these results are not presented in the supplied manuscript.
Questions for authors

How many reconstructions and independent runs contribute to each reported accuracy? Can the authors reconcile the percentages with Eq. (31) and provide confidence intervals?

Can matched adapters trained with and without selected target examples demonstrate that the recovered information depends on those examples, rather than only on improved target recognition?

How is QGQ_G instantiated for StarGAN v2? What source images are used, and how much target information is recoverable from those images before refinement?

Do the improvements persist against a tuned, simpler differential objective and SMI-AW under matched computational budgets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EdmJ (1/3): Differential guidance and computational cost (W1/Q4)

We thank Reviewer EdmJ for the careful review. We have uploaded a revised manuscript with all substantive changes highlighted in blue. Below we address W1 and Q4, which ask how much of VisDelta's gain comes from differential guidance (LΔ\mathcal L_\Delta) rather than the surrounding ranking, transformation, and selection pipeline, why we use the hard-negative differential margin (Δm\Delta m) instead of the simple score difference (gΔ=s1−s0g_\Delta=s_1-s_0), and whether the accuracy gains justify the runtime cost.

1. Isolating differential guidance from the surrounding pipeline

Our central contribution is the adapter-induced gain in target separation, Δm=m1−m0\Delta m=m_1-m_0, while candidate ranking, latent constraints, transformations, and multi-view selection support its optimization. Table 5 (p. 10) and Appendix E.12 (p. 25) in the revised manuscript separate these contributions on CelebA with Qwen2.5-VL-7B. All variants share the same candidate pools (N=512,K=8N=512,K=8), transformations, and selection views (L=10L=10). The table below summarizes the single-run Acc@1/5 results (seed 42), three-seed Acc@1 (mean ± std), evaluator feature distance, and per-target runtime across these controlled variants:

MethodAcc@1 ↑Acc@5 ↑Three-seed Acc@1δeval\delta_{\mathrm{eval}} ↓Time (s/target) ↓
VisDelta w/o LΔ\mathcal L_\Delta (rescaled)21.50%48.50%21.50% ± 0.50%158.92069.1
SMI-AW (T=50T=50)22.00%49.50%22.17% ± 0.76%157.86468.4
SMI-AW (T=100T=100)22.50%50.00%22.50% ± 0.50%156.410114.5
SMI-AW + VisDelta pipeline (w/o LΔ\mathcal L_\Delta)23.00%50.50%22.83% ± 0.76%155.62069.5
VisDelta w/ score difference gΔg_\Delta23.50%51.00%23.33% ± 0.76%154.36084.2
SMI-AW + LΔ\mathcal L_\Delta (matched pipeline)25.00%51.50%24.83% ± 0.76%151.24097.8
VisDelta26.00%52.00%26.00% ± 1.00%149.38598.6

Two controls establish the contribution of LΔ\mathcal L_\Delta:

  • Preserving the total guidance weight: Removing LΔ\mathcal L_\Delta and rescaling (λans,λmar)(\lambda_{\mathrm{ans}},\lambda_{\mathrm{mar}}) to (0.75,0.75)(0.75,0.75) preserves the default coefficient sum of 1.50. Acc@1 rises from 18.50% in the unrescaled ablation (Table 3, p. 9) to 21.50%, whereas full VisDelta reaches 26.00%, a further 4.50 percentage points (pp).
  • Holding the surrounding pipeline fixed: Adding our ranking, transformation, and selection pipeline to SMI-AW raises Acc@1 from 22.00% to 23.00%. Adding LΔ\mathcal L_\Delta within that exact pipeline raises it to 25.00%, and full VisDelta reaches 26.00%, achieving a paired three-seed gain of 3.17 ± 0.29 pp over SMI-AW with the pipeline alone (95% CI: [2.45, 3.88] pp).

Together, these controls show that LΔ\mathcal L_\Delta contributes a clear recovery gain beyond both the surrounding pipeline and the total guidance weight (Table 5 also reports FaceNet distance, FID, CS, and LPIPS).

2. Why compare target separation instead of only the target score?

While the simple score difference gΔ=s1−s0g_\Delta=s_1-s_0 measures how much the adapter increases the target score alone, our differential margin compares this increase against competing conditions (Appendix A.1, Eq. 22, p. 15):

Δm(x,ct;qt)=gΔ(x,ct;qt)−1H∑c−∈HtgΔ(x,c−;qt).\Delta m(x,c_t;q_t)=g_\Delta(x,c_t;q_t)-\frac{1}{H}\sum_{c^-\in\mathcal H_t}g_\Delta(x,c^-;q_t).

Any additive score shift shared by the target and selected hard negatives cancels out, so Δm\Delta m isolates the adapter's improvement in target separation. Replacing Δm\Delta m with gΔg_\Delta under the same default weights (λΔ=1,τ=1\lambda_\Delta=1,\tau=1) lowers Acc@1 from 26.00% to 23.50%, giving Δm\Delta m a paired three-seed advantage of 2.67 ± 0.29 pp (95% CI: [1.95, 3.38] pp; Appendix E.12) and confirming the benefit of calibrating against hard negatives.

3. Recovery gained per computational budget

Because LoRA updates only the language model, the frozen vision encoder and connector are shared across M0M_0 and M1M_1 (Appendix B.1, pp. 17–18). Reusing their visual features for each transformed image yields a runtime of 98.6 s/target for VisDelta versus 68.4 s for SMI-AW (1.44×). Doubling SMI-AW's refinement steps (T=100T=100) requires 114.5 s/target and reaches 22.50% Acc@1, so VisDelta achieves 3.50 pp higher accuracy in less time than this expanded-budget baseline.

AuthorsOpenReview ↗

Response to Reviewer EdmJ (2/3): Dependence on private training data (W2/Q2)

This comment addresses W2 and Q2, which ask whether reconstructions depend on the private fine-tuning data rather than pre-existing identity knowledge in the base VLM or public generator, and how we test the training-data dependence in Eq. (26). In the revised manuscript, Tables 4 and 11 (pp. 10, 25) and Appendix E.11 (p. 24) evaluate synthetic identities, anonymized labels, and counterfactual training-subset controls on Qwen2.5-VL-7B.

1. Recovery with synthetic identities and anonymized labels

We constructed 100 StyleGAN2 identities with random two-word pseudonyms, generating 20 images per identity via anchor-latent perturbations and FaceNet consistency checks (14 adaptation, 3 validation, and 3 held-out images). We also replaced all 530 FaceScrub names with anonymous labels (ID_001–ID_530) in both adaptation and attack prompts. The table below summarizes target recovery accuracy under both settings:

SettingMethodAcc@1 ↑Acc@5 ↑
Synthetic identitiesBase Only1.00%5.00%
Synthetic identitiesSMI-AW22.00%48.00%
Synthetic identitiesVisDelta28.00%55.00%
Anonymized FaceScrubBase Only0.19%0.94%
Anonymized FaceScrubSMI-AW17.92%42.64%
Anonymized FaceScrubVisDelta21.89%46.79%

While Base Only stays near chance (1.00% and 0.19%), VisDelta reaches 28.00% and 21.89% Acc@1 (three-seed Acc@1: 27.67% ± 0.58% and 21.76% ± 0.58% and 21.76%±0.58%21.76\%\pm0.58\%), exceeding SMI-AW by 6.00 and 3.96 percentage points (pp) (Table 4 also includes Final CE Only, CS, and LPIPS).

To compare proximity to training versus unseen images of the same identity without set-size bias, δtrain\delta_{\mathrm{train}} averages nearest-neighbor FaceNet distance over 20 random draws of 3 training images, while δheld\delta_{\mathrm{held}} uses the 3 held-out images. VisDelta reconstructions are closer to the adaptation images than to the held-out images on both synthetic identities (0.812 vs. 0.882) and anonymized FaceScrub (0.797 vs. 0.858, both p<0.0001p<0.0001 via two-sided paired tt-tests across 100 and 530 identities), showing that recovery remains effective under newly assigned labels and aligns more closely with the adaptation images.

2. Switching the training subset while keeping identity fixed

To test the training-data dependence in Eq. (26), we trained matched group-level counterfactual adapters on CelebA. Each identity's 14 adaptation images are split into disjoint seven-image subsets DA\mathcal D_A and DB\mathcal D_B, and adapters M1(A)M_1^{(A)} and M1(B)M_1^{(B)} each train on 1,400 images with identical initialization, batch-order seed, and 10-epoch schedule. The table below reports target accuracy, mean nearest-neighbor FaceNet distance to each seven-image subset, subset membership inference (MIA) AUC, and condition retrieval (Table 11):

ConfigurationAcc@1 ↑Acc@5 ↑Distance to DA\mathcal D_A ↓Distance to DB\mathcal D_B ↓MIA AUC ↑R@1 ↑R@5 ↑
Base model M0M_013.00%35.00%1.1041.10650.00%18.50%41.00%
Single score s1s_1 on M1(A)M_1^{(A)}20.50%46.50%0.9160.94254.20%51.50%78.00%
Identity-excluded adapter12.50%34.50%1.1081.11050.40%19.00%41.50%
VisDelta on M1(A)M_1^{(A)}25.50%51.50%0.8380.91468.40%64.00%86.50%
VisDelta on M1(B)M_1^{(B)}25.00%51.00%0.9190.84367.90%63.50%86.00%

Switching the training subset reverses which subset the reconstruction is closer to: training on DA\mathcal D_A gives 0.838 vs. 0.914, whereas training on DB\mathcal D_B gives 0.843 vs. 0.919 in favor of DB\mathcal D_B (a 0.076 gap in both directions, p<0.0005p<0.0005 across 200 identities), demonstrating training-subset-specific alignment within fixed identities.

3. Identity exclusion and behavioral evidence

We also trained ten adapters that each exclude 20 target identities and replace their 280 images with non-target images to preserve the 2,800-image budget. Evaluating each excluded identity on its corresponding adapter reduces Acc@1 from 26.00% (Table 1, p. 7) to 12.50%, close to Base Only (13.00%). Using Δm\Delta m, subset MIA distinguishes 1,400 member images from 1,400 same-identity nonmembers with 68.40%/67.90% AUC (versus 50.40% near chance on excluded adapters), and condition retrieval reaches 64.00%/63.50% R@1. Together, these controls show that VisDelta recovers recognizable target appearances and training-subset visual cues learned from the private fine-tuning data.

AuthorsOpenReview ↗

Response to Reviewer EdmJ (3/3): Evaluation protocol and manuscript corrections (W3–W5, Q1/Q3)

This comment addresses W3 and Q1 (clarifying the target-level accuracy denominator in Eq. 31 and reporting multi-seed uncertainty), W4 and Q3 (clarifying the unconditional dog generator and Algorithm 1 notation), and W5 (fixing the four incomplete sentences and figure/table references).

1. Target-level accuracy and reproducible uncertainty estimates (W3/Q1)

Thank you for pointing out the mismatch between the original percentages and Eq. (31), which we have corrected throughout the revised manuscript. The original script classified multiple transformed views per selected reconstruction and pooled predictions over valid views (skipping failed face detections), resulting in a view-level denominator.

Retaining the same selected reconstructions and frozen evaluator, we recomputed Acc@1/5 using one prediction per final untransformed reconstruction, matching Eq. (31). Denominators are fixed at 530 identities for FaceScrub, 200 for CelebA, and 120 breeds for Stanford Dogs, with a 90% central-crop fallback (224×224224\times224) when face detection fails so no target is skipped (Section 5.1, p. 7; Appendix C, pp. 18–19). For seed 42, VisDelta's Top-1/Top-5 counts are 131/530 and 270/530 on FaceScrub (24.72%/50.94%) and 52/200 and 104/200 on CelebA (26.00%/52.00%). We updated Tables 1–3 (pp. 7–9), Tables 7–9 (pp. 22–23), and Figures 5, 6, 8–10 to report seed 42 under this target-level protocol.

To quantify run-to-run variation, we ran paired seeds 42, 43, 44 on Qwen2.5-VL-7B, sharing initial latent codes and transformation draws across methods within each seed. The table below summarizes the multi-seed mean ±\pm standard deviation from Table 10 (p. 23):

DatasetMethodAcc@1 ↑Acc@5 ↑
FaceScrubBase Only11.38% ± 0.29%33.96% ± 0.50%
FaceScrubFinal CE Only19.18% ± 0.76%44.59% ± 0.97%
FaceScrubSMI-AW21.45% ± 0.58%48.30% ± 0.86%
FaceScrubVisDelta24.97% ± 0.61%51.07% ± 0.76%
CelebABase Only13.00% ± 0.50%35.17% ± 0.76%
CelebAFinal CE Only20.50% ± 1.00%46.33% ± 1.26%
CelebASMI-AW22.17% ± 0.76%49.83% ± 1.04%
CelebAVisDelta26.00% ± 1.00%52.17% ± 0.76%

VisDelta's paired Acc@1 gains over SMI-AW are 3.52±0.113.52\pm0.11 pp on FaceScrub (95% CI: [3.25, 3.79] pp) and 3.83±0.293.83\pm0.29 pp on CelebA (95% CI: [3.12, 4.55] pp; two-sided Student's tt, df=2df=2). Separately, 10,000 paired bootstrap resamples of seed-42 targets give identity-level Acc@1 CIs of [21.13%, 28.49%] and [20.00%, 32.50%] (see Appendix E.10, pp. 22–23, for the Top-1 hit counts of VisDelta and SMI-AW by seed). Both paired seed intervals exclude zero, confirming consistent gains under the corrected target-level evaluation.

2. Dog generation uses latent codes only (W4/Q3)

The dog experiments use unconditional StyleGAN2 pretrained on AFHQ-Dog, as clarified in Section 5.1 (p. 7) and Appendix B.2 (p. 18), citing Karras et al. (2020) for StyleGAN2 and Choi et al. (2020) for AFHQ. Candidate generation follows

zn∼N(0,I),wn=Gmap(zn),xn=Gsyn(wn).z_n\sim\mathcal N(0,I),\qquad w_n=G_{\mathrm{map}}(z_n),\qquad x_n=G_{\mathrm{syn}}(w_n).

No source image or reference dog photo is used, and we removed the unused symbols QG\mathcal Q_G and ξn\xi_n from Algorithm 1 (p. 16) so the threat model, text, and pseudocode consistently specify generation from sampled latent codes.

3. Manuscript corrections (W5)

We corrected all four locations in the revised manuscript:

  • p. 3 & Appendix D (p. 19): Completed the sentences describing selection across freshly sampled transformations and introducing Algorithm 1's three-stage pipeline.
  • Appendix A.3 (p. 16) & Figure 11 (p. 26): Added the explicit reference to Table 11 (p. 25) and aligned Figure 11's labeling and caption with CelebA.
AuthorsOpenReview ↗

Follow-up on our response

We appreciate the care you put into your review and the constructive feedback you shared. We have posted our response and uploaded the revised manuscript, with changes highlighted in blue. As the discussion closes later today, we would welcome your thoughts on whether our revisions address your concerns. We would be glad to answer any further questions. Thank you again for your time.

Reviewer_YhLQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VisDelta is a model inversion attack against a released LoRA adapter for a VLM. Given white-box access to the base model M0M_0, the adapter (hence M1M_1), the full condition set, the query template and a public generator, it samples N=512N = 512 latents, mines H=4H = 4 hard-negative conditions on M1M_1 (eq. 13), ranks candidates by a guidance loss and keeps K=8K = 8, refines each with T=50T = 50 projected-Adam steps under random differentiable augmentations inside an ℓ2\ell_2 ball (eqs. 14-16), and selects by a mean/worst-view combination over L=10L = 10 fresh views (eqs. 17-21). The guidance JguideJ_{\text{guide}} (eq. 11) combines target likelihood under M1M_1, a tanh-squashed target margin under M1M_1, and the new term LΔ\mathcal{L}_{\Delta} (eq. 10): the tanh-squashed differential margin Δm=m1−m0\Delta m = m_1 - m_0 (eq. 7), the increase in target separation induced by the adapter.

Validation on Qwen2.5-VL-7B, LLaVA-1.6-7B, MiniCPM-V-2.6 and InternVL3-8B with FaceScrub, 200 CelebA identities and 120 Stanford Dogs breeds. On CelebA/Qwen2.5-VL, Acc@1/5 of 26.38/52.47 vs 22.57/50.18 for SMI-AW, the single external baseline (Table 1). Ablations (Table 3) show the differential term is worth 7.35 pp and that wrong-adapter and permuted-condition controls fall to 6-7%; Fig. 6 shows accuracy rising with LoRA scale. The differential-margin idea is sound and the controls are convincing that the adapter contributes signal; the claims are weakened because no variance is reported for 2-4 pp gains on a stochastic attack, and because the "private images" framing is tested only on public figures and generic classes, where the base-only pipeline already recovers some targets.

Critical strengths
  • Originality: The differential margin is a clean, reusable idea. Defining Δm=m1−m0\Delta m = m_1 - m_0 (eqs. 6-7) and showing (lines 230-233) that it equals the target's response gain minus the mean hard-negative gain discounts both the base model's prior confidence and adapter effects shared across conditions. Table 3 supports it: removing LΔ\mathcal{L}_{\Delta} costs 7.35 pp Acc@1 (26.38 to 19.03), the wrong-adapter control collapses to 6.24% and the permuted-condition control to 7.36%, and Fig. 6 shows accuracy increasing monotonically with adapter scale. Together these establish that the recovered signal is attributable to the adapter.
  • Clarity: The threat model (Section 3, lines 181-187) is explicit about what the attacker has and lacks, the pipeline is specified with every budget (NN, KK, TT, HH, LL), and all methods are stated to share splits, queries, candidate budgets and evaluation (lines 351-352).
Critical weaknesses
  • Soundness: No measure of variation anywhere, and the reported precision is inconsistent with the stated protocol. Tables 1-3 report more than thirty quantities to two decimals with no seeds, standard deviations, confidence intervals or number of runs. The attack is stochastic (512 random latents, 8 refined candidates, random augmentations per step and per selection view), and the headline advantage over the only external baseline is 3.81/2.29 pp on CelebA and 3.42/2.63 pp on FaceScrub (Table 1). Moreover, with 200 CelebA identities (line 346), one reconstruction per target (eq. 21) and Acc@K defined as a binary indicator averaged over targets (Appendix C, lines 899-908), accuracy can only move in 0.5-point steps, yet Table 1 reports 26.38% and 52.47%. Either multiple attack runs were averaged without being reported, or the target count or aggregation differs from the text. Until this is explained, none of the point differences in Tables 1-3 can be interpreted.
  • Significance: The privacy claim is broader than the experiments. The abstract and contribution 1 speak of "private images" and "private visual information," but the identities are FaceScrub and CelebA public figures and the classes are Stanford Dogs breeds. Table 1 shows the Base Only pipeline reaches 13.29% Acc@1 on CelebA and 11.63% on FaceScrub (no Base Only result for Stanford Dogs), so the base model with the public FFHQ prior already recovers some targets before any adapter is loaded; which individual identities the base model knows is not measured. The image-level metrics are paired comparisons with one held-out reference image per target (Appendix B lines 859-863; Appendix C lines 899-940), not with any training image, so they measure similarity to the identity, which a class prototype also achieves. The adaptation images are withheld from the attacker (Appendix B.1, lines 859-863) and the ethics statement says the public benchmarks simulate protected fine-tuning data (lines 502-505), so the simulated threat is legitimate; what the experiments do not establish is sample-level leakage (similarity to any specific withheld training image) or generalisation to identities the base model does not know. The supported claim is that the adapter increases the attacker's ability to synthesise a recognisable image of an identity it was trained on; "private visual information" should be narrowed to that.
Questions for authors

Questions that could change my assessment

  1. How many targets were attacked per dataset, how many independent attack runs per target, and how were Tables 1-3 aggregated, given that 200 targets with one reconstruction each cannot produce 26.38%? Please then report the key rows with at least three paired attack seeds (same latents and augmentations across methods) as mean ±\pm std, and separately a bootstrap interval over identities. Score-change criterion: if the protocol is clarified and a paired confidence interval for the VisDelta-minus-SMI-AW difference across attack seeds excludes zero on both face datasets (with identity-level uncertainty reported separately), Soundness rises to 3 and I would raise the Rating to 3; if the paired interval includes zero on either dataset or the protocol cannot be reconciled with the tables, Soundness falls to 1.
  2. Can you run one experiment on identities the base VLM does not know (e.g. a held-out set of non-celebrity faces, or synthetic identities generated from the same StyleGAN2 and used to fine-tune the adapter) and report a nearest-neighbour distance to the specific private training images, with a same-identity held-out image as control? Score-change criterion: recovery of visual content for base-unknown identities with training-image similarity above the held-out-image control would substantiate the privacy claim and raise Informativeness to 3 and the Rating to 3; failure would require narrowing the claim to identity-level leakage for known identities, with the Rating unchanged.

Clarifications and minor points (do not affect my assessment)

  1. Table 3 shows VisDelta without LΔ\mathcal{L}_{\Delta} (19.03%) below SMI-AW (22.57%) and Final CE Only (20.84%); add a PPA- or PLG-MI-style baseline with the VLM as scorer under the same budgets, and present the contribution as the differential loss term if that baseline plus LΔ\mathcal{L}_{\Delta} matches VisDelta.
  2. The threat model assumes the attacker knows the complete condition set C\mathcal{C} and the query template; discuss how C\mathcal{C} is obtained for a released adapter whose task is not documented.
  3. Hard negatives are mined on M1M_1 only (eq. 13); a sentence on why not on Δm\Delta m would help.
  4. SMI-AW's FID is better on both datasets (Table 1); the abstract's framing omits it.
  5. A short mitigation discussion (DP-LoRA, adapter noise, reducing α/r\alpha/r as Fig. 6 suggests) would strengthen the security framing.
  6. The base-vs-adapted contrast is the same signal LoRA-Leak uses for membership inference; acknowledge the lineage.
  7. The embedded figure asset on p. 22 (Fig. 11) is labelled "Facescrub" while its caption says CelebA.
4 comments
AuthorsOpenReview ↗

Response to Reviewer YhLQ (1/3): Evaluation accuracy and statistical reliability (W1/Q1)

We thank Reviewer YhLQ for the careful review and constructive suggestions. We have uploaded a revised manuscript with all substantive changes highlighted in blue. Below we address W1 and Q1, which ask why the original accuracy percentages did not match integer ratios over the target sets, whether Table 1 reports a single run or a multi-seed mean, and whether VisDelta's gains are statistically significant across attack seeds.

1) Unifying evaluation to target-level accuracy

Thank you for catching the mismatch between our original evaluation script and Eq. (31). We apologize for reporting view-level accuracy and have corrected the affected results throughout the revised manuscript. Previously, the script classified multiple transformed views of each selected reconstruction and pooled predictions over valid views (skipping failed face detections), resulting in a view-level denominator.

Retaining the same selected reconstructions and frozen evaluator, we recomputed Acc@1/5 using one final untransformed reconstruction per target, matching Eq. (31). Denominators are fixed at 530 identities for FaceScrub, 200 for CelebA, and 120 breeds for Stanford Dogs, with a 90% central-crop fallback (224×224224\times224) for face-detection failures so no target is excluded. While the attack's internal multi-view selection remains unchanged, evaluation now strictly follows Eq. (31) across Tables 1–3 (pp. 7–9), Tables 7–9 (pp. 22–23), and Figures 5, 6, 8–10 (pp. 8–9, 20–21) (Figure 7 shows optimization losses and is unaffected).

2) Evaluator protocol and updated main results

Appendix C (pp. 18–19) specifies the independent Inception-v3 evaluator: 14 training and 3 validation images per target (excluding the 3 held-out reference images), attaining validation Top-1/5 of 96.42%/99.25% on FaceScrub (1,590 images), 94.83%/98.67% on CelebA (600), and 86.39%/97.22% on Stanford Dogs (360).

Table 1 (p. 7) reports single-run results for seed 42 (while three-seed statistics are reported separately in Table 10). VisDelta achieves 24.72%/50.94% Acc@1/5 on FaceScrub (131/530 and 270/530 integer hits) versus SMI-AW's 21.32%/48.11%, and 26.00%/52.00% on CelebA (52/200 and 104/200 hits) versus 22.00%/49.50%, yielding gains of 3.40/2.83 and 4.00/2.50 percentage points (pp), respectively.

3) Paired attack seeds and separate identity-level uncertainty

We evaluated three independent attack seeds (42, 43, 44) on Qwen2.5-VL-7B, sharing the same 512 initial latent codes and transformation draws per target within each seed. The table below summarizes the per-seed Top-1 hit counts, cross-seed mean ± sample std, and identity-level 95% CIs (from 10,000 paired bootstrap resamples of seed-42 targets, which quantify target-sampling uncertainty separately from seed variation):

DatasetMethodTop-1 hits (42, 43, 44)Acc@1 (mean ± std)Acc@5 (mean ± std)Identity 95% CI (Acc@1)
FaceScrubBase Only60, 62, 5911.38% ± 0.29%33.96% ± 0.50%[8.68%, 14.15%]
FaceScrubFinal CE Only101, 106, 9819.18% ± 0.76%44.59% ± 0.97%[15.85%, 22.64%]
FaceScrubSMI-AW113, 117, 11121.45% ± 0.58%48.30% ± 0.86%[17.92%, 25.09%]
FaceScrubVisDelta131, 136, 13024.97% ± 0.61%51.07% ± 0.76%[21.13%, 28.49%]
CelebABase Only26, 27, 2513.00% ± 0.50%35.17% ± 0.76%[8.50%, 18.00%]
CelebAFinal CE Only41, 43, 3920.50% ± 1.00%46.33% ± 1.26%[15.00%, 26.50%]
CelebASMI-AW44, 46, 4322.17% ± 0.76%49.83% ± 1.04%[16.50%, 28.00%]
CelebAVisDelta52, 54, 5026.00% ± 1.00%52.17% ± 0.76%[20.00%, 32.50%]

For the paired seed-level Acc@1 difference (VisDelta minus SMI-AW), the mean ± std is +3.52 ± 0.11 pp on FaceScrub (two-sided Student's tt 95% CI, df=2df=2: [+3.25, +3.79] pp) and +3.83 ± 0.29 pp on CelebA (95% CI: [+3.12, +4.55] pp; see Table 10, p. 23, and Appendix E.10, pp. 22–23). Under the corrected target-level evaluation, VisDelta outperforms SMI-AW in Acc@1 in all three paired runs on each dataset, and both 95% confidence intervals for the mean gain lie entirely above zero.

AuthorsOpenReview ↗

Response to Reviewer YhLQ (2/3): Evidence of recovery from private fine-tuning data (W2/Q2)

This comment addresses W2 and Q2, which ask whether VisDelta recovers visual information from the private fine-tuning images rather than relying on celebrity associations already present in the base VLM, and how our claims in the text align with this evidence. In the revised manuscript, we added three experiments with Qwen2.5-VL-7B (Tables 4 and 11, pp. 10, 25; Appendix E.11, p. 24) to test dependence on the private adaptation data beyond celebrity-name associations.

1) Synthetic identities with random pseudonyms

We constructed 100 identities using the public StyleGAN2-FFHQ generator by perturbing distinct anchor latents and retaining 20 images per identity via FaceNet filtering and visual checks (14 adaptation, 3 validation, and 3 held-out images), paired with random two-word pseudonyms. The independent 100-way evaluator attains 97.00%/99.67% validation Top-1/5 (Appendix E.11, p. 24).

2) Anonymized labels for real identities

We replaced all 530 FaceScrub names with ID_001–ID_530 in both adaptation and attack prompts to test recovery without supplying original celebrity names.

To compare training and held-out images fairly with equal reference-set sizes, δtrain\delta_{\mathrm{train}} averages nearest-neighbor FaceNet distances over 20 draws of 3 training images per identity, while δheld\delta_{\mathrm{held}} uses the 3 held-out images. The table below summarizes the results from Table 4 (p. 10):

DatasetMethodAcc@1 ↑Acc@5 ↑δtrain\delta_{\mathrm{train}} ↓δheld\delta_{\mathrm{held}} ↓Δδface\Delta\delta_{\mathrm{face}}
Synthetic IDsBase Only1.00%5.00%1.1921.194+0.002
Synthetic IDsSMI-AW22.00%48.00%0.8720.911+0.039
Synthetic IDsVisDelta28.00%55.00%0.8120.882+0.070
Anonymized FaceScrubBase Only0.19%0.94%1.2051.204−0.001
Anonymized FaceScrubSMI-AW17.92%42.64%0.8850.918+0.033
Anonymized FaceScrubVisDelta21.89%46.79%0.7970.858+0.061

Here Δδface=δheld−δtrain>0\Delta\delta_{\mathrm{face}}=\delta_{\mathrm{held}}-\delta_{\mathrm{train}}>0 indicates closer alignment with the adaptation images. While Base Only stays at chance, VisDelta exceeds SMI-AW by 6.00 and 3.96 pp in Acc@1 (three-seed Acc@1: 27.67% ± 0.58% and 21.76% ± 0.58%). Its reconstructions are significantly closer to training than to held-out images in both settings (two-sided paired tt-tests across n=100n=100 and 530530; p<0.0001p<0.0001; Table 4 also reports Final CE Only, CS, and LPIPS).

3) Changing the training subset within the same identity

For each of 200 CelebA identities, we split its 14 adaptation images into disjoint 7-image subsets DA\mathcal D_A and DB\mathcal D_B and trained matched adapters M1(A)M_1^{(A)} and M1(B)M_1^{(B)}. The table below summarizes target accuracy, mean nearest-neighbor FaceNet distance to each subset, subset MIA AUC, and condition retrieval from Table 11 (p. 25):

ConfigurationAcc@1 ↑Distance to DA\mathcal D_A ↓Distance to DB\mathcal D_B ↓Subset MIA AUC ↑R@1 ↑
Base model M0M_0 (s0s_0)13.00%1.1041.10650.00%18.50%
Single adapted score (s1s_1 on M1(A)M_1^{(A)})20.50%0.9160.94254.20%51.50%
Identity-excluded adapter Δθ−\Delta\theta^{-}12.50%1.1081.11050.40%19.00%
VisDelta on M1(A)M_1^{(A)}25.50%0.8380.91468.40%64.00%
VisDelta on M1(B)M_1^{(B)}25.00%0.9190.84367.90%63.50%

Reconstruction alignment follows the training subset: training on A gives 0.838 to A vs. 0.914 to B, whereas training on B reverses this to 0.843 to B vs. 0.919 to A (paired tt-tests across 200 identities, p<0.0005p<0.0005 in both directions). Using Δm\Delta m, subset MIA separates 1,400 training images from 1,400 same-identity held-out-subset images at 68.40%/67.90% AUC, and condition retrieval reaches 64.00%/63.50% R@1. Excluding 20 target identities per adapter (10 adapters, 2,800 training images each) reduces Acc@1 from 26.00% (Table 1) to 12.50%. With neither subset used for training, the control yields a near-chance AUC of 50.40%.

On the 100 synthetic identities with random pseudonyms (Table 4), VisDelta achieves 28.00% Acc@1 versus 1.00% for Base Only. Its reconstructions are significantly closer to the private training images than to an equal number of held-out images of the same identity (mean nearest-neighbor FaceNet distance: 0.812 vs. 0.882, paired p<0.0001p<0.0001). Together, these results show that released adapters reveal both recognizable target appearances and visual cues specific to the training subset, as reflected in the revised Abstract, Introduction (pp. 1–2), and Conclusion (p. 10).

AuthorsOpenReview ↗

Response to Reviewer YhLQ (3/3): Additional comparisons and clarifications (Minor 1–7)

This comment addresses Minor 1–7, covering controlled ablations of LΔ\mathcal L_\Delta and baseline scope (Minor 1), condition-set assumptions (Minor 2), mining hard negatives under M1M_1 (Minor 3), the FID trade-off (Minor 4), defense evaluations (Minor 5), and crediting LoRA-Leak (Minor 6).

1) Differential guidance and controlled baselines (Minor 1)

In the original ablation (Table 3, p. 9), removing LΔ\mathcal L_{\Delta} reduced the remaining guidance-weight sum from 1.50 to 0.50 relative to λprior=0.02\lambda_{\mathrm{prior}}=0.02. Adding a rescaled control (λans,λmar)=(0.75,0.75)(\lambda_{\mathrm{ans}},\lambda_{\mathrm{mar}})=(0.75,0.75) raises Acc@1 from 18.50% to 21.50% (close to SMI-AW's 22.00%), while full VisDelta retains a 4.50 pp advantage (26.00%).

We also isolated LΔ\mathcal L_{\Delta} within a matched ranking, transformation, and selection pipeline on CelebA with Qwen2.5-VL-7B. The table below summarizes the key variants from Table 5 (p. 10) and Appendix E.12 (p. 25):

ConfigurationAcc@1 (seed 42)Acc@1 (3-seed mean ± std)
VisDelta without LΔ\mathcal L_{\Delta}, rescaled weights21.50%21.50% ± 0.50%
SMI-AW + VisDelta pipeline, without LΔ\mathcal L_{\Delta}23.00%22.83% ± 0.76%
SMI-AW + LΔ\mathcal L_{\Delta}, matched pipeline25.00%24.83% ± 0.76%
VisDelta26.00%26.00% ± 1.00%

Adding LΔ\mathcal L_{\Delta} improves matched-pipeline SMI-AW from 23.00% to 25.00%, and full VisDelta gains a further 1.17 ± 0.29 pp across paired seeds (95% CI: [0.45, 1.88] pp; Table 5 also reports doubled-step SMI-AW and gΔ=s1−s0g_{\Delta}=s_1-s_0). Because PPA and PLG-MI target closed-set classifier logits rather than autoregressive VLM token sequences, we focused on controlled comparisons against the VLM-specific baseline SMI-AW under matched candidate pools, transformations, and selection settings.

2) Knowledge of the condition set (Minor 2)

Section 3 (p. 4) explicitly states that our evaluation assumes access to the complete condition set C\mathcal C. When task documentation is unavailable, public probe queries may help construct a candidate vocabulary, for example by identifying tokens with positive logit shifts from M0M_0 to M1M_1.

3) Why mine negatives under M1M_1? (Minor 3)

Section 4.3 (p. 6) clarifies that ht(c)h_t(c) selects conditions that actively compete with the target under the adapted model M1M_1. A large s1−s0s_1-s_0 can instead reflect a rise from a very low base score while the adapted score remains far below the target, so mining under M1M_1 identifies relevant competitors for Δm\Delta m.

4) Reporting the FID trade-off (Minor 4)

The Abstract (p. 1) and Section 5.2 (p. 7) now explicitly report SMI-AW's lower FID (109.86 vs. 112.31 on FaceScrub and 109.74 vs. 111.92 on CelebA), while showing that VisDelta achieves the best Acc@1, Acc@5, δface\delta_{\mathrm{face}}, δeval\delta_{\mathrm{eval}}, and CS on both datasets (Table 1, p. 7).

5) Mitigations and the utility trade-off (Minor 5)

In Table 6 (p. 10) and Appendix E.13 (p. 25), we evaluate three mitigations on CelebA with Qwen2.5-VL-7B (Task Acc. on 600 validation images; attack metrics on 200 targets; arrows indicate defender-preferred direction). The table below summarizes the results:

ConfigurationTask Acc. ↑Acc@1 ↓Acc@5 ↓δface\delta_{\mathrm{face}} ↑FID ↑LPIPS ↑
No defense (γ=1.0\gamma=1.0)91.50%26.00%52.00%0.886111.920.711
Reduced scale (γ=0.5\gamma=0.5)86.50%20.50%44.50%0.976118.450.734
Weight noise (σ=0.02\sigma=0.02)82.50%17.50%41.00%1.028124.800.749
DP-LoRA84.50%15.00%37.50%1.074128.600.758

DP-LoRA (εDP≈8\varepsilon_{\mathrm{DP}}\approx8, δ=10−5\delta=10^{-5}) reduces Acc@1 from 26.00% to 15.00% with a 7.00 pp utility drop, achieving the lowest attack accuracy among tested defenses while retaining higher task utility than weight noise (84.50% vs. 82.50%).

6) Credit to LoRA-Leak and correction to Figure 11 (Minor 6–7)

Section 2 (p. 3) now explicitly credits Ran et al. (2025) (LoRA-Leak) for using the pretrained base model as a reference to calibrate adaptation leakage in membership inference. We also corrected the dataset label in Figure 11 (p. 26) to match the CelebA caption.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your careful and constructive review. We have posted our response and uploaded the revised manuscript, with changes highlighted in blue. As the discussion closes later today, we would appreciate hearing whether our response addresses your concerns. We would be happy to clarify anything that remains unclear. Thank you again for your time and effort.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the Area Chair and Reviewers: Summary of Revisions and Additional Experiments

We thank the Area Chair and reviewers for their time and thoughtful feedback. Below, we summarize our contributions, the main questions raised by the reviewers, and how we addressed them. We have uploaded the revised PDF, with substantive changes highlighted in blue. We list these changes and their locations at the end of this comment.

Summary of contributions. We study model inversion against released VLM LoRA adapters without access to private fine-tuning images, and propose VisDelta, which compares paired responses of the public base VLM M0M_0 and LoRA-adapted VLM M1M_1 on the same generated image and query to guide latent-space reconstruction across four VLMs and three benchmarks.

1) Target-level evaluation and multi-seed statistical evidence (EdmJ W3/Q1, uyVh Q3, YhLQ W1/Q1)

We aligned all Acc@1/5 results with Eq. (31): one final untransformed reconstruction per target, with fixed denominators of 530 (FaceScrub), 200 (CelebA), and 120 (Stanford Dogs), using a 90% central-crop fallback for failed face detections so every target is counted. Using the same saved reconstructions and frozen evaluator, we updated Tables 1–3 and 7–9 and Figures 5, 6, and 8–10, and documented the complete evaluator protocol in Appendix C (pp. 18–19).

The table below summarizes the three paired attack seeds (42, 43, 44) from Table 10 (p. 23), where methods share initial latent codes and transformations within each seed (mean ± std; 95% CIs via paired Student's tt distribution, df=2df=2):

DatasetVisDelta Acc@1SMI-AW Acc@1Gain (pp; 95% CI)
FaceScrub24.97% ± 0.61%21.45% ± 0.58%+3.52 [+3.25, +3.79]
CelebA26.00% ± 1.00%22.17% ± 0.76%+3.83 [+3.12, +4.55]

Both paired intervals exclude zero, confirming consistent recovery gains across attack seeds under the target-level protocol (Table 10 and Appendix E.10, pp. 22–23 also report 10,000-resample identity bootstrap CIs).

2) Dependence on private adaptation data (EdmJ W2/Q2, uyVh W1/Q1, YhLQ W2/Q2)

In Table 4 (p. 10), on 100 synthetic StyleGAN2 identities with random pseudonyms, VisDelta reaches 28.00% Acc@1 (vs. 1.00% Base Only and 22.00% SMI-AW); with anonymized FaceScrub labels (ID_001–ID_530), VisDelta reaches 21.89% (vs. 0.19% and 17.92%). Using size-matched 3-image reference sets, reconstructions are closer to actual adaptation images than to held-out images of the same identity, with FaceNet distance gaps of 0.070 and 0.061 (both p<0.0001p<0.0001). Splitting CelebA targets by distance to 10,000 generated StyleGAN2-FFHQ samples yields a 4.00 pp gain over SMI-AW in both near and far groups.

In Table 11 (p. 25; Appendix E.11, p. 24), training matched adapters on disjoint 7-image subsets DA\mathcal D_A and DB\mathcal D_B per identity reverses reconstruction alignment toward the trained subset (0.838 vs. 0.914 on M1(A)M_1^{(A)}; 0.843 vs. 0.919 on M1(B)M_1^{(B)}; p<0.0005p<0.0005), while excluding target identities reduces Acc@1 from 26.00% to 12.50%, showing that recovery captures recognizable target appearances and training-subset visual cues from private adaptation data.

3) Controlled isolation of differential guidance (EdmJ W1/Q4, uyVh W2/Q2, YhLQ Minor 1)

SMI-AW optimizes latent codes using only scores from the adapted VLM M1M_1. In Table 5 (p. 10; Appendix E.12, p. 25), holding the surrounding pipeline fixed and inserting LΔ\mathcal L_\Delta into SMI-AW raises Acc@1 from 23.00% to 25.00% (+2.00 pp in each seed), and full VisDelta reaches 26.00%. Rescaling VisDelta without LΔ\mathcal L_\Delta to keep the default total guidance weight (1.50) gives 21.50% (+4.50 pp for LΔ\mathcal L_\Delta), and our hard-negative margin Δm\Delta m outperforms simple score difference gΔ=s1−s0g_\Delta=s_1-s_0 by +2.67 pp (95% CI: [1.95, 3.38] pp). Doubling SMI-AW to 100 steps yields 22.50% at 114.5 s/target, whereas VisDelta reaches 26.00% at 98.6 s, confirming a clear gain from differential guidance.

4) Mitigations and clarifications (YhLQ Minor 2–6, EdmJ W4–W5/Q3)

In Table 6 (p. 10; Appendix E.13, p. 25), DP-LoRA (εDP≈8,δ=10−5\varepsilon_{\mathrm{DP}}\approx8, \delta=10^{-5}) lowers Acc@1 from 26.00% to 15.00% while retaining 84.50% task accuracy (vs. 91.50% without defense).

Blue-highlighted revisions by location:

  • pp. 1–2, 10 (Abstract, Intro, Conclusion): Target-level metrics, FID trade-off, and target/subset recovery wording.
  • pp. 3–4, 6 (Sec. 2–4.3): LoRA-Leak credit, selection sentence, condition-set assumption, and M1M_1 negative mining.
  • pp. 7–10 (Sec. 5.1–5.4, Tables 1–6, Figs. 5–6): Target-level results, AFHQ-Dog StyleGAN2, and new Tables 4–6.
  • pp. 16, 18–26 (Alg. 1, App. A.3–E.13, Tables 7–11, Figs. 8–11): Evaluator protocol, multi-seed Table 10, counterfactual Table 11, and text/caption fixes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.