- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TGAP, a lightweight residual adapter applied to post-resampler visual tokens before transmission in a split VLM pipeline. With the source VLM frozen, the adapter is trained to preserve utility, suppress correct private-attribute answers, and increase the uncertainty of an auxiliary attribute predictor, while limiting changes to the original tokens. Evaluation on a paired privacy–utility benchmark shows improved trade-offs over hard token masking and attention masking in several model settings. Additional experiments assess attribute recovery by separately trained attackers with frozen LLM backbones.
Critical strengths
The proposed post-resampler token modulation offers a lightweight way to reduce measured privacy leakage while preserving utility-task performance in several evaluated settings. The paper evaluates both semantic and representation leakage. Compared with hard masking and attention masking, TGAP achieves better privacy–utility trade-offs in several evaluated settings.
Critical weaknesses
1. Limited methodological novelty
TGAP combines utility preservation, private-label suppression, entropy maximization, and representation manipulation. These ideas have substantial precedents: MaxEnt-ARL uses entropy-based privacy suppression, STPrivacy modifies visual token embeddings, and Shredder learns perturbations while freezing the pretrained network. The authors should clarify the technical novelty.
References:
- Roy and Boddeti. Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach. CVPR 2019. Paper · Code
- Li et al. STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognition. ICCV 2023. Paper · Code
- Mireshghallah et al. Shredder: Learning Noise Distributions to Protect Inference Privacy. ASPLOS 2020. Paper
2. Insufficient baseline comparisons
The main comparisons use hard masking and attention masking. Neither learns a transformation under comparable supervision. Stronger comparisons should adapt learned representation defenses and learned-noise methods to the same interface, with matched training budgets and attacker protocols.
Appendix E.7 provides additional controls, but does not establish a matched comparison with learned token-space defenses.
Relevant baselines:
- MaxEnt-ARL. Paper · Code
- STPrivacy. Paper · Code
- Mireshghallah et al. Shredder: Learning Noise Distributions to Protect Inference Privacy. ASPLOS 2020 paper
- Author-released code, corresponding to the 2019 self-supervised v2 paper, rather than the ASPLOS version.
3. Limited wearable validity
The benchmark consists of single images from Reddit and VQA datasets, which do not adequately capture continuous, first-person wearable use. Consequently, the evaluation leaves temporal privacy accumulation untested. Although the paper acknowledges this proxy setting, a fixed, reproducible evaluation set and experiments on real wearable sequences are needed to substantiate claims about wearable deployment.
4. Restricted attacker adaptation
The adaptive attacks freeze the LLM backbone and train only the bridge and prediction head, limiting the attacker’s adaptation capacity. The authors should also evaluate attacks that adapt the backbone through LoRA to assess whether the privacy gains persist against a stronger attacker.
Reference:
5. Inconsistent statistics in Table 4
Table 4 reports mean ± standard deviation over three seeds, but several entries are mathematically incompatible with three nonnegative scores. For three such scores, the sample standard deviation cannot exceed sqrt(3) times the mean. Phi-3.5-V TGAP P.Cont is 0.002 ± 0.012, which violates this bound even after accounting for rounding. Other violations include Phi-3.5-V TGAP P.Acc (0.061 ± 0.134), P.Rel (0.006 ± 0.015), and P.WR (0.003 ± 0.013), as well as InternVL2.5 TGAP P.Cont (0.013 ± 0.036). The authors should release the per-seed scores and aggregation code, explain these discrepancies, and correct the table.
Questions for authors
Please refer to the weakness.