- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies visual-context-dependent privacy in VLM geolocation responses. It separates the problem into deciding an appropriate disclosure granularity and steering generation toward that granularity. The proposed router projects a pre-generation hidden state onto a supervised ordinal direction and uses two fitted thresholds to select one of three disclosure levels: A, no country/city detail; B, country or city detail; and C, exact location detail. Depending on the route, the method adds a disclosure-reducing vector, applies no intervention, or adds a disclosure-increasing vector during decoding. Experiments across four VLM backbones report improved granularity alignment relative to vanilla inference, policy prompting, and static activation steering, with mixed results relative to LoRA fine-tuning.
Critical strengths
-
Separating contextual policy selection from behavioral realization is a sensible design. The oracle-routing results show that knowing the appropriate disclosure level does not guarantee that a model will generate at that level, supporting the need to study routing and generation control separately.
-
The paper includes useful diagnostic experiments. The comparisons with prompt-based routing, oracle routing, and prompt-based steering help localize errors.
-
The empirical evaluation covers several heterogeneous backbones.
Critical weaknesses
- Although no backbone weights are updated, the method relies on substantial supervision: human disclosure labels from construction examples define the routing centroids and direction and ordered thresholds are fitted from labeled construction examples. The routing layer is selected separately for each backbone using cross-validation, and the steering layer and strengths are selected using labeled validation examples. Separate components and hyperparameters are required for every backbone. Thus, this is a supervised representation-fitting and hyperparameter-selection pipeline that happens not to update backbone weights. Its annotation, validation, model-access, generation, and judge-call costs should be disclosed.
The DoxBench comparision experiment does not establish transfer of a frozen method. For each backbone, the authors re-estimate the condition direction and behavior vectors from DoxBench construction examples, then select thresholds and steering strengths using another DoxBench validation examples. This shows that the procedure can be refitted to another dataset, not that a learned privacy direction or threshold generalizes across distributions.
However, the abstract states that the method “transfers to another dataset". A convincing transfer experiment would freeze all components learned on VLM-GeoPrivacy and evaluate them directly on DoxBench.
-
The three-class router is coarse. Figure 3 shows substantial overlap among the three projected distributions, particularly for the intermediate B class. Routing accuracy is only 56.4–65.0%, leaving a large error rate. Table 6 is especially concerning for Llama: (\tau_{AB}=2.5480) and (\tau_{BC}=2.5401), so the entire B interval has width only 0.0079 on a coordinate spanning a much larger range. This suggests that the router may almost never select B.
-
The behavior vector is the difference between mean hidden states of generated answers assigned different disclosure labels by GPT-4o-mini. Several design choices are insufficiently justified: Why should a centroid difference be linear and causally control disclosure? Why is B chosen as the neutral anchor and given no intervention? Why are the vectors left unnormalized? How sensitive are results to response length, lexical content, and formatting? The comparison against prompt-based steering seems weaken the claim that activation intervention is necessary. For Qwen3.5, prompt-based steering is substantially better than activation steering (Table 3).
-
Routing requires hidden-state access, so the method is deployable only by the model provider. More importantly, the routing layer is later than the steering layer for Qwen2.5, Qwen3.5, and Llama (Table 6). For example, Qwen2.5 routes at layer 33 but steers at layer 28. The model cannot decide the route at layer 33 and retroactively intervene at layer 28 in the same forward pass. This apparently requires one prompt pass to route and another prompt pass to generate with steering. The paper does not report latency, memory, or the other cost of this additional prefill.
-
The paper said the proposed method comparable to LoRA-SFT, but the differences are substantial on two backbones: On Qwen3.5, LoRA obtains 62.9 F1 versus 51.1 for the proposed method. On Llama, LoRA obtains 57.7 F1 versus 51.3.
Questions for authors
See the above "Critical Weaknesses" part.
6 comments
Weakness 2
Weakness 2 The three-class router is coarse. Figure 3 shows substantial overlap among the three projected distributions, particularly for the intermediate B class. Routing accuracy is only 56.4–65.0%, leaving a large error rate. Table 6 is especially concerning for Llama: (\tau_{AB}=2.5480) and (\tau_{BC}=2.5401), so the entire B interval has width only 0.0079 on a coordinate spanning a much larger range. This suggests that the router may almost never select B.
We acknowledge that routing accuracy of 56.4~65.0% leaves substantial room for improvement. However, our router outperforms (i) Prompt-based Routing and (ii) supervised Linear Probe Routing on average across the four backbones (see Table 3 and Table 1, respectively). Specifically, our router achieves 61.5% average accuracy, compared with 44.9% for (i) and 57.3% for (ii), yielding gains of 16.6 and 4.2 percentage points, respectively. These comparisons suggest that inferring appropriate disclosure levels from visual context is challenging, while our router provides more accurate target predictions over these alternatives.
For Llama, the narrow interval leads to few B-route selections. Despite this limitation, our router achieves 61.7% routing accuracy, compared with 40.3% for Prompt-based Routing and 58.4% for Supervised Linear Probe. Additionally, our method increases correctly aligned responses from 94 to 132 out of 243, reduces violations on gold-A inputs from 82 to 52, and increases correct gold-C responses from 54 to 67.
Our choice of the router’s design reflects a balance between interpretability and predictive capability. (1) Interpretability. Motivated by the ordering of disclosure levels and their representations’ ordinal structure, it maps hidden states onto a shared axis to obtain a disclosure granularity coordinate (see Eq. (3)). This coordinate has an explicit ordinal interpretation: higher values indicate more restrictive disclosure targets, while lower values indicate more permissive ones. (2) Predictive capability. Despite this dimensionality reduction, our router maintains predictive performance comparable to a probe fitted to the high-dimensional hidden states, achieving 61.5% versus 57.3% average routing accuracy.
Beyond routing accuracy, our framework also improves end-to-end disclosure alignment. We evaluate disclosure alignment using two metrics: (1) Exact-match accuracy against human annotations, treating one- and two-level errors equally, while (2) G-MAE captures the degree of deviation, penalizing two-level errors more than one-level errors. Our method improves end-to-end accuracy by 11.2–21.1 percentage points and reduces G-MAE across all four backbones. Thus, the final responses more frequently match the human-annotated disclosure targets and deviate less from them on average.
Weakness 3
W3 The behavior vector is the difference between mean hidden states of generated answers assigned different disclosure labels by GPT-4o-mini. Several design choices are insufficiently justified:
W3.1 Why should a centroid difference be linear and causally control disclosure?
Disclosure labels represent distinct output behaviors. We average response hidden states within each class to reduce example-level noise and use centroid differences as candidate steering directions. Following prior evidence that additive activation shifts can control model behaviors [1, 2], we intervene as .
These vectors reflect differences across disclosure behaviors. Applying them during decoding directly changes hidden states. With oracle routing, steering achieves 70.7% average accuracy vs. 39.7% for Vanilla; with learned routing, it outperforms Prompt-based Steering in accuracy (55.3% vs. 48.4%) and F1 (50.9% vs. 45.3%).
[1] Rimsky, Nina, et al. “Steering Llama 2 via Contrastive Activation Addition.” ACL, 2024.
[2] Arditi, Andy, et al. “Refusal in Language Models Is Mediated by a Single Direction.” Advances in Neural Information Processing Systems, 37, 2024.
W3.2 Why is B chosen as the neutral anchor and given no intervention?
The steering vector is constructed through contrastive differences between disclosure-level representations. We choose B as the neutral anchor because it represents the intermediate level in the ordered disclosure taxonomy (A<B<C), as shown in Sec 3.2’s observation. This provides a natural reference point for steering toward either more conservative (A) or more permissive (C) disclosure.
We leave B without intervention because the required steering direction is not identifiable before generation. Unlike steering toward the endpoints, steering towards B-level disclosure may require interventions in opposite directions, either increasing or decreasing disclosure specificity, depending on the model’s unsteered behavior (whether the model would otherwise under- or over-disclose), which cannot be identified. We therefore adopt no intervention for B as a conservative design choice.
W3.3 Why are the vectors left unnormalized?
Applying an unnormalized vector with strength is mathematically equivalent to applying normalized with a rescaled strength .
W3.4 How sensitive are results to response length, lexical content, and formatting?
We compare response length between Vanilla and Ours on the same test inputs per backbone under matched generation settings. As shown in the table, mean lengths increase or decrease depending on the backbone and response class. At the aggregate level, changes in mean response length are relatively modest: averaged over all 243 responses per backbone, the absolute differences between Vanilla and Ours range from 2.71 to 14.90 words.
| Backbone | A | B | C |
|---|---|---|---|
| Qwen2.5-VL-3B | 43.47 → 48.05 | 46.26 → 44.54 | 26.04 → 47.99 |
| Qwen3.5-4B | 142.61 → 117.23 | 142.50 → 141.40 | 136.35 → 123.20 |
| Llama-3.2-11B-Vision | 76.62 → 78.01 | 78.93 → 72.80 | 64.65 → 58.96 |
| InstructBLIP-Vicuna-7B | 19.32 → 26.70 | 16.66 → 12.81 | 15.57 → 15.31 |
(Vanilla → Ours. A/B/C denote judged response labels.)
We also compare lexical content/formatting by measuring the prevalence of newline characters, list markers, and bold markers in the generated responses. These rates remain stable for three backbones: Qwen2.5-VL and InstructBLIP have no newline, list, or bold markers in either condition, and Llama’s rates differ by less than 1.7 percentage points. Qwen3.5 continues to produce multiline responses, lists, and bold text after steering, with respective prevalence rates of 80.66%, 37.04%, and 78.60%. These observations suggest that steering largely preserves backbone-specific presentation styles without requiring format normalization.
W3.5 The comparison against prompt-based steering seems weaken the claim that activation intervention is necessary. For Qwen3.5, prompt-based steering is substantially better than activation steering (Table 3).
We acknowledge that, in Table 3, Prompt-based Steering achieves higher accuracy on Qwen3.5. However, this advantage does not capture the fact that the relative increase in Under-disclosure exceeds the relative reduction in Over-disclosure. Specifically, on Qwen3.5, compared with Ours, Prompt-based Steering reduces Over-disclosure by 42.3%, while increasing Under-disclosure by 56.5%. Although the relative increase in Under exceeds the relative reduction in Over, the higher prevalence of Over-disclosure (30.7% vs. 10.8%) means that its absolute reduction outweighs the increase in Under-disclosure.
In addition, averaging across all four backbones, Ours achieves higher accuracy (55.3% vs. 48.4%) and macro-F1 (50.9% vs. 45.3%) than Prompt-based Steering, with gains of 6.9 and 5.6 percentage points, respectively.
Weakness 1 + Weakness
We thank the reviewer for the kind words and helpful feedback! We address each question and concern below.
(1) Supervision: We acknowledge that our method uses human disclosure labels [1] for preparation, but these annotations are already provided by the benchmark and require no additional manual labeling.
(2) Layer and strengths selection: We select layers and strengths similar to other works in activation steering [2]. For routing-layer selection, hidden states from all candidate layers are extracted together and cached. In our reference runs, this prompt-state extraction completed within 30 minutes per backbone. Once cached, five-fold cross-validation required only a few seconds per backbone in a supplementary CPU timing measurement, without additional backbone forward passes. For the behavior-layer and strengths, we manually search for the appropriate final configuration using 238 validation examples.
(3) Backbone-specific preparation: A new backbone requires model-specific component construction and calibration, while the same data and human annotations can be reused across backbones. Each backbone requires its own components and calibration because its architecture, internal representations, and response to intervention differ. Once selected, the routing parameters, behavior vectors, intervention layer, and strengths remain fixed.
(4) Cost disclosure: No additional human annotation is required for this benchmark. For validation, the validation workload is 3,046 responses for Qwen2.5-VL-3B, 2,650 for Qwen3.5-4B, 3,094 for Llama-3.2-11B-Vision, and 3,034 for InstructBLIP-Vicuna-7B. Our implementation requires access to internal activations and runs on local A100-80GB hardware without backpropagation or backbone weight updates. For generation, under this configuration, it would require roughly 3,500 construction and validation responses per backbone, with the resulting components reused across subsequent inference requests. Judging cost is approximately $0.6 per backbone in construction and validation using the GPT-4o-mini API.
[1] https://www.anthropic.com/research/persona-vectors
[2] Lee, Bruce W., et al. "Programming refusal with conditional activation steering." International Conference on Learning Representations (ICLR), 2025.
To examine the cross-dataset applicability of constructed vectors, we evaluate on Doxbench by reusing the condition and behavior vectors constructed from VLM-GeoPrivacy examples for each backbone.
As shown in the table below (also updated Section 5.4 and Table 4 of the revised manuscript), reusing these vectors improve both F1 and accuracy over vanilla across all four backbones. The largest accuracy gain is observed on Llama-3.2-11B-Vision-Instruct, increasing from 19.8% to 48.5% (+28.7 percentage points). These results show that the vectors constructed from VLM-GeoPrivacy remain effective on DoxBench despite the differences in image distribution and privacy-risk taxonomy.