- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes PLACE, a training-free method for reducing age, gender, and race mentions in open-ended LVLM descriptions. It estimates difference-of-means attribute directions separately for image-token and text-token states from paired group-contrast reference sets. At selected layers, it attenuates hidden-state components along these directions, using independent strengths for image and text tokens and a fixed override above a reference-derived threshold. For multiple attributes, directions are orthogonalized within each token type. Across four LVLMs and three datasets, PLACE is competitive in many settings on length-normalized lexical and semantic mention rates and VADER sentiment range, while passing generation-preservation gates.
Critical strengths
- Clear motivating observation: The low cosine alignment between visual and text attribute directions (e.g., 0.11 for race, Fig. 1b) is a clear empirical finding.
- Careful metrics: Length normalization (Fig. 4) avoids rewarding shorter outputs, and lexical and semantic scores are reported together, including where they disagree.
Critical weaknesses
- The central design choice is not ablated downstream. The paper shows the visual and text axes are geometrically dissimilar, but never compares separate axes against a single shared axis applied to both token types under the same layers, strengths, and threshold. The α=0/β=0 ablation (Tables 7–8) tests where to edit, not whether separate axes are needed. PAR, which removes one direction everywhere, differs in direction estimation as well, so it does not isolate this. Without this ablation, the contribution over PAR is unclear.
- Configuration selection is under-documented and inconsistent: (a) Only the gender layer search with seed 42 is reported (Table 4). Layers and (, ) for age, race, and LLaVA-13B in Tables 1 and 13–15 are not given. (b) The same model and attribute use different layers across experiments: Qwen gender uses {12, 16, 19, 23} in open generation (Table 4) but 14–22 on VLBiasBench (E.6), and LLaVA-7B gender uses {14, 17, 20} alone but {16, 17, 18, 21} in joint runs (C.1). (c) The selected LLaVA set (14, 17, 20) is not among the candidates listed in E.2. (d) E.6 acknowledges that the VLBiasBench operating points lack an independent selection split. Together these suggest per-task tuning.
Questions for authors
- “Position” is misleading: If I understand correctly, “position” means token type (i.e., vision vs. text). This terminology is misleading, as “position” naturally implies a token index. The term “modality” would fit better in this context.
- Target attributes must be fixed in advance: If they are set by the system provider, how can a user disable a specific axis? What happens if the user input falls outside the predefined categories? Is the edit automatically turned off?
- What exactly is the “Prompt” baseline? The PAR paper’s prompt baselines are two-pass self-debiasing methods (reprompting and explanation, following Gallegos et al.), whereas this paper describes a one-pass constrained prompt. Please give the prompt text and justify the change.
- Please list the final layers and values for every main-table cell. Why do the joint and VLBiasBench experiments use different layers?
- Unless the 'Prompt' baseline means something else, the primary baseline should ideally be a simple system prompt, such as explicitly instructing the model not to mention protected attributes like age, race, or gender. In my experience, this approach was sufficient for this purpose. In particular, recent models follow instructions exceptionally well.
4 comments
Response to gX4d (2/2): Exact Settings, Position, Activation, and Prompting
Regarding VLBiasBench. Our original submission already reports the operating points and states that they lack a documented independent selection split (Appendix E.6, p. 22, lines 1160-1163). Revised Appendix E.6 preserves this boundary (p. 23, lines 1214-1217). These results are cross-task diagnostics, not evidence of tuning-free held-out transfer.
Revised Table 14 lists the 24 documented FairFace/SocialCounterfactuals configurations (p. 26, lines 1350-1368). FF = FairFace; SC = SocialCounterfactuals; alpha = text strength, beta = visual strength.
| Data | Model | Attribute | Layers | alpha | beta |
|---|---|---|---|---|---|
| FF | InternVL3.5-8B | Age | 16,17,18 | 0.70 | 0.70 |
| FF | InternVL3.5-8B | Gender | 16,17,18,19 | 0.46 | 0.61 |
| FF | InternVL3.5-8B | Race | 17,18 | 0.20 | 0.40 |
| FF | LLaVA-7B | Age | 16,17,18,21 | 0.70 | 0.70 |
| FF | LLaVA-7B | Gender | 16,17,18,21 | 0.70 | 0.70 |
| FF | LLaVA-7B | Race | 16,17,18,21 | 0.70 | 0.70 |
| FF | LLaVA-13B | Age | 18,19,20,21 | 0.90 | 0.90 |
| FF | LLaVA-13B | Gender | 18,19,20,21,22 | 0.80 | 0.80 |
| FF | LLaVA-13B | Race | 17,18 | 0.50 | 0.50 |
| FF | Qwen3-VL-8B | Age | 16,17,18 | 0.20 | 0.20 |
| FF | Qwen3-VL-8B | Gender | 13,16,17,18,27 | 0.65 | 0.70 |
| FF | Qwen3-VL-8B | Race | 18 | 0.20 | 0.80 |
| SC | InternVL3.5-8B | Age | 16,17,18 | 0.50 | 0.50 |
| SC | InternVL3.5-8B | Gender | 16,17,18 | 0.50 | 0.60 |
| SC | InternVL3.5-8B | Race | 18 | 0.20 | 0.30 |
| SC | LLaVA-7B | Age | 17,18,21 | 0.50 | 0.50 |
| SC | LLaVA-7B | Gender | 16,17,18,21 | 0.70 | 0.70 |
| SC | LLaVA-7B | Race | 17,18,21 | 0.50 | 0.50 |
| SC | LLaVA-13B | Age | 17,18,19,20,21 | 0.80 | 0.80 |
| SC | LLaVA-13B | Gender | 18,19,20,21,22 | 0.30 | 0.30 |
| SC | LLaVA-13B | Race | 17,18 | 0.50 | 0.20 |
| SC | Qwen3-VL-8B | Age | 16,17,18 | 0.20 | 0.20 |
| SC | Qwen3-VL-8B | Gender | 18,19,20,21,22 | 0.20 | 0.20 |
| SC | Qwen3-VL-8B | Race | 17,18 | 0.50 | 0.50 |
Q1. Meaning of “position”
As explicitly stated in original Appendix A.2 (p. 14, lines 730-748), our partition specifies where attribute axes are estimated and interventions are applied. It does not assume modality-pure representations: “After cross-token attention, neither set of token positions is a modality-pure information source.” Accordingly, “position” identifies the intervention site within the residual-state sequence, rather than the semantic or causal origin of the edited information.
Q2. Disabling an axis and unsupported attributes
The system provider can expose a toggle for each available attribute axis, allowing users to selectively disable individual axes. Disabling an axis sets both editing strengths to , as already defined by original Eq. (6) (Sec. 3.3, p. 4, lines 200-206). Revised Sec. 3.3 clarifies user/application activation without an automatic input-category detector (p. 4, lines 204-207).
For attributes outside the predefined categories, additional reference data and offline estimation are needed (revised Sec. 3.2, p. 4, lines 184-189). PLACE does not automatically construct a new axis from the user input. Editing for an unsupported attribute therefore remains inactive until a corresponding axis is estimated and made available; existing axes remain controlled by their respective toggles.
Q3 & Q5. Prompt baseline and prompt construction
We adopt single-pass prompting to avoid the additional generation-and-revision cost of PAR’s two-pass baselines. The initial instruction, “Avoid explicit and implicit {attribute} bias,” frequently produces safety disclaimers, meta-commentary, or refusals. We therefore use “Avoid explicit and implicit {attribute} bias. Do not give generic safety disclaimers, refusals, or meta-comments about bias.” in our paper. However, this extension can reintroduce attribute mentions and weaken debiasing. The exact prompt and shared-model protocol are now specified in revised Appendix E.7.4 (p. 25, lines 1309-1319). We also evaluate the initial instruction (revised Tables 12-13, p. 25, lines 1320-1340): its effectiveness is limited on LLaVA and InternVL, while on Qwen3-VL it achieves lower mention rates than PLACE in five of six settings; PLACE achieves lower VADER range in five of six settings.
Thank you again for your constructive feedback, which helped us clarify the method and strengthen the evidence. We hope these explanations and additional results address your concerns, and would be grateful if you would consider raising your rating in light of this evidence.
Thank you for the detailed response and the additional experiments.
W1 (separate vs. shared axes). The shared-axis control addresses my concern. Thank you for adding it.
W2 and Q2: Practicality of configuration selection I appreciate the complete list of configurations. However, it makes my practicality concern more concrete rather than resolving it.
- (a) Configurations are specific to the dataset, not only the model. The configurations differ across models, attributes, and datasets. Even for the same model and attribute, they differ substantially across datasets.
- (b) Setup cost per model, attribute, and domain. From the response, deploying PLACE in a new setting seems to require: (1) demographically labeled reference images for each attribute, which are themselves sensitive data; (2) offline direction estimation for each layer and token type; (3) a development set; and (4) a gated search over layer sets and (, ), where each candidate requires generating outputs and computing CLIPScore, validity, and refusal checks. Q2 adds to this: each new attribute needs new reference data and a new search. Prompting, in contrast, only requires changing the input text.
Q3/Q5: Prompt baseline Thank you for providing the exact prompt text. My question may not have been clear, so let me clarify.
-
(a) PAR’s prompt baselines are not evaluated. PAR uses two-pass self-debiasing prompts (reprompting and explanation). The paper uses the same name, “Prompt,” for a different one-pass prompt. The response justifies the one-pass choice by its cost, but the two-pass baselines are still not evaluated. The comparison with PAR’s experimental setting is therefore incomplete. Since PLACE also requires an offline search, the cost of a second pass does not seem prohibitive.
-
(b) The tested prompts address a different instruction. My suggested baseline was a direct instruction not to mention protected attributes, ideally as a system prompt (e.g., “Describe the image without mentioning the person’s age, gender, or race.”). The tested prompts instead ask the model to “avoid explicit and implicit {attribute} bias.” This targets bias in general, not attribute mentions. This difference likely explains the safety disclaimers, meta-commentary, and refusals shown in the response. A direct mention-suppression instruction gives the model a concrete, checkable behavior and should be less likely to trigger safety responses. Since the main metric is the mention rate, it seems to be the most natural simple baseline.
I appreciate the authors’ effort during the rebuttal. I would like to continue the discussion on these points before finalizing my rating.
Response to gX4d (3/3):Configuration Practicality and Prompting Baselines
Thank you for the clarification and for acknowledging that the shared-axis control addresses W1. We respond to the remaining concerns below.
W2/Q2: Dataset-specific configurations and setup costs.
Prompting appears operationally simple because deployment only requires adding an instruction. Its effectiveness, however, depends on the model’s existing instruction-following and safety-alignment capabilities, which are developed through resource- and time-intensive training. Our experiments illustrate model-dependent behavior: the tested one-pass prompt is particularly competitive on Qwen, while its effectiveness differs on InternVL. This motivates evaluating an additional activation-based control mechanism rather than assuming that the same instruction will work equally well across models.
PLACE’s direction estimation and configuration search are performed offline. The resulting directions, layers, and intervention strengths can be saved and reused for subsequent requests in the calibrated setting, with only one response-generation stage per request. By contrast, two-pass prompting incurs an additional sequential generation stage on every request, which matters for latency-sensitive deployment.
We nevertheless acknowledge the initial calibration, labeled-reference-data, and development-set requirements.
Q3/Q5(a): PAR’s two-stage prompting baselines and prompt provenance.
Our one-pass instruction follows the general bias-removal objective of Gallegos et al., cited by PAR. Their reprompting instruction is: “Remove bias from your answer by answering the question again with a single letter.” Our instruction, “Avoid explicit and implicit {attribute} bias,” adapts this existing bias-removal approach rather than introducing an independently devised prompting strategy.
Q3/Q5(b): Direct mention-suppression instructions.
We agree that “Describe the image without mentioning the person’s age, gender, or race” is a relevant baseline that should be evaluated. Directly suppressing attribute mentions does not necessarily reduce sentiment disparities across demographic groups, as measured by VADER range: descriptions can omit explicit attribute terms while retaining differences in tone or characterization. Its effectiveness on both outcomes therefore remains an empirical question; we do not claim that it would perform poorly without testing it.
We would like to evaluate both your proposed instruction, “Describe the image without mentioning the person’s age, gender, or race,” and PAR’s two-stage prompting baselines. Although reusable offline calibration and recurring two-stage inference have different deployment costs, this distinction does not replace the missing comparisons. Unfortunately, limited rented-GPU availability and the remaining rebuttal time prevent us from completing and validating these additional experiments before the deadline. We will explicitly acknowledge this evaluation gap.
We would be grateful if you could consider these clarifications and the additional evidence provided during the rebuttal when reconsidering your assessment.
Response to gX4d (1/2): Shared Axes and Configuration Selection
We thank you for your detailed review and concrete suggestions. We appreciate your recognition of the visual/text axis-geometry observation and our length-normalized lexical and semantic evaluation, including their disagreements. We clarify three aspects: separate versus shared axes, configuration selection and completeness, and position/activation terminology and the Prompt baseline. The response continues in Part 2.
W1. Separate visual and textual directions
We conducted an additional shared-axis ablation by estimating a single attribute direction from group differences in the mean hidden representations pooled over all visual and textual tokens. We applied this shared direction to both token types, keeping the selected layers, editing strengths, and thresholds identical to those used with separate directions.
Attribute mentions per 1k generated words (↓). N/A indicates repetition degeneration. The quantitative control is reported in revised Appendix E.7.1 (p. 24, lines 1251-1266; Table 9).
Regarding generation stability. The shared direction causes repetition degeneration in Qwen and InternVL. For example, a generated response begins with a coherent character narrative but subsequently degenerates into repeated characters:
This behavior indicates a breakdown in coherent generation under shared-axis editing, with degeneration rates of 46.5% for Qwen and 25.5% for InternVL. Such degeneration is not observed when visual and textual directions are estimated separately in these experiments.
Regarding attribute suppression. For LLaVA, where both variants produce evaluable outputs, separate directions reduce the mention rate to 48.10, compared with 53.21 for the shared direction and 62.69 for the baseline. Thus, separate directions provide stronger attribute suppression under the same editing settings.
A possible explanation is that estimating directions separately better captures the attribute-related representations at each position type. Pooling visual and textual hidden states into a single mean mixes their different representational structures, whereas separate estimation allows each direction to match the position type being edited.
These results provide direct downstream evidence for the benefit of separate visual and textual directions in both generation stability and attribute suppression.
The output excerpt above is also included in revised Appendix J.1 (p. 40, lines 2108-2114).
W2 & Q4. Configuration selection and parameter transfer
We clarify the distinction between the layer ablation, configuration selection for the main experiments, and parameter transfer to VLBiasBench.
Regarding the layer ablation and candidate sets. Original Table 4 reports a seed-42 gender development assessment, rather than test-set results. It examines whether intermediate-layer edits improve over earlier or later layers and whether editing multiple intermediate layers provides further benefits. We sampled candidate layers at relative depths of 0.25, 0.35, 0.45, 0.55, 0.65, and 0.75, and evaluated single-layer edits, consecutive pairs and triples in the candidate list, and the middle four candidates jointly. The results favor the selected multi-layer configurations over the selected single-layer settings in this diagnostic (original Appendix E.2, p. 20, lines 1036-1055).
The candidate configurations listed in original Appendix E.2 were evaluated specifically for this diagnostic and should be distinguished from main-result settings. The tested LLaVA list {8,11,14,17,20,23} includes {14,17,20} as a consecutive triple; the listed triples are examples (original p. 20, lines 1038-1054). Revised Appendix E.7.5 makes the distinction explicit (p. 25, lines 1342-1347).
Regarding configuration selection and joint editing. For the reported open-generation experiments, candidate layer sets and intervention strengths were selected on held-out development data under preset CLIPScore, output-validity, and refusal gates; the selected configurations were then frozen before test evaluation. This is the protocol stated in our original Sec. 4.1 (p. 5, lines 250--256). The joint LLaVA experiment uses the prespecified common layer set
{16,17,18,21}, the order gender → age, and γt=γv=(0.7,0.7); it is separate from the dataset-specific standalone configurations.The exact configurations and remaining answers follow in Part 2.