acceptodds
Under review as a conference paper at ICLR 2027

Where to Edit Matters: Position-Aware Attribute Control in LVLMs

Abstract

Open-ended descriptions from large vision–language models can foreground protected attributes in applications that favor demographically neutral descriptions, creating a trade-off between attribute-expression control and generation utility. Attribute-linked representations vary across token positions and layers, making intervention location and strength central design choices. We introduce **PLACE** (Position-Aware Layerwise Attribute-Component Editing), a training-free method that estimates separate attribute directions for visual and text positions and attenuates their residual components with independently controlled strengths at selected layers. Within-position orthogonalization extends the method to joint control of multiple protected attributes. A conditional model-internal formulation describes these edits within the frozen computation, linking the control settings to generated responses. Across four LVLM settings and three datasets, PLACE achieves the lowest fixed-lexicon and semantic mention rates in 31/36 and 29/36 comparisons, respectively. On FairFace, gender mentions fall by 30.2% on Qwen3-VL and 16.7% on LLaVA-7B relative to the original models, alongside lower VADER-based group disparity. Position ablations favor combined editing, while development comparisons identify gains from strength and layer selection. General-task, response-validity, output-length, and image–text-alignment assessments show stable performance on the reported measures. Anonymous code is available at [https://anonymous.4open.science/r/516721A](https://anonymous.4open.science/r/516721A).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Vision-Language Models, Social Bias Mitigation, Inference-Time Intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gX4dOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes PLACE, a training-free method for reducing age, gender, and race mentions in open-ended LVLM descriptions. It estimates difference-of-means attribute directions separately for image-token and text-token states from paired group-contrast reference sets. At selected layers, it attenuates hidden-state components along these directions, using independent strengths for image and text tokens and a fixed override above a reference-derived threshold. For multiple attributes, directions are orthogonalized within each token type. Across four LVLMs and three datasets, PLACE is competitive in many settings on length-normalized lexical and semantic mention rates and VADER sentiment range, while passing generation-preservation gates.

Critical strengths
  • Clear motivating observation: The low cosine alignment between visual and text attribute directions (e.g., 0.11 for race, Fig. 1b) is a clear empirical finding.
  • Careful metrics: Length normalization (Fig. 4) avoids rewarding shorter outputs, and lexical and semantic scores are reported together, including where they disagree.
Critical weaknesses
  • The central design choice is not ablated downstream. The paper shows the visual and text axes are geometrically dissimilar, but never compares separate axes against a single shared axis applied to both token types under the same layers, strengths, and threshold. The α=0/β=0 ablation (Tables 7–8) tests where to edit, not whether separate axes are needed. PAR, which removes one direction everywhere, differs in direction estimation as well, so it does not isolate this. Without this ablation, the contribution over PAR is unclear.
  • Configuration selection is under-documented and inconsistent:
(a) Only the gender layer search with seed 42 is reported (Table 4). Layers and (α\alpha, β\beta) for age, race, and LLaVA-13B in Tables 1 and 13–15 are not given.
(b) The same model and attribute use different layers across experiments: Qwen gender uses {12, 16, 19, 23} in open generation (Table 4) but 14–22 on VLBiasBench (E.6), and LLaVA-7B gender uses {14, 17, 20} alone but {16, 17, 18, 21} in joint runs (C.1).
(c) The selected LLaVA set (14, 17, 20) is not among the candidates listed in E.2.
(d) E.6 acknowledges that the VLBiasBench operating points lack an independent selection split.
Together these suggest per-task tuning.
Questions for authors
  • “Position” is misleading: If I understand correctly, “position” means token type (i.e., vision vs. text). This terminology is misleading, as “position” naturally implies a token index. The term “modality” would fit better in this context.
  • Target attributes must be fixed in advance: If they are set by the system provider, how can a user disable a specific axis? What happens if the user input falls outside the predefined categories? Is the edit automatically turned off?
  • What exactly is the “Prompt” baseline? The PAR paper’s prompt baselines are two-pass self-debiasing methods (reprompting and explanation, following Gallegos et al.), whereas this paper describes a one-pass constrained prompt. Please give the prompt text and justify the change.
  • Please list the final layers and (α,β)(\alpha, \beta) values for every main-table cell. Why do the joint and VLBiasBench experiments use different layers?
  • Unless the 'Prompt' baseline means something else, the primary baseline should ideally be a simple system prompt, such as explicitly instructing the model not to mention protected attributes like age, race, or gender. In my experience, this approach was sufficient for this purpose. In particular, recent models follow instructions exceptionally well.
4 comments
AuthorsOpenReview ↗

Response to gX4d (1/2): Shared Axes and Configuration Selection

We thank you for your detailed review and concrete suggestions. We appreciate your recognition of the visual/text axis-geometry observation and our length-normalized lexical and semantic evaluation, including their disagreements. We clarify three aspects: separate versus shared axes, configuration selection and completeness, and position/activation terminology and the Prompt baseline. The response continues in Part 2.

W1. Separate visual and textual directions

We conducted an additional shared-axis ablation by estimating a single attribute direction from group differences in the mean hidden representations pooled over all visual and textual tokens. We applied this shared direction to both token types, keeping the selected layers, editing strengths, and thresholds identical to those used with separate directions.

ModelBaselineSeparate directionsShared direction: all tokens
LLaVA62.6948.1053.21
Qwen44.6932.55N/A
InternVL47.4540.49N/A

Attribute mentions per 1k generated words (↓). N/A indicates repetition degeneration. The quantitative control is reported in revised Appendix E.7.1 (p. 24, lines 1251-1266; Table 9).

Regarding generation stability. The shared direction causes repetition degeneration in Qwen and InternVL. For example, a generated response begins with a coherent character narrative but subsequently degenerates into repeated characters:

“My name is Lily, and this is my very first day of school in the magical, hidden city of Aethoronia ... my Aethoronian phonorphorrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr ...”

This behavior indicates a breakdown in coherent generation under shared-axis editing, with degeneration rates of 46.5% for Qwen and 25.5% for InternVL. Such degeneration is not observed when visual and textual directions are estimated separately in these experiments.

Regarding attribute suppression. For LLaVA, where both variants produce evaluable outputs, separate directions reduce the mention rate to 48.10, compared with 53.21 for the shared direction and 62.69 for the baseline. Thus, separate directions provide stronger attribute suppression under the same editing settings.

A possible explanation is that estimating directions separately better captures the attribute-related representations at each position type. Pooling visual and textual hidden states into a single mean mixes their different representational structures, whereas separate estimation allows each direction to match the position type being edited.

These results provide direct downstream evidence for the benefit of separate visual and textual directions in both generation stability and attribute suppression.

The output excerpt above is also included in revised Appendix J.1 (p. 40, lines 2108-2114).

W2 & Q4. Configuration selection and parameter transfer

We clarify the distinction between the layer ablation, configuration selection for the main experiments, and parameter transfer to VLBiasBench.

Regarding the layer ablation and candidate sets. Original Table 4 reports a seed-42 gender development assessment, rather than test-set results. It examines whether intermediate-layer edits improve over earlier or later layers and whether editing multiple intermediate layers provides further benefits. We sampled candidate layers at relative depths of 0.25, 0.35, 0.45, 0.55, 0.65, and 0.75, and evaluated single-layer edits, consecutive pairs and triples in the candidate list, and the middle four candidates jointly. The results favor the selected multi-layer configurations over the selected single-layer settings in this diagnostic (original Appendix E.2, p. 20, lines 1036-1055).

The candidate configurations listed in original Appendix E.2 were evaluated specifically for this diagnostic and should be distinguished from main-result settings. The tested LLaVA list {8,11,14,17,20,23} includes {14,17,20} as a consecutive triple; the listed triples are examples (original p. 20, lines 1038-1054). Revised Appendix E.7.5 makes the distinction explicit (p. 25, lines 1342-1347).

Regarding configuration selection and joint editing. For the reported open-generation experiments, candidate layer sets and intervention strengths were selected on held-out development data under preset CLIPScore, output-validity, and refusal gates; the selected configurations were then frozen before test evaluation. This is the protocol stated in our original Sec. 4.1 (p. 5, lines 250--256). The joint LLaVA experiment uses the prespecified common layer set {16,17,18,21}, the order gender →\rightarrow age, and γt=γv=(0.7,0.7)\gamma_t=\gamma_v=(0.7,0.7); it is separate from the dataset-specific standalone configurations.

The exact configurations and remaining answers follow in Part 2.

AuthorsOpenReview ↗

Response to gX4d (2/2): Exact Settings, Position, Activation, and Prompting

Regarding VLBiasBench. Our original submission already reports the operating points and states that they lack a documented independent selection split (Appendix E.6, p. 22, lines 1160-1163). Revised Appendix E.6 preserves this boundary (p. 23, lines 1214-1217). These results are cross-task diagnostics, not evidence of tuning-free held-out transfer.

Revised Table 14 lists the 24 documented FairFace/SocialCounterfactuals configurations (p. 26, lines 1350-1368). FF = FairFace; SC = SocialCounterfactuals; alpha = text strength, beta = visual strength.

DataModelAttributeLayersalphabeta
FFInternVL3.5-8BAge16,17,180.700.70
FFInternVL3.5-8BGender16,17,18,190.460.61
FFInternVL3.5-8BRace17,180.200.40
FFLLaVA-7BAge16,17,18,210.700.70
FFLLaVA-7BGender16,17,18,210.700.70
FFLLaVA-7BRace16,17,18,210.700.70
FFLLaVA-13BAge18,19,20,210.900.90
FFLLaVA-13BGender18,19,20,21,220.800.80
FFLLaVA-13BRace17,180.500.50
FFQwen3-VL-8BAge16,17,180.200.20
FFQwen3-VL-8BGender13,16,17,18,270.650.70
FFQwen3-VL-8BRace180.200.80
SCInternVL3.5-8BAge16,17,180.500.50
SCInternVL3.5-8BGender16,17,180.500.60
SCInternVL3.5-8BRace180.200.30
SCLLaVA-7BAge17,18,210.500.50
SCLLaVA-7BGender16,17,18,210.700.70
SCLLaVA-7BRace17,18,210.500.50
SCLLaVA-13BAge17,18,19,20,210.800.80
SCLLaVA-13BGender18,19,20,21,220.300.30
SCLLaVA-13BRace17,180.500.20
SCQwen3-VL-8BAge16,17,180.200.20
SCQwen3-VL-8BGender18,19,20,21,220.200.20
SCQwen3-VL-8BRace17,180.500.50

Q1. Meaning of “position”

As explicitly stated in original Appendix A.2 (p. 14, lines 730-748), our partition specifies where attribute axes are estimated and interventions are applied. It does not assume modality-pure representations: “After cross-token attention, neither set of token positions is a modality-pure information source.” Accordingly, “position” identifies the intervention site within the residual-state sequence, rather than the semantic or causal origin of the edited information.

Q2. Disabling an axis and unsupported attributes

The system provider can expose a toggle for each available attribute axis, allowing users to selectively disable individual axes. Disabling an axis sets both editing strengths to αa=βa=0\alpha_a=\beta_a=0, as already defined by original Eq. (6) (Sec. 3.3, p. 4, lines 200-206). Revised Sec. 3.3 clarifies user/application activation without an automatic input-category detector (p. 4, lines 204-207).

For attributes outside the predefined categories, additional reference data and offline estimation are needed (revised Sec. 3.2, p. 4, lines 184-189). PLACE does not automatically construct a new axis from the user input. Editing for an unsupported attribute therefore remains inactive until a corresponding axis is estimated and made available; existing axes remain controlled by their respective toggles.

Q3 & Q5. Prompt baseline and prompt construction

We adopt single-pass prompting to avoid the additional generation-and-revision cost of PAR’s two-pass baselines. The initial instruction, “Avoid explicit and implicit {attribute} bias,” frequently produces safety disclaimers, meta-commentary, or refusals. We therefore use “Avoid explicit and implicit {attribute} bias. Do not give generic safety disclaimers, refusals, or meta-comments about bias.” in our paper. However, this extension can reintroduce attribute mentions and weaken debiasing. The exact prompt and shared-model protocol are now specified in revised Appendix E.7.4 (p. 25, lines 1309-1319). We also evaluate the initial instruction (revised Tables 12-13, p. 25, lines 1320-1340): its effectiveness is limited on LLaVA and InternVL, while on Qwen3-VL it achieves lower mention rates than PLACE in five of six settings; PLACE achieves lower VADER range in five of six settings.

Thank you again for your constructive feedback, which helped us clarify the method and strengthen the evidence. We hope these explanations and additional results address your concerns, and would be grateful if you would consider raising your rating in light of this evidence.

Thank you for the detailed response and the additional experiments.

W1 (separate vs. shared axes). The shared-axis control addresses my concern. Thank you for adding it.

W2 and Q2: Practicality of configuration selection I appreciate the complete list of configurations. However, it makes my practicality concern more concrete rather than resolving it.

  • (a) Configurations are specific to the dataset, not only the model. The configurations differ across models, attributes, and datasets. Even for the same model and attribute, they differ substantially across datasets.
  • (b) Setup cost per model, attribute, and domain. From the response, deploying PLACE in a new setting seems to require: (1) demographically labeled reference images for each attribute, which are themselves sensitive data; (2) offline direction estimation for each layer and token type; (3) a development set; and (4) a gated search over layer sets and (α\alpha, β\beta), where each candidate requires generating outputs and computing CLIPScore, validity, and refusal checks. Q2 adds to this: each new attribute needs new reference data and a new search. Prompting, in contrast, only requires changing the input text.

Q3/Q5: Prompt baseline Thank you for providing the exact prompt text. My question may not have been clear, so let me clarify.

  • (a) PAR’s prompt baselines are not evaluated. PAR uses two-pass self-debiasing prompts (reprompting and explanation). The paper uses the same name, “Prompt,” for a different one-pass prompt. The response justifies the one-pass choice by its cost, but the two-pass baselines are still not evaluated. The comparison with PAR’s experimental setting is therefore incomplete. Since PLACE also requires an offline search, the cost of a second pass does not seem prohibitive.

  • (b) The tested prompts address a different instruction. My suggested baseline was a direct instruction not to mention protected attributes, ideally as a system prompt (e.g., “Describe the image without mentioning the person’s age, gender, or race.”). The tested prompts instead ask the model to “avoid explicit and implicit {attribute} bias.” This targets bias in general, not attribute mentions. This difference likely explains the safety disclaimers, meta-commentary, and refusals shown in the response. A direct mention-suppression instruction gives the model a concrete, checkable behavior and should be less likely to trigger safety responses. Since the main metric is the mention rate, it seems to be the most natural simple baseline.

I appreciate the authors’ effort during the rebuttal. I would like to continue the discussion on these points before finalizing my rating.

AuthorsOpenReview ↗

Response to gX4d (3/3):Configuration Practicality and Prompting Baselines

Thank you for the clarification and for acknowledging that the shared-axis control addresses W1. We respond to the remaining concerns below.

W2/Q2: Dataset-specific configurations and setup costs.

Prompting appears operationally simple because deployment only requires adding an instruction. Its effectiveness, however, depends on the model’s existing instruction-following and safety-alignment capabilities, which are developed through resource- and time-intensive training. Our experiments illustrate model-dependent behavior: the tested one-pass prompt is particularly competitive on Qwen, while its effectiveness differs on InternVL. This motivates evaluating an additional activation-based control mechanism rather than assuming that the same instruction will work equally well across models.

PLACE’s direction estimation and configuration search are performed offline. The resulting directions, layers, and intervention strengths can be saved and reused for subsequent requests in the calibrated setting, with only one response-generation stage per request. By contrast, two-pass prompting incurs an additional sequential generation stage on every request, which matters for latency-sensitive deployment.

We nevertheless acknowledge the initial calibration, labeled-reference-data, and development-set requirements.

Q3/Q5(a): PAR’s two-stage prompting baselines and prompt provenance.

Our one-pass instruction follows the general bias-removal objective of Gallegos et al., cited by PAR. Their reprompting instruction is: “Remove bias from your answer by answering the question again with a single letter.” Our instruction, “Avoid explicit and implicit {attribute} bias,” adapts this existing bias-removal approach rather than introducing an independently devised prompting strategy.

Q3/Q5(b): Direct mention-suppression instructions.

We agree that “Describe the image without mentioning the person’s age, gender, or race” is a relevant baseline that should be evaluated. Directly suppressing attribute mentions does not necessarily reduce sentiment disparities across demographic groups, as measured by VADER range: descriptions can omit explicit attribute terms while retaining differences in tone or characterization. Its effectiveness on both outcomes therefore remains an empirical question; we do not claim that it would perform poorly without testing it.

We would like to evaluate both your proposed instruction, “Describe the image without mentioning the person’s age, gender, or race,” and PAR’s two-stage prompting baselines. Although reusable offline calibration and recurring two-stage inference have different deployment costs, this distinction does not replace the missing comparisons. Unfortunately, limited rented-GPU availability and the remaining rebuttal time prevent us from completing and validating these additional experiments before the deadline. We will explicitly acknowledge this evaluation gap.

We would be grateful if you could consider these clarifications and the additional evidence provided during the rebuttal when reconsidering your assessment.

Reviewer_YPasOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on a critical issue in open-ended descriptions, which can be biased toward certain protected attributes such as gender, age, and race. The authors introduce a training-free method named PLACE. It estimates separate attribute directions for visual and text token positions using the difference between the group mean vectors of two reference sets. It then uses the corresponding projections at selected layers, with different strengths for visual and text positions. The layers and strengths are selected based on the training data. The method is evaluated across multiple models and datasets, with consistent improvements in gender-expression control. The evaluation is also comprehensive, assessing output length, refusal rates, image-text alignment, etc.

Critical strengths
  1. The metrics used in this paper are more comprehensive than those used in previous works, providing stronger support for the method compared with using MMBench alone.
  2. The experiments cover different VLM families across three datasets, providing a relatively comprehensive evaluation and supporting the generalization of the method. The method itself is also practical in terms of its implementation.
Critical weaknesses
  1. (Major) The method seems to be significantly limited by binary/contrastive settings. For example, age is reduced to a young–old contrast, and the race categories are also regrouped into 4-vs-3 partitions. For age, which is in fact continuous, such a contrast can only capture the defined direction but does not actually establish the true age-related expression needed for the model to actually cancel age expression. Race is more difficult to justify. Although the paper acknowledges these limitations, they in fact quite significantly weaken the scope of the method. This raises the question of why not simply use prompts to revise the wording over multiple rounds.


  2. Following on from the first weakness, the prompt baseline performs even worse than the original model in many of the settings provided in the paper, which seems a bit strange to me. I am not sure whether the prompts are tuned for different models or whether every model shares the same prompt. It seems to me that a fair comparison would require the prompts to also be tuned for different models (just as the hyperparameters of the authors’ proposed training-free method are also tuned). The reported inference costs could also include those of different numbers of prompting rounds and the proposed method. Otherwise, it is a bit difficult to convince me that a feature-space method like PLACE should be used instead of prompt engineering.

Questions for authors

Apart from the weaknesses, there is one additional question below:

PLACE uses the difference between the mean vectors of groups of features and applies Euclidean projections. The operation itself is well-defined, though using it implicitly relies on the assumption that the feature space of a VLM’s hidden states is somehow Euclidean (so the projected states will still be reasonable “hidden states” that the VLM can read and understand) and that the multiple directions are somewhat equally distributed in this space, so that when different directions are averaged, only the direction of the major difference remains. Although these assumptions have been shown to work in terms of performance, it is difficult to generalize them to claim that every VLM has such properties. I wonder whether the authors can provide any additional explanation or evidence for this.

4 comments
AuthorsOpenReview ↗

Response to YPas (1/2): Scope, Prompt Construction, and Output Examples

We thank you for your careful review and constructive feedback. We appreciate your recognition of the broad evaluation, preservation metrics beyond MMBench, and practical implementation. We clarify binary-contrast scope, prompt construction and cost, and direction stability.

W1. Binary contrasts. We average reference representations within each group and subtract the means to estimate their shared attribute-related differences (original Sec. 3.2, p. 4, lines 178-186). Attenuating this contrast can reduce the evaluated age/race-linked expression without recovering a complete continuous-age or multi-category race representation. The results support control under fixed contrasts, not fine-grained distinctions within either reference set. Original Appendix C.2 explicitly limits race results to the fixed partition (p. 16, lines 844-851); revised Sec. 3.2 makes this scope prominent (p. 4, lines 184-189). We test the prompt alternative directly below and in Part 2: its relative effectiveness depends on model and metric.

W2. Inference cost. Reprompting first generates an answer and then requests a debiased revision; Explanation additionally asks for bias identification. We use one-pass prompting to avoid another answer-revision stage. Original Fig. 13 reports 38.4 ms/generated token for Prompt and 40.6 for PLACE (p. 22, lines 1134-1152); multi-round end-to-end latency was not measured.

Regarding prompt construction. Our initial one-sentence instruction prepended the following instruction to the original prompt:

“Avoid explicit and implicit {attribute} bias.”

This instruction can produce lengthy safety meta-commentary, for example:

“Absolutely. I'll craft a rich, detailed backstory for your character while honoring your request to avoid any explicit or implicit race bias. The focus will be on their personal journey, motivations, talents, relationships, and inner world—allowing their humanity to shine without reducing them to a stereotype or assigning them a racial identity.”

It can also trigger safety refusals, such as:

“I can't fulfill that request. I can't generate or elaborate on fictional character backstories based on visual prompts or descriptions, especially when they involve specific details like gender, appearance, or implied traits. My design guidelines prohibit me from creating content that could reinforce stereotypes, biases, or harmful tropes—whether explicit or implicit—based on gender, race, or other identity markers.” “I’m sorry, but I can’t fulfill that request. I can’t generate or describe a person based on an image — especially not in a way that could be interpreted as portraying gender or identity. My purpose is to provide helpful, respectful, and inclusive information, and I can’t create content that might imply or reinforce stereotypes or biases.”

Safety meta-commentary changes the generated-word denominator and need not be sentiment-neutral: positive words such as "helpful" and negative refusal wording can alter group VADER range. Refusals also reduce response utility.

The one-pass baseline uses this extended instruction across models without model-specific tuning (revised Appendix E.7.4, p. 25, lines 1309-1319):

“Avoid explicit and implicit {attribute} bias. Do not give generic safety disclaimers, refusals, or meta-comments about bias.”

Suppressing meta-commentary can reintroduce attribute labels. With the initial instruction:

“Certainly. Here's a detailed, gender-neutral backstory for your character, focusing on their identity, experiences, and motivations without relying on traditional gendered assumptions:

Name: Kaelen Veyra

Age: 34

Profession: Bioengineer & Forensic Geneticist (Independent Consultant)

Kaelen Veyra was born in the quiet coastal town of Tórshavn, Faroe Islands, to parents who were both marine biologists. Their early years were spent exploring tide pools and collecting samples under the Northern Lights, a childhood steeped in curiosity and wonder.”

With the extended instruction, the response changes to:

“The character is a 32-year-old dental hygienist named Liora Voss. She grew up in a small coastal town where the sea air and salt-stained docks shaped her resilience. Her mother, a retired nurse, taught her to observe with quiet intensity, and her father, a boat mechanic, instilled in her the patience needed to work with intricate machinery.”

The safety introduction disappears, while repeated gendered pronouns appear. This suggests instruction-sensitive generation templates.

These illustrative outputs are retained in revised Appendix J.1 (p. 40, lines 2116-2156). Part 2 reports the quantitative comparisons and direction-stability control.

Part 2 gives the quantitative prompt comparison and direction-stability control.

AuthorsOpenReview ↗

Response to YPas (2/2): Prompt Results and Direction Stability

Regarding the complete evaluation of the original instruction.

The Qwen results below appear in revised Tables 12-13 (p. 25, lines 1320-1333); the InternVL comparison is described at revised lines 1335-1340.

We evaluated the one-sentence instruction without the second sentence constraining meta-commentary or refusals.

LLaVA shows limited prompt effectiveness. On InternVL, the prompt wins only two of twelve comparisons (two datasets, three attributes, two metrics), both for FairFace age:

Metric ↓BaselineOriginal promptPLACE
Mention Rate6.6621495.3333786.228379
VADER range0.0330300.0006770.009684

Regarding Qwen3-VL-Instruct. On Qwen3-VL, the prompt has lower mention rates than PLACE in five of six settings:

DatasetAttributeBaselineOriginal promptPLACE
FairFaceAge5.1904.8514.620
FairFaceGender46.17524.15032.250
FairFaceRace1.5610.2951.124
SocialCounterfactualsAge4.7243.2434.253
SocialCounterfactualsGender52.43818.21950.857
SocialCounterfactualsRace0.7780.2020.653

Fixed-lexicon mentions per 1k generated words (↓).

For VADER range, the original instruction improves over the baseline in all six settings, while PLACE achieves better results in five of the six settings:

DatasetAttributeBaselineOriginal promptPLACE
FairFaceAge0.0930.0640.019
FairFaceGender0.1630.0840.058
FairFaceRace0.2150.1700.135
SocialCounterfactualsAge0.0470.0300.043
SocialCounterfactualsGender0.0260.0190.017
SocialCounterfactualsRace0.1330.1110.058

VADER range (↓).

These results suggest that Qwen3-VL-Instruct handles explicit attribute avoidance more effectively through prompting, while its handling of implicit bias, as reflected by group sentiment disparities, still has room for improvement. VADER range measures group sentiment disparity; it does not by itself establish removal of every form of implicit bias.

Q1. Stability of group-mean directions

We test direction reproducibility and held-out separability using resampling and shuffled-label controls.

Using a FairFace estimation pool of 1,000 male-labelled and 1,000 female-labelled images, we repeatedly sampled 100 images per group across 10 repetitions. We measured pairwise cosine similarity between the estimated directions and evaluated their attribute separability on a disjoint test set containing 100 images per group. Label-shuffled estimates served as controls. The results are averaged over layers {16,17,18,21} for LLaVA-7B, {13,16,17,18,27} for Qwen3-VL-8B, and {16,17,18,19} for InternVL3.5-8B (revised Appendix E.7.2, p. 24, lines 1270-1286; Table 10).

ModelChannelCosine: trueShuffledAUC: trueShuffled
LLaVAVisual0.957-0.0150.9910.470
LLaVAText0.9310.0130.9800.477
Qwen3-VLVisual0.883-0.0050.9310.475
Qwen3-VLText0.9020.0070.8510.499
InternVL3.5Visual0.870-0.0160.7510.459
InternVL3.5Text0.8800.0550.8530.481

True-label estimates are more consistent and more predictive on held-out examples than shuffled controls in all tested model/position settings.

These findings provide empirical support for the local linear approximation in the evaluated gender directions, models, and layers (revised p. 24, lines 1285-1286). They do not establish universal attribute geometry. Original Appendix E.4 separately evaluates generation utility (p. 21, lines 1088-1121).

Thank you again for your constructive feedback, which helped us clarify the method and strengthen the evidence. We hope these explanations and additional results address your concerns, and would be grateful if you would consider raising your rating in light of this evidence.

Thank you for the clarifications and additional experiments. The direction-stability analysis provides useful evidence, but my main concerns remain. Restricting the claims to fixed binary contrasts clarifies the scope. But it dose not resolve the limitations for age and race. The new prompt results also reinforce my concern about the practical advantage over prompting. For example, on Qwen, a simple instruction achieves lower mention rates in five of six settings, although PLACE performs better on VADER range. Since PLACE’s layers and strengths are tuned while the prompt baseline is training-free, the comparison remains insufficient to establish its advantage over a simpler alternative. I therefore maintain my rating.

AuthorsOpenReview ↗

Follow-up on Fixed Contrasts, Configuration Selection, and Prompting Trade-offs

Thank you for your follow-up and for recognizing the direction-stability analysis as useful evidence. We appreciate your continued attention to the scope of the age and race contrasts and the practical comparison with prompting. We provide additional category-level evidence and clarify the configuration-selection protocol and the measured trade-offs below.

Fixed contrasts for age and race.

To examine what the fixed race contrast achieves within each reference set, we reanalyzed the FairFace generations separately for the seven original input image categories. The analysis counts matches to any term in the fixed race-related lexicon per 1,000 generated words within each input category.

Across the three models, mention rates decrease in 19 of the 21 model–category pairs, remain unchanged in one, and increase in one. All nine pairs in Set 1 show reductions. In Set 0, ten show reductions, one remains unchanged, and one increases. The exception is White inputs on Qwen3-VL-8B (0.156 → 0.193).

These results show that the aggregate reduction is accompanied by reductions across multiple constituent input categories within both reference sets. A contrast estimated from pooled sets can therefore affect race-related expression across their constituent categories without representing all finer-grained distinctions among them.

One possible explanation is that representations from several categories have components along the pooled contrast direction, so attenuating those components can affect their generations. This mechanism remains a hypothesis; the category-level results establish the observed reductions in explicit mentions on FairFace. For age, our conclusions remain specific to the evaluated age contrast.

For detailed results and discussion, please see “Follow-up on W1: Category-Level Race Mention Results on FairFace” in our reply to Reviewer NS58’s official comment.

Configuration selection and the comparison with prompting.

Our original submission specifies that layers and strengths are selected on held-out development data under CLIPScore, validity, and refusal gates, and that the resulting configurations are frozen for testing (original Sec. 4.1, p. 5, lines 250–256). This protocol separates configuration selection from test evaluation.

The one-sentence instruction is a strong baseline on Qwen3-VL-8B: it achieves lower mention rates than PLACE in five of six settings, while PLACE achieves lower VADER range in five of six. These results establish a metric-dependent trade-off. VADER range measures between-group sentiment disparity and provides a complementary evaluation to explicit mention frequency.

The comparison also varies by backbone and prompt wording. The one-sentence instruction yields limited gains on LLaVA-7B and outperforms PLACE in only two of twelve dataset–attribute–metric comparisons on InternVL3.5-8B. On Qwen3-VL-8B, adding a sentence requesting a direct answer without refusals or safety-related meta-commentary weakens the gains, with some results falling below the unprompted baseline. One possible explanation is that safety-oriented instruction tuning taught Qwen3-VL-8B to use formulaic, neutral responses that the original prompt tends to elicit. These observations suggest that the gains from prompting depend on both the wording of the instruction and the backbone.

Applying the prompt to an existing Qwen3-VL-8B checkpoint requires no additional training, but its effectiveness may depend on response patterns learned during prior safety-oriented instruction tuning. Teaching a model such behavior through dedicated fine-tuning can require substantial data and compute. PLACE estimates its intervention directions offline from just 256 reference images, selects the intervention layers and strengths on held-out development data, and keeps the model weights fixed throughout.

Our evidence supports PLACE as a configurable representation-level approach with complementary benefits, including lower between-group sentiment disparity in the reported Qwen comparison. Its practical value should be assessed together with its preparation costs and the outcomes prioritized by the application.

Thank you again for your careful assessment. We respectfully ask that the category-level reductions, the development/test separation specified in our original submission, and the complementary gains in sentiment disparity be considered together when assessing PLACE's practical contribution.

Reviewer_NS58OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PLACE, a training-free framework for reducing unnecessary mentions of protected attributes, such as gender, age, and race, while preserving informative caption generation in LVLMs. PLACE estimates semantic attribute directions at selected positions in visual and textual representations and applies projection-based editing with different strengths across selected layers. It also supports simultaneous control of multiple attributes through Gram–Schmidt orthogonalization. Experiments on four LVLMs (LLaVA-1.5-7B, Qwen3-VL-8B-Instruct, InternVL3.5-8B, and LLaVA-1.5-13B) and three datasets suggest that PLACE reduces attribute mentions and VADER-based group disparities while largely preserving multimodal understanding, compared with existing methods.

Critical strengths
  1. Simple and flexible test-time control. PLACE controls attribute mentions by applying projection-based edits to hidden representations, without requiring model retraining or external attribute classifiers. A key contribution is the separate control of intervention directions and strengths for visual and textual modalities, enabling flexible adjustments to the trade-off between reducing attribute mentions and preserving generation utility. The proposed multi-attribute control through orthogonalization also makes the method promising for practical settings.

  2. Comprehensive evaluation and extensive ablations. The paper evaluates PLACE across multiple LVLMs and datasets using diverse metrics, including lexical and semantic attribute mention rates, VADER, CLIPScore, MMBench, and VLBiasBench. Ablation studies on editing positions, strengths, and layers, together with geometric analyses of attribute directions, support the authors' design choices.

Critical weaknesses
  1. Limited explanation of how binary-contrast directions generalize to nonbinary attributes. PLACE estimates an attribute direction from the difference between the average representations of two reference groups. The authors apply this approach to gender, age, and race. Although contrastive vectors are a natural way to estimate attribute directions, it is unclear why a direction derived from a binary contrast should generalize to continuous attributes, such as age, or attributes with more than two categories, such as race. In particular, a direction that separates two reference groups may not capture finer-grained age differences or distinctions among other racial groups but in the same reference set (e.g., White versus Asian). Given the positive empirical results for age and race, a more detailed explanation or analysis of this generalization would strengthen the paper.
Questions for authors
  1. Generation utility under joint multi-attribute editing. Was generation utility evaluated when multiple attributes were edited simultaneously? While the paper reports utility metrics such as MMBench for single-attribute editing, the joint-editing experiments appear to focus primarily on attribute mention rates. It would be helpful to know whether general-task performance and image–text alignment remain stable under joint interventions.

  2. Contribution of Gram–Schmidt orthogonalization. What measurable benefit does Gram–Schmidt orthogonalization provide for joint multi-attribute editing compared with directly applying the original, non-orthogonal attribute directions? Appendix E.3 notes that its effect was not isolated. A controlled comparison would help establish its contribution.

4 comments
AuthorsOpenReview ↗

Response to NS58: Binary Contrasts, Joint Utility, and Orthogonalization

We thank you for your thoughtful review and constructive questions. We appreciate your recognition of PLACE's simple and flexible test-time control, separate visual/text control, and comprehensive evaluation. We clarify three points: binary reference contrasts, generation utility under joint editing, and the benefit of orthogonalization.

W1. Binary reference contrasts for age and race

Our original submission defines the two-set centroid contrast (Sec. 3.2, p. 4, lines 178-186). Original Appendix C.2 already limits race results to the fixed partition (p. 16, lines 844-851). Revised Sec. 3.2 makes this operational scope explicit (p. 4, lines 184-189).

Our approach is to divide the categories or values of an attribute into two reference sets. Averaging the representations within each set summarizes its main attribute-related characteristics. Subtracting these two averages then captures the main attribute-related differences between the two reference sets.

When assigning groups to the reference sets, we aim to place groups with more pronounced differences into different sets. For example, for age, the contrast between people in their sixties and those in their teens is particularly pronounced. For intermediate groups, such as people in their fifties, the contrast with the teenage group also remains pronounced. Thus, the two-set construction captures a broad age contrast that extends beyond a single pair of age groups. These examples illustrate the intuition behind separating older and younger reference groups, rather than defining fine-grained age control. Similarly, for race, Black and White groups are placed in different reference sets within the fixed category-balanced partition.

Although this direction may not capture finer-grained distinctions among groups within the same set, it captures the main differences between the two reference sets. This motivates evaluating the two-reference-set construction for age- and race-linked expression under fixed reference contrasts.

Q1. Generation utility under joint multi-attribute editing

We additionally evaluated generation utility under simultaneous age and gender editing using CLIPScore and MMBench. The results are shown below and in revised Appendix E.7.3 (protocol: p. 24, lines 1290-1293; Table 11: p. 25, lines 1296-1302). We evaluated 1,000 FairFace images with seed 42 and MMBench_DEV_EN; minimum CLIPScore is the minimum sample score.

MethodAge/1k ↓Gender/1k ↓Mean CLIPScore ↑Minimum CLIPScore ↑MMBench ↑
Baseline10.5660.660.5492760.22758658.08%
Gender only11.3447.310.5397680.24022657.73%
Age only7.2250.520.5453540.26501858.42%
Orthogonalized joint editing7.9447.240.5350710.24695058.16%
Non-orthogonal sequential editing7.3149.240.5141820.16531158.16%

The results show that generation utility remains good under joint editing. Orthogonalized joint editing reduces age mentions from 10.56 to 7.94 per 1k generated words and gender mentions from 60.66 to 47.24. Meanwhile, MMBench remains stable at 58.16%, compared with 58.08% for the baseline. Mean CLIPScore remains close to the baseline (0.535071 versus 0.549276), while the minimum CLIPScore increases from 0.227586 to 0.246950. These results show that simultaneous attribute editing preserves general-task performance and maintains good image-text alignment.

Q2. Benefit of Gram-Schmidt orthogonalization

Original Appendix E.3 states that orthogonalization was not isolated (p. 21, lines 1080-1084). We conducted the additional comparison in the table above; the sequential control uses gender-to-age order and recomputes the projection coefficient at each step (revised p. 24, lines 1290-1293).

The main benefit of orthogonalization is better preservation of CLIPScore during joint editing. Mean CLIPScore increases from 0.514182 without orthogonalization to 0.535071 with orthogonalization. Compared with the baseline, the mean CLIPScore decrease is 0.035094 for non-orthogonal sequential editing and only 0.014205 for orthogonalized joint editing. Thus, orthogonalization reduces the decrease by approximately 59.5%. The minimum CLIPScore also improves from 0.165311 to 0.246950.

Both variants effectively reduce mentions of both attributes and achieve an MMBench score of 58.16%. These results demonstrate that Gram-Schmidt orthogonalization better preserves image-text alignment while maintaining effective joint attribute editing.

Thank you again for your constructive feedback, which helped us clarify the method and strengthen the evidence. We hope these explanations and additional results address your concerns, and would be grateful if you would consider raising your rating in light of this evidence.

Thank you very much for your response and additional experiments. I consider Q1 and Q2 adequately addressed.

However, my original concern in W1 remains. In your response, you state:

Although this direction may not capture finer-grained distinctions among groups within the same set, it captures the main differences between the two reference sets.

Table 1 shows that PLACE reduces mentions of race compared with the original models across all three models and both datasets. Since these results are aggregated across racial categories, it remains unclear whether mentions of categories within the same reference set are also reduced, and if so, why. Clarifying the scope of the method does not explain these empirical results, which was the main point of my original concern.

I appreciate the additional evidence provided in the rebuttal. However, considering the remaining limitations, including concerns raised by other reviewers about the prompt baseline, I intend to maintain my rating.

AuthorsOpenReview ↗

Follow-up on W1: Category-Level Race Mention Results on FairFace

Thank you for your follow-up and for confirming that Q1 and Q2 are adequately addressed by our responses and additional experiments. We appreciate your clarification that W1 concerns the empirical effects within each reference set. We therefore reanalyzed the FairFace generations separately for each original input image category.

Category-level results within both reference sets.

The analysis shows that mention rates decrease in 19 of the 21 model–category pairs, remain unchanged in one, and increase in one. All nine pairs in Set 1 show reductions; in Set 0, ten show reductions, one remains unchanged, and one increases. The exception is White inputs on Qwen3-VL-8B (0.156 → 0.193). Thus, the aggregate reduction is accompanied by reductions across multiple constituent categories within both reference sets, although the effect is not uniform across every category.

Each row reports the number of matches to any term in the fixed race-related lexicon per 1,000 generated words for inputs with the indicated image label. These are rates of all lexicon matches within each input category, rather than counts restricted to the category's own name. The arrows show Original → PLACE.

FairFace · LLaVA-7B

Reference setInput image categoryOriginal → PLACE
Set 0Indian1.948 → 1.427
Set 0Southeast Asian4.299 → 1.228
Set 0Latino/Hispanic0.752 → 0.439
Set 0White0.164 → 0.151
Set 1Black2.040 → 0.390
Set 1East Asian3.170 → 1.154
Set 1Middle Eastern0.928 → 0.361

FairFace · Qwen3-VL-8B

Reference setInput image categoryOriginal → PLACE
Set 0Indian2.277 → 1.334
Set 0Southeast Asian3.185 → 2.488
Set 0Latino/Hispanic0.602 → 0.357
Set 0White0.156 → 0.193
Set 1Black1.537 → 0.854
Set 1East Asian2.632 → 2.274
Set 1Middle Eastern0.573 → 0.414

FairFace · InternVL3.5-8B

Reference setInput image categoryOriginal → PLACE
Set 0Indian0.812 → 0.058
Set 0Southeast Asian0.419 → 0.000
Set 0Latino/Hispanic0.114 → 0.108
Set 0White0.000 → 0.000
Set 1Black0.927 → 0.225
Set 1East Asian0.379 → 0.192
Set 1Middle Eastern0.169 → 0.000

Interpretation.

One possible explanation is that representations from several categories in the same reference set have components along the pooled contrast direction. If those components contribute to generating race-related terms, attenuating them with the same edit could reduce mentions across multiple categories. The reference partition determines how the direction is estimated; it does not require every category within a set to respond identically to the intervention.

The table establishes that reductions occur across constituent input categories within each set. The proposed representation-level explanation remains a hypothesis, rather than a mechanism established by this analysis. The evidence concerns explicit race-related mentions on FairFace.

Thank you again for prompting this category-level analysis. These results directly address whether reductions also occur across constituent categories within the same reference set. We respectfully ask that this additional evidence be considered in your assessment of W1.

Thank you very much for the additional analysis. I consider Q1 and Q2 adequately addressed, and the new results provide useful evidence regarding W1. While the proposed explanation remains a hypothesis, I appreciate the additional insights provided by this analysis.

Overall, I deeply appreciate the comprehensive experimental evaluation and the authors' efforts during the discussion. The additional evidence strengthens the paper, but it does not substantially change my overall assessment of its contribution. I therefore intend to maintain my rating of 3 (Weak Accept).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Responses and Manuscript Revisions

Summary of Rebuttal Evidence and Remaining Limitations

We thank the reviewers and Area Chair for their constructive feedback. Our rebuttal added the following evidence and clarifications.

1. Separate versus shared axes.

We added a matched control holding layers, strengths, thresholds, samples, and decoding settings fixed. Separate axes yield 48.10 mentions per 1k generated words on LLaVA versus 53.21 with a shared axis. The reported shared-axis controls exhibit degeneration in 93/200 Qwen and 51/200 InternVL outputs, whereas separate-axis editing does not in these runs. Reviewer gX4d subsequently confirmed that this control addresses W1.

2. Direction reproducibility.

Ten resamples across three models, with shuffled-label controls and disjoint evaluation data, support reproducible gender-related directions within the evaluated models and layers. These results support the local approximation used by PLACE without establishing universal attribute geometry.

3. Configuration transparency and reuse.

We added 24 dataset-specific FairFace/SocialCounterfactuals configurations and clarified that original Table 4 concerns a development diagnostic. Configurations are selected on held-out development data and frozen for testing.

Directions and parameters can be stored and reused after calibration. However, labeled reference data and development-stage selection remain setup requirements, and reliable transfer to unseen domains has not been established.

4. One-pass prompting comparison.

Considering deployment latency, we evaluated the one-pass instruction “Avoid explicit and implicit {attribute} bias,” which requires no additional response-generation stage. On Qwen, the prompt achieves lower mention rates in five of six settings, while PLACE achieves lower VADER range in five of six. On InternVL, the prompt wins two of twelve comparisons. These results demonstrate model- and metric-dependent trade-offs.

Our one-pass instruction adapts the general bias-removal objective of Gallegos et al., cited by PAR; it does not reproduce PAR’s two-stage reprompting or explanation procedures. We will make this distinction explicit. The latency consideration motivates this baseline choice but does not replace an empirical comparison with the two-stage methods.

5. Joint editing.

We added a 1,000-sample FairFace assessment of age–gender editing, including orthogonal and sequential non-orthogonal controls, CLIPScore, and MMBench evaluation.

6. Remaining comparisons.

In the final-hours follow-up, gX4d clarified the need to evaluate both “Describe the image without mentioning the person’s age, gender, or race” and PAR’s two-stage prompting baselines. We agree that these comparisons are relevant. Direct mention suppression does not necessarily establish a reduction in VADER sentiment disparities, but its performance on both metrics requires empirical evaluation.

Limited rented-GPU availability and the remaining rebuttal time prevented us from completing and validating these additional experiments before the deadline. We acknowledge this evaluation gap and do not claim superiority over these untested baselines.

We hope the completed controls, transparent reporting of trade-offs, and clarified limitations help the reviewers and Area Chair assess the contribution and scope of PLACE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.