Where to Edit Matters: Position-Aware Attribute Control in LVLMs

Open-ended descriptions from large vision–language models can foreground protected attributes in applications that favor demographically neutral descriptions, creating a trade-off between attribute-expression control and generation utility. Attribute-linked representations vary across token positions and layers, making intervention location and strength central design choices. We introduce **PLACE** (Position-Aware Layerwise Attribute-Component Editing), a training-free method that estimates separate attribute directions for visual and text positions and attenuates their residual components with independently controlled strengths at selected layers. Within-position orthogonalization extends the method to joint control of multiple protected attributes. A conditional model-internal formulation describes these edits within the frozen computation, linking the control settings to generated responses. Across four LVLM settings and three datasets, PLACE achieves the lowest fixed-lexicon and semantic mention rates in 31/36 and 29/36 comparisons, respectively. On FairFace, gender mentions fall by 30.2% on Qwen3-VL and 16.7% on LLaVA-7B relative to the original models, alongside lower VADER-based group disparity. Position ablations favor combined editing, while development comparisons identify gains from strength and layer selection. General-task, response-validity, output-length, and image–text-alignment assessments show stable performance on the reported measures. Anonymous code is available at [https://anonymous.4open.science/r/516721A](https://anonymous.4open.science/r/516721A).