acceptodds
Under review as a conference paper at ICLR 2027

One Failure, Many Repairs: Module-Level Repair Spaces in Vision–Language Models

Abstract

When a vision–language model (VLM) gets a question wrong, is there only one internal way to make it right? From one LLaVA-OneVision-0.5B checkpoint (Base) we fine-tune, with each of two training seeds, all seven non-empty subsets of the vision encoder (V), projector (P), and language model (L) under identical data, exposure, and example order, and trace every error on 13 tasks of 2D geometry through the design; since untrained modules stay bitwise equal to Base, trained modules can also be transplanted, reverted, and recombined. We ask three questions (values: seed 1101/1104). RQ1: which repairs exist? Of Base errors, 77.9/76.8% are repaired by some single-module adaptation and 45.2/43.5% by two or three different ones (44.0/50.0% when every model's task-level answer prior is removed); repairs that only a pair or the triple make are rare and form a different, non-additive regime. RQ2: what does joint training build? It mostly reorganizes available repairs instead of adding new ones (at most 7.0/8.5% of a joint set's repairs are made by no single set); V co-trained with L keeps only 38/39% of its stand-alone gain; the synergy of the analysed V+L pair-emergent repairs mostly survives recombination across independent runs, so it mostly does not require run-private code; and the training path can move which module carries a repair, at an accuracy cost when V trains first. RQ3: why are several repairs available? In the case analysed in depth (G02), Base already encodes the needed information, and V-only and L-only adaptation fix the failure along different causal routes, V-only through image keys and values that the frozen language model reads, L-only by changing L's own text computation; both meet in one late decision state, so where parameters adapt is not where the repair is computed. A behavioural failure need not correspond to a unique defect or repair location.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.