acceptodds
Under review as a conference paper at ICLR 2027

One Failure, Many Repairs: Module-Level Repair Spaces in Vision–Language Models

Abstract

When a vision–language model (VLM) gets a question wrong, is there only one internal way to make it right? From one LLaVA-OneVision-0.5B checkpoint (Base) we fine-tune, with each of two training seeds, all seven non-empty subsets of the vision encoder (V), projector (P), and language model (L) under identical data, exposure, and example order, and trace every error on 13 tasks of 2D geometry through the design; since untrained modules stay bitwise equal to Base, trained modules can also be transplanted, reverted, and recombined. We ask three questions (values: seed 1101/1104). RQ1: which repairs exist? Of Base errors, 77.9/76.8% are repaired by some single-module adaptation and 45.2/43.5% by two or three different ones (44.0/50.0% when every model's task-level answer prior is removed); repairs that only a pair or the triple make are rare and form a different, non-additive regime. RQ2: what does joint training build? It mostly reorganizes available repairs instead of adding new ones (at most 7.0/8.5% of a joint set's repairs are made by no single set); V co-trained with L keeps only 38/39% of its stand-alone gain; the synergy of the analysed V+L pair-emergent repairs mostly survives recombination across independent runs, so it mostly does not require run-private code; and the training path can move which module carries a repair, at an accuracy cost when V trains first. RQ3: why are several repairs available? In the case analysed in depth (G02), Base already encodes the needed information, and V-only and L-only adaptation fix the failure along different causal routes, V-only through image keys and values that the frozen language model reads, L-only by changing L's own text computation; both meet in one late decision state, so where parameters adapt is not where the repair is computed. A behavioural failure need not correspond to a unique defect or repair location.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, mechanistic interpretability, fine-tuning, module-level repair, factorial design, module transplantation, activation patching, geometric perception

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sDvCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors discuss whether VLM prediction errors can be repaired through different module adaptations, via controlled fine-tuning and a series of analyses. They provide some observations on repair multiplicity and how different modules interact during joint training. However, this study only relies on a single Llava-onevision 0.5B mLLM. Neither family nor scale effect is discussed sufficiently. Furthermore, the core findings remain largely descriptive, and the corresponding evidence is still based on a single task.

Critical strengths

This paper shows a controlled experimental analysis in detail. The full-factorial design, module transplantation and causal interventions compose an overall analysis of how different VLM modules can repair observed behaviors, beyond some simple result comparisons.

Critical weaknesses
  1. Severely limited scope and generalizability. The entire paper only relies on a single Llava-onevision 0.5B model with very limited baseline performance. There is no comparison across different families or sizes. Whether the findings from that model can still be applied to more general models or different tasks remains unclear.

  2. Mechanistic discussions beyond descriptive observations are insufficient. The core repair multiplicity findings are mostly descriptive, and discussions or explanations on the joint-training dynamics are limited. Casual evidence emphasized by the authors is still based on a single task.

Questions for authors
  1. Are the key findings robust enough on repair multiplicity and joint training, if more VLMs and tasks are included?

  2. Can you provide any further mechanisms that determine repair routes during joint training, and verify them, rather than just optimization effects and interface incompatibility?

1 comment
AuthorsOpenReview ↗

Response to Reviewer sDvC: added models, weak baseline, joint-training routes, causal scope

Thank you for the review. You raised three concerns: a single model (W1/Q1), a weak baseline, and no mechanism for what joint training does (W2/Q2). In short: of three added models, two reproduce all three of our repair-space statements and the third only one, and the findings about joint training transfer only in part; all four models are weak on these tasks, so the concern about the baseline stands; and we now have causal evidence on where repairs are computed and, from an exploratory test on G02 (shape selection), that training order can move the route, but none on why joint training picks one.

W1 / Q1 (other models). We repeated the behavioural experiment on three more models, training all seven sets with the paper's recipe: Qwen2-VL-2B and SmolVLM-Instruct, each with about 2.2B parameters, and SmolVLM-500M-Instruct with 0.5B. We fixed three predictions before training. Over half of the Base errors would be repaired by a single-module set, over a quarter by two or three of them separately, and under a quarter of each joint set's repairs would be made by no single-module set. All three hold in every run of Qwen2-VL-2B and SmolVLM-Instruct. In every SmolVLM-500M run only the third holds, and the first two shares are only 38.3–41.8% and 18.0–18.2% (Appendix I). We found no technical cause for this failure, and model size is confounded with image tiling between the two SmolVLM models. All added runs come from a separate pipeline with our untuned recipe.

These models do not answer your concern about the weak baseline. None of them has more than about 2.2B parameters, and none already solves these tasks: Base accuracy is 25.3–32.8% for the four models (Section 6). Our design needs seven fine-tuning runs per model and seed, plus the order runs. We judged that a 7B-scale model with repeated seeds could not be trained and checked with our compute within the discussion period, and we used that compute for three models with three seeds each. A 7B-scale model is now running, and we will add its results to the final version.

The findings about joint training transfer only in part. In LLaVA-OV, co-training costs V much of the repair it achieves alone, and L accounts for more of VL's repairs than V (Section 4.2, Figure 4). We tested both in the added models by transplanting VL's V, and separately VL's L, into Base. In eight of the nine added runs VL's V still repairs fewer Base errors than the V-only set, though by less than in any LLaVA-OV seed. In the ninth, SmolVLM-500M seed 1102, it repairs 94 more, with a 95% interval from −48 to +223.

The second finding does not transfer to the SmolVLM models. Before scoring any transplant we predicted that VL's L, placed alone in Base, would make more of VL's repairs than VL's V. It does in all three Qwen2-VL-2B runs and in none of the six runs of the two SmolVLM models. V makes more, with the interval of the difference above zero, in all three SmolVLM-Instruct runs and in SmolVLM-500M seed 1102.

W2 / Q2 (mechanism). The module-level results rest on interventions (transplants and reversions), but our causal analysis of how a repair is computed covers only G02 and B02 (circle contact) in LLaVA-OV-0.5B. On both tasks the V-only repair and the L-only repair follow different routes, and for each, final-token patches first recover half of the gain at layer 14. On G02 projecting V's final-token change onto L's recovers V's effect; on B02 the two changes align only in part (Section 4.3, Figure 6). Patching the jointly trained model itself on G02 shows that it, like the L-only set, repairs in text-side computation (Appendix E.4). We did not patch B04 (overlapping-shape counts).

Two questions stay open: why co-training lowers what V repairs alone, and how joint training picks a route. On the first, a control freezes L but keeps its gradients in the shared clipping norm. In all eight seeds, at 10% of training, where this control stops, its V update points almost the same way as the V-only update and is no smaller. In VL, then, it is L's movement that turns V's update away from the V-only direction (Appendix D.6). We have not shown that this is why V loses repairs (Appendix D.7).

On the second question, training order matters in LLaVA-OV. In an exploratory test, when V trains alone for the first half, the final model's G02 repair depends far more on early-layer states at the image positions than when L trains first or both train together (eight seeds, Appendix E.5). If each module gets the same number of updates, one after the other, the V-first model still shows this dependence in weaker form (seven seeds). Since these equal-update runs have no joint phase, they show that order can move the route but not how joint training picks one. On the three added models, which we did not patch, the module trained first carries more of the repairs in all nine pairs of equal-update runs, whether L, V or neither carries more under joint training (Appendix I.1).

Reviewer_QPrCOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the same failure in a vision–language model can be corrected through adaptations to different modules. Starting from a shared LLaVA-OneVision-0.5B checkpoint, the authors fine-tune all seven non-empty combinations of the vision encoder, projector, and language model on 13 geometric reasoning and perception tasks. They characterize item-level repair patterns and use module transplantation, reversion, recombination, and activation interventions to investigate how these repairs are realized. The main findings are that many failures admit multiple single-module repairs, joint training often reorganizes existing repair solutions, and different adaptations can affect downstream computation through distinct routes.

Critical strengths
  1. A systematic framework for comparing repairs at the example level. The shared initialization and complete factorial design allow the authors to compare which specific failures are corrected by different adaptation choices. Module transplantation and reversion extend this analysis beyond aggregate accuracy. This is a useful experimental framework for studying adaptation in modular multimodal systems.
  2. Mechanistic evidence beyond behavioral overlap. The analysis of G02 suggests that vision-only and language-only adaptation can improve the same predictions through different upstream routes while affecting a shared downstream decision direction. The joint-training and recombination experiments also provide evidence that independently available repairs need not be preserved during joint adaptation. These findings make the paper more informative than a comparison of fine-tuning configurations alone.
Critical weaknesses
  1. The distinctive mechanistic contribution and resulting takeaways are insufficiently distilled. The observation that multiple sufficiently expressive modules can be adapted to correct the same examples is not, by itself, particularly surprising. Although the paper contains stronger findings, it does not clearly prioritize which mechanistic regularities constitute its main advance beyond this observation. This issue is amplified by the presentation. The paper would benefit from a single explanatory figure showing a concrete failed example, the alternative adaptation choices, the transplantation procedure, and the resulting mechanistic findings. A dedicated conclusion should also distinguish established findings from task-specific observations and explain what decisions they support—for example, when to consider single-module adaptation, when joint training merits investigation, and what transplantation results can and cannot diagnose. Without this synthesis, the significance and practical usefulness of the extensive experiments remain harder to assess.
  2. The observed repair categories may depend substantially on the optimization recipe. A single-module run failing to repair an example does not establish that the module lacks the capacity to do so. Its failure could reflect the learning rate, training duration, or interference from multitask training. Consequently, categories such as pair-emergent and triple-emergent should be interpreted as outcomes of the tested procedure, rather than evidence that multiple modules inherently need to change. The authors acknowledge this limitation, but targeted learning-rate and training-budget sensitivity experiments would materially strengthen the central claims. The training-order experiment also changes the optimization opportunities available to each module, making it difficult to isolate ordering from unequal training exposure and learning-rate schedules.
Questions for authors
  1. Clarification—mechanistic novelty: Beyond the prevalence of overlapping repairs, which one or two findings do the authors regard as the central new mechanistic insights? Which are supported across tasks, and which should remain explicitly task-specific?
  2. Additional evidence—optimization robustness: Do the main repair categories and joint-training conclusions persist when single-module runs receive alternative learning rates or longer training? A targeted analysis on G02 and B04 would be particularly informative. Can the training-order result also be tested with more closely matched per-module optimization budgets?
  3. Presentation clarification: Could the authors provide an overview figure connecting a concrete example to the factorial training design, module interventions, and main findings? This would help distinguish behavioral repair overlap from evidence about the underlying computation.
2 comments
AuthorsOpenReview ↗

Response to Reviewer QPrC (1/2): W1/Q1 central findings, W2/Q2 learning rate and budget

Thank you for the review. You asked what the central insight is (W1/Q1), how far the results depend on the learning rates and the budget (W2/Q2), and for an overview figure (Q3). In short: Table 1 and a new Conclusion state three findings and their reach; new controls vary the learning-rate ratio, the budget, the single-module recipes and, with equal updates, the order of training; and these controls made us narrow one claim, because which failures seem to need joint training depends on the recipe and the run. A second comment covers the order control, two checks on B04 (overlapping-shape counts) and Q3.

W1 / Q1 (the central insight). We agree that the submitted version left its central finding implicit. In LLaVA-OV-0.5B a failure often has several single-module repairs, these repairs take different internal routes on the two tasks we patched, and most errors that a joint set repairs are ones that a single-module set also repairs. Pooled over the 13 tasks, the first and the third hold in all eight seeds, although the stricter test of the third in Section 4.2 fails in seed 1105. Multiplicity on its own is to be expected. The paper measures it (RQ1) and builds RQ2 and RQ3 on it.

Repairs that no single module makes are few. Even in each seed's joint set with the most of them, they are 7.5 ± 1.7% of its repairs and over 10% only in seed 1105. Only 2.6% of the Base errors, most of them on B04, are repaired by a joint set in a majority of seeds and by no single module in any seed. The Conclusion (Section 5) advises retuning or re-running a single module and checking other seeds before calling a failure joint-only.

We patched only LLaVA-OV, on G02 (shape selection) in full and on B02 (circle contact) in reduced form. On both tasks the V-only and the L-only repair take different routes. On G02 projecting V's final-token change onto L's recovers V's effect, and on B02 the two changes align only in part.

In an exploratory test, patching the joint model on G02 shows that training order changes how it repairs G02, more weakly with equal updates (Section 4.3, Appendices E.4–E.5). This is evidence, not a mechanism: what decides the route when both modules train from the start is open.

W2 / Q2 (dependence on the training recipe). You are right that the set of Base errors that seem to need two modules depends on the recipe, and even on the run. In LLaVA-OV, two results survive the controls below. Failures still often have several single-module repairs. And when VL's L and VL's V are each placed alone into Base, L accounts for more of VL's repairs than V, pooled over tasks, at every learning-rate ratio and budget tried. Among the added models the first fails in all three SmolVLM-500M runs, and the second holds only in the three Qwen2-VL-2B runs (Appendix I).

For the learning-rate ratio, we retrained VL in seeds 1103 to 1105 with the V-to-L ratio moved four-fold in either direction at the same geometric mean (Appendix E.1). We added seed 1104 to this control and the single-module ones below after the results of seeds 1103 and 1105 were known. Against a re-run of VL at the paper's ratio, favouring V raises V's share of VL's repairs from 21.9–24.7% to 43.5–45.1%, still 15.5–20.9 pp below L's, and costs 4.4–6.6 pp of accuracy. Favouring V makes V account for more of VL's repairs than L on B04, but not on G02, where L accounts for more even when VL is trained for twice the updates at the paper's ratio (Appendix E.3). Pooled over tasks, doubled VL stays L-dominant in all three seeds, with a shift toward V that lies above zero in seeds 1104 and 1105 but within the 10 pp we had set for stable dominance.

For single modules, in seeds 1103 to 1105 we trained only V at L's learning rate and only L at V's, re-ran each at the paper's rates, and trained each alone for twice the updates at the paper's rates on a stretched schedule (Appendix E.3). The re-runs are not exact replicates of the paper's V-only and L-only sets, because the clipping norm spans different modules.

The test concerns VL's pair-specific items, the Base errors that VL repairs and both the V-only and the L-only set leave wrong. Pooled over tasks, no single-module run repairs half of them, the bar fixed in advance. On B04 the best run per seed repairs 16.2–56.6% of them. It is a retuned run in seeds 1103 and 1105 and the L-only re-run at the paper's rates in seed 1104 (22 of 136 items). G02 has only one to three such items per seed, too few to show whether a recipe converts them.

In a post hoc recount over the three seeds we treated every single-module control run as one more run of its module. The share of Base errors that only a pair of modules repairs then falls from 5.8–6.2% to 2.0–2.7%, and the re-runs at the paper's rates alone account for 39.0–64.5% of the items that leave this class, and for most of them in seed 1104. The abstract and Sections 3.3, 4.1 and 5 now state this dependence.

AuthorsOpenReview ↗

Response to Reviewer QPrC (2/2): W2/Q2 order with equal updates, B04 checks, Q3 overview figure

In short: with equal updates the effect of training order is weaker in LLaVA-OV but remains, and it appears in all nine pairs of runs on the added models; of two further checks on B04 (overlapping-shape counts), neither settles why its joint-only repairs arise; and Figure 1 is the new overview.

W2 / Q2, continued (training order with equal updates). You are right that in our order experiment the module trained first also received more updates. In the new control one module trains alone for updates 1–1220 and the other for updates 1221–2439. We ran both orders in seeds 1102–1108, adding all but 1103 and 1105 after the results of those two were known, and compare them with the original design in the same seeds (Figure 5, Appendix E.2). The control separates order from the number of updates, but because V and L never train together here, it says nothing about a joint phase with equal updates.

The carry index is the share of the final model's repairs made by its V alone in Base minus the share made by its L alone. Training V first instead of L first raises the carry index by +90.7 to +121.1 pp. That is less than the +140.6 to +163.8 pp of the original design, and the two intervals do not overlap in any seed, so the effect is weaker in this control but does not disappear. We had predicted that the module trained first would account for more of the repairs. This holds in all seven L-first models but in only six of the V-first ones, because in seed 1105 neither carries detectably more.

The control has a cost in accuracy. In six of the seven seeds each order scores below a re-run of our simultaneous schedule, with an interval that excludes zero, but the seed without a cost differs between the two orders. We also patched both final models on G02 (shape selection; Appendix E.5). Restoring Base's image-position states at layer 3 removes 31.9–37.8% of the V-first model's gain, compared with 51.7–55.2% for the original V-first model in the same seeds. In the L-first model no tested layer removes more than 10.7%.

On the three added models we ran both orders with our separate pipeline and untuned recipe, three pairs of runs per model. In all nine pairs the module trained first carries more of the repairs, whether L, V or neither carries more under joint training. Which order is less accurate differs between models. In Qwen2-VL-2B seeds 1101 and 1102 the run with V first failed our check that its first update reproduces that of the V-only set, in gradient norm only. Repeats of that update did not differ at the 0.05 level (in seed 1101 we ran this test after its run had started), but we report both checks as failed (Appendix I.1).

W2 / Q2, continued (two further checks on B04). The first check concerns interference from the other tasks. It uses VL's pair-emergent B04 items, which no single set repairs. We retrained V and L separately on B04 alone, with the same number of B04 exposures (6,000) but 189 updates instead of 2,439. In every seed, neither repairs more than 9% of these items. Under a rule we had fixed in advance, this argues against interference in four seeds. In the other four the test is inconclusive, because the retrained V reproduces under half of the original V-only set's B04 repairs. Because the number of updates is not matched, and B04's joint-only repairs also depend on the recipe and on the run, we regard this test as weak evidence either way (Appendices H.6 and E.3).

The second check concerns our own statement that VL's pair-emergent B04 repairs concentrate on even counts. The L-only set answers an even count on at most 3.4% of all B04 items, against 57.0% for Base. We therefore re-scored B04 with only the candidates of the correct parity (Appendix H.5). With that scoring the L-only set picks the right even count more often than Base in every seed, so an output prior explains most of its failures on even counts. For VL's pair emergence on even counts the check is inconclusive. Our pre-set rule for a prior effect, a fall of the share of Base's even-count errors that VL repairs and neither V nor L does by more than half, with an interval above zero, is met only in seed 1106 (28.7% to 7.9%). In the other seven seeds the interval of the change includes zero: the share falls in four (in seed 1108 from 21.1% to 12.2%) and rises in three, and its mean does not fall (16.8 ± 7.7% to 20.4 ± 16.1%). Nor does the check show whether an even item needs a V and an L trained jointly rather than separately. We revised the even-count statements in Section 4.1 and Appendix H.4 accordingly.

Q3 (overview figure). We added Figure 1. It follows one item of B02 (circle contact) that Base gets wrong through the three research questions: the V-only and the L-only set both repair it (b), in seed 1101 VPL's V alone repairs it in Base and VPL still repairs it when its V is reset to Base (c), and (d) sketches our task-level patching results on G02 and B02, whose values Figure 6 shows.

Reviewer_EcN8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether the same failure in a VLM can be corrected by independently fine-tuning different modules. Using LLaVA-OneVision-0.5B, the authors fine-tune all seven non-empty combinations of the vision encoder, projector, and language model across 13 diagram-based geometry tasks. The results show that many failures can be corrected through multiple, alternative single-module adaptations. Joint training generally reallocates existing repair capabilities rather than creating entirely new ones. Through module transplantation and causal interventions, the authors further demonstrate that different adaptations can follow distinct computational routes while reaching the same decision state. The central conclusion is that a model's behavioral failure does not necessarily identify a unique internal defect or repair location.

Critical strengths
  1. The paper moves beyond comparing fine-tuning strategies by investigating whether individual failures admit multiple, distinct repair solutions.
  2. The factorial training setup controls data exposure and example order, while module transplantation, reversion, and recombination provide useful ways to investigate repair sufficiency and ownership.
  3. Rather than relying solely on aggregate accuracy, the authors examine which specific errors each adaptation fixes, which repairs are lost, and which emerge through joint training.
  4. The causal patching experiments provide evidence that vision-only and language-only adaptations can follow different computational routes yet converge on a common decision state.
  5. The study covers 13 tasks, evaluates two seeds in depth, and includes additional behavioral checks with a third seed. It also examines alternative explanations, including answer biases and potential shortcuts.
Critical weaknesses
  1. The detailed causal analysis focuses primarily on the G02 reference task. It remains unclear whether the same mechanisms explain repair multiplicity across other tasks or VLM architectures.
  2. Pair and triple emergent repairs show relatively poor replication across seeds outside B04. This weakens confidence that these effects represent stable properties of the model rather than training variability.
  3. Transplantation establishes whether a module can preserve a repair in the Base model's context, but this does not necessarily identify where the computation is performed in the original jointly trained model. The ownership analysis, therefore, depends on the operational definitions used.
  4. Correctness is determined by ranking predefined answer options using conditional log-likelihoods. The resulting repair patterns may differ under free-form generation or alternative scoring procedures.
  5. The study provides compelling evidence of multiple repair routes in specific settings, but its findings on diagram-based geometry do not yet establish that similar repair spaces occur in more general multimodal reasoning tasks.
Questions for authors

Please see the mentioned weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer EcN8 (1/2): W1 other tasks and architectures, W2 joint-only stability

Thank you for the review. You asked whether the causal picture holds beyond one task and one model (W1), whether the emergent classes are stable (W2), where the joint model itself computes (W3), and how the results depend on scoring (W4) and on the domain (W5). In short: patching of a second task now covers eight seeds, and there the two repairs align only in part, whereas on G02 (shape selection) projecting V's final-token change onto L's recovers V's effect; two of three added models reproduce all three of our repair-space statements, and the third only one of them; and the joint-only classes turned out to depend on the run and the recipe, so we now limit claims about pair emergence to B04 (overlapping-shape counts). This comment answers W1 and W2, and a second one W3 to W5. All section, figure and appendix numbers refer to the revised PDF.

W1 (one task, one model). The second task is B02 (circle contact). The submitted version applied a reduced form of the G02 patching design to B02 in two seeds, and we have now run it in all eight. The rule for a shared decision state was fixed before any aggregate B02 patching result. The other claim thresholds were set after seeds 1101 and 1104, so only the other six seeds are replications. In every seed the V-only and the L-only set again repair B02 through different routes, and for both, final-token patches first recover half of the gain at layer 14. But the two repairs align only in part. In each seed we project the V-only change at the final token onto the L-only change. This recovers 1.03 ± 0.04 of the V-only effect on G02 but 0.35 ± 0.03 on B02 (mean ± SD over seeds), and the rule we had fixed for a shared decision state like G02's is met in no seed. We therefore do not claim a single mechanism behind multiplicity (Appendix G.4, Section 4.3, Figures 1d and 6c). Patching still covers only two of the 13 tasks, and not B04.

We did not patch other models. Instead we trained all seven sets on Qwen2-VL-2B, SmolVLM-Instruct and SmolVLM-500M-Instruct, three runs per model, and transplanted VL's V and VL's L into Base. Before training we wrote down three predictions: more than half of the Base errors would have a single-module repair, more than a quarter would have two or three such repairs, and for each joint set fewer than a quarter of its repairs would come from no single-module set. All three hold in all three runs of Qwen2-VL-2B and of SmolVLM-Instruct. On SmolVLM-500M only the third holds, for VL in seed 1102 by point estimate only. In its three runs just 38.3–41.8% of the Base errors have a single-module repair and 18.0–18.2% have two or three, and we could not find a technical cause for this.

The transplants also differ between models. In all three Qwen2-VL-2B runs VL's L, placed alone into Base, makes more of VL's repairs than VL's V does. In the six runs of the two SmolVLM models L never does, and in four of them V makes more, with the interval of the difference above zero. Because these runs come from a separate pipeline with our untuned recipe and three runs per model, we do not attribute the differences to model family, size or tokenisation (Appendix I).

W2 (stability of the emergent classes). You are right, and your point changed our claims. Outside B04, whether an item is pair- or triple-emergent mostly does not carry over from one run to another. On B04 such items recur more often, but even there the items that only joint sets repair change with the recipe and with the run.

Across seeds, we counted how often an item that is pair- or triple-emergent in one seed gets a single-module repair in another. This happens to 74.0 ± 5.6% of such items outside B04 and to 29.3 ± 8.0% of them on B04 (mean ± SD over eight seeds). We call a Base error seed-robust joint-only when a joint set repairs it in most seeds and no single-module set repairs it in any seed. Such errors are 2.6% [1.7, 3.6] of all Base errors, 16.3% of the Base errors on B04 and 0.9% of those on the other tasks. Multiplicity is much more stable. For 42.4% [39.8, 44.9] of all Base errors, at least two single-module sets each repair the error in five or more of the eight seeds (Appendix C.9).

The recipe matters as well. In each of three seeds we added six runs that train V or L alone, at the paper's rates, at the other module's learning rate, or for twice the updates on a stretched schedule. Of the B04 errors that VL repairs and neither the V-only nor the L-only set does, the best added run repairs 16.2–56.6%, depending on the seed. In two seeds that best run uses retuned settings, but in seed 1104 it is an L-only run at the paper's own rates, which repairs 22 of 136 such errors. This run is not an exact re-run of the paper's L-only set, because it clips gradients over a different set of modules (Appendix E.3). The abstract, Table 1 and Sections 3.3 and 4.1 now state that the emergent classes depend on the run and the recipe.

AuthorsOpenReview ↗

Response to Reviewer EcN8 (2/2): W3 where the joint model computes, W4 generation, W5 domain

In short, for W3 to W5: we patched the jointly trained model itself on G02 (shape selection), and we now read a transplant only as what a module repairs alone in Base (W3); free-form generation lowers the multiplicity counts, in part through how answers are read, and changes the joint-only share less than it varies across seeds (W4); and we tested no task outside 2D geometry, which we are running now (W5).

W3 (where the joint model computes). You are right that a transplant only shows what a module can repair by itself in Base, and Section 3.3 now defines ownership in this operational sense. To see where the joint model itself computes its repair, we patched it directly. We did this only on G02, where the transplants assign VL's repairs to L.

In all eight seeds we patched VL one layer at a time (Section 4.3, Appendix E.4). We measure a repair by its margin gain over Base, where the margin is the log-likelihood lead of the correct answer over the best wrong one. Giving VL the image-position states of Base costs it at most 12.3% of this gain at any layer we tested. In the other direction, giving Base the text-position states of VL recovers half of VL's gain by layer 6 or 7. We had fixed no threshold in advance for the loss at a single layer, so we report these numbers without a test.

We then patched the models from the training-order runs (Appendix E.5). In an exploratory test, when V was trained first, the model loses 51.7–56.5% of its gain if Base's image-position states are put back at layer 3. In the L-first model, in VL and in its re-run, no tested layer costs more than 14.4%. A second comparison, the direction of the final-token change, did not separate the routes: all eight seeds contradicted our prediction that the V-first model's final-token change would lie more along the V-only change.

In those runs the first module also trained for longer. We therefore ran a control in seven seeds that trains the two modules one after the other with the same number of updates. It separates order from the number of updates, though without a joint phase it cannot show how joint training with equal updates would behave. The image-side dependence is weaker there but still present. At layer 3 the V-then-L control loses 31.9–37.8% of its gain, less than the original V-first model, while the L-then-V control loses at most 10.7% at any tested layer.

Two limits remain. First, the order runs show that the order of training can move the route, but not what decides the route when both modules train together from the start. Second, we still do not know where VL computes the repairs that only it makes. On G02 VL makes at most three per seed that neither the V-only nor the L-only set makes. One of the two rules we had fixed for these items failed in six of the seven seeds that have any, but with so few items neither outcome is evidence.

W4 (free-form generation). We re-scored Base and all seven sets by greedy generation in all eight seeds (Section 4.1, Appendix C.3). We parse each output with three rules, all fixed before any generation existed and one of them primary, and count an output that is not a legal answer as wrong.

Generation lowers multiplicity, the share of Base errors that two or more single-module sets repair, by an amount that depends on how answers are read. If we take the errors that Base makes under ranking and score the trained sets by generation, this share falls from 41.4–53.9% to 26.2–40.2% under the primary rule and to 38.3–49.8% under a rule we added post hoc. The primary rule ignores letter case, so on B03 (circled letter) it can read no answer (the letters A–Z and a–z) of any model. The post hoc rule keeps case and also reads number words. The comparison is itself post hoc: we chose it after reading the generations of two seeds. Under the primary rule, most of the repairs that ranking finds and generation misses are rows where the generation cannot be read as a legal answer; the trained sets give a legal answer in 61–87% of their generations, and in 80–97% under the post hoc rule. Where a generation is legal under the primary rule, it agrees with the top-ranked answer on 99.4–100.0% of rows for the sets that train L, and on 93.9–96.8% for P, V and VP.

The share of Base errors that only joint sets repair moves less between the two scorings than it varies across seeds. Under ranking it is 5.5–9.0% across seeds. On ranking's Base errors, none of the four rules (three pre-set, one post hoc) changes it by more than 1.02 pp, and with each scoring's own Base errors by at most 2.00 pp. The paper keeps ranking as its main scoring because ranking can score every task.

W5 (other domains). We agree. The revision tests no task outside 2D geometry, for LLaVA-OV or the three added models, and its abstract, Introduction, Limitations and Conclusion restrict the claims accordingly. We are now running the seven-set design on tasks outside 2D geometry and will add the results to the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: what the revision shows, what we narrowed, and what remains open

Thank you for the reviews, and we apologise for the late response. To address your concerns, we expanded the main study from two to eight seeds, evaluated all seven module subsets on three additional VLMs with three seeds each, extended activation patching on a second task to eight seeds, and added training-order, learning-rate, and budget controls. Completing and validating these experiments with limited access to shared compute took longer than expected. We have uploaded a revised PDF, and all section, table and appendix numbers below refer to it. It now calls the tasks by name; our replies keep the codes of the submission, mainly G02 (shape selection), B02 (circle contact) and B04 (overlapping-shape counts).

What the revision shows

  1. One failure, several repairs. In LLaVA-OV-0.5B, 77.2 ± 0.6% of Base errors are repaired by training a single module and 46.1 ± 4.0% by each of two or three (mean ± SD over eight seeds, up from two). Prespecified versions of these statements hold in every run of two added 2.2B models and fail in every run of SmolVLM-500M (Section 4.1, Appendix I).
  2. What joint training builds. Most errors that a joint set repairs are also repaired by a single-module set. The module trained first carries more of the repairs. With equal updates this effect is weaker in LLaVA-OV, where a V trained first carries detectably more in six of seven seeds, and it appears in all nine pairs of runs on the added models (Section 4.2).
  3. Routes. On both tasks we patched, the V-only and the L-only repair take different routes. On G02 they converge at the final token, where projecting V's change onto L's recovers V's effect; on B02, now patched in all eight seeds, the two changes align only in part. Patching the joint model on G02 shows that the modules that adapt need not determine where it computes the repair: VL repairs in text-side computation, but with V trained first the joint model depends on early image states (exploratory; Section 4.3).

What we narrowed

  • Whether a failure seems to need joint training depends on the recipe and the run. We now limit claims about pair emergence to B04, and treat joint-only classes as outcomes of a run and a recipe.
  • L's dominance under joint training holds for LLaVA-OV and, more weakly, for Qwen2-VL-2B, but not for the two SmolVLM models.
  • Free-form generation lowers the multiplicity counts, partly through how answers are read (Appendix C.3).

What remains open

  • Stronger models. All four models have a Base accuracy of 25.3–32.8% on these tasks and at most about 2.2B parameters. Our design needs seven fine-tuning runs per model and seed, and we spent our compute on three more models with three seeds each.
  • Tasks outside 2D geometry, and patching beyond two tasks in one model.
  • What decides the carrier when modules train together, and why co-training lowers what V repairs alone.

We are currently running further experiments on these points and will add the results to the final version. Because of the limited time, some of the new controls cover fewer seeds than the main study (three or seven of the eight), the added models have three seeds each, and patching on B02 uses a reduced design. Runs for the missing control seeds and for further seeds of the added models are now running, and the full B02 design will follow; we will add all of these to the final version as well.

For each new analysis we fixed the items, statistics and decision rules before looking at its results, and we report the tests that failed (the exceptions are listed at the start of Appendix E and in Appendix I). The added models ran in a separate pipeline. Two of its validation checks missed their criterion: a loss check, whose criterion we then redefined on fp32 logits, where both sides agree, and a comparison of cached with full-forward scoring of the multi-token answers of B06 (grid rows/columns; 15 of 528 rows flipped), after which we kept full forward passes for the added models. A check on the first update of two Qwen2-VL-2B order runs failed and is reported as failed (Appendices I and I.1). Table 1 states each finding with its evidence and reach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.