- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors discuss whether VLM prediction errors can be repaired through different module adaptations, via controlled fine-tuning and a series of analyses. They provide some observations on repair multiplicity and how different modules interact during joint training. However, this study only relies on a single Llava-onevision 0.5B mLLM. Neither family nor scale effect is discussed sufficiently. Furthermore, the core findings remain largely descriptive, and the corresponding evidence is still based on a single task.
Critical strengths
This paper shows a controlled experimental analysis in detail. The full-factorial design, module transplantation and causal interventions compose an overall analysis of how different VLM modules can repair observed behaviors, beyond some simple result comparisons.
Critical weaknesses
-
Severely limited scope and generalizability. The entire paper only relies on a single Llava-onevision 0.5B model with very limited baseline performance. There is no comparison across different families or sizes. Whether the findings from that model can still be applied to more general models or different tasks remains unclear.
-
Mechanistic discussions beyond descriptive observations are insufficient. The core repair multiplicity findings are mostly descriptive, and discussions or explanations on the joint-training dynamics are limited. Casual evidence emphasized by the authors is still based on a single task.
Questions for authors
-
Are the key findings robust enough on repair multiplicity and joint training, if more VLMs and tasks are included?
-
Can you provide any further mechanisms that determine repair routes during joint training, and verify them, rather than just optimization effects and interface incompatibility?
Response to Reviewer sDvC: added models, weak baseline, joint-training routes, causal scope
Thank you for the review. You raised three concerns: a single model (W1/Q1), a weak baseline, and no mechanism for what joint training does (W2/Q2). In short: of three added models, two reproduce all three of our repair-space statements and the third only one, and the findings about joint training transfer only in part; all four models are weak on these tasks, so the concern about the baseline stands; and we now have causal evidence on where repairs are computed and, from an exploratory test on G02 (shape selection), that training order can move the route, but none on why joint training picks one.
W1 / Q1 (other models). We repeated the behavioural experiment on three more models, training all seven sets with the paper's recipe: Qwen2-VL-2B and SmolVLM-Instruct, each with about 2.2B parameters, and SmolVLM-500M-Instruct with 0.5B. We fixed three predictions before training. Over half of the Base errors would be repaired by a single-module set, over a quarter by two or three of them separately, and under a quarter of each joint set's repairs would be made by no single-module set. All three hold in every run of Qwen2-VL-2B and SmolVLM-Instruct. In every SmolVLM-500M run only the third holds, and the first two shares are only 38.3–41.8% and 18.0–18.2% (Appendix I). We found no technical cause for this failure, and model size is confounded with image tiling between the two SmolVLM models. All added runs come from a separate pipeline with our untuned recipe.
These models do not answer your concern about the weak baseline. None of them has more than about 2.2B parameters, and none already solves these tasks: Base accuracy is 25.3–32.8% for the four models (Section 6). Our design needs seven fine-tuning runs per model and seed, plus the order runs. We judged that a 7B-scale model with repeated seeds could not be trained and checked with our compute within the discussion period, and we used that compute for three models with three seeds each. A 7B-scale model is now running, and we will add its results to the final version.
The findings about joint training transfer only in part. In LLaVA-OV, co-training costs V much of the repair it achieves alone, and L accounts for more of VL's repairs than V (Section 4.2, Figure 4). We tested both in the added models by transplanting VL's V, and separately VL's L, into Base. In eight of the nine added runs VL's V still repairs fewer Base errors than the V-only set, though by less than in any LLaVA-OV seed. In the ninth, SmolVLM-500M seed 1102, it repairs 94 more, with a 95% interval from −48 to +223.
The second finding does not transfer to the SmolVLM models. Before scoring any transplant we predicted that VL's L, placed alone in Base, would make more of VL's repairs than VL's V. It does in all three Qwen2-VL-2B runs and in none of the six runs of the two SmolVLM models. V makes more, with the interval of the difference above zero, in all three SmolVLM-Instruct runs and in SmolVLM-500M seed 1102.
W2 / Q2 (mechanism). The module-level results rest on interventions (transplants and reversions), but our causal analysis of how a repair is computed covers only G02 and B02 (circle contact) in LLaVA-OV-0.5B. On both tasks the V-only repair and the L-only repair follow different routes, and for each, final-token patches first recover half of the gain at layer 14. On G02 projecting V's final-token change onto L's recovers V's effect; on B02 the two changes align only in part (Section 4.3, Figure 6). Patching the jointly trained model itself on G02 shows that it, like the L-only set, repairs in text-side computation (Appendix E.4). We did not patch B04 (overlapping-shape counts).
Two questions stay open: why co-training lowers what V repairs alone, and how joint training picks a route. On the first, a control freezes L but keeps its gradients in the shared clipping norm. In all eight seeds, at 10% of training, where this control stops, its V update points almost the same way as the V-only update and is no smaller. In VL, then, it is L's movement that turns V's update away from the V-only direction (Appendix D.6). We have not shown that this is why V loses repairs (Appendix D.7).
On the second question, training order matters in LLaVA-OV. In an exploratory test, when V trains alone for the first half, the final model's G02 repair depends far more on early-layer states at the image positions than when L trains first or both train together (eight seeds, Appendix E.5). If each module gets the same number of updates, one after the other, the V-first model still shows this dependence in weaker form (seven seeds). Since these equal-update runs have no joint phase, they show that order can move the route but not how joint training picks one. On the three added models, which we did not patch, the module trained first carries more of the repairs in all nine pairs of equal-update runs, whether L, V or neither carries more under joint training (Appendix I.1).