- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents VLMintune, a library for developing and comparing visual instruction-tuning methods. Its main idea is to place model changes, trainable parameters, method-specific inputs, supervision, loss, and checkpoint loading within a common method interface. Data processing, training, generation, and scoring are shared. The paper evaluates seven methods on Qwen2.5-VL-3B-Instruct across TextVQA, VizWiz, and image-based ScienceQA. These 21 runs show that methods with different training mechanisms can use the same workflow. The results also show task-dependent differences between the tested methods.
Critical strengths
- The interface addresses a useful practical problem. Some tuning methods require changes to inputs, hidden states, supervision, and checkpoint loading, beyond selecting trainable parameters. Bringing these changes into one method interface is a sensible design. Including model reconstruction is particularly useful for methods whose behavior depends on intervention locations and masks, not just saved weights.
- The paper reports a consistent end-to-end comparison across different types of methods. The seven methods share data processing and evaluation within each task. Section 5.1 also reports fresh checkpoint loading, matching validation IDs, and full evaluation coverage. These checks strengthen the comparison. The authors clearly state that parameter counts and compute budgets are not matched, and that the results do not establish general method rankings.
Critical weaknesses
- The main benefit for method development is not directly demonstrated. Table 1 compares built-in features, but this does not show how much work the proposed interface saves when adding or changing a method. This matters because reduced integration effort is the main motivation. Existing tools already cover parts of this workflow: ms-swift provides custom tuner preparation, saving, and loading, along with loss customization, while LLaVA Steering Factory supports MoReS within a shared training and evaluation pipeline. Appendix D provides a useful modification example, but no comparison of the required changes in another framework. It also notes that a new method name does not activate the built-in MoReS preprocessing path. The paper should explain this dependency and show which work the new interface removes in a concrete extension task.
- The validation does not yet establish that the shared implementations preserve the intended method behavior. Section 5.1 checks completed runs and full prediction coverage, but does not report whether outputs match before and after checkpoint loading, or whether intervention masks and supervised tokens match the intended settings. These checks are especially relevant because reconstruction and method-specific inputs are central contributions. The paper also leaves several structural choices in recipe code, while providing limited details about ranks, target modules, and intervention settings for the reported runs. Table 3 has no untuned-backbone result, so it is unclear how much each recipe changes task performance. The authors acknowledge these limits, but the current evidence remains stronger for workflow completion than for correct method reuse.
Questions for authors
- What checks verify the intervention positions and supervised-token masks? Have you compared outputs before saving and after fresh loading on the same inputs? Please also provide the structural settings and code version used for each recipe in Table 3.
- What scores does the untuned Qwen2.5-VL-3B-Instruct model obtain under the same evaluation setup? This would help determine how much the reported recipes improve or reduce performance on each task.
- Could you add experiments on a more recent VLM to check whether the shared interface and the main findings also hold beyond Qwen2.5-VL-3B-Instruct? Testing a few methods with different training mechanisms on one newer model would help assess how well the library works across models.