acceptodds
Under review as a conference paper at ICLR 2027

VLMintune: A Unified Interface for Developing Visual Instruction Tuning Methods

Abstract

Developing a visual instruction-tuning method often requires changes beyond model parameters, including inserted modules, forward hooks, token masks, supervision, objectives, and inference reconstruction. When these changes are scattered across paper-specific code, reusing a method requires substantial integration effort. We present VLMintune, a research library that treats the complete tuning method as the reusable unit. Each tuning recipe specifies how a method modifies a supported VLM, what it trains, any method-specific inputs, supervision, or loss, and how its learned behavior is saved and reconstructed. The surrounding data processing, backbone-native formatting, training, inference, and evaluation remain shared. VLMintune supports weight adaptation, inserted modules, representation interventions, and alternative token supervision. We implement seven representative methods and evaluate them on Qwen2.5-VL-3B-Instruct across TextVQA, VizWiz, and ScienceQA, yielding 21 full-data runs that demonstrate consistent within-platform comparison of structurally different tuning methods.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
VLM, multimodal learning, library

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mQBQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VLMintune, a library for developing and comparing visual instruction-tuning methods. Its main idea is to place model changes, trainable parameters, method-specific inputs, supervision, loss, and checkpoint loading within a common method interface. Data processing, training, generation, and scoring are shared. The paper evaluates seven methods on Qwen2.5-VL-3B-Instruct across TextVQA, VizWiz, and image-based ScienceQA. These 21 runs show that methods with different training mechanisms can use the same workflow. The results also show task-dependent differences between the tested methods.

Critical strengths
  1. The interface addresses a useful practical problem. Some tuning methods require changes to inputs, hidden states, supervision, and checkpoint loading, beyond selecting trainable parameters. Bringing these changes into one method interface is a sensible design. Including model reconstruction is particularly useful for methods whose behavior depends on intervention locations and masks, not just saved weights.
  2. The paper reports a consistent end-to-end comparison across different types of methods. The seven methods share data processing and evaluation within each task. Section 5.1 also reports fresh checkpoint loading, matching validation IDs, and full evaluation coverage. These checks strengthen the comparison. The authors clearly state that parameter counts and compute budgets are not matched, and that the results do not establish general method rankings.
Critical weaknesses
  1. The main benefit for method development is not directly demonstrated. Table 1 compares built-in features, but this does not show how much work the proposed interface saves when adding or changing a method. This matters because reduced integration effort is the main motivation. Existing tools already cover parts of this workflow: ms-swift provides custom tuner preparation, saving, and loading, along with loss customization, while LLaVA Steering Factory supports MoReS within a shared training and evaluation pipeline. Appendix D provides a useful modification example, but no comparison of the required changes in another framework. It also notes that a new method name does not activate the built-in MoReS preprocessing path. The paper should explain this dependency and show which work the new interface removes in a concrete extension task.
  2. The validation does not yet establish that the shared implementations preserve the intended method behavior. Section 5.1 checks completed runs and full prediction coverage, but does not report whether outputs match before and after checkpoint loading, or whether intervention masks and supervised tokens match the intended settings. These checks are especially relevant because reconstruction and method-specific inputs are central contributions. The paper also leaves several structural choices in recipe code, while providing limited details about ranks, target modules, and intervention settings for the reported runs. Table 3 has no untuned-backbone result, so it is unclear how much each recipe changes task performance. The authors acknowledge these limits, but the current evidence remains stronger for workflow completion than for correct method reuse.
Questions for authors
  1. What checks verify the intervention positions and supervised-token masks? Have you compared outputs before saving and after fresh loading on the same inputs? Please also provide the structural settings and code version used for each recipe in Table 3.
  2. What scores does the untuned Qwen2.5-VL-3B-Instruct model obtain under the same evaluation setup? This would help determine how much the reported recipes improve or reduce performance on each task.
  3. Could you add experiments on a more recent VLM to check whether the shared interface and the main findings also hold beyond Qwen2.5-VL-3B-Instruct? Testing a few methods with different training mechanisms on one newer model would help assess how well the library works across models.
Reviewer_VkJpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VLMintune is a research library for developing, reusing, and comparing visual instruction tuning methods for VLMs. Its key idea is to treat the complete tuning method as the reusable unit: each method is a “recipe” implementing a common TrainingMethod interface for model preparation, trainable state, runtime inputs, supervision/loss changes, saving, and checkpoint reconstruction, while data processing, formatting, training, generation, and VQA evaluation are shared. The paper contrasts VLMintune with component libraries, general training frameworks, and evaluation toolkits. It releases seven methods, two backbone adapters, and five VQA dataset adapters, and evaluates all seven on Qwen2.5-VL-3B-Instruct across TextVQA, VizWiz, and image-only ScienceQA in 21 full-data runs. Results are task-dependent: LoRA leads on TextVQA and ScienceQA, VL-Adapter on VizWiz, and MoReS/ReFT are competitive on TextVQA but weaker on ScienceQA. The authors note limitations: non-parameter-matched comparisons, one backbone, one seed, and fixed learning rates.

Critical strengths
  1. The paper’s main conceptual contribution is well motivated: treating the full visual instruction tuning method as the reusable implementation unit is a sensible response to the heterogeneity of modern tuning approaches. The TrainingMethod interface covers preparation, training behavior, supervision, loss, saving, and reconstruction, which is broader than a typical PEFT adapter abstraction.

  2. The seven recipes span weight adaptation, quantized adaptation, inserted modules, representation interventions, and alternative supervision. This is a stronger demonstration than showing several variants of LoRA. The inclusion of MoReS, ReFT, and L2T makes the “unified interface” claim more meaningful.

  3. Requiring each recipe to define how the adapted model is saved and reconstructed on a fresh backbone is a valuable design choice. It reduces the risk that methods with runtime interventions are evaluated incorrectly and makes the shared inference/evaluation pipeline more credible.

Critical weaknesses
  1. The evaluation does not directly validate the library’s main claim. The paper claims VLMintune enables method development, reuse, and comparison through a common workflow. But the experiments mainly show that seven recipes can be run and evaluated. There is no ablation of the abstraction itself: no comparison against direct per-method implementations, no measurement of code reuse, no developer/user study, no assessment of how much effort is required to add a new method or dataset, and no analysis of runtime or engineering overhead.

  2. Benchmark comparisons are underpowered and confounded. All results use one backbone, one seed, fixed learning rates, and no hyperparameter sweep. Different methods also have very different trainable parameter counts and configurations. For example, QLoRA reports 119.734M adapter parameters versus LoRA’s 14.967M, and L2T trains 3.123B parameters. The paper acknowledges that the protocol is not parameter-matched, but the practical recommendations still risk being read as method rankings.

  3. Missing baselines limit interpretation. There is no untuned backbone baseline, so the reader cannot tell how much each method improves over the base model. There are also no comparisons to the original method-specific implementations or frameworks. This means the paper cannot establish whether VLMintune’s implementations reproduce expected behavior or whether the shared workflow introduces differences.

Questions for authors

The library is presented as supporting heterogeneous visual instruction tuning, but the empirical study covers one backbone and single-image VQA only. Two backbone adapters are claimed, but only Qwen2.5-VL-3B-Instruct is evaluated. The five dataset adapters are not all used in experiments. There is no evidence for multi-image, interleaved, video, or other VLM families. The “unified” claim is therefore plausible but only partially demonstrated.

Reviewer_1gr9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the substantial integration effort required when developing visual instruction-tuning methods by introducing VLMintune, a library that treats tuning methods as reusable units. Focusing on method development, VLMintune provides a framework for training VLMs with different training recipes. The unified tuning workflow is intended to facilitate method comparisons by integrating general post-training procedures into a common experimental pipeline, thereby separating method-specific behavior from the surrounding experimental workflow. The experiments analyze the effects of different training mechanisms and derive practical recommendations accordingly.

Critical strengths
  • The paper proposes a unified workflow for general post-training on VLMs, allowing comparisons to focus on method-specific differences while reducing the influence of variations in the surrounding experimental workflow.
  • VLMintune supports seven methods, two backbone adapters, and five dataset adapters, and is evaluated through 21 full-data runs across three datasets. Based on the experimental results, the paper analyzes the performance of different post-training strategies and derives practical insights, demonstrating a reasonable depth of analysis.
Critical weaknesses
  • As a unified workflow, VLMintune is evaluated experimentally on only one base model, Qwen2.5-VL-3B-Instruct, which provides limited evidence of its applicability across different backbones. Consequently, the generalizability of the findings derived from these experiments also remains unclear. Experiments on different base models, covering at least several representative methods, would provide further validation of the results reported in Section 5.2 and strengthen the reliability of the paper's conclusions.
  • The experiments underlying the practical recommendations use only a single random seed. This may be insufficient to support the conclusions in Section 5.3, which are partly based on relatively small performance differences.
Questions for authors
  • More intuitive comparisons or visualizations, in addition to the code provided in the supplementary material, would help illustrate the extent to which VLMintune reduces integration effort and further substantiate the advantages of its unified workflow.
  • Comparisons of engineering overhead, such as GPU memory consumption, would provide further insights into the code quality and implementation efficiency of the unified framework. Such evidence would help readers better understand how VLMintune's implementation compares with that of other libraries in terms of practical efficiency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.