acceptodds
Desk rejected at ICLR 2027

Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal instruction tuning can be compute-inefficient when a fixed training budget is spread across large image–video pools whose examples vary substantially in utility. We present Goal-Driven Data Optimization (GDO), a framework that computes six sample descriptors for each candidate and constructs optimized 1x training subsets for different goals. Under a fixed one-epoch Qwen3-VL-8B-Instruct training and evaluation protocol on 32 H20 GPUs, GDO reaches the benchmark-specific Uni-10x references with fewer training examples and attains higher endpoint accuracy. Each reference is the lowest-scoring endpoint among four historical 512k-sample uniform branches, making the comparison descriptive rather than a matched causal estimate. GDO reaches these references after 35.4k samples on MVBench, 26.6k on VideoMME, 27.3k on MLVU, and 34.7k on LVBench, while improving endpoint accuracy by 1.38, 1.67, 3.08, and 0.84 percentage points, respectively. The gains are largest on MVBench and MLVU and smaller on LVBench, consistent with weaker alignment between its ultra-long-video setting and the short-video and image-dominant training pool. The four profiles span low-loss, coverage-oriented, and temporally focused allocations; among them, Temp+ gives the strongest overall endpoint profile.

Primary area:
datasets and benchmarks
Keywords:
multimodal instruction tuning, data optimization, data efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Z1rmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Goal-Driven Data Optimization (GDO) for multimodal instruction tuning. GDO computes six descriptors for each training example and constructs goal-specific subsets through a shared scoring function and different feasibility constraints. The method is evaluated with Qwen3-VL-8B-Instruct on several video understanding benchmarks.

Critical strengths
  1. The paper studies an important practical problem: allocating a limited multimodal instruction-tuning budget across heterogeneous image and video data.

  2. The paper is unusually transparent about its experimental protocol and includes subtask analyses and component ablations.

Critical weaknesses
  1. The main baseline is not a matched control. For each benchmark, the paper uses the lowest-scoring endpoint among four historical 512k-sample uniform runs as the reference, which systematically favors GDO and makes the reported gains difficult to interpret.

  2. The “faster convergence” claim is not based on directly evaluated sample counts. The reported Peak Match values use interpolated, post-processed trajectories with one checkpoint removed and pseudo-sample-count positions, so the claimed 14–19× reductions are not directly measured.

Questions for authors
  1. Can the authors compare GDO against fresh uniform/random subsets under exactly the same training budget and random seeds?

  2. Can the authors report actual evaluated checkpoints at the claimed Peak Match sample counts instead of interpolated values?

  3. How do the conclusions change if a single fixed uniform baseline is used for all benchmarks rather than the weakest branch selected separately for each benchmark?

Reviewer_1AYFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Goal-Driven Data Optimization (GDO), a pipeline for selecting multimodal instruction-tuning data according to goals such as low loss, broad coverage, or stronger temporal supervision. GDO computes six sample descriptors, applies a shared ranking score and goal-specific feasibility constraints, and constructs subsets from LLaVA-OneVision image QA and LLaVA-Video short-video QA data.

The authors train Qwen3-VL-8B-Instruct for one epoch on 32 H20 GPUs and evaluate four profiles on MVBench, VideoMME, MLVU, and LVBench. Table 2 shows trade-offs across the profiles: MinLoss and Diverse are below the frozen reference on LVBench by 1.36 and 0.32 pp, respectively, while Temp is 0.23 pp below it on MVBench. Temp+ uses 53.3k examples and has the strongest endpoint result on all four benchmarks, with gains of 1.38, 1.67, 3.08, and 0.84 pp, respectively.

Critical strengths

Strength 1

The four GDO profiles share the same scoring backbone and training protocol while changing subset construction and allocation constraints. This provides a useful view of how profile choices relate to the results under the stated setup.

Strength 2

The paper reports subtask outcomes as well as aggregate scores. Table 3 includes adverse results, including a -3.8 pp delta for Temp+ on MLVU Ego, while Appendix Figure 7 shows a broader MVBench subtask heatmap and Figure 8 includes representative subtask trajectories with both gains and regressions.

The appendix also gives a useful account of component effects. Table 5 reports endpoint ablations for MVBench, VideoMME, and MLVU; Figure 6 plots ablation trajectories for all four benchmarks, including LVBench. The effects are not uniformly positive: for example, removing VDS or PPL slightly improves MLVU, while removing self-consistency lowers MLVU by 3.16 pp.

Critical weaknesses

Weakness 1

The central efficiency comparison uses a relatively favorable frozen reference and estimated crossing points. For each benchmark, the reference is the lowest endpoint among four historical 512k-sample UNI-10x branches, rather than a matched-budget uniform subset. The paper does not report a uniform or random subset trained at a matching GDO budget, so it is unclear how much of the reported advantage comes from GDO’s selection rather than from the reference choice.

For the trajectory figures, the paper retains four of five saved checkpoints after removing one noisy point, and linearly interpolates the first crossing. The authors disclose this procedure and state that the comparisons are descriptive, not matched causal controls. This is therefore a limitation of the evidence, not an undisclosed protocol choice. Still, the reported sample-reduction ratios should be interpreted as estimates against the selected reference, not as directly measured speedups attributable to GDO.

A matched-budget uniform subset for Temp+, together with the endpoint values of all four 512k branches, would make the efficiency claim easier to assess. Otherwise, the headline reductions should remain explicitly framed as descriptive estimates against the selected reference.

Weakness 2

The related-work discussion does not address recent work directly relevant to multimodal instruction-data selection. XMAS studies data selection for vision-language model fine-tuning using cross-modal alignment trajectories, while DOSE uses pretrained models to assess multimodal data quality and alignment (XMAS, ICML 2026; DOSE, Findings of ACL 2026). Both were available before the ICLR 2027 paper deadline.

Their methods and settings differ from GDO, especially in its mixed image/video pool and goal-specific temporal constraints; their existence does not by itself negate GDO’s contribution. However, the paper should explain more clearly what new insight or capability those profile constraints provide beyond combining descriptors with a fixed score. The authors should position GDO against these methods and state whether temporal allocation in mixed image/video tuning is the principal contribution.

Questions for authors

Question 1

Could the authors report a uniform subset comparison at the Temp+ budget of 53.3k and provide the endpoint scores of all four 512k UNI-10x branches? This would help separate the effect of GDO selection from the choice of the frozen reference.

Question 2

How should readers distinguish GDO’s methodological contribution from recent multimodal instruction-data selection methods such as XMAS and DOSE? Is goal-specific temporal allocation in a mixed image/video pool the principal novelty?

Reviewer_tcWkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Goal-Driven Data Optimization (GDO), a data selection framework for multimodal instruction tuning. GDO combines six sample descriptors, including video dependence, temporal relevance, and difficulty, with heuristic scoring and subset composition constraints. The authors evaluate four goal profiles using Qwen3-VL-8B-Instruct on several video understanding benchmarks, reporting improved accuracy and data efficiency compared with historical uniform-sampling references.

Critical strengths

S1. Empirical evaluation. The paper evaluates several data selection configurations across four video understanding benchmarks and includes component ablations.

Critical weaknesses

W1. Questionable baseline comparison. The paper compares GDO against the lowest-performing result among four historical uniform-sampling runs, selected separately for each benchmark. This is a weak reference that systematically favors the proposed approach. Without comparisons against existing data selection methods (e.g., [1–4]), it is difficult to assess whether GDO actually improves data efficiency.

W2. Misleading convergence claims. The paper repeatedly describes its results as faster convergence, but the evidence primarily concerns sample efficiency: reaching a particular benchmark accuracy with fewer training examples. This does not establish faster convergence in the usual optimization sense.

W3. Limited novelty and insufficient methodological justification. The method largely combines existing data-quality signals and heuristic selection constraints. The scoring coefficients and goal profiles appear hand-designed, with limited justification for their effectiveness. In particular, the weighting schemes in Equations (9)–(11) are introduced without explaining how the coefficients were determined or whether they were tuned against downstream performance. This raises concerns about hyperparameter sensitivity and potential benchmark-specific tuning.

W4. Poor presentation. The manuscript is repetitive and unnecessarily verbose, with several terms introduced without adequate explanation. In addition, the use of \cite where \citep would be appropriate makes many sentences awkward to read. These issues make it difficult to identify the main contributions and follow the experiments.

References

[1] Lyu et al. (2025). Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning.

[2] Xia et al. (2024). LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML.

[3] Kung et al. (2023). Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks. EMNLP.

[4] Nayak et al. (2026). A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn’t). ICML.

Questions for authors

Q1. Baseline comparisons. Can the authors compare GDO against existing data selection methods (e.g., [1–4]) and matched random-subset baselines under the same training budget?

Q2. Weight justification and sensitivity. How were the coefficients in Equations (9)–(11) determined? Can the authors provide a sensitivity analysis to assess whether the improvements are robust to these choices?

Q3. Motivation of goal profiles. What motivates the four goal profiles (MinLoss, Diverse, Temp, and Temp+)? Why were these particular objectives and selection constraints chosen?

Q4. Goal-specific evaluation. How is the success of each goal measured? Although the paper specifies different selection constraints, the evaluation primarily reports the same downstream accuracy metrics for all profiles. Could the authors clarify how these metrics validate each profile's intended objective?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.