acceptodds
Under review as a conference paper at ICLR 2027

The Value of Training Data Is Not Static: Optimization-Aware Adaptive Selection for Multimodal Instruction Tuning

Abstract

Multimodal instruction tuning relies on large‑scale image‑text datasets, making effective data selection critical for improving training efficiency. Existing approaches mostly adopt static offline scoring or measure visual dependence at inference time, yet these metrics cannot directly quantify how a training example’s gradient update will benefit the current evolving multimodal model. To address this gap, we propose **DOVAS**, a region‑based dynamic data‑allocation framework that combines offline visual‑evidence scoring with online optimizer‑aligned utility estimation. DOVAS first partitions the instruction corpus into fixed, semantically stratified regions. During training, it periodically re‑weights regional sampling probabilities by estimating the predicted impact of candidate optimizer updates without modifying region membership. This design lets the model draw adaptively useful training data throughout the optimization trajectory. Experiments on standard multimodal instruction‑tuning benchmarks demonstrate that DOVAS consistently outperforms competitive static data‑selection baselines. Using only half of the full‑data training compute budget, it surpasses the performance of full‑dataset training across multiple model families. These results demonstrate that the training‑time value of multimodal samples is not a static intrinsic property but depends on model state and optimizer dynamics.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
data selection, visual instruction tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FQ9bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DOVAS, a dynamic data-allocation framework for multimodal instruction tuning. It first partitions the training corpus into fixed regions using instruction semantics and offline scores measuring answer compatibility, visual dependence, and correct-image specificity. During training, it periodically estimates how regional optimizer updates would affect held-out answer and visual counterfactual objectives, then adjusts regional sampling probabilities accordingly.

Experiments on LLaVA-665K and Vision-Flan-186K, with additional evaluations on Qwen models, report competitive or better aggregate performance than full-data training under a 50% online compute budget that includes dynamic selection but excludes offline preparation. Ablations examine region construction, proxy objectives, optimizer-aligned estimation, and dynamic versus fixed allocation, while repeated runs support the method’s consistency relative to random sampling and its static variant.

Critical strengths

A distinctive and well-motivated allocation design. DOVAS combines fixed visual-evidence regions with optimizer-aligned utility estimates based on answer and visual counterfactual objectives. This provides a practical way to adapt the training mixture without repeatedly scoring the entire corpus. The combination offers a meaningful methodological contribution beyond static visual-dependence scoring.

Strong empirical results with substantive supporting controls. The method achieves competitive or better aggregate performance than full-data training under half the online compute budget, with improvements across many individual benchmarks. Comparisons against fixed mixtures, raw-gradient estimation, and reduced objective sets support the main design choices, while repeated runs demonstrate consistent gains over random sampling and the static variant. These results make the approach promising beyond its conceptual motivation.

Critical weaknesses
  1. The benefit of real-time feedback is not fully isolated from that of a time-varying curriculum. The initialized and average-mixture controls support changing the allocation during training, but both remove temporal scheduling as well as feedback from the current model state. They therefore do not establish whether ongoing utility estimation is needed, or whether replaying a previously obtained allocation schedule would achieve similar gains. This distinction matters because state-dependent allocation is a central contribution and incurs additional online computation.

  2. The connection between proxy utility and generalization remains only partially validated. The update-level diagnostics show that predicted utility tracks changes in a proxy objective, while the downstream ablations support the usefulness of the proposed objectives overall. However, neither directly establishes that regions ranked more highly at a given checkpoint produce larger improvements on independent downstream tasks. Such evidence would strengthen the interpretation that the allocator tracks changing training value, rather than simply optimizing a useful but imperfect surrogate.

Questions for authors

I would like to understand whether the gains require feedback from the current model state, or whether a predetermined, time-varying curriculum would suffice. Could you record the regional sampling probabilities from one DOVAS run and replay that schedule under different training seeds, without online utility estimation? Comparing replay with live DOVAS at matched parameter-update counts would help isolate the contribution of feedback; a matched-online-compute comparison would show whether that contribution justifies its overhead.

The utility diagnostics establish agreement with changes in the proxy objective, but I am less certain how well the resulting regional rankings predict generalization gains. At a few checkpoints, could you start from the same model and optimizer states, briefly train on each region separately, and compare the observed improvements on independent validation tasks with the predicted utilities? This would directly test whether regions assigned higher utility are also more beneficial beyond the proxy objectives.

2 comments

Thank you for dedicating your time and effort to reviewing our paper. We are deeply encouraged by your positive comments and recognition of our work. Below, we provide detailed responses to address your concerns.

W1 & Q1: The existing static controls do not distinguish a time-varying curriculum from real-time feedback.

We thank the reviewer for this comment. The initialized-mixture and average-mixture controls in Table 9 show that fixed allocations do not recover the performance of DOVAS. Following the reviewer’s suggestion, we additionally conduct a cross-seed sampling-schedule replay experiment.

Replay retains the time-varying regional probabilities recorded from a DOVAS run, but does not estimate utility from the current state of the target run. It therefore preserves the temporal curriculum while removing current-state feedback.

Let K denote the number of parameter updates performed by DOVAS. Both target runs perform K updates with a global batch size of 16, yielding 16K cumulative training-example visits, and use the same learning-rate schedule.

MethodSource of regional probabilitiesParameter updatesCumulative training-example visitsRel. (%)
DOVASReal-time feedback from the current model stateK16K101.1
ReplayA prerecorded schedule from another seedK16K100.2

This result supports the additional value of current-state feedback in the evaluated cross-seed comparison.

W2 & Q2: The connection between proxy utility and independent downstream gains has not been directly established.

We thank the reviewer for this suggestion. We introduce a regional short-training experiment that starts each branch from the same model and optimizer states and directly measures the gains obtained on independent GQA and TextVQA validation sets.

The experiment uses LLaVA-v1.5-7B and LLaVA-665K, with checkpoints selected at approximately 10%, 50%, and 90% of the online compute budget. At each checkpoint, we record the six regional utility scores actually used for allocation. We then initialize separate branches from the same model, AdamW, and learning-rate scheduler states, and train each branch exclusively on one region for 100 parameter updates. All branches are evaluated before and after training on the same fixed, independent GQA and TextVQA validation sets.

Let θt\theta_t denote the checkpoint model and θt,b100\theta_{t,b}^{100} the model obtained after 100 updates on region b. Let SGS_G and STS_T denote GQA accuracy and the official TextVQA VQA accuracy, respectively, both expressed on a 0–100 scale. The observed gain is defined as:

Δt,b=12[SG(θt,b100)−SG(θt)+ST(θt,b100)−ST(θt)].\Delta_{t,b}=\frac{1}{2}\left[S_G(\theta_{t,b}^{100})-S_G(\theta_t)+S_T(\theta_{t,b}^{100})-S_T(\theta_t)\right].

Thus, Δt,b\Delta_{t,b} is the arithmetic mean of the accuracy improvements on the two tasks, measured in percentage points.

We use the actual allocation score:

Ut,b=A^t,b+V^t,b+S^t,b,U_{t,b}=\widehat A_{t,b}+\widehat V_{t,b}+\widehat S_{t,b},

and calculate the within-checkpoint rank correlation:

ρt=Spearman⁡([Ut,0,…,Ut,5],[Δt,0,…,Δt,5]).\rho_t=\operatorname{Spearman}\left([U_{t,0},\ldots,U_{t,5}],[\Delta_{t,0},\ldots,\Delta_{t,5}]\right).

Training progressSpearman across six regionsHighest-predicted-utility regionGain from that region (pp)Random-control gain (pp)Advantage over random (pp)
10%0.943Region 5+0.24+0.09+0.15
50%0.886Region 4+0.20+0.10+0.10
90%0.829Region 3+0.09+0.04+0.05

Strong positive rank correlations are observed at all three checkpoints, indicating that regions assigned higher predicted utility generally produce larger gains on the independent tasks. The highest-predicted-utility region also outperforms the random control at each stage, providing further empirical support for the ability of the utility score to identify regions with greater local training value.

Thank you for the detailed response. My concerns are partially addressed, I will remain my positive score.

Reviewer_GCZsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes DOVAS, a dynamic data selection framework for multimodal instruction tuning that adapts the sampling distribution to the evolving model and optimizer states. DOVAS first partitions training examples into fixed evidence regions using offline scores based on answer compatibility, visual gain, and correct-image specificity. During training, it estimates the utility of candidate optimizer updates for each region using answer and visual counterfactual objectives, then periodically adjusts regional sampling probabilities. Experiments on LLaVA-665K and Vision-Flan-186K show that DOVAS outperforms competitive data selection baselines and achieves higher aggregate performance than full-data training with only 50% of its online compute budget. Additional experiments across model architectures, ablation studies, and repeated training runs support the effectiveness of the proposed allocation strategy.

Critical strengths
  1. Well-motivated problem formulation. The paper addresses an important limitation of static data selection in multimodal instruction tuning. The utility of training examples can change as the model learns, making fixed selection scores insufficient for identifying useful data throughout training. The observed changes in regional utility provide empirical motivation for adapting the training distribution to the current model state.

  2. Practical integration of offline partitioning and online allocation. DOVAS combines fixed evidence-based regions with dynamic sampling probabilities, allowing the training distribution to adapt without repeatedly rescoring individual examples or reconstructing the partition. Its utility estimation considers answer prediction, visual dependence, and correct-image specificity while incorporating the current optimizer state. This design provides a reasonable balance between adaptive selection and computational overhead.

  3. Consistent improvements over competitive baselines. DOVAS achieves the highest aggregate performance on both LLaVA-665K and Vision-Flan-186K among the compared methods. With an online compute budget equal to 50% of full-data training, it achieves relative performance of 101.1% and 109.7%, respectively. The evaluation includes competitive selection methods such as OPUS, VIG, and VisNec, providing meaningful evidence of the practical effectiveness of DOVAS.

  4. Comprehensive ablations supporting the proposed components. The paper evaluates evidence-based region construction against random and semantic-only partitions, isolates the contribution of visual counterfactual objectives, and compares optimizer-aligned utility against raw-gradient scoring. The average-mixture static control is particularly informative because it preserves the aggregate sampling proportions of a completed DOVAS run while removing their dependence on the current training state. DOVAS consistently outperforms this control, supporting the importance of adapting the allocation over time.

  5. Broad evaluation and evidence of repeatability. The experiments cover two instruction-tuning datasets and three model architectures, including LLaVA-v1.5-7B, Qwen2.5-VL-7B, and Qwen3.5-4B. The paper examines multiple compute budgets and the sensitivity to major allocation hyperparameters. Repeated experiments across five training seeds provide additional evidence that the observed gains are stable under the fixed offline preparation. The explicit accounting of online selection FLOPs strengthens the transparency of the evaluation.

Critical weaknesses
  1. Limited methodological novelty relative to existing data selection approaches. Several central components of DOVAS build on established ideas, including dynamic data selection, visual dependence scoring, and optimizer-aware utility estimation. Related methods such as LESS and OPUS have already explored the use of optimizer information to estimate the influence of training data. The primary contribution of DOVAS appears to lie in combining these techniques with multimodal evidence regions and counterfactual objectives. Although the ablation studies demonstrate the usefulness of individual components, comparisons with existing methods such as OPUS change multiple factors simultaneously, including the selection objectives, granularity, and update procedure. This makes it difficult to isolate the methodological contribution beyond the integration of existing techniques.

  2. Incomplete assessment of end-to-end computational efficiency. The reported compute budget includes parameter training and online selection but excludes offline annotation and preparation. For DOVAS, these operations include semantic clustering, negative-image retrieval and verification, and scoring the training data under multiple visual conditions using Qwen3.5-9B. The scorer-size ablation shows that offline scoring quality substantially affects downstream performance, making its computational cost relevant to practical deployment. Although the paper clearly defines its online FLOP accounting, it does not establish whether the complete pipeline remains more efficient after accounting for offline preparation. Reporting end-to-end FLOPs, GPU-hours, wall-clock time, and the potential amortization of offline costs across multiple training runs would strengthen the efficiency claims.

  3. Confounding factors in the ablation of visual objectives. Table 4 shows consistent improvements when visual and specificity objectives are added to the answer-only utility. However, each utility component is standardized separately before the components are summed and passed through a softmax with a fixed temperature. Adding components can change the scale of the combined utility and the concentration of the resulting sampling distribution, even when the components contain highly correlated information. Consequently, the observed improvements cannot be attributed exclusively to the additional visual information. A comparison with matched sampling entropy or a normalization of the combined utility, together with an analysis of correlations among the three components, would provide stronger evidence for their independent contributions.

  4. Limited statistical evidence for improvements over strong external baselines. The repeated experiments across five training seeds support the stability of DOVAS relative to Random and DOVAS-Static. However, comparable repeated-run results are not reported for strong external baselines such as OPUS, VIG, and VisNec. On LLaVA-665K, DOVAS achieves 101.1% relative performance compared with 100.0% for VisNec, leaving some uncertainty about whether this advantage is robust to training variability in both methods. Additional repeated runs or uncertainty estimates for the strongest external baselines would make the comparative performance claims more convincing.

Questions for authors
  1. Reliability of utility ranking within individual checkpoints. Appendix F.3 reports a pooled Pearson correlation of 0.926 between predicted and observed update improvements across 48 comparisons collected at three training stages. However, the allocation mechanism depends on ranking regions within each checkpoint, while the reported diagnostic objective differs from the standardized and EMA-smoothed utility used for allocation. Could the authors report checkpoint-wise rank correlations or the stability of regional utility rankings across different probe samples? This would help establish how reliably the estimated utilities support actual allocation decisions.

  2. Consistency between the region definition and compute accounting. Equation (7) and Algorithm 1 define MM positive-evidence regions together with Region 0, suggesting M+1M+1 regions in total. However, Appendix D.1 calculates the default regional probe cost using 6×8=486\times8=48 examples. If the default setting uses M=6M=6, the stated region definition appears to imply seven regions. Could the authors clarify whether Region 0 is included in the regional probes and how its computational cost is accounted for? This clarification would help reconcile the algorithm with the reported selection overhead.

3 comments

We sincerely express our gratitude for dedicating your valuable time to providing insightful suggestions that can enhance our paper. Your praise regarding our methodology, experiments, and contributions have greatly encouraged and motivated us. Our detailed responses to all of your concerns are presented below.

W1: The methodological contribution primarily combines existing techniques, and the comparison with OPUS does not isolate the additional contributions.

We thank the reviewer for this comment. DOVAS introduces a specific allocation design for multimodal instruction tuning: it organizes data into visual-evidence regions, evaluates candidate updates using answer and visual counterfactual objectives, and adjusts the regional training distribution under a limited online estimation budget.

The existing ablations examine the utility of region construction, objective composition, and update estimation separately. Table 8 changes region construction while retaining the proxy objectives and allocation mechanism. Table 4 changes the objectives while keeping the regions and optimizer-aligned estimator fixed. Table 10 replaces the optimizer-aligned update vector with the raw gradient within the same allocation framework. These results support the effectiveness of the corresponding design choices.

Following the reviewer’s suggestion, we additionally introduce a frozen-preconditioner control to isolate an update-estimation difference relevant to OPUS. This control retains DOVAS’s region construction, A/V/S objectives, and dynamic allocation mechanism, and changes only the candidate update vector used for utility estimation.

The experiment uses LLaVA-v1.5-7B on LLaVA-665K. Let K denote the number of parameter updates completed by full DOVAS under the original 50% online budget. The new control also performs K updates and follows the same learning-rate schedule.

DOVAS applies a separate virtual AdamW update to each regional gradient:

(θt,b+,st,b+)=AdamW⁡(θt,st,gt,b),(\theta_{t,b}^{+},s_{t,b}^{+}) = \operatorname{AdamW}(\theta_t,s_t,g_{t,b}),

ut,b=θt−θt,b+.u_{t,b}=\theta_t-\theta_{t,b}^{+}.

Each candidate starts from the same current model and optimizer states. The candidate gradient participates in the first-moment, second-moment, and parameter-update computations, after which the virtual update is discarded.

The additional control uses a frozen-preconditioner approximation inspired by OPUS. Within a scoring round, all six regions share the second-moment scaling determined by the current optimizer history, rather than recomputing that scaling for each candidate regional gradient.

The two variants use either the full candidate update ut,bu_{t,b} or its approximation u~t,b\widetilde u_{t,b} to evaluate the same three utilities:

V_{t,b}=\langle u_{t,b},g_{v,t}\rangle,\qquad S_{t,b}=\langle u_{t,b},g_{s,t}\rangle.$$ The frozen-preconditioner control replaces only $u_{t,b}$ with $\widetilde u_{t,b}$ in these expressions. Subsequent standardization, EMA smoothing, softmax allocation, and inner-product computation remain unchanged. | Method | Allocation objectives | Parameter updates | Rel. (%) | |---|---|---:|---:| | DOVAS | A+V+S | K | 101.1 | | DOVAS with OPUS-style frozen-preconditioner estimation | A+V+S | K | 99.2 | **Together with the existing ablations, this result supports the additional value of the update-estimation choice within the proposed allocation framework.** > **W2: End-to-end efficiency excludes offline scoring, whose importance is reinforced by the scorer-size ablation.** To improve comparability, we use the same Qwen3.5-9B scorer for VisNec, VIG, and DOVAS, reducing differences attributable to scoring-model capability. We also adapt OPUS to multimodal instruction tuning. Whereas prior data selection studies commonly describe their budgets using the retained data fraction, our protocol additionally matches online FLOPs and charges online selection to the common budget, rather than granting it additional free compute. We agree that offline preparation remains relevant to practical efficiency. We therefore report the complete GPU-hour breakdown for DOVAS and the closest-performing baseline, VisNec [1], using the same scoring model. | Method | Offline preparation (GPU-hours) | Parameter training (GPU-hours) | Online selection (GPU-hours) | Total GPU-hours | Peak GPU memory | Rel. (%) | |---|---:|---:|---:|---:|---:|---:| | VisNec | 28.4 | 17.4 | — | 45.8 | 108.4 | 100.0 | | DOVAS | 27.1 | 15.8 | 1.9 | 44.8 | 108.4 | 101.1 | DOVAS achieves higher performance under the common online FLOP budget, while the measured total GPU-hour cost is slightly lower after including offline preparation. We will clarify in the final manuscript that “50% compute” denotes an online budget equal to 50% of the Full Data baseline’s parameter-training FLOPs, with online selection included.

W3: Adding visual objectives may change utility scale and sampling concentration.

We thank the reviewer for identifying this potential confound. We agree that adding V and S under a fixed softmax temperature may alter the concentration of the sampling distribution.

To test this explanation directly, we introduce an entropy-matched A-only control using the LLaVA-v1.5-7B and Vision-Flan-186K setting of Table 4.

We first extract the regional sampling probabilities at each refresh from the full DOVAS run and compute their normalized entropy:

H~(π)=−1log⁡6∑b=05πblog⁡πb.\widetilde H(\pi)=-\frac{1}{\log 6}\sum_{b=0}^{5}\pi_b\log\pi_b.

The A-only control computes its regional probabilities using its own standardized, EMA-smoothed answer utility. At each refresh, its temperature is adjusted so that the absolute difference from the normalized entropy of the corresponding DOVAS distribution does not exceed 10−410^{-4}.

MethodObjectivesMean absolute normalized entropy differenceEntropy-matching success rate (%)Rel. (%)
Full DOVASA+V+S——109.7
Entropy-matched A-onlyA0.000013100105.1

This result supports that the benefit of the combined objectives cannot be explained solely by lower sampling entropy or a more concentrated distribution. The visual objectives provide additional useful information for regional allocation.

W4: Small improvements over VisNec, VIG, and OPUS lack sufficient statistical evidence.

We thank the reviewer for this concern. At the 50% budget, DOVAS outperforms Full Data on 8 of the 10 benchmarks and achieves the best or tied-best performance on 7 of the 10 benchmarks among the compared methods. To summarize performance across tasks rather than emphasize a particular benchmark, we use aggregate relative performance, following [1].

Regarding training variability, Table 11 already reports five-seed results for DOVAS, Random, and DOVAS-Static. We additionally provide five-run results for VisNec and OPUS below. Entries report Rel. (%), with the mean and sample standard deviation in the final column.

MethodRun 1Run 2Run 3Run 4Run 5Mean ± SD
VisNec99.699.7100.5100.1100.1100.00 ± 0.36
OPUS97.996.899.197.798.197.92 ± 0.83

Q1: Pooled Pearson correlation does not validate within-checkpoint regional rankings, and the diagnostic objective differs from the allocation score.

We thank the reviewer for this suggestion. To assess the rankings used for allocation more directly, we evaluate checkpoints at approximately 10%, 50%, and 90% of training progress using the actual allocation score:

Ut,b=A^t,b+V^t,b+S^t,b.U_{t,b}=\widehat A_{t,b}+\widehat V_{t,b}+\widehat S_{t,b}.

At each checkpoint, we calculate:

ρt=Spearman⁡([Ut,0,…,Ut,5],[Δt,0,…,Δt,5]),\rho_t=\operatorname{Spearman}\left([U_{t,0},\ldots,U_{t,5}],[\Delta_{t,0},\ldots,\Delta_{t,5}]\right),

where Δt,b\Delta_{t,b} denotes the mean improvement in GQA and TextVQA accuracy after short training on region b.

Specifically, we start from the same model and optimizer states and train separately on each of the six regions for 100 parameter updates. We then measure the observed gains on fixed, independent GQA and TextVQA validation sets and compute the regional rank correlation within each checkpoint.

Training progress10%50%90%
Spearman correlation0.9430.8860.829

The actual allocation scores remain strongly correlated with the observed downstream gains when rankings are evaluated separately within each checkpoint. These results provide direct empirical support for the relationship between the proposed regional utility and downstream training value at the evaluated checkpoints.

Q2: Consistency between the default number of regions and the 6×8 regional probe count.

We thank the reviewer for pointing out this ambiguity. The default configuration contains five positive-evidence regions and an additional Region 0 for examples that do not satisfy the real-image condition. Thus:

M=5,K=M+1=6.M=5,\qquad K=M+1=6.

Region 0 participates in utility estimation in the same way as the other regions. Eight probe examples are drawn from each region at every refresh, giving:

6×8=486\times8=48

regional probe examples in total.

Region 0 is therefore already included in the reported default regional probe cost. We will make the notation consistent across Eq. (7), Algorithm 1, Figure 3, and the accompanying text: M will denote the number of positive-evidence regions, and K the total number of regions. We will also state the default values explicitly to avoid interpreting “six bins” as six positive-evidence regions.

Thank you for the detailed response. The control with matched sampling entropy addresses my concern about the visual objective ablation. The additional baseline runs strengthen the performance comparisons. The ranking analysis at individual checkpoints supports the reliability of the utility estimates. The clarification of the region count resolves my question about compute accounting.

Some concerns remain regarding methodological novelty. The additional experiments address most of my concerns and strengthen the empirical evidence. I will increase my soundness score from 2 to 3.

Reviewer_jLDmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

DOVAS is a dynamic data-selection framework for multimodal instruction tuning. It first partitions training data offline into fixed, semantically clustered evidence regions using answer compatibility, visual gain, and correct-image specificity scores. During training, it periodically estimates how a candidate optimizer update from each region would affect held-out proxy objectives—answer loss, real-vs-blind image loss, and real-vs-negative image loss—using a virtual optimizer step with the current optimizer state. These utilities set region sampling probabilities, while region boundaries and proxy pools remain fixed. On LLaVA-665K and Vision-Flan-186K, DOVAS outperforms full-data aggregate performance and many selection baselines under a 50% online-compute cap, with ablations supporting dynamic allocation, visual counterfactual objectives, and optimizer-aligned utility estimation.

Critical strengths
  1. The paper correctly identifies a real limitation of offline selection: data value changes during training. The distinction between stable region boundaries and adaptive region sampling is sensible and avoids repeatedly rescoring the entire pool.

  2. Applying a virtual optimizer step with the current optimizer state, rather than using raw gradient similarity, is a strong design choice. The ablation against raw-gradient allocation supports its benefit.

  3. The real-vs-blind and real-vs-negative objectives go beyond answer loss and target whether updates improve the use of visual evidence. The ablation shows that answer-only dynamic allocation can underperform DOVAS-Static, whereas adding visual objectives reverses this. This is an important, honest finding.

Critical weaknesses
  1. DOVAS requires semantic clustering, Qwen3.5-9B scoring under real, blind, and negative conditions, negative-image retrieval and verification, and proxy-pool construction. This is excluded from the compute budget for all methods, but DOVAS depends heavily on it. The paper’s efficiency claim is therefore about online compute, not end-to-end cost. For a single training run, the offline pipeline may dominate; amortization must be demonstrated.

  2. Compute accounting is FLOP-based and does not capture wall-clock or memory. The paper counts training and online selection FLOPs, but excludes state copying, memory traffic, offline preparation, evaluation, and configuration search. The “50% compute” claim is therefore narrow. Measured GPU-hours, peak memory, and end-to-end runtime would make the practical claim much stronger.

  3. Fixed regions limit adaptivity. Only region-level sampling probabilities change. Region membership, semantic-cluster shares, and within-region uniform sampling stay fixed. A misassigned sample or a harmful region cell cannot be downweighted individually. This may be sufficient, but it weakens the “dynamic data selection” framing relative to sample-level or cluster-level adaptive methods.

Questions for authors
  1. DOVAS exceeds full-data aggregate performance, but some individual benchmarks are lower. At the 30% budget, the advantage over random is small and task-level results are mixed; strong gains appear mainly at 50–70% budgets. The unweighted “Rel.” average may obscure which capabilities improve or regress. Baselines also lack variance estimates and significance tests in the main tables.

  2. OPUS is adapted to the multimodal setting, and VisNec/VIG use the same offline scorer. Other selectors may have different offline costs and tuning. Since offline preparation is excluded, the comparison measures online-budget behavior rather than total selection-and-training cost. A matched end-to-end budget would be more convincing.

3 comments

We sincerely thank you for the precious time and effort in providing a wealth of suggestions to enhance the quality of our paper. We have carefully read all the comments and provide detailed point-by-point responses as follows. Hopefully, we can adequately address your concerns.

W1 & W2: Offline preparation is excluded, and single-run efficiency and cost amortization remain unclear.

We thank the reviewer for highlighting the distinction between the online compute budget and end-to-end cost. We first clarify the motivation for our accounting protocol. Prior data selection studies, including [1, 2], commonly report the retained data fraction as a measure of the training-data budget, with the selected subset remaining fixed during subsequent training. DOVAS, however, dynamically samples from predefined data regions according to the current training state. A fixed subset fraction therefore cannot fully capture the additional computation required for utility estimation and adaptive allocation.

For this reason, our FLOP budget includes both parameter training and online selection. Selection consumes the same budget and correspondingly reduces the computation available for parameter updates; it does not receive additional unaccounted compute. Under this protocol, DOVAS uses approximately 43% of the training data, compared with approximately 50% for the baselines.

To reduce variation attributable to the scoring model, we use the same Qwen3.5-9B scorer when reproducing the relevant baselines. The table below reports the complete GPU-hour breakdown for DOVAS and VisNec [1], the closest-performing baseline.

MethodOffline preparation (GPU-hours)Parameter training (GPU-hours)Online selection (GPU-hours)Total GPU-hoursPeak GPU memoryRel. (%)
VisNec28.417.4—45.8108.4100.0
DOVAS27.115.81.944.8108.4101.1

Under the common online FLOP budget, DOVAS achieves higher performance than VisNec. In this measured configuration, its total GPU-hour cost is also slightly lower after including offline preparation. We will clarify throughout the final manuscript that “50% compute” refers to an online budget equal to 50% of the Full Data baseline’s parameter-training FLOPs, including the cost of online selection.

W3: Fixed regions, semantic-cluster shares, and uniform within-cell sampling limit the granularity of adaptation.

DOVAS dynamically adjusts the sampling probabilities of fixed regions, rather than assigning adaptive weights to individual examples within each region. Consequently, examples within a region receive higher or lower sampling probabilities together as the regional allocation changes.

Region-level estimation provides a practical way to control the cost of repeated selection. Whereas methods such as [3, 4] use optimization-related signals to assess candidate examples, DOVAS estimates utility at the regional level. The region-construction ablation in Table 8 and the granularity study in Figure 3 show that this design improves performance while maintaining a moderate online selection overhead.

We nevertheless acknowledge the reviewer’s concern about adaptation granularity. We will consistently describe the method as “state-dependent regional allocation.” This terminology emphasizes that the allocation changes with the current model and optimizer states, while region membership remains fixed.

Q1: Improvements are not uniform across tasks, gains at the 30% budget are modest, and the aggregate metric and statistical evidence require clarification.

We thank the reviewer for this observation. At the 50% budget, DOVAS outperforms Full Data on 8 of the 10 benchmarks and achieves the best performance on 7 of the 10 benchmarks among the compared methods. These results support its effectiveness across multiple tasks, although they do not imply uniform improvement on every benchmark.

To summarize performance across diverse downstream tasks rather than emphasize a single benchmark, we report aggregate relative performance, following [1]. Specifically, Rel. is the mean of the benchmark-wise scores normalized by the corresponding Full Data scores.

Regarding statistical evidence, Table 11 already reports five-seed results for DOVAS, Random, and DOVAS-Static. We additionally provide five-run results for VisNec and OPUS below. All entries are Rel. (%), and the final column reports the mean and sample standard deviation.

MethodRun 1Run 2Run 3Run 4Run 5Mean ± SD
VisNec99.699.7100.5100.1100.1100.00 ± 0.36
OPUS97.996.899.197.798.197.92 ± 0.83

Q2: A shared scorer and baseline adaptations do not establish end-to-end cost fairness.

To improve comparability, VisNec, VIG, and DOVAS use the same Qwen3.5-9B scorer, reducing differences attributable to scoring-model capability. We also adapt OPUS, a competitive dynamic selection method, to multimodal instruction tuning.

While prior data selection studies commonly describe their budgets using the retained data fraction, our comparison additionally matches online FLOPs so that the computation required for online selection is charged to the common budget rather than provided as additional free compute.

We agree that this distinction alone does not establish end-to-end cost equivalence. We therefore provide the complete GPU-hour breakdown against VisNec in our response to Weakness 1 and Weakness 2. In the measured setting, DOVAS achieves higher performance with a slightly lower total GPU-hour cost. This measurement complements the original comparison under a common online FLOP budget.

[1] VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning.

[2] Selective Training for Large Vision Language Models via Visual Information Gain.

[3] OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration.

[4] LESS: Selecting Influential Data for Targeted Instruction Tuning.

Thank you for the detailed response. Most of my concerns have been addressed. I will maintain my positive score.

Comments on the paper

2 comments
Area_Chair_g8exOpenReview ↗

Please read the response from authors and discuss

Dear reviewer fellows,

The author rebuttal has been posted. The rebuttal period is a critical part of our review process, allowing authors to clarify misunderstandings and address specific concerns.

Please take a moment to read the author response, engage in the discussion by replying to the authors' responses, and update your review if the rebuttal changes your assessment.

Your active participation is vital to ensuring a fair and thorough evaluation of the submissions. A constructive discussion helps us make the most informed decisions.

Thank you for your time and valuable contributions to the review process.

Best regards,

Your AC

Dear reviewers,

We hope we were able to resolve most of your primary concerns through our rebuttal. We would be happy to provide further clarification on any point!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.