acceptodds
Withdrawn from ICLR 2027

Rethinking Influence-Based Data Selection: A Curvature-Aware Sampling Framework for Visual Instruction Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting informative training data is central to efficient multimodal instruction tuning. While existing methods widely rank examples via first-order, pointwise influence on validation loss, such filtering provides no guarantees on the validation risk after subset retraining. To address this limitation, we introduce a budget-constrained Poisson sampling framework that optimizes inclusion probabilities to minimize the expected validation risk after subset retraining. Crucially, we show that the validation curvature governs the retrained risk and derive a second-order risk surrogate endowed with a closed form optimal inclusion rule. We further establish an excess risk bound that decays as with the sampling budget , and characterize the relationship between existing influence function methods and the proposed framework. Extensive experiments across multiple vision language models, multimodal benchmarks, and selection budgets demonstrate the effectiveness of our method; notably, with an expected selection budget of only 6.4%, it retains 98.38% of the performance of training on full dataset.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Influence Function, Dataselection, Visual Instruction Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_ns2QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAIS, a curvature aware data selection framework for visual instruction tuning. It formulates selection as budget constrained Poisson sampling that minimizes expected validation risk after weighted subset retraining. A second order approximation yields closed form inclusion probabilities based on validation curvature along each sample’s influence direction, together with an asymptotic excess risk bound. Experiments across several vision language models and multimodal benchmarks show improved aggregate performance over uniform sampling and existing influence based selection methods.

Critical strengths
  1. Principled sampling formulation. The paper directly optimizes expected validation risk after budget constrained subset retraining and derives a closed form curvature aware sampling rule with an excess risk guarantee. This provides a coherent alternative to heuristic pointwise ranking and clarifies its relationship to classical influence based sampling (Section 3).

  2. Broad empirical validation. CAIS is evaluated across multiple selection budgets, vision language model families, and multimodal benchmarks, with comparisons to several relevant baselines. Its consistent aggregate gains, particularly at the smallest budget, provide meaningful evidence that validation curvature can improve data selection in practice (Section 4 and Appendix B).

Critical weaknesses
  1. The reported efficiency is not end to end. CAIS first fine tunes a Reference model on the complete 625K candidate set, then extracts features for all candidates, selects a subset, and retrains from the base model. The reported training savings exclude Reference training and selection, while candidate feature extraction alone takes 8.37 of the 9.03 hours reported for the selection pipeline (Section 4.3 and Appendix B.1). Thus, the claim that 6.4% of the data retains 98.38% of full performance describes only the final retraining set and does not establish net savings for a one time use case. This substantially weakens the paper’s central efficiency motivation unless the authors provide complete cost accounting and a realistic amortization scenario.

  2. The novelty is insufficiently positioned against closely related work. The discussion overlooks established validation guided subset selection methods such as GLISTER [1], which directly optimizes held out validation performance, and recent targeted instruction selection methods such as GIST [2], which models coupled, non diagonal validation geometry under LoRA. These works are directly relevant to the paper’s validation risk objective and its motivation for moving beyond first order influence. Without clearly distinguishing CAIS from these formulations, it is difficult to assess which aspects constitute a substantive advance. A conceptual comparison and, where feasible, an empirical comparison with the closest scalable methods are needed.

[1] Killamsetty, Krishnateja, et al. “GLISTER: Generalization Based Data Subset Selection for Efficient and Robust Learning.” AAAI 2021. [2] Min, Guanghui, et al. “GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry.” ICML 2026.

Questions for authors
  1. Since CAIS requires training a full data Reference model before selection, why would a practitioner retrain a new model from the base checkpoint on the selected subset rather than directly use the already trained Reference model? Please clarify the practical deployment or amortization setting and report the corresponding end to end compute cost.

  2. Do all experiments use the Horvitz Thompson weighted loss with inverse inclusion weights 1/πi1/\pi_i during subset retraining? If so, how are potentially large weights handled in optimization? If not, how does the theoretical analysis apply to the implemented method?

  3. How accurately do the CountSketch and Gauss Newton approximations preserve the exact curvature scores and resulting inclusion probabilities? Can the authors report score correlation, selected subset overlap, or downstream sensitivity to the sketch dimension and damping coefficient?

Reviewer_oPNTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a curvature-aware influence-based data selection method for visual instruction tuning. It formulates budget-constrained selection as designing Poisson inclusion probabilities that minimize expected validation risk after Horvitz–Thompson-weighted retraining. The authors show that influence-function-based sampling is a first-order special case driven by validation-gradient outer products, and introduce a validation-curvature term that yields a closed-form sampling rule. They provide an excess-risk bound and scale the method with CountSketch gradients and Gauss–Newton Hessian estimates. Experiments on LLaVA-Vicuna-7B over LLaVA-V-625K, plus Qwen3-VL-4B/8B, show that the method achieves the best aggregate relative performance among selectors at 40K, 100K, and 200K budgets, reaching 98.38% of full-data performance with only 6.4% of candidates at 40K. A cost analysis shows feature extraction dominates runtime, while solving the sampling design itself is cheap.

Critical strengths
  1. Framing selection as minimizing expected validation risk under Poisson sampling with HT-weighted retraining is clean and connects data selection to optimal subsampling theory. The derivation of influence-based sampling as a first-order special case is a useful conceptual contribution.

  2. The excess-risk bound and comparison with uniform and influence-based sampling are stronger than most data-selection papers provide. The paper also proves that the proposed design can have lower asymptotic risk than uniform and scalar influence-based sampling under stated conditions.

  3. Good empirical gains under tight budgets. The largest improvement appears at the 40K budget, where CAIS retains 98.38% of full-data relative performance using only 6.4% of candidates. This is the most compelling practical result.

Critical weaknesses
  1. The cost accounting is incomplete and can be misleading. The method requires a full-data reference model and candidate feature/curvature extraction. In the 40K timing run, selection takes about 9.03 hours, of which 8.37 hours is feature extraction, while one training epoch takes only 0.50 hours. The reported compute savings exclude feature extraction and selection. Thus the practical benefit is mainly for multi-epoch training or repeated reuse, not for a single-epoch selection-and-train pipeline. The “training-free” framing should be qualified.

  2. Validation construction is target-specific and may overstate generalization. The validation set is a random 20% of each benchmark’s test examples, with the remaining 80% used for testing. This is a legitimate target-aware setup, and the split is disjoint, but selection is directly aligned with the seven evaluation benchmarks. There is no evaluation on unseen benchmarks or held-out task families. The results therefore measure benchmark-targeted selection, not general-purpose subset quality.

  3. The main guarantee concerns exact coefficients at a fixed reference checkpoint. In practice, CAIS uses sketched gradients, Gauss–Newton Hessian estimates, pseudo-label draws, damping, and a coefficient floor. The paper provides a sensitivity bound for relative score error, but it does not give a full finite-sample guarantee for the actual sketched pipeline. Key approximations are not ablated systematically.

Questions for authors
  1. “Rel.” is an average of normalized scores across nine metrics, and CAIS is not uniformly best. For example, at 100K its MME-C is substantially below LESS and Full, while its aggregate is highest. The weighting of metrics is implicit and may favor certain behaviors. Per-metric wins/losses and worst-case performance should be discussed more explicitly.

  2. Main results use three runs, but external comparisons lack confidence intervals or significance tests. At 200K, CAIS’s advantage over Uniform is only 0.07 points, which is unlikely to be meaningful without repeated runs. The strong claims should be tempered at larger budgets.

Reviewer_wFTLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CAIS for visual instruction data selection. It derives budget-constrained Poisson sampling probabilities from a curvature-aware approximation of validation risk after subset retraining, and uses gradient sketching and Gauss–Newton curvature estimates to make the computation tractable.

Critical strengths
  1. The paper addresses the important problem of selecting informative multimodal instruction data to improve fine-tuning efficiency.
  2. The curvature-aware formulation of budget-constrained sampling is interesting.
Critical weaknesses
  1. Several closely related studies on multimodal instruction data selection, particularly in LLaVA-665K-based settings, are missing from the discussion or comparisons [1–7]. ICONS [1] is especially relevant, as it extends LESS to visual instruction data selection.
  2. The evaluation setting makes direct comparison with established methods such as ICONS[1] and COINCIDE[2] difficult. Evaluation on shared benchmarks and aligned budgets would make the relative performance easier to assess.
  3. As stated in lines 2275–2277, the reference model is first trained on the full dataset. This introduces substantial computational overhead that is not accounted for in the cost analysis in Table 7. This also raises concerns about the fairness of the comparisons if the baselines do not use comparably trained reference models.
  4. The implementation details of the reproduced baselines are insufficiently reported, making it difficult to assess comparison fairness.
  5. The efficiency analysis lacks comparisons with other data selection baselines.
  6. The paper states that the validation set combines seven tasks, with 20% of the benchmark test examples randomly sampled within each task. However, the exact composition of this multi-task validation set and its impact on the trained model’s out-of-distribution (OOD) performance remain unclear.

[1] ICONS: Influence Consensus for Vision-Language Data Selection. arXiv:2501.00654.

[2] Concept-skill Transferability-based Data Selection for Large Vision-Language Models. EMNLP, 2024.

[3] Less is More: High-value Data Selection for Visual Instruction Tuning. ACM Multimedia, 2025.

[4] Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness. ICCV, 2025.

[5] CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization. NeurIPS, 2025.

[6] Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models. NeurIPS, 2025.

[7] PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection. EMNLP, 2026

Questions for authors

Could the authors provide a matched comparison with COINCIDE/ICONS at the commonly used 20% budget, or clearly clarify the rationale for modifying the evaluation protocols?

Reviewer_pV4bOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Curvature-Aware Influence Sampling (CAIS), which uses validation-curvature-weighted influence scores to optimize data inclusion probabilities under a sampling budget for visual instruction tuning. It derives a second-order risk surrogate and a conditional excess-risk bound, and evaluates a computationally approximated implementation on LLaVA and Qwen3-VL models across multiple benchmarks and selection budgets.

Critical strengths

The evaluation across multiple VLMs, benchmarks, and selection budgets provides useful empirical evidence that curvature-aware sampling can preserve much of the full-data performance when training on small subsets.

Critical weaknesses
  1. Limited novelty relative to established optimal subsampling theory. Poisson sampling, inverse-probability weighting, and influence-based optimal inclusion rules were already studied by [1]. More directly, [2] derives a square-root sampling rule based on the quadratic form g⊤H−1QH−1gg^\top H^{-1}QH^{-1}g, explicitly allowing the target-loss Hessian to define QQ. CAIS uses essentially the same score structure with validation curvature. The paper does not adequately distinguish its additional theoretical contribution from these precedents, making the methodological advance appear highly incremental.

  2. The theoretical guarantee is insufficiently connected to the implementation. The excess-risk bound requires strong local stability, optimization accuracy, and a vanishing validation gradient for the advertised o(B−1)o(B^{-1}) rate. However, the experiments use sketched curvature estimates and train fresh adapters from the base model rather than continuing from the reference checkpoint. The paper does not establish that the required conditions or approximation accuracy hold in these experiments, leaving the guarantee’s practical relevance unclear.

  3. The claimed efficiency is not established end to end. Appendix B.1 requires full-data training of a reference model before selection, while the reported timing excludes this cost. Feature extraction and selection already take approximately 8.5 hours, compared with about 0.5 hours for the final subset training. Consequently, training on a 6.4% subset does not itself demonstrate computational savings, and the practical benefit remains unclear without accounting for reference training and selection overhead.

Reference:

[1] Daniel Ting and Eric Brochu. Optimal Subsampling with Influence Functions. NeurIPS, 2018.

[2] Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards a Statistical Theory of Data Selection under Weak Supervision. ICLR, 2024.

Questions for authors
  1. How does CAIS differ technically from the curvature-weighted optimal sampling rule in [2], and what substantive theoretical contribution remains beyond this existing formulation?

  2. What evidence supports the local stability, optimization accuracy, and score-approximation conditions required by the guarantee in the actual VLM experiments?

  3. What are the end-to-end computational savings when full-data reference training, feature extraction, and selection are all included?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.