- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CAIS, a curvature aware data selection framework for visual instruction tuning. It formulates selection as budget constrained Poisson sampling that minimizes expected validation risk after weighted subset retraining. A second order approximation yields closed form inclusion probabilities based on validation curvature along each sample’s influence direction, together with an asymptotic excess risk bound. Experiments across several vision language models and multimodal benchmarks show improved aggregate performance over uniform sampling and existing influence based selection methods.
Critical strengths
-
Principled sampling formulation. The paper directly optimizes expected validation risk after budget constrained subset retraining and derives a closed form curvature aware sampling rule with an excess risk guarantee. This provides a coherent alternative to heuristic pointwise ranking and clarifies its relationship to classical influence based sampling (Section 3).
-
Broad empirical validation. CAIS is evaluated across multiple selection budgets, vision language model families, and multimodal benchmarks, with comparisons to several relevant baselines. Its consistent aggregate gains, particularly at the smallest budget, provide meaningful evidence that validation curvature can improve data selection in practice (Section 4 and Appendix B).
Critical weaknesses
-
The reported efficiency is not end to end. CAIS first fine tunes a Reference model on the complete 625K candidate set, then extracts features for all candidates, selects a subset, and retrains from the base model. The reported training savings exclude Reference training and selection, while candidate feature extraction alone takes 8.37 of the 9.03 hours reported for the selection pipeline (Section 4.3 and Appendix B.1). Thus, the claim that 6.4% of the data retains 98.38% of full performance describes only the final retraining set and does not establish net savings for a one time use case. This substantially weakens the paper’s central efficiency motivation unless the authors provide complete cost accounting and a realistic amortization scenario.
-
The novelty is insufficiently positioned against closely related work. The discussion overlooks established validation guided subset selection methods such as GLISTER [1], which directly optimizes held out validation performance, and recent targeted instruction selection methods such as GIST [2], which models coupled, non diagonal validation geometry under LoRA. These works are directly relevant to the paper’s validation risk objective and its motivation for moving beyond first order influence. Without clearly distinguishing CAIS from these formulations, it is difficult to assess which aspects constitute a substantive advance. A conceptual comparison and, where feasible, an empirical comparison with the closest scalable methods are needed.
[1] Killamsetty, Krishnateja, et al. “GLISTER: Generalization Based Data Subset Selection for Efficient and Robust Learning.” AAAI 2021. [2] Min, Guanghui, et al. “GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry.” ICML 2026.
Questions for authors
-
Since CAIS requires training a full data Reference model before selection, why would a practitioner retrain a new model from the base checkpoint on the selected subset rather than directly use the already trained Reference model? Please clarify the practical deployment or amortization setting and report the corresponding end to end compute cost.
-
Do all experiments use the Horvitz Thompson weighted loss with inverse inclusion weights during subset retraining? If so, how are potentially large weights handled in optimization? If not, how does the theoretical analysis apply to the implemented method?
-
How accurately do the CountSketch and Gauss Newton approximations preserve the exact curvature scores and resulting inclusion probabilities? Can the authors report score correlation, selected subset overlap, or downstream sensitivity to the sketch dimension and damping coefficient?