- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies instruction selection for supervised fine-tuning when target concepts receive unequal pretrained support. It introduces a prior barrier comparing a target concept's support with that of competing concepts, and derives posterior concentration and predictive risk bounds within a Bayesian concept-space model. The proposed PASS method constructs reference-derived concepts, estimates distinguishing evidence from desired and incorrect response representations, and selects instructions using barrier-aware weights, diminishing coverage rewards, and redundancy penalties. Its allocation weights satisfy , combining the estimated barrier with existing pretrained support. Experiments compare two 7B backbones and two selection budgets across five downstream capabilities. PASS reports competitive aggregate performance and small improvements over equal per-capability allocation.
Critical strengths
- An explicit formulation of pretrained support asymmetry.
The paper separates the target concept's initial pretrained disadvantage from the distinguishing evidence provided by supervision. This provides a coherent motivation for considering both existing model support and candidate utility when allocating a limited fine-tuning budget.
- Promising empirical budget efficiency.
The evaluation includes two backbones, two budgets, and seven selection baselines under shared fine-tuning settings. On Mistral-7B, PASS with 5K examples achieves 51.5, exceeding the strongest reported 10K baseline score of 50.9. These results suggest potential practical value, subject to establishing their robustness.
Critical weaknesses
- The algorithmic increment over closely related adaptive selection methods is insufficiently isolated.
MIG, published in Findings of ACL 2025, already models diminishing information gains across semantic labels and explicitly evaluates the exponential saturation function . BIDS, published in Findings of EMNLP 2025, normalizes candidate-target utilities and iteratively favors underrepresented targets. Its Appendix A.7 further demonstrates this mechanism using hidden-representation similarities through RDS-BIDS. Thus, adaptive coverage and representation-based balancing are established ingredients.
PASS's more distinctive components appear to be its desired/incorrect-response evidence and pretrained barrier/support weights. Although Section 6 acknowledges related approaches, the experiments do not establish the added value of these components over the shared coverage mechanism. In particular, Table 2 compares against equal allocation across five capabilities, rather than retaining the same concepts and objective while removing barrier-based weighting. This leaves the principal methodological contribution insufficiently demonstrated.
- The evidence supporting the barrier-based explanation and the reported improvements remains incomplete.
The theory concerns a Gibbs posterior with independent and identically distributed supervision, whereas the experiments use LoRA fine-tuning on adaptively selected heterogeneous instructions. The correspondence between the cosine-margin surrogates and theoretical prior ratios or likelihood evidence is not validated. Corollary 1 also establishes sufficient supervision conditions, rather than necessary head-tail sample requirements. Empirically, Table 1 reports only aggregate scores without repeated-run uncertainty. PASS's margins over the strongest baselines are approximately 0.9, 0.9, 0.2, and 0.0 points. PASS and RANDOM both report 65.6 for Qwen2.5-7B with a 10K budget, while Table 2's allocation gains are only 0.26–0.56 points. Consequently, neither the reliability of the improvements nor their attribution to prior-barrier modeling is sufficiently established.
Questions for authors
-
Could the authors clarify the incremental contribution relative to MIG and BIDS, including RDS-BIDS? A controlled comparison using the same concept partition, evidence matrix, and coverage objective, with uniform versus barrier/support-based weights, would directly test the proposed contribution.
-
Do the estimated barriers predict actual supervision requirements or subsequent fine-tuning gains beyond simpler concept-level signals such as reference error rate or response loss? Results stratified by barrier would help validate the proposed explanation. Please also clarify the intended scope of the connection between the Gibbs-posterior analysis and practical LoRA fine-tuning.
-
Could the authors provide per-benchmark results and variability across repeated selection and fine-tuning runs? In particular, are the Table 2 improvements larger than run-to-run variation, and what is the unrounded PASS-RANDOM difference for Qwen2.5-7B with a 10K budget?
-
Could the authors specify the calibration functions, aggregation rule, clustering criterion, and complete training and evaluation configuration? Please clarify how Equation 13 handles and how the redundancy term is defined when during greedy initialization.