- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CAP, a model-aware data selection method that compares reference responses with the model's own generations and measures representation divergence in shallow and deep layers. It then scores candidate examples using the residual obtained by regressing deep-layer divergence on shallow-layer divergence. Experiments on Marin-8B-Base across mathematics, code, and science tasks demonstrate the effectiveness of CAP, with particularly strong results on mathematical reasoning.
Critical strengths
- The paper identifies two interesting layer-wise patterns: activation stable rank follows a U-shaped profile, while the weight profile exhibits an approximately inverted-U shape. Based on these observations, the paper proposes CAP, a model-aware data selection method.
- The experimental results demonstrate the generality and robustness of CAP across multiple tasks, with particularly notable improvements in mathematical reasoning.
Critical weaknesses
- The comparison with recent methods is insufficient. The authors are encouraged to include more recent data selection methods from the past two years to better demonstrate the performance advantages of CAP.
- The organization could be improved. For example, the proposed method could be presented in a separate section to improve clarity and readability.
- The connection between the theoretical analysis and the proposed method needs further clarification. Although the paper identifies complementary layer-wise rank patterns, it remains unclear how these observations directly motivate the design of CAP, particularly the shallow/deep layer split and the regression-based scoring mechanism.
Questions for authors
Please see the content in Weaknesses.
We thank the reviewer for the positive evaluation and the helpful suggestions on positioning and presentation.
Thank you for this suggestion. We would like to explain how the baselines were chosen, and we add a comparison that highlights what distinguishes CAP from recent methods.
How we chose baselines. Rather than comparing against every method, we cover each major family of selection signals with a well-established representative, all under one controlled protocol (same pool, budget, data order, seed and SFT recipe). Six baselines × three domains × four budgets, plus CAP, amount to 84 full-parameter SFT runs (96 with DSIR, added below). Recent methods are largely newer members of these families:
Within this landscape, CAP combines two properties: it is target-free, and it scores training value directly from the model's internal representations. Unlike gradient-based methods (LESS, G-Vendi, GRACE) it is inference-only, at roughly the FLOP cost of one forward pass per candidate.
Comparison with target-aware selection. LESS and DSIR are given the GSM8K training split as their math target. Every math checkpoint was also evaluated on MATH500, which is no method's target:
Table R1: Math at the 8k budget (Marin-8B-Base, 80k NuminaMath pool)
We will add DSIR and the MATH500 results to the paper, and we will consider further extending the baseline set with recent methods such as MIWV and RDS+ in the revision.
The method is presented in Sec. 4 ("CAP: Counterfactual Assimilation Profile for Model-Aware Data Selection"), but we agree it can be made easier to find. We will (i) shorten Sec. 3 to the observation and its two tested predictions, moving the formal capacity model to the appendix; (ii) open Sec. 4 with an algorithm box (inputs → greedy counterfactual → per-layer divergence → depth split → pool-level regression → z-score → top-k); and (iii) give the experimental setup its own section.
Each design choice corresponds to one element of the analysis and is supported by data:
The resulting selections generalize beyond the benchmark used for evaluation: CAP is the best selector on the untargeted MATH500 (Table R1). The theory does not fix the exact split points (0.33 / 0.66) or the horizon R. These are empirical defaults, and Table 5 shows CAP is robust to R across nearly an order of magnitude.