- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes H-MSAE for selecting compact instruction-tuning datasets. It learns sparse concepts at multiple resolutions and builds a soft hierarchy from their activation patterns. This selection procedure combines quality-weighted concept coverage, length quotas, and stochastic greedy sampling. Across six model settings, H-MSAE achieves the highest aggregate scores among the evaluated subset selectors. It also cuts end-to-end selection time to roughly one-fifth of that of MIG. The paper includes component ablations, additional data-pool experiments, and a local theoretical analysis that motivates balancing quality and diversity.
Critical strengths
-
The method offers a useful way to obtain concept coverage without generating natural-language tags for the entire dataset. The learned sparse features and hierarchy can also be reused across selection runs.
-
The efficiency gains are practically meaningful. End-to-end selection takes approximately 7.4 hours, compared with 36.6 hours for MIG, and the paper provides a breakdown of these costs.
Critical weaknesses
-
The ablations support the individual components, but do not fully establish the benefit of concept coverage beyond quality and length control. Random sampling and quality-only selection within the same length bins and quotas would provide useful controls. Without them, it is difficult to assess how much the proposed coverage mechanism adds.
-
The advance over prior SAE-based data selection is not sufficiently clear. The paper builds on an existing MSAE and reports useful component ablations, but lacks a direct comparison with the closely related method of Yang et al. (2025). A clearer comparison of their selection mechanisms would help establish the significance of the hierarchical formulation.
Questions for authors
-
Can you clarify what is new in the hierarchical coverage formulation beyond combining multi-resolution sparse representations, quality weighting, and length constraints? This would help me assess the methodological contribution.
-
Can you compare H-MSAE with the two baselines using the same length bins and quotas: random sampling and select the highest-quality examples based on DEITA scores? This would help show whether concept coverage adds value beyond quality and length control.
3 comments
Thanks for adding the quota-matched baselines and comparing against GreedSelect and SimScale. That addresses my main concerns, so I've raised my score to 3.
Thank you very much for updating your recommendation score. We are happy to see that our rebuttal solved your concerns.
We thank the reviewer for acknowledging the practicability and efficiency of our proposed method. We address your concerns below.
Q1. Can you clarify what is new in the hierarchical coverage formulation beyond combining multi-resolution sparse representations, quality weighting, and length constraints?
A1. Beyond simply combining existing components, our key novelty lies in explicitly constructing and exploiting cross-resolution concept relationships for hierarchical data selection. Unlike Yang et al. (2025), which employs SAE-based selection based on flat feature coverage or overlap, H-MSAE builds upon multi-resolution sparse representations learned by MSAE to model hierarchical semantic redundancy. However, MSAE alone does not guarantee reliable parent–child relationships, and feature absorption may persist.
We therefore introduce two additional coupled mechanisms: 1. Post-hoc parent construction that infers soft cross-resolution parent–child relations from concept co-activation, without retraining MSAE; and 2. Hierarchy-aware concave coverage that propagates concept evidence to ancestors and applies diminishing returns to account for semantic redundancy across resolutions. Together, these mechanisms provide a post-hoc structural correction to MSAE-based coverage, designed to mitigate the effects of residual feature absorption. Crucially, the inferred hierarchy directly determines marginal selection gains, rather than merely providing additional features.
W2. Lacking a direct comparison with the closely related method of Yang et al. (2025).
A2. We directly reproduce the selection procedure in Algorithm 1 of Yang et al. (2025), including instruction-length ordering, iterative concept-set updates, and the original GreedSelect and SimScale selection rules. We evaluate both methods on Qwen3-1.7B and Qwen3-4B using 50K selected examples, three-epoch SFT, and three training seeds. As shown in the following table, H-MSAE consistently outperforms both Yang baselines across all comparisons.
W1/Q2. Can you compare H-MSAE with the two baselines using the same length bins and quotas?
A3. We compare with two additional baselines using the same length bins and integer quotas as H-MSAE: (1) Random, which randomly selects examples within each bin, and (2) Quality-only, which selects examples with the highest DEITA quality scores within each bin. All methods select 50K examples from the same candidate pool and follow the same three-epoch SFT protocol.
On Qwen3-4B-Base, we evaluate all methods across three training seeds shown in the following table. H-MSAE outperforms Random by +2.22 and Quality-only by +0.72 points in paired Final Avg, with consistent positive improvements across all three seeds. Against Quality-only, the improvement is primarily driven by objective evaluation (+1.21 points). Although Quality-only is a strong baseline, the consistent positive gains across all three seeds support the effectiveness of hierarchical concept coverage beyond quality weighting and length constraints.
Thanks for your insightful and constructive comments. We sincerely hope you can reconsider the recommendation score if our explanations and additional experiments solve your concerns.