acceptodds
Under review as a conference paper at ICLR 2027

H-MSAE: Hierarchical Sparse Concept Coverage for Efficient and Diverse Instruction Data Selection

Abstract

Large instruction-tuning corpora contain substantial semantic redundancy, motivating the selection of compact subsets that preserve useful supervision while reducing training cost. Existing diversity-aware selectors often rely on generated semantic tags and explicit graph structures, introducing considerable annotation and preprocessing overhead. We introduce H-MSAE, a hierarchical sparse instruction selection framework that organizes multi-resolution concepts learned by a Matryoshka sparse autoencoder into a coarse-to-fine hierarchy. Quality-weighted, diminishing-return coverage with ancestor propagation promotes high-quality and non-redundant semantic coverage, while pool-relative length constraints regulate subset composition. We motivate this design through a local quality-diversity analysis of subset SFT utility and examine it using example-quality and representation-space diversity diagnostics. On Tulu 3 benchmark, H-MSAE achieves the highest Final Avg among the evaluated 50K selectors across scales and initialization. The selected subset also exhibits the highest mean quality and effective rank among the evaluated methods. By avoiding full-pool generative tagging and supporting efficient sparse selection, H-MSAE reduces measured end-to-end runtime by approximately relative to MIG.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Instruction Data Selection, Sparse Autoencoders, Hierarchical Semantic Coverage

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wPTdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes H-MSAE for selecting compact instruction-tuning datasets. It learns sparse concepts at multiple resolutions and builds a soft hierarchy from their activation patterns. This selection procedure combines quality-weighted concept coverage, length quotas, and stochastic greedy sampling. Across six model settings, H-MSAE achieves the highest aggregate scores among the evaluated subset selectors. It also cuts end-to-end selection time to roughly one-fifth of that of MIG. The paper includes component ablations, additional data-pool experiments, and a local theoretical analysis that motivates balancing quality and diversity.

Critical strengths
  • The method offers a useful way to obtain concept coverage without generating natural-language tags for the entire dataset. The learned sparse features and hierarchy can also be reused across selection runs.

  • The efficiency gains are practically meaningful. End-to-end selection takes approximately 7.4 hours, compared with 36.6 hours for MIG, and the paper provides a breakdown of these costs.

Critical weaknesses
  • The ablations support the individual components, but do not fully establish the benefit of concept coverage beyond quality and length control. Random sampling and quality-only selection within the same length bins and quotas would provide useful controls. Without them, it is difficult to assess how much the proposed coverage mechanism adds.

  • The advance over prior SAE-based data selection is not sufficiently clear. The paper builds on an existing MSAE and reports useful component ablations, but lacks a direct comparison with the closely related method of Yang et al. (2025). A clearer comparison of their selection mechanisms would help establish the significance of the hierarchical formulation.

Questions for authors
  • Can you clarify what is new in the hierarchical coverage formulation beyond combining multi-resolution sparse representations, quality weighting, and length constraints? This would help me assess the methodological contribution.

  • Can you compare H-MSAE with the two baselines using the same length bins and quotas: random sampling and select the highest-quality examples based on DEITA scores? This would help show whether concept coverage adds value beyond quality and length control.

3 comments

We thank the reviewer for acknowledging the practicability and efficiency of our proposed method. We address your concerns below.


Q1. Can you clarify what is new in the hierarchical coverage formulation beyond combining multi-resolution sparse representations, quality weighting, and length constraints?

A1. Beyond simply combining existing components, our key novelty lies in explicitly constructing and exploiting cross-resolution concept relationships for hierarchical data selection. Unlike Yang et al. (2025), which employs SAE-based selection based on flat feature coverage or overlap, H-MSAE builds upon multi-resolution sparse representations learned by MSAE to model hierarchical semantic redundancy. However, MSAE alone does not guarantee reliable parent–child relationships, and feature absorption may persist.

We therefore introduce two additional coupled mechanisms: 1. Post-hoc parent construction that infers soft cross-resolution parent–child relations from concept co-activation, without retraining MSAE; and 2. Hierarchy-aware concave coverage that propagates concept evidence to ancestors and applies diminishing returns to account for semantic redundancy across resolutions. Together, these mechanisms provide a post-hoc structural correction to MSAE-based coverage, designed to mitigate the effects of residual feature absorption. Crucially, the inferred hierarchy directly determines marginal selection gains, rather than merely providing additional features.


W2. Lacking a direct comparison with the closely related method of Yang et al. (2025).

A2. We directly reproduce the selection procedure in Algorithm 1 of Yang et al. (2025), including instruction-length ordering, iterative concept-set updates, and the original GreedSelect and SimScale selection rules. We evaluate both methods on Qwen3-1.7B and Qwen3-4B using 50K selected examples, three-epoch SFT, and three training seeds. As shown in the following table, H-MSAE consistently outperforms both Yang baselines across all comparisons.

ModelMethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
Qwen3-1.7BH-MSAE63.01 ± 0.3629.56 ± 0.4546.29 ± 0.37—
Yang GreedSelect61.31 ± 0.3426.75 ± 0.8544.03 ± 0.58+2.26
Yang SimScale59.85 ± 0.0927.28 ± 0.4143.56 ± 0.20+2.73
Qwen3-4BH-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13—
Yang GreedSelect74.70 ± 0.5736.77 ± 0.3755.74 ± 0.39+3.12
Yang SimScale74.75 ± 0.3538.14 ± 0.4556.45 ± 0.07+2.41

W1/Q2. Can you compare H-MSAE with the two baselines using the same length bins and quotas?

A3. We compare with two additional baselines using the same length bins and integer quotas as H-MSAE: (1) Random, which randomly selects examples within each bin, and (2) Quality-only, which selects examples with the highest DEITA quality scores within each bin. All methods select 50K examples from the same candidate pool and follow the same three-epoch SFT protocol.

On Qwen3-4B-Base, we evaluate all methods across three training seeds shown in the following table. H-MSAE outperforms Random by +2.22 and Quality-only by +0.72 points in paired Final Avg, with consistent positive improvements across all three seeds. Against Quality-only, the improvement is primarily driven by objective evaluation (+1.21 points). Although Quality-only is a strong baseline, the consistent positive gains across all three seeds support the effectiveness of hierarchical concept coverage beyond quality weighting and length constraints.

MethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
H-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13-
Quality-only + quotas76.41 ± 0.2839.86 ± 0.6658.14 ± 0.43+0.72
Random + quotas75.06 ± 1.1938.22 ± 0.1756.64 ± 0.66+2.22

Thanks for your insightful and constructive comments. We sincerely hope you can reconsider the recommendation score if our explanations and additional experiments solve your concerns.

Thanks for adding the quota-matched baselines and comparing against GreedSelect and SimScale. That addresses my main concerns, so I've raised my score to 3.

Thank you very much for updating your recommendation score. We are happy to see that our rebuttal solved your concerns.

Reviewer_wxHwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes H-MSAE, a hierarchical sparse concept coverage framework for efficient and diversity-aware instruction data selection. The method employs a Matryoshka Sparse Autoencoder (MSAE) to extract multi-resolution sparse concepts from frozen conversation embeddings and constructs a soft concept hierarchy through cross-resolution co-activation. A quality-weighted concave coverage objective, combined with ancestor propagation and pool-relative length quotas, is optimized using stochastic greedy selection.

The authors further provide a local quality–diversity analysis of subset SFT utility and empirically evaluate the method on Tulu 3 across multiple model scales and architectures. H-MSAE achieves the highest aggregate score among the evaluated 50K selection methods on Qwen3 models, with particularly notable improvements in subjective evaluation. It also reduces measured end-to-end selection time by approximately 4.96× compared with MIG.

The paper addresses a practically important problem and presents promising results. However, several aspects of the experimental design and methodological validation leave the independent contribution of hierarchical sparse coverage insufficiently established.

Critical strengths

S1. A practically relevant and computationally efficient data selection framework.

The paper addresses semantic redundancy in large-scale instruction-tuning corpora, where effective subset selection can substantially reduce downstream training costs. Replacing expensive full-pool generative tagging with sparse concepts learned from frozen embeddings is a promising direction. The reported reduction in end-to-end selection time from 36h34m for MIG to 7h22m for H-MSAE is a meaningful practical advantage.

S2. A coherent integration of semantic coverage, sample quality, and subset composition.

The proposed framework combines multi-resolution sparse concepts, cross-resolution ancestor propagation, quality-weighted diminishing-return coverage, and length-aware selection constraints. These components address complementary concerns in instruction data selection, and the formulation is conceptually intuitive.

S3. Extensive downstream evaluations across model scales and architectures.

The experiments include multiple Qwen3 model sizes, both base and instruction-tuned initialization, and additional evaluations on Llama and Mistral models. The study also considers multiple selection budgets and additional data pools. The consistent aggregate improvements across the primary Qwen3 experiments provide encouraging evidence for the effectiveness of the selected subsets.

S4. Useful component ablations and efficiency analysis.

The paper includes ablations of multi-resolution reconstruction, ancestor propagation, quality weighting, and length constraints. The separate profiling of prerequisite computation, structure construction, and core sampling is particularly useful for understanding the sources of computational savings.

Critical weaknesses

W1. Insufficient evidence for the independent contribution of the learned concept hierarchy.

Although the ablation study demonstrates the effectiveness of ancestor propagation, it remains unclear whether the improvements arise from meaningful coarse-to-fine semantic relationships or simply from additional feature propagation and coverage. Since the hierarchy is constructed based on statistical co-activation, its semantic validity is not directly established. Consequently, the specific advantage of hierarchical sparse coverage over simpler flat or multi-resolution coverage remains insufficiently justified.

W2. Potential confounding effects of data length and training-token budget.

The proposed length-aware quotas explicitly influence the length distribution of selected examples. Under a fixed example budget and epoch-based SFT, different selection methods may result in different total training-token counts and optimization steps. Although length control is evaluated through ablation, the paper does not sufficiently disentangle the contribution of semantic coverage from the potential effects of data length and training exposure. This makes it difficult to precisely attribute the observed improvements to the proposed selection objective.

W3. Limited connection between the theoretical analysis and the practical selection objective.

Theorem 4.1 establishes a quality–diversity decomposition under specific assumptions about gradient reliability, alignment, and covariance. However, the proposed method optimizes quality-weighted semantic coverage in a learned sparse representation space, rather than the gradient-based quantities appearing in the theorem. The empirical analysis also relies on surrogate metrics, such as DEITA quality scores and embedding effective rank. Therefore, the theoretical results primarily provide an intuitive motivation rather than a direct justification for the effectiveness of the proposed objective.

W4. Limited consistency and uncertainty analysis of the reported performance gains.

Although H-MSAE achieves the highest aggregate scores in the primary evaluations, its improvements are more pronounced in subjective evaluation than in objective benchmarks. For example, H-MSAE underperforms MIG in objective average on Qwen3-8B-Base, while its overall improvement on Mistral-7B-v0.3 is only 0.07 points. Furthermore, the paper does not report variability across independent downstream training runs or statistical uncertainty for the reported scores. These observations make it difficult to assess the robustness and generalizability of the performance improvements.

Questions for authors

Q1. What is the independent contribution of the learned sparse concept hierarchy?

While the ablation results suggest that ancestor propagation is beneficial, it remains unclear whether the improvement comes from the meaningful coarse-to-fine semantic relationships captured by the learned hierarchy or simply from additional feature propagation and coverage. How should the contribution of the hierarchical structure itself be distinguished from that of multi-resolution sparse representations and flat concept coverage?

Q2. To what extent are the performance gains attributable to semantic coverage rather than differences in data length and training exposure?

H-MSAE explicitly incorporates length-aware quotas that may favor longer examples. Under a fixed example budget and epoch-based SFT, different selected subsets may contain substantially different numbers of training tokens and result in different optimization steps. How can the improvements attributed to hierarchical semantic coverage be disentangled from the potential effects of length distribution and training-token budget?

Q3. How directly does the theoretical quality–diversity analysis support the proposed selection objective?

Theorem 4.1 characterizes subset utility in terms of gradient reliability and curvature-normalized directional diversity under specific assumptions. However, H-MSAE optimizes quality-weighted semantic coverage in a learned sparse representation space, and the empirical diagnostics rely on surrogate metrics. To what extent do these quantities reflect the gradient-based properties established in the theorem, and how might the theoretical conclusions depend on the underlying assumptions?

Q4. How robust and consistent are the reported improvements across evaluation settings?

The improvements appear to be more pronounced in subjective evaluation than in objective benchmarks. For instance, H-MSAE achieves a higher Final Avg than MIG on Qwen3-8B-Base despite a lower objective average, while the improvement on Mistral-7B-v0.3 is relatively small. How should these differences be interpreted, and to what extent do they affect the conclusions regarding the robustness and generalizability of H-MSAE?

4 comments

We thank the reviewer for acknowledging the practicability of our proposed method and our extensive evaluations and ablations. We address your concerns below.


W1/Q1. What is the independent contribution of the learned sparse concept hierarchy?

A1. Our key contribution is not merely using multi-resolution sparse representations, but explicitly constructing cross-resolution concept relationships and incorporating them into the marginal gains of hierarchical coverage. While MSAE provides features at multiple resolutions, it does not explicitly specify how their cross-resolution overlap should be handled during data selection. H-MSAE addresses this through post-hoc parent construction and ancestor-aware concave coverage, allowing inferred concept relationships to directly influence sample selection.

To disentangle these contributions, we introduce three controlled ablations:

  1. Fine-only: Uses only the finest-resolution features, serving as a single-resolution reference.
  2. Flat multi-resolution: Retains features at all resolutions without ancestor propagation, isolating the benefit of multi-resolution coverage.
  3. Shuffled-parent: Randomizes parent–child connections while preserving multi-resolution representations, parent-degree structure, and ancestor propagation, isolating the contribution of the inferred parent topology.

The following table reports results over three training seeds.

ModelMethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
4B-BaseH-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13—
Fine-only76.91 ± 0.4738.16 ± 0.6057.54 ± 0.18+1.32
Flat multi-resolution77.58 ± 0.2539.85 ± 0.1658.71 ± 0.20+0.15
Shuffled-parent77.29 ± 0.1939.66 ± 0.4058.48 ± 0.12+0.39
8B-BaseH-MSAE80.65 ± 0.2842.85 ± 0.1661.75 ± 0.19—
Fine-only79.72 ± 0.2441.48 ± 0.4060.60 ± 0.31+1.16
Flat multi-resolution80.20 ± 0.1142.85 ± 0.1961.52 ± 0.09+0.23
Shuffled-parent80.11 ± 0.1942.66 ± 0.3361.39 ± 0.21+0.37

The results reveal three observations. First, flat multi-resolution coverage improves over fine-only by +1.17/+0.93 Final Avg points on 4B/8B, confirming that multi-resolution representations provide substantial benefits. Second, H-MSAE further improves over flat multi-resolution by +0.15/+0.23 points, indicating an additional benefit from hierarchy-aware coverage. Third, H-MSAE outperforms shuffled-parent by +0.39/+0.37 points, while shuffled-parent performs slightly worse than flat multi-resolution. This suggests that arbitrary ancestor propagation is insufficient and that the inferred parent topology provides useful structural information for guiding coverage.

Importantly, the inferred hierarchy does not simply introduce additional features, rather, it directly affects sample selection through ancestor-aware marginal coverage gains. These ablations distinguish the substantial contribution of multi-resolution representations from the additional contribution of learned cross-resolution relationships, providing empirical evidence for the value of the proposed hierarchical selection mechanism.

W2/Q2. To what extent are the performance gains attributable to semantic coverage rather than differences in data length and training exposure?

A2. To disentangle these effects, we conduct two complementary controlled comparisons: (1) length-quota-matched selection baselines and (2) fixed supervised-token budget comparisons.

(1) Controlling for length distribution. We compare H-MSAE against two baselines using identical length bins and per-bin quotas: Quality-only, which selects the highest-quality examples within each bin without semantic coverage optimization, and Random, which samples uniformly within each bin. This controls the length-allocation policy while varying the selection strategy.

The following table shows the results on Qwen3-4B-Base with 3 repeated runs. Under the same length quotas, H-MSAE improves Final Avg over Quality-only by +0.72 points, including +1.21 points on objective benchmarks, and outperforms Random by +2.22 points. These results suggest that the gains cannot be explained solely by the length-allocation policy.

MethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
H-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13—
Quality-only + quotas76.41 ± 0.2839.86 ± 0.6658.14 ± 0.43+0.72
Random + quotas75.06 ± 1.1938.22 ± 0.1756.64 ± 0.66+2.22

(2) Controlling for supervised-token budget. We further compare H-MSAE with MIG under an identical budget of 123.73M supervised tokens on Qwen3-4B-Base. We choose this budget to closely match the supervised tokens consumed by H-MSAE in Table 1. The results are shown in the table below.

MethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
H-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13—
MIG77.00 ± 0.5938.58 ± 0.1957.79 ± 0.32+1.07

Under the matched supervised-token budget, H-MSAE retains a +1.07 improvement in Final Avg over MIG, with consistent improvement across all seeds. These results indicate that H-MSAE's advantage comes from algorithmic design rather than just greater supervised-token exposure.


W3/Q3. How directly does the theoretical quality–diversity analysis support the proposed selection objective?

A3. We clarify that Theorem 4.1 provides a theoretical motivation for jointly considering quality and diversity, rather than a direct guarantee for H-MSAE's hierarchical coverage objective.

Specifically, under a local second-order approximation of SFT, homogeneous target-gradient alignment and self-curvature, and a quality-dependent gradient covariance model with uncorrelated fluctuations, the theorem shows that expected target-risk reduction increases with gradient reliability and curvature-normalized directional diversity. These assumptions isolate their contributions; when they do not hold, additional terms may affect subset utility, and the decomposition need not retain the same form.

H-MSAE operationalizes these two principles through tractable surrogate signals: DEITA scores approximate supervision quality, while diminishing-return sparse concept coverage encourages non-redundant semantic information. The learned hierarchy further accounts for redundancy across resolutions through ancestor propagation. However, semantic coverage is not mathematically equivalent to gradient-space diversity, and we do not establish a formal bound connecting the two. Thus, the theorem motivates the design principles, while the effectiveness of the specific hierarchical objective is supported empirically, including comparisons against flat multi-resolution coverage and shuffled-parent relations.

W4/Q4. How robust and consistent are the reported improvements across evaluation settings?

A4. We conduct additional three-seed downstream training experiments to assess the consistency and uncertainty of the reported improvements, based on which we clarify the cross-model performance differences.

(1) Three-seed reproducibility and uncertainty analysis. To assess whether the observed improvements are robust to downstream training randomness, we conduct additional experiments with three independent training seeds on Qwen3-1.7B/4B/8B and Mistral-7B following the original settings of Table 1 and Table 6.

ModelMethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
Qwen3-1.7BH-MSAE63.01 ± 0.3629.56 ± 0.4546.29 ± 0.37—
MIG62.13 ± 0.3228.34 ± 0.2745.24 ± 0.11+1.05
Qwen3-4BH-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13—
MIG76.67 ± 0.2037.94 ± 0.5457.30 ± 0.24+1.56
Qwen3-8BH-MSAE80.65 ± 0.2842.85 ± 0.1661.75 ± 0.19—
MIG80.84 ± 0.0941.42 ± 0.4261.13 ± 0.18+0.62
Mistral-7BH-MSAE61.27 ± 0.6142.88 ± 0.6952.07 ± 0.65—
MIG60.97 ± 0.2541.42 ± 0.5151.20 ± 0.27+0.87

(2) Interpreting performance differences across models. Across three training seeds, H-MSAE achieves positive mean Final Avg improvements over MIG on Qwen3-1.7B/4B/8B (+1.05/+1.56/+0.62 points) and Mistral-7B-v0.3 (+0.87 points). The improvement on Qwen3-8B is primarily driven by subjective evaluation (+1.43 points), while its objective score remains competitive. Notably, the original single-run result on Mistral showed only a +0.07-point improvement, whereas the three-seed average shows a larger gain. Overall, these results support the consistency of aggregate gains.

Beyond downstream performance, H-MSAE provides a 4.96× end-to-end data-selection speedup over MIG (7h22m vs. 36h34m) by avoiding costly full-pool generative tagging, offering a substantial efficiency advantage even when performance gains vary across models.


Thanks for your insightful and constructive comments. We sincerely hope you can reconsider the recommendation score if our explanations and additional experiments solve your concerns.

Dear Reviewer wxHw,

As the discussion period is approaching its conclusion, we would like to send a gentle reminder regarding our responses. We are sincerely grateful for your time and effort devoted to carefully evaluating our manuscript. To address your concerns, we have provided point-by-point responses to all the questions and concerns raised in the initial reviews.

We would be grateful if you could let us know whether our responses have adequately addressed your concerns. We would be happy to answer any further questions and would be delighted to provide additional clarifications.

Best regards, The Authors

Reviewer_m9wMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes H-MSAE for selecting instruction-tuning subsets without downstream task labels. Frozen conversation embeddings are encoded with a Matryoshka sparse autoencoder, and cross-resolution co-activation defines a soft concept hierarchy. Selection maximizes quality-weighted concave coverage with ancestor propagation, subject to pool-derived length quotas. Experiments select 50K examples from Tulu 3 and evaluate several model scales, initializations, and families, with additional data-pool comparisons in the appendix. The paper reports improved aggregate downstream performance and an approximately fivefold reduction in end-to-end selection time relative to MIG.

Critical strengths
  1. The method offers a practical alternative to full-pool generative tagging. Table 2 includes embedding, scoring, representation learning, and selection prerequisites: H-MSAE takes 7h22m versus 36h34m for MIG on the reported hardware. Separating this end-to-end result from cached selection-stage timings makes the efficiency contribution useful and interpretable.
  2. The evaluation goes beyond one model setting. Table 1 shows the best Final Avg among the compared 50K selectors at three Qwen3 scales, and Appendix A.2 includes an instruction-tuned initialization, Llama and Mistral models, and additional pools. Table 4 also tests ancestor propagation, multi-resolution reconstruction, quality weighting, and length control, providing evidence for the combined design.
Critical weaknesses
  1. The gains are insufficiently separated from variation and training-composition effects. Table 1 improves Final Avg over MIG by 0.79–0.99 points, while Table 6 shows only a 0.07-point advantage on Mistral. Downstream results lack multi-run uncertainty, and equal example counts with three packed epochs need not imply equal token or optimizer-step budgets when length quotas change the subset. Table 4 uses a different 256-step protocol and removes length control, but does not compare hierarchy-based selection against a quality baseline with the same length quotas. This limits attribution of the improvement to hierarchical coverage.
  2. The quality–diversity analysis gives only indirect support for the implemented selector. Theorem 4.1 assumes homogeneous target-gradient alignment and self-curvature, uncorrelated fluctuations, and a specific quality-dependent covariance model. Its diversity is measured in target-Hessian gradient geometry, whereas the selector uses sparse embedding coverage. Section 5.4 acknowledges that DEITA scores and embedding effective rank do not directly measure the theorem's quantities; DEITA is also used for selection itself. The analysis therefore does not yet explain why the learned hierarchy is preferable to a simpler coverage representation.
Questions for authors
  1. For one representative model, can you provide repeated-run uncertainty and actual training-token/optimizer-step counts for H-MSAE and MIG, together with a quality-based control using H-MSAE's identical length quotas? This would clarify whether the reported margin persists under comparable training exposure.
  2. Can you clarify the intended scope of Theorem 4.1 and provide a small diagnostic connecting hierarchical coverage to gradient redundancy, or a control replacing learned parent relations with shuffled relations? The existing no-propagation ablation is useful, but does not distinguish meaningful hierarchy from generic propagation.
3 comments

We thank the reviewer for acknowledging the practicability of our proposed method and the comprehensive evaluation. We address your concerns below.


W1/Q1. Can you provide repeated-run uncertainty, training tokens/steps, and a length-controlled quality-only baseline?

A1. We provide additional three-seed experiments, actual training-token and optimizer-step counts, a token-matched comparison, and a quality-only baseline with identical length quotas.

3-seed repeated-run. We conduct additional experiments on Qwen3-4B-Base using three training seeds under the SFT protocol of Table 1. Compared with MIG, H-MSAE shows a consistent improvement of 1.56 points. Due to differences in sequence packing, H-MSAE and MIG perform 309 and 264 optimizer steps, respectively. We further examine a separate token-matched experiment to ensure fair comparison.

MethodStepsAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
H-MSAE30977.62 ± 0.2040.11 ± 0.0858.86 ± 0.13-
MIG26476.67 ± 0.198737.94 ± 0.5457.30 ± 0.24+1.56

Training exposure and token-matched comparisons. We additionally compare H-MSAE with MIG under an identical budget of 123.73M supervised tokens on Qwen3-4B-Base. We choose this budget to closely match the supervised tokens consumed by H-MSAE in Table 1. The results are shown in the table below.

MethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
H-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13-
MIG77.00 ± 0.5938.58 ± 0.1957.79 ± 0.32+1.07

Under the matched supervised-token budget, H-MSAE retains a +1.07 improvement in Final Avg over MIG, with consistent improvement across all seeds. These results indicate that H-MSAE's advantage comes from algorithmic design rather than just greater supervised-token exposure.

Comparison with a length-controlled quality-only baseline. We compare H-MSAE with a quality-only baseline that selects the highest-quality examples within the same length bins and integer quotas, without semantic coverage optimization.

MethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
H-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13-
Quality&length-only76.41 ± 0.2839.86 ± 0.6658.14 ± 0.43+0.72

We show that H-MSAE achieves a +0.72-point Final Avg improvement across all paired seeds, primarily driven by a +1.21-point objective improvement. These results support the benefit of semantic coverage beyond quality-based selection under identical length quotas, while acknowledging that supervised-token exposure differs between the two methods.


W2/Q2. Can you clarify the intended scope of Theorem 4.1, and provide a control replacing learned parent relations with shuffled relations?

A2. Scope of Theorem 4.1. We clarify that Theorem 4.1 provides a theoretical motivation for quality–diversity trade-offs in subset selection, rather than a direct guarantee for the proposed hierarchical coverage objective. Specifically, under a local second-order approximation of SFT, homogeneous gradient alignment and self-curvature, and a quality-dependent gradient covariance model, the theorem decomposes expected target-risk reduction into contributions from gradient reliability and directional diversity. H-MSAE operationalizes these principles through quality weighting and diminishing-return hierarchical concept coverage. Intuitively, examples covering distinct semantic concepts may induce less redundant gradient directions, making semantic concept coverage a tractable surrogate for directional diversity. However, the theorem does not guarantee the optimality of the learned hierarchy. We clarify these limitations in the revision.

Shuffled-parent experiment. We report an additional shuffled-parent experiment to justify the learned hierarchy. The shuffled-parent control randomly reassigns parent–child connections while preserving the multi-resolution sparse representations, parent-degree structure, and ancestor propagation mechanism.

ModelMethodAvgobj ↑Avgsub ↑Final Avg ↑Δ Final ↑
4B-BaseH-MSAE77.62 ± 0.2040.11 ± 0.0858.86 ± 0.13-
Shuffled-parent77.29 ± 0.1939.66 ± 0.4058.48 ± 0.12+0.39
8B-BaseH-MSAE80.65 ± 0.2842.85 ± 0.1661.75 ± 0.19-
Shuffled-parent80.11 ± 0.1942.66 ± 0.3361.39 ± 0.21+0.37

H-MSAE outperforms shuffled-parent on average at both scales, with consistent improvements. These results show that learned parent relations can provide additional benefits beyond generic hierarchical propagation.


Thanks for your insightful and constructive comments. We sincerely hope that our explanations and additional experiments solve your concerns.

Thank you for the response, which addresses my main concerns. I remain supportive of acceptance. My current score of weak accept already reflects this assessment, so I will keep it.

Thanks very much for your timely reply. We are glad to see that our rebuttal solved your concerns.

Comments on the paper

4 comments

We thank the reviewer for carefully considering our rebuttal and acknowledging that the additional experiments strengthen the empirical evidence. We sincerely appreciate your constructive feedback.

We would like to reiterate that our key contribution is explicitly constructing and exploiting cross-resolution concept relationships for hierarchical data selection. To further substantiate this contribution, we introduced fine-only, flat multi-resolution, and shuffled-parent ablations. In particular, H-MSAE consistently outperforms fine-only selection across all paired runs, demonstrating the benefits of multi-resolution concept coverage. We also provided additional comparisons with existing data selection methods, length-quota-matched baselines, repeated-run evaluations, and supervised-token-controlled experiments to strengthen our empirical analysis.

Regarding your remaining reservations, we would greatly appreciate further clarification on the following points:

  1. Methodological novelty: Which specific aspects of our proposed hierarchy-aware selection mechanism remain insufficiently differentiated from prior methods?
  2. Task generalization: Are you referring to generalization across downstream evaluation tasks, model architectures, or training datasets? We would appreciate clarification on the specific setting you consider insufficiently evaluated.
  3. Synthesized data quality: Could you clarify what this concern refers to in the context of our paper? H-MSAE selects subsets from existing instruction-tuning data rather than generating synthetic training examples. We would also appreciate clarification regarding the reference to "synthesis ablations" in your latest comment.

We sincerely appreciate your time and consideration. Any additional clarification would help us better understand and address your remaining concerns.

As the discussion period is approaching its conclusion, we would like to send a gentle reminder regarding our responses. We are sincerely grateful for your time and effort devoted to carefully evaluating our manuscript. To address your remaining concerns, we have provided additional explanations.

We would be grateful if you could further specify your remaining concerns and let us know whether our responses have adequately addressed them. We would be happy to answer any further questions and would be delighted to provide additional clarifications.

Thank you for the clarifications and additional experiments. I apologize for not carefully checking my second comment, which contained inaccurate wording about task generalization and synthesized data quality. I withdraw those concerns.

Some reservations remain. Table 8 shows modest gains from the explicit hierarchy over flat multi-resolution coverage, at 0.15 and 0.23 points. The theory primarily provides design motivation, with limited direct evidence that greater hierarchical concept coverage corresponds to greater gradient-direction diversity. The subjective evaluation would also benefit from fuller configuration details and stronger robustness validation.

Nevertheless, the controlled comparisons and repeated runs strengthen the empirical evidence, and the complete method offers meaningful performance and efficiency benefits. I have therefore raised my score to 3 (Weak acceptance).

We sincerely thank the reviewer for the clarification, thoughtful reassessment, and willingness to update the score.

We also appreciate your remaining constructive feedback. These suggestions will help us further improve the clarity and presentation of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.