- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies targeted instruction data selection that improves a designated task while preserving performance on other tasks. It proposes STAR, which uses FFN activations from a frozen language model to identify Task-Shared and Target-Specific neurons, constructs separate activation subspaces, and combines prototype-alignment scores for cluster-aware subset selection. Experiments across five target tasks, four OOD benchmarks, two selection budgets, and two backbone models show improved target and known non-target performance, together with competitive OOD results.
Critical strengths
-
Practically meaningful problem formulation. The paper identifies an important limitation of existing targeted selection methods: improving a designated task may reduce performance on other tasks. Framing selection as an asymmetric balance between target specialization and cross task utility is well motivated, and STAR provides a simple mechanism for representing these two objectives through separate activation subspaces (Section 3).
-
Broad empirical evaluation. STAR is evaluated across five target tasks, four OOD benchmarks, two selection budgets, and two backbone models, with additional ablations on neuron selection, score fusion, and clustering. The results provide promising evidence that the proposed design improves target specialization while retaining utility across the known reference tasks (Sections 4 and 5).
Critical weaknesses
-
The evidence for broad cross task utility is limited. STAR directly uses all five reference tasks to construct its Task Shared signal, so the reported Non Target Avg. measures performance on tasks that already participate in selection rather than utility on genuinely unseen tasks. On the fully held out Cross Target OOD evaluation, STAR scores 40.08, slightly below Random at 40.17 and BIDS at 40.36; on Qwen2.5, it also underperforms BIDS on this metric. The current results therefore support improved balancing among known reference tasks more strongly than the broader claim of preserving cross task utility.
-
The source of the improvement is not sufficiently isolated through matched baselines. STAR differs from the main target specific baselines in several respects: it uses all five reference sets, separates two activation subspaces, combines target and shared scores, and applies cluster based selection. Although the appendix extends MONA and NAIT to multiple reference tasks, the paper does not compare against other strong targeted methods such as GIST under the same information setting, or against a simple combination of a target specific selector and a task balanced selector. Consequently, it remains unclear whether the gains require the proposed neuron decomposition or can be obtained through simpler score fusion with the same reference supervision.
Questions for authors
-
How were , , the numbers of selected neurons, and the number of clusters chosen? Were any of these hyperparameters selected using performance on the benchmarks reported in the main results?
-
Are the reported results based on a single fine tuning run? The standard deviations in Table 3 appear to measure variation across target settings rather than variation across random seeds. If repeated runs are available, do the main improvements remain consistent?
-
How would STAR compare with GIST, LESS, or a simple fusion of a target specific selector and a task balanced selector when all methods receive the same five reference sets? Such a comparison would help determine whether the gains require the proposed neuron decomposition or primarily result from additional non target supervision.
I would be happy to raise my score if the authors can convincingly address these concerns, particularly through comparisons under matched reference supervision.
9 comments
W2/Q3 — Five-Reference and Pipeline-Matched Baselines
2. Comparisons under Matched Reference Supervision (W2 / Q3)
We address the reviewer's proposed alternatives through native and pipeline-matched five-reference baselines (R1/R1b), simple target-plus-balanced score fusion (R18), and a fixed-cluster Unified Union ablation (R2).
(1) Five-reference baselines, including GIST and LESS.
LESS, MONA, NAIT, and GIST receive the same five reference sets as STAR. In the pipeline-matched versions, target/non-target scores are fused with after percentile-rank normalization. All use MiniBatchKMeans (), the same cluster-wise quotas and remainder filling, and the same 5% data budget and SFT configuration. Clustering uses each method's own feature space. Configurations are fixed, with no additional hyperparameter search; the protocol is detailed in Comment 5.
Table R1. Five-reference, pipeline-matched comparisons (Llama-3.1-8B, 5%).
| Method | MMLU | MMLU-Pro | TyDiQA | XCOPA | BBH | GSM8K | MATH | MBPP | H-Eval | Target Avg. | Non-Target Avg. | Aligned OOD | Cross-Target OOD |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LESS-matched | 59.66 | 35.26 | 72.25 | 62.58 | 62.22 | 56.56 | 18.10 | 44.40 | 40.24 | 59.02 | 58.38 | 39.05 | 38.19 |
| MONA-matched | 64.32 | 36.21 | 70.30 | 62.36 | 62.31 | 54.74 | 18.98 | 46.80 | 43.29 | 59.69 | 58.18 | 40.21 | 38.80 |
| NAIT-matched | 64.24 | 36.93 | 70.52 | 62.62 | 64.54 | 55.80 | 17.50 | 48.20 | 43.90 | 60.66 | 59.12 | 40.24 | 39.18 |
| GIST-matched | 63.51 | 36.05 | 69.49 | 62.78 | 63.80 | 55.19 | 17.72 | 46.80 | 44.51 | 59.76 | 59.40 | 40.27 | 38.92 |
| STAR | 64.81 | 37.28 | 72.01 | 63.25 | 65.65 | 62.93 | 19.94 | 48.60 | 43.90 | 62.80 | 60.21 | 41.09 | 40.08 |
We also retain the native five-reference implementations from Appendix B.2, which preserve each baseline's selection procedure. Table R1b places their results alongside the matched versions; native results are aggregated from Table 9 of the original paper.
Table R1b. Native five-reference and pipeline-matched implementations (Llama-3.1-8B, 5%).
| Method | Implementation | Target Avg. | Non-Target Avg. | Aligned OOD | Cross-Target OOD |
|---|---|---|---|---|---|
| MONA-5ref | Native | 61.90 | 59.52 | 39.92 | 38.97 |
| MONA-matched | Pipeline-matched | 59.69 | 58.18 | 40.21 | 38.80 |
| NAIT-5ref | Native | 61.49 | 60.00 | 40.34 | 39.42 |
| NAIT-matched | Pipeline-matched | 60.66 | 59.12 | 40.24 | 39.18 |
| STAR | Shared/Specific subspaces | 62.80 | 60.21 | 41.09 | 40.08 |
Against the strongest native five-reference result in each column, STAR improves Target Avg. by +0.90, Non-Target Avg. by +0.21, Aligned OOD by +0.75, and Cross-Target OOD by +0.66. In particular, it improves over NAIT-5ref by +1.31 Target / +0.21 Non-Target, and over MONA-5ref by +0.90 Target / +0.69 Non-Target. Thus, the target gains coexist with slightly higher known-task utility and higher related OOD transfer under the same reference supervision.
Relative to the strongest pipeline-matched result in each column, the gains are +2.14 / +0.81 / +0.82 / +0.90, respectively. The two comparisons serve complementary purposes: native implementations preserve each baseline's selection procedure, while matched variants control common score processing and cluster-based selection. The following controls examine simple fusion and scoring-space structure directly.
W2/Q3 — Simple NAIT + BIDS Score-Fusion Control
(2) Native target-specific + task-balanced score fusion (continued).
We construct NAIT + BIDS using native target-only NAIT scores and native BIDS scores computed from all five reference sets. The fused selector therefore has access to the same five reference sets as STAR. Let denote the percentile rank over the candidate pool. We use the fixed equal-weight fusion
The control selects the top- candidates by the fused score, where is the 5% selection budget. It uses score mixing only. Table R18 includes the native component results for comparison.
Table R18. Simple score fusion vs. STAR (Llama-3.1-8B, 5%).
| Method | MMLU | MMLU-Pro | TyDiQA | XCOPA | BBH | GSM8K | MATH | MBPP | H-Eval | Target Avg. | Non-Target Avg. | Aligned OOD | Cross-Target OOD |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NAIT (component) | 64.24 | 37.30 | 71.45 | 62.35 | 64.17 | 55.50 | 18.76 | 48.20 | 40.24 | 60.71 | 59.15 | 39.66 | 39.06 |
| BIDS (component) | 63.90 | 35.46 | 69.85 | 62.53 | 63.24 | 56.79 | 17.70 | 45.80 | 45.73 | 59.92 | 59.92 | 40.36 | 40.36 |
| NAIT + BIDS | 62.85 | 35.86 | 68.79 | 62.49 | 64.26 | 62.17 | 16.26 | 47.20 | 40.85 | 61.05 | 59.88 | 38.87 | 39.76 |
| STAR | 64.81 | 37.28 | 72.01 | 63.25 | 65.65 | 62.93 | 19.94 | 48.60 | 43.90 | 62.80 | 60.21 | 41.09 | 40.08 |
The fused selector achieves 61.05 Target Avg., 59.88 Non-Target Avg., and 39.76 Cross-Target OOD. STAR exceeds it on all nine individual benchmark columns and all four aggregates: +1.75 Target, +0.33 Non-Target, +2.22 Aligned OOD, and +0.32 Cross-Target OOD.
This directly answers the proposed simple-combination control: the evaluated equal-weight fusion does not reproduce STAR's joint outcome. The next comparison holds STAR's neuron coordinates and cluster assignments fixed to test the contribution of separate scoring subspaces.
W1 — Known Non-Target Utility and Held-Out OOD Evidence
We thank the reviewer for recognizing the practical asymmetric objective and STAR's simple Shared/Specific design. We have added five-reference comparisons including GIST and LESS, a native NAIT+BIDS score-fusion control, a fixed-cluster subspace ablation, and three-seed SFT results. These additions directly address matched supervision and the source of the gains; we also clarify the scope of cross-task preservation.
1. Known Non-Target Utility and Held-Out OOD Transfer (W1)
Non-Target Avg. measures utility on the four other known reference tasks, whose small reference sets participate in selection. The OOD benchmarks (MMLU-Pro, XCOPA, MATH, HumanEval) are excluded from selection and assess transfer to held-out benchmarks related to known capability families. They do not establish preservation of arbitrary unseen skills.
Aligned OOD. STAR has the highest Aligned OOD among the methods evaluated in each of the seven reported model-budget settings: Llama-3.1-8B at 5%/1%, Qwen2.5-7B at 5%/1%, Qwen3.5-9B-Base at 5%, Qwen1.5-MoE at 5%, and OLMoE at 5%. The five dense-model comparisons include Random and BIDS; the two MoE comparisons include Random and NAIT-5ref, which receives the same five reference sets as STAR. These results support transfer of target-task improvements to paired, held-out benchmarks.
Cross-Target OOD. This metric averages all four OOD benchmarks over all five target settings, including the aligned settings. Table R17 reports this broader aggregate consistently with the original Table 3.
Table R17. Cross-Target OOD across seven settings. The baseline column compares the evaluated LESS, MONA, NAIT, and GIST variants, including native five-reference versions where available. Both MoE comparisons use NAIT-5ref. The NAIT+BIDS fusion control is reported separately in R18.
STAR exceeds the strongest targeted baseline listed in each setting. It exceeds Random in four of seven settings, with gaps of 0.05-0.10 points in the other three. Against BIDS, STAR is 0.17-0.70 points lower in all five settings with BIDS results. For Llama-3.1-8B at 5%, STAR obtains 40.08, versus 40.17 for Random and 40.36 for BIDS; STAR's three-seed mean is 40.10 ± 0.48, while those Random and BIDS results are single-run.
We will qualify the claim as improved target specialization with stronger average utility on known reference tasks and competitive transfer to related held-out OOD benchmarks.