acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Targeted Data Selection for Instruction Tuning: Balancing Target Specialization and Cross-Task Utility

Abstract

Instruction data selection aims to identify a compact subset of training examples that improves downstream performance under a fixed data budget. Existing target-specific methods primarily optimize relevance to a designated target task, but improvements on the target may coincide with cross-task degradation, while task-balanced methods typically treat multiple tasks symmetrically without explicitly prioritizing the target. We therefore rethink targeted data selection for instruction tuning as an asymmetric objective: improving target specialization while preserving cross-task utility. To address this objective, we propose STAR (Task-Shared and Target-Specific Activation-based Ranking), which uses cross-task FFN activation statistics from a frozen LLM to construct two complementary neuron sets. Task-Shared neurons exhibit consistently high and relatively balanced activation across reference tasks, while Target-Specific neurons exhibit stronger activation for the designated target than for the remaining reference tasks. These neuron sets define separate activation subspaces, where STAR measures candidate alignment with all reference-task prototypes and the target prototype, respectively, and combines the two scores for selection. We evaluate STAR across five target tasks and four OOD benchmarks under selection budgets of 5% and 1%. Under the 5% budget, STAR achieves the highest Target Avg. of 62.80 and Non-Target Avg. of 60.21, along with the best aligned OOD performance and cross-target OOD transfer among target-specific methods. STAR remains effective under the 1% budget and across backbone models. These results demonstrate that STAR improves target specialization while preserving cross-task utility under limited budgets. The code is available at https://anonymous.4open.science/r/STAR-42E7/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Instruction Tuning, Data Selection, Neuron Activations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cRsBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies targeted instruction data selection that improves a designated task while preserving performance on other tasks. It proposes STAR, which uses FFN activations from a frozen language model to identify Task-Shared and Target-Specific neurons, constructs separate activation subspaces, and combines prototype-alignment scores for cluster-aware subset selection. Experiments across five target tasks, four OOD benchmarks, two selection budgets, and two backbone models show improved target and known non-target performance, together with competitive OOD results.

Critical strengths
  1. Practically meaningful problem formulation. The paper identifies an important limitation of existing targeted selection methods: improving a designated task may reduce performance on other tasks. Framing selection as an asymmetric balance between target specialization and cross task utility is well motivated, and STAR provides a simple mechanism for representing these two objectives through separate activation subspaces (Section 3).

  2. Broad empirical evaluation. STAR is evaluated across five target tasks, four OOD benchmarks, two selection budgets, and two backbone models, with additional ablations on neuron selection, score fusion, and clustering. The results provide promising evidence that the proposed design improves target specialization while retaining utility across the known reference tasks (Sections 4 and 5).

Critical weaknesses
  1. The evidence for broad cross task utility is limited. STAR directly uses all five reference tasks to construct its Task Shared signal, so the reported Non Target Avg. measures performance on tasks that already participate in selection rather than utility on genuinely unseen tasks. On the fully held out Cross Target OOD evaluation, STAR scores 40.08, slightly below Random at 40.17 and BIDS at 40.36; on Qwen2.5, it also underperforms BIDS on this metric. The current results therefore support improved balancing among known reference tasks more strongly than the broader claim of preserving cross task utility.

  2. The source of the improvement is not sufficiently isolated through matched baselines. STAR differs from the main target specific baselines in several respects: it uses all five reference sets, separates two activation subspaces, combines target and shared scores, and applies cluster based selection. Although the appendix extends MONA and NAIT to multiple reference tasks, the paper does not compare against other strong targeted methods such as GIST under the same information setting, or against a simple combination of a target specific selector and a task balanced selector. Consequently, it remains unclear whether the gains require the proposed neuron decomposition or can be obtained through simpler score fusion with the same reference supervision.

Questions for authors
  1. How were ww, β\beta, the numbers of selected neurons, and the number of clusters chosen? Were any of these hyperparameters selected using performance on the benchmarks reported in the main results?

  2. Are the reported results based on a single fine tuning run? The standard deviations in Table 3 appear to measure variation across target settings rather than variation across random seeds. If repeated runs are available, do the main improvements remain consistent?

  3. How would STAR compare with GIST, LESS, or a simple fusion of a target specific selector and a task balanced selector when all methods receive the same five reference sets? Such a comparison would help determine whether the gains require the proposed neuron decomposition or primarily result from additional non target supervision.

I would be happy to raise my score if the authors can convincingly address these concerns, particularly through comparisons under matched reference supervision.

9 comments
AuthorsOpenReview ↗

W1 — Known Non-Target Utility and Held-Out OOD Evidence

We thank the reviewer for recognizing the practical asymmetric objective and STAR's simple Shared/Specific design. We have added five-reference comparisons including GIST and LESS, a native NAIT+BIDS score-fusion control, a fixed-cluster subspace ablation, and three-seed SFT results. These additions directly address matched supervision and the source of the gains; we also clarify the scope of cross-task preservation.

1. Known Non-Target Utility and Held-Out OOD Transfer (W1)

Non-Target Avg. measures utility on the four other known reference tasks, whose small reference sets participate in selection. The OOD benchmarks (MMLU-Pro, XCOPA, MATH, HumanEval) are excluded from selection and assess transfer to held-out benchmarks related to known capability families. They do not establish preservation of arbitrary unseen skills.

Aligned OOD. STAR has the highest Aligned OOD among the methods evaluated in each of the seven reported model-budget settings: Llama-3.1-8B at 5%/1%, Qwen2.5-7B at 5%/1%, Qwen3.5-9B-Base at 5%, Qwen1.5-MoE at 5%, and OLMoE at 5%. The five dense-model comparisons include Random and BIDS; the two MoE comparisons include Random and NAIT-5ref, which receives the same five reference sets as STAR. These results support transfer of target-task improvements to paired, held-out benchmarks.

Cross-Target OOD. This metric averages all four OOD benchmarks over all five target settings, including the aligned settings. Table R17 reports this broader aggregate consistently with the original Table 3.

Table R17. Cross-Target OOD across seven settings. The baseline column compares the evaluated LESS, MONA, NAIT, and GIST variants, including native five-reference versions where available. Both MoE comparisons use NAIT-5ref. The NAIT+BIDS fusion control is reported separately in R18.

Model / budgetRandomBIDSBest targeted baselineSTAR
Llama-3.1-8B, 5%40.1740.3639.42 (NAIT-5ref)40.08 (3 seeds: 40.10 ± 0.48)
Llama-3.1-8B, 1%40.6540.8239.0040.60
Qwen2.5-7B, 5%48.7850.0148.7149.31
Qwen2.5-7B, 1%49.0249.1447.4548.92
Qwen3.5-9B-Base, 5%56.4957.3556.3657.18
Qwen1.5-MoE-A2.7B, 5%34.37—32.91 (NAIT-5ref)35.27
OLMoE-1B-7B, 5%24.57—23.91 (NAIT-5ref)24.63

STAR exceeds the strongest targeted baseline listed in each setting. It exceeds Random in four of seven settings, with gaps of 0.05-0.10 points in the other three. Against BIDS, STAR is 0.17-0.70 points lower in all five settings with BIDS results. For Llama-3.1-8B at 5%, STAR obtains 40.08, versus 40.17 for Random and 40.36 for BIDS; STAR's three-seed mean is 40.10 ± 0.48, while those Random and BIDS results are single-run.

We will qualify the claim as improved target specialization with stronger average utility on known reference tasks and competitive transfer to related held-out OOD benchmarks.

AuthorsOpenReview ↗

W2/Q3 — Five-Reference and Pipeline-Matched Baselines

2. Comparisons under Matched Reference Supervision (W2 / Q3)

We address the reviewer's proposed alternatives through native and pipeline-matched five-reference baselines (R1/R1b), simple target-plus-balanced score fusion (R18), and a fixed-cluster Unified Union ablation (R2).

(1) Five-reference baselines, including GIST and LESS.

LESS, MONA, NAIT, and GIST receive the same five reference sets as STAR. In the pipeline-matched versions, target/non-target scores are fused with λ=0.5\lambda=0.5 after percentile-rank normalization. All use MiniBatchKMeans (C=100C=100), the same cluster-wise quotas and remainder filling, and the same 5% data budget and SFT configuration. Clustering uses each method's own feature space. Configurations are fixed, with no additional hyperparameter search; the protocol is detailed in Comment 5.

Table R1. Five-reference, pipeline-matched comparisons (Llama-3.1-8B, 5%).

MethodMMLUMMLU-ProTyDiQAXCOPABBHGSM8KMATHMBPPH-EvalTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
LESS-matched59.6635.2672.2562.5862.2256.5618.1044.4040.2459.0258.3839.0538.19
MONA-matched64.3236.2170.3062.3662.3154.7418.9846.8043.2959.6958.1840.2138.80
NAIT-matched64.2436.9370.5262.6264.5455.8017.5048.2043.9060.6659.1240.2439.18
GIST-matched63.5136.0569.4962.7863.8055.1917.7246.8044.5159.7659.4040.2738.92
STAR64.8137.2872.0163.2565.6562.9319.9448.6043.9062.8060.2141.0940.08

We also retain the native five-reference implementations from Appendix B.2, which preserve each baseline's selection procedure. Table R1b places their results alongside the matched versions; native results are aggregated from Table 9 of the original paper.

Table R1b. Native five-reference and pipeline-matched implementations (Llama-3.1-8B, 5%).

MethodImplementationTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
MONA-5refNative61.9059.5239.9238.97
MONA-matchedPipeline-matched59.6958.1840.2138.80
NAIT-5refNative61.4960.0040.3439.42
NAIT-matchedPipeline-matched60.6659.1240.2439.18
STARShared/Specific subspaces62.8060.2141.0940.08

Against the strongest native five-reference result in each column, STAR improves Target Avg. by +0.90, Non-Target Avg. by +0.21, Aligned OOD by +0.75, and Cross-Target OOD by +0.66. In particular, it improves over NAIT-5ref by +1.31 Target / +0.21 Non-Target, and over MONA-5ref by +0.90 Target / +0.69 Non-Target. Thus, the target gains coexist with slightly higher known-task utility and higher related OOD transfer under the same reference supervision.

Relative to the strongest pipeline-matched result in each column, the gains are +2.14 / +0.81 / +0.82 / +0.90, respectively. The two comparisons serve complementary purposes: native implementations preserve each baseline's selection procedure, while matched variants control common score processing and cluster-based selection. The following controls examine simple fusion and scoring-space structure directly.

AuthorsOpenReview ↗

W2/Q3 — Simple NAIT + BIDS Score-Fusion Control

(2) Native target-specific + task-balanced score fusion (continued).

We construct NAIT + BIDS using native target-only NAIT scores and native BIDS scores computed from all five reference sets. The fused selector therefore has access to the same five reference sets as STAR. Let PR⁡\operatorname{PR} denote the percentile rank over the candidate pool. We use the fixed equal-weight fusion

Sfusion(x,t)=0.5PR(StNAIT(x))+0.5PR(S1:5BIDS(x))S_{fusion}(x,t) = 0.5 PR(S_t^{NAIT}(x)) + 0.5 PR(S_{1:5}^{BIDS}(x))

The control selects the top-BB candidates by the fused score, where BB is the 5% selection budget. It uses score mixing only. Table R18 includes the native component results for comparison.

Table R18. Simple score fusion vs. STAR (Llama-3.1-8B, 5%).

MethodMMLUMMLU-ProTyDiQAXCOPABBHGSM8KMATHMBPPH-EvalTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
NAIT (component)64.2437.3071.4562.3564.1755.5018.7648.2040.2460.7159.1539.6639.06
BIDS (component)63.9035.4669.8562.5363.2456.7917.7045.8045.7359.9259.9240.3640.36
NAIT + BIDS62.8535.8668.7962.4964.2662.1716.2647.2040.8561.0559.8838.8739.76
STAR64.8137.2872.0163.2565.6562.9319.9448.6043.9062.8060.2141.0940.08

The fused selector achieves 61.05 Target Avg., 59.88 Non-Target Avg., and 39.76 Cross-Target OOD. STAR exceeds it on all nine individual benchmark columns and all four aggregates: +1.75 Target, +0.33 Non-Target, +2.22 Aligned OOD, and +0.32 Cross-Target OOD.

This directly answers the proposed simple-combination control: the evaluated equal-weight fusion does not reproduce STAR's joint outcome. The next comparison holds STAR's neuron coordinates and cluster assignments fixed to test the contribution of separate scoring subspaces.

Reviewer_LzLaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a targeted data selection algorithm, called STAR, for LLM instruction tuning. STAR addresses the issue of cross-task degradation, which exists in previous target-specific selection algorithms. STAR extracts FFN neuron activations from a frozen LLM and applies z-score normalization to build cross-task activation profiles. It then identifies task-shared and target-specific neuron subsets, and selects instruction tuning data by scoring candidate examples and using cluster-aware sampling. STAR demonstrates the ability to improve target-task performance and maintain cross-task utility.

Critical strengths

This paper has valid motivations. It observes that selecting data for specific tasks could compromise an LLM’s capabilities on other tasks. The activation normalization method has a solid mathematical grounding. It utilizes the ℓ2\ell_2-norm of the down-projection columns to accurately reflect the actual vector magnitude written into the residual stream. The z-score normalization effectively removes background noise. The method proposed in this paper is less computationally intensive compared with previous work, like LESS, making it more applicable for large-scale data selection.

Critical weaknesses

Unfair Baseline Comparison:

In Tables 1–3, baselines (LESS, MONA, NAIT) only access the reference set of a single target task, whereas STAR uses the reference sets of all 5 tasks (1 Target + 4 Non-Target) to build its profiles. Evaluating Non-Target Avg. on those exact 4 non-target tasks gives STAR an unfair advantage via prior information leakage.

Limited Model Diversity & MoE Applicability:

Evaluation is restricted to dense models (Llama-3.1-8B), leaving its generalizability to modern Mixture-of-Experts (MoE) architectures (e.g., Qwen-MoE, DeepSeek-MoE) unclear. Because MoE uses sparse Top-kk routing, most experts remain inactive (zero activation) per token, creating zero-inflated distributions that can break STAR’s z-score normalization and quantile scoring.

Questionable Underlying Assumptions:

  • Polysemantic Neurons: Individual FFN neurons are highly polysemantic, compressing multiple unrelated concepts. A hard Top-KK cutoff on raw neurons merely acts as a surface-level lexical/domain filter (e.g., matching code keywords or math symbols) rather than isolating true reasoning quality from noisy data.
  • Activation ≠\neq Gradient Utility: High activation on a frozen base model simply means the model already represents that pattern well; it does not reflect gradient utility for fixing model weaknesses during SFT.
Questions for authors
  1. Fair Baseline Comparison: Since STAR uses all 5 reference sets during selection, how do the baselines (LESS, MONA, NAIT) perform across all main tables if given access to the same 5 reference sets?

  2. Applicability to MoE Models: Can STAR be applied to Mixture-of-Experts (MoE) models (e.g., Qwen-MoE, Mixtral)? How does it handle zero-inflated activations from sparse Top-kk routing during z-score normalization?

  3. Polysemantic Neurons: Since raw FFN neurons are polysemantic, how does STAR ensure Top-KK selection captures true reasoning quality rather than just matching surface-level tokens (e.g., code keywords or math symbols)?

  4. Activation vs. Gradient Utility: High activation on a frozen model means it already recognizes that pattern well. Why is selecting high-activation samples better for SFT than targeting patterns where the base model is weak?

9 comments
AuthorsOpenReview ↗

W1/Q1 — Fair Five-Reference Baseline Comparison

We thank the reviewer for recognizing the practical motivation, normalized FFN contribution measure, and efficiency of gradient-free selection. Your questions motivated five-reference comparisons, two MoE evaluations, and a lexical-scoring control, which we present below.

1. Fair Reference Supervision (W1 / Q1)

In the Appendix section “Comparison with Multi-Reference Baselines” of our original submission, we have already provided five-reference comparisons with MONA and NAIT, addressing the concern of unequal reference supervision. To further address the reviewer's concerns about information leakage, we further introduce pipeline-matched comparisons with LESS, MONA, NAIT, and GIST (Min et al., ICML 2026), all using the same five reference sets and clustering procedure as STAR. The matched comparison uses:

  • Scoring: target/non-target fusion with fixed λ=0.5\lambda=0.5 (Appendix B.2), with percentile-rank normalization before fusion.
  • Selection: MiniBatchKMeans (C=100C=100), the same cluster quotas and remainder filling, with each method clustering in its own feature space.
  • Training: the same candidate pool, 5% data budget, and SFT setup.

Table R1. Five-reference, pipeline-matched comparison (Llama-3.1-8B, 5%).

MethodMMLUMMLU-ProTyDiQAXCOPABBHGSM8KMATHMBPPH-EvalTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
LESS-matched59.6635.2672.2562.5862.2256.5618.1044.4040.2459.0258.3839.0538.19
MONA-matched64.3236.2170.3062.3662.3154.7418.9846.8043.2959.6958.1840.2138.80
NAIT-matched64.2436.9370.5262.6264.5455.8017.5048.2043.9060.6659.1240.2439.18
GIST-matched63.5136.0569.4962.7863.8055.1917.7246.8044.5159.7659.4040.2738.92
STAR64.8137.2872.0163.2565.6562.9319.9448.6043.9062.8060.2141.0940.08

We report the native five-reference implementations alongside their matched variants to make both comparisons explicit.

Table R1b. Native five-reference and pipeline-matched implementations (Llama-3.1-8B, 5%). Native results are aggregated from Table 9 of the original paper.

MethodImplementationTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
MONA-5refNative61.9059.5239.9238.97
MONA-matchedPipeline-matched59.6958.1840.2138.80
NAIT-5refNative61.4960.0040.3439.42
NAIT-matchedPipeline-matched60.6659.1240.2439.18
STARShared/Specific subspaces62.8060.2141.0940.08

STAR leads on all four aggregates against native MONA/NAIT-5ref and the matched baselines. Against the strongest native method in each column, the gains are +0.90 Target, +0.21 Non-Target, +0.75 Aligned OOD, and +0.66 Cross-Target OOD. Against the strongest matched method, they are +2.14 / +0.81 / +0.82 / +0.90, respectively; STAR also leads on seven of the nine individual benchmark columns in R1. Native implementations preserve each baseline's selection procedure, while matched variants control common processing. Together, they show that the advantage persists under equal reference supervision.

Target-agnostic BIDS also uses all five references: STAR achieves 62.80 Target / 60.21 Non-Target, compared with 59.92 / 59.92 for BIDS.

The reference examples (at most 100 per task) come from training, validation, or official few-shot sources and are disjoint from evaluation test sets (Appendix Table 6). Non-Target Avg. measures utility on known reference tasks. The four related OOD benchmarks are excluded from selection, so the OOD gains additionally support transfer to held-out benchmarks within related capability families.

AuthorsOpenReview ↗

W2/Q2 — Routing-Aware MoE Features and Zero-Inflation Diagnostics

2. Extension to Sparse MoE Architectures (W2 / Q2)

The original paper evaluates Llama-3.1-8B and Qwen2.5-7B; additional dense-backbone results include Qwen3.5-9B-Base (R8 in our response to gnAX). To address sparse routing directly, we implement STAR on Qwen1.5-MoE-A2.7B (with a shared expert) and OLMoE-1B-7B-0125 (without shared experts).

Routing-aware neuron contribution. For expert neuron u=(l,e,j)u=(l,e,j), we use

1+\omega_{e,s}\cdot |a_{u,s}(x)| \cdot \lVert W_u^{\mathrm{dn}}\rVert_2 \right).$$ Here $\omega_{e,s}$ is the actual expert-output routing weight, including applicable normalization or scaling. It is zero for an unrouted expert; a shared expert uses its actual output gate. This incorporates the expert's contribution to the model output. Instrumentation preserves the forward pass (maximum observed logit difference: 0 across four ranks). Subsequent steps retain STAR's fixed defaults: 500 background examples, z-score normalization, Shared/Specific neuron selection with $K=4096$ per set, prototype cosine similarity, rank fusion with $w=0.5$, and $C=100$ clustering. **Table R12. Activation sparsity and coverage diagnostics on background and reference sets.** | Diagnostic | Qwen1.5-MoE-A2.7B | OLMoE-1B-7B-0125 | | ---------------------------------------------------------- | ----------------- | ---------------- | | Token-level zeros in routed-expert neurons | 93.33% | 87.50% | | Example-level zeros in routed-expert neurons | **1.97%** | **3.89%** | | Routed-expert coverage per reference task (layer × expert) | 1,440 / 1,440 | 1,024 / 1,024 | | Coordinates with zero background standard deviation (SD) | 0 | 0 | | Coordinates with background SD ≤ 1e-6 | 0.0061% | 0.0054% | STAR aggregates contributions over all tokens in each example. Writing $r_e(x)$ for the fraction of tokens routed to expert $e$ and $\bar c_u(x)$ for the mean contribution on those tokens gives $$c_u(x)=r_e(x)\cdot \bar c_u(x).$$ with $c_u(x)=0$ when no token routes to that expert. Token-level zero rates of **93.33% / 87.50%** therefore fall to **1.97% / 3.89%** at the example level. Every reference task covers all routed experts, no coordinate has zero background SD, and fewer than **0.01%** have SD at most $10^{-6}$. Z-score normalization and percentile ranks do not require Gaussian activations. The relevant concerns include degenerate variance and sparse observations; R12 directly checks these properties in the evaluated models. Example-level aggregation provides denser features for normalization and prototypes, while routing frequency contributes potentially useful task-selective information through $r_e(x)$. The downstream tests below assess whether these features support effective selection.
AuthorsOpenReview ↗

W2/Q2 — Downstream STAR Results on Two MoE Backbones

Downstream evaluation on the two MoE models (continued).

We evaluate Random, NAIT-5ref, and STAR under the same 5% budget. NAIT-5ref and STAR receive the same five reference sets, and all methods use fixed configurations without additional hyperparameter search. The nine benchmark columns and four aggregate metrics follow the main paper.

Table R13. Results on MoE backbones (5% budget).

ModelMethodMMLUMMLU-ProTyDiQAXCOPABBHGSM8KMATHMBPPH-EvalTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
Qwen1.5-MoERandom58.9028.4156.9458.7139.0747.9910.1035.8040.2447.7447.7434.3734.37
Qwen1.5-MoENAIT-5ref60.0428.2357.2258.0939.4456.2514.2036.2039.6349.8346.9735.0432.91
Qwen1.5-MoESTAR61.1629.4058.3559.4940.7455.5016.3636.8041.4650.5148.2136.6835.27
OLMoERandom54.1619.8648.3254.4931.6743.3714.7819.209.1539.3439.3424.5724.57
OLMoENAIT-5ref54.9018.7049.4054.7533.0644.6615.1020.009.7640.4037.9724.5823.91
OLMoESTAR55.2320.0149.0855.9835.1946.1716.2420.6010.9841.2539.9225.8024.63

Relative to NAIT-5ref, STAR improves Target Avg. by 0.68 / 0.85, Non-Target Avg. by 1.24 / 1.95, Aligned OOD by 1.64 / 1.22, and Cross-Target OOD by 2.36 / 0.72 points on Qwen1.5-MoE / OLMoE, respectively.

STAR also improves Target Avg. over Random on both models. Its Cross-Target OOD is 35.27 vs. 34.37 on Qwen1.5-MoE and remains close to Random on OLMoE (24.63 vs. 24.57). Together with the routing diagnostics, these results demonstrate effective data selection on both shared-expert and non-shared-expert architectures, including gains over a targeted baseline with equal reference supervision.

Reviewer_gnAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper frames targeted instruction data selection as an asymmetric objective: improving a designated target task while preserving performance on the other reference tasks. The proposed method, STAR, uses FFN activations from a frozen LLM to identify Task-Shared and Target-Specific neuron sets. It scores candidate examples by their similarity to task prototypes in the two subspaces, combines the scores through percentile-rank fusion, and selects a subset using cluster-aware selection.

Critical strengths

The paper clearly motivates prioritizing a designated target while accounting for performance on the remaining tasks.

STAR extracts selection signals from a frozen model without gradient computation or additional LLM training. Background normalization, separate Shared/Specific representations, and rank-based fusion have clear motivations, and the selected neuron sets can be directly inspected.

The evaluation covers five target tasks, four OOD benchmarks, two selection budgets, and a second backbone. Detailed task-level results for the main setting help readers assess both the benefits and costs of specialization.

Critical weaknesses

W1. The contribution of the Shared/Specific decomposition is not fully isolated.

STAR differs from the baselines in its representations, score processing, and clustering. Without clustering, its Non-Target Avg. drops to 59.39 (Table 11), below NAIT-5ref (60.00) and MONA-5ref (59.52), although its Target Avg. remains higher. This suggests that clustering contributes substantially to the final cross-task advantage.

Table 5 supports the usefulness of the selected neurons, so these observations do not imply that all gains come from clustering. However, changing the neuron sets changes both scoring and clustering features. The comparison therefore does not directly distinguish selecting informative neurons from assigning them to separate Shared/Specific subspaces. Controls using the same selected neurons and fixed cluster assignments, alongside matched normalization and clustering for the multi-reference baselines, would clarify the decomposition’s contribution.

W2. The endpoint ablations depend on an unclear clustering implementation.

Figure 3 provides useful evidence of a target/non-target trade-off, but the clustering implementation is unclear. The main text defines a concatenation of the Shared and Specific representations, while Appendix A.2 mentions a “weighted” concatenation without specifying the weights.

If ww changes only scoring, clustering may still use both representations at w=0w=0 and w=1w=1, so the removed branch can still affect selection. These would not be clean single-signal ablations. If ww also changes the clustering representation, the sweep changes both scoring and clustering. This distinction needs clarification to interpret each branch’s contribution.

W3. The presentation of cross-task preservation needs qualification.

Table 8 generally supports improved cross-task utility: STAR beats Random in 17 of 20 non-target comparisons. However, GSM8K drops by 2.73 points when targeting MMLU, and TyDiQA drops by 2.35 points when targeting GSM8K. The benefit is therefore average rather than uniform.

Figure 1 also presents a favorable subset of results. For the MMLU-target setting, its three displayed tasks give average gains of +1.49 for STAR and +0.80 for BIDS. Including all five tasks, using Tables 1 and 8, changes these to approximately +0.43 and +0.70. Showing all five tasks would better represent the trade-off in this setting.

W4. Evaluation across budgets and model generations is limited.

At the 1% budget, Non-Target Avg. and Cross-Target OOD Avg. are missing. The central claim of improving specialization while preserving cross-task utility is therefore not fully tested under the tighter budget.

The experiments are also limited to two older model families, Llama-3.1 and Qwen2.5. The Qwen2.5-7B comparison covers only 5%, without Random, MONA, or task-level results. Since STAR relies on model-specific FFN activations, testing a more recent backbone, such as Qwen3.5 model, would help establish whether its selection signals remain effective across model generations.

W5. Data selection time is not reported.

STAR requires activation extraction over approximately 270K candidates, followed by scoring and clustering. The paper does not report how long this selection process takes or compare it with the baselines, leaving its practical time overhead unclear.

Questions for authors
  1. (W1) Can MONA-5ref and NAIT-5ref be compared using the same score normalization, clustering procedure, and tuning budget as STAR?

  2. (W2) What weights are used for the clustering representation, and do they depend on ww? Can the endpoint ablations remove the unused branch from both scoring and clustering, with OOD results also reported across ww?

  3. (W4) Can the authors provide the missing 1% cross-task results and complete the Qwen2.5-7B baseline comparisons? Can they also evaluate STAR on a recent backbone, such as Qwen3.5-9B-Base, using the same tasks and strong baselines?

  4. (W1) Can the authors compare the separate subspaces against a common representation built from the union of the selected Shared and Specific neurons, while keeping the original cluster assignments fixed? This would help distinguish the benefit of selecting informative neurons from the benefit of assigning them to different scoring subspaces.

12 comments
AuthorsOpenReview ↗

W1/Q1 — Native and Matched Five-Reference Comparisons

We thank the reviewer for recognizing the practical objective, gradient-free framework, and experimental coverage. To address the central attribution concern, we provide native and pipeline-matched five-reference comparisons, together with a fixed-cluster Unified Union control (discussed in our subsequent comments). We also clarify the weight sweep and provide additional ablations, backbone evaluations, and cost measurements.

1. Matched Reference Supervision and Selection Procedures (W1 / Q1)

We extend LESS, MONA, NAIT, and GIST (Min et al., ICML 2026) to use the same five reference sets as STAR. GIST is a recent gradient-based selection method that uses SVD-derived task-specific subspaces to align training gradients with target directions, providing an additional strong comparator.

The pipeline-matched implementations use:

  • Reference-score fusion: Target and non-target scores are combined with the fixed λ=0.5\lambda=0.5 in Appendix B.2.
  • Score processing and clustering: Percentile-rank normalization, MiniBatchKMeans with C=100C=100, identical cluster quotas, and the same remainder-filling procedure. Each method clusters its own representations.
  • Training: The same candidate pool, selection budget, and SFT configuration.
  • Hyperparameters: No method receives an additional hyperparameter search. Baseline-specific defaults follow the original publications/released implementations. STAR uses its fixed defaults from Section 4.1 and Appendix A.2: w=β=0.5w=\beta=0.5, Ksh=Ksp=4096K^{sh}=K^{sp}=4096, and C=100C=100. The sensitivity studies diagnose these choices and do not select the main configuration.

Table R1. Matched comparisons on Llama-3.1-8B (5% budget).

MethodMMLUMMLU-ProTyDiQAXCOPABBHGSM8KMATHMBPPH-EvalTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
LESS-matched59.6635.2672.2562.5862.2256.5618.1044.4040.2459.0258.3839.0538.19
MONA-matched64.3236.2170.3062.3662.3154.7418.9846.8043.2959.6958.1840.2138.80
NAIT-matched64.2436.9370.5262.6264.5455.8017.5048.2043.9060.6659.1240.2439.18
GIST-matched63.5136.0569.4962.7863.8055.1917.7246.8044.5159.7659.4040.2738.92
STAR64.8137.2872.0163.2565.6562.9319.9448.6043.9062.8060.2141.0940.08

STAR achieves the highest values on all four aggregate metrics. Its Target Avg. of 62.80 exceeds the strongest matched baseline, NAIT-matched (60.66), by 2.14 points, while improving Non-Target Avg. by 0.81 points over GIST-matched (59.40).

To assess whether these improvements depend on the common processing pipeline, we additionally compare against native five-reference implementations in the next comment.

Reference: Min et al. (2026), GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry, ICML 2026. https://proceedings.mlr.press/v306/min26a.html

AuthorsOpenReview ↗

W1/Q1 — Native and Matched Five-Reference Comparisons

Native five-reference comparisons (continued).

To distinguish the effect of equal reference supervision from that of the matched processing pipeline, we also evaluate native MONA-5ref and NAIT-5ref, preserving their original selection procedures. These results are aggregated from Table 9 of the original submission.

Table R1b. Native five-reference vs. pipeline-matched implementations (Llama-3.1-8B, 5%).

MethodImplementationTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
MONA-5refNative61.9059.5239.9238.97
MONA-matchedPipeline-matched59.6958.1840.2138.80
NAIT-5refNative61.4960.0040.3439.42
NAIT-matchedPipeline-matched60.6659.1240.2439.18
STARShared/Specific subspaces62.8060.2141.0940.08

Against the strongest native five-reference baseline in each column, STAR improves Target Avg. by 0.90, Non-Target Avg. by 0.21, Aligned OOD by 0.75, and Cross-Target OOD by 0.66.

Importantly, the matched pipeline does not uniformly improve baseline performance. The native comparisons preserve baseline-specific selection procedures, while the matched variants control score processing and clustering. Together, these comparisons show that STAR's advantage persists under equal reference supervision. The fixed-cluster experiment in our next comment further isolates the contribution of separate scoring subspaces.

AuthorsOpenReview ↗

W1/Q4 — Fixed-Cluster Test of Shared/Specific Subspaces

2. Isolating the Shared/Specific Scoring Subspaces (W1 / Q4)

The reviewer correctly identifies clustering as a useful component. In the original Table 11, removing it decreases Target Avg. from 62.80 to 62.67 (−0.13), Non-Target Avg. from 60.21 to 59.39 (−0.82), and Cross-Target OOD from 40.08 to 39.87 (−0.21).

To determine whether separate scoring subspaces contribute with clustering held fixed, we conduct the requested Unified Union experiment. It pools STAR's selected Task-Shared and Target-Specific neuron coordinates into one representation, then computes both scores using the corresponding task prototypes in that common space.

Unified Union and STAR preserve:

  • The same selected neuron coordinates;
  • The original STAR cluster assignments and selection quotas;
  • The scoring formulations, percentile-rank fusion, and hyperparameters (w=0.5w=0.5, β=0.5\beta=0.5).

The controlled change is whether the Shared and Specific scores use separate subspaces or the common neuron union.

Table R2. Separate vs. unified scoring subspaces with fixed neurons and cluster assignments (Llama-3.1-8B, 5% budget).

MethodMMLUMMLU-ProTyDiQAXCOPABBHGSM8KMATHMBPPH-EvalTarget Avg.Non-Target Avg.Aligned OODCross-Target OOD
Unified Union64.3835.7269.7462.2063.5257.8517.1446.2040.2460.3458.7538.8338.48
STAR64.8137.2872.0163.2565.6562.9319.9448.6043.9062.8060.2141.0940.08

STAR exceeds Unified Union on all nine benchmark columns and all four aggregates: +2.46 Target, +1.46 Non-Target, +2.26 Aligned OOD, and +1.60 Cross-Target OOD.

This result directly supports the benefit of separate Shared/Specific scoring spaces with neurons and clustering held fixed. A plausible explanation is that the common representation introduces dimensions less relevant to each signal, reducing the discrimination of its prototype similarities. Together with the original clustering ablation, these results establish that both the scoring-space structure and cluster-aware selection contribute to the final outcome.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

Dear Area Chairs and Reviewers,

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We appreciate their recognition of STAR's practical motivation, Shared/Specific design, gradient-free efficiency, and empirical results. Their comments have helped us strengthen the experimental evidence and clarify the scope of our contributions.

We have conducted additional experiments and analyses to address the main concerns, summarized below.

1. Fair comparisons and contribution of Shared/Specific decomposition (gnAX, LzLa, cRsB). We provide five-reference comparisons with LESS, MONA, NAIT, and GIST, alongside native baselines, a simple NAIT+BIDS fusion, and a fixed-cluster Unified Union control. STAR maintains its advantage under equal reference supervision, while the ablation confirms the benefit of separate scoring subspaces.

2. Clustering and hyperparameter analysis (gnAX, cRsB). We clarify the clustering implementation and report sensitivity analyses for the fusion weight, neuron counts, and cluster counts.

3. Generalization and robustness (gnAX, LzLa, cRsB). We extend evaluations across selection budgets, three dense backbones, and two MoE architectures, and report three-seed SFT results. These experiments further support STAR's effectiveness across different settings.

4. Representation analysis and efficiency (gnAX, LzLa). We introduce a lexical-scoring control, analyze the connection between activation alignment and gradient utility, and report measured data-selection costs.

We hope that our responses and additional evidence have satisfactorily addressed the reviewers' concerns and clarified the contributions of our work.

Authors of Submission 12971

Thank you for your follow-up and for confirming that the additional five-reference comparisons address your concern regarding fair supervision. We would like to share a brief note to contextualize the native five-reference results:

  1. Head-to-head gains across all four metrics (rather than column-wise maxima): In Table R1b, the reported +0.90 / +0.21 / +0.75 / +0.66 margins compare STAR against the column-wise best across two different baselines. Evaluated head-to-head, STAR outperforms MONA-5ref across all four metrics by +0.90 Target / +0.69 Non-Target / +1.17 Aligned OOD / +1.11 Cross-Target OOD, and outperforms NAIT-5ref by +1.31 Target / +0.21 Non-Target / +0.75 Aligned OOD / +0.66 Cross-Target OOD. Notably, NAIT-5ref comes closest on Non-Target Avg., while showing noticeably lower target performance (e.g., -2.88 points behind STAR on GSM8K target, revised Table 13), whereas STAR achieves the highest target score in 5/5 target settings.

  2. Solid gains on Target Avg. and held-out OOD benchmarks: Equalizing five-reference supervision naturally narrows the gap on Non-Target Avg., which is evaluated on tasks whose reference sets are used during selection and aligns with our asymmetric problem formulation. At the same time, STAR retains clear gains on Target Avg. (+0.90 to +1.31) and on held-out OOD benchmarks excluded from selection (+0.66 to +1.17)—for context, the spread among the original target-specific baselines on this benchmark is ~0.9 points.

  3. Consistent gains on two MoE backbones: As shown in our W2/Q2 response (Table R13 / revised Table 18), STAR also consistently outperforms the same native NAIT-5ref baseline across all four metrics on both Qwen1.5-MoE (+0.68 Target, +1.24 Non-Target, +1.64 Aligned OOD, +2.36 Cross-Target OOD) and OLMoE (+0.85 Target, +1.95 Non-Target, +1.22 Aligned OOD, +0.72 Cross-Target OOD).

We hope these details provide helpful context for the five-reference comparisons. Thank you again for your time and feedback!

Dear Reviewer LzLa,

Thank you very much for your constructive feedback and for raising your score. We truly appreciate your time and support, and will make sure to include the additional results in the final version.

Best regards,

Authors of Submission 12971

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.