acceptodds
Under review as a conference paper at ICLR 2027

ToolSAE: Data Selection for Tool Learning via Capability-Relevant Neuron Activations

Abstract

Tool calling is a foundational capability that enables large language model (LLM) agents to interact with the external world. Training such capabilities increasingly relies on large-scale tool-calling traces, making it critical to identify the most useful traces from these extensive datasets. Existing data selection methods typically prioritize alignment with target examples or generic criteria such as quality and diversity, yet these signals often capture only surface features (e.g., schema keywords), while failing to reflect the specific tool-calling capabilities to be strengthened. In this work, we introduce ToolSAE, a capability-guided data selection framework that explicitly identifies tool-capability neurons in the SAE latent space and selects training examples based on their activations. We employ sparse autoencoders to decompose polysemantic model activations into sparse, interpretable representations. Through contrastive neuron profiling and token-level diagnosis, ToolSAE identifies capability-specific neurons while filtering out those responding primarily to surface formatting. It then ranks candidate traces by their weighted activations over the retained neurons, prioritizing examples that engage tool-calling behaviors. Empirical evaluations show that training on just 5% of the tool-calling data selected by ToolSAE achieves tool-calling performance competitive with full-data training while maintaining comparable general-domain performance. Further analysis shows that the identified neurons capture tool-calling semantics and provide effective signals for capability-guided data selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool Calling, Data Selection, Sparse Autoencoders

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ndAWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ToolSAE, which selects tool-calling training data using SAE feature activations from within the model. The method first applies a Top-K SAE to decompose hidden states into sparse features, then screens candidate features based on three criteria: activation differences between tool and non-tool samples, enrichment across behavior categories, and penalties for format/data-source shortcuts. Token-level diagnostics are used to filter out features that primarily respond to formatting markers. Finally, candidate samples are ranked by validation scores combined with IDF-weighted activation strength. The experiments mainly use Qwen3.5-9B-Base, selecting approximately 6,800 records from 135,910 candidates, and fine-tuning them together with a fixed set of 32k general instructions. Results show that using only 5% tool data outperforms full-data training on BFCL v4's Non-Live AST and Multi-Turn benchmarks, though it still trails full training on Live AST. On general evaluations, overall performance is comparable to other 5% selection methods. The paper also provides component ablations, feature analysis, and preliminary data-reuse experiments on Qwen3-14B-Base.

Critical strengths
  1. Compared to approaches that only pursue global feature coverage, this paper explicitly distinguishes tool-behavior-related features from format- and source-related features, and combines sample-level filtering with token-level diagnostics. This represents a worthwhile incremental direction relative to existing activation-based selection methods. As shown in Table 11, removing either the token-level diagnostics or the shortcut penalty causes some key metrics to decline, providing preliminary support for the contribution of these steps.

  2. Under strictly controlled final SFT configurations in the main experiments, the 5% tool-calling data outperforms the full dataset on Non-Live AST and Multi-Turn; ablations show that removing the shortcut penalty or token-level diagnostics significantly harms the corresponding performance. These results have practical implications for tool-calling training under low-data-budget settings.

Critical weaknesses
  1. The paper lacks a direct comparison with the most closely related recent activation-driven selection methods, which limits the assessment of its relative advantages and the boundaries of its contribution. MoNA (NeurIPS 2025), which leverages monosemantic SAE activations for task-oriented sample selection, and NAIT (ICLR 2026), which filters instruction data based on capability-relevant activation directions, are both directly relevant to this work. Existing baselines such as Diversity-SAE are insufficient for judging the paper's advantages over these works.
Questions for authors
  1. Could the authors provide the results of a direct comparison between the proposed method and MoNA or NAIT? These comparisons should be conducted under the same tool candidate pool, selection budget, and training settings. In particular, they should include MoNA, which likewise adopts SAE representations, so that the competitiveness of the proposed method relative to these closest neighboring approaches can be properly assessed.
2 comments

Thank you for your valuable suggestions and the time you have dedicated to reviewing our work. Your feedback has been very helpful in improving our research.

W1 / Q1. Direct comparison with activation-driven selection methods

We compared ToolSAE with MoNA [1] and IGDS [2]. MoNA ranks traces by their generalized Jaccard similarity to a target-task prototype in SAE activation space. IGDS identifies task-relevant SAE features using activation frequency and intervention-based validation, then ranks candidates by those features' activations. Each method selected 6,795 examples (approximately 5%) from the same pool of 135,910 candidates. We used the same 9B base model, 32,000 general instructions, and final SFT configuration across methods.

MethodNon-Live ASTLive ASTMulti-Turn
ToolSAE86.9874.7519.88
Random (avg @ 3)85.7072.7914.29
MoNA87.5273.990.25
IGDS80.3371.3215.25

Downstream performance. ToolSAE scores highest on Multi-Turn (19.88), by 4.63 points over IGDS and 19.63 points over MoNA. It also scores highest on Live AST (74.75), by 0.76 points over MoNA and 3.43 points over IGDS. MoNA scores 0.54 points higher than ToolSAE on Non-Live AST. In this comparison, all traces selected by MoNA were single-turn interactions. During Multi-Turn evaluation, the resulting model frequently repeated tool calls until termination.

Contribution of tool-specific screening. In submitted Table 11, the ablations retain 512 features. Removing the shortcut penalty reduces Multi-Turn from 19.88 to 16.25, and omitting token-level diagnosis reduces it to 17.50. Both ablations also lower Non-Live and Live AST scores.

[1] Da Ma et al. Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations. In Advances in Neural Information Processing Systems, 2025.

[2] Ling Shi et al. From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 2026.

Reviewer_ndAWOpenReview ↗

Reviewer Response

Thank you for the additional experiments and detailed response. The direct comparisons with MoNA and IGDS address my main concern regarding the lack of comparisons with closely related activation-driven data selection methods.The results provide stronger evidence for the effectiveness of ToolSAE, particularly its advantage on Multi-Turn tasks. I consider my original concern to be substantially addressed. The additional experiments strengthen the paper's empirical contributions, and I will maintain my current positive rating.

Reviewer_B5ABOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ToolSAE, a capability-guided data selection framework applied to the task of supervised fine-tuning for LLM tool learning. The framework consists of three main components: (1) a Top-K Sparse Autoencoder (SAE) representation module that decomposes polysemantic hidden states into sparse, monosemantic neuron activations; (2) a tool-capability neuron identification pipeline combining sample-level contrastive profiling and token-level diagnosis, which isolates neurons specific to tool-calling behaviors while filtering out shortcuts related to surface formatting and data source biases; and (3) a capability-guided scoring mechanism that ranks and selects the top 5% of candidate traces based on their IDF-weighted activations over the retained neurons. Experiments on BFCL v4 and general-domain benchmarks using Qwen3.5-9B-Base verify that training on the 5% subset selected by ToolSAE outperforms full-data training on Non-Live AST and Multi-Turn tasks while preserving general instruction-following capabilities.

Critical strengths
  1. The paper explores a capability-guided data selection paradigm based on model-internal neuron activations to improve the data efficiency of tool-calling fine-tuning. Instead of relying on small target reference sets or generic difficulty metrics, isolating capability-relevant features in the SAE latent space offers a well-motivated direction for curating agentic training data.
  2. The proposed methodology carefully addresses the issue of models overfitting to superficial formatting patterns in tool-calling traces. By combining contrastive neuron profiling—which balances relevance, behavioral enrichment, and metadata shortcut penalties—with token-level diagnosis, the framework effectively filters out neurons that merely respond to JSON delimiters or schema boilerplate.
  3. The authors conduct a thorough empirical evaluation across seven selection baselines in Section 3.2, complemented by detailed component ablations in Section 3.3 and qualitative neuron analyses in Section 3.4. It is encouraging to see that retaining only 5% of the 136k candidate pool achieves competitive performance against full-pool training and shows promising cross-model transferability in Appendix C.1.
Critical weaknesses

W1. The neuron identification pipeline relies heavily on hand-crafted heuristics and manual rules. While the two-stage filtering framework in Section 2.3 is intuitive, its implementation depends on numerous pre-defined rules and manual thresholds. Specifically, computing behavioral enrichment (Equation 6) and token-level diagnosis (Equation 8) requires manually designing priority-ordered category rules (Appendix A.2, Table 4) and regular-expression-like token region definitions (Appendix A.3, Table 7), alongside multiple hard filtering thresholds in Appendix A.5 (e.g., α=0.5\alpha=0.5, γ=0.8\gamma=0.8). This makes the method somewhat engineered for the current ChatML format and dataset pool, and it could be clearer how easily this profiling pipeline adapts to new tool schemas or agentic tasks without manual recalibration.

W2. The sample-level aggregation and additive scoring mechanism may oversimplify multi-step interactions. In Section 2.2 (Equation 4), token-level activations across an entire conversation are compressed into a single vector via dimension-wise max pooling, which is then summed linearly in Section 2.4 (Equation 9) to score candidates. Max pooling only captures peak single-token activations and discards sequential or compositional dependencies across turns, while the additive scoring does not explicitly prevent redundancy among the selected examples. Consequently, the scoring function may implicitly favor longer traces or specific dataset compositions (given the skewed source-category distribution in Appendix A.4, Table 8) rather than balanced capability coverage.

W3. Empirical comparisons and result robustness could be further strengthened. In Section 3.2 (Table 1), the performance gain over Random selection on Non-Live AST Avg. is relatively marginal (86.98 vs. 85.70), and variance across multiple training seeds is not reported for ToolSAE. Moreover, while Section 4 discusses recent SAE-based selection methods like Diversity-SAE and MoNA, the experiments in Section 3.2 only compare against Diversity-SAE and omit target-aligned SAE baselines like MoNA. Including such comparisons and clarifying whether key hyperparameters (such as kf=512k_f=512 in Section 3.3, Figure 4-b) were tuned on an independent validation split would make the empirical claims more solid.

Questions for authors
  1. Were the filtering thresholds in Appendix A.5 and the neuron count kf=512k_f=512 (Section 3.3) tuned on an independent validation set or directly on the BFCL evaluation splits?
  2. What is the actual distribution of the 5% ToolSAE-selected subset across data sources, tool categories (cf. Table 8), and average sequence length compared to the full pool?
  3. How might the hand-crafted category rules and ChatML-specific token templates (Appendix A.2 & A.3) be adapted to unseen agentic formats without manual recalibration?
2 comments

We sincerely appreciate your careful review and constructive suggestions.

W1 / Q1. Parameter choice and sensitivity

We set the filtering thresholds and the default feature count of 512 heuristically and assessed them on the shared 120-example reference set used by the target-guided baselines. BFCL v4 was used only for downstream evaluation and sensitivity analysis after the default settings were fixed. Appendix A.5 details the thresholds, and Figure 4 and Table 11 report the sensitivity results.

Number of retained features. We tested 128, 256, 512, and 1,024 features. The default setting of 512 achieved the highest Non-Live AST score (86.98). Increasing the number to 1,024 reduced Non-Live AST performance, while Multi-Turn performance remained nearly unchanged (19.88 vs. 19.75).

Profiling weight and token-level filtering. Increasing the profiling weight α\alpha from 0.5 to 1 reduced Multi-Turn performance from 19.88 to 15.13. We also evaluated the contribution of token-level filtering separately. With 512 features retained, removing token-level filtering reduced Multi-Turn performance from 19.88 to 17.50.

W2 / Q2. Composition of the 5% ToolSAE subset

We appreciate the suggestion to examine how the selected subset differs from the full candidate pool.

Source distribution. Table 1 compares the source distributions of the full candidate pool and the selected subset. The largest changes are a decrease in Glaive's share (46.62% to 31.58%) and an increase in xLAM's share (44.15% to 55.32%).

Table 1: Source distribution in the candidate pool and the 5% ToolSAE subset

SourceFull poolToolSAE subset
xLAM44.15%55.32%
Glaive46.62%31.58%
ToolACE7.76%12.89%
When2Call1.47%0.21%

Behavior categories. Table 2 compares the behavioral category distributions of the full candidate pool and the selected subset, using the category definitions in Appendix A.2. Parallel-call examples show the largest increase, from 26.20% of the full pool to 46.61% of the selected subset.

Table 2: Behavioral category distribution in the candidate pool and the 5% ToolSAE subset

CategoryFull poolToolSAE subset
Parallel call26.20%46.61%
Simple call6.53%2.18%
Argument-heavy9.27%6.78%
Multi-tool selection8.65%8.42%
Tool-response grounding46.91%35.79%
Reject irrelevance2.00%0.22%
Non-tool0.42%0.00%

W2. Evaluating potential length bias in ToolSAE

We acknowledge that max pooling can favor longer traces. By retaining only the strongest activation within a trace, it discards activation timing and frequency, while longer traces provide more opportunities for a feature to activate. Moreover, scoring each trace independently does not explicitly penalize redundancy among selected examples.

Trace length analysis. We measured the sequence lengths of all 135,910 candidate traces and the 6,795 traces selected by ToolSAE. The selected traces have an average length of 881.91 tokens, 38.5% longer than the full-pool average of 636.72 tokens. All traces fit within the 8,192-token training limit.

As described in Appendix A.3, the shortcut penalty uses surface statistics including total message-content length, first-user-query length, and serialized tool-schema length. These statistics are measured in characters. The penalty reduces the profiling scores of features whose activations are strongly correlated with these or other surface statistics.

Length-based selection baseline. To examine whether the performance gains can be explained by trace length alone, we constructed a length-based baseline by selecting the longest 5% of candidate traces according to their token counts. We used the same 9B base model, general instructions, and final SFT configuration as ToolSAE.

Table 3: Length-based selection, Random, and ToolSAE on Qwen3.5-9B-Base

Selection methodNon-Live ASTLive ASTMulti-TurnGeneral Avg
Random (avg x 3 )85.7072.7914.2977.33
Longest 5%82.7373.1517.1275.52
ToolSAE86.9874.7519.8877.53

ToolSAE outperforms length-based selection by 4.25 points on Non-Live AST, 1.60 points on Live AST, 2.76 points on Multi-Turn, and 2.01 points on General Avg. These results indicate that selecting longer traces alone does not account for ToolSAE's performance gains, suggesting that its activation-based selection captures useful signals beyond sequence length.

W1 / Q3. Adapting the pipeline to another agent format

ToolSAE can be adapted to new tool-calling formats that expose the same underlying structured fields.

Reusable behavior classification rules. The behavior classification rules operate on structured records before serialization. Once a new format is mapped to the same message roles, tool calls, arguments, and observations, the existing rules can be reused. For example, the parallel-call rule continues to count the number of calls within each message.

Format-specific token mapping. Token-level shortcut diagnosis relies on a mapping that identifies delimiters and other formatting tokens in rendered traces. For the template change below, we updated the delimiter definitions and token-span mapping.

Our main experiments use the Qwen3.5 ChatML-style template (Appendix A.5), which represents tool calls using function and parameter tags. To evaluate format adaptation, we retained the Qwen3.5-9B backbone but switched to the Qwen3 NoThink template, which encodes each call as JSON within a tool-call block. The original behavior rules produced the same category assignments for all 135,910 records. With the updated token-span mapping, ToolSAE selected 6,795 examples under the new template using the original shortcut threshold (0.8) and feature count (512).

W3. ToolSAE training seed variation and matched SAE baselines

Thank you for the valuable suggestion. We conducted additional experiments to evaluate training seed variation and compare ToolSAE with closely related SAE-based selection methods.

ToolSAE across training seeds. We fixed the selected subset and repeated 9B SFT with three different random seeds. ToolSAE achieved 87.01±0.4487.01 \pm 0.44 on Non-Live AST, 73.69±0.9773.69 \pm 0.97 on Live AST, and 19.59±0.2619.59 \pm 0.26 on Multi-Turn (mean ±\pm sample SD).

Matched SAE baselines. We also compared ToolSAE with MoNA [1] and IGDS [2], using Random as a reference. MoNA ranks examples by the similarity of their SAE activations to a prototype constructed from target-task examples. IGDS identifies task-relevant SAE features based on activation frequency and intervention-based validation effects, then ranks examples by their activations on these features. All methods use the same candidate pool and 5% selection budget, with identical general instructions and final SFT settings. Table 4 reports the results.

Table 4: Matched comparison on Qwen3.5-9B-Base

MethodNon-Live ASTLive ASTMulti-TurnGeneral Avg
ToolSAE86.9874.7519.8877.53
Random (avg @ 3)85.7072.7914.2977.33
MoNA87.5273.990.2575.57
IGDS80.3371.3215.2575.71

ToolSAE scores highest on Multi-Turn, Live AST, and General Avg. in Table 4. MoNA scores highest on Non-Live AST. In this comparison, all traces selected by MoNA were single-turn interactions. During Multi-Turn evaluation, the resulting model frequently repeated tool calls until termination.

[1] Da Ma et al. Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations. In Advances in Neural Information Processing Systems, 2025.

[2] Ling Shi et al. From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 2026.

Reviewer_qKgDOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ToolSAE selects tool-calling SFT data using SAE latent activations. It identifies tool-relevant features through contrastive profiling and token-level diagnosis, then ranks examples by weighted activations. Experiments use a fixed 5% tool-data budget with shared general data and final SFT conditions. Results are competitive with full-data training and improve over random selection, mainly on Multi-Turn, while general-domain scores remain comparable.

Critical strengths
  1. Under the same 5% data budget and final training setup, ToolSAE improves Multi-Turn by 5.59 points over Random and 2.76 points over DSIR
  2. The diagnosis and shortcut-penalty ablations support the tool-specific filtering design. These results provide evidence that the filtering contributes to better data selection.
Critical weaknesses
  1. My main concern is whether the tool-specific design adds sufficient value beyond existing SAE-based selection.
  • IGDS (ACL 2026) already identifies task-relevant SAE features and selects examples by their activations, yet it is not discussed.
  • MoNA (NeurIPS 2025) is cited but not evaluated.
  • The contribution therefore rests on the tool-specific feature identification and filtering, whose advantage over these closest methods has not been demonstrated under matched training conditions.
  1. The experiments do not clearly show what the full SAE-based procedure adds over simpler alternatives.
  • SAE improves AST scores over raw hidden states, but raw hidden-state selection reports 20.25 on Multi-Turn versus 19.88, where ToolSAE shows its largest gain over Random. With the same SAE features, uniform weighting remains competitive on AST, and score-only weighting matches the full Multi-Turn score (Table 11). Full weighting improves AST over score-only, but its additional benefit on Multi-Turn remains unclear, as does the benefit of using sparse rather than raw hidden-state features on this task.
  1. The evidence for a reliable advantage across training runs and models is limited.
  • The Multi-Turn gain over Random is meaningful at 5.59 points, while Non-Live and Live gains are 1.28 and 1.96 points. The 0.20-point change in the general average mainly supports preservation of general performance.
  • Random is averaged over three seeds, but dispersion and comparable repeated results for ToolSAE are not reported. As the authors acknowledge, the 14B experiment changes training settings and lacks a matched 5% Random control.
  • It therefore provides limited evidence that the advantage over random selection transfers to another model.
Questions for authors
  1. What do the tool-specific changes add beyond IGDS and MoNA, and are matched comparisons with the closest SAE-based selection method available?

  2. Are repeated final-training results available for ToolSAE and Random? For the 14B experiment, are there results using 5% random data under the same training settings?

3 comments

Thank you very much for the time and effort you devoted to our work. Your insightful comments have greatly strengthened our work.

W1 / Q1. Motivation and distinction from MoNA and IGDS

Motivation. A feature that activates strongly on parallel calls may simply respond to the repeated JSON delimiters in their serialization, rather than to parallel-call behavior itself. This ambiguity motivates contrastive profiling across traces. Specifically, we examine whether a feature is enriched in parallel-call traces and whether its activation is more closely associated with tool-call content or formatting patterns. Enrichment alone is insufficient: a feature may be prevalent in parallel-call traces simply because they share repeated syntax. We therefore perform token-level diagnosis to identify which tokens account for its strongest activations, distinguishing behavior-related responses from those driven by formatting.

Distinction from SAE-based selection. MoNA [1] represents a target task using the average SAE activation pattern of reference examples and ranks candidates by their generalized Jaccard similarity to this prototype. IGDS [2] also identifies individual task-relevant SAE features. Specifically, it selects features that activate frequently on target examples, evaluates the effect of amplifying each feature on validation performance, and uses the selected features to rank candidate examples.

Both MoNA and IGDS use SAE activations to identify task-relevant training data. The key distinction is that ToolSAE goes a step further: it explicitly examines whether activations within tool traces reflect the demonstrated tool-use behavior or are instead driven by their serialization. We operationalize this distinction through contrastive profiling across traces and token-level diagnosis within each trace.

ToolSAE thus combines two complementary signals: which traces activate a feature and which tokens trigger its activation. We use both criteria to identify features whose activations capture tool-use behavior rather than superficial formatting, and then use these features to rank training examples. This distinction is substantiated by the findings of Lin et al. [3].

[1] Da Ma et al. Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations. In Advances in Neural Information Processing Systems, 2025.

[2] Ling Shi et al. From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 2026.

[3] Qiqiang Lin et al. Robust Function-Calling for On-Device Language Models via Function Masking. In The Thirteenth International Conference on Learning Representations, 2025.

W1 / Q1. Comparison with general SAE-based data selection methods

We compared MoNA and IGDS on the same 9B base model. All methods selected 6,795 examples (approximately 5%) from the same pool of 135,910 candidates and used the same 32,000 general instructions and final SFT configuration.

Table 1: ToolSAE vs. MoNA, IGDS, and Random on Qwen3.5-9B-Base (5% selection budget)

MethodNon-Live ASTLive ASTMulti-TurnGeneral Avg
ToolSAE86.9874.7519.8877.53
Random (mean over 3 selections)85.7072.7914.2977.33
MoNA87.5273.990.2575.57
IGDS80.3371.3215.2575.71

ToolSAE achieves the highest Multi-Turn score (19.88), exceeding Random by 5.59 points, IGDS by 4.63 points, and MoNA by 19.63 points. MoNA performs best on Non-Live AST (87.52), while ToolSAE achieves the highest scores on Live AST (74.75) and General Avg (77.53). We further examined the examples selected by MoNA and found that all of them contain only a single interaction turn. During Multi-Turn evaluation, the resulting model frequently repeated tool calls until termination, which is consistent with its low Multi-Turn score (0.25).

The matched comparisons show that ToolSAE achieved the highest Multi-Turn score in this setting. The ablations below examine the contributions of the individual components.

W2. Contributions of SAE representations, weighting, and feature screening

Table 11 in Appendix C.2 compares the full ToolSAE pipeline with simpler alternatives. Replacing SAE features with raw hidden states tests the effect of the representation. Varying the weights with the SAE features fixed isolates the effect of weighting.

Table 2: SAE representation and feature-weighting ablations on Qwen3.5-9B-Base

Selection variantNon-Live ASTLive ASTMulti-Turn
Raw hidden states81.1770.2620.25
SAE, uniform weights86.3175.3618.25
SAE, validation-score-only weights85.3673.8319.88
ToolSAE86.9874.7519.88

SAE representation. Compared with raw hidden-state selection, SAE-based selection improves Non-Live AST by 5.81 points and Live AST by 4.49 points. Raw hidden-state selection scores 0.37 points higher on Multi-Turn (20.25 vs. 19.88).

Weighting strategy. With the SAE features fixed, adding IDF to validation-score weighting improves Non-Live AST by 1.62 points and Live AST by 0.92 points. Both settings score 19.88 on Multi-Turn. Compared with uniform weighting, full weighting scores 0.67 points higher on Non-Live AST and 1.63 points higher on Multi-Turn. Uniform weighting scores 0.61 points higher on Live AST.

Feature screening. We evaluate the contributions of the shortcut penalty and token-level filtering in separate ablations.

Table 3: Shortcut penalty and token-level filtering ablations on Qwen3.5-9B

Feature-identification variant (submitted Table 11)Non-Live ASTLive ASTMulti-Turn
Full ToolSAE86.9874.7519.88
Without shortcut penalty84.0271.2516.25
Without token-level filtering79.8371.1717.50

Relative to full ToolSAE, removing the shortcut penalty reduces Non-Live AST, Live AST, and Multi-Turn scores by 2.96, 3.50, and 3.63 points, respectively. Removing token-level filtering with 512 features retained reduces the same scores by 7.15, 3.58, and 2.38 points, respectively, relative to full ToolSAE.

W3 / Q2. Variation across Random data selections and matched controls for 14B transfer

9B: Variation across the three original Random selections. We clarify that the three Random seeds refer to independent random data selections, rather than repeated SFT runs with different training seeds. Each seed independently samples 6,795 (5%) tool-calling examples from the same pool of 135,910 candidates. Since our objective is data selection, variation across independently sampled subsets directly measures how the choice of training data affects downstream performance.

Table 4 reports the BFCL scores for the three Random selections alongside the ToolSAE result from submitted Table 1.

Table 4: ToolSAE and three Random data selections on Qwen3.5-9B-Base

MethodNon-Live ASTLive ASTMulti-Turn
Random (mean ±\pm SD)85.70±1.8985.70 \pm 1.8972.79±0.7372.79 \pm 0.7314.29±2.3814.29 \pm 2.38
ToolSAE86.9874.7519.88

ToolSAE outperforms the Random avg by 1.28, 1.96, and 5.59 points on Non-Live AST, Live AST, and Multi-Turn, respectively. Notably, its Multi-Turn score also exceeds the best of the three Random selections by 3.13 points. The reported standard deviations quantify variability across independently sampled data subsets, rather than optimization variability across repeated SFT runs.

ToolSAE across training seeds. We fixed the selected subset and repeated 9B SFT with three different random seeds. ToolSAE achieved 87.01±0.4487.01 \pm 0.44 on Non-Live AST, 73.69±0.9773.69 \pm 0.97 on Live AST, and 19.59±0.2619.59 \pm 0.26 on Multi-Turn (mean ±\pm sample SD).

14B: Matched Random controls for the transfer experiment. Appendix C.1 evaluates cross-model transfer by applying the 5% subset selected using the 9B SAE configuration to Qwen3-14B-Base. To establish a matched baseline, we additionally trained the same 14B model using three independently sampled 5% Random subsets, keeping the final training settings identical to those of the transfer experiment. Each subset contains 6,795 tool-calling examples.

Table 5: ToolSAE transfer, three Random selections, and Full-data training on Qwen3-14B-Base

Selection seed / methodNon-Live ASTLive ASTMulti-TurnGeneral Avg
Random (mean ±\pm SD)87.09±1.4087.09 \pm 1.4076.18±0.9276.18 \pm 0.9213.08±0.8513.08 \pm 0.8569.00±0.6269.00 \pm 0.62
ToolSAE-transfer91.3376.6813.3869.33
Full90.5477.8714.2567.69

ToolSAE-transfer scores 4.24 points higher than the matched Random mean on Non-Live AST. The differences are smaller on Live AST (+0.50), Multi-Turn (+0.30), and General Avg. (+0.33). Compared with Full-data training, ToolSAE-transfer scores higher on Non-Live AST (+0.79) and General Avg. (+1.64) and lower on Live AST (-1.19) and Multi-Turn (-0.87). The 5% runs used a global batch size of 64 for two epochs. The Full run used a global batch size of 128 for one epoch. We will include the Random baseline results in Appendix C.1.

The matched MoNA and IGDS comparisons address my concern about prior work. The 4.63-point Multi-Turn gain over IGDS at 9B is meaningful. My remaining concern is what the SAE representation and IDF weighting add. The gain over random selection alone cannot establish that. On Multi-Turn, raw hidden states score 20.25 versus 19.88 for ToolSAE, and removing IDF leaves the score at 19.88. At 14B, the gain over random is 4.24 points on Non-Live AST but only 0.30 on Multi-Turn. I am raising my rating from 1 to 2 (weak rejection).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.