- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ToolSAE, which selects tool-calling training data using SAE feature activations from within the model. The method first applies a Top-K SAE to decompose hidden states into sparse features, then screens candidate features based on three criteria: activation differences between tool and non-tool samples, enrichment across behavior categories, and penalties for format/data-source shortcuts. Token-level diagnostics are used to filter out features that primarily respond to formatting markers. Finally, candidate samples are ranked by validation scores combined with IDF-weighted activation strength. The experiments mainly use Qwen3.5-9B-Base, selecting approximately 6,800 records from 135,910 candidates, and fine-tuning them together with a fixed set of 32k general instructions. Results show that using only 5% tool data outperforms full-data training on BFCL v4's Non-Live AST and Multi-Turn benchmarks, though it still trails full training on Live AST. On general evaluations, overall performance is comparable to other 5% selection methods. The paper also provides component ablations, feature analysis, and preliminary data-reuse experiments on Qwen3-14B-Base.
Critical strengths
-
Compared to approaches that only pursue global feature coverage, this paper explicitly distinguishes tool-behavior-related features from format- and source-related features, and combines sample-level filtering with token-level diagnostics. This represents a worthwhile incremental direction relative to existing activation-based selection methods. As shown in Table 11, removing either the token-level diagnostics or the shortcut penalty causes some key metrics to decline, providing preliminary support for the contribution of these steps.
-
Under strictly controlled final SFT configurations in the main experiments, the 5% tool-calling data outperforms the full dataset on Non-Live AST and Multi-Turn; ablations show that removing the shortcut penalty or token-level diagnostics significantly harms the corresponding performance. These results have practical implications for tool-calling training under low-data-budget settings.
Critical weaknesses
- The paper lacks a direct comparison with the most closely related recent activation-driven selection methods, which limits the assessment of its relative advantages and the boundaries of its contribution. MoNA (NeurIPS 2025), which leverages monosemantic SAE activations for task-oriented sample selection, and NAIT (ICLR 2026), which filters instruction data based on capability-relevant activation directions, are both directly relevant to this work. Existing baselines such as Diversity-SAE are insufficient for judging the paper's advantages over these works.
Questions for authors
- Could the authors provide the results of a direct comparison between the proposed method and MoNA or NAIT? These comparisons should be conducted under the same tool candidate pool, selection budget, and training settings. In particular, they should include MoNA, which likewise adopts SAE representations, so that the competitiveness of the proposed method relative to these closest neighboring approaches can be properly assessed.
2 comments
Reviewer Response
Thank you for the additional experiments and detailed response. The direct comparisons with MoNA and IGDS address my main concern regarding the lack of comparisons with closely related activation-driven data selection methods.The results provide stronger evidence for the effectiveness of ToolSAE, particularly its advantage on Multi-Turn tasks. I consider my original concern to be substantially addressed. The additional experiments strengthen the paper's empirical contributions, and I will maintain my current positive rating.
Thank you for your valuable suggestions and the time you have dedicated to reviewing our work. Your feedback has been very helpful in improving our research.
W1 / Q1. Direct comparison with activation-driven selection methods
We compared ToolSAE with MoNA [1] and IGDS [2]. MoNA ranks traces by their generalized Jaccard similarity to a target-task prototype in SAE activation space. IGDS identifies task-relevant SAE features using activation frequency and intervention-based validation, then ranks candidates by those features' activations. Each method selected 6,795 examples (approximately 5%) from the same pool of 135,910 candidates. We used the same 9B base model, 32,000 general instructions, and final SFT configuration across methods.
Downstream performance. ToolSAE scores highest on Multi-Turn (19.88), by 4.63 points over IGDS and 19.63 points over MoNA. It also scores highest on Live AST (74.75), by 0.76 points over MoNA and 3.43 points over IGDS. MoNA scores 0.54 points higher than ToolSAE on Non-Live AST. In this comparison, all traces selected by MoNA were single-turn interactions. During Multi-Turn evaluation, the resulting model frequently repeated tool calls until termination.
Contribution of tool-specific screening. In submitted Table 11, the ablations retain 512 features. Removing the shortcut penalty reduces Multi-Turn from 19.88 to 16.25, and omitting token-level diagnosis reduces it to 17.50. Both ablations also lower Non-Live and Live AST scores.
[1] Da Ma et al. Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations. In Advances in Neural Information Processing Systems, 2025.
[2] Ling Shi et al. From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 2026.