acceptodds
Under review as a conference paper at ICLR 2027

Localizing RL-Induced Tool Use to a Single Crosscoder Feature

Abstract

Fine-tuning through RL reshapes the internal representations of language models to enable agentic behaviors such as tool use, yet the mechanistic basis of these changes remains poorly understood. While RL substantially improves structured tool-call generation, it is unclear which features emerge, which are preserved, and whether identified features can be leveraged for retraining-free behavioral control. In this work, we show that isolate a compact set of RL-specific features that mediate tool-calling capability1 in . Across a 48-crosscoder hyperparameter sweep, encode-decode reconstruction improves the RL model’s tool correctness by +31.1 ± 9.7 pp and passively transfers toolcalling ability to the frozen base model by +6.8 ± 5.0 pp which we call a . Our findings show that DFC partitioning concentrates RL-introduced capability into a minimal, steerable feature set that enables runtime behavioral control of agentic LLMs.

Primary area:
interpretability and explainable AI
Keywords:
Methods (probing, steering, causal interventions), Concept Discovery (e.g., SAEs, dictionary learning), Applications of interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2iFFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper applies Dedicated Feature Crosscoders to a base/ToolRL-fine-tuned Qwen2.5-3B pair to study where tool-related behavioral changes are represented. It reports reconstruction-induced behavioral transfer and large steering effects from a single model-exclusive feature. The evaluation includes 48 crosscoder variants, comparisons with shared and unpartitioned features, and steering sweeps evaluated on 40 prompts per cell.

Critical strengths

The comparison between exclusive, shared, and unpartitioned dictionaries addresses a concrete localization question. Examining feature budgets and reconstruction behavior is more informative than presenting one successful activation edit. The finding that the strongest features respond to narrow formatting markers is also useful and is appropriately discussed in Section 6.7.

Critical weaknesses
  • The main metric does not establish task-appropriate tool use. Section 3.4 defines tool_correctness as fuzzy matching a called name to an available tool in the prompt, without checking the requested task, arguments, or execution outcome. Figure 5 exposes the consequence: after steering, the model invents a user request and then calls a tool for that invented request, which is scored as correct. Before steering, it asks for clarification. I appreciate the narrower terminology in the limitations, but this rubric cannot distinguish useful tool-calling propensity from inappropriate or hallucinated calls.

  • The headline single-feature result is selected from steering sweeps on a small reused evaluation set. The +65-point result and its interval describe a selected configuration on 40 prompts, rather than an independently validated effect of a frozen feature/layer/strength choice. This matters because the central claim is compact, reliable localization, not merely the existence of a high-scoring steering configuration.

Questions for authors
  1. How do the results change when existing outputs are scored for the requested task, including tool choice and argument correctness, rather than whether the called tool appears in the prompt?
  2. What is the complete input for Figure 5? Please distinguish the supplied user request from model-generated text and explain why the steered call is correct for that request.
  3. Which data were used to select the feature, layer, and steering strength? Was the selected configuration subsequently evaluated on untouched prompts, without further selection?
  4. Does ablating the identified feature impair tool use in the unsteered RL model? How does this compare with norm-matched random or shared-feature controls? This would distinguish a feature that can induce the behavior from one on which the model normally relies.
  5. Equation (1) applies the additional penalty to both A-exclusive-plus-shared and B-exclusive-plus-shared features, whereas Section 3.3 describes an exclusive-only penalty. Which objective was implemented, and does the penalty comparison also change regularization of the shared features?
Reviewer_nU4yOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how RL fine-tuning for tool use changes the internal representations of a language model, using Dedicated Feature Crosscoders, which split the crosscoder dictionary into A-exclusive, B-exclusive, and shared features. The authors compare ToolRL-Qwen2.5-3B (Model A) with Qwen2.5-3B (Model B) and train 48 crosscoders with different configurations. They find that replacing the activations with the crosscoder reconstruction improves Model A’s tool correctness by 31.2 pp on average and also improves Model B by 6.8 pp. They then rank the A-exclusive features by Cohen’s d between tool-use and general text, and steer Model A by amplifying these features: a single feature at layer 13 gives a +65 pp improvement, while the standard crosscoder needs 33 features to reach its peak of +70 pp.

Critical strengths

The paper's focus is interesting and reasonable. Using model diffing to understand differences before and after RL training on tool-use behaviors is an important direction and has drawn attention.

Transparency: The paper reports all 48 hyperparameter configurations' results, with code and model checkpoints available. Therefore, it is easy to follow and reproduce the paper's results and verify the arguments.

Critical weaknesses

The evaluation metric is misleading: To my understanding, the criterion for tool calls only judges whether a tool name in the model output occurs in the tool list in the task prompt, regardless of the correctness of tool choices and arguments. Figure 5 is a failure case for the evaluation: the trained model first fabricates the user's query and then uses a tool to answer it. I think this should not be a desired behavior for the trained model.

The separation in the UMAP is determined by design: The DFC forces the decoder vectors of exclusive features to be zero on the other model's side, so the two groups are orthogonal by construction, not evidence.

DFC being better than CrossCoder is statistically unsupported: In Table 9, the confidence intervals show a big overlap between the performance of DFC features and CrossCoder features even when the number of features is 1 or 2. So the claim that DFC is better than CrossCoder requires more evidence, and increasing the number of evaluation prompts may help. Moreover, the authors mention "partially refute the DFC's design intent" in Section 7, which contradicts the title, which proposes that tool calling is localized to a single feature.

Questions for authors

Why is the evaluation set small?: Only 100/40 tasks are judged during the scanning/steering stage, respectively, while the evaluation set of ToolRL has a much larger number.

Why does the ToolRL model have only 19% accuracy on its own evaluation set? The evaluation configuration may hurt the real performance of the ToolRL model; for example, max_new_tokens=200 may truncate the thinking process. In the original ToolRL paper, the same model (Qwen2.5-3B with GRPO) reaches 53% on BFCL V3, which is a harder benchmark than the current evaluation because it additionally evaluates the tool choice and arguments.

The reference model should be the instruct model. The ToolRL model is trained from Qwen2.5-3B-Instruct, but this paper uses the pretrained Qwen2.5-3B as Model B, so the difference between the two models comes from both instruction tuning and RL. I think this setting is wrong, and the paper should compare against the instruct model.

The reconstruction misses critical details. How is the reconstruction implemented? It is not clear to me at which layer the activations are replaced, and what is fed into Model A’s side of the crosscoder when only Model B is generating. If the encoder reads both models’ activations, I think Model B may directly use Model A’s information, which could explain the capability spillover.

Reviewer_mcRuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies crosscoders to a Qwen2.5-3B base model and its ToolRL-fine-tuned counterpart. Dedicated Feature Crosscoders partition the learned dictionary into model-exclusive and shared features. The study evaluates 48 variants, reports improved tool-related scores after activation reconstruction, and identifies RL-exclusive features whose amplification changes tool-call generation. Its central claims are capability spillover to the frozen base model and localization of a large steering effect to a single crosscoder feature.

Critical strengths
  1. The paper investigates a relevant post-training interpretability question. Comparing related base and RL models through shared and exclusive sparse features is a reasonable approach to studying changes in tool-call behavior. The reconstruction, partition, and steering experiments provide several complementary observations.

  2. The manuscript exposes useful limitations and negative results. It reports that base-model format accuracy does not improve, that the architecture-level spillover difference is not significant, and that important features resemble structural-template detectors. These observations help identify a narrower and more defensible interpretation of the results.

Critical weaknesses
  1. The scoring rubric does not establish correct tool use. Section 3.4 defines tool correctness as fuzzy matching a tool name available in the prompt, without requiring the task-appropriate tool, correct arguments, or execution success. Figure 5 additionally presents a steered response that supplies its own user request before calling a tool. This could reward tool-emission propensity or an invented task rather than correct behavior. The central gains need reevaluation against explicit user goals and complete valid calls.

  2. Steering sufficiency does not establish capability localization. Amplifying one feature can change output behavior without showing that the underlying capability resides in that feature. The paper lacks a necessity test and matched nonzero control directions. B-exclusive steering is zero by construction, as Section 6.5 acknowledges, so it cannot establish behavioral specificity. These concerns are particularly important because the identified features resemble formatting detectors.

  3. The capability-spillover interpretation is underdetermined. The base model never acquires valid tool-call formatting under the reported metric, so its 6.8-point gain is not demonstrated executable tool use. The reconstruction protocol also needs to state explicitly whether base-model inference uses only base activations or receives information from the RL branch. Without that clarification and appropriate reconstruction controls, the result does not establish passive transfer of an RL-induced capability.

  4. Selection and dependence weaken the statistical claims. Steering maxima and their confidence intervals are reported after choosing settings on the same 40 prompts, without fresh-sample validation. Tests across 48 related crosscoders or multiple steering cells also require accounting for shared prompts and correlated configurations. Additionally, a nonsignificant difference does not establish the claimed interchangeability of the architectures.

  5. The exclusive-penalty experiment does not isolate exclusivity as written. Equation 1 applies the additional penalty to both fA∪shf_{A\cup sh} and fB∪shf_{B\cup sh}, so shared features are penalized as well. This conflicts with the description of varying only the exclusive penalty and confounds the interpretation that penalization pushes information into the shared partition. The implemented objective and a fixed-shared-penalty comparison need clarification.

  6. The single-feature saturation claim is stronger than the reported table. Section 6.1 states that steering one feature matches steering all seven A-exclusive features. Appendix Table 9 instead reports a best gain of +65.0 points for one feature and +47.5 for seven. This supports an observed optimum at a small subset, rather than a flat saturation curve. The selected optimum also lacks independent replication.

Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.