acceptodds
Under review as a conference paper at ICLR 2027

When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMs

Abstract

Before invoking external tools, an agentic LLM must select among a K-way action space: executing a call, seeking clarification, answering directly, or declining. While internal activation steering can alter these pre-execution decisions, conventional aggregate metrics obscure where altered states land and what collateral damage they inflict. We present SAKIKO, an auditing framework that formalizes representation repair via directional error discovery, router-conditioned intervention, destination-resolved verification, and prospectively frozen statistical licensing. Across seven LLMs on When2Call and MetaTool, channel-keyed interventions induce direction-specific net gains in five models; across three sealed evaluations, none of 59 budget-matched random directions matches calibrated target gain. Crucially, destination auditing shows that behavioral movement does not equal repair: an intervention achieving +55 net gain corrupts over half of the baseline-correct decisions it touches, and promising point estimates on Qwen3-4B and Gemma-2-9B are formally declined due to finite-sample uncertainty. SAKIKO establishes the necessity of outcome-resolved adjudication before claiming internal repair.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, tool-use LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cc2wOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper shows that activation steering can be used to improve the model's ability to choose whether to call a tool or do something else (e.g. ask for clarification, refuse, etc.). It compares it to other steering baselines and finds it works better than those. It attempts to make broader claims about these results showing it is possible to "repair' some mistakes that a model makes using these targeted interventions.

Critical strengths

Potentially novel approach to evaluating some claims about model internals: The paper might be attempting to understand internal representations of tool choice. Though it is hard for me to understand what exactly is shown, see critical weaknesses.

Critical weaknesses

Soundness of the approach: The approach lacks comparison with the relevant baselines. My understanding is that the goal is to evaluate whether activation steering can be used to improve how models handle tool choice, and does so using supervised data. To justify progress along this goal, it is crucial to compare the proposed approach to simple supervised fine-tuning approaches such as simply fine-tuning a LoRA. The evaluations are also very narrow, mostly relying on What2Call, without making other measurements of the impact of such steering on simple benchmarks to evaluate how much performance degrades (like agent capabilities benchmarks). The paper claims that prior work might miss degradation due to only evaluating binary outcomes, but the more standard way of measuring such degradations is simply to run capability evaluations, which this paper does not do.

Clarity of the question being investigated and its motivation: I am not sure that my understanding of the problem this paper attempts to solve is correct. If it is correct, it has the soundness issues described above. If this is not the problem the paper attempts to solve, it does a poor job at explaining what problem it attempts to solve. A discussion or conclusion section would help the reader understand what the key takeaways are and how this represents progress towards a goal that could matter to AI practitioners or evaluators.

Questions for authors
  • Are you making claims beyond the direct usefulness of these steering interventions and about how mechanistically this task works?
  • If so, what are these claims, and why do they matter?
  • If not, how does the proposed method compare to simple baselines?
  • Why not rely on capability evaluations to measure model degradation?

Minor:

  • Could you make the figures less dense and make the font size match the font size of the text? Figure 2 in particular is very hard to read.
Reviewer_EdoZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SAKIKO, a framework for auditing activation interventions on pre-execution tool decisions. It discovers directional error channels, trains a linear router to condition channel-specific activation steering, and distinguishes target arrivals, transitions to other incorrect actions, and damage to baseline-correct decisions. A prospectively frozen statistical gate determines whether a configuration supports a specified repair claim. Experiments demonstrate that aggregate improvements can coexist with collateral damage, and that promising point estimates need not satisfy the chosen evidential criteria.

Critical strengths
  1. The paper provides a useful diagnostic decomposition of intervention outcomes. Its examples make clear that leaving a source error does not imply reaching the correct action, and that aggregate gains can hide losses among exposed baseline-correct cases.

  2. The evaluation includes budget-matched random directions and additional structural controls, separates historical from prospectively frozen evaluations, and reports unsuccessful or inconclusive settings. These practices help constrain overinterpretation of steering results.

Critical weaknesses
  1. The practical decision value of the audit beyond end-to-end utility remains insufficiently demonstrated. An intervention may legitimately improve overall outcomes despite some collateral errors; conversely, a correct pre-execution action does not establish successful tool execution or task completion. The experiments primarily evaluate action selection rather than completed tasks, failure costs, and intervention overhead. They therefore do not establish that the proposed destination metrics and licensing decisions select interventions with better final utility than a direct outcome-based evaluation. The weakness is not that all collateral damage must be eliminated, but that the additional adjudication criteria have not been adequately connected to consequential outcomes.

  2. The diagnostic decomposition is not developed into a corresponding corrective mechanism. Channel-conditioned activation steering and router gating provide an intervention apparatus, but the audit primarily accepts or declines fixed configurations rather than jointly optimizing error transitions, preservation, or task-level costs. An auditing contribution need not introduce a new correction algorithm; however, in the absence of such a mechanism, stronger evidence is needed that this particular decomposition yields actionable insight beyond a more detailed accounting of classification outcomes. The present results do not sufficiently establish that independent methodological contribution.

Questions for authors
  1. Can the audit be shown to change intervention selection in a way that improves held-out end-to-end task success or cost-sensitive utility compared with selection based on aggregate final outcomes? Such evidence would clarify its practical value beyond descriptive diagnostics.

  2. Can the destination and collateral information be used to choose or adapt the router, direction, or dose and demonstrably improve the correction–preservation trade-off against aggregate-gain-based tuning? Alternatively, what consequential failure does the audit reliably predict that direct task evaluation misses?

Reviewer_yGFSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper works on the topic of fixing pre-execution tool decisions with steering vectors. Specifically, it argues that aggregated metrics often hide new failures introduced by steering. These are identified as either 1. previously correct decisions becoming incorrect, or 2. previously incorrect decisions changing into a new but still incorrect decision. The paper proposes setting stricter standards on these hidden failures as a new evaluation procedure

Critical strengths
  • The paper identifies a real failure mode of current benchmarks
  • Their proposed evaluation includes scoring against a random baseline which i think is a very useful decision.
Critical weaknesses
  • While useful, i think the paper mainly points out best practices, making the unique contributions appear quite thin
  • While the threshold numbers are defended as pre-registered, the specific numbers are never justified
  • I personally found the prose and the nomenclature used hard to follow
Questions for authors

I would suggest showing examples of public results that break down when looking at the hidden failures that you identified, and maybe other benchmarks similar to this one?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.