acceptodds
Under review as a conference paper at ICLR 2027

Localizing Behaviors from Generations, Not MCQA Options

Abstract

Task settings that elicit a single-token response from a Language Model are widely used in interpretability research to causally localize the circuits underlying the task. Yet, this formulation is poorly suited to complex behaviors such as factual recall, reasoning, summarization, and style transfer. Prior work therefore often studies these behaviors through multiple-choice question answering (MCQA) formulations, or leaves them unexplored altogether. In these constrained settings, a model is said to exhibit the target behavior when it predicts the aligned answer choice. We propose, and defend, an alternative: localizing behavioral circuits directly from free-form generative responses. Across five model families and four tasks with matched MCQA and generative variants, we find that circuits localized from free-form responses are more precise than their MCQA-derived counterparts. We employ mechanistic analyses to explain this difference: MCQA introduces intermediate computations that are absent during direct generation, and contains high-rank representations. Taken together, our results suggest that localizing circuits from free-form responses provides a more direct and effective basis for model control.

Primary area:
interpretability and explainable AI
Keywords:
localization, large language models, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ySbxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares attention heads localized through attribution patching on multiple-choice and free-form tasks, then evaluates their usefulness for activation steering. It reports that free-form localization improves free-form steering at small head budgets, while MCQA localization performs better under MCQA evaluation. Additional analyses examine head overlap, representation rank, steering-vector readouts, and residual-stream interventions.

Critical strengths
  1. The paper investigates an important limitation of MCQA-based circuit localization and motivates the need to study model behaviors under more realistic free-form generation settings.

  2. The evaluation covers four behaviors across five language models, with controlled comparisons between MCQA and free-form localization under different steering budgets and evaluation formats.

Critical weaknesses
  1. Limited support for the main claim. The empirical results largely reflect a format-matching effect: free-form localization performs better on free-form evaluation, while MCQA localization performs better on MCQA evaluation. The advantage also vanishes at larger head budgets. It is therefore unclear whether the proposed approach provides a fundamentally better localization method.

  2. Insufficient mechanistic evidence. The lower-dimensional representations and logit-lens results do not necessarily imply more precise or causally meaningful circuits. In particular, the relatively weak transfer observed in Figure 4 raises concerns about whether the proposed causal abstraction is adequately supported by the experiments.

  3. Questionable evaluation of steering effectiveness. The qualitative examples in Appendix H show that steering often introduces hallucinations or substantially changes the original response content. This raises concerns about whether the reported improvements reflect precise behavioral control rather than unintended changes in generation. More rigorous evaluations of content preservation and off-target effects are needed.

Questions for authors

Can the authors provide length-matched or supervision-matched attribution experiments to distinguish the effect of response format from the effect of the likelihood objective?

Reviewer_GkR4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether attention heads localized using multiple-choice question answering (MCQA) provide an appropriate basis for interpreting and controlling free-form LLM behavior. The authors use attribution patching to select heads separately from MCQA and free-form contrasts, then evaluate them through activation steering. Free-form localization is generally more effective on free-form evaluations at small head budgets, while MCQA evaluation favors MCQA localization. The paper also analyzes head overlap, residual-space dimensionality, aggregate logit-lens readouts, and residual-stream interchange interventions. These results document format-dependent localization, but I find the additional insight into the underlying behavioral mechanisms limited.

Critical strengths
  1. Clear reporting of limitations. The authors report the MCQA reversal, the occupation-swap preference for single-token localization, and hallucination or replacement of non-target content with both approaches. These findings clarify the approach's limits and implications for model control.
Critical weaknesses
  1. Limited methodological novelty and incremental contribution. The localization–steering pipeline, including localization from free-form completions, has already been explored (Sankaranarayanan et al.). The main addition is a controlled MCQA/free-form comparison rather than a new localization method. The different demands of answer selection and generation make format dependence unsurprising. The analyses provide limited explanation of the selected heads' causal roles, and limit evidence for selective model improvement. I find the methodological and explanatory advance insufficient for acceptance.

  2. Mechanistic interpretation needs stronger validation. Table 2 reports aggregate steering-direction readouts across selected heads, not individual-head readouts (§6.3). Selection and representativeness of the model–task examples remain unclear. The summed log-likelihood contrast may be driven by a few tokens rather than distributed behavioral evidence (§3.1); concentration alone is not invalidating, but incidental wording could dominate head rankings. Existing residual-stream interventions provide causal evidence without directly validating the attribution-based head rankings (§6.4).

Questions for authors
  1. Head-selection validation. On a manageable subset, how do attribution-based rankings compare with exact head-level activation-patching effects under the same contrastive objective and intervention sites? Does the small-budget free-form advantage survive, addressing dependence on the first-order approximation?

  2. Token contributions and readout coverage. What are the signed and absolute per-token contributions to free-form head attribution scores? If a few tokens dominate, do they reflect target behavior or incidental wording that changes the rankings? How were the Table 2 model–task pairs and analysis settings selected, and do the remaining pairs support the same interpretation?

Reviewer_Pgw6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether the response format used for localization changes which attention heads are found for a behavior. It compares localizing from multiple-choice (MCQA) responses with localizing from free-form generations, keeping the behavior, model, attribution method, steering method and evaluation fixed. Experiments cover four behaviors and five models.

Under FF evaluation, FF-localized heads steer better at small head budgets, and the gap disappears at 10%. Under MCQA evaluation the result reverses.

The authors explain the difference mechanistically. The two head sets barely overlap, FF heads write to a lower-dimensional subspace that reads out concept words rather than answer letters, and MCQA involves an extra chain from the request to the option position to the answer letter. Section 7 shows that the conclusion flips for a single-token cloze version of the bias task.

Critical strengths
  1. Clean controlled design. Only the response format changes, and the full cross of localization, steering vector and evaluation format is run on five models with random baselines. This directly tests whether MCQA is a good proxy, so the result is useful even though it turns out largely symmetric.

  2. Thorough mechanistic analysis. Head overlap, depth, subspace dimension, logit lens and causal interventions tell a consistent story. The letter chain in Section 6.4 and the prefill experiment in Appendix D are convincing, and the authors report a counterexample instead of hiding it.

Critical weaknesses

Weaknesses:

1. The claims go beyond the evidence.

  • "More precise" is never defined or measured. Localization quality is judged only by steering success, which tests controllability, not whether these heads compute the behavior.
  • Unit-normalized vectors with a shared α give different effective strengths across heads of different scale.
  • Each localization wins on its own format (FF +0.075 under FF evaluation, MCQA +0.067 under MCQA evaluation, k ≤ 0.1). The FF advantage is gone by k = 0.1, Figure 5 is mixed, and Section 7 reverses the result.
  • Each setting has 50 test items, α appears to be chosen on them, and there are no confidence intervals or significance tests.

The evidence supports "localize in the format you deploy in", not "free-form is better".

2. The mechanistic evidence does not isolate response format.

  • Section 7 changes format and contrast design at once, and the result flips.
  • d90 may reflect input differences, since MCQA prompts contain long option text. "An order of magnitude" does not match 65.2 vs. 102.6.
  • Logit-lens readouts at different depths are not comparable, and MCQA heads sit deeper. Table 2 shows only 7 of 19 pairs, and on bias the MCQA heads point to "she/her", not letters.
  • The FF causal evidence is weak: peak transfer is about 0.09.

Minor: the cross-format results from Section 4.3 are missing; there are "Section ??" references, inconsistent notation, and miscounts ("five model families", "five behaviors").

Questions:

  • As written, Eq. 1 decreases toward the target behavior, which conflicts with the text and with top-signed head selection. Is this a typo or the implementation?
  • Was α chosen on the test items? How stable are the main gaps across seeds or data splits?
  • Which inputs and positions are used for d90, and does the gap hold on identical inputs?
  • How were the 7 pairs in Table 2 chosen?
  • Does the ranking hold under a necessity test (ablating the selected heads), or with steering strength matched by activation norm?
  • Could you separate format from contrast design, e.g., with noisier FF contrasts or cloze-style localization on other tasks?
  • Could you repeat the token readout with a tuned lens or Jacobian lens?
Questions for authors

Please see the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.