acceptodds
Under review as a conference paper at ICLR 2027

Amplification Without Repair: Causal Limits of Activation Steering in Abstract Reasoning

Abstract

Activation steering modifies hidden representations to control language-model behavior. Current approaches select interpretable features or learn directions from demonstrations, seeking improvements in reasoning. How these interventions translate into correction on new examples remains unclear. We investigate this transfer through a mechanistic study of text-serialized ARC-AGI-1, ARC-AGI-2, and symbolic Bongard-LOGO. Feature audits combine input controls, activation localization, and random-controlled interventions; support-adaptation experiments examine gradient alignment, decision boundaries, and support aggregation. Steering improves output validity and produces concept-specific held-out margin gains, while decision repair remains sparse. Fit/check gradient alignment differs between initially correct and incorrect concepts across development and fresh-combination confirmation. Aggregating support gradients amplifies both groups' responses, strengthening correct and incorrect preferences. These findings connect correction to directional transfer and boundary crossing, motivating interventions that target held-out errors and evaluations that track repairs and harms alongside margin gains.

Primary area:
interpretability and explainable AI
Keywords:
Activation steering, spatial reasoning, text-based spatial representations, causal interventions, mechanistic interpretability, task adaptation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_x1QmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the limitations of activation steering for abstract reasoning on ARC-AGI and Bongard-LOGO. Using SAE feature analysis, gradient alignment, and support-consensus interventions, the authors study whether steering directions derived from support examples transfer to unseen instances. They find that steering can improve output validity and prediction margins, but rarely corrects errors and sometimes reinforces incorrect predictions. These results suggest that improvements in these metrics do not necessarily translate into more reliable reasoning.

Critical strengths
  1. The experiments cover ARC-AGI-1, ARC-AGI-2, and Bongard-LOGO, with useful controls such as norm-matched random interventions and donor-concept controls. Together with the gradient-alignment analysis, these help clarify when steering works and where it falls short.
  2. The distinction between increasing prediction margins and correcting errors is an important point of the paper. I also appreciate the validation on 115 fresh concept combinations and the authors’ willingness to report that an initial hypothesis did not replicate.
Critical weaknesses
  1. The main gradient-alignment and support-consensus analyses are limited to Gemma-2-9B-it on Bongard-LOGO. Without broader validation, it is hard to tell whether the observed failure modes generalize to other models and tasks.
  2. The relationship between gradient alignment and correction remains unclear. Misalignment may help explain why interventions fail, but it could also reflect whether the model’s initial prediction was correct. The emphasis on pairwise ranking margins also makes it difficult to assess how much the interventions improve instance-level classification accuracy or reasoning.
Questions for authors
  1. Could the authors test whether the main gradient-alignment and support-consensus findings hold on other models, such as Qwen3 or DeepSeek-R1?
  2. Does gradient alignment still predict correction outcomes after controlling for baseline margins, task difficulty, and gradient magnitudes?
  3. Do the conclusions about error correction hold when evaluated using instance-level classification accuracy rather than pairwise ranking margins?
  4. How sensitive are these findings to the intervention layer, optimization objective, and intervention strength?
Reviewer_XhiVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper studies whether activation steering transfers from demonstrations to held-out reasoning errors. It combines an SAE-feature audit on serialized ARC tasks with support-adaptation experiments on symbolic Bongard-LOGO.
  • The analysis separates valid output, exact task success, score changes, and wrong-to-correct decisions. It includes matched-norm random directions, directions from other concepts, dose sweeps, and a frozen confirmation protocol on 115 fresh combinations of shared primitives.
  • The main finding is that favorable average effects can largely strengthen decisions the model already gets right. Support aggregation does not reliably repair held-out errors, and one development-set coherence finding fails to replicate.
Critical strengths
  • The evaluation thoughtfully separates control from improved real reasoning. The paper checks whether a feature changes formatting, task scores, or actual decisions, and compares it with random and other-concept directions at matched norms. This makes the evidence more informative than an activation correlation or an improved average score alone.
  • I like the confirmation study. The authors freeze the protocol, test fresh concept combinations, and report that the support-coherence result does not hold up. The finding that aggregation can reinforce existing errors is useful even without a new state-of-the-art method. Reporting repairs and harms separately is the right emphasis.
Critical weaknesses
  • The strongest negative result has limited model coverage. The full support-transfer, dose, and consensus analysis is on Gemma-2-9B-it. The other-model support pilots use only five concepts each, and the R1 pilot does repair its single initially wrong pair (Appendix F). The ARC experiments also start from very low exact-solution accuracy. I would like to see more results on diverse model set if possible.
  • The main diagnostic is still a step removed from ordinary task success. Much of the mechanistic analysis uses the ranking of a positive-negative pair rather than generated solutions or standard per-example classification. That is a reasonable diagnostic, but I would like a clearer link from this result to the broader claims about abstract reasoning. Prior task/function-vector work and SAE-Steering evaluate different interventions and task outcomes; the current study does not directly overturn those results.
Questions for authors
  • Does the correct-versus-wrong amplification pattern hold in a full second-model cohort, beyond the five-concept pilots? How should the successful R1 repair affect the scope of the conclusion?
  • Do the same intervention conditions show the pattern under ordinary query classification or generated-answer accuracy, rather than pair ranking alone?
  • How does the audited feature-selection/intervention setup compare with the published SAE-Steering procedure on a common task? If a direct comparison is not available, can the main claims be stated more explicitly as applying to the tested interventions?
Reviewer_gMQBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether activation steering corrects held-out reasoning errors. It audits SAE features on text-serialized ARC tasks, then examines support-trained activation vectors on symbolic Bongard-LOGO through gradient alignment, dose responses, and support aggregation. Steering improves output validity or margins more consistently than it repairs decisions. In the confirmation study, support-gradient consensus strengthens both correct and incorrect preferences, while the development-set relationship between support coherence and transfer does not replicate.

Critical strengths
  1. The evaluation distinguishes output validity, margin gains, and actual correction. For example, R1 feature 26683 raises valid-grid counts from 92 to 182 out of 419 ARC prompts, but produces only one exact solution. This helps establish what the intervention actually controls.
  2. The confirmation study strengthens the empirical contribution. Consensus yields an overall margin gain of approximately +0.078 but −0.141 among initially wrong concepts. The authors also report the failed coherence replication openly.
Critical weaknesses

The replicated alignment difference useful, but it does not fully establish what limits decision repair. Equations 3–5 describe how a direction changes the check margin, while it remains unclear whether the initially wrong pairs are correctable at the same site and vector norm with a better direction. The matched check-supervised diagnostic proposed in Appendix K would help distinguish poor direction construction from limited correction capacity. I also could not find the confirmation donor contrasts needed to assess whether the concept-specific advantage established in development carries over to this cohort.

Questions for authors
  1. How do matched check-supervised and support-derived directions compare in repair rates on initially wrong confirmation pairs?
  2. Does the alignment difference in Table 2 persist when comparing concepts with similar absolute baseline margins and task difficulty?
  3. On Gemma’s full confirmation cohort, do the learned and consensus directions retain an advantage over matched donor-concept directions, especially among initially wrong pairs?
Reviewer_qRYVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how activation steering translates into correction on new examples through a mechanistic study of text-serialized ARC-AGI-1, ARC-AGI-2, and symbolic Bongard-LOGO. It combines feature audits with support-adaptation experiments examining gradient alignment, decision boundaries, and support aggregation. The tested interventions improve output validity and produce concept-specific held-out margin gains, while decision repair remains sparse. Confirmation on 115 fresh concept combinations shows that aggregating support gradients can strengthen both correct and incorrect preferences.

Critical strengths
  1. The paper separates improvements in output validity and answer margins from actual error correction. Its analysis of gradient alignment and decision boundaries helps explain why steering can strengthen existing predictions without fixing mistakes.

  2. The study uses norm-matched random and donor-concept controls and checks its findings on 115 fresh concept combinations under a fixed protocol. Reporting both repairs and harms, including results that do not replicate, makes the analysis more credible.

Critical weaknesses
  1. The main mechanistic results come from Gemma-2-9B-it, while the Qwen3 and R1 support-adaptation pilots cover only five concepts each. It remains unclear whether the reported failure pattern holds more broadly across models.

  2. The paper lacks a diagnostic comparison with a direction aligned to the held-out gradient at the same layer and intervention norm. This would help distinguish poor transfer of support-derived directions from limitations of the chosen intervention site.

  3. The main mechanistic analysis uses Gemma's layer 20. The paper does not systematically test whether the findings persist across other layers or different choices of support examples and held-out pairs within each concept.

  4. No code is provided, making it harder to reproduce the feature-selection and intervention pipelines and verify the reported results.

Questions for authors
  1. Have you extended the Qwen3 and R1 experiments beyond the five-concept pilots, and do the main findings still hold?

  2. As a diagnostic, can a direction aligned with the held-out gradient repair errors at the same layer and intervention norm?

  3. Do the findings hold across different intervention layers and support/check splits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.