acceptodds
Under review as a conference paper at ICLR 2027

SHADOW ANCHORING: GROUNDING HUMAN- INTERPRETABLE CONCEPTS IN TRANSFORMERS VIA SPARSE AUTOENCODERS

Abstract

Interpretability methods for language models are mostly post hoc: probes and sparse autoencoders (SAEs) are fit after training to discover features the model happened to learn. We study a proactive alternative, concept anchoring, in which designated latent coordinates are trained to carry human-specified concepts, and ask whether it can be retrofitted into a pretrained transformer without degrading language modeling (an "interpretability tax"). We use continuous RGB color in CSS code as a machine-checkable testbed, since hex literals provide exact three-dimensional ground truth inside natural, syntactically constrained text. We introduce SAE shadow anchoring: supervision is applied to three latents of a frozen, pretrained SAE at an intermediate layer of Qwen2.5-Coder-0.5B, and gradients flow through the frozen SAE into LoRA adapters. Compared with anchoring the raw residual stream, shadow anchoring reaches higher color recovery with a smaller language-modeling cost relative to unconstrained LoRA fine-tuning. Clamping the anchored latents shifts generated color values in a dose-dependent way, keeps generated CSS syntactically valid, and introduces no spurious colors on general Python and JavaScript prompts. The representation transfers across six syntactic contexts, and Hewitt-Liang control tasks indicate the probes measure geometry rather than lookup shortcuts. Separately, digit-pair tokenization lets a 38M-parameter transformer trained from scratch generalize to unseen hex codes.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, sparse autoencoders, concept anchoring, activation steering, language models, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CEVjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Authors propose SAE shadow anchoring, a concept-anchoring method that freezes a pretrained SAE at an intermediate layer of Qwen2.5 model and backpropagates RGB color supervision through its encoder into LoRA adapters. Using CSS hex literals as exact ground truth, it finds Layer 8 as a semantic peak and achieves near-perfect color recovery while reducing the language-modeling tax relative to unconstrained fine-tuning.

Critical strengths

Using the method proposed authors managed to reduce the damage of language fluency (interpretability tax) as compared to unconstrained fine-tuning.

Using RGB color codes as their test is quite clever. It’s a good starting point to test their theory.

Critical weaknesses
  1. Authors only test their framework on Qwen2.5 model and one concept type and one domain. This is not enough to claim the generality of the framework.
  2. Do authors only measure the interpretability tax on held-out CSS data? If so, it would be good to measure it also on other datasets like general code benchmarks for example.
  3. I think it’s not very clear from the experiments whether SAEs really add anything here. Linear probing already recovers color with r = 0.94 and one could use probe directions as an anchoring baseline. So why do we need SAEs at all here?
Questions for authors

See Weaknesses

Reviewer_NLB9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a learning objective that encourages "anchoring" of some part of a hidden state (either LLM or SAE) to take the value of some ground truth concept vector. This is used in two ways:

  1. During pretraining, three elements of the hidden state on the final token of a hex string literal representing a colour are encouraged to take the values of the corresponding RGB values.

  2. For a pretrained model, an SAE is applied at some layer and the elements of its hidden state most correlated with RGB channels are selected. A lora adapter is trained with gradients back-propagated through the SAE encoder such that the selected SAE elements are encouraged to take the RGB values of the inputted hex strings.

This is evaluated on hex literals from a set of CSS files.

Critical strengths
  1. The central idea of the paper, encouraging a set of neurons in the hidden state of a model to encode concept information, is quite interesting and directly tackles the post-hoc shortcoming of methods like SAE. I think this method would be of some interest to the field and could lead to interesting further work.

  2. The experimental validation on the chosen task is considerable and thorough. While I have some misgivings about the particular task chosen (see next section), the authors comprehensively explore their method's performance within that task.

Critical weaknesses
  1. This paper is quite hard to follow. For instance, in the methods section phases 3 and 4 are mentioned, before phases 1 and 2 are ever discussed or explained, which is much later in the paper (the experimental set-up corresponding to phase 1 is never labelled as such). I think you could also make it clearer what it means for a coordinate to be anchored or non-anchored and what the intuition for desiring this is in the first place. The task that you are evaluating your method on, reading RGB values for input hex codes from these anchored hidden state elements is also not made clear in the experimental setup section.

  2. The paper relies heavily on a very simple task. The RGB prediction task is low dimensional, structured and non-polysemantic. I think this paper would be strengthened by investigating whether your method is useful for identifying and steering more complicated concepts like factuality or sentiment.

Questions for authors

How do you expect sparser density in the SAE, as would be the case in a TopKSAE for instance, would effect your method and results? Would you have greater difficulty selecting SAE hidden state elements to anchor?

What happens if you select random SAE latent elements and anchor them instead of selecting the 3 most correlated with the RGB channels?

Reviewer_DpE9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests whether continuous RGB values in code can be assigned to designated coordinates of a pretrained transformer. It probes Qwen2.5-Coder-0.5B for color information, trains a layer-8 sparse autoencoder (SAE), then freezes the SAE and optimizes LoRA adapters with language-modeling, three-latent anchor, and off-coordinate leakage losses. The main comparison is against unconstrained LoRA training and anchoring three raw residual-stream coordinates. On held-out CSS windows, SAE anchoring improves RGB readout (MAE 0.0305 versus 0.0589 for raw anchoring) and has a smaller top-1 accuracy decrement relative to unconstrained LoRA (0.10 versus 0.26 percentage points; Table 2). SAE-decoder interventions shift generated colors on ten CSS prompts, with large off-target channel shifts; separately, red-latent readout correlates with red-channel values across six code formats (pp. 6–8).

Critical strengths

The testbed permits exact supervision and quantitative readout. RGB channels can be checked directly in code, making both the anchoring loss and held-out representation results more concrete than a solely qualitative interpretability example (pp. 2–6, Tables 1–2).

The paper includes an unconstrained LoRA control and reports adverse effects. Table 2 separates domain-adaptation gains from the measured anchor penalty. The authors also disclose off-target channel movement under interventions (pp. 6–7), which helps assess the actual control achieved.

Critical weaknesses

Figure presentation needs improvement. E.g. raw expressions left in the figures make it look like no human review of this paper before submission. More models should be tested to show efficacy of the method proposed.

The principal SAE-versus-raw comparison confounds the SAE with coordinate selection and a different leakage penalty. The raw baseline anchors arbitrary residual dimensions 0–2, whereas the SAE baseline selects latents already correlated with RGB on a calibration split (pre-anchoring selected-latent probe R² 0.2673 versus 0.0022 for three random latents; Table 1). As the authors note on p. 6, a raw baseline anchored along probe-aligned directions is missing. Equation 2 also penalizes 893 unanchored raw residual dimensions in the raw arm but 7,165 unanchored ReLU SAE latents in the SAE arm. The same nominal penalty weight therefore imposes different constraints; the raw penalty's pressure on almost the entire residual vector could itself contribute to the larger language-modeling cost. Table 2 reads the base and raw-anchored models from arbitrary raw coordinates, but Phase 4 from already color-correlated SAE latents; a same-latent, unanchored readout control is missing. The 0.17 percentage-point in-domain accuracy advantage and MAE improvement do not isolate an SAE-specific benefit. Only one training seed is tested (p. 6), and the accuracy tax is measured only on held-out CSS windows. The generation experiments do not establish that anchoring enables selective control. The intervention adds frozen SAE-decoder directions to the residual stream; it is tested on the anchored Phase 4 model without a corresponding steering test on the unadapted or raw-anchored model. At the largest green steering scale, green increases by 0.303 while off-target blue increases by 0.361 and red by 0.243 (Fig. 3/Table 3), even though the intervention assigns negative coefficients to the off-target latents. The blue increase exceeds the target increase; the paper also reports a red-clamped output with R = 0.021 rather than a saturated red (p. 6). The reported steering means across ten color prompts have no per-prompt spread or uncertainty. The ten-prompt general-code check assesses syntax and color intrusion, but lacks a correctness metric; its intervention fires once at the final prompt token rather than at a color-literal position (p. 7). One displayed quicksort completion changes the base case from length ≤1 to ≤2, which would return an unsorted two-element input (Table 6, p. 13). The CAA baseline's actual injected residual norms and scale tuning are not reported. Table 4 tests red-channel readout across formats without unadapted or raw-anchored controls, so it does not attribute that transfer to anchoring. These limits matter because steering is a central claimed payoff of creating the anchor.

Questions for authors

5.2.1: How was the layer search done exactly? How was "six sampled layers" sampled?

Do the same SAE-decoder steering vectors produce similar effects on the unadapted or raw-anchored model? For each Red, Green, and Blue clamp, please report target-minus-off-target channel changes and the actual injected residual norms for both SAE steering.

Under the same LoRA budget, with raw residuals anchored along color-aligned linear-probe directions and the leakage penalty matched or ablated in both arms, do the RGB MAE and language-modeling advantages persist across independent training seeds? Please include a same-latent readout of the unanchored model and a general-code language-modeling metric alongside the held-out CSS metric.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.