acceptodds
Under review as a conference paper at ICLR 2027

Semantic State Atlas: State-Conditioned Coordinates for Activation Intervention

Abstract

Activation steering can alter model behavior without updating weights, but an edit calibrated at one hidden state need not retain the same effect at another state or representation stream. Local intervention directions can weaken, affect a different behavior, or alter unintended predictions as the anchor changes. We study this problem as semantic persistence. Semantic State Atlas represents an intervention with anchor-specific local coordinates and explicitly transfers those coordinates between charts. Each chart selects directions that respond to the requested target while limiting changes to held-out non-target predictions. Transition operators are trained so that a transported edit reproduces the source edit's measured target responses at the destination, with consistency imposed only across semantically equivalent routes. A denoising-based validity surrogate further regularizes edited states toward locally supported regions. On controlled semantic-binding tasks, Chart and Atlas perform almost identically at one hop (91.5% versus 91.6%), but the gap grows to 73.0% versus 85.1% at three hops and 53.2% versus 71.8% at six hops. A matched state-conditioned MLP reaches 76.7% and 59.3%, respectively, indicating that state conditioning alone does not account for the full gain. With paired stream maps fixed, mean cross-stream success rises from 63.8% to 67.5%. Single-step benchmark gains are smaller, while nested procedural edits and added inference cost remain limitations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Activation Intervention, Semantic Persistence, Representation Steering, Vision Language Models, Multimodal Representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3Ge6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to keep the effect of activation steering consistent across different hidden states in VLMs. Instead of using a fixed steering vector, the authors propose to build local coordinate systems for each state and learn transitions to transfer the intervention between them. They also introduce a denoising-based step to keep the edited activations valid. The experiments show that the method works better for longer multi-hop paths, while the improvements on standard benchmarks are relatively small.

Critical strengths

The paper presents an interesting idea of representing activation interventions using local coordinates tailored to each hidden state, rather than relying on a fixed global steering direction

Critical weaknesses

There are three primary weaknesses: (1) lacking justification for semantic persistence, (2) large inference-time complexity and (3) lack of justification for each component of the proposed method.

For the first point, it remains unclear why semantic persistence is necessary in practical VLM applications. In particular, under what realistic inference scenarios must an intervention be repeatedly transported across multiple anchors, rather than applying an independently constructed intervention at each state? The experiments demonstrate improvements on predefined multi-hop paths, but the connection between these paths and real-world inference scenarios is insufficiently established. Consequently, it is difficult to assess whether the proposed framework addresses a significant limitation of existing activation-steering methods.

The next point is a concern about additional computation during inference. For unseen anchors, it requires constructing local chart bases through activation retrieval and behavioral probe evaluations, computing state-dependent transitions through a neural network and a matrix solve, and performing iterative optimization of edited hidden states using a denoising-based validity objective. Moreover, lthough the paper reports inference latency, the efficiency comparison is primarily against independent local charts rather than popular activation-steering methods. Given the considerable computational overhead and relatively modest improvements on single-step benchmarks, it remains unclear whether the performance gains justify the additional inference cost.

Finally, the method combines several components whose necessity and interactions are not sufficiently motivated from a unified perspective. Although ablation studies are provided, a clearer explanation on how the way each components are designed that way is appreciated.

Questions for authors
  1. It would be helpful if the authors could provide a clearer step-by-step walkthrough of the extraction and inference phases, explaining the role of each component, why it is needed, and how much computational overhead it introduces, especially during inference.
  2. It would be nice to see comparisons with some widely used activation-steering methods to better understand the effectiveness and efficiency of Atlas. Also, could the authors elaborate a bit more on why semantic persistence is important in practice, perhaps with a concrete example where existing steering methods struggle, or mechanistic interpretation for it?
Reviewer_xFHtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of semantic persistence in activation steering, where an intervention calibrated at one hidden state may not retain the same effect at another state or representation stream. The paper proposes Semantic State Atlas, which represents interventions with anchor-specific local coordinates and explicitly transfers these coordinates between charts. Learned transition operators aim to preserve the source intervention's target effects while limiting changes to non-target predictions, with path consistency imposed across semantically equivalent routes and a denoising-based validity surrogate for edited states. Experiments on controlled semantic-binding tasks show substantial improvements in multi-hop persistence and modest gains in cross-stream success, while improvements on single-step benchmarks remain limited.

Critical strengths
  1. The problem formulation of the paper is well-motivated. The paper addresses an important but relatively underexplored issue in activation steering, which is preserving the semantic effect of an intervention across changing internal representations.

  2. The multi-hop experiments provide strong empirical support for the proposed approach. Atlas outperforms independent charts and the matched MLP baseline, especially on longer transition paths. The ablation results further support the proposed design choices.

Critical weaknesses
  1. The generalizability of the method beyond controlled settings remains unclear. The strongest results come from tasks with explicitly defined semantic intents and evaluation paths, while gains over independent charts on public VLM benchmarks remain small. It is unclear how much the proposed framework would help in practice.

  2. The multi-hop evaluation uses predefined anchor paths, which are not shown to correspond to actual autoregressive trajectories. It remains unclear whether the observed improvements would extend to natural multi-step reasoning during generation.

Questions for authors
  1. As the strongest results are obtained on controlled semantic-binding tasks with predefined semantic intents and target/non-target probes, to what extent does the proposed method rely on these task-specific specifications, and how would it handle new semantic intents not seen during training?
  2. How are the multi-hop paths constructed, given that the anchors do not necessarily correspond to actual inference steps? How well do these paths reflect representation changes during actual model inference?
  3. Although the matched MLP baseline uses the same task-level path-consistency supervision, it differs from Atlas in both its parameterization and the formulation of the consistency loss. Can the authors further disentangle the contributions of the explicit coordinate transport structure and the choice of consistency objective to the observed improvements?
Reviewer_57duOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies semantic persistence in Vision-Language Model (VLM) activation steering: keeping an intervention effective as internal states change across multi-hop reasoning or modality streams. The authors propose Semantic State Atlas, which constructs local coordinate charts around each anchor state (balancing target sensitivity against non-target leakage) and learns transition operators (G_{ab}) with path consistency to transport intervention coordinates between charts, alongside a denoising validity regularizer. While single-step gains are minimal, Atlas substantially improves multi-hop (3-to-6 hop) and cross-stream persistence over independent charts and a matched state-conditioned MLP.

Critical strengths

Framing multi-hop and cross-modal steering decay through local coordinate charts and learned transition operators is a clean, well-motivated conceptual idea.

The experimental controls are strong and well-designed—particularly the Matched MLP baseline (isolating the benefit of explicit coordinate transport from generic state conditioning) and component ablations.

The paper is very transparent about its limitations.

Critical weaknesses

Because of this multi-stage setup, constructing charts and running validity refinement at unseen anchors adds noticeable computational and latency overhead

Many of the individual building blocks (local subspace selection, denoising validity surrogates, and state-dependent steering) are adapted from existing literature, and combining them creates a very complex, heavy pipeline (requiring pre-trained stream denoisers, paired alignment maps, SVD + finite-difference probe evaluations per chart, and unrolled projected-gradient steps)

Questions for authors

Given that gains on standard public VLM benchmarks are very small, what are the main real-world VLM deployment scenarios where 3-to-6 hop activation transport across pre-defined anchor nodes is needed in practice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.