acceptodds
Under review as a conference paper at ICLR 2027

Utility-Preserving Relabeling Reveals Semantic Sensitivity in Language-Model Decision Making

Abstract

A decision should not change when an action is given a different label but its payoff stays the same. We study whether language models satisfy this form of strategic invariance. We introduce CSS-Bench, a benchmark of Prisoner’s Dilemma, Chicken, and Stag Hunt games in which action labels are swapped while the underlying payoff structure remains fixed. Across nine open-weight models, relabeling changes decision accuracy substantially, with absolute gaps of up to 83.4 percentage points across model-vocabulary cells. The effect varies in both magnitude and direction across models and vocabularies, and is not restricted to semantically loaded labels. We then trace the effect inside the models. Label-related information becomes linearly decodable at an average normalized depth of 0.25, while activation patching identifies causal influence at a substantially later depth of 0.76 across 36 model-vocabulary cells. Thus, information about action labels can be represented well before it becomes causally relevant to the decision. Causal steering further changes decision behavior, but transfers substantially less reliably to unseen labels than the underlying representation. Together, these results show that language models can change their decisions when action labels are swapped without changing the underlying payoffs, and provide a controlled framework for studying how linguistic information influences decision-making.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, linear probing, activation steering, strategic reasoning, game-theoretic evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gdLXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

CSS-Bench tests whether LLM choices change when action labels are swapped while payoffs stay fixed. The paper finds label sensitivity across nine models and investigates it with probes, activation patching, and steering. Its main mechanistic claim is that label information becomes decodable early but affects decisions much later.

Critical strengths
  1. The paired design is useful. Both prompts use the same labels and tokens, making logit comparisons and activation interventions straightforward.

  2. The authors report several limiting results, including sign reversals across templates, poor steering transfer to unseen labels, and failed geometric localization. This helps readers see where the evidence stops.

Critical weaknesses
  1. My first concern is whether these models can already solve the payoff task. Averaging the two label assignments in Table 2 gives 43.0% accuracy, with 31 of 36 cells below 50%. There is no base accuracy with labels such as A/B, and the correct option in the Prisoner's Dilemma is always Slot B. Simply changing order moves Pythia-410M from 27.8% to 54.6% (Table 16). Label effects could therefore include strong position or word preferences; they do not by themselves establish semantic interference with payoff-based decisions. Neutral labels also have the largest average effect, which weakens the semantic framing. The word pools are tiny: moral has only 4/2 words (Table 4), reused across 3,000 pairs per cell. Instance-level significance does not establish generalization across words. Semantically empty labels, balanced positions, and per-word-pair results are needed to tell what the benchmark measures.

  2. The mechanism analysis leaves straightforward alternative explanations open. Both probes train only on canonical prompts, where the payoff-optimal slot and positive-word slot are identical (Section 6.1, Appendix H). Reading word position is enough to solve the probe task; this does not identify payoff information. Patching only replaces the final position, leaving label information elsewhere for later layers to reread, so late recovery need not be specific to the label mechanism. The 0.25 and 0.76 depths measure different things and do not establish a causal delay. Steering has a related issue: the correct counterfactual answer is always the negative word, so favoring that word class can raise accuracy. Unseen-label accuracy is only 44%. Decorrelation in probe training, multi-position and payoff-only patching controls, and steering on canonical prompts are needed to rule out these explanations.

Questions for authors
  1. What accuracy do the models reach with A/B-style labels and balanced correct-option positions? Please break this down by game and report effects by word pair. This would directly affect my reading of the semantic-sensitivity claim.

  2. What are the probes' absolute early-layer accuracies? Can they decode the payoff-optimal slot when trained on both canonical and counterfactual prompts?

  3. Does recovery stay late when label positions or multiple positions are patched layer by layer? Does changing only payoffs produce a similar curve?

  4. What happens when the same steering direction is applied to canonical prompts? Why does Qwen2.5-7B moral score 50.5%/48.3% in Table 2 but 3.2%/95.8% in Table 13? Is this due to the word split? Please also check the direction sign in Eq. 10.

Minor: check the sign conventions in Tables 2 and 15 and Appendix M. Some claims in Appendix A lack citations, and Evans et al. is missing from the references.

Reviewer_HmPWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CSS-Bench to test whether language models preserve their decisions when action labels are swapped while payoffs remain fixed. Across three matrix games, four label vocabularies, and nine open-weight models, the authors report substantial sensitivity, including to neutral labels. Linear probing, activation patching, and steering investigate the associated internal representations, with label information reportedly becoming decodable earlier than the layers identified through patching. Held-out-label evaluation and cross-game transfer further assess generalization, showing that steering transfers less reliably to unseen labels.

Critical strengths
  1. CSS-Bench constructs matched prompt pairs that preserve numerical payoffs and the same two label words while swapping their action assignments. This provides a clear experimental framework for assessing whether decisions remain consistent under payoff-preserving relabeling.
  2. The paper supplements behavioral evaluation with linear probing, activation patching, and steering. Examining both information decodability and intervention effects is a useful attempt to explain the observed sensitivity, giving the study potential explanatory value beyond reporting accuracy differences.
Critical weaknesses
  1. Prior work has already examined action-label effects in matrix games [1] and selection bias associated with output tokens [2]. CSS-Bench extends this line of work with richer label vocabularies and procedurally generated payoff instances, but the paper does not sufficiently establish what its behavioral findings add beyond these known effects. Consequently, the overall contribution depends substantially on whether the mechanistic analyses provide a reliable new explanation.
  2. The mechanistic evidence does not clearly separate payoff reasoning from label processing. Probes A and B have identical training targets because the optimal slot always carries the positive label in canonical prompts. Counterfactual failure can therefore reflect a label-position shortcut. Moreover, Eq. 9 measures recovery of a fixed token preference, although correct behavior requires changing the output token after relabeling. Such recovery can reflect output remapping without isolating the cause of inconsistent decisions, weakening the mechanistic interpretation of the probe–patching depth gap.

[1] Nathan Herr et al. Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Games. arXiv:2407.04467, 2024.

[2] Chujie Zheng et al. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR, 2024.

Questions for authors
  1. Are Probes A and B trained on identical canonical inputs and target labels? If so, what evidence in the current experiments distinguishes payoff decoding from a shortcut based on the positive label’s position?
  2. How does the interpretation of Eq. 9 account for the output-token change required by correct relabeling? Do the current analyses distinguish recovery associated with inconsistent action choices from recovery occurring on pairs where both original answers are correct?
Reviewer_YoHbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the formal framework of "Strategic Invariance" to investigate how Large Language Models (LLMs) operate as decision-makers. Using advanced mechanistic interpretability techniques—such as activation patching, probing, and residual stream manipulation—the authors analyse whether and how models preserve the utility of their internal representations under semantic transformations or relabelling. The work includes empirical experiments on models like Pythia-410M to map how these interventions affect the model's decision trajectories.

Critical strengths

The theoretical framework of strategic invariance is well-formalised and offers an innovative lens through which to study the decision-making processes of LLMs. The rigorous application of mechanistic interpretability tools (such as steering and residual stream analysis) to empirically validate hypotheses about semantic projections is a strong methodological contribution. Furthermore, the extensive appendix (which includes cross-topology transfer and probing experiments) demonstrates the depth of the investigation.

Critical weaknesses

The primary critical weakness concerns the scalability and ecological validity of the findings. Much of the strongest empirical evidence (such as patching recovery) appears anchored to relatively small-scale models (e.g., the Pythia-410M family). It remains unclear whether the invariance dynamics observed in the residual stream remain consistent in modern language models with tens of billions of parameters (e.g., Llama-3 or Qwen), where networks tend to exhibit greater polysemanticity and decision circuits are significantly more distributed.

Additionally, although utility-preserving relabelling is conceptually fascinating, mapping these concepts to real-world out-of-distribution decision tasks would require broader empirical validation beyond the synthetic or highly controlled datasets used in this experiment.

Questions for authors

Have you conducted (or do you plan to conduct) experiments to verify whether strategic invariance and the residual stream manipulation results scale predictably to larger decision-making models (e.g., at the 7B–70B scale)?

Regarding cross-topology transfer, could you clarify how much the results depend on specific tokenisation choices, and how the framework would handle more severe latent-space discrepancies between heterogeneous architectures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.