acceptodds
Under review as a conference paper at ICLR 2027

Represented But Not Composed: How Numerical Comparisons Enter Language-Model Decisions

Abstract

We show that a language model can represent a rule's numerical comparison without that representation causally controlling the verdict. We test this across four instruction-tuned models (Llama, Qwen, Gemma, Mistral) in two controlled rule-following settings. In a simple US Federal Aviation Administration (FAA) altitude rule, changing the stated ceiling strongly changes the verdict in the three models with a sufficiently strong FAA decision signal. In a more complex anti-structuring rule, where an amount–threshold comparison must combine with how the transaction is split, changing the threshold barely changes the verdict even when only one comparison changes. We contrast this numerical comparison with rule polarity (whether the rule permits or prohibits the action). This is not a representation failure: threshold-dependent information reaches the decision layer even when the gold answer is held fixed. In Llama, this representation is graded and causally movable, yet moving it across the threshold shifts the verdict by only 2.0–2.3% of the polarity-control effect; the identical intervention on the polarity decision direction serves as a positive control and flips the verdict completely. Across all four models, the late-layer components that drive the decision respond strongly to qualitative action structure but negligibly to the stated threshold. The same dissociation appears behaviorally: in Llama, telling the model that the full amount is below the threshold repairs 0 of 115 over-flagged cases, while telling it the resulting qualitative judgment that the split does not evade reporting repairs 115 of 115. Gemma matches this exactly; Qwen shows the same asymmetry less completely; Mistral shows partial repair. The numerical comparison is represented, graded, and causally movable, yet barely controls the verdict in this task: it is represented but not composed.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, causal interventions, numerical reasoning, compositional reasoning, representation analysis, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_XmNUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether numerical information represented inside a language model actually influences its rule-following decisions. In a controlled bank-reporting task, the correct verdict depends on both how a transaction is split and how its amounts compare with a stated reporting threshold. Across four instruction-tuned models, the authors find that threshold-dependent information reaches the decision position, while the examined decision-related components respond strongly to splitting and weakly to threshold changes. In Llama, they further identify a representation predictive of the amount–threshold relation: interventions that move its decoded value across the threshold change the verdict by only 2.0–2.3% of the effect of their permit/prohibit control. Behaviorally, supplying the correct numerical comparison often fails to repair errors, whereas supplying the resulting non-evasion judgment is much more effective. A simpler drone-altitude task shows that numerical thresholds can strongly influence decisions when a single comparison determines the answer.

Critical strengths

They test multiple layers, expand interventions beyond one direction, include verdict-matched contrasts, and replicate several findings across four models. They also report inconclusive experiments and differences between models.

Critical weaknesses

The test set is too narrow to establish a reliable conclusion and to support the work's broad claim. The authors could test more popular public datasets.

The writing, especially the introduction, is hard to follow due to many unclear terms, making it hard to understand the research question.

The threshold and amount remain at their own token positions, and later heads at the last token can re-read them. It means the patch isn’t a clean test of “the decision doesn’t read T.”

Questions for authors

How do your research question and observations generalize to more common cases?

1 comment
AuthorsOpenReview ↗

Response to XmNU

Thank you for distinguishing model-family replication from task generalization and for identifying the alternative-pathway problem.

Generalization. We completed a 512-prompt Llama screen that matches numerical values and qualitative facts across comparison-only and combined rules, neutral and fictional banking contexts, two phrasings, and both YES/NO mappings. With YES meaning FLAGGED, accuracy is 116/128 for comparison and 128/128 for the combined rule. Reversing the mapping yields 61/128 and 63/128. None of the prespecified strata establishes a reliable composition-specific failure. These results do not support generalizing the original banking failure to numerical/qualitative rule combination broadly. The fictional banking policy is simpler than the original regulatory task; it is a controlled comparison, not a public-dataset replication.

We have not added a public benchmark and will not claim coverage beyond the tested settings. Four model families provide replication within the historical prompts, whereas the new screen is Llama-only. Surface-identity repetitions also do not create independent transaction types: the original verdict-matched cohort contains three transaction types repeated over eight identities.

Alternative pathways and intervention validity. We agree that patching the final prompt-position residual leaves other positions and coordinates available for subsequent computation. A small effect from that patch cannot establish that the model never reads the threshold. Moreover, our new identity-controlled representation test retains decodable sign information but does not validate the proposed numerical intervention direction. We will withdraw the inference that this intervention demonstrates an independently computed comparison failing to affect the verdict.

The narrower component result remains: under the tested donor-output swaps, selected polarity-aligned components respond much more along the measured readout to action edits than to threshold edits. This is a statement about particular interventions, components and a readout—not exclusion of other routes or a claim that the readout captures all numerical influence. Final verdict effects were not measured in that component experiment and will not be implied.

We will simplify the introduction and state the behavioral task, intervention site, measured outcome and inference limits before presenting results. The title and conclusions will be narrowed accordingly. The follow-ups clarify the boundary of the finding, rather than establish the broader interpretation in the submitted headline.

Reviewer_yAHEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts to characterize a dissociation between numerical information that is represented in the residual stream and the causal effect that this information should have on a downstream decision. Authors demonstrate this with a threshold judgment task, and demonstrate that representations of threshold information do not change model decisions.

Critical strengths
  1. Investigating the dissociation between computing information from inputs (e.g., threshold - value) and integrating that information in a downstream decision circuit is an interesting idea. It is not obvious that information that requires computation to extract would be discarded, rather than integrated into a broader, causally efficacious circuit. This work has the potential to go after this type of question, but does not, as it only investigates threshold representations.

  2. Section 4.4 quickly alludes to a useful analysis method, where components are isolated based on their impact on a particular causal subspace, and input perturbations are used to characterize these components.

Critical weaknesses
  1. The core insight -- that information may be decodable but not causally implicated in model behavior -- is already extremely well established in the field, and has in many ways motivated the development of causal methods in mechanistic interpretability.

  2. A typical mechanistic interpretability study proceeds by first identifying a behavioral target to investigate, validating that models do or do not perform this behavior, and then rigorously investigate why their behavior looks the way it does. This work delays the basic behavioral evaluation until section 4.5. Merely running this experiment first would immediately reveal that models are insensitive to the numerical threshold, and only are reacting to a heuristic based on splitting. This obviates the rest of the mechanistic analysis: of course the model responses are not sensitive to representations of numerical thresholds. The model never acted like it was sensitive to these values! It might be interesting to mechanistically investigate what would make these models sensitive to numerical thresholds, but that is not the explanatory target investigated here.

  3. The writing is hard to follow, as the paper jumps directly to describing specific results/interpretations of the experimental results without a sufficient description of the task setting, or without sufficient framing to contextualize the broader contribution. Similarly, the task presentation is overly confusing. Presenting the concrete example from Appendix A in the main text would be substantially more clear.

Questions for authors

What do sections 4.2 and 4.3 add to our understanding of models, beyond what is revealed in section 4.5?

3 comments
AuthorsOpenReview ↗

Response to yAHE

We agree that decodability without causal influence is not itself a novel contribution, and that the behavioral target should precede the mechanistic analysis. The more specific question is which input changes affect the same intervention-defined components, rather than whether a decodable threshold direction changes an already threshold-insensitive verdict.

What the component experiment adds. We select candidate heads by polarity-aligned writes, include the late-band MLPs, and transplant each donor component's output into the same anchor context. The measured outcome is the change in a polarity projection at the fixed readout layer. It is not the final verdict logit. This controlled comparison finds a stronger response to replacing split deposits with a single deposit than to changing the stated threshold.

We recomputed effect magnitudes to avoid cancellation of positive and negative per-case effects. For each component, we average the absolute per-case effect, take the median over selected components, and normalize by the corresponding polarity statistic:

ModelAction edit / polarityThreshold edit / polarity
Llama77.2%3.1%
Qwen115.9%6.8%
Gemma104.5%3.5%
Mistral107.6%1.4%

These descriptive results use 30 scenarios per model. The ordering survives removing components at the readout layer and including all screened candidates without the additional causal-effect cutoff. At the scenario level, the magnitude of the median action effect exceeds that of the median threshold effect in all 30 scenarios of each model. This is not universal component-wise dominance: Qwen's layer-19 MLP is an exception, and we will correct that wording.

The added information is localization of differential input sensitivity to the same tested component outputs and readout. Behavioral threshold insensitivity alone does not specify which component interventions produce this contrast. Nevertheless, selection favors polarity-aligned components, the action edit changes both wording and numerical surface, and other numerical pathways remain possible. These results do not establish why the selectivity arises, an exclusively qualitative circuit, or complete absence of numerical influence.

What the original numerical analyses establish. Your objection to Sections 4.2–4.3 is substantive. A new paired identity control retains sign decodability but fails to validate the numerical intervention direction. We consequently cannot use those sections to establish an available, computed comparison that is subsequently ignored. We will remove that inference and distinguish exploratory representation results from independently validated causal evidence.

We also ran the suggested simpler behavioral comparison. Across 512 matched prompts, the combined numerical/qualitative rule is correct in 128/128 cases when YES denotes FLAGGED; the comparison-only task is correct in 116/128. Reversed answer mappings perform poorly on both. Thus this screen does not support a general composition failure. It bounds the interpretation to the original task context rather than extending the headline to synthetic rules.

We will reorganize the paper around a concrete behavioral example, the matched input contrasts, and the component-selectivity experiment you identified as useful. The contribution we can defend is this bounded characterization of input sensitivity, not a new demonstration that decodability and causal use can diverge.

Thank you for your reply.

This response is still very difficult to parse. More importantly, it is unclear to me why we would care about the input components that affect an intervention-defined component, if the model's behavior does not change either way.

Also, I identified the analysis method as useful, but the results of that analysis are undermined by the model's insensitivity to the core experimental manipulation in the first place.

I will be maintaining my score.

Thank you for clarifying. An unchanged answer does not tell us how the model’s internal components respond to different inputs. Our experiment tests that question: swapping the outputs of the same selected components changes the internal polarity score much more when the action changes than when the threshold changes. This identifies a difference in internal sensitivity, although it does not by itself explain why the final answer ignores the threshold. We will make that distinction clearer and simplify the presentation.

Reviewer_y692OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates whether language models use numerical information when applying a rule that combines numerical and qualitative conditions. The authors find that numerical information is decodable from internal activations, but interventions on extracted numerical directions have little effect on the verdict.

Critical strengths

None

Critical weaknesses
  1. The evidence does not clearly distinguish failure to combine a computed comparison from failure to compute the relevant comparison.

This is my main concern. The authors acknowledge that the signal could reflect the threshold, the amount and threshold separately, or a derived relation. Moreover, changing the extracted coordinate does not independently establish that the intervention changes the comparison used by the model’s own rule computation. The results therefore establish weak influence of the tested numerical features more clearly than they establish a failure to combine an already computed comparison with other conditions.

  1. The behavioral repair experiment provides limited evidence for the proposed mechanism.

Supplying “the amount is below the threshold” still requires the model to derive its consequence for reporting and evasion. Supplying “the split does not evade reporting” provides almost the requested answer. Their different effectiveness therefore does not, by itself, explain why the original inference fails. The false-conclusion control usefully argues against indiscriminate copying, but does not remove this difference or establish that repair operates through the mechanism identified by the activation experiments.

3 The presentation makes the central argument unnecessarily difficult to assess.

Unnecessary jargon, verbose phrasing, and repeated abstract descriptions obscure the task and experimental logic. This makes it harder to understand what each experiment establishes and whether the evidence supports the central claim. The regulatory framing further requires readers to unpack reporting, splitting, and evasion before understanding the underlying numerical test. The framing also adds complexity without clearly establishing its value. The paper studies numerical rule execution, but it is unclear why this requires regulatory contexts rather than simple numerical rule questions. These contexts make the task harder to follow and introduce differences in wording and domain knowledge that complicate interpretation.

Questions for authors
  1. What evidence distinguishes an available but unused comparison from numerical features that never form the comparison required by the rule ? Is intervention validity assessed independently of the coordinate being patched ?

  2. What evidence shows that the banking failure comes from combining conditions, rather than from the particular wording or context of the banking task ? Can the authors use a simple synthetic rule and progressively add conditions while keeping the vocabulary and numerical comparisons matched. This would make the task easier to understand and better isolate the source of failure.

  3. What mechanistic conclusion does the supplied-comparison versus supplied-conclusion experiment establish beyond showing that providing the conclusion makes the task easier ?

  4. Can the authors extend the causal analysis to answers produced after explicit reasoning, to test whether the same explanation holds beyond immediate verdicts.

3 comments
AuthorsOpenReview ↗

Response to y692

Thank you for identifying the distinction between a decodable numerical signal and a computed comparison available to the decision. We completed two follow-ups addressing representation validity and your proposed comparison-versus-combined-rule test. Their results require us to narrow the submitted interpretation.

Independent representation validation. We reran 126 numerical prompts alongside 126 matched counterparts with customer and bank identities held constant. The original identity assignment was correlated with numerical margin, so it cannot provide clean evidence for the numerical representation claim. In Llama at layer 22, the constant-identity ridge probe retains 95.2% sign accuracy, but within-side margin accuracy falls from 90.5% in the paired original arm to 69.0%. More importantly, the proposed intervention direction fails our prespecified validation requirements. Thus the follow-up establishes residual decodability, not a validated comparison variable that can be manipulated independently. We will remove the claim that the original intervention establishes failure to compose an already computed comparison. Reading back the overwritten coordinate is not independent evidence that its semantic content was changed.

Matched rule-combination screen. We tested 512 Llama prompts crossing numerical values, a binary split condition, comparison-only versus combined rules, neutral versus fictional banking contexts, two phrasings and two explicit answer mappings. The comparison rule is FLAGGED iff x>T; the combined rule is FLAGGED iff split AND x>T. Both tasks contain the same case facts, including the split fact. All four logical combinations are included.

Answer mappingComparison correctCombined rule correct
YES = FLAGGED116/128128/128
YES = CLEAR61/12863/128

The combined rule succeeds on every case under the first mapping, across both contexts and phrasings. Reversing the mapping substantially degrades both tasks. None of the eight context/phrasing/mapping strata meets the prespecified criterion for a reliable comparison-to-combination failure. We therefore cannot attribute the original banking errors to adding a qualitative condition in general. This simple fictional policy differs from the historical anti-structuring task; the screen bounds that explanation rather than reproducing all of the original task's demands. We will report the full crossed result, including the unfavorable mappings.

Behavioral repair and inference regime. The supplied-comparison versus supplied-conclusion contrast shows that different explicit guidance changes behavior differently. Providing the non-evasion conclusion supplies information close to the requested judgment. The false-conclusion control limits an indiscriminate-copying account, but does not establish a composition mechanism or connect the behavioral repair to the numerical intervention. We will present it as behavioral evidence only. We have not extended the causal experiment to post-reasoning verdicts. Intermediate-judgment outputs under other prompts do not establish that the immediate-verdict mechanism persists after reasoning.

The surviving mechanistic observation is narrower: selected polarity-aligned components show substantially greater readout sensitivity to action edits than to threshold edits in the tested banking prompts. We will revise the title, abstract and conclusions around that observation, explain the task with a concrete example first, and distinguish each measurement from its interpretation. We are not asking that the submitted “already computed but not composed” claim be accepted on the follow-up evidence.

Thank you for the response.

The follow-up results reinforce my main concerns, and the rebuttal remains difficult to read. I will maintain my score.

Thank you for considering the follow-up results. They clarify the distinction between the broader composition hypothesis and the narrower component-selectivity finding. We will make that distinction central to the paper and simplify the presentation so that each experiment’s contribution and limitations are clear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.