acceptodds
Under review as a conference paper at ICLR 2027

Same Task Performance, Different Intervention Readouts: Cross-Run Variation under In-Context Rule Aliasing

Abstract

Causal edits can yield different readouts from models with nearly identical task behavior. We study 26.1M-parameter transformers on a synthetic assignment language where RECENCY and RARITY select the same answer on every training document. All 75 runs across 25 configurations reach accuracy at least 0.999, yet under an answer-preserving multiplicity edit, the sign fraction—the fraction of valid pairs with positive log-odds displacement—spans more than 0.3 across seeds in 13 cells, reaching 0.879; there a strong positive tail contrasts with mostly near-zero negatives. Replacing the probe batch moves this fraction by at most 0.034 in four extreme runs. These are differences in intervention sensitivity rather than answer policy: on generated documents where the rules disagree, rarity-answer rates are at most 0.003 in the main grid. Supervision that distinguishes the rules changes the readout. A RARITY-labeled arm supplies a one-sided trained control for positive displacement; in the RECENCY-labeled arm, observed dispersion is lower. The dispersion comparisons remain exploratory and are not adjusted for cell selection. Strong directional readouts occur before runs meet the retrieval diagnostics. Intervention readouts should be reported with behavioral tests, independent training runs, checkpoint, probe population and gating policy rather than treated as configuration-level mechanism labels.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, intervention sensitivity, in-context learning, underspecification, reproducibility

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LTgHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether models with nearly identical task performance exhibit consistent responses to the same input intervention. The authors construct a synthetic assignment task in which two candidate rules, RECENCY and RARITY, always produce identical answers on the training distribution. They then introduce an answer-preserving multiplicity inversion that changes the relative frequencies of competing values while keeping the correct answer unchanged. Intervention sensitivity is measured through changes in the log-odds ratio between the old and new values. The main finding is that similar task performance does not guarantee reproducible intervention responses. Importantly, the authors also show that such differences should not be directly interpreted as evidence of different answer-selection policies or internal mechanisms.

Critical strengths
  1. A controlled experimental setting for studying intervention reproducibility.
  2. Extensive empirical evidence and careful interpretation of intervention measurements. It remains unclear to what extent the observed cross-run variation generalizes to realistic language understanding or reasoning tasks, where competing cues are rarely perfectly equivalent.
Critical weaknesses
  1. Insufficient motivation and unclear positioning of the central research contribution. The paper does not clearly establish the broader research problem or the gap it aims to address.
  2. Limited generalizability and insufficient explanation of the observed intervention differences.
  3. The paper suffers from poor organization and unclear exposition, making it difficult to follow the central research question, understand the motivation, and distinguish the main contributions from the extensive experimental details.
Questions for authors

Please refer to the weaknesses above.

Reviewer_atU9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether models with nearly identical task accuracy give reproducible intervention readouts. It trains transformers on a synthetic assignment language in which RECENCY and RARITY agree on every training answer. Across 75 runs in 25 cells, accuracy is at least 0.999, but an answer-preserving multiplicity edit produces different log-odds shifts across runs.

Tests on rule disagreements show that these shifts do not identify the answer policy. The experiments examine probe sampling, training streams, checkpoints, schedules, model scale, and corpus controls. Rule-disambiguating supervision provides a positive control and exploratory dispersion comparisons. The paper recommends reporting run variation and behavioral checks alongside intervention results.

Critical strengths

The large run differences survive useful checks. Figure 1 shows variation at saturated accuracy, and Tables 5–6 show positive tails beyond small sign changes near zero. Changing probe batches moves the selected extreme runs by at most 0.034; changing training streams at fixed initialization still gives large variation.

The paper checks what the readout actually measures. Rarity-answer rates are at most 0.003 on generated disagreements, and 98.68% of edited pairs retain the correct argmax. The trained RARITY arm provides a positive control. Failed controls are also reported.

Critical weaknesses

The main result is variation in an out-of-distribution confidence response; its connection to rule aliasing remains unclear. The models still overwhelmingly choose RECENCY on disagreements. No head passes the original joint intervention criterion, and the geometry test fails to distinguish the intended direction from controls. The non-aliased comparison changes exposure to edited inputs as well as labels. These results leave open whether aliasing matters beyond ordinary between-run variation in off-distribution logits.

The broader evidence for dispersion is limited. The extended baseline includes the three seeds used to select the widest cell, and contrasts using only new seeds are smaller. The rendered-language and pretrained-model results sit near the sign-fraction ceiling, so they do not reproduce the dispersion result outside the original surface. An independent task or a replication away from the ceiling would strengthen the contribution.

Questions for authors

Can you compare rule-distinguishing labels while holding the input and exposure distribution fixed, using the same held-out probe population? This would help isolate the role of aliasing.

Training uses every non-padding token, but the rule equivalence concerns query answers. Does the result persist with answer-only loss, or does the auxiliary loss help explain the consistent RECENCY behavior?

Can you reproduce the dispersion in a second task or a rendered/pretrained setting away from the readout ceiling, with cells and seeds chosen before inspecting their range?

Reviewer_xte9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the observations that models trained with the same structure, task and training configuration can respond differently to the same intervention on evaluation inputs.

The authors experiments several settings like 1) data sampling, 2) random seeds, 3) training checkpoints, 4) supervision, and 5) model size to analyze this.

This results show large variation in intervention-induced log-odds changes across random seeds.

Overall, the paper demonstrates that even when models achieve nearly identical ACC on the same task, their probabilistic outputs to input interventions can vary across training runs.

Critical strengths
  • The paper empirically shows that models trained on the same task with the same structure and configuration, but different random seeds, can exhibit different log-odds changes under the same input intervention on evaluation datasets despite achieving nearly identical task ACC.
Critical weaknesses
  1. Underspecification (https://arxiv.org/abs/2011.03395) is well-known problem in ML, and the authors already cite this work. But, specifically, multiple models can achieve nearly identical task performance while exhibiting different behaviors under distribution shifts or interventions. This paper shows similar results in a specific synthetic task (retrieval like) setting, but I couldn't find any new insights from their analysis beyond what is already known.

  2. I also couldn't understand the practical significance of the observed log-odds variation. Why is such variation problematic in practice? In what settings would it matter, and what specifically needs to be addressed? I think the paper should explain these points more clearly.

  3. The authors also don't suggest any concrete solution to the problem mentioned in 2. Even the proposed disagreement-supervision experiments are described by the authors as exploratory evidence (In line 454-457), rather than as a validated solution.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.