acceptodds
Under review as a conference paper at ICLR 2027

Answer Candidates Reshape Language Model Decisions: Layer-wise Dynamics and Causal State Interventions

Abstract

Multiple-choice benchmarks usually treat answer options as a passive interface for measuring what a language model knows. We show that answer candidates actively shape how models form answers. For each question, we construct paired inputs that preserve the question and correct answer while replacing only wrong options with unrelated alternatives from the same task. Across six Llama and Qwen models ranging from 3B to 14B parameters and fifteen tasks, we find that benchmark distractors delay (the top-ranked candidate) the last change in the model's favored option in 86 of 90 model-task settings, shorten the span over which the correct answer maintains a consistent lead (persistent correct-answer preference), and reduce accuracy by 20.2 percentage points. We then test whether these internal differences causally affect predictions. Specifically, we transfer the hidden state at the final token of the strongest wrong answer between paired inputs. This intervention improves accuracy by 8.78 percentage points and exceeds norm-matched random and cross-question controls. A separate 360-question panel further shows that reversing the transfer direction reverses the aggregate behavioral effect. These results show that answer candidates are not passive evaluation interfaces. Instead, they actively shape language model computation and causally influence final decisions. The code is provided in the supplementary material.

Primary area:
interpretability and explainable AI
Keywords:
LLM mechanistic interpretability, Layer-wise decision dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HPZROpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper investigates how answer-candidate composition reshapes layer-wise preference formation in decoder-only language models and whether candidate-dependent hidden states causally influence final predictions. It compares paired benchmark and unrelated-option inputs across six Llama and Qwen models, 15 MCQA tasks, followed by full-state activation-patching experiments on six tasks using panels of 36, 180, and 360 questions.

Critical strengths

A large sum of empirical evaluation. The paper evaluates six Llama and Qwen models across fifteen general and domain-specific tasks under different settings, resulting in a large collection of paired model-question records. The consistency of the reported patterns across models, tasks, and alternative readout settings provides broad empirical support for the main observations.

Critical weaknesses
  1. The distinction from prior activation-patching work should be discussed in greater depth. The paper applies full-state activation patching at a selected token and layer and measures the resulting behavioral shift. Similar state-transfer interventions and the sensitivity of MCQA predictions to answer options have been studied previously. The paper should discuss deeper such as which model components implement the observed effect rather focusing the phenomenon.

  2. Mechanistic expainasion beyond layer-wise observations should be discussed. The paper provides an empirical characterization of how answer preferences and intervention effects vary across layers. Prior information-flow studies have further localized such dynamics to specific attention heads, MLPs, and computational paths. A component- or path-level analysis would help explain why the identified layers influence the final decision.

Questions for authors

Please find the weakness above.

6 comments
AuthorsOpenReview ↗

Response for Reviewer HPZR [1/2]

Weakness 1: How does this work differ from prior activation-patching studies, and which components contribute to the observed effect?

Our work addresses a distinct question from prior activation-patching studies: how does changing only distractor content reshape a model's preference for the same correct answer? We construct paired inputs that keep the question stem, correct answer and its position, and option count fixed while replacing only incorrect options. We track the emergence of sustained correct-answer preference across layers and use the same strongest incorrect option as a fixed competitor in our interventions. Across 90 model–task settings, benchmark distractors delayed the last preference revision in 86 settings and shortened the persistent correct-preference span. Full-state transfer on an independent 180-question set improved mean accuracy by 8.78 percentage points. Its effect on the correct-versus-fixed-distractor score gap also exceeded both norm-matched random and cross-question controls.

(1) Identifying the state components contributing to the effect. We conducted component-removal experiments on Llama-3.2-3B and Qwen3-4B-Base using MMLU (80 questions) and CSQA (120 questions). We separately removed the state difference at the intervention-layer input, the attention output difference, and the MLP output difference, followed by downstream recomputation. Removing the input-state difference produced the largest mean attenuation in all four model–task settings. Removing the MLP output difference also significantly attenuated the transfer effect on CSQA for both models after Holm correction, whereas removing the attention output difference yielded no significant effect.

ModelTaskInput-state removal ↑Attention-output removal ↑MLP-output removal ↑
Llama-3.2-3BMMLU0.512 (0.581)0.067 (0.093)0.086 (0.110)
Llama-3.2-3BCSQA1.217 (1.214)0.040 (0.032)0.190 (0.122)
Qwen3-4B-BaseMMLU0.353 (0.252)0.009 (0.005)0.042 (0.030)
Qwen3-4B-BaseCSQA0.975 (0.649)−0.007 (0.003)0.068 (0.043)

(2) Tracing the upstream sources of state differences. We identified upstream sources on an independent discovery set and evaluated them on held-out questions by swapping module outputs between paired inputs, recomputing the state at the original intervention site, and repeating the transfer. Introducing unrelated-distractor outputs partially restored the transfer gain, whereas the reverse swap attenuated it. The identified sources were input embeddings and Block 0 MLP for Llama-3.2-3B, and Block 0 attention and MLP for Qwen3-4B-Base. All 16 restoration and blocking tests passed Holm correction.

ModelUpstream moduleTaskRestoration gain ↑Blocking loss ↑
Llama-3.2-3BInput embeddingsMMLU0.363 (0.348)0.534 (0.632)
Llama-3.2-3BInput embeddingsCSQA1.605 (1.868)1.294 (1.666)
Llama-3.2-3BBlock 0 MLPMMLU0.247 (0.274)0.447 (0.503)
Llama-3.2-3BBlock 0 MLPCSQA1.366 (1.521)1.000 (1.269)
Qwen3-4B-BaseBlock 0 attentionMMLU0.289 (0.264)0.330 (0.220)
Qwen3-4B-BaseBlock 0 attentionCSQA0.999 (0.812)0.867 (0.615)
Qwen3-4B-BaseBlock 0 MLPMMLU0.290 (0.258)0.310 (0.224)
Qwen3-4B-BaseBlock 0 MLPCSQA1.010 (0.815)0.866 (0.640)

Note: All entries report mean ΔG (sample variance), where G is the normalized score gap between the correct answer and the fixed distractor. In the first table, larger values indicate greater attenuation of the full-state transfer effect after component removal. In the second table, restoration measures the gain over the original-question baseline, while blocking measures the loss relative to full-state transfer. Variances are calculated across questions after averaging option rotations within each question.

Together, these results connect the distractor-dependent state differences to specific upstream modules and identify the residual and MLP contributions underlying their downstream effects. We will incorporate these analyses and clarify the distinction from prior activation-patching work in the revised manuscript.

AuthorsOpenReview ↗

Response for Reviewer HPZR [2/2]

Weakness 2: Which model components and information-flow pathways explain how layer-wise effects influence the final decision?

To move beyond layer-wise observations and identify specific computational components, we conducted a two-stage component screening followed by independent confirmation on Qwen3-4B-Base.

(1) Systematically identifying influential attention heads. Starting from the original intervention layer (Layer 9), we screened the attention and MLP outputs at the answer-prediction position across the subsequent 26 layers, evaluating 52 candidate components. The highest-ranked candidates were the attention output at Layer 24 and the MLP output at Layer 29. We then individually tested all 64 attention heads in these two layers using bidirectional restoration and blocking interventions. The screening score was defined as the smaller of the task-balanced mean restoration and blocking effects. Only heads with positive effects in both directions and output changes exceeding the numerical error bound were retained. Exactly three heads met these criteria:

Discovery rankAttention head (layer, head)RestorationBlockingScreening score
1(24, 30)0.056840.071860.05684
2(29, 0)0.000950.007980.00095
3(24, 0)0.002090.000380.00038

All layer and head indices are zero-based. We also performed head screening on Llama-3.2-3B, but no head met the criterion for confirmation.

(2) Confirming the causal contribution of the selected head group. We froze these three heads and jointly intervened on their outputs at the answer-prediction position using independent confirmation sets of 120 CSQA and 80 MMLU-General questions. Compared with norm-matched random interventions, the selected head group produced the following effects:

ModelTaskQuestionsRestoration advantage ↑Blocking advantage ↑
Qwen3-4B-BaseCSQA1200.06246 (0.00421)0.07543 (0.00801)
Qwen3-4B-BaseMMLU-General800.00808 (0.00154)0.01416 (0.00138)

Note: Values represent mean (sample variance) of normalized score-gap changes beyond norm-matched random interventions. Both restoration and blocking passed Holm correction on CSQA, while blocking alone passed on MMLU-General.

(3) Identifying the primary individual contributor and examining information flow. We further performed individual interventions on the three selected heads. On the CSQA confirmation set, Layer 24, Head 30 showed the clearest individual contribution: its restoration and blocking effects were 0.06357 and 0.07482, respectively, close to the corresponding raw means of the three-head joint intervention (0.06418 and 0.07650). In comparison, the individual effects of the other two heads were close to zero.

Attention headRestoration ↑Blocking ↑
Layer 24, Head 300.06357 (0.00415)0.07482 (0.00858)
Layer 29, Head 0−0.00015 (0.00046)−0.00152 (0.00066)
Layer 24, Head 00.00152 (0.00055)0.00046 (0.00080)

Furthermore, Key-Value (K/V) interventions showed that jointly exchanging K/V information across all context positions accessible to the answer-prediction token changed the score gap in both restoration and blocking directions. Together, these findings connect systematic component screening, independent head-level confirmation, individual head contributions, and contextual information-flow interventions, providing a more detailed mechanistic explanation of how distractor-dependent representations influence final answer preferences. We will incorporate these analyses into the revised manuscript.

AuthorsOpenReview ↗

Follow-up to Reviewer HPZR

Thank you for your careful review and constructive questions. We would greatly appreciate a brief reply on whether our rebuttal addresses your concerns.

We clarified how our study differs from prior activation-patching work, traced part of the distractor effect to early model components, and identified a specific attention head in Qwen3-4B-Base that influences answer preferences on CommonsenseQA.

Thank you again for your time and consideration.

Reviewer_ephGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies LLM behavior on multiple-choice tasks, looking at a variant where all the wrong answer choices have been replaced by "random" choices (from other questions). This should make the tasks substantially easier (as they indeed find), and they study some effects of this perturbation wrt to the internal LLM states. In particular they look at the final token position before producing the answer choice, and show that in the "unrelated" (perturbed) setting they answer label token has been locked in an earlier layer than in the "benchmark" (original) setting. They also show that transferring a state from a particular layer in from the last token position of the strongest distractor answer, can affect the accuracy in the expected way.

They conclude that "answer candidates actively shape how models form answers", although it's not so clear why that is a surprise, or how we can make use of that insight.

Critical strengths

Performs a moderately interesting perturbation of multiple-choice questions, with some (mostly intuitive) conclusions about internal states and causal effects from distractor hidden states. Clearly presented with copious amount of data, including worrying about the effect of how the hidden states are projected to a final output.

Critical weaknesses

Not so clear whether these insights are surprising in some way, or whether they could be utilized for some purpose, like improving our general understanding of transformer workings.

Questions for authors

Given the variation in your measurements across the different models and datasets (e.g., Fig 2), can we deduce something about the nature of particular models/datasets?

5 comments
AuthorsOpenReview ↗

Response for Reviewer ephG [1/2]

Weakness: What is surprising about these findings, and how can they advance our understanding of Transformer computation?

Thank you for raising this important question. The central and somewhat unexpected finding is that changing only incorrect options can substantially alter a model's internal answer-formation process even when its final answer remains correct. This reveals a distinction between final prediction accuracy and the underlying computational process, showing that accuracy alone does not fully capture how models respond to candidate context.

(1) Distractor content reshapes internal decision trajectories independently of final correctness. We further analyzed 28,143 paired model–question records for which both input versions were answered correctly. Even within this subset, benchmark distractors delayed the last preference revision by an average of 5.29 percentage points of model depth and shortened the persistent correct-preference span by 4.95 percentage points. This direction was observed in 85 of 90 model–task settings. Thus, distractors influence not only whether a model answers correctly but also how it internally arrives at the correct answer.

(2) Distractor-dependent states reshape competition among answer candidates. Among 900 model–question records with at least three options, state transfer reduced the predicted probability of the fixed incorrect competitor by an average of 9.78 percentage points, while increasing the correct answer's probability by 5.18 percentage points and the combined probability of the remaining incorrect options by 4.60 percentage points. This shows that distractor-dependent states redistribute probability across the candidate set rather than simply shifting preference from the selected distractor to the correct answer. Our additional component-removal experiments further showed that removing the state difference already present at the intervention-layer input produced the largest mean attenuation in all four model–task settings. Removing the MLP output difference also significantly reduced the transfer effect on CSQA for both models. These findings reveal how candidate-dependent information is carried across layers and influences subsequent answer competition.

(3) Layer-wise measurements provide diagnostic value beyond final accuracy. They identify questions whose internal decision processes are sensitive to distractor changes even when the final answers remain correct. This provides a new analytical tool for characterizing candidate sensitivity, auditing distractor robustness in MCQA benchmarks, and understanding how Transformers form their answers. We also explored early-exit and selective-review applications, but these experiments did not establish consistent accuracy gains. The demonstrated utility of our measurements is therefore their ability to diagnose internal decision dynamics beyond what final accuracy alone can reveal.

AuthorsOpenReview ↗

Response for Reviewer ephG [2/2]

Questions: What do the variations across models and datasets reveal about their sensitivity to distractor content?

We further analyzed the variations across models and tasks in Figure 2. Our results reveal substantial model- and task-dependent differences in distractor sensitivity, and identify the model's uncertainty on the original question as an informative predictor of these differences.

(1) Models and datasets exhibit different levels of sensitivity to distractor changes. These differences persist both in the full sample and among questions answered correctly under both input conditions. WinoGrande exhibits substantially larger delays in the last preference revision than MMLU High School Geography. Across models, Llama-3.1-8B shows a larger average delay than Qwen3-14B-Base. Within the Qwen3 series, the average delay decreases progressively from the 4B to the 8B and 14B checkpoints.

ComparisonModel or taskLast-revision delay: allLast-revision delay: both correctPersistent-span change: allPersistent-span change: both correct
ModelLlama-3.1-8B+9.23+8.71−15.73−8.62
ModelQwen3-4B-Base+4.75+4.02−9.40−2.32
ModelQwen3-8B-Base+4.55+3.58−8.44−2.53
ModelQwen3-14B-Base+3.14+2.76−5.73−1.91
TaskWinoGrande+11.18+12.18−20.28−11.64
TaskMMLU High School Geography+1.55+1.86−2.57−1.23
TaskMMLU-General+4.37+3.68−10.51−2.78
TaskCommonsenseQA+4.24+4.60−7.63−3.14

Note: Values represent mean differences between benchmark-distractor and unrelated-distractor conditions, in percentage points of model depth. Model averages are calculated across 15 tasks, and task averages across six models. The both-correct subset includes records answered correctly under both conditions. Negative persistent-span changes indicate a shortened span.

(2) Model uncertainty predicts the extent of distractor-induced changes in decision trajectories. We performed grouped cross-validation on 40,890 paired records, grouping by normalized question stem so that records associated with the same question remained in the same fold across models. Adding subject, option-count, and textual features to model and task features yielded only modest improvements. Incorporating the entropy of the original answer distribution further improved held-out prediction for both layer-wise measures.

Prediction targetModel and task only+ Subject and text features+ Original-answer entropy
Last revision depth0.02800.03000.0967
Persistent correct-preference span0.07320.07670.1612

Note: Values are held-out R² from grouped cross-validation.

Across all 90 model–task settings, higher entropy on the original input was consistently associated with later final preference revisions and shorter persistent correct-preference spans. This identifies model uncertainty as an informative predictor of distractor sensitivity and provides an inference-time signal for identifying questions whose internal decision trajectories are more susceptible to candidate changes.

(3) The relationship between model scale and distractor sensitivity is task-dependent. To separate model-scale effects from task differences, we compared Qwen3-4B, 8B, and 14B Base checkpoints using identical questions, distractor positions, and text versions. We measured how much the last preference revision shifted when an unrelated incorrect option was replaced with a relevant but still incorrect distractor.

TaskShared questions8B − 4B14B − 4B
MMLU-General4−2.78−1.82
CommonsenseQA3+0.28+0.44
ARC-Challenge5+0.28+0.89

Note: Values represent differences in the last-revision delay effect between relevant incorrect distractors and unrelated incorrect options, in percentage points of model depth.

Larger checkpoints showed lower distractor sensitivity on MMLU-General but slightly higher sensitivity on CSQA and ARC-Challenge. This indicates that the relationship between model scale and distractor sensitivity is task-dependent rather than uniformly monotonic. Overall, the variations in Figure 2 reveal that internal decision sensitivity reflects the interaction between model uncertainty and task-specific candidate context, providing a basis for characterizing decision dynamics across models and datasets.

AuthorsOpenReview ↗

Follow-up to Reviewer ephG

Thank you for your careful review and for asking what the differences across models and tasks reveal. We would greatly appreciate a brief reply on whether our rebuttal addresses your question.

Our analysis shows that distractors can delay when a model settles on the correct answer even when its final answer stays the same, and that the model’s uncertainty helps predict the size of this delay across models and tasks.

Thank you again for your time and consideration.

Reviewer_G9fjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes and experimentally validates the view that, when LLMs answer multiple-choice questions, answer candidates participate in the process of answer formation rather than merely serving as a passive interface. By replacing incorrect options with other unrelated options from the same task, the authors observe corresponding changes in model answering behavior across multiple models and tasks. They further use hidden-state intervention experiments to demonstrate that candidate-related context can causally influence model decisions.

Critical strengths
  1. The paper validates the authors' claim through a relatively comprehensive and systematic set of experiments. The authors control relevant variables and observe relatively consistent phenomena across multiple models and experimental tasks, providing support for the robustness of the results.
  2. Multiple intervention experiments on hidden states, conducted without changing the input text, provide fairly strong causal evidence and support the authors' conclusions.
Critical weaknesses
  1. The research motivation is insufficiently substantiated. The authors repeatedly state that "multiple-choice benchmarks usually treat answer options as a passive interface" and use this as an implicit premise motivating the study. However, they do not explain the precise meaning of this claim or provide references to substantiate it. For either humans or LLMs, the fact that answer options can influence judgments on multiple-choice questions hardly seems surprising. Consequently, the significance of the paper's main undertaking—experimentally refuting this assumption—is questionable.
  2. The paper's core contribution is insufficient. Prior work has already investigated how changing option content affects LLM predictions. The paper's contribution therefore rests primarily on its additional layer-wise observations and state-intervention results. However, the authors do not adequately explain what substantive new insights or understanding these results provide. Moreover, the authors neither propose novel experimental methods nor use their findings to pursue applications aimed at improving model accuracy on multiple-choice questions.
  3. The intervention experiments lack further mechanistic explanation. The interventions are implemented by transferring contextual states at the final token positions of answer candidates. However, the results remain compatible with other straightforward explanations. For example, the advantage of same-question paired transfer over cross-question controls may arise partly from a closer match in context. Additional position and layer-depth control experiments describe which intervention sites are effective but still do not adequately explain why those sites are effective.
Questions for authors
  1. Please clarify which evaluation practices or interpretive conventions are meant by the statement that "multiple-choice benchmarks usually treat answer options as a passive interface." How does this paper revise that understanding, and what valuable new insights does it offer?
  2. Could the authors further report changes in the relative scores or probabilities of all answer candidates before and after intervention, and clarify whether the intervention effect is primarily concentrated on the fixed wrong competitor or also involves other candidates?
7 comments
AuthorsOpenReview ↗

Response to Reviewer G9fj [1/2]

Weakness 1: What motivates this study beyond the known sensitivity of predictions to answer options?

Our motivation is to investigate how changing distractor content reshapes the internal process of answer formation when the question and correct answer remain unchanged. This question is motivated by the gap between final accuracy and the underlying decision process, which conventional evaluation does not capture. We will replace the phrase "passive interface" in the abstract and conclusion with this more precise research question.

To demonstrate the significance of this distinction, we analyzed 28,143 paired model–question observations for which both input versions produced the correct answer. Even within this subset, benchmark distractors delayed the last preference revision by an average of 5.29 percentage points of model depth and shortened the persistent correct-preference span by 4.95 percentage points. This pattern was observed in 85 of 90 model–task settings, showing that distractors alter internal decision trajectories even when final correctness is unchanged.

SampleLayer-wise measureUnrelated distractorsOriginal distractorsMean change (depth pp)
All 40,890 paired observationsLast revision depth72.14%77.49%+5.34
All 40,890 paired observationsPersistent correct-preference span37.13%26.68%−10.45
Both correct, 28,143 pairsLast revision depth69.44%74.73%+5.29
Both correct, 28,143 pairsPersistent correct-preference span41.71%36.76%−4.95

Note: Changes are calculated as original minus unrelated distractors using unrounded values. The full-panel results summarize Figure 2; the both-correct subset includes questions answered correctly under both conditions.

Weakness 2: What substantive new insights do the trajectory and intervention results provide?

Our contribution is a systematic characterization of how distractor-dependent representations evolve across layers and causally influence final answer preferences. The submitted paper showed that full-state transfer at the fixed competitor's boundary changes the final choice, with the aggregate behavioral effect reversing when the transfer direction is reversed. Combined with the both-correct analysis above and the candidate-level probability results in Question 2, these findings show that changing distractors affects not only prediction outcomes but also the internal trajectory and competitive relationships through which answers are formed.

These measurements provide a complementary diagnostic framework for understanding Transformer computation, distinguishing models that achieve the same accuracy but follow different internal decision trajectories, and assessing distractor robustness in MCQA benchmarks. We also explored offline early-exit and selective-review strategies under matched computational or review budgets, but did not establish consistent accuracy improvements. The demonstrated utility of our approach is therefore to reveal and characterize internal decision dynamics that conventional accuracy-based evaluation cannot capture.

AuthorsOpenReview ↗

Response to Reviewer G9fj [2/2]

Weakness 3: Could closer context matching explain the transfer effect, and why does the intervention site matter?

(1) The transfer effect depends on donor-state content beyond contextual matching. We compared an unrelated replacement with a meaning-preserving rephrase of the same incorrect option. Both donors shared the same question, edited option slot, recipient input, and matched update norm. Across two models and three tasks (40 shared questions per task), the unrelated replacement produced larger gains in the normalized correct-versus-competitor score gap in all six model–task settings, with mean advantages ranging from 0.166 to 1.037. Editing and patching the strongest distractor also produced larger gains than applying the corresponding operation to the second-strongest distractor in all six settings. Both sets of contrasts passed Holm correction.

TaskShared questionsUnrelated replacement over rephrase ↑Strongest over second-strongest distractor ↑
MMLU-General400.476 (0.247)0.859 (0.404)
ARC-Challenge400.592 (0.374)1.061 (1.019)
CommonsenseQA400.752 (0.438)1.039 (0.714)

Note: Values are mean (sample variance) of paired contrasts in the normalized correct-versus-fixed-competitor score gap. The two models are averaged within each shared question before calculating the sample variance across questions. Both comparisons use norm-matched updates. The strongest-versus-second-strongest comparison captures the combined influence of candidate rank and intervention site.

(2) Upstream component analyses explain why the selected boundary state carries an effective intervention signal. On independent MMLU-General and CSQA questions, removing the state difference already present at the intervention-layer input produced the largest average attenuation for both Llama-3.2-3B and Qwen3-4B-Base. Further upstream-source interventions showed that swapping selected early module outputs and recomputing the later boundary state changed the transfer effect. The implicated sources were the input embeddings and Block 0 MLP for Llama-3.2-3B, and Block 0 attention and MLP for Qwen3-4B-Base. All 16 restoration and blocking tests passed Holm correction. These findings connect the effectiveness of the selected intervention site to candidate-dependent representations formed in earlier computations.

Question 1: Which evaluation practices are meant by the term "passive interface"?

We refer specifically to the conventional practice of reporting accuracy on a fixed, provided candidate set, as in the original MMLU evaluation protocol (Hendrycks et al., 2021, Section 4.1). This does not imply that answer options cannot influence human or model judgments. Rather, accuracy on a fixed candidate set does not reveal whether predictions remain stable when distractor content changes or whether the same correct prediction is reached through different internal processes. Our study complements conventional accuracy evaluation by measuring paired distractor sensitivity and the associated internal decision trajectories. We will clarify this distinction and replace the phrase "passive interface" in the abstract and conclusion.

Question 2: How are the probabilities of all answer candidates redistributed after intervention?

The largest probability change occurs at the fixed strongest incorrect competitor, but the intervention also changes the probabilities assigned to the correct answer and the remaining incorrect options. The candidate-level results are summarized below.

CandidateBefore transferAfter transferChange (pp): mean (sample variance)
Correct answer55.06%60.24%+5.18 (80.22)
Fixed incorrect competitor21.39%11.61%−9.78 (165.96)
Other incorrect candidates, combined23.55%28.15%+4.60 (49.16)

Note: The analysis includes 900 model–question observations with at least three options from the 180-question forward-transfer panel. Probabilities are normalized over the presented options and averaged over option rotations within each question. Variances are calculated across model–question observations.

In score space, the fixed competitor's score relative to the mean score of the other incorrect candidates decreased by 0.829 logit units. In the separate bidirectional-transfer panel, reverse transfer increased the fixed competitor's probability by 6.04 percentage points on questions with at least three options. We have also prepared a candidate-level export containing before-and-after scores and probabilities for every option, including two-option questions, which will be included in the revised supplementary materials.

AuthorsOpenReview ↗

Follow-up to Reviewer G9fj

Thank you for your careful review and constructive questions. We would greatly appreciate a brief reply on whether our rebuttal addresses your concerns.

We showed that changing distractors can shift when the model settles on the correct answer even when its final choice stays the same, and that the intervention produces the largest probability change for the strongest incorrect option.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Brief Follow-up on Our Rebuttal

Dear Reviewers,

Thank you for the time you have devoted to our submission. As the discussion period draws to a close, we would be grateful if the reviewers could briefly let us know whether our responses address their main concerns.

If any questions remain, we would be happy to clarify them before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.