acceptodds
Under review as a conference paper at ICLR 2027

Before The Question Ends: Provisional Answer Formation In Language Models

Abstract

A language model may form an answer preference before a question is complete. Whether this early preference causally influences its final decision after question completion remains unclear. In binary Yes/No question answering, we introduce the preference score (PS) as a behavioral measure of provisional answer preference. Across four models, final-answer accuracy is lower when the correct answer conflicts with the early preference than when it aligns with that preference. To examine the causal role of provisional answer preference, we intervene on question-prefix representations and find that they carry this preference and influence final-answer computation. At the circuit level, question-prefix circuits selected solely for their contribution to final-answer computation also carry provisional preference. Steering against the early preference and suppressing these circuits both make final decisions less likely to follow the initially favored answer. Our findings provide causal evidence that preferences formed before question completion bias final decisions even after the complete question is available.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Large Language Models, Autoregressive Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xgn6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper looks at LLMs answering Yes/No questions, investigating to what extent the models have an early preference (while reading the first part of the question) for an answer, and how such early preference correlates with final answer and accuracy. They measure this answer preference score (PS) by aborting the question part way and looking at the Yes/No logits following an answer prompt. In the cleanest setting, for NP-BoolQ with pairs of questions sharing at least 80% common prefix, but opposite answers, they show a substantial gap in accuracy between gold answers that aligns with that early preference vs the unaligned ones (e.g., 96.1% vs 82.4% for Qwen3-8B).

The paper then shows a correlation between swapping certain hidden states in the question text (given different paraphrased passages in CONDAQA) and their preference scores as well as final answer - using this to conclude (?) a causal relationship between such provisional answer preference (encoded in hidden states) and final answer. There are further experiments learning an activation steering vector indicating the answer preference, and showing the expected effect of applying this vector to early hidden state translating to final answer shifts. Finally, they do circuit analysis using position-aware edge attribution patching, arriving at similar conclusions about the causality of early answer preference.

Critical strengths

Provides interesting, direct evidence for the intuitive notion that LLMs are biased by answer preferences that get formed while reading the question (and background passage).

Considers a reasonably diverse set of models (albeit of limited size) and datasets (limited by the requirement to naturally involve some contrastive cases), including complementary approaches in support of hypothesis.

Critical weaknesses

The activation swapping in Section 5.1 seems to establish correlation more than causation, although it feels like the later sections makes a stronger case.

It is not clear what the takeaways would be for future model training or usage, in terms of overcoming such biases.

Questions for authors

Did you measure the total probability mass on the provisional answers? I imagine if the prefix is sufficiently ambiguous, like "Is it legal to? Please answer yes or no.", models might want to push back rather than following the yes/no instruction. It might still be directionally valid of course, as indicated in your Fig 2, but could be interesting to see to what extent this is what the model would naturally output (probably varies a lot between models).

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the thoughtful questions which helped strengthen our analysis.

Weakness 1: Causal interpretation of activation swapping (Section 5.1)

We thank the reviewer for encouraging us to clarify what activation swapping establishes and how it contributes to the overall causal argument. To highlight differences in provisional preference as clearly as possible, we paired inputs with the same question and GT answer, using meaning-preserving passage paraphrases to hold answer-relevant semantic content fixed.

Activation swapping shows that question-prefix hidden states carry provisional answer preference and causally influence both provisional PS and the final-answer margin. The changes in both provisional preference score and the final-answer margin correlate with the pre-swap preference difference between the paired inputs. These correlations connect the causal effects of the representations to provisional preference.

The swapping results also provide the basis for the steering analysis in Section 5.2. We select a layer with strong preference transfer and fit a steering vector at that layer. Steering shifts both provisional preference scores and final-answer margins, providing a more targeted test of the causal role of provisional preference.

We will revise the last paragraph of Section 5.1 to clarify what activation swapping establishes:

Activation swapping therefore shows that question-prefix hidden states carry provisional answer preference and causally influence both provisional PS and the final-answer margin. The pre-swap provisional preference difference between the two inputs correlates with the changes in both provisional preference and the final-answer margin caused by swapping. These correlations connect the causal effects of the representations to provisional preference.

Weakness 2: Takeaways for model training and usage

We thank the reviewer for encouraging us to clarify the practical implications of our findings. Our results highlight the importance of revising an early answer preference when later question words require the opposite answer. The lower accuracy on conflicting questions identifies this as a capability that current models struggle with.

These findings highlight the need for datasets that explicitly target preference revision. Shared-prefix pairs with opposite correct answers, as in our NP-BoolQ shared-prefix setting, provide a suitable format. A model cannot answer both questions correctly by following the same prefix preference. Such datasets could provide training supervision that encourages models to use later question information to revise an initial preference.

For usage, these pairs can serve as targeted evaluations of models’ ability to handle answer-changing completions. High accuracy on aligned questions can keep overall accuracy high even when accuracy on conflicting questions is much lower. Reporting aligned and conflicting accuracy separately helps users identify which models are better at revising an incorrect initial preference.. Future work could also examine whether provisional preferences bias final decisions in vision-language models. Shared-prefix pairs with a fixed image could extend this dataset design to multimodal question answering.

Question 1: Probability mass on Yes and No at the provisional readout

We thank the reviewer for highlighting the importance of checking whether Yes/No outputs remain likely for incomplete questions. Table R4 reports the mean total probability mass on the Yes and No token sets used for PS (Appendix A.5).

Model0%20%40%60%80%100%
Qwen3-8B1.0001.0001.0001.0001.0001.000
Llama-3.1-8B0.9970.9570.9880.9961.0001.000
Gemma-3-12B1.0001.0001.0001.0001.0001.000
Phi-40.1220.7880.9420.9940.9991.000

Table R4: Mean probability mass on the Yes/No token sets. Results use the 1,048 CONDAQA validation questions. Columns indicate word-based question progress; 0% includes the passage but no question words.

At the 20% and 40% checkpoints, where prefixes may still be highly ambiguous, the first three models assign at least 0.95 mean probability mass to Yes/No. Phi-4 assigns 0.788 and 0.942, respectively, so Yes/No still receives most of its probability mass on average. The clearest exception is Phi-4 at 0%, with a mean mass of 0.122. These results indicate that Yes/No outputs generally receive substantial probability mass under our prompt even when much of the question is still missing.

Thank you for the careful responses! My overall assessment of the work stays the same.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful reading and follow-up. Your questions helped us clarify how activation swapping contributes to the causal argument, examine whether Yes/No outputs remain likely for incomplete questions, and articulate the implications for evaluating models’ ability to revise early preferences. We appreciate your constructive feedback and the opportunity to make the evidence and its implications clearer.

Reviewer_u9CqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether large language models develop provisional answer preferences before a question is fully presented, and how these early preferences influence final predictions. The authors examine this question through behavioral analysis, shared-prefix comparisons, hidden-state interventions, and circuit-level analysis. The experimental results suggest that provisional answer preferences are associated with final decisions, and that question-prefix representations can influence final-answer computation.

Critical strengths
  1. An interesting research question with a degree of novelty. The paper extends the study of answer formation in language models to the stage before the question is fully presented. Investigating provisional answer preferences and their influence on final decisions provides a meaningful perspective on understanding the decision-making process of language models.
  2. A systematic methodology with complementary experimental designs. The authors employ shared-prefix comparisons, activation swapping, and activation steering to examine the phenomenon from different perspectives. These experiments investigate the relationship between provisional preferences and final answers, the information carried by question-prefix representations, and the effects of interventions on final predictions.
Critical weaknesses
  1. The circuit-level analysis does not convincingly establish a mechanism specific to provisional answer preferences. In Section 6, the authors attempt to explain the internal mechanisms of provisional answer preferences through circuit analysis. However, Table 3 shows that partial or complete suppression of the selected circuits reduces the aligned–conflicting accuracy gap while also substantially decreasing accuracy on aligned examples. A reduction in the accuracy gap under circuit suppression is therefore insufficient, by itself, to establish that the selected circuits specifically mediate provisional answer preferences, rather than supporting general answer computation. Without stronger evidence of circuit specificity, the experiments in Section 6 provide limited support for the paper's claimed mechanistic explanation.
  2. The central causal interpretation is not sufficiently supported by the current interventions. In Sections 5.1 and 5.2, the authors investigate the influence of provisional answer preferences by swapping hidden states at question-prefix positions and applying steering along directions associated with the Preference Score (PS). While these experiments demonstrate that interventions on prefix representations can influence final predictions, they do not adequately isolate provisional answer preference as the specific causal factor. Swapping hidden states may simultaneously modify other semantic or contextual information, while steering along a PS-related direction may also produce more general shifts in answer preference. Consequently, the observed effects do not conclusively establish that provisional answer preferences themselves are responsible for the changes in final decisions. Since this causal relationship constitutes a central contribution of the paper, the lack of sufficient causal isolation substantially weakens the main claim.
Questions for authors
  1. Regarding circuit specificity: What evidence distinguishes the identified circuits from more general answer-computation pathways? Could the authors provide comparisons with size-matched random circuits or other high-attribution circuits to establish whether the observed effects are specific to provisional answer preferences?
  2. Regarding causal interpretation: Could the authors provide additional analyses to disentangle the information associated with early PS from other semantic or contextual information encoded in prefix representations, thereby providing stronger support for the causal interpretation of provisional answer preferences?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the insightful questions on specificity and constructive review.

Weakness 1 and Question 1: Circuit specificity

We thank the reviewer for suggesting control comparisons that help clarify the functional role of the question-prefix circuit within answer computation. We have added these comparisons to Appendix D.3 (Table 13).

The question-prefix circuit C(0.3)C(0.3) is selected by attribution to the final-answer margin, with destinations restricted to the first 80% of the question. Our analysis characterizes how this part of the answer pathway contributes to provisional preference.

Control comparisons. On the 854 NP-BoolQ shared-prefix pairs, we suppressed C(0.3)C(0.3) and two size-matched controls using the same procedure, ae←γaea_e \leftarrow \gamma a_e. Each circuit contains 20,400 connections for Qwen3-8B and 15,057 for Llama-3.1-8B.

  • Uniform random: connections sampled uniformly from all candidate connections, using three random seeds.
  • Question-suffix: the highest-ranked connections under the same attribution ranking whose destinations lie in the last 20% of the question or at the question mark. This provides another high-attribution part of the answer pathway operating at later question positions.
ModelSuppressed circuitγAlignedConflictingOverallGapΔGap [95% CI]
Qwen3-8BNone–96.182.489.313.7–
Question-prefix C(0.3)C(0.3)0.590.583.386.97.2−6.5 [−11.0, −2.0]
058.585.972.2−27.4−41.1 [−49.4, −33.2]
Uniform random (×3)0.596.482.089.214.5+0.7 [−0.4, +2.0]
096.281.388.714.8+1.1 [−1.0, +3.8]
Question-suffix0.592.164.878.527.3+13.6 [+7.9, +19.3]
085.316.150.769.3+55.6 [+47.8, +62.5]
Llama-3.1-8BNone–95.075.185.019.9–
Question-prefix C(0.3)C(0.3)0.590.079.684.810.4−9.5 [−15.2, −4.7]
081.482.081.7−0.6−20.5 [−27.4, −14.0]
Uniform random (×3)0.594.774.484.520.3+0.4 [−1.0, +1.7]
093.973.183.520.8+0.9 [−1.0, +2.6]
Question-suffix0.586.055.070.531.1+11.1 [+4.3, +17.3]
087.314.150.773.3+53.3 [+44.2, +61.8]

Table 13: Circuit suppression with control circuits on NP-BoolQ shared-prefix pairs. Accuracies are percentages. Gap is aligned minus conflicting accuracy, and ΔGap is its change from no suppression, both in percentage points. Overall is accuracy across all questions. Uniform random values are means over three seeds. Brackets show 95% passage-cluster bootstrap CIs.

The key result is the improvement on conflicting questions. Complete suppression of the question-prefix circuit C(0.3)C(0.3) improves conflicting accuracy from 82.4% to 85.9% for Qwen3-8B and from 75.1% to 82.0% for Llama-3.1-8B. Size-matched random suppression leaves the gap nearly unchanged. Complete question-suffix suppression instead reduces conflicting accuracy to 16.1% and 14.1% and widens the gap, while causing larger overall accuracy losses.

Although aligned accuracy can decline when suppression weakens a preference that favors the correct answer, the more informative evidence comes from conflicting questions. Here, the provisional preference favors the incorrect answer, so the observed accuracy gain is consistent with reducing its influence on the final decision.

The contrasting effects of prefix and suffix suppression support distinct contributions within the answer pathway, with prefix computations carrying provisional preference and later question computations helping revise it.

AuthorsOpenReview ↗

Official Comment by Authors

Weakness 2 and Question 2: Causal specificity of provisional preference

We thank the reviewer for suggesting analyses that distinguish information associated with provisional preference from other information in question-prefix representations. Below, we compare swapping only the vPSv_{\mathrm{PS}} component with swapping components along other directions capturing variation between the paired inputs. The next comment examines whether the effects could instead reflect a general Yes/No bias.

Q2: Separating provisional preference from other information in the prefix states

Activation swapping in Section 5.1 transfers all differences between the paired prefix states. Here, we examine the contribution of the component associated with provisional preference by swapping along one direction at a time.

We compare the fitted vPSv_{\mathrm{PS}} direction with four control directions, PC1–PC4. These controls capture large differences between the paired representations outside vPSv_{\mathrm{PS}}. To construct them, we collect hidden-state differences at each question-prefix position in the training pairs and remove their component along vPSv_{\mathrm{PS}}. We then obtain the four leading principal components without mean-centering. They capture the largest remaining squared differences along mutually orthogonal directions.

For each direction, a swap A←BA \leftarrow B replaces only that component of AA's prefix states with the corresponding component from BB. All other components remain unchanged. We evaluate on CONDAQA validation and test paraphrase pairs, performing swaps in both directions at the layer selected for steering.

Score shift. We measure how a swap A←BA \leftarrow B along a single direction changes AA's provisional PS and final-answer margin relative to their values without swapping. For both measurements, a change counts as positive when it follows the direction from AA's original provisional PS to BB's, and negative when it goes in the opposite direction. We report the mean signed change across swaps in PS units, equivalent to averaging sign(δ(s))Δt(s)\mathrm{sign}(\delta(s))\Delta_t(s) using the definitions in Eq. (2).

Energy. Energy measures the size of a one-direction replacement relative to a full swap. For each swap, we divide the squared hidden-state difference along the chosen direction by the total squared hidden-state difference, summing both quantities over the prefix positions. We report the median fraction across swaps as a percentage. For example, 1% energy means that the chosen direction accounts for 1% of the total squared difference between the paired prefix states for a typical pair.

Swapped directionEnergy (%)Shift in PSt⋆\mathrm{PS}_{t^\star}Shift in PST\mathrm{PS}_T
Full swap (reference)100+1.131 [+1.034, +1.236]+0.850 [+0.761, +0.945]
vPSv_{\mathrm{PS}}0.15+0.185 [+0.162, +0.207]+0.136 [+0.115, +0.158]
PC10.62+0.051 [+0.039, +0.063]+0.024 [+0.013, +0.037]
PC20.87+0.013 [+0.004, +0.021]−0.004 [−0.013, +0.005]
PC30.31+0.028 [+0.018, +0.039]+0.013 [+0.004, +0.024]
PC40.60+0.037 [+0.027, +0.047]+0.005 [−0.005, +0.015]

Table 10: Single-direction swaps in Qwen3-8B. Brackets show 95% passage-cluster bootstrap CIs.

Table 10 shows that the component along vPSv_{\mathrm{PS}} carries less energy than any of PC1–PC4, yet produces larger mean shifts in both provisional PS and the final-answer margin. For Qwen3-8B, the shifts are 3.6 and 5.7 times those of PC1 for PSt⋆\mathrm{PS}_{t^\star} and PST\mathrm{PS}_T, respectively. In this model, swapping this component alone reproduces about 16% of the full-swap shift, whereas two of the four principal directions do not detectably shift the final-answer margin.

We fitted vPSv_{\mathrm{PS}} from provisional PS differences within meaning-preserving paraphrase pairs that share the same question and GT answer. This controlled setting was designed to highlight provisional-preference differences while holding answer-relevant semantic content fixed. Learning from these controlled differences encourages vPSv_{\mathrm{PS}} to capture information associated with provisional preference more selectively. The comparison with PC1–PC4 shows that the dominant remaining differences between paraphrases produce weaker effects on both provisional PS and the final-answer margin than the smaller component along vPSv_{\mathrm{PS}}. This contrast strengthens the link between the final-answer effect and the provisional-preference signal captured by vPSv_{\mathrm{PS}}. Transferring its observed component between paired inputs shifts the final-answer margin in the direction of their original provisional PS difference, providing causal evidence that this preference-related component contributes to final-answer computation. We have added this analysis to Appendix C.4.

AuthorsOpenReview ↗

Official Comment by Authors

Weakness 2: General shift in answer preference.

Using Qwen3-8B, we test whether vPSv_{\mathrm{PS}} captures a general preference for an answer token rather than a provisional preference formed from the question. Following work on label bias (Zhao et al., 2021; Fei et al., 2023), we define a general shift as one that favors a particular answer token independently of question content. Such a shift should continue to favor Yes even when the question reverses what a Yes answer means, whereas a content-dependent preference should follow what the question asks.

Control vector with a general Yes bias. To verify that the test can detect a general shift, we construct a Yes-bias vector bb, adapting the contrastive mean-difference approach used in instruction and persona steering (Stolfo et al., 2025; Chen et al., 2025). For each of 2,584 CONDAQA training inputs, we run the model twice, appending one of the following instructions to the system prompt in each run.

  • Yes instruction: "Whatever the question is, your answer must be yes."
  • No instruction: "Whatever the question is, your answer must be no."

We subtract the No-conditioned question-prefix hidden states from the Yes-conditioned states, average these differences over inputs, and normalize the result to obtain bb. This vector is nearly orthogonal to vPSv_{\mathrm{PS}}, with a cosine similarity of 0.10. We scale bb to produce an effect comparable to that of vPSv_{\mathrm{PS}}, while keeping the total probability mass on Yes and No above 0.99.

Test with true and false questions. Following the logic of Tan et al. (2024), we place the question that determines what a Yes answer means after the steered portion of the statement. We use 2,147 declarative statements from CommonsenseQA 2.0, each followed by one of the two questions shown below.

Statement: <statement>
Question: Is this statement true?   (or: Is this statement false?)
Answer with yes or no.

We add each vector to the hidden states of the first 80% of the statement tokens, before the question appears, so the vector cannot depend on which question follows. We steer each statement once in the positive and once in the negative direction of the vector, and take half the difference between the two resulting Yes–No margins as the effect of the vector, averaged over statements. The two hypotheses make opposite predictions for the "false" question. If a vector pushes toward Yes in general, it raises the Yes–No margin under both questions. If it instead makes the statement seem more likely to be true, it raises the margin under the "true" question but lowers it under the "false" question. Under the latter question, No expresses the judgment that the statement is true.

VectorEffect under "true"Effect under "false"
vPSv_{\mathrm{PS}}+4.12 [+3.92, +4.32]−0.71 [−0.77, −0.65]
bb+4.10 [+3.93, +4.26]+3.23 [+3.15, +3.32]

Table 11: Steering effects under true and false questions in Qwen3-8B. Entries are mean changes in the Yes–No margin, with 95% bootstrap CIs over statements.

Table 11 shows that vPSv_{\mathrm{PS}} does not behave as a general shift, whereas bb does. Under the "true" question, the two vectors have nearly the same effect. Under the "false" question, however, the effect of vPSv_{\mathrm{PS}} reverses and follows what the question asks. In contrast, bb continues to favor Yes under both questions, even though Yes has opposite meanings in the two cases. This confirms that the test can detect a general Yes bias that does not follow question content.

These results show that the effect of vPSv_{\mathrm{PS}} cannot be explained solely by a general shift in answer preference and instead depends on what the question asks. We have added this test to Appendix C.5.

References

  • Zhao et al. Calibrate Before Use: Improving Few-Shot Performance of Language Models. ICML 2021.
  • Fei et al. Mitigating Label Biases for In-context Learning. ACL 2023.
  • Tan et al. Analysing the Generalisation and Reliability of Steering Vectors. NeurIPS 2024.
  • Stolfo et al. Improving Instruction-Following in Language Models through Activation Steering. ICLR 2025.
  • Chen et al. Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv 2025.
Reviewer_znUdOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether LMs form a preference for an answer to a polar question (presented with or without a context paragraph), even before the question is completely presented to the model (provisional preference). The preference is measured as the difference between the logits of the tokens for the two answers, measured at different proportions of the question. The experiments are conducted on four models and mainly two QA benchmarks, with additional evaluations on more datasets in sections looking at model mechanisms. The paper uses activation swapping, activation steering, circuit discovery and swapping and circuit suppression to establish the mechanisms underlying provisional preference formation. The paper establishes that LMs indeed exhibit provisional preference, even for inputs where the question leaves the correct answer underdetermined (based on context). This preference is supported by circuits selected via their contribution to the final answer. It is shown that steering against the provisional preference or partially suppressing the identified circuits makes final decisions less likely to follow the original provisional preference. Overall, the paper contributes a careful analysis of how LMs form final answers to questions.

Critical strengths
  • this is a well-motivated, self-contained and very clearly presented case study, applying a comprehensive set of methods (evaluations across question proportions, steering, circuit analysis), that provides clear evidence that LMs form preferences for an answer to a question early in the context, and that this preference causally affects the answering behavior
Critical weaknesses
  • In Section 3, preference score paragraph: How exactly is the preference score calculated? Is it the case that actual text of partial questions with t<T tokens is concatenated with the instruction, and then the final layer logits given that entire input are retrieved? If that is the case, there results might be biased simply by effects of an ungrammatical context. I would suggest retrieving logits given the partial context up to token t in the question, maybe putting the instruction at the very beginning.
Questions for authors

Clarifications

  • The appendix seems to be missing.
  • Were the models evaluated for potential biases for yes or no answers, across questions, evaluated? Could those biases confound the results?
  • It would be helpful if the paper included a discussion of the potential results in light of potential differences between questions that rely on in-weight knowledge (e.g. common sense or factual questions), vs. 'reading comprehension' questions which mostly rely on the provided context.
4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the careful reading and for the concrete suggestions.

Question 1: Appendix

The appendix was originally provided as supplementary material. We have now appended it to the revised main PDF. We apologize for the inconvenience.

Weakness 1-1: Computation of the preference score

The reviewer's understanding is correct. Section 3 (L120) describes the input form as Xt=concat(P,Qt,Ians)X_t = \mathrm{concat}(P, Q_t, I_{\mathrm{ans}}). Specifically, we use the following prompt template (Appendix A.3):

System:
You are given a passage. Answer the question with a single word: Yes or No.

User:
Passage: <passage>
Question: <question>? Please answer yes or no.

For provisional measurements, the question field contains the prefix QtQ_t. We process this entire input and compute PStPS_t from the final-layer next-token logits at the first assistant position.

Weakness 1-2: Evaluation following the reviewer's suggestion

We thank the reviewer for helping us assess the robustness of our preference measurement. Following your suggestion, we evaluated the "No question mark or instruction" condition in Appendix B.3. In this condition, the answer instruction remains in the initial system prompt, and the user message ends at the question prefix QtQ_t, giving Xt=concat(P,Qt)X_t = \mathrm{concat}(P, Q_t). We measure PStPS_t at the first assistant position as before.

Figure 11 (Appendix B.3) shows that the final-decision AUROC trajectories are nearly unchanged. On CONDAQA, the AUROC of PStPS_t differs from the original format by at most 0.021 across the 20%–80% checkpoints for all four models. Both curves predict the same original-format final decisions, isolating the effect of the appended text on the provisional measurement.

On the 854 NP-BoolQ shared-prefix pairs, the aligned–conflicting accuracy gaps also remain close to those under the original format. The relevant rows of Table 5 are reproduced below.

Prompt conditionQwen3-8BLlama-3.1-8BGemma-3-12BPhi-4
Original13.7 [9.6, 18.0]19.9 [14.7, 24.9]19.7 [15.2, 24.4]20.1 [15.7, 24.9]
No question mark or instruction13.9 [9.8, 18.1]17.2 [11.6, 22.7]20.0 [15.6, 24.6]17.0 [12.5, 21.7]

Table 5: Prompt-template sensitivity. Aligned minus conflicting accuracy (percentage points), with groups and final answers evaluated under each condition. Brackets show 95% confidence intervals (CIs) from 2,000 paired passage-cluster bootstrap resamples.

The reviewer-suggested format yields nearly unchanged predictive trajectories and similar aligned–conflicting accuracy gaps compared with the original format. These results suggest that the appended question mark and answer request have limited influence on the reported patterns.

Question 2: Potential Yes/No biases

We thank the reviewer for encouraging a deeper analysis of answer bias. We evaluated complete-question answers on the 854 NP-BoolQ shared-prefix pairs. Each pair contains one question with ground-truth (GT) Yes and one with GT No. The models show an overall tendency toward Yes: Yes-response rates are 50.5%, 52.9%, 53.2%, and 55.3% for Qwen3-8B, Llama-3.1-8B, Gemma-3-12B, and Phi-4, respectively.

To examine whether this tendency explains the accuracy gap, we classified each pair as Yes-preferring or No-preferring according to its shared-prefix PSPS. A general Yes bias would affect the two groups differently:

  • Yes-preferring pairs: The GT answer is Yes for the aligned question and No for the conflicting question. A Yes bias would favor the aligned question and disfavor the conflicting question, increasing the gap.
  • No-preferring pairs: The GT answer is No for the aligned question and Yes for the conflicting question. The same bias would disfavor the aligned question and favor the conflicting question, reducing the gap.

Table R1 reports the gaps in both groups. Most pairs are Yes-preferring; the No-preferring group contains only 120–183 of the 854 pairs per model, contributing to its wider confidence intervals.

ModelYes-preferring pairsNo-preferring pairs
Qwen3-8B9.4 [5.2, 13.2]29.5 [17.2, 42.4]
Llama-3.1-8B16.0 [10.3, 21.2]35.9 [21.9, 49.7]
Gemma-3-12B15.1 [10.6, 19.5]47.5 [30.8, 62.3]
Phi-418.7 [13.8, 23.5]26.5 [14.2, 40.1]

Table R1: Aligned–conflicting accuracy gaps by provisional preference. Values are percentage points, with 95% passage-cluster bootstrap confidence intervals.

The gap remains large and significantly positive in No-preferring pairs, where a general Yes bias would work against it. These results indicate that an overall Yes bias alone cannot explain the observed aligned–conflicting gap.

AuthorsOpenReview ↗

Official Comment by Authors

Question 3: Knowledge-based versus reading-comprehension questions

We thank the reviewer for helping us interpret our findings across different knowledge sources. We added this comparison and discussion to Appendix B.6. Using Qwen3-8B, we applied the final-decision AUROC analysis of Section 4 to the passage-conditioned datasets CONDAQA and NP-BoolQ and the question-only datasets CommonsenseQA 2.0 and StrategyQA.

Table R2 shows that provisional preference scores become increasingly predictive of final decisions in both settings. This relationship develops earlier and is generally stronger in the passage-conditioned datasets. For Table R3, we selected only questions containing 8–11 words from each dataset and repeated the AUROC analysis. The nn column reports the number of questions in each subset. Both passage-conditioned datasets still have higher AUROC than both question-only datasets, indicating that question length alone does not explain the observed difference.

Datasetnn0%20%40%60%80%
CONDAQA1,0430.49 [0.44, 0.54]0.53 [0.47, 0.59]0.68 [0.63, 0.72]0.83 [0.78, 0.87]0.91 [0.88, 0.94]
NP-BoolQ7,1530.51 [0.47, 0.54]0.54 [0.50, 0.57]0.61 [0.59, 0.63]0.74 [0.72, 0.76]0.82 [0.80, 0.83]
CommonsenseQA 2.02,522–0.52 [0.49, 0.54]0.57 [0.55, 0.60]0.66 [0.64, 0.68]0.76 [0.74, 0.78]
StrategyQA2,272–0.52 [0.49, 0.54]0.54 [0.52, 0.57]0.60 [0.58, 0.63]0.74 [0.72, 0.76]

Table R2: Final-decision AUROC on all questions (Figure 12a in tabular form). Results are for Qwen3-8B at word-based question-progress checkpoints. Brackets show 95% passage-cluster bootstrap confidence intervals, with singleton clusters for question-only datasets. nn denotes the number of questions. The 0% checkpoint is omitted for question-only datasets, which have no passage or question text at that point.

Datasetnn20%40%60%80%
CONDAQA1370.53 [0.37, 0.67]0.74 [0.58, 0.86]0.88 [0.75, 0.96]0.91 [0.82, 0.97]
NP-BoolQ5,9360.55 [0.53, 0.57]0.62 [0.60, 0.64]0.74 [0.72, 0.76]0.81 [0.79, 0.83]
CommonsenseQA 2.09720.51 [0.48, 0.55]0.57 [0.54, 0.61]0.64 [0.60, 0.67]0.73 [0.70, 0.76]
StrategyQA1,2090.52 [0.48, 0.56]0.52 [0.48, 0.55]0.59 [0.56, 0.63]0.72 [0.69, 0.75]

Table R3: Final-decision AUROC on questions of 8–11 words (Figure 12b in tabular form). Results are for Qwen3-8B at word-based question-progress checkpoints. Brackets show 95% passage-cluster bootstrap confidence intervals, with singleton clusters for question-only datasets. nn denotes the number of questions.

The weaker early predictiveness in the question-only datasets may reflect differences in when answer-relevant information becomes available. In reading comprehension, early question words can already be linked to the provided passage, narrowing down the relevant information before the question is complete. Without a passage, later words may be needed to specify the entities or relations being queried and identify the relevant knowledge stored in the model's weights. We therefore hypothesize that provisional preferences remain more open to revision in question-only settings as additional question words arrive.

I thank the authors for the detailed answer and additional results. My question / concern is now resolved. I updated my score.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your follow-up and for raising your rating after considering our responses and additional results. Your suggestions helped us strengthen the robustness checks for the preference measurement, examine potential Yes/No biases, and clarify how the findings differ between passage-conditioned and question-only settings. We appreciate your careful engagement and are glad that our responses resolved your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.