- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies the setting of question ordering during prompting: their primary finding is that the model’s accuracy (pass@4) score on a question is affected by its position in a context (e.g., does that question come before or after other questions). In their experiments, they consider 4 different question permutations (T1/T2/T3/T4) across three different Qwen models across a variety of language modeling benchmarks (heavily math reasoning). They find that when a question is asked later in the context, the model places a disproportionately large amount of attention on earlier tokens in the context in earlier layers. They then design STAIR: a method to “re-address” how much attention is given to preceding tokens based on a new query; they show that in most settings STAIR raises QA performance compared to Native prompting. STAIR introduces a small set of parameters that need to be trained prior to use.
Critical strengths
-
I think the experimental setting and metrics used are fair and relevant to the current literature surrounding this work. While STIAR does introduce a small computational overhead when training additional parameters, the rest of the evaluation matches the visible history, current problem, generation seed, and decoding budget (lines 376-377). This makes it easy to compare the performance of STAIR to vanilla and native decoding.
-
The topic of how to manage the effect of a model’s prior context on later inference is pressing; many current solutions study this problem from the perspective of intervening on the context. This paper takes an interesting approach of intervening on the model activations (which may be relevant in settings where users are free-form interacting with a model outside of a harness which can better manage context).
Critical weaknesses
-
I am comparing the results in Figure 1D and Figure 2D; from Figure 1D it seems that Vanilla is the best performing QA strategy (in ⅞ settings) and that STAIR basically matches its performance; since error bars are omitted I suspect there is no significant difference in performance between STAIR and vanilla? In which case, why not just clear the context history and use vanilla T1 prompting? I know the authors use Figure 2D to motivate that in some cases, prior steps aid later steps, but if you look at this graph temporally (T2 -> T4), it seems that T4 “wrong→correct” is lower than T2. Overall at each Tx the net change is negative and it gets worse over T2,3,4. It really does just make you wonder if the cheapest and simplest option (resetting the context) is really just the best one.
-
Tables 1 and 2 are missing error bars; it doesn’t seem like the performance gain from STAIR is substantially better than Vanilla in most settings.
Questions for authors
-
Only a single model family is considered. I am wondering if STAIR (and the finding that the ordering of a question) generalizes to other open-weight model families? (e.g., LLama3)?
-
How does STAIR affect the model performance in a non-QA setting (e.g., longer form text completion)? Would you remove stair for non-QA tasks?
5 comments
Response to Reviewer xVvH (Part 2 / 3)
2. Statistical support and comparisons with Vanilla (W2)
Your emphasis on statistical uncertainty is important for interpreting comparisons with Vanilla. The revised Figure 1d now displays paired pointwise 95% intervals for STAIR relative to both Native and Vanilla. We also report complete per-turn Avg@4, Pass@4, and paired intervals across all three Qwen models and four benchmarks (Tables 28–30).
We estimate uncertainty using 4,000 paired bootstrap replicates, accounting for question and actual-session clustering while preserving matched conditions and each problem's four sampled responses.
The following representative comparisons illustrate both directions of historical influence:
| Model / Benchmark | Vanilla | Native | STAIR | STAIR − Native (95% CI) | STAIR − Vanilla (95% CI) |
|---|---|---|---|---|---|
| Qwen3-4B Instruct / MATH-500 | 95.00 | 94.10 | 94.63 | +0.53 [−0.41, 1.48] | −0.37 [−1.48, 0.90] |
| Qwen3.5-4B / MATH-500 | 96.00 | 94.78 | 96.27 | +1.48 [0.43, 2.68] | +0.27 [−1.06, 1.79] |
| Qwen3.5-4B / AIME 2025 | 60.83 | 40.56 | 52.22 | +11.67 [3.16, 20.77] | −8.61 [−18.72, 1.25] |
| Qwen3.5-4B / GPQA-Diamond | 63.64 | 75.25 | 75.38 | +0.13 [−2.62, 3.06] | +11.74 [7.26, 16.84] |
| Qwen3.5-9B / GPQA-Diamond | 68.81 | 79.88 | 75.93 | −3.96 [−6.96, −1.14] | +7.11 [3.16, 11.58] |
All values are Avg@4 (%). Vanilla reports independent solving at T1; Native and STAIR report mean T2–T4 results.
These results establish two complementary findings.
First, STAIR can mitigate substantial history-induced interference. On AIME 2025/Qwen3.5-4B, Native drops from 60.83% (Vanilla) to 40.56%, while STAIR recovers 11.67 points [3.16, 20.77], approximately 57.5% of the observed loss.
Second, resetting is not universally optimal. On GPQA-Diamond, both Qwen3.5 models perform substantially better with retained history than with Vanilla. STAIR achieves +11.74 [7.26, 16.84] over Vanilla on Qwen3.5-4B and +7.11 [3.16, 11.58] on Qwen3.5-9B, even though the latter trails Native. These positive paired intervals show that clearing history is not always the best-performing choice.
We further strengthened the analysis underlying Table 1 by conditioning on the correctness of the actual first historical response. When T1 is incorrect, Native falls 19.72 points below Vanilla [−25.08, −14.38], while STAIR recovers +8.21 points [4.34, 12.22] over Native. When T1 is correct, STAIR improves over Native by +0.76 points [0.15, 1.40] (Table 33).
Finally, we expanded the fixed-history T4 comparisons across three decoding-seed groups. On Qwen3.5-4B/AIME 2025, STAIR achieves 68.33 ± 2.50% Avg@4, compared with 46.11 ± 3.37% for Native, with positive improvements in every group (Tables 4, 32).
Together, these analyses clarify the central distinction: Vanilla is a strong reference when history can be discarded, while STAIR improves how historical computation is used when conversational history is retained.
Response to Reviewer xVvH (Part 3 / 3)
3. Generalization to other model families and non-QA tasks (Questions)
(1) Other open-weight model families and the effect of question position.
Your question about whether both the history effect and STAIR generalize beyond Qwen is particularly important for establishing the scope of our findings. We extended STAIR beyond Qwen by independently training a controller on DeepSeek-R1-Distill-Llama-8B, using its own OpenThoughts response data.
Importantly, this experiment evaluates both the effect of retained history and STAIR's ability to improve historical readout.
Across three decoding-seed groups on AIME 2025:
| Condition | Mean Avg@4 (%) |
|---|---|
| Vanilla (T1) | 34.17 ± 0.83 |
| Native (T2–T4) | 23.98 ± 0.32 |
| STAIR (T2–T4) | 28.80 ± 0.58 |
Retaining history reduces accuracy by 10.19 percentage points relative to Vanilla, while STAIR recovers 4.81 points, with positive gains in every decoding-seed group (Table 24).
Thus, both the history-induced performance change associated with later turns and the benefit of learned historical readout extend to a Llama-based backbone.
We additionally compared SFT and RL checkpoints within the same 1.5B training lineage. At the SFT stage, Vanilla, Native, and STAIR achieve 25.56%, 14.72%, and 20.00%; at the RL stage, they achieve 28.89%, 20.83%, and 27.22%, respectively. These results further show that retained history affects subsequent reasoning across different post-training stages, with positive STAIR gains in all three decoding-seed groups at both stages (Tables 26–27).
(2) Non-QA tasks and long-form generation.
Your question about non-QA generation is important for understanding whether historical readout can benefit tasks beyond benchmark-style reasoning. To evaluate a task beyond conventional question answering, we applied Qwen3.5-9B's original math-trained controller directly to HumanEval+ code generation, without code-domain fine-tuning.
Across all 164 HumanEval+ tasks:
| Condition | Avg@4 (%) |
|---|---|
| Vanilla (T1) | 87.50 |
| Native (T2–T4) | 86.18 |
| STAIR (T2–T4) | 87.91 |
STAIR improves over Native by +1.73 points [−0.54, 4.23], with positive Avg@4 gains at every later turn (Tables 25, 34).
At the point-estimate level, the unchanged math-trained controller fully recovers the performance loss caused by retaining history and slightly exceeds Vanilla. This provides encouraging evidence that learned historical readout can transfer beyond mathematical reasoning to structured code generation without task-specific fine-tuning.
STAIR itself is not restricted to QA outputs. It adjusts current-prompt representations during prefill, while leaving the backbone and subsequent decoding path unchanged. The same mechanism can therefore be applied to general text generation.
We would not categorically disable STAIR for non-QA tasks. Its usefulness depends on whether the task can benefit from retained historical computation. For open-ended long-form prose, task-specific evaluations of coherence, relevance, and instruction following would help determine when to enable the auxiliary readout. The STAIR branch can also be disabled at inference, recovering the Native computation whenever auxiliary historical access is not desired.
Closing
Thank you again for your thoughtful review. Your comments have helped us clarify an important distinction: resetting history is effective when prior context is unnecessary, whereas STAIR investigates how to improve reasoning when history is retained for conversational continuity.
The additional Vanilla comparisons, paired uncertainty estimates, history-quality analysis, Llama-based evaluation, and HumanEval+ transfer results provide a clearer picture of when historical computation helps, when it interferes, and how learned query-side readout can improve its use.
We hope these revisions provide a stronger basis for evaluating the motivation, reliability, and broader applicability of STAIR.
Response to Reviewer xVvH (Part 1 / 3)
We thank you for your careful review and for recognizing the value of our controlled comparisons and activation-level approach to retained history. Your questions have helped us clarify the role of resetting, statistical evidence, and generalization.
In response, we have strengthened the Vanilla comparisons, added paired uncertainty estimates to Figure 1d and the appendix, extended evaluation to a Llama-based backbone, and evaluated cross-task transfer to code generation. We address your concerns below.
1. Why learn to use history instead of simply resetting it? (W1)
Your question about why one should not simply reset the context is fundamental to the motivation of this work. We agree that resetting is a reasonable and effective choice when previous context is unnecessary. However, our work addresses a complementary setting: when conversational continuity requires retaining history, can the model learn to use the computation produced by earlier turns more effectively?
In real interactions, users frequently ask independent questions, switch topics, pursue follow-up discussions, and later return to earlier tasks within the same conversation. Conversation continuity does not necessarily imply task continuity. Resetting after every task switch discards the preceding interaction, even when users may wish to revisit it. Existing summarization and selection methods manage what history to retain; STAIR instead investigates how current queries can access the historical computation that remains available.
We deliberately evaluate independent, self-contained problems to isolate history effects without direct semantic dependencies between tasks.
Regarding Figure 2d, your observation about the negative net trend is correct. For Qwen3-4B Instruct on MATH-500, the net effects of retained history at T2, T3, and T4 are −0.50, −0.95, and −1.25 percentage points, respectively.
However, the central observation in Figure 2d is that history changes answers in both directions, rather than simply causing uniform degradation. Among 2,000 paired responses at each turn, the numbers changing from wrong to correct are 37, 40, and 35, while those changing from correct to wrong are 47, 59, and 60, respectively.
Thus, positive answer changes persist at every turn, but negative changes increasingly outweigh them under Native as the conversation progresses. This is the motivation for studying selective historical readout: rather than assuming that retained history is uniformly helpful or harmful, we seek to adjust how its computational states influence the current problem.
Importantly, other model–task combinations show a net benefit from retaining history, as detailed in §2 below. Therefore, the negative trend in Figure 2d describes one controlled setting, not a universal property of retained conversations.
Historical computation also offers information beyond ordinary visible context. In thinking models such as Qwen3.5, earlier intermediate reasoning traces are not carried forward as visible text in subsequent turns; the ordinary conversation retains the post-thinking answer text. STAIR separately preserves K/V states produced during those earlier reasoning steps, allowing current queries to access that computation without restoring the intermediate reasoning text to the visible prompt.
This is precisely why we investigate activation-level historical access as a complement to context management, rather than treating Reset and STAIR as competing solutions to an identical problem.