acceptodds
Under review as a conference paper at ICLR 2027

The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

Abstract

As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reliability, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_KTWHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reveals that while adding task-irrelevant context to LLM inputs barely affects overall benchmark accuracy, it causes significant per-example prediction flips. By prepending meaningless "pseudo-words" to test questions, the authors demonstrate a two-sided effect where irrelevant text degrades performance on some examples while arbitrarily improving it on others. To expose these risks, they introduce two metrics, Instability (INS) and Worst-tail Degradation (WTD), and show that these context-induced failures are highly model-specific, persist across various training stages, and are only partially mitigated by test-time reasoning.

Critical strengths

The paper successfully identifies a flaw in how LLMs are typically evaluated, proving that standard aggregate accuracy metrics can mask severe, per-example reliability risks. The introduction of Instability (INS) and Worst-tail Degradation (WTD) provides a concrete framework for future researchers to isolate harmful prediction flips.

The authors conduct a comprehensive evaluation across a wide variety of state-of-the-art models (including versions of GPT, DeepSeek, Grok, Mistral, and Gemini) and diverse datasets (MMLU-Pro, GPQA, HLE, SimpleQA).

Critical weaknesses

The paper is primarily observational and does not propose any novel methods or mitigations. While it identifies the problem and shows that test-time reasoning acts as a partial mitigation, it does not propose a new architectural or training-based cure for the instability.

The qualitative analysis of why models flip their answers (e.g., introducing calculation errors vs. solving the wrong problem) relies on LLM-based annotations rather than human expert evaluation. The lack of human verification leads me to question the reliability of some of these results.

Measuring INS and WTD requires extensive repeated sampling and complex boostrapping procedures, which the authors note have significant efficiency drawbacks. Further, the use of pseudo-words is quite an artificial setup, and it is unclear whether the findings and results would generalize well to more noisy real-world data and applications (e.g. the coding agent mentioned in the intro).

Questions for authors

Your evaluation relies on extensive repeated sampling and bootstrap procedures to accurately calculate INS and WTD. Given that this is computationally prohibitive for continuous evaluation during model training, have you explored any proxy metrics that could reliably estimate a model's Worst-tail Degradation in a single pass?

The qualitative error analysis uses LLM-based annotation to categorize the types of prediction flips. However, given that these questions are complex enough that the models themselves are failing when distracted, how can we be confident that the LLM judge isn't exhibiting its own reasoning blind spots during evaluation? Did you conduct any human-expert validation on a subset of these annotations to confirm the error categorizations?

What insights do your findings contribute to developing mitigations for the issues you have identified and raised in your paper?

5 comments

Thank you for your positive assessment of our metrics and evaluation, and for the constructive questions. We address each point below.


W1 / Q3: No mitigation; what insights do the findings give?

While our main contribution is a measurement, we tested three more candidate mitigations as preliminary results. All use the paper's local-model protocol (MMLU, 500 questions, 10 pseudo-word contexts of 1k tokens; App. G).

Model scale (Qwen3.5, 0.8B → 9B).

SizeBase: AVGBase: INSBase: WTDInstruct: AVGInstruct: INSInstruct: WTD
0.8B46.36.626.447.27.530.3
2B57.27.126.555.77.127.9
4B70.03.915.170.46.322.5
9B72.63.713.873.76.421.2

Scale reduces instability but does not remove it (WTD 26.4 → 13.8 for base, 30.3 → 21.2 for instruct).

Looped depth (Ouro-2.6B, 2 → 4 recurrent loops).

LoopsAVGINSWTD
254.05.119.0
358.84.515.6
459.13.912.3

More loops reduce instability for Ouro-2.6B (WTD −6.7, significant) while accuracy rises.

An explicit system prompt: "The conversation may contain irrelevant or meaningless text. Ignore it and answer only the final question."

ModelSystem promptAVGINSWTD
Olmo-3-7B-Instructdefault56.718.869.1
Olmo-3-7B-Instruct+ ignore instruction57.018.868.2
Qwen3.5-9Bdefault73.76.421.2
Qwen3.5-9B+ ignore instruction73.75.219.7

Telling the model to ignore irrelevant text barely helps.

Conclusion. Larger models and more loops reduce the instability, but they cost more compute and do not remove it, while a cheap fix such as prompting barely works. We leave a more systematic study of mitigations to future work.


W2 / Q2: Are the LLM error-type annotations reliable?

Why we did not rely on human annotation. Reliable human labels would require domain experts for every question. GPQA questions were written and validated by PhD-level experts, and even experts in the same domain reach only 65% accuracy, while skilled non-experts with web access reach 34%. HLE questions were individually contributed and reviewed by domain experts. Labelling one pair requires full understanding of two answers and locating where they diverge. Expert annotation at this scale is beyond what we can crowdsource reliably.

Why LLM judges are suitable here. The judge does not need to solve the question. As specified in our prompt (Appendix C), its task is to compare the incorrect answer against the correct one and pick one code from a defined taxonomy, while given the ground truth. We believe state-of-the-art LLMs are suitable and reliable for this task.

Three more LLM judges on all pairs. We re-labelled every pair with gpt-6.1-sol, claude-sonnet-5.5 and gemini-4-argon, using the same prompt, and took their majority vote.

  • The majority vote agrees with the original gpt-5.5 labels on 83% of pairs (Cohen's κ = 0.75). Fleiss' κ across all four judges is 0.72.
  • The paper's conclusions hold under the majority vote:
Finding (Table 2)OriginalMajority Vote
Most common error type: MMLU-Pro / HLE / SimpleQACalculation / Rule / FactCalculation / Rule / Fact
Wrong Calculation share, MMLU-Pro: degraded vs improved62% vs 37%57% vs 37%
Wrong Calculation share, GPQA: degraded vs improved43% vs 23%42% vs 15%

W3 / Q1: Measurement cost and a single-pass proxy

  • Fewer samples are enough to rank models. We re-ran the analysis on subsamples of the existing data. For our 500-question setting, 5-17 samples per question (instead of 40) recover the model ranking by WTD and INS at Spearman ρ ≥ 0.9, about 3× cheaper.
  • Log-prob as a single-pass proxy. In our open-model experiments, we already compute INS and WTD from the probability of the ground-truth answer instead of repeated sampling. This takes one deterministic forward pass per prompt, and it needs no noise-floor correction. It is cheap enough for model developers to track instability continuously during training.

W3: Pseudo-words are artificial; do findings transfer to real-world noise?

Pseudo-words are a deliberately content-free stress test: any effect cannot come from semantic interference. Our ablation study (Sec. 3.3, Fig. 4) tested three other context types: random tokens (TOK), random webpages from FineWeb-Edu (WEB), and unrelated question-answer histories (SQA).

  • Pseudo-words are not the most harmful context. They give the highest WTD on MMLU-Pro but the lowest on SimpleQA, so other contexts can cause even larger worst-case drops.
  • The affected questions overlap across context types. Per-question changes correlate at mean r = 0.37 (MMLU-Pro) and 0.24 (SimpleQA) for gpt-5.4. Without sampling noise, the correlation reaches r = 0.70 for local models (App. G).
  • Even short contexts matter. As few as 10 pseudo-word tokens already induce non-trivial INS and WTD (Fig. 5).

Beyond QA, irrelevant context also lowers end-to-end task success for a real coding agent on SWE-bench; see our response to Reviewer fE9E, W3/Q3.

Hi Reviewer KTWH:

Thank you again for your thoughtful review. We conducted additional experiments and analysis to address your concerns.

We hope these results strengthen the paper's empirical contribution and practical implications.

Please let us know if anything remains unclear; we’re happy to elaborate.

Reviewer_fE9EOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how adding irrelevant context changes LLM performance on QA tasks. Across several models and benchmarks, improvements on some questions and declines on others can cancel out, leaving aggregate accuracy relatively stable. The authors measure changes in individual questions and the most affected examples, account for sampling noise, and examine how the results vary with context type, context length, reasoning effort, and training stage.

Critical strengths
  1. Looking at individual questions reveals changes that aggregate accuracy can hide. Testing multiple models, benchmarks, and context types gives a useful picture of how broadly this happens.

  2. The authors use repeated sampling and noise correction to separate context effects from random variation. Evaluating the most affected questions with separate samples also makes the reported degradation more convincing.

Critical weaknesses
  1. Shuffling answer options, changing spaces or line breaks, and rephrasing instructions are missing as control conditions. Comparing these perturbations under the same evaluation would help establish what adding irrelevant context reveals beyond general prompt sensitivity. I find the additional contribution unclear without this comparison.

  2. The main results use disabled or minimal reasoning, while increasing reasoning effort substantially reduces the measured instability. The paper leaves the practical importance of the remaining effects insufficiently established. A clearer account of how many questions remain reliably affected under stronger reasoning would help assess the significance of the findings.

  3. Agents can encounter irrelevant retrieved code and tool outputs during a task, but the experiments do not evaluate this process. The conversation-history setting still measures an answer to a standalone question after earlier QA exchanges. It leaves open whether context-induced mistakes persist, accumulate, or get corrected through later interaction, so the implications for agent reliability remain uncertain.

Questions for authors
  1. What evidence distinguishes the reported instability from what ordinary changes to prompt wording, formatting, or option order would produce?

  2. Using the existing samples, how many questions still show a reliable loss in correctness at high reasoning effort?

  3. What evidence supports extending the QA findings to final task success in agents that can revise their actions through further interaction?

4 comments

Thank you for your careful review and the concrete suggestions, which led to several of our new experiments. We address each point below.


W1 / Q1: What does irrelevant context reveal beyond ordinary prompt sensitivity?

We ran the three controls you suggested on Olmo-3-7B-Instruct (MMLU, 500 questions), with the same questions and the same log-prob estimator as the context conditions. CHA denotes the paper's pseudo-word context.

  • Option shuffle: 10 different reorderings of the options, gold label remapped.
  • Whitespace: 10 different whitespace and line-break edits.
  • Instruction rephrase: 10 different paraphrases of the answer instruction.

1. Magnitude: most prompt edits are a small effect.

WhitespaceInstruction rephraseOption shuffleCHA, 10 tokensCHA, 1k tokens
INS3.42.817.56.818.8
WTD12.711.150.125.869.1
  • Whitespace and instruction edits cause 5-6× less worst-tail damage than 1k tokens of pseudo-word context.
  • Option shuffle has similar INS, but its WTD is significantly lower than 1k tokens of pseudo-word context.
  • At matched edit size, adding 10 meaningless tokens is more harmful than instruction rewording: about 2× the INS and WTD.

2. Context and prompt edits affect different questions. As in the paper, we compute the Pearson correlation of the per-question change Δ_i between two conditions:

CHA (1k) vs.r
Other context types: random tokens / web text / unrelated QA0.80 / 0.70 / 0.55
Prompt edits: whitespace / option shuffle / instruction rephrase0.41 / 0.38 / 0.28
  • Every other context type correlates with pseudo-word context more strongly than any prompt edit does.
  • All correlations are positive: questions the model is unsure about move somewhat under any perturbation. But the questions that context breaks are mostly not the ones that prompt edits break.

Conclusion. Irrelevant context is a fragility separate from prompt-format sensitivity. Prompt-robustness checks miss most of it.


W2 / Q2: How many questions remain reliably degraded at high reasoning effort?

We define "reliable" as replicating on independent samples. All results below use existing samples from the paper (MMLU-Pro, high effort; the three values are gpt-5.4 / gpt-5.5 / gpt-oss-120b).

  1. Held-out contexts. We select the 25 most-degraded questions using 5 of the 10 contexts and half of the baseline trials. The 5 unseen contexts still lower their accuracy by 10.0 / 12.4 / 12.8 points (p ≤ 0.002).
  2. Independent runs. The questions most degraded at medium effort lose 8.8 / 8.8 / 7.4 points again in the separate high-effort run (p ≤ 0.002).
  3. Count above noise. At high effort, the number of questions with a ≥25-point drop exceeds the noise floor by 2-5 per 500 (up to 1%). At the lowest effort it is 13-29 per 500.

Conclusion. Reasoning shrinks the affected set (from 13-29 to 2-5 questions per 500) but does not remove it, and going from medium to high effort adds little. High reasoning effort is therefore not a full solution for deployment. Even at high effort, up to 1% of inputs are reliably flipped by irrelevant context, and users cannot predict which ones. Running every query at high effort also adds cost and latency.


W3 / Q3: Do context-induced mistakes persist, accumulate, or get corrected in agents?

(a) SWE-bench-verified. We use gpt-oss-120b with bash, python and editor tools, with 50 tasks × 20 runs per condition. We add 10k tokens of pseudo-word context at the start of the system prompt and compare it with the clean baseline.

  • 10k tokens of irrelevant context lower the solve rate from 42.7% to 37.3%.
  • The damage enters mostly through where the agent commits its edit: the share of final patches on the gold file falls by 6.2 pts, while the quality of the fix changes little. An early commitment to the wrong file is rarely corrected. With the irrelevant context, the agent also tests less often after editing (−5.7 pts).
  • The agent is not aware of the noise. It never mentions the irrelevant context in its reasoning (0/1,000 runs).

(b) A controlled multi-turn tool-use task. Olmo-3-7B-Instruct uses lookup / think / submit tools over 18 rounds. In each round, the user names 3 keys, and the agent looks up their values and submits a sum. We vary two factors:

  • Task structure: at the end of each round, the agent submits either the sum for this round (independent) or the running total of all rounds so far (cumulative).
  • Feedback: after each round, the agent receives no feedback, correctness feedback ("correct" / "incorrect"), or the ground truth (the true total).

Each of the 6 setups has 100 items (different key/value sets), and each item has 10 clean and 10 noisy rollouts. In the noisy rollouts, one tool result at round 8 contains 1k tokens of pseudo-word context. All rounds before round 8 are identical in both branches.

  • Persistence depends on the task. One noisy tool result causes immediate damage in every setup (−19 to −49 pts at round 8). With independent rounds, the error is gone by the next round. With a running total, the wrong total is carried to the end, although later steps are computed correctly on top of it. Being told "incorrect" every round does not help (0%). Being told the true total repairs the error within one round.
Task / feedbackAccuracy at round 8 (clean → noisy)Errors corrected at round 9Errors ever corrected laterAccuracy at round 18 (clean → noisy)
Independent, none96 → 7779%100%94 → 93
Independent, correctness97 → 7388%100%90 → 87
Independent, ground truth98 → 7374%99%92 → 86
Cumulative, none96 → 560%0%68 → 42
Cumulative, correctness94 → 450%0%63 → 30
Cumulative, ground truth94 → 4587%99%91 → 85
  • To test accumulation, we add noise to the tool results of three rounds (8, 11 and 14) instead of one, in the cumulative task without feedback. Each noisy round breaks more rollouts and almost none recover (1%), so noisy accuracy drops again after every noisy round. At round 18, only 14% of noisy rollouts are right, against 42% with one noisy round and 68% with none.
Noisy roundsAccuracy at round 8 (clean → noisy)Round 11Round 14Round 18
896 → 5684 → 5079 → 4868 → 42
8, 11, 1496 → 5684 → 2879 → 1968 → 14
  • The agent's behavior changes, but it does not notice its own errors. It becomes more cautious after noise. It looks up keys it already knows again, instead of reusing their values (reuse falls from 38% to 22% in the independent task and from 29% to 20% in the cumulative task, no feedback). It does this even when its noisy-round answer was right (independent task, reuse at the next round: 78% clean vs 57% after a correct noisy round), so it reacts to the noise, not to its error. Before any feedback, if we ask the agent whether its answer was correct, the agent is about as confident in wrong answers as in right ones in the cumulative task (AUROC 0.48-0.69), suggesting it is not aware of its error caused by noisy context.

Conclusion. An agentic setup does not automatically correct these errors caused by task-irrelevant context. When an error enters the agent's state, it persists and accumulates. Only informative feedback repairs it. The agent's own reaction to noise cannot repair a corrupted state. The QA-level instability we measure is therefore an entry point for agent failures.

Hi Reviewer fE9E:

Thank you again for your constructive feedback. We conducted additional experiments that directly address your three concerns.

We hope these new results help clarify the paper's novelty and practical significance.

Please let us know if anything remains unclear; we’re happy to elaborate.

Thank you for the additional experiments. The prompt controls and high-effort analysis address my first two questions. The SWE-bench result is useful, though it remains unclear whether the effect from synthetic pseudowords in the system prompt generalizes to realistic distractors such as irrelevant code or logs. I’m keeping my rating at 2.

Comments on the paper

1 comment

We thank both reviewers for their thoughtful feedback. We are glad that both reviewers recognized the core strengths of the paper:

  • The problem matters. Per-example analysis "reveals changes that aggregate accuracy can hide" (fE9E). The paper "identifies a flaw in how LLMs are typically evaluated" (KTWH).
  • The metrics are useful. Introducing INS and WTD "provides a concrete framework for future researchers to isolate harmful prediction flips" (KTWH).
  • The evaluation is broad. It covers "multiple models, benchmarks, and context types" (fE9E): GPT, DeepSeek, Grok, Mistral and Gemini models on MMLU-Pro, GPQA, HLE and SimpleQA (KTWH).

We summarize the added evidence below.

ConcernNew evidenceKey result
Is this just prompt sensitivity? (fE9E)Option shuffle, whitespace and instruction-rephrase controls under the identical estimatorWhitespace and instruction edits cause 5-6× less worst-tail damage, and prompt edits hit different questions than context does
How many questions remain reliably affected at high reasoning? (fE9E)Replication on held-out contexts and independent runs, plus noise-corrected counts, using existing samples2-5 of 500 questions (up to 1%) stay reliably degraded at high effort; the most affected questions still lose 10-13 pts under unseen contexts
Does it matter for agents? (fE9E, KTWH)(a) Trajectory analysis of a SWE-bench coding agent; (b) a controlled multi-turn tool-use task that measures persistence, accumulation and repairSWE-bench solve rate drops 5.4 pts. In the tool-use task, errors in carried state are almost never corrected without informative feedback
Are the LLM error annotations reliable? (KTWH)Three more LLM judges on all 975 pairs, combined by majority voteThe majority vote agrees with the original labels at κ = 0.75, and the paper's error-type conclusions hold
Mitigations and measurement cost (KTWH)(a) Three candidate mitigations; (b) Sampling analysisNone of the three removes the instability; 5-17 samples per question are enough to rank models (~3× cheaper)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.