acceptodds
Under review as a conference paper at ICLR 2027

InertiaBench: When LLMs Acknowledge Corrections but Still Give Old Answers

21 views

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in multi-turn conversations, yet how they handle user corrections remains poorly understood. To investigate this, we perform experiments where a user revises the premises, goal, or constraints after the model answers and later asks the full revised question. We observe an intriguing phenomenon: models acknowledge a revised premise, goal, or constraint but still give the old answer, a failure we term thinking inertia, which accuracy-based benchmarks conflate with other errors. In this paper, we introduce InertiaBench, a diagnostic benchmark that examines whether models acknowledge a correction, apply it in later answers, and recover when reminded. Concretely, we pair problems from extensive datasets with LLM-generated corrections and standalone revised questions, cross-check both reference answers with distinct LLMs, and manually modify a stratified sample. Finally, we gather 5,996 instances spanning 4 domains and 3 correction types, and responses are classified as the old answer, the correct revised answer, or another error. To evaluate the InertiaBench performance of popular LLMs, we conduct comprehensive experiments on 17 models, illustrating that each model gives the old answer to 18–21% of revised questions, usually after acknowledging the correction, and that 43–77% of these failures persist after a reminder, more often for reasoning models (68%) than standard LLMs (54%). Our code and data are available at https://anonymous.4open.science/r/InertiaBench-4165.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Multi-turn Evaluation, Thinking Inertia, User Corrections, Reasoning Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rPaxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work presents a benchmark, InertiaBench, for evaluating the behavior of LLMs in the case where the LLM acknowledges its mistake but still outputs the wrong answer. This is a wide-studied topic in NLP. This work proposes semi-automatic pipeline for the dataset construction. The results were further reviewed by human annotators. The resultant dataset includes 5,996 instances in 12 settings from 4 domains.

Critical strengths
  • This work addresses an important but yet to clearly solved weakness in many LLMs.
  • The manuscript is easy to follow.
  • The size of the benchmark data is moderate (5,996 instances).
  • The quality of the data were reviewed by human annotators.
Critical weaknesses
  • Many of LLMs involved in the experiments are outdated. Some more recent LLMs such as Gemini 3.x, GPT-5.x, and Qwen 3.x could be added in experiments.
  • The associations between performance and thinking effort (and length of thinking trajectory) could be further analyzed.
  • In each instance, the old, wrong answer was pre-generated by another LLM in the construction pipeline but not real wrong answer generated by the LLM under test. This style of evaluation does not measure the true performance of the LLM in the real-world usage. It could be unrealistic to mock the wrong answer for a question that can be perfectly solved by the LLM under test.
Questions for authors
  • How do the thinking effort and trajectory length affect the performance?
  • It could be interesting to see the comparison between thinking and non-thinking modes from the same LLM.
2 comments
AuthorsOpenReview ↗

Reply (1/2) to W1–W2 and Q1–Q2

Thank you for your time and efforts! Your comments pushed us to extend model coverage and to separate two quantities that are easy to conflate: configured reasoning effort and realized trajectory length. Here are our responses to your raised concerns and questions.

W1. Newer model coverage.

We added Qwen3.8-Max and Kimi-K2.6 (9–13 November 2026), each on 600 instances across the same 12 domain × correction-type cells, with thinking off/on at the final turn. The mismatch persists in the newer models at rates comparable to the original configurations:

ModelCAS ≥ 4CAS ≥ 4 & old answer at T4Same item, fresh old answer
Qwen3.8-Max99.2%13/122 (10.7%)6/122 (4.9%)
Kimi-K2.697.5%12/122 (9.8%)3/122 (2.5%)
Original 11 configurations, same items91.7%17/120 (14.2%)—

On repeatedly solvable items, human-confirmed immediate mismatches are 6/98 (Qwen) and 8/98 (Kimi), versus 0/98 and 1/98 in held-out fresh calls (pooled p = 0.012). GPT-5.x and Gemini 3.x were outside our available API access; our claims refer to the tested configurations. (Added models and settings: Appendix M.1, p. 40; screened results: Section 4.3, Table 4, p. 9, and Appendix M.4, p. 42.)

W2 & Q1. Thinking effort and trajectory length.

(1) Configured budget (Qwen, single-request T1+T3 condition, fixed items): old-answer rates are 19/122 (thinking off), 20/122 (512-token cap), 19/122 (2,048-token cap), and 9/122 (no explicit reasoning-budget cap) — the off→uncapped comparison has 10:0 discordant pairs, exact p = 0.002; the full 600-item set moves 20.0% → 15.0% (39:9, p < 0.001). More available computation helps; small caps do not. (Section 5, p. 10; Appendix M.5, pp. 42–43, Table 20, p. 43.) (2) Realized length (quartiles of observed reasoning-token counts): no common monotonic relationship with old-answer matching across the four model × condition groups (e.g., 12.9 → 15.6 → 19.0 → 7.4% for Qwen/FULL), and the longest quartile shows the lowest revised-task accuracy (e.g., 51.0% → 14.2%). Extra computation can help on a fixed item set, while longer traces on harder items need not indicate better answers; we report the two analyses separately. (Appendix M.5, Table 21, p. 43.)

Q2. Thinking vs. non-thinking within the same model.

On the common 122-item set (earlier turns generated once and held fixed; only the final-turn setting varies): Qwen changes 6.6% → 1.6% (FULL), 4.9% → 3.3% (fresh), 15.6% → 7.4% (T1+T3); Kimi changes 0.8% → 2.5%, 2.5% → 4.9%, and 13.9% → 12.3%. The improvement is real but model-dependent; in every setting the single-request correction presentation keeps the highest old-answer rate, so reasoning mode changes the size of the gap, not its location. (Appendix M.4, Table 19, p. 42; Appendix M.5, pp. 42–43.)

We have highlighted these analyses in Sections 4.3 and 5, and Appendix M.1, M.4–M.5 of the revised manuscript.

AuthorsOpenReview ↗

Reply (2/2) to W3 and informativeness

W3. The old answer is the model's own, not an injected one. We are glad to clarify the protocol: the tested model itself generates the initial answer (T2), and that response stays in the dialogue; the construction pipeline only supplies scoring references, never an assistant message. (Section 3.1, p. 4; Section 3.3, p. 6.) A valid revision can also make an initially correct answer obsolete, so a model that solves the original task is a feature of the design, not a flaw: the question is whether it adapts when the user changes the premises, goal, or constraints. The response analysis distinguishes the two quantities explicitly:

(1) Across 3,274 dialogues from 11 original configurations, 45.8% of actual T2 responses match the original reference (67.3% and 68.7% in the first 300 new-model dialogues).

(2) Among 620 archival T8 old-reference matches, 75.6% also match that same model's own T2 answer.

(3) Persistence is strongest exactly where the old answer is the model's own: when T2 matched the reference, the T4 reply returns it in 27.1% of dialogues (n = 1,501), versus 6.7% otherwise (n = 1,773). (Appendix M.5, “Initial-answer provenance,” p. 43.)

Old-reference matching (the full-set label) and repeating one's own earlier answer are reported as separate quantities in Section 3.3 and Appendix M.5.

On informativeness.

Respectfully, the specific relationship we measure is less studied than it may appear: prior benchmarks evaluate refutation handling, memory updates, or accuracy under evolving intent, but none links what the model says about a correction to the answer it then gives within the same dialogue (Table 1 of the paper). The revision adds what we believe is genuinely new evidence: human-confirmed cases in which a model accurately restates a correction and, in the same reply, answers the old task — on items it solves three times out of three in isolation (14/196 vs. 1/196 held-out, p = 0.012) — together with the matched presentation, history, and reasoning-effort controls above. (Section 4.3, pp. 8–9, Table 4; Appendix M.4, p. 42; Section 5, p. 10, and Appendix M.5, Table 20, p. 43.)

We have highlighted the provenance analysis in Appendix M.5 and the contribution framing in Sections 1 and 4.3 of the revised manuscript.

Reviewer_tS4eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InertiaBench tests whether LLMs say they understand a correction but later give the old answer. It measures how accurately the model restates the correction, whether its final answer matches the old or updated reference answer, and whether a reminder helps it recover. The paper tests 17 model configurations across 12 task settings.

Critical strengths
  • The benchmark links an accurate restatement of the correction to later answer outcomes in the same conversation. This gives useful information that accuracy alone cannot show.
  • It separates three outcomes: the old answer, the correct updated answer, and another wrong answer. This helps explain how models fail.
  • It gives detailed descriptions of how the tasks were built and includes extra analyses.
Critical weaknesses
  • The paper does not show how accurate its LLM judge is. The main results depend on an LLM judge that scores whether models accept corrections and give old or updated answers. These judgments may be difficult across the different generated tasks. However, the paper does not report human checks or comparisons with other judges for these scores. The human review covers the tasks and reference answers, while the judge agreement reported for RFC covers a different measure. The authors should check a sample of response scores with human reviewers and show whether scoring errors change the main results.

  • The paper does not show how often models give the old answer without the earlier conversation. Appendix K reports accuracy on the updated question in a fresh conversation, but not the old-answer rate. Without this comparison, it is unclear how much old-answer behavior comes from the earlier conversation rather than ordinary mistakes. The authors should compare old-answer rates on the same questions using the existing BL-3 responses.

  • Giving the old answer does not establish that the model forgot or failed to apply the correction. A model may correctly apply the updated conditions but make a later reasoning or calculation error, or guess an option, and still produce the old reference answer. Accurately restating the correction also does not establish that the model can solve the updated task. The paper therefore measures acknowledgment followed by old-answer matching, but this observation alone does not identify why the model failed. The authors should compare matched cases where the model can reliably solve the updated question in isolation and test whether the earlier conversation increases old-answer matching. Even that comparison would support a history-related effect rather than prove forgetting or non-application.

  • The evaluation does not include newer models. The OpenAI results use GPT-4.1 and o4-mini, and the Anthropic results use Claude 4.6 models. Newer generations are now available, so these results do not show whether the same pattern holds for current models. The authors should state when the experiments were run and add results for representative newer models available before their evaluation cutoff, or limit their claims to the tested versions.

Questions for authors
  1. Did human reviewers check the LLM judge’s scores for correction acceptance and final answers? If so, how often did they agree, and what mistakes did they find?
  2. Do the main results change when another LLM judge scores the responses or when people correct the judge’s mistakes?
  3. On the same questions, how often do models give the old answer in BL-3 compared with the full conversation?
  4. Among questions the model can reliably solve in isolation, does the full conversation increase old-answer matching? What evidence distinguishes forgetting or failure to apply the correction from a reasoning error after applying it?
  5. Why are Claude-4.6-Sonnet and GPT-4.1 included in the main evaluation but missing from the AnU analysis? Appendix F explains that records need both CAS and answer labels, but why are these labels unavailable for those models and some Gemini responses? Could the missing records affect the reported averages?
  6. When were the experiments run, and why were newer OpenAI and Anthropic models not included? Does the same pattern hold for representative newer models available before the evaluation cutoff?
4 comments
AuthorsOpenReview ↗

Reply (1/3) to W1 and Q1–Q2

Thank you for your time and efforts! Your separation of task validation from validation of the response scores pinpointed exactly what was missing, and we have now completed both. Here are our responses to your raised concerns and questions.

W1 & Q1. Human validation of the acknowledgment (CAS) and answer labels.

Two annotators (blind to model identity and automatic labels, working without LLM assistance) completed a response-label audit. Annotator agreement, judge accuracy against final human labels, and direct confirmation of the joint acknowledgment–answer event are reported separately:

Human-audit checkResult
Answer labels, inter-annotator agreement211/230 (91.74%), κ = 0.84
Answer judge vs. final human labels (expanded audit)386/410 (94.15%)
Old-answer label precision70/75 (93.33%)
CAS ≥ 4, inter-annotator agreement228/250 (91.20%), κ = 0.79
CAS judge vs. final human judgments219/250 (87.60%)
Immediate joint-event candidates confirmed130/132 (98.48%); 4/125 more among negative controls
Delayed joint-event candidates confirmed190/200 (95.00%); 6/200 more among negative controls

(Revised manuscript: Section 4.3, p. 9; Appendix M.2, pp. 40–41, Table 16.)

Human corrections go in both directions: the immediate audit (TP = 130, FP = 2, FN = 4, TN = 121) flips 2 positives and 4 negatives; the delayed audit flips 10 positives and 6 negatives. All 14 capability-screened immediate events (see Q4) are human-confirmed, 14/14. (Appendix M.2, pp. 40–41.)

Q2. Do the main results change under another judge or human-corrected labels?

We re-scored 6,805 historical records (all 17 configurations, ≤ 50 per cell) with an independent strict judge (DeepSeek-V4), holding CAS labels fixed: pooled AnU 18.25% vs. 18.74% under the original labels; per-configuration ranges 14.3–22.0% vs. 14.3–22.7%; largest single change 2.3 pp. The AnU pattern is preserved. Complementary checks: 91.7% cross-judge answer-label agreement (810 responses), 97.3% repeated-scoring agreement (864), 95.0% CAS-threshold agreement with the original judge (600 random), and 45.7% on a deliberately threshold-adversarial CAS sample (300), which we disclose but do not treat as representative. Under human-corrected labels, 98.48% of immediate and 95.00% of delayed candidates remain confirmed, so the reported events survive correction. (Appendix M.2, Table 16 and Table 17, p. 41; alternative-judge AnU and pilot scoring: p. 41.)

We have highlighted the full audit matrices in Section 4.3 and Appendix M.2 (Tables 16–17) of the revised manuscript.

AuthorsOpenReview ↗

Reply (2/3) to W2–W3 and Q3–Q4

W2 & Q3. Old answers in BL-3 compared with the full conversation. On the same questions, re-scored original responses: 19.0% (FULL) vs. 22.2% (BL-3) over 3,246 available pairs from 11 configurations. The new matched evaluation (identical instance-specific T7 in both conditions) finds the same direction in all four configurations:

ModelT8 thinkingPairsFULLBL-3FULL / BL-3 on 122 solver-supported items
Qwen3.8-MaxOff59917.5%19.7%8/122 vs. 6/122
Qwen3.8-MaxOn60014.0%18.0%2/122 vs. 4/122
Kimi-K2.6Off60016.0%20.5%1/122 vs. 3/122
Kimi-K2.6On60016.0%21.0%3/122 vs. 6/122

(Revised manuscript: Section 4.3, Table 4, p. 9; Appendix M.4, Table 19, p. 42.)

We report this plainly: retained history does not raise the aggregate old-answer rate. It does interact with the preceding reply, though: when T4 applied the correction, FULL is below BL-3 (all p ≤ 0.035); when T4 already gave the old answer (16–19% of dialogues, containing 58–78% of all final old answers), FULL exceeds BL-3 by +4.2 to +10.6 pp (max p = 0.023). Every T8 old answer on the 122-item set followed CAS(T4) ≥ 4, so the delayed joint counts equal the subset old-answer counts above. (Appendix M.4, p. 42, including Table 19 and the exploratory trajectory analysis.)

W3 & Q4. Reliably solvable items, and what distinguishes the failure modes.

Screening for items solved in three isolated attempts retains 172 (Qwen) and 169 (Kimi) items; FULL vs. held-out fresh T8 old-answer rates there are 1.7/1.2%, 1.2/2.9%, 0.6/1.2%, and 1.2/1.8% — no delayed excess. The measurable mismatch instead sits at the acknowledgment turn itself: on screened items (98 per model), CAS ≥ 4 and an old answer occur in the same T4 reply in 6/98 and 8/98 cases, versus 0/98 and 1/98 in held-out fresh calls (pooled 14/196 vs. 1/196, p = 0.012, CI +2.5–11.5 pp), human-confirmed 14/14. On distinguishing explanations: (1) timing — the old answer occurs in the very reply that restates the correction, leaving no intervening turn in which it could be forgotten; (2) competence — each task was solved 3/3 in isolation; (3) persistence — after such a mismatch, repeating the update leaves 165/208 (79.3%) old answers, re-deriving from the updated conditions leaves 160/208 (76.9%), and only explicitly flagging the error reduces this to 84/208 (40.4%). We therefore present thinking inertia strictly as a behavioral acknowledgment–application mismatch, and have removed mechanism-implying statements. (Screened immediate and delayed controls: Section 4.3, p. 9, and Appendix M.4, p. 42; supplementary T4-failure reminders: Appendix M.5, p. 43.)

We have highlighted these controls in Sections 4.3–4.4 and Appendix M.4–M.5 of the revised manuscript.

AuthorsOpenReview ↗

Reply (3/3) to W4 and Q5–Q6

Q5. Missing CAS and joint labels for Claude-4.6-Sonnet, GPT-4.1, and some Gemini records. The gaps came from an incomplete CAS scoring pass, not from selection by outcome. We extended scoring to all retained records:

ModelCAS labels before → afterRetainedGap leftAnU before → after
GPT-4.10 → 5,9175,9170— → 18.30%
Claude-4.6-Sonnet1,445 → 4,1894,201122.63%* → 18.36%
Gemini-2.5-Pro4,747 → 5,6125,612019.34% → 20.62%
Gemini-2.5-Flash5,813 → 5,8985,898019.35% → 19.53%

(Revised manuscript: Appendix M.3, Table 18, p. 41; coverage accounting and aggregate rates, p. 42.)

*The 1,445-row preliminary set covered only Sonnet's three algorithmic/numeric cells (the lowest-AnU domain, 3.1%) plus 14 CB-CE rows, which explains the unrepresentative 2.63%; it was never part of the published 15-model analysis, whose 15,400 positive count we reproduce exactly. The 12 remaining Sonnet records: 11 have empty T4 replies (nothing to score) and 1 was rejected by the provider's content filter (1 in 10,212 calls); they keep their T8 labels but exit the joint rates rather than count as negatives. Aggregate impact: 17.70% → 17.86% (record-weighted) and 17.72% → 17.88% (equal-weight model mean) — +0.16 pp under either weighting. The reported averages are stable. (Appendix M.3, p. 42.)

W4 & Q6. Experimental dates and newer models.

Original result-file timestamps span 23 March – 6 May 2026 (filesystem metadata). (For original model coverage, see Appendix I, Table 8, p. 29.) The added evaluation ran 9–13 November 2026 with qwen3.8-max (Alibaba Cloud Bailian) and kimi-k2.6 (Moonshot) on 600 instances with thinking off/on at T8; a 32,768-token output limit; Qwen temperature/top-p 0.7/0.8 (off) and 0.6/0.95 (on); Kimi API defaults; scoring by deepseek-v4-pro-0813; instance-specific T7 rewritten by qwen-max. (Added-run dates, endpoints, and inference settings: Appendix M.1, p. 40.) Newer OpenAI, Anthropic, and Gemini generations were outside our available API access, so our claims are limited to the tested configurations. The pattern does hold in the newer models: substantive restatement in 99.2% (Qwen) and 97.5% (Kimi) of dialogues, yet immediate acknowledgment–answer mismatches in 13/122 and 12/122 cases (vs. 6/122 and 3/122 on identical fresh asks), plus the human-confirmed screened events (6/98, 8/98) described under Q4. (Human-confirmed screened outcomes: Section 4.3, Table 4, p. 9, and Appendix M.4, p. 42.)

We have highlighted the coverage accounting in Appendix M.3 and the evaluation conditions in Appendix M.1 of the revised manuscript.

Thank you for the detailed response and the additional experiments, which address several of my concerns. My main concern, however, remains. The proposed metric cannot distinguish a failure to apply the correction from other errors that merely reproduce the same uncorrected answer. As a result, the benchmark's ability to diagnose that specific failure mode has yet to be established, and I will therefore keep my original score.

Reviewer_5DkGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new four-exchange evaluation benchmark - 1) answer an original task; 2) acknowledge a vliad revision; 3) answer a buffer question; and 4) answer a fully restated revised task. The benchmark contains ~6k records spanning different source domains and revision types and multiple language models were evaluated in the paper.

Critical strengths
  • This paper offers a useful diagnostic dataset beyond aggregate accuracy.
  • The paper was also executed decently with relatively broad LLM coverage.
Critical weaknesses

The main weakness of this benchmark is the reported errors may not be caused by the interference introduced by conversation history. A model could give the old answer simply because it gets the revised question wrong so much as an additional old-answer rates w/ and wo/ history is needed to tease out this confounding factor.

Questions for authors

How often does the model give the old answer when asked the revised question in a fresh conversation, compared with the full conversation? Were the final rewritten questions independently checked against their expected answers?

2 comments
AuthorsOpenReview ↗

Reply (1/2) to the critical weakness and Q1

Thank you for your time and efforts! We really appreciate the constructive review. The matched with/without-history comparison you asked for is exactly the right control, and it has substantially sharpened our measurements. Here are our responses to your raised concern and questions.

W1 & Q1. Old-answer rates with and without conversation history.

We agree that an old answer at T8 becomes evidence about conversation history only when compared with a matched no-history baseline. We therefore ran this comparison in two ways: (1) re-scoring the available original FULL/BL-3 responses, and (2) a new matched evaluation on 600 instances with one fixed instance-specific revised question (T7) per item, where FULL retains the dialogue and BL-3 asks the identical T7 in a fresh conversation. (Revised manuscript: Section 4.3, pp. 8–9, Table 4, p. 9; Appendix M.4, p. 42.)

ModelThinking at T8PairsFULL (with history)BL-3 (fresh, identical T7)
Qwen3.8-MaxOff59917.5%19.7%
Qwen3.8-MaxOn60014.0%18.0%
Kimi-K2.6Off60016.0%20.5%
Kimi-K2.6On60016.0%21.0%

Re-scored original responses point the same way: 19.0% (FULL) vs. 22.2% (BL-3) over 3,246 pairs from 11 configurations. The aggregate old-answer rate therefore does not increase with history in our data, and we report this plainly. What the aggregate hides is where old answers arise; three further results localize them:

(1) Revision presentation matters. On the 122 solver-supported items (see Q2), delivering the same revision as a correction (original task + update in one request) yields the old answer far more often than the restated task alone: 19/122 vs. 6/122 (Qwen, thinking off; McNemar 16:3, p = 0.004) and 17/122 vs. 3/122 (Kimi; 16:2, p = 0.001), in the same direction in all four configurations. (Section 4.3, p. 9; Appendix M.4, Table 19, p. 42.)

(2) Human-confirmed mismatches on repeatedly solvable items. Screening items each model solved in three isolated attempts (98 items per model), an accurate acknowledgment (CAS ≥ 4) and an old answer occur in the same T4 reply in 6/98 (Qwen) and 8/98 (Kimi) dialogues, versus 0/98 and 1/98 in a held-out fourth fresh call (pooled 14/196 vs. 1/196; item-level 10:1, exact p = 0.012; item-bootstrap 95% CI +2.5–11.5 pp). Two annotators confirmed all 14 events and the validity of their tasks. (Section 4.3, p. 9; Appendix M.2, pp. 40–41, Table 16; Appendix M.4, p. 42.)

(3) Two opposite trajectories average out. When the model applied the correction at T4, FULL old-answer rates are lower than BL-3 (all p ≤ 0.035). When T4 already contained the old answer — only 16–19% of dialogues, yet holding 58–78% of all final old answers — FULL exceeds BL-3 by +4.2 to +10.6 pp (Kimi thinking-on: 75/113 vs. 63/113, 18:6, p = 0.023). (Appendix M.4, “Exploratory trajectory analysis,” p. 42.)

We have highlighted these matched controls in Section 4.3 and Appendix M.4 of the revised manuscript.

AuthorsOpenReview ↗

Reply (2/2) to Q2 and the ratings

Q2. Were the final rewritten questions independently checked against their expected answers? Yes — on the exact final questions used in the paired evaluation, by two independent routes.

(1) Reference-free solving. DeepSeek-V4 and Kimi-K2.6 each solved the original and rewritten questions without seeing any reference:

Solver check (600 items)Items
Original question → original reference reproduced314/600
Revised question → revised reference reproduced203/600
Same solver reproduces both references for one item122/600 (union; DeepSeek 106, Kimi 100)

(Revised manuscript: Appendix M.1, p. 40.)

(2) Human task review. Two annotators confirm both references and T7 rewrite fidelity for 320/355 reviewed items (90.14%), including 110/122 of the solver-supported set and all tasks underlying the 14 screened immediate events. (Section 3.2, pp. 5–6; Appendix M.2, pp. 40–41, Table 16.)

On 167/600 items a solver derives the old reference from the revised question alone; this is exactly the item-level baseline tendency your concern describes. Every main contrast is therefore paired within each evaluated model against the same restated question, and the headline analyses use the 122 items whose two references were each reproduced by at least one independent solver. The construction-time audit of the released benchmark (1,200 stratified instances; correction validity, T7 incorporation, buffer independence) is described in Section 3.2 and Appendix E. (Section 3.2, p. 5; Appendix E, pp. 26–27.)

We have highlighted the solver checks and task audits in Section 4.3 and Appendix M.1–M.2 of the revised manuscript, and will release the exact question versions, solver outputs, and audit dispositions.

On presentation and informativeness.

Thank you for the honest ratings — they pushed us to reorganize the evidence. The revision reports each outcome next to its matched control (Section 4.3), separates the immediate (T4) and delayed (T8) endpoints with explicit cohorts (Appendix M), and states what each comparison does and does not establish. What the benchmark adds beyond prior multi-turn evaluations: it scores what the model says about an update and the answer it then gives within the same dialogue, and separates old answers from revised answers and other errors. The added models restate corrections substantively in 97.5–99.2% of dialogues, so the mismatch we measure concerns answer identity after acknowledgment, not noticing the change. (For the joint measurement and screened outcomes, see Section 3.3, p. 6, and Section 4.3, pp. 8–9; for prior-work comparison, see Section 2 and Table 1, p. 3.)

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of all rebuttal responses

Dear reviewers, AC, SAC, and PC,

We sincerely thank you for the time and effort you have put into the review process.

In this work, we (1) identify thinking inertia — models accurately acknowledge a user correction yet still give the old answer; (2) design the Corrupt-and-Correct protocol, tracking each correction through acknowledgment, application, and recovery; (3) build InertiaBench (5,996 instances, 4 domains × 3 correction types) with dual-solver-checked references and human-audited samples; and (4) evaluate 17 model configurations, and in the revision add matched history/presentation controls, capability screening, and expanded human audits.

During the rebuttal period we completed new experiments and human audits addressing every raised concern: matched FULL/BL-3 old-answer comparisons, independent checks of the final rewritten questions, human validation of all response labels, completed CAS coverage, two newer models, reasoning-budget and trajectory-length analyses, and initial-answer provenance. We believe that we have addressed all reviewers' concerns with these supplemented experiments, audits, and clarifications, now reflected in the revised manuscript (Section 4.3, Appendices K and M).


✅ 1. Reviewer 5DkG's major concerns: history confound & T7 validation (initial rating: 2)

Matched with/without-history comparison; independent checks of the final rewritten questions.

(1) Matched controls: FULL does not exceed BL-3 in aggregate (19.0% vs. 22.2% on 3,246 historical pairs; 14.0–17.5% vs. 18.0–21.0% on the new 600-instance evaluation); the correction-presentation contrast is significant (p = 0.004 / 0.001); a trajectory analysis separates the two opposite subgroups (16–19% of dialogues hold 58–78% of final old answers). (Section 4.3, Table 4, p. 9; Appendix M.4, Table 19 and trajectory analysis, p. 42.)

(2) Human-confirmed immediate mismatches on repeatedly solvable items: 14/196 vs. 1/196 held-out (p = 0.012). (Section 4.3, p. 9; Appendix M.4, p. 42.)

(3) T7 checks: reference-free dual-solver support (122-item cohort) plus human task audits (320/355 confirmed; all 14 core tasks pass). (Section 3.2, pp. 5–6; Appendix M.1–M.2, pp. 40–41, Table 16.)

✅ 2. Reviewer tS4e's major concerns: label validity, controls, coverage, versions (initial rating: 2)

Human checks of judge scores; judge sensitivity; BL-3 comparison; solvable subsets; missing labels; dates and newer models.

(1) Completed two-annotator audits: judge–human agreement 94.15% (answers) and 87.60% (CAS); joint events confirmed 98.48% (immediate) and 95.00% (delayed); corrections go in both directions. (Appendix M.2, Table 16, p. 41.)

(2) Alternative-judge re-scoring of 6,805 records preserves AnU (18.25% vs. 18.74%; max change 2.3 pp). (Appendix M.2, p. 41.)

(3) CAS coverage completed for all 17 configurations; aggregate AnU changes by only +0.16 pp; all 12 remaining records accounted for. (Appendix M.3, pp. 41–42, Table 18.)

(4) Dates documented (original: 23 Mar – 6 May 2026; added: 9–13 Nov 2026); the pattern replicates on Qwen3.8-Max and Kimi-K2.6. (Added-run metadata: Appendix M.1, p. 40; screened new-model evidence: Section 4.3, Table 4, p. 9.)

✅ 3. Reviewer rPax's major concerns: model coverage, reasoning effects, provenance (initial rating: 2)

Newer models; thinking effort and trajectory length; thinking on/off; source of the initial answer.

(1) Two newer models added across the 12 cells; the mismatch persists (13/122 and 12/122 immediate events; 6/98 and 8/98 human-confirmed on solvable items). (For added-model settings and human-confirmed screened events, see Appendix M.1, p. 40, and Section 4.3, Table 4, p. 9.)

(2) Configured budget: uncapped reasoning reduces old answers 19/122 → 9/122 (p = 0.002); realized length: no monotonic relation — the two are reported separately. (Section 5, p. 10; Appendix M.5, Tables 20–21, p. 43.)

(3) Thinking on/off compared within each model and each presentation condition. (Appendix M.4, Table 19, p. 42; Appendix M.5, pp. 42–43.)

(4) T2 is generated by the tested model itself; own-answer persistence is 27.1% vs. 6.7%. (Section 3.1, p. 4; Appendix M.5, p. 43.)

In summary, we believe that we have addressed all reviewers' concerns with the supplemented experiments, human audits, matched controls, and clarifications in the updated manuscript's main body and appendix, and we are grateful for the engaged and constructive reviews that led to these improvements.

Dear Reviewers,

Thank you all for your time and effort in reviewing and responding. Just a gentle reminder to reviewers to take a look at the authors' responses and share any updated thoughts on whether your main concerns have been addressed and, if not, to indicate which points are not well addressed. The author–reviewer discussion period ends on November 18, so I would greatly appreciate your input before then. Thank you again for your time and contributions!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.