- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents a benchmark, InertiaBench, for evaluating the behavior of LLMs in the case where the LLM acknowledges its mistake but still outputs the wrong answer. This is a wide-studied topic in NLP. This work proposes semi-automatic pipeline for the dataset construction. The results were further reviewed by human annotators. The resultant dataset includes 5,996 instances in 12 settings from 4 domains.
Critical strengths
- This work addresses an important but yet to clearly solved weakness in many LLMs.
- The manuscript is easy to follow.
- The size of the benchmark data is moderate (5,996 instances).
- The quality of the data were reviewed by human annotators.
Critical weaknesses
- Many of LLMs involved in the experiments are outdated. Some more recent LLMs such as Gemini 3.x, GPT-5.x, and Qwen 3.x could be added in experiments.
- The associations between performance and thinking effort (and length of thinking trajectory) could be further analyzed.
- In each instance, the old, wrong answer was pre-generated by another LLM in the construction pipeline but not real wrong answer generated by the LLM under test. This style of evaluation does not measure the true performance of the LLM in the real-world usage. It could be unrealistic to mock the wrong answer for a question that can be perfectly solved by the LLM under test.
Questions for authors
- How do the thinking effort and trajectory length affect the performance?
- It could be interesting to see the comparison between thinking and non-thinking modes from the same LLM.
2 comments
Reply (2/2) to W3 and informativeness
W3. The old answer is the model's own, not an injected one. We are glad to clarify the protocol: the tested model itself generates the initial answer (T2), and that response stays in the dialogue; the construction pipeline only supplies scoring references, never an assistant message. (Section 3.1, p. 4; Section 3.3, p. 6.) A valid revision can also make an initially correct answer obsolete, so a model that solves the original task is a feature of the design, not a flaw: the question is whether it adapts when the user changes the premises, goal, or constraints. The response analysis distinguishes the two quantities explicitly:
(1) Across 3,274 dialogues from 11 original configurations, 45.8% of actual T2 responses match the original reference (67.3% and 68.7% in the first 300 new-model dialogues).
(2) Among 620 archival T8 old-reference matches, 75.6% also match that same model's own T2 answer.
(3) Persistence is strongest exactly where the old answer is the model's own: when T2 matched the reference, the T4 reply returns it in 27.1% of dialogues (n = 1,501), versus 6.7% otherwise (n = 1,773). (Appendix M.5, “Initial-answer provenance,” p. 43.)
Old-reference matching (the full-set label) and repeating one's own earlier answer are reported as separate quantities in Section 3.3 and Appendix M.5.
On informativeness.
Respectfully, the specific relationship we measure is less studied than it may appear: prior benchmarks evaluate refutation handling, memory updates, or accuracy under evolving intent, but none links what the model says about a correction to the answer it then gives within the same dialogue (Table 1 of the paper). The revision adds what we believe is genuinely new evidence: human-confirmed cases in which a model accurately restates a correction and, in the same reply, answers the old task — on items it solves three times out of three in isolation (14/196 vs. 1/196 held-out, p = 0.012) — together with the matched presentation, history, and reasoning-effort controls above. (Section 4.3, pp. 8–9, Table 4; Appendix M.4, p. 42; Section 5, p. 10, and Appendix M.5, Table 20, p. 43.)
We have highlighted the provenance analysis in Appendix M.5 and the contribution framing in Sections 1 and 4.3 of the revised manuscript.
Reply (1/2) to W1–W2 and Q1–Q2
Thank you for your time and efforts! Your comments pushed us to extend model coverage and to separate two quantities that are easy to conflate: configured reasoning effort and realized trajectory length. Here are our responses to your raised concerns and questions.
W1. Newer model coverage.
We added Qwen3.8-Max and Kimi-K2.6 (9–13 November 2026), each on 600 instances across the same 12 domain × correction-type cells, with thinking off/on at the final turn. The mismatch persists in the newer models at rates comparable to the original configurations:
On repeatedly solvable items, human-confirmed immediate mismatches are 6/98 (Qwen) and 8/98 (Kimi), versus 0/98 and 1/98 in held-out fresh calls (pooled p = 0.012). GPT-5.x and Gemini 3.x were outside our available API access; our claims refer to the tested configurations. (Added models and settings: Appendix M.1, p. 40; screened results: Section 4.3, Table 4, p. 9, and Appendix M.4, p. 42.)
W2 & Q1. Thinking effort and trajectory length.
(1) Configured budget (Qwen, single-request T1+T3 condition, fixed items): old-answer rates are 19/122 (thinking off), 20/122 (512-token cap), 19/122 (2,048-token cap), and 9/122 (no explicit reasoning-budget cap) — the off→uncapped comparison has 10:0 discordant pairs, exact p = 0.002; the full 600-item set moves 20.0% → 15.0% (39:9, p < 0.001). More available computation helps; small caps do not. (Section 5, p. 10; Appendix M.5, pp. 42–43, Table 20, p. 43.) (2) Realized length (quartiles of observed reasoning-token counts): no common monotonic relationship with old-answer matching across the four model × condition groups (e.g., 12.9 → 15.6 → 19.0 → 7.4% for Qwen/FULL), and the longest quartile shows the lowest revised-task accuracy (e.g., 51.0% → 14.2%). Extra computation can help on a fixed item set, while longer traces on harder items need not indicate better answers; we report the two analyses separately. (Appendix M.5, Table 21, p. 43.)
Q2. Thinking vs. non-thinking within the same model.
On the common 122-item set (earlier turns generated once and held fixed; only the final-turn setting varies): Qwen changes 6.6% → 1.6% (FULL), 4.9% → 3.3% (fresh), 15.6% → 7.4% (T1+T3); Kimi changes 0.8% → 2.5%, 2.5% → 4.9%, and 13.9% → 12.3%. The improvement is real but model-dependent; in every setting the single-request correction presentation keeps the highest old-answer rate, so reasoning mode changes the size of the gap, not its location. (Appendix M.4, Table 19, p. 42; Appendix M.5, pp. 42–43.)
We have highlighted these analyses in Sections 4.3 and 5, and Appendix M.1, M.4–M.5 of the revised manuscript.