- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether task-relevant truth can be recovered from LLM hidden states even when generated answers are incorrect, using controlled Fact–Rule–Action tasks and semantic interventions. The authors further propose FRA-ER, a hidden-state-based error detection method that improves error prediction over existing baselines across multiple QA tasks and model families.
Critical strengths
The paper provides well-designed controlled experiments and extensive empirical evaluations, demonstrating the effectiveness of hidden-state-based error detection across multiple tasks.
Critical weaknesses
-
Limited novelty and missing citation. The paper overlooks closely related prior work, such as Reasoning Models Know When They're Right (COLM 2025), and does not clearly establish its novelty beyond existing hidden-state correctness probing methods.
-
Disconnect between contributions. The controlled FRA experiments and natural QA error detection seem loosely connected, as FRA-ER essentially reduces to supervised correctness probing without explicit constraint instantiation.
Questions for authors
What evidence connects the truth–decision disagreement observed in controlled FRA tasks to the effectiveness of FRA-ER on natural QA, where error detection is performed through direct supervised probing?