- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper decomposes model–oracle log-ratio disagreement into mean and variance, studies their behavior around hallucination onset, and uses oracle-guided decoding to reduce downstream hallucinations.
Critical strengths
The empirical distinction between short-lived entropy spikes and persistent oracle-referenced disagreement is investigated across multiple model families.
Critical weaknesses
-
The “fixed-KL” causal interpretation is incorrect. Equation 4 preserves (E_q[\delta]), where (\delta=\log(p_M/p_O)), rather than the KL of the intervened distribution. Under the stated constraint, [ \mathrm{KL}(q|p_O)=\mathrm{KL}(q|p_M)+\mathrm{KL}(p_M|p_O). ] Thus the intervention does not isolate variance at fixed KL for the actual sampling distribution.
-
Prediction 4 overstates what follows from the decomposition. Inability to recover an oracle-dependent quantity does not imply inability to predict hallucinations. Results for selected entropy and consistency detectors cannot establish the claimed limitation of all (p_M)-only detectors.
Questions for authors
What is the actual (\mathrm{KL}(q|p_O)) after intervention? What establishes detector blindness beyond the specific proxies evaluated?