- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the transfer of internal factuality probing from direct answers to chain-of-thought responses. Under one locked protocol that crosses budget, training style, and feature pipeline orthogonally, it finds that the measured transfer gap depends strongly on the evaluation protocol. It thereby studies target-style supervision and claim-paired affine alignment as recovery methods, and audits automatic factuality labels to distinguish asserted errors from truncations and parser artifacts.
Critical strengths
- This paper separates the pipeline main effect from the cross-style effect. It shows that the original 0.804 collapse was manufactured by changing the budget, span rule, and pipeline at once, which is a useful correction method for hallucination detection.
- The six-feature surface screen is label-free and catches a parser artifact that cross-validation, held-out agreement, and probe AUROC all missed.
Critical weaknesses
- The paper's main contribution is a correction of its own prior claim. It is unclear how much of the contribution stands independently of the original incorrect headline.
- Localization becomes a bottleneck. Automated claim localization costs −0.217 AUROC, and the paper explicitly states that localization dominates end-to-end performance.
- The paper leans heavily on appendix blocks and cross-references rather than presenting a clean narrative. It is difficult to identify the single main result.
Questions for authors
- Gemma's failure is reported but not explained. Do you have a mechanistic hypothesis to explain this phenomenon?