- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
There are signals in language models before they answer a question that would show that their answer would be wrong. This paper proposes a cheap trick to nudge the model near the start of its answer so it’s less likely to lock into a bad line of reasoning.
Critical strengths
- The method proposed is simple and practical to use. It would be easily adopted if it works.
- Evaluation covers 10 models across many benchmark types, which is broad and comprehensive.
Critical weaknesses
- The motivation and the methods are not connected. The probing results and the compression analysis use different models and tasks, but the method never used the probe.
- The title claim “narrow the knowledge-output gap” is not tested. No evidence to show that the problems that the probe predicted failure can be fixed by EHETI.
- The main “be sloppy” prompt is not justified. The method may be sensitive to wording depending on the exact negative prompt.
- A probe that predicts wrong answers might be just detecting hard questions. This is different from the model having the right answer hidden inside. There is no discussion separating these two cases.
Questions for authors
- Why does your majority voting over 32 samples only have slightly more tokens than single-trajectory methods? There should be 32x more.
- Why does your GPT-OSS results on HMMT look better than BRUMO? This is not the case for the other models.
- Token units are 100 million which for AIME is several million tokens per problem. That is abnormally long reasoning. Could you check the units please?
Thank you for your efforts. If you have any further questions about our responses, we would be very happy to answer them.