- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents a method that uses proxy information (cumulative entropy, confidence drop, and layerwise stability) to help the model determine when to defer the answer. An interesting point is that layerwise hidden states can be used to assess the reliability of token predictions. With this method, the decoding strategy encourages a roughly left-to-right generation order without fixed blocks and achieves a good speed–accuracy trade-off.
Critical strengths
The layerwise hidden states can be used to assess the reliability of token predictions, which gives some new messgae to the communities. With this method, The decoding strategy encourages a roughly left-to-right generation order without fixed blocks and achieves a good speed–accuracy trade-off.
Critical weaknesses
The improvements are minimal, and in some cases, the method performs worse than other methods. On Dream GSM8K, accuracy increases only from 81.73% to 81.80% (+0.07 percentage points), while throughput decreases from 30.03 to 24.56 TPS (an approximately 18.2% reduction), comparing with Fast-Dllm. I would be interested in seeing whether the authors could demonstrate a distinctive benefit of their method beyond existing ordering strategies. Does the cumulative entropy criterion capture task-specific dependencies, or does it primarily impose a left-to-right ordering bias?
Questions for authors
-
Writing and presentation. I think the writing of this paper could be improved:
- Figure 2 would be better placed in Section 2 after the concept of stability is introduced, and there is too much explanatory text in the figure.
- I found Section 2 confusing. I understand that the authors want to explain the motivation behind the different components, but it might be better to introduce a preliminaries section that clearly defines all the concepts before moving on to the analysis.
- Sentences such as “This follows Figure 2c: as confidence drop increases, the error rate rises from 14.2% to 46.4% for , but only from 1.8% to 6.9% for ” could be polished.
-
TPS evaluation at different batch sizes. Regarding the TPS evaluation, projecting intermediate-layer hidden states through the output head may introduce substantial computational overhead. I suspect this method could perform worse at larger batch sizes. Could the authors provide results for different batch sizes?