- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies order and shape calibration. Theory and experiments link token-level errors to limited output diversity. The exact measurements make this a useful diagnostic study. However, uncertainty in the token labels limits support for the broader conclusions. The uniform oracle also leaves the cause of its gains unclear. These concerns affect the main interpretation of the results and support my weak rejection.
Critical strengths
The controlled tasks provide strong evidence. Section 5.2 uses tasks with known sets of valid outputs. The authors can therefore calculate validity and diversity exactly. These measurements avoid uncertainty from judgments about text quality. The results support the proposed causes of diversity loss in these tasks.
Critical weaknesses
A token can have a valid continuation even when its greedy completion fails. Thus, greedy decoding can mark a valid token as invalid. Appendix A.3 includes useful checks with additional continuations. In Table 3, recall at precision 0.8 increases from 0.09 with greedy decoding to 0.34 with ten sampled continuations. The trade-off remains, but its measured size depends on the procedure used to assign labels. This uncertainty weakens broad claims about open-ended generation.
The oracle in Section 4.4 selects tokens labeled valid and gives them equal probability. It therefore changes both token selection and relative probabilities. Its gains do not isolate the effect of token order errors. Table 1 selects the other methods at validity of at least 0.8. However, the table does not report each method’s measured validity. The comparison therefore does not establish whether the diversity gains remain at the same validity rate.
Questions for authors
What fraction of tokens labeled invalid by greedy decoding receive a valid label with ten sampled continuations? How do these label changes affect the estimated limits of rank-based filtering?
Do the Table 1 gains remain when the methods have the same measured validity rate? Please report each method’s validity. Please also test an oracle that preserves the model’s relative probabilities among tokens labeled valid.