- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the evaluation of token selection in DLM. It identifies that prior adaptive decoders determines which token to commit and how many to commit at each decoding step, which complicates the comparison of different selectors. The authors conduct controlled experiments by fixing and balancing the number of committed tokens across different selectors and find that their rankings can change under different decoding schedules. Based on these observations, the paper proposes a Decision-Value protocal, which suggests evaluating selectors w/ their count selection rules.
Critical strengths
- The paper controls token selection under varying decoding schedules. This helps illustrate how the evaluation results can be affected by the choice of decoding budget etc.
Critical weaknesses
- The main finding is that changing the number of committed tokens across decoding steps can change the performance ranking of different selectors. However, is this somewhat expected and known? For example, in Fast-dLLM, changing the confidence threshold affects how many tokens are committed at each step, and can naturally lead to different decoding trajectories and accuracy. This seems similar to the effect of hyperparameter tuning in existing decoding methods. I think this effect has been already considered in all prior work that people will always compare performance and efficiency together. The proposed DVP mainly summarizes some reasonable evaluation practices, and I don't see much new insight beyond these observations.
- The paper lacks understanding of why the observed ranking reversals happen. Most formulations are definitions of the experimental setup and comparison metrics. The paper shows that changing schedules can affect selector performance, but lacks what properties of the selector lead to these reversals. I would expect more insights beyond reporting empirical differences.
- Im concerned about the practical significance of the findings. The paper shows that changing the decoding schedule can change the ranking of token selectors. However, in practice, we usually compare complete decoding methods, each with its own schedule and hyperparameters. It is unclear how often the reported ranking changes would actually lead to selecting a worse decoder. In Section 5.2, the authors do not find such cases on GSM8K, and the cases on ASDiv involve the additional entropy-based selector. This makes me question how serious this issue is for existing decoding methods.
Questions for authors
- What is the main technical contribution of DVP beyond standard evaluation practices? Comparing selectors under matched budgets and evaluating complete decoders using accuracy and latency seem to be quite natural.
- How often do the observed ranking reversals actually lead to choosing a different complete decoder among existing published methods? The strongest examples in Section 5.2 appear to involve the additional entropy-based decoder.
- Since the number of committed tokens can already be controlled by hyperparameters in existing methods, what new insights does DVP provide beyond showing that changing these configurations can affect selector rankings?