- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines a limitation of entropy-minimizing selection, i.e., it prioritizes confident selection, but does not consider whether the selected candidate is informative or not. For example, a concentrated distribution can have low entropy regardless of which candidate is selected. This paper aims to address this issue by using residual entropy. The authors proves that the residual entropy D <= 0, when the pairwise ordering of candidates by trust agrees with their ordering by informativeness. The authors conducted empirical validations, real-pipeline validation and downstream applications to evaluate the proposed approach.
Critical strengths
-
The research question is interesting and this paper points out a concrete limitation of entropy minimization for candidate selection. I am convinced that the informativeness of selected candidates are also important, which, however, is not fully considered in the existing entropy minimization approach. The proposed residual objective is simple, and Proposition 2 gives an exact decomposition. Corollaries 3-4 provide explicit sufficient conditions that can be checked at a given state.
-
The experiments include useful negative results. Appendix D.5 reports no significant accuracy gain from selecting the initially advantaged expert when participation status is held fixed. Appendix E then examines the condition when there is expert redundancy. These results help clarify when and where the proposed method brings benefits or not.
Critical weaknesses
-
The theoretical contribution is narrower than the paper’s claim, as it does not always guarantee informative selection. The decomposition (the 2nd equation in the abstract) by itself does not guarantee that minimizing D will always help select an informative candidate or improve downstream performance. Corollary 3 assumes pairwise ordering agreement between beta_t and v_t/beta_t. Corollary 4 assumes that the candidate maximizing beta_t also maximizes v_t/beta_t, and adds a margin condition. Thus, these corollaries show that D< 0 follows when the specified conditions hold; they do not establish that optimizing the residual-entropy objective produces that ordering.
-
The class-imbalance experiment does not establish that the proposed mechanism adds value beyond a direct count-based baseline. Algorithm 2 computes each class’s sample count from the training labels and uses it to define the class-level scarcity signal. The benchmark, in turn, is constructed by substantially reducing the training examples of one class. In this setup, selecting the least frequent class is a natural baseline for identifying the target class, and the paper does not report how that baseline performs. It makes it unclear what the proposed selection mechanism adds beyond a signal that directly reflects the way the benchmark was constructed. The authors should report a count-based baseline and clarify whether the target remains identifiable from counts under their experimental protocol.
-
The MoE experiments provide limited evidence that the proposed objective selects the advantaged expert, but do not convincingly connect that ability to the downstream gains. Appendix D.3 reports that the residual objective selects the advantaged expert in 2/25 runs, versus 0/25 for the other objectives. The authors themselves do not treat it as evidence of selection accuracy. Appendix E shows that choosing the better expert can improve performance, but does not test whether the proposed method can identify that expert. Thus, the paper does not convincingly connect the method’s selection ability to downstream gains.
Questions for authors
-
How should the ordering-guarantee statements in Section 5.2 and Appendix E.1 be interpreted, given the ordering assumptions in Corollaries 3-4? What evidence supports the stronger claim that optimizing residual entropy produces a useful ordering?
-
Is there any restriction in Algorithm 2 that prevents selecting the class with the smallest training count? If not, what does residual selection add to this direct rule?