- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The main idea of the paper is to investigate the influence of imbalanced training priors on classification decisions. The theoretical formulation part is professional, and the results are promising. However, there are some issues related to the methodology, novelty, experiments, and uncertainty modeling.
Critical strengths
- The paper addresses the impact of class imbalance on the minority-class decision boundaries. 2.Its analysis includes fisher consistency, importance-weighted risk minimization, the gaussian boundary displacement, and minority-region problem.
- The predictive separation from dirichlet makes a simple framework to train the uncertainty-aware classifiers.
- The evaluation includes different modalities and fixed-label LLM classification
- The evaluation includes several few-shot groups in which the majority-class performance is maintained as well.
Critical weaknesses
1- It is very importatn to distigush better the contribution and main novelty of the paper from the past and conventional class re-weithing methods.
-
There are some assumptions that limit the generality of the model. for example the theorem 3 assumes known priors, pure prior shift, and independent optimization. Also the theorem 1 requires an ordering-stability condition .
-
There are some mismatchs between theory and the implementation. The theorem 3 shows exact importance weighting, whereas the implementation uses clipped weights and minibatch self-normalization. Fixed clipping can change the effective target then the theoretical guarantee does not directly apply.
-
There are also some insufficient experimental controls. To determine whether improvement comes from conventional reweighting rather than from the evidential formulation, a class-weighted cross-entropy model with the same head is needed. You should also add more complete baseline variability and statistical significance.
-
Examples include “Equation equation 13”, “The paper provide”, and repeated equation-reference wording in the supplementary section.
-
The Cao et al. (2019) reference appears twice with identical bibliographic information. Also, there are several references contain incomplete author formatting or extraneous text.
-
Tables 1 and 2 are dense and some figure labels are small.
-
Important selected hyperparameter values should also be provided.
Questions for authors
1- please clear the main novelty of your work versus the other classical methods.
2- Please clear the generality of your work, in the appreance of many limitations and constraints you defined.
2 comments
Assuming known priors only confirms the correctness of the prior information (all of which comes from the target domain), and has nothing to do with generalizability. The theme of this paper is evidential optimization of Fisher consistency theory on long-tailed datasets. The same head classes and baselines have also been given in the paper (see Table 1 and Table 2), and the comparison is between the accuracy of the original baseline and that after Fisher optimization.
Regarding weight clipping, please refer to lines 266–269 of the manuscript. It is solely intended to prevent training explosion caused by extremely large weights during training; in experiments, a factor of 10 is already a quite exaggerated threshold.