- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a failure mode of second-stage correction in long-tailed recognition. When the first stage already addresses class imbalance, fitting a correction using the model's predictions on its own training images can further hurt rare classes. The paper attributes this behavior to class-dependent overconfidence on training examples, which makes rare classes appear easier than they are on unseen data.
The proposed method fits class-balanced matrix scaling with off-diagonal and intercept regularization (ODIR) on out-of-fold predictions, selects its regularization, shrinkage, and logit-adjustment strength through internal cross-validation, and folds the resulting correction into the final classifier head. A binary response model explains when reusing training predictions reverses the correction direction and characterizes the optimal correction strength within the proposed deployment family. Experiments compare prediction sources across several correction rules and demonstrate improvements on CIFAR-LT, ImageNet-LT, and iNaturalist2018.
Critical strengths
-
A useful finding about how second-stage correction can fail. The paper shows that a model's predictions on its training images can mislead a subsequent correction, especially for rare classes. Even when the first stage already addresses class imbalance, the second stage can further favor frequent classes and hurt rare-class performance. The effect is substantial at the overall accuracy level: in the CIFAR-100-LT IF100 setting of Table 3, fitting on training logits yields 55.33% top-1 accuracy, 5.06 percentage points below LA. The binary analysis provides an interpretable explanation of how this reversal can occur. This combination of an informative observation and a useful explanation is a central reason I favor acceptance.
-
Convincing controlled experiments and a straightforward remedy. Tables 3 and 4 keep the correction method fixed while changing the predictions used for fitting. In particular, the matched in-fold and out-of-fold comparisons use the same fold models, helping isolate the effect of whether the prediction-generating model has seen the fitting images. Figure 3 examines the effect across several correction rules. Together with the results across seeds in Appendix D.2, these experiments provide persuasive evidence that the fitting source matters and support using out-of-fold predictions. Since the fitted correction can be folded into the existing linear classifier head, it requires no additional remapper computation at inference.
Critical weaknesses
-
The connection between the binary reversal criterion and the multiclass measurements needs stronger justification. Appendix C.3 defines the multiclass contrast as the true-class logit minus the largest competing logit, and the head–tail center as half the difference between the many-shot and few-shot per-class contrast means. However, the strongest competing class can vary across examples and need not belong to the opposite shot group, whereas the theory concerns a fixed binary head–tail contrast. The center also summarizes only the many-shot and few-shot groups, while Figure 6(a) relates it to changes in overall multiclass accuracy. It is therefore unclear when these statistics preserve the theoretical reversal condition or predict the behavior of the full multiclass remapper. Figure 5(b) also needs to specify whether its pairwise thresholds come from separately fitted binary corrections or are derived from the deployed multiclass remapper. These gaps limit how directly the multiclass diagnostics validate the theoretical mechanism.
-
The characterization of data-fitted methods in Section 2 needs qualification. Section 2 includes Prior2Posterior among methods characterized as fitting on predictions from images the model has seen. However, in its logit-adjusted setting, Prior2Posterior estimates the effective prior using held-out validation predictions and averages this estimate with one derived from training predictions (Bhat et al., WACV 2025, Section 4.3, Eqs. (28)–(29)). The characterization should account for this use of held-out predictions. The paper already acknowledges held-out calibration and stacking, as well as stratified K-fold predictions for first-stage difficulty estimation in long-tailed recognition. The related-work discussion should therefore distinguish the proposed second-stage construction, reversal analysis, and empirical findings from the established principle of using held-out predictions, and explain more precisely how the proposed approach differs from prior methods such as Prior2Posterior.
Questions for authors
-
Clarification of the multiclass mechanism validation: What assumptions or empirical evidence justify using the statistics in Appendix C.3 to predict the binary reversal condition, particularly when the strongest competing class varies across examples? For Figure 5(b), are the pairwise thresholds obtained from separately fitted binary corrections or derived from the deployed multiclass remapper? If the latter, how is a scalar threshold defined when the remapped logit difference can depend on other classes' logits? Please clarify which theoretical predictions these measurements directly test.
-
Clarification of the relationship to Prior2Posterior: Could the authors revise the characterization in Section 2 to account for Prior2Posterior's use of held-out validation predictions in prior estimation, and clarify what the proposed cross-fitted remapper and reversal analysis add relative to that approach?
-
Clarification of training cost: Appendix E acknowledges the low accuracy gain per unit of training compute, with improvements of 1.59 and 0.78 percentage points on ImageNet-LT and iNaturalist2018. Appendix C.1 states that five fold models use the same GLMC recipe as the full model. Assuming equal epoch counts, one full-data run plus five runs on approximately 80% of the data corresponds to roughly five times the aggregate sample-training workload of a single full-data run. Could the authors report the actual total GPU-hours, wall-clock time, and hardware used, including out-of-fold prediction generation, correction fitting, and hyperparameter selection? A brief quantitative cost summary in the main text would make the practical trade-off clearer.
5 comments
Response to Review LewP
(b) Answers to the three parts of Question 1.
- [Why the statistics of Appendix C.3] We agree that the strongest competing class varies across examples. We use the contrast of each image (its true-class logit minus the largest other logit) because this contrast decides whether the image is classified correctly, whichever class competes. For two classes, this contrast is the of Section 4, and the center (half the difference between the average contrast of the many-shot classes and that of the few-shot classes) is exactly the center of Theorem 1. For more classes, we use the center, computed before any correction is fitted, only to predict the direction of the change in top-1 (Figure 7(a), formerly Figure 6(a)).
- [Thresholds of Figure 6(b), formerly Figure 5(b)] The thresholds come from neither a separately fitted binary correction nor the deployed remapper. For each pair of a many-shot class and a few-shot class (1085 pairs), the threshold is the midpoint of the class-wise averages of the logit difference between the two classes, which by Lemma 1 is the threshold that a fit on these logits gives. This threshold depends only on the two logits of the pair, so it is a single number. We have revised the caption to state the source of these thresholds.
- [Predictions tested] These measurements directly test two predictions of Theorem 1. First, the threshold from reused logits lies on the head side in all 1085 pairs, while the threshold from out-of-fold logits lies near the optimal threshold (Figure 6(b)). Second, when the in-sample center is negative, the remapper fitted on reused logits (, ) lowers test top-1 over all classes, including the medium-shot classes, by raising many-shot accuracy and lowering few-shot accuracy. This prediction holds in all 27 class-balanced settings of Section 5.3 (Figure 7(a)).
2. On Prior2Posterior. (For Weakness 2 and Question 2)
Thank you for this correction, and for your detailed suggestions on Prior2Posterior.
- [Correction] We apologize for the inaccurate description. We had only noted that Prior2Posterior calculates the effective prior of the model using its reused logits, and overlooked that, when the model is trained with logit adjustment, Prior2Posterior also estimates the effective prior on a held-out validation set. Section 2 now reads: "Prior2Posterior calculates the effective prior of the model using its reused logits, together with a held-out validation set for models trained with logit adjustment."
- [Difference] Fitting on held-out predictions is an established principle, as you note. Our contribution is theoretical: we characterize the risk that memorization brings to a second stage fitted on reused logits, and prove when this risk arises (Theorem 1, and the new Appendix B.5 for any number of classes). Viewed through this analysis, Prior2Posterior can be seen as a trade-off that requires no additional training. Its held-out estimate is free of memorization, while the training-set estimate it averages in is not. Our out-of-fold fit avoids memorization on every training image at the cost of training the fold models, and we use it to verify the theory.
Response to Review LewP
3. On training cost. (For Question 3)
We agree that a discussion on computational overhead is crucial. The overhead comes mainly from cross-fitting, the established technique we use to verify the theory, and we believe the value of the theory does not depend on this technique. We report the measured cost in the new Appendix D.6 (Table 15, reproduced below), including the hardware and the cost of logit export, remapper fit and selection.
- [Detecting the risk is free] On a class-balanced first stage, our theory assesses the risk at no cost. The criterion of Theorem 1 needs no fold model. It is computed on the training set alone and predicts the direction correctly in all 27 class-balanced settings (Section 5.3).
- [Data as the limiting resource] Long-tailed recognition is limited by data, and tail data are costly to collect. Our analysis reveals that the training set itself still holds information that existing second stages do not use.
- [Value of the gain] On a first stage without balancing, the gain remains large. On the cross-entropy first stage, cRT and LWS improve the first stage by 2.34 and 1.89 points. Once memorization is avoided with the out-of-fold fit, these gains rise to 3.70 and 3.42 (Figure 1).
- [Cost analysis] We find that on CIFAR-100-LT, three fold models already match the gain of five. Across the three datasets, three fold models take 2.1 to 2.5 times the GPU-hours of the full model (table below). Exporting the logits and fitting the correction take about 2% of the training cost. Inference is unchanged, since the correction is absorbed into the classifier head.
- [Main text] Per your suggestion, Section 5 now summarizes this trade-off.
| GPU-hours on a single GPU | CIFAR-100-LT | ImageNet-LT | iNaturalist2018 |
|---|---|---|---|
| GPU | L40S (48 GB) | A100 (80 GB) | A100 (80 GB) |
| CPU cores | 28 | 32 | 32 |
| Full model | 0.84 | 12 | 40 |
| Three fold models | 2.12 | 25.5 | 93 |
| Five fold models | 3.53 | 42.5 | 155 |
| Logit export, remapper fit and selection | 0.03 | 0.75 | 3 |
Finally, the theory only requires logits of images the model was not trained on. Any cheaper way to obtain such logits satisfies the same theory, and we hope our analysis will guide the development of more efficient second stages.
We believe these revisions and the new analyses in the appendix have addressed your concerns and have strengthened the paper. We are happy to provide any further clarification during the discussion period. Thank you once again for your valuable guidance.
Sincerely,
The Authors of Submission 16359
Response to Review LewP
Dear Reviewer LewP,
Thank you for such a detailed and rigorous review. We are glad that the paper offered you "a useful finding about how second-stage correction can fail" and that you found the controlled experiments convincing. Your feedback on the connection between the theory and the multiclass measurements led to the new Appendix B.5, which proves the multiclass counterpart of Theorem 1. Following the comments of the other two reviewers, we have restructured the abstract and the introduction following the discovery process below.
Accordingly, we have changed "Memorized Logits" in the title to "Memorization".
Beyond this, we have only added the experiments and proofs requested by the reviewers, and further polished sentences that were hard to read. The theorems, proofs and experimental results are unchanged, and all revisions are highlighted in green.
Thank you for raising this crucial point. Although the closed form of Theorem 2 relies on the two-class Gaussian model, we have proved that the risk of reversal in Theorem 1 holds for any number of classes and any logit distribution (Proposition 2, new Appendix B.5), and the multiclass criterion uses neither a contrast nor a competing class.
(a) Theoretical response.
[Multiclass criterion] The multiclass criterion proved in the new Appendix B.5 (Proposition 2) uses no contrast. This criterion compares the average probability of the head classes with their class prior. The probability of the head classes is the softmax probability of all head classes against all other classes, so the two groups are fixed and no competing class has to be selected for each example.
[What is proved] For the softmax cross-entropy, a fit stops when the average probability of the head classes equals their class prior on the fitting data (Lemma 2). The offset fitted on the reused logits and the optimal offset on test data have opposite signs if and only if
pˉin,H(0)<C∣H∣<pˉf,H(0),
where pˉin,H(0) and pˉf,H(0) are the average probabilities of the head classes on the reused logits and on test data before any offset, and ∣H∣/C is their class prior (Proposition 2). Any fraction of the fitted offset then increases the test cross-entropy. In the binary Gaussian model this condition reduces to Δ>df>0 of Theorem 1.
[Predictions tested] The test cross-entropy increases with the applied fraction of the fitted offset, as Proposition 2 states, in all 12 settings of Appendix B.5 that satisfy the condition. These settings combine two first stages, three seeds and two head groups (many-shot vs. the rest; many- and medium-shot vs. few-shot).