acceptodds
Under review as a conference paper at ICLR 2027

Memorized Logits Reverse Post-Hoc Correction in Long-Tailed Recognition

Abstract

Two-stage methods for long-tailed recognition freeze a trained backbone and refit the classifier on the backbone's own training images. When the first stage is already class-balanced, this refit favors the head classes it is meant to rein in. The cause is memorization. A model inflates the true-class logit of its training images, most for the rarest classes, and its training-set outputs therefore overstate the tail. In a Gaussian response model we prove that a fitted correction takes its direction from the center of the logits it is fit on. Memorized logits send it toward the head, out-of-fold logits toward the tail, and the optimal strength has a closed form. Cross-fitted logit remapping therefore scores each training image with a fold model trained without it, fits a matrix-scaling correction on those logits, and folds it into the classifier head. It ranks first among all compared methods on CIFAR-LT with further gains on ImageNet-LT and iNaturalist2018, whereas the same correction fit on memorized logits underperforms logit adjustment. Where a correction is fit matters more than which correction is fit. Code is available at https://anonymous.4open.science/r/crossfit_remap-BEC8.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-tailed recognition, post-hoc correction, memorization, cross-fitting, logit adjustment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LewPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a failure mode of second-stage correction in long-tailed recognition. When the first stage already addresses class imbalance, fitting a correction using the model's predictions on its own training images can further hurt rare classes. The paper attributes this behavior to class-dependent overconfidence on training examples, which makes rare classes appear easier than they are on unseen data.

The proposed method fits class-balanced matrix scaling with off-diagonal and intercept regularization (ODIR) on out-of-fold predictions, selects its regularization, shrinkage, and logit-adjustment strength through internal cross-validation, and folds the resulting correction into the final classifier head. A binary response model explains when reusing training predictions reverses the correction direction and characterizes the optimal correction strength within the proposed deployment family. Experiments compare prediction sources across several correction rules and demonstrate improvements on CIFAR-LT, ImageNet-LT, and iNaturalist2018.

Critical strengths
  1. A useful finding about how second-stage correction can fail. The paper shows that a model's predictions on its training images can mislead a subsequent correction, especially for rare classes. Even when the first stage already addresses class imbalance, the second stage can further favor frequent classes and hurt rare-class performance. The effect is substantial at the overall accuracy level: in the CIFAR-100-LT IF100 setting of Table 3, fitting on training logits yields 55.33% top-1 accuracy, 5.06 percentage points below LA. The binary analysis provides an interpretable explanation of how this reversal can occur. This combination of an informative observation and a useful explanation is a central reason I favor acceptance.

  2. Convincing controlled experiments and a straightforward remedy. Tables 3 and 4 keep the correction method fixed while changing the predictions used for fitting. In particular, the matched in-fold and out-of-fold comparisons use the same fold models, helping isolate the effect of whether the prediction-generating model has seen the fitting images. Figure 3 examines the effect across several correction rules. Together with the results across seeds in Appendix D.2, these experiments provide persuasive evidence that the fitting source matters and support using out-of-fold predictions. Since the fitted correction can be folded into the existing linear classifier head, it requires no additional remapper computation at inference.

Critical weaknesses
  1. The connection between the binary reversal criterion and the multiclass measurements needs stronger justification. Appendix C.3 defines the multiclass contrast as the true-class logit minus the largest competing logit, and the head–tail center as half the difference between the many-shot and few-shot per-class contrast means. However, the strongest competing class can vary across examples and need not belong to the opposite shot group, whereas the theory concerns a fixed binary head–tail contrast. The center also summarizes only the many-shot and few-shot groups, while Figure 6(a) relates it to changes in overall multiclass accuracy. It is therefore unclear when these statistics preserve the theoretical reversal condition or predict the behavior of the full multiclass remapper. Figure 5(b) also needs to specify whether its pairwise thresholds come from separately fitted binary corrections or are derived from the deployed multiclass remapper. These gaps limit how directly the multiclass diagnostics validate the theoretical mechanism.

  2. The characterization of data-fitted methods in Section 2 needs qualification. Section 2 includes Prior2Posterior among methods characterized as fitting on predictions from images the model has seen. However, in its logit-adjusted setting, Prior2Posterior estimates the effective prior using held-out validation predictions and averages this estimate with one derived from training predictions (Bhat et al., WACV 2025, Section 4.3, Eqs. (28)–(29)). The characterization should account for this use of held-out predictions. The paper already acknowledges held-out calibration and stacking, as well as stratified K-fold predictions for first-stage difficulty estimation in long-tailed recognition. The related-work discussion should therefore distinguish the proposed second-stage construction, reversal analysis, and empirical findings from the established principle of using held-out predictions, and explain more precisely how the proposed approach differs from prior methods such as Prior2Posterior.

Questions for authors
  1. Clarification of the multiclass mechanism validation: What assumptions or empirical evidence justify using the statistics in Appendix C.3 to predict the binary reversal condition, particularly when the strongest competing class varies across examples? For Figure 5(b), are the pairwise thresholds obtained from separately fitted binary corrections or derived from the deployed multiclass remapper? If the latter, how is a scalar threshold defined when the remapped logit difference can depend on other classes' logits? Please clarify which theoretical predictions these measurements directly test.

  2. Clarification of the relationship to Prior2Posterior: Could the authors revise the characterization in Section 2 to account for Prior2Posterior's use of held-out validation predictions in prior estimation, and clarify what the proposed cross-fitted remapper and reversal analysis add relative to that approach?

  3. Clarification of training cost: Appendix E acknowledges the low accuracy gain per unit of training compute, with improvements of 1.59 and 0.78 percentage points on ImageNet-LT and iNaturalist2018. Appendix C.1 states that five fold models use the same GLMC recipe as the full model. Assuming equal epoch counts, one full-data run plus five runs on approximately 80% of the data corresponds to roughly five times the aggregate sample-training workload of a single full-data run. Could the authors report the actual total GPU-hours, wall-clock time, and hardware used, including out-of-fold prediction generation, correction fitting, and hyperparameter selection? A brief quantitative cost summary in the main text would make the practical trade-off clearer.

5 comments
AuthorsOpenReview ↗

Response to Review LewP

Dear Reviewer LewP,

Thank you for such a detailed and rigorous review. We are glad that the paper offered you "a useful finding about how second-stage correction can fail" and that you found the controlled experiments convincing. Your feedback on the connection between the theory and the multiclass measurements led to the new Appendix B.5, which proves the multiclass counterpart of Theorem 1. Following the comments of the other two reviewers, we have restructured the abstract and the introduction following the discovery process below.

  • [Starting point] In our reproduction, the gain of SAMN (CVPR 2026) over the GLMC first stage is unstable (Figure 1).
  • [Observation] From this, we find that long-tailed second stages are not always effective, because memorization puts them at risk of insufficient correction, and even wrong correction.
  • [Proof] The correction depends on two quantities, the optimal boundary dfd_f and the memorization Δ\Delta. Fitted on the training images (reused logits), it stops at df−Δd_f-\Delta (Theorem 1). The first stages of earlier work were heavily imbalanced, so df>Δd_f>\Delta and the correction showed only an insufficient gain. Stronger first stages lower dfd_f and larger models raise Δ\Delta, so the risk is more likely to appear.
  • [Verification] The theory predicts that a fit on data the model has not seen stops at the optimal boundary dfd_f. We verify this prediction with standard cross-fitting and a simple correction.
  • [Result] As shown in the new Figure 1, multiple second-stage methods such as SAMN all obtain consistent improvements across datasets, imbalance factors and seeds.

Accordingly, we have changed "Memorized Logits" in the title to "Memorization".

Beyond this, we have only added the experiments and proofs requested by the reviewers, and further polished sentences that were hard to read. The theorems, proofs and experimental results are unchanged, and all revisions are highlighted in green.

1. On the multiclass measurements and the binary theory. (For Weakness 1 and Question 1)

Thank you for raising this crucial point. Although the closed form of Theorem 2 relies on the two-class Gaussian model, we have proved that the risk of reversal in Theorem 1 holds for any number of classes and any logit distribution (Proposition 2, new Appendix B.5), and the multiclass criterion uses neither a contrast nor a competing class.

(a) Theoretical response.

  • [Multiclass criterion] The multiclass criterion proved in the new Appendix B.5 (Proposition 2) uses no contrast. This criterion compares the average probability of the head classes with their class prior. The probability of the head classes is the softmax probability of all head classes against all other classes, so the two groups are fixed and no competing class has to be selected for each example.

  • [What is proved] For the softmax cross-entropy, a fit stops when the average probability of the head classes equals their class prior on the fitting data (Lemma 2). The offset fitted on the reused logits and the optimal offset on test data have opposite signs if and only if

    pˉin,H(0)<∣H∣C<pˉf,H(0),\bar p_{\mathrm{in},\mathcal H}(0)<\frac{|\mathcal H|}{C}<\bar p_{f,\mathcal H}(0),

    where pˉin,H(0)\bar p_{\mathrm{in},\mathcal H}(0) and pˉf,H(0)\bar p_{f,\mathcal H}(0) are the average probabilities of the head classes on the reused logits and on test data before any offset, and ∣H∣/C|\mathcal H|/C is their class prior (Proposition 2). Any fraction of the fitted offset then increases the test cross-entropy. In the binary Gaussian model this condition reduces to Δ>df>0\Delta>d_f>0 of Theorem 1.

  • [Predictions tested] The test cross-entropy increases with the applied fraction of the fitted offset, as Proposition 2 states, in all 12 settings of Appendix B.5 that satisfy the condition. These settings combine two first stages, three seeds and two head groups (many-shot vs. the rest; many- and medium-shot vs. few-shot).

AuthorsOpenReview ↗

Response to Review LewP

(b) Answers to the three parts of Question 1.

  • [Why the statistics of Appendix C.3] We agree that the strongest competing class varies across examples. We use the contrast of each image (its true-class logit minus the largest other logit) because this contrast decides whether the image is classified correctly, whichever class competes. For two classes, this contrast is the TT of Section 4, and the center (half the difference between the average contrast of the many-shot classes and that of the few-shot classes) is exactly the center of Theorem 1. For more classes, we use the center, computed before any correction is fitted, only to predict the direction of the change in top-1 (Figure 7(a), formerly Figure 6(a)).
  • [Thresholds of Figure 6(b), formerly Figure 5(b)] The thresholds come from neither a separately fitted binary correction nor the deployed remapper. For each pair of a many-shot class and a few-shot class (1085 pairs), the threshold is the midpoint of the class-wise averages of the logit difference between the two classes, which by Lemma 1 is the threshold that a fit on these logits gives. This threshold depends only on the two logits of the pair, so it is a single number. We have revised the caption to state the source of these thresholds.
  • [Predictions tested] These measurements directly test two predictions of Theorem 1. First, the threshold from reused logits lies on the head side in all 1085 pairs, while the threshold from out-of-fold logits lies near the optimal threshold (Figure 6(b)). Second, when the in-sample center is negative, the remapper fitted on reused logits (λ=1\lambda=1, τ=0\tau=0) lowers test top-1 over all classes, including the medium-shot classes, by raising many-shot accuracy and lowering few-shot accuracy. This prediction holds in all 27 class-balanced settings of Section 5.3 (Figure 7(a)).

2. On Prior2Posterior. (For Weakness 2 and Question 2)

Thank you for this correction, and for your detailed suggestions on Prior2Posterior.

  • [Correction] We apologize for the inaccurate description. We had only noted that Prior2Posterior calculates the effective prior of the model using its reused logits, and overlooked that, when the model is trained with logit adjustment, Prior2Posterior also estimates the effective prior on a held-out validation set. Section 2 now reads: "Prior2Posterior calculates the effective prior of the model using its reused logits, together with a held-out validation set for models trained with logit adjustment."
  • [Difference] Fitting on held-out predictions is an established principle, as you note. Our contribution is theoretical: we characterize the risk that memorization brings to a second stage fitted on reused logits, and prove when this risk arises (Theorem 1, and the new Appendix B.5 for any number of classes). Viewed through this analysis, Prior2Posterior can be seen as a trade-off that requires no additional training. Its held-out estimate is free of memorization, while the training-set estimate it averages in is not. Our out-of-fold fit avoids memorization on every training image at the cost of training the fold models, and we use it to verify the theory.
AuthorsOpenReview ↗

Response to Review LewP

3. On training cost. (For Question 3)

We agree that a discussion on computational overhead is crucial. The overhead comes mainly from cross-fitting, the established technique we use to verify the theory, and we believe the value of the theory does not depend on this technique. We report the measured cost in the new Appendix D.6 (Table 15, reproduced below), including the hardware and the cost of logit export, remapper fit and selection.

  • [Detecting the risk is free] On a class-balanced first stage, our theory assesses the risk at no cost. The criterion of Theorem 1 needs no fold model. It is computed on the training set alone and predicts the direction correctly in all 27 class-balanced settings (Section 5.3).
  • [Data as the limiting resource] Long-tailed recognition is limited by data, and tail data are costly to collect. Our analysis reveals that the training set itself still holds information that existing second stages do not use.
  • [Value of the gain] On a first stage without balancing, the gain remains large. On the cross-entropy first stage, cRT and LWS improve the first stage by 2.34 and 1.89 points. Once memorization is avoided with the out-of-fold fit, these gains rise to 3.70 and 3.42 (Figure 1).
  • [Cost analysis] We find that on CIFAR-100-LT, three fold models already match the gain of five. Across the three datasets, three fold models take 2.1 to 2.5 times the GPU-hours of the full model (table below). Exporting the logits and fitting the correction take about 2% of the training cost. Inference is unchanged, since the correction is absorbed into the classifier head.
  • [Main text] Per your suggestion, Section 5 now summarizes this trade-off.
GPU-hours on a single GPUCIFAR-100-LTImageNet-LTiNaturalist2018
GPUL40S (48 GB)A100 (80 GB)A100 (80 GB)
CPU cores283232
Full model0.841240
Three fold models2.1225.593
Five fold models3.5342.5155
Logit export, remapper fit and selection0.030.753

Finally, the theory only requires logits of images the model was not trained on. Any cheaper way to obtain such logits satisfies the same theory, and we hope our analysis will guide the development of more efficient second stages.

We believe these revisions and the new analyses in the appendix have addressed your concerns and have strengthened the paper. We are happy to provide any further clarification during the discussion period. Thank you once again for your valuable guidance.

Sincerely,

The Authors of Submission 16359

Reviewer_nJfUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the usage of a cross-validation when fitting a new classification-head to the backbone for the long-tail image classification task. The main intuition is that cross validation can disentangle the model bias (mentioned as memorization) from the long-tail data bias and achieve a better head/tail class tradeoff. Also, it performs a theoretical analysis showing the benefit of cross validation during the second stage of head-fitting and it tests this theory across CIFAR-LT, ImageNet-LT and iNaturalist18 datasets showing good performance across imbalance-ratios and generalizability on top of other methods. It achieves achieves competitive results.

Critical strengths

The major benefits are the theoretical analysis and strong performance.

Critical weaknesses

-The paper is poorly written. There are many points that look unreadable like: page 3 l148. "The responses a correction is fit on determine which classification errors it can correct". I think that the current version uses a very difficult to comprehend language.

-The motivation is unclear, reading the introduction I don't understand the problem and why do we need to do cross-validation. I understand that cross-validation should increase the performance but the current motivation is not delivered properly. Some visualizations or visual explanations or intuitive explanations could help to connect the issue of insufficient second-stage head tuning to cross validation and the proposed method.

-The novelty is unclear to me because on the one hand the individual components i.e. two-stage training, logit-adjustment and cross-validation are well known and and on the other, their interplay is not well delivered, nor presented.

Questions for authors

Suggestions: The paper could be significantly enhanced by using a language that is easy to understand and use more visual and intuitive explanations and figures. Also, I believe that the two-stage classifier learning has many previous works and papers thus it is best to do a very comprehensive literature review and comparisons. Maybe a Table that compares the previous works could help.

3 comments
AuthorsOpenReview ↗

Response to Review nJfU

Dear Reviewer nJfU,

Thank you for your careful reading and constructive feedback. We are encouraged that you recognized the theoretical analysis and the strong performance of the paper. We apologize for the lack of clarity in the writing. Per your suggestion, we have added two figures. Figure 1 shows the problem and our improvement on multiple published two-stage methods such as SAMN (CVPR 2026), and Figure 3 illustrates Theorem 1. Following the discovery process below, we have also restructured the abstract and the introduction, and rewritten complex or misleading sentences.

  • [Starting point] In our reproduction, the gain of SAMN over the GLMC first stage is unstable (Figure 1).
  • [Observation] From this, we find that long-tailed second stages are not always effective, because memorization puts them at risk of insufficient correction, and even wrong correction.
  • [Proof] The correction depends on two quantities, the optimal boundary dfd_f and the memorization Δ\Delta. Fitted on the training images (reused logits), it stops at df−Δd_f-\Delta (Theorem 1). The first stages of earlier work were heavily imbalanced, so df>Δd_f>\Delta and the correction showed only an insufficient gain. Stronger first stages lower dfd_f and larger models raise Δ\Delta, so the risk is more likely to appear.
  • [Verification] The theory predicts that a fit on data the model has not seen stops at the optimal boundary dfd_f. We verify this prediction with standard cross-fitting and a simple correction.
  • [Result] As shown in the new Figure 1, multiple second-stage methods such as SAMN all obtain consistent improvements across datasets, imbalance factors and seeds.

Accordingly, we have changed "Memorized Logits" in the title to "Memorization".

Beyond this, we have only added the experiments and proofs requested by the reviewers, and further polished sentences that were hard to read. The theorems, proofs and experimental results are unchanged, and all revisions are highlighted in green.

1. On writing. (For Weakness 1)

Thank you for pointing to the exact sentence. Following your pointer, we have revised the writing throughout the paper. The sentence you pointed out now reads: "Because the first-stage model has memorized the training images, a correction fitted on these images is biased toward the head classes."

2. On motivation and the role of cross-validation. (For Weakness 2)

We apologize for the lack of clarity. Beyond restructuring the abstract and the introduction and revising the writing throughout the paper, we further clarify the problem and the role of cross-fitting below.

  • [Problem] We prove that existing long-tailed second stages are at risk of insufficient correction, and even wrong correction (Theorem 1). This risk appears on multiple published second-stage methods such as SAMN (new Figure 1). Due to memorization, the logit of the ground-truth label is larger on training images than on test images, especially for the tail classes, and thus the head bias estimated on the training images is smaller than the true bias. A second stage fitted on these images therefore removes too little of the bias, and moves in the wrong direction when memorization exceeds the bias (new Figure 3).
  • [Role of cross-fitting] Our theory predicts that the risk disappears once the correction is fitted on logits free of memorization. We verify this prediction with cross-fitting, a simple and established technique, so that the comparison focuses on memorization. In Table 3, with the same models and the same amount of data, the same correction underperforms logit adjustment when fitted on reused logits, and outperforms logit adjustment when fitted on logits free of memorization.
AuthorsOpenReview ↗

Response to Review nJfU

3. On novelty and the comparison with two-stage methods. (For Weakness 3 & Suggestions)

Thank you for this suggestion. We agree that two-stage training, logit adjustment and cross-validation are each well known. Our novelty is identifying a risk that existing long-tailed second-stage methods overlook. Because of memorization, a second stage fitted on the training images can correct too little, and even correct wrongly (Theorem 1). The new Appendix B.5 extends this result to any number of classes.

  • [Comparison table] We have added Table 14 (new Appendix D.5), which covers 11 representative methods in two groups. Prior-based methods such as logit adjustment apply a fixed correction, which cannot accurately represent the head bias that remains in a given first stage. Data-fitted methods, from cRT and LWS (ICLR 2020) to LOS (ICLR 2025) and SAMN (CVPR 2026), instead estimate this bias from the training set. For each method, we list in the table what the second stage adjusts and the data from which the parameters are obtained.
  • [Position] Most data-fitted methods fit their parameters on reused logits. Our work is complementary to such methods. Appended to SAMN, MaxNorm, cRT and LWS, an out-of-fold fit improves each method in all 11 settings of Figure 1.
  • [Interplay of the components] The three components are connected by memorization. In two-stage training, a data-fitted second stage is fitted on reused logits, where memorization enters. Logit adjustment fits nothing on data, so it is unaffected by memorization and serves as the reference. Cross-fitting provides logits free of memorization, on which the fitted second stage rises from below this reference to above it.

We believe these revisions and the new analyses in the appendix have addressed your concerns and have strengthened the paper. We are happy to provide any further clarification during the discussion period. Thank you once again for your valuable guidance.

Sincerely,

The Authors of Submission 16359

Dear Reviewer nJfU,

We hope this message finds you well.

We have posted a response to your comments and revised the paper accordingly. Following your suggestions, we have restructured the abstract and the introduction, added Figure 1 and Figure 3, and added Table 14, which compares 11 representative second-stage methods.

We would greatly appreciate it if you could take a moment to check whether our response has addressed your concerns. We are happy to provide any further clarification.

Sincerely,

The Authors of Submission 16359

Reviewer_VwEBOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how memorization affects post-hoc classifier correction in long-tailed recognition. The authors show that fitting corrections on training-set logits can introduce bias, particularly for rare classes. They propose cross-fitted logit remapping, which uses out-of-fold predictions to improve classifier calibration and performance. Experiments on CIFAR-LT, ImageNet-LT, and iNaturalist2018 demonstrate improvements over several existing methods.

Overall, long-tailed is a practical problem, but i feel this paper introduce too more questions when addressing it.

Critical strengths
  1. Interesting problem: The paper identifies a meaningful issue in post-hoc correction caused by memorization, particularly for minority classes.
  2. Theoretical analysis: The authors provide theoretical explanations for the observed bias and the benefits of out-of-fold correction.
  3. Empirical validation: The method is evaluated on multiple long-tailed recognition benchmarks, with ablation studies supporting the proposed approach.
Critical weaknesses
  1. Limited methodological novelty: The core approach relies on established techniques, including cross-fitting, matrix scaling, and logit adjustment. The main contribution is their combination and application to long-tailed recognition rather than a fundamentally new learning method.
  2. High computational overhead: The method requires training multiple additional fold models. This substantially increases training costs, while improvements on ImageNet-LT and iNaturalist2018 are relatively modest (+1.59% and +0.78%, respectively). A systematic efficiency comparison is needed.
  3. Limited generalizability of theoretical claims: The theoretical analysis primarily relies on simplified binary Gaussian assumptions and a label-shift setting. It remains unclear how well these conclusions extend to realistic multiclass settings and more complex distribution shifts.
Questions for authors

see above

4 comments

Dear Reviewer VwEB,

Thank you very much for your highly valuable comments in this year's heavy reviewing season. We are greatly encouraged that you recognized the "interesting problem" the paper identifies, its theoretical analysis and its empirical validation. We apologize that the lengthy description of the procedure in the abstract obscured our finding: memorization makes long-tailed post-hoc correction insufficient, and even reverses it (Theorem 1). We have restructured the abstract and the introduction following the discovery process below.

  • [Starting point] In our reproduction, the gain of SAMN (CVPR 2026) over the GLMC first stage is unstable (Figure 1).
  • [Observation] From this, we find that long-tailed second stages are not always effective, because memorization puts them at risk of insufficient correction, and even wrong correction.
  • [Proof] The correction depends on two quantities, the optimal boundary dfd_f and the memorization Δ\Delta. Fitted on the training images (reused logits), it stops at df−Δd_f-\Delta (Theorem 1). The first stages of earlier work were heavily imbalanced, so df>Δd_f>\Delta and the correction showed only an insufficient gain. Stronger first stages lower dfd_f and larger models raise Δ\Delta, so the risk is more likely to appear.
  • [Verification] The theory predicts that a fit on data the model has not seen stops at the optimal boundary dfd_f. We verify this prediction with standard cross-fitting and a simple correction.
  • [Result] As shown in the new Figure 1, multiple second-stage methods such as SAMN all obtain consistent improvements across datasets, imbalance factors and seeds.

Accordingly, we have changed "Memorized Logits" in the title to "Memorization".

Beyond this, we have only added the experiments and proofs requested by the reviewers, and further polished sentences that were hard to read. The theorems, proofs and experimental results are unchanged, and all revisions are highlighted in green.

1. On novelty. (For Weakness 1)

We agree that cross-fitting, matrix scaling and logit adjustment are established techniques. Our novelty is identifying a risk that existing long-tailed second-stage methods overlook. Because of memorization, a second stage fitted on the training images can correct too little, and even correct wrongly (Theorem 1). We choose simple components such as matrix scaling to avoid introducing other factors, so that the comparison focuses on the fitting data. Each of five second-stage methods underperforms logit adjustment when fitted on reused logits, and outperforms logit adjustment once only the fitting data are changed to out-of-fold logits (Figure 4). This verifies our theory that the same correction improves performance once memorization is avoided.

2. On computational cost. (For Weakness 2)

We agree that a discussion on computational overhead is crucial. The overhead comes mainly from cross-fitting, the established technique we use to verify the theory, and we believe the value of the theory does not depend on this technique.

  • [Detecting the risk is free] On a class-balanced first stage, our theory assesses the risk at no cost. The criterion of Theorem 1 needs no fold model. It is computed on the training set alone and predicts the direction correctly in all 27 class-balanced settings (Section 5.3).
  • [Data as the limiting resource] Long-tailed recognition is limited by data, and tail data are costly to collect. Our analysis reveals that the training set itself still holds information that existing second stages do not use.
  • [Value of the gain] On a first stage without balancing, the gain remains large. On the cross-entropy first stage, cRT and LWS improve the first stage by 2.34 and 1.89 points. Once memorization is avoided with the out-of-fold fit, these gains rise to 3.70 and 3.42 (Figure 1).
  • [Cost analysis] We find that on CIFAR-100-LT, three fold models already match the gain of five. Across the three datasets, three fold models take 2.1 to 2.5 times the GPU-hours of the full model (new Appendix D.6, table below). Inference is unchanged, since the correction is absorbed into the classifier head.
GPU-hours on a single GPUCIFAR-100-LTImageNet-LTiNaturalist2018
GPUL40S (48 GB)A100 (80 GB)A100 (80 GB)
CPU cores283232
Full model0.841240
Three fold models2.1225.593
Five fold models3.5342.5155
Logit export, remapper fit and selection0.030.753

Finally, the theory only requires logits of images the model was not trained on. Any cheaper way to obtain such logits satisfies the same theory, and we hope our analysis will guide the development of more efficient second stages.

3. On the generality of the theory. (For Weakness 3)

Thank you for this important question. Although the closed form of Theorem 2 relies on the two-class Gaussian model, we have proved that the risk of reversal in Theorem 1 holds for any number of classes and any logit distribution (Proposition 2, new Appendix B.5). Consider a second-stage correction that adds an offset to the logits of the head classes. The fitted offset and the optimal offset have opposite signs exactly when the head classes receive less probability than their class prior on the reused logits and more on test data. Proposition 2 writes this condition as

pˉin,H(0)<∣H∣C<pˉf,H(0),\bar p_{\mathrm{in},\mathcal H}(0)<\frac{|\mathcal H|}{C}<\bar p_{f,\mathcal H}(0),

where pˉin,H(0)\bar p_{\mathrm{in},\mathcal H}(0) and pˉf,H(0)\bar p_{f,\mathcal H}(0) are the average probabilities of the head classes on the reused logits and on test data before any offset, and ∣H∣/C|\mathcal H|/C is their class prior. The condition contains no Gaussian parameter and holds for any number of classes CC. Applying any fraction of the fitted offset then increases the test cross-entropy, as observed in all 12 CIFAR-100-LT settings that satisfy this condition.

  • [Why the reversal generalizes] The reversal follows from the softmax cross-entropy itself, so it needs no Gaussian assumption and holds for any number of classes. Lemma 2 shows where a fitted head offset stops, and Proposition 2 gives the if-and-only-if condition under which the fitted and optimal offsets have opposite signs.
  • [Optimal strength in practice] Without a closed form, the best strength is still found in practice. Across all 27 class-balanced settings, the strength selected on out-of-fold logits alone, without test labels, comes within 0.10 points of the best top-1 accuracy on average (Section 5.3).

In long-tailed recognition, the training set is long-tailed and the test set is balanced, while the images of each class follow the same distribution in both, so our theory covers the task this paper addresses.

We believe these revisions and the new analyses in the appendix have addressed your concerns and have strengthened the paper. We are happy to provide any further clarification during the discussion period. Thank you once again for your valuable guidance.

Sincerely,

The Authors of Submission 16359

Reviewer_VwEBOpenReview ↗

Clarification on memorization, stability, and theoretical novelty

Thanks for your feedback and i still have some questions

Q1. How does the proposed memorization mechanism differ from existing notions of stability and forgetting?

Both the proposed analysis and continual-learning stability compare predictions under different data exposure conditions. To some extend, you are addressing a similar problem with continue learing, that model is learning with {S/x_i} but need to adapt to x_i subsequently. I would be interesting to interprete it from the long-tailed perspective, but make it dificult to understand the difference. The apparent distinction is that continual learning studies the loss of previously acquired information after subsequent updates, whereas this paper studies the in-sample prediction bias created by including xix_i in first-stage training. Could the authors clarify whether the main novelty is specifically the finding that this membership-dependent bias is class-dependent and can reverse a post-hoc long-tail correction, rather than a new notion of stability itself?

Also, it would be helpful to cite papers on sharpness-aware minimization and its application on long-tailed problem.

I will keep my score now before clarifying the difference.

Dear Reviewer VwEB,

Thank you very much for the follow-up and for finding the long-tailed perspective interesting.

Our core novelty is identifying a risk overlooked by data-fitted long-tailed second stages: memorization makes their correction insufficient, and even reverses it. We characterize this risk (Theorem 1 and Proposition 2), and show that a fit on out-of-fold logits restores the correct direction (Corollary 1).

Regarding your question, we would like to clarify it through two questions.

1. Is our paper old wine in a new bottle?

No. The notion of memorization is established; the failure it reveals here is not. We keep both the name and the definition of memorization from Feldman [1], as cited in Section 2 of the submission. This definition compares the predictions on xix_i with and without xix_i in the training set. It does not explain when this difference reverses a long-tailed correction fitted on the training images, nor whether such a fit can expose the reversal. We answer both.

  • [The correction can reverse] The correction depends on half the difference in memorization between the tail and head classes, Δ\Delta, which acts in the opposite direction to the head bias dfd_f that the correction is meant to remove. Theorem 1 proves that a correction fitted on the training images stops at df−Δd_f-\Delta instead of the optimal boundary dfd_f, and reverses toward the head classes once Δ\Delta exceeds dfd_f. For a correction that adds an offset to the logits of the head classes, Proposition 2 (Appendix B.5) gives the if-and-only-if condition for this reversal, for any number of classes and any logit distribution. In line with this, each of five second-stage methods underperforms logit adjustment when fitted on the training images, and outperforms it once fitted on out-of-fold logits (Figure 4).
  • [The training objective favors the reversal] On the training images, the reversed correction is still optimal: its cross-entropy and balanced error are both best at full strength, while its test error rises with strength (Theorem 1). Fitting the correction or selecting its strength on these images therefore favors the reversal.

The theoretical novelty is therefore the finding you describe, that memorization can reverse a long-tailed correction, together with the fact that the training objective still favors this reversal.

2. How does memorization relate to stability and forgetting in continual learning?

We agree with the distinction you draw. Catastrophic forgetting is a loss of previously acquired knowledge [2], whereas memorization is an inflation of the logits of training images. The former is seen as an accuracy drop on previously learned tasks, and the latter as overconfidence on the very images the model was trained on.

  • [No subsequent update] Continual learning trains a model on new tasks and asks whether it retains previously acquired knowledge. This retention is the stability side of the stability–plasticity dilemma [2]. In our setting, all models are trained once and never updated afterward, so no model learns on S∖xiS\setminus x_i and then adapts to xix_i, and the first-stage model loses no previously acquired knowledge.
  • [Memorization in long-tailed recognition] Instead, the logits of the first-stage model on its own training images are inflated relative to unseen images, so the head bias that a long-tailed second stage is meant to remove looks smaller on these logits than it is.

3. On sharpness-aware minimization.

Thank you for suggesting the works on sharpness-aware minimization. We have added them to Section 2, highlighted in green. Sharpness-aware minimization seeks flat minima to improve generalization [3], and its applications to long-tailed recognition improve the generalization of tail classes during training [4, 5]. ImbSAM [5] finds that the generalization bottleneck of class-imbalanced recognition "primarily lies in the severe overfitting for tail classes with limited training data", which agrees with the memorization we measure. These methods improve the training of the first stage, whereas our analysis concerns the logits on which the second stage fits its correction.

Thank you again for your suggestions, which have greatly improved the paper. In preparing this response, we have also learned a lot. We would be grateful for your further feedback.

Sincerely, The Authors

[1] Vitaly Feldman. Does learning require memorization? A short tale about a long tail. In STOC, 2020.

[2] Matthias De Lange et al. A continual learning survey: Defying forgetting in classification tasks. TPAMI, 2021.

[3] Pierre Foret. Sharpness-aware minimization for efficiently improving generalization. In ICLR, 2021.

[4] Harsh Rangwani. Escaping saddle points for effective generalization on class-imbalanced data. In NeurIPS, 2022.

[5] Yixuan Zhou.ImbSAM: A closer look at sharpness-aware minimization in class-imbalanced recognition. In ICCV, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.