Memorized Logits Reverse Post-Hoc Correction in Long-Tailed Recognition

Two-stage methods for long-tailed recognition freeze a trained backbone and refit the classifier on the backbone's own training images. When the first stage is already class-balanced, this refit favors the head classes it is meant to rein in. The cause is memorization. A model inflates the true-class logit of its training images, most for the rarest classes, and its training-set outputs therefore overstate the tail. In a Gaussian response model we prove that a fitted correction takes its direction from the center of the logits it is fit on. Memorized logits send it toward the head, out-of-fold logits toward the tail, and the optimal strength has a closed form. Cross-fitted logit remapping therefore scores each training image with a fold model trained without it, fits a matrix-scaling correction on those logits, and folds it into the classifier head. It ranks first among all compared methods on CIFAR-LT with further gains on ImageNet-LT and iNaturalist2018, whereas the same correction fit on memorized logits underperforms logit adjustment. Where a correction is fit matters more than which correction is fit. Code is available at https://anonymous.4open.science/r/crossfit_remap-BEC8.