acceptodds
Withdrawn from ICLR 2027

CAFE: Closed-Loop Classifier-Aware Feature Enrichment for Long-Tailed Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-tailed recognition remains challenging due to classifier bias induced by highly imbalanced training data. Existing information augmentation methods often overlook which tail samples actually require augmentation and whether generated features remain beneficial as the classifier evolves. To address these issues, we propose Classifier-Aware Feature Enrichment (CAFE), a two-stage classifier fine-tuning framework that establishes a closed-loop augmentation process. Guided by the evolving classifier, CAFE identifies challenging tail samples and selectively augments them using informative features transferred from data-rich classes. The generated features are continuously evaluated, and beneficial synthetic features are retained while redundant or detrimental ones are removed. The updated classifier subsequently guides the next round of sample identification and feature enrichment, allowing the augmentation process to adapt continuously to the evolving classifier. Extensive experiments on CIFAR10-LT, CIFAR100-LT, ImageNet-LT, and iNaturalist 2018 demonstrate the effectiveness of CAFE and show that it achieves competitive performance against state-of-the-art methods. Code will be made publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long-tailed recognition, image classification, imbalanced learning, deep learning.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7PYqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper deals with the long-tailed recognition problem. Different from previous works which performs tail feature augmentation in a static and classifier-agnostic way, this paper proposes a framework to determine which tail samples need to be augmented during training, which is dynamic and classifier-aware. Extensive experiments verify the effectiveness of proposed method.

Critical strengths
  • This paper is generally well written and easy to follow.

  • The high-level idea is reasonable. Performing tail feature augmentation in a dynamic view and classifier-aware way is reasonable and is rarely studied in previous works.

Critical weaknesses
  • The proposed method is too heuristic and the whole framework is too complex. The designed metric for tail sample selection is heuristic. From someone, it is hard to say whether it is better than other alternatives, e.g., in Eq. (7), using previous defined mm may be better? The minor empirical improvement cannot fully justify the induced complexity. There are too many hyper-parameters to determine. I doubt whether it can really work in practice.

  • The complex design is not fully justified. The authors should conduct comprehensive ablations to justify each designed component. Otherwise, it is hard to understand the necessity and superiority of a specific component.

Questions for authors
  • Some details are unclear or not described. E.g., section 3.3.1, the author states the donor class is selected based on the metric defined in Eq. (7). But the author didn't provide how to select. For section 3.3.2, the authors didn't clearly described how αt\alpha_t and MM determined.

  • The necessity of aux. real classifier should be justified and discussed. The aux. real classifier has effect in section 3.2 and 3.4.2. I just doubt if it is necessary to introduce such a classifier. It is also possible to compute relevant metrics with only one classifier. How much benefits will this choice bring?

Reviewer_epc4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Classifier-Aware Feature Enrichment (CAFE), a second-stage classifier fine-tuning framework for long-tailed recognition with a frozen backbone. A Detector uses the averaged logits of a primary classifier and a real-data-only auxiliary classifier to identify misclassified or low-margin tail samples. A Generator enriches these samples through channel-wise soft fusion with head- or medium-class donors, guided by prototype similarity and the current rival class. A Curator checks candidate validity and evaluates candidate subsets through virtual updates of the auxiliary classifier, measuring class-balanced cross-entropy risk changes on a support set drawn from real training features. Accepted features enter a synthetic memory and are subsequently re-evaluated and removed when they cease to be useful. Experiments cover CIFAR10-LT, CIFAR100-LT, ImageNet-LT, and iNaturalist 2018, using different Stage-I methods, component ablations, lifecycle diagnostics, and three-run results on CIFAR-LT.

Critical strengths
  1. A meaningful treatment of synthetic-feature usefulness as something that changes during classifier training. CAFE considers which samples need enrichment, whether generated features help classification on real data, and whether previously accepted features remain useful. The auxiliary classifier and class-balanced risk criterion provide a concrete mechanism for evaluating candidate updates beyond feature similarity or local margin improvement. This is a useful direction because features that help at one point in training need not remain helpful as the classifier changes. The comparison with V5 provides initial evidence for the value of continuing to reassess features after their initial acceptance.

  2. Broad reported improvements and an ablation design that targets relevant alternatives. CAFE improves several Stage-I methods across standard benchmarks. For example, BCE3S+CAFE improves CIFAR10-LT and CIFAR100-LT accuracy at IF=100 by 2.51 and 1.80 percentage points, respectively, while GLMC+CAFE improves ImageNet-LT overall accuracy from 56.30 to 57.88. Figure 1 includes comparisons with other Stage-II methods using the same CE-trained backbone, and Table 8 provides three-run CIFAR results consistent with the main-table averages. The ablation design targets relevant alternatives, including real-only fine-tuning, hard-example replay, uncurated generation, and permanent retention of accepted features.

Critical weaknesses
  1. Important operational definitions and experimental protocols are insufficiently specified or inconsistently reported.

    The actual CIFAR class partition needs clarification. Section 4.1 defines tail classes as having at most 20 training examples. Under the standard CIFAR-LT construction, the smallest CIFAR10-LT class contains 50 examples even at IF=100, and the smallest CIFAR100-LT class at IF=10 also contains 50. Thus, none of the reported CIFAR10-LT settings, nor CIFAR100-LT at IF=10, contains a tail class under the stated rule; CIFAR10-LT at IF=10 contains no medium class either. These sets enter the Detector quantile, donor selection, and tail-risk constraint, so the ambiguity affects the algorithm itself. A different operational partition may have been used, but it needs to be specified.

    Other details that determine the method’s behavior also need clarification: the precise Consistent criterion, candidate-subset construction, support-set size and resampling, numerical Curator thresholds, and memory capacity and handling of a full memory. Appendix A.3 states that λsyn\lambda_{\mathrm{syn}}, ρsyn\rho_{\mathrm{syn}}, and ηfloor\eta_{\mathrm{floor}} are set independently for each dataset without reporting their values or selection protocol.

    There are also discrepancies between the figures and tables. Figure 1 is labeled IF=100, but its CE and CAFE values match the IF=10 entries in Table 1. Figure 6 reports full-model accuracies of 79.39 and 72.39 at IF=50 and IF=100, compared with 79.71 and 72.87 in Tables 1 and 8. These gaps are 0.32 and 0.48 percentage points, whereas the corresponding ranges across the three reported runs in Table 8 are approximately 0.01 and 0.02 points. The paper should reconcile the checkpoints, configurations, and aggregation used in these figures and tables, and specify which sources of randomness vary across the repeated runs.

  2. The evidence does not yet sufficiently establish the utility proxy’s relevance to the primary classifier or the benefit of generation over meaningful hard-example replay.

    The Curator evaluates virtual updates at the auxiliary classifier fϕf_\phi on a support set sampled from real training features, whereas accepted synthetic updates are applied to the primary classifier fθf_\theta. A positive risk reduction at fϕf_\phi on this support set does not by itself establish a benefit at fθf_\theta on independent data. Table 5 reports accuracy changes for accepted and rejected candidates, but does not identify the evaluation data, classifier, or candidate counts clearly enough to resolve this question.

    The replay control V3 also has an important ambiguity. Appendix A.2 states that selected real features are directly replayed while the Curator remains active. However, Section 3.4.1 describes Consistent in terms of target–rival margin improvement under both classifiers. If improvement is measured against the source feature, an unchanged replayed feature cannot strictly improve its margin relative to itself. The treatment of ties, the checks retained or bypassed in V3, and the amount of additional replay therefore matter. Without this information, V3 does not yet provide a clear basis for attributing the improvement to generated features beyond hard-example replay.

Minor comment on positioning: The donor-selection and fusion components should be positioned more explicitly relative to Chu et al. (ECCV 2020, Feature Space Augmentation for Long-Tailed Data) and H2T. Chu et al. use class-level confusion information to guide donor selection, whereas CAFE uses the current sample-level rival. Setting αt=1\alpha_t=1 in Eq. (8) gives channel replacement, closely related to H2T. Explaining these connections would help clarify what the sample-level selection, utility screening, and continued re-evaluation add to existing feature augmentation methods.

Additional discussion: From an optimization perspective, CAFE can be interpreted as proposing feature perturbations and selecting the induced classifier updates subject to class-balanced real-data risk constraints. A first-order expansion of Eqs. (12)–(14) relates the utility score to alignment between synthetic and real-data gradients, while finite-step utility also depends on curvature and update scale. Memory retention then determines the cumulative training exposure assigned to the accepted candidates. This perspective provides a concrete explanation for why usefulness can change during training and helps distinguish the roles of generation, utility screening, and lifecycle management. It would be useful to discuss whether lifecycle management mainly removes updates that have become harmful, reallocates training exposure toward newly useful candidates, or combines both effects.

Questions for authors
  1. Operational definitions and configuration. What class partitions are actually used for augmentation targets, donors, and groupwise utility constraints on each CIFAR-LT setting? Please provide the precise Consistent criterion, candidate-subset and support-set construction, numerical Curator thresholds, memory configuration, and dataset-specific synthetic-update parameters, together with the hyperparameter-selection protocol. Reporting consistency and run scope. Please reconcile Figure 1 and Figure 6 with Tables 1 and 8. Do the runs in Table 8 vary the Stage-I checkpoint, the long-tailed training subset, Stage-II randomness, or some combination of these?

  2. Interpretation of the replay and lifecycle controls. How does V3 apply Consistent when the replayed feature is identical to its source? On one representative setting, could the authors clarify or revise this control so that replayed real features bypass source-relative margin-improvement checks while retaining utility screening, and compare it with V1, V4, and full CAFE using the same Stage-I checkpoint and comparable classifier-update budgets (reusing existing runs where possible)? Please report primary-classifier accuracy, admitted sample counts, and actual update counts, and clarify the memory policies used in V3 and V5, including what happens when the memory is full.

  3. Transfer of the utility signal and the update being evaluated. Which classifier, evaluation data, and candidate sampling procedure are used in Table 5? Does the virtual update in Eq. (12) include momentum and the post-update MaxNorm operation specified in Appendix A.3, or does it intentionally use a simplified gradient step? Using candidate subsets and checkpoints from existing runs, could the authors examine whether the original auxiliary-classifier utility scores predict risk reductions after the corresponding update to the primary classifier under its actual optimizer, evaluated on held-out real data that do not participate in acceptance? Comparing utility-accepted and utility-rejected candidates that have passed the validity checks, with their counts, mean risk changes, and positive-utility proportions, would directly strengthen the evidence for the Curator.

Reviewer_b8dVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Classifier-Aware Feature Enrichment (CAFE), a Stage-II classifier fine-tuning method for long-tailed recognition. After freezing a Stage-I feature extractor, CAFE repeatedly:

(i) detects difficult tail examples using the target-versus-rival margin averaged over a primary classifier and an auxiliary classifier trained only on real features;

(ii) creates candidate tail features by channel-wise fusion with features from head/medium classes;

(iii) filters and maintains the candidates using feature-validity constraints and a virtual classifier update evaluated on a class-balanced support set.

The retained synthetic memory is used to update the primary classifier, whose new state guides the next round, yielding the claimed closed loop. Experiments on CIFAR10-LT, CIFAR100-LT, ImageNet-LT, and iNaturalist 2018 report consistent improvements over the corresponding Stage-I models, with component ablations, sensitivity studies, lifecycle visualizations, computational-cost analysis, and three-run CIFAR results.

Critical strengths

(i) The closed-loop formulation addresses a genuine limitation of static feature augmentation. CAFE conditions both which tail samples are augmented and which synthetic features survive on the evolving classifier. This is consequential because it gives the method a coherent motivation beyond indiscriminately enlarging every tail class, and the dynamic-memory ablation (V5 versus V6) suggests that re-evaluation/removal contributes beyond initial candidate screening.

(ii) The method is evaluated across several standard long-tailed benchmarks and Stage-I representations. CAFE improves CE and BCE3S on CIFAR-LT and improves GLMC/ProCo-based models on ImageNet-LT or iNaturalist 2018. The paper also includes component ablations, sensitivity analysis, cost measurements, and three independent CIFAR runs. This breadth makes the empirical trend more credible than a result tied to one dataset or one representation learner.

Critical weaknesses

(i) The incremental contribution over H2T is not sufficiently isolated. H2T already transfers feature channels from head classes to tail classes during Stage-II classifier adjustment, and CAFE adopts a closely related head-to-tail feature-generation mechanism. CAFE conceptually extends this process with classifier-aware defect detection, candidate curation, and dynamic memory maintenance; however, its directly reported improvement over H2T is modest. The current experiments do not provide a strictly matched comparison that progressively adds these components to the same H2T-style generator. Therefore, it remains unclear how much of the gain is specifically attributable to the proposed closed-loop lifecycle rather than differences in implementation, hyperparameter tuning, or evaluation protocol.

(ii) The robustness and tuning burden of the threshold-heavy framework are insufficiently evaluated. CAFE introduces several interacting hyperparameters, including the fusion strength �0��0�, defect-selection quantile �1��1�, and the Curator thresholds �2��2�, �3��3�, �4��4�, �5��5�, and �6��6�. The paper reports sensitivity analyses only for �7��7� and �8��8�, while the remaining thresholds directly determine candidate admission and memory maintenance. It is unclear whether these values are fixed across datasets, imbalance factors, backbones, and Stage-I methods, or tuned separately for each setting. Since these thresholds may interact, the current analysis is insufficient to establish the robustness and practical tuning cost of the complete framework.

Questions for authors

Could the authors clarify the essential algorithmic difference between CAFE and H2T beyond adding the Detector and Curator? More importantly, could the authors provide a matched experiment using the same Stage-I checkpoint, H2T-style generator, training schedule, checkpoint-selection rule, and evaluation protocol, and then progressively add classifier-aware detection, initial curation, and dynamic memory removal? This would clarify whether the closed-loop lifecycle provides a consistent gain over head-to-tail feature transfer itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.