acceptodds
Under review as a conference paper at ICLR 2027

Beyond Standalone Accuracy: Class-Frequency Correspondence Shapes Conditional Expert Gain in Long-Tailed Recognition

Abstract

Evaluating an additional expert in multi-expert long-tailed recognition requires accounting for the correctness already covered by the existing expert set. Standalone accuracy cannot distinguish whether an auxiliary expert repeats existing coverage or recovers residual errors. We therefore define conditional expert gain to measure correctness contributed beyond existing coverage and establish a set-conditioned view of complementary expert value. Using this view, we identify class-frequency correspondence as a key factor shaping residual specialization, the distribution of newly recovered correctness across the class-frequency spectrum. Controlled correspondence interventions reveal an accuracy-gain inversion: reversing the class-to-frequency assignment raises standalone accuracy while sharply reducing conditional gain and shifting recovery away from rare classes. Class-accuracy-preserving analysis further shows that this gain shift is not explained by class-wise accuracy allocation alone, revealing an additional within-class residual-overlap contribution. Guided by this insight, we propose Frequency-Structured Auxiliary Learning (FSAL), which uses a shared correspondence to orient frequency-dependent auxiliary scores within the inherited two-stage structure and expert budget, enforcing one class-wise frequency assignment across the auxiliary objective. On CIFAR-100-LT, FSAL achieves 59.5% Overall Top-1 accuracy and 44.1% Few-class accuracy at IR=100, with the highest Overall Top-1 among the compared methods at all three imbalance ratios. The same correspondence-dependent recovery shift extends to ImageNet-LT, while results on iNaturalist 2018 further support transfer under natural class imbalance. Code is available at Supplement.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long-tailed learning, Multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tGzmOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies auxiliary-expert complementarity in long-tailed recognition by conditional expert gain. Controlled class-frequency reassignment shows an accuracy–gain inversion and changes in tail-error recovery. The proposed FSAL improves recognition performance across three benchmarks.

Critical strengths

The accuracy–gain inversion is an interesting observation, supported by fixed-reference interventions, graded correspondence changes, and representation controls.

FSAL achieves consistent benchmark improvements, particularly on underrepresented classes.

Critical weaknesses

FSAL combines frequency-aware mechanisms, but ablations are insufficient to isolate the benefit of the proposed shared result beyond the combined objective. Hyperparameter sensitivity does not resolve this concern.

Conditional gain measures correctness coverage rather than ensemble improvement. Although Appendix B acknowledges this distinction, its practical value for selecting experts under the actual aggregation rule remains insufficiently explained.

Questions for authors

Can you isolate the contributions of temperature, prior adjustment, margin, and deferred weighting under matched training and tuning budgets?

How reliably does conditional gain rank candidates by their actual incremental ensemble accuracy?

Table 3 implies an aligned–reversed gain difference of 6.92 percentage points, whereas Table 9 reports 3.33. Which evaluation settings explain this difference?

Reviewer_L5RwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how class-frequency correspondence affects the complementary value of auxiliary experts in long-tailed recognition. The authors introduce conditional expert gain to quantify the additional correct predictions contributed by an expert beyond an existing expert set. They further analyze how class-frequency correspondence shapes residual error recovery and propose Frequency-Structured Auxiliary Learning (FSAL), which uses a shared correspondence to guide class-frequency-dependent logit transformations and deferred reweighting. Experiments on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 demonstrate the impact of class-frequency correspondence on expert complementarity and show that FSAL achieves competitive classification performance.

Critical strengths
  1. Frequency-Structured Auxiliary Learning (FSAL) is well motivated by the finding that class-frequency correspondence affects which classes' residual errors an auxiliary expert can correct. The use of a shared correspondence for logit transformation and deferred reweighting closely aligns the method with this finding.
  2. Controlled experiments comparing aligned, permuted, and reversed correspondence, together with graded correspondence sweeps, cross-Stage-I representation evaluations, and a cross-objective comparison with KPS, provide supporting evidence for the consistency of the observed correspondence-dependent effects.
Critical weaknesses

The relationship between conditional expert gain and final ensemble performance remains insufficiently established. Although Appendix B.1 acknowledges that additional correctness coverage need not translate directly into ensemble accuracy gains, the paper does not empirically quantify their relationship. Consequently, the extent to which conditional expert gain reflects an auxiliary expert's actual contribution to final ensemble performance remains unclear.

Questions for authors
  1. Could the authors provide qualitative visualizations of samples recovered under different correspondence settings to illustrate the underlying mechanism and the resulting performance differences?
  2. Could the authors provide component-wise ablations to isolate the contributions of the main terms in the FSAL objective?
  3. Could the authors further analyze the relationship between conditional expert gain and final ensemble accuracy to clarify the practical value of this metric for evaluating multi-expert systems?
Reviewer_cH4POpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies an issue in multi-expert long-tail recognition: when the system already has several classification experts, how should the value of a new expert be evaluated? The author points out that the traditional single-model classification accuracy cannot measure whether a new expert truly complements the samples that the existing experts cannot correctly identify. To this end, the paper proposes Conditional Expert Gain, which measures the complementary value by calculating the proportion of samples that the new expert can correctly identify but the existing expert set has not yet identified. Furthermore, the authors investigate the correspondence between class identity and training sample frequency (Class-Frequency Correspondence) and propose Residual Specialization to describe the distribution of the new expert's correct recognition ability across different class frequencies. In terms of methodology, the paper proposes Frequency-Structured Auxiliary Learning (FSAL), which controls category-related temperature, prior adjustment, margin, and delayed sample reweighting through a unified category frequency mapping, thereby improving the learning process of auxiliary experts. The authors conducted experiments on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018. Table 1 shows that under the conditions of IR=100 in CIFAR-100-LT, FSAL achieves 59.5% Overall Top-1 and 44.1% Few-class Accuracy, which is an improvement of 2.7 and 4.2 percentage points respectively compared to TS-MOF's 56.8% and 39.9%. The paper also investigates the changes in expert complementarity through category frequency mapping interventions, different evaluation priors, and various model configurations.

Critical strengths

1: Evaluating the complementary value of new experts based on the existing expert set provides a meaningful analytical perspective for multi-expert long-tail recognition. Traditional classifier evaluation mainly focuses on independent accuracy, but in multi-expert systems, experts with higher independent accuracy may not necessarily bring greater system benefits. If two experts predict correctly on exactly the same samples, then the value of the second expert in expanding the coverage of correct predictions may be very small. Conditional Expert Gain explicitly characterizes the contribution of a new expert to the existing correct prediction coverage. This perspective is particularly meaningful for long-tail recognition, as auxiliary experts may specifically recover samples in rare categories that have not been correctly identified by other experts, even when the overall accuracy is not high. The paper further discusses the set dependence of conditional gains and the submodular properties of the coverage function, and distinguishes the complementary variations from different sources through category condition decomposition. These analyzes not only propose a new loss function but also attempt to explain the formation mechanism of complementarity in multi-expert systems.

2: The intervention experiments on the correspondence of category frequencies are quite systematic, and performance improvements have been achieved on multiple long-tail datasets.

Critical weaknesses

1: Conditional Expert Gain measures the ideal correctness coverage, rather than the classification gain that an actual multi-expert reasoning system can achieve. The connection between the two still requires more direct validation.

This is the issue I believe is most worth discussing in depth in the paper. Paper definition: $

G(e∣S)=B(S∪{e})−B(S)G(e\mid S)=B(S\cup\{e\})-B(S)

$

Among them, �1��1� represents the proportion of samples for which at least one expert in the expert set made a correct prediction. This definition is clear and reasonable in mathematical terms. However, in practical reasoning, the system usually cannot know in advance which expert is correct on the current sample. Therefore, adding new experts increases the theoretical coverage of correct predictions, but it does not mean that the system can directly utilize these additional correct predictions. For example, a certain auxiliary expert may correctly predict the samples that the existing experts fail on, but if the final fusion strategy cannot identify situations where this auxiliary expert should be trusted, then this complementarity may not necessarily translate into an improvement in the final Top-1 accuracy. The theoretical analysis of the paper mainly revolves around coverage and residual error recovery, while the final system performance still relies on the expert fusion mechanism of TS-MOF. Although the experiments indeed show that FSAL improves the final classification accuracy, the paper does not sufficiently establish the following relationship: Increased conditional expert gain → Fusion mechanism effectively utilizes complementary predictions → Improved final classification accuracy. Therefore, the current conditional gain is more suitable to be interpreted as a potential complementarity indicator, rather than being directly equated to the additional expert gain in actual deployment.

This difference is important because it directly affects the practical significance of the core analytical metrics of the paper. 2: FSAL combines various existing frequency-aware techniques under a unified correspondence, but current experiments are not sufficient to fully prove that the "shared correspondence" is the key source of performance improvement. FSAL uses multiple frequency-related mechanisms:

  • Category-related temperature adjustment.
  • Logit adjustment related to prior probabilities.
  • Margin related to category frequency.
  • Delayed sample reweighting.

The basic ideas of these techniques have been extensively studied in long-tail recognition. The core methodological innovation of the paper is to unify category frequency mapping, allowing these mechanisms to share a consistent category direction. This design has a certain degree of rationality, but it is necessary to further distinguish between two different sources of improvement: The first is the performance gains brought by combining multiple existing long-tail learning techniques. The second is the additional benefits brought by using a unified category frequency correspondence for these techniques. The current experiment demonstrates the variations in category correspondence under normal, random, and reverse settings, and a related target analysis is also provided in the appendix. But merely proving that the frequency correspondence of errors reduces performance does not sufficiently demonstrate that the unified correspondence combination method is superior to the reasonably configured conventional frequency-aware targets. Especially since the method in the paper inherits the two-stage framework of TS-MOF, it is necessary to more directly compare versions that use the same components but do not adopt the FSAL unified design. If this factor cannot be clearly isolated, the contribution of the FSAL method may be more akin to an effective integration of existing frequency-aware objectives rather than a new auxiliary expert learning mechanism.

Questions for authors

Question 1: What is the quantitative relationship between the conditional expert gain and the classification gain after actual fusion? Currently, �0��0� measures the incremental coverage of the set of correct predictions by the expert. It is recommended that the authors report the conditional gain, the change in Top-1 performance after actual fusion, and the upper bound achievable by ideal Oracle fusion in the same experiment. Especially, we hope to see whether the changes in conditional gain under different category frequency correspondences can reliably predict the final performance changes of the real system. This will help demonstrate that conditional expert gain is not only a useful analytical metric but also has direct implications for the design and selection of practical multi-expert models.

Question 2: Can you provide a complete frequency-aware target decomposition experiment to verify the independent contribution of the unified correspondence? FSAL also includes temperature, prior adjustment, margin, and delayed reweighting. I hope the authors will compare the performance of using each module individually, gradually combining the modules, and when all modules share or do not share the same frequency correspondence. Especially, when each component uses the true category frequency information, does FSAL still significantly outperform the simple combination of these existing methods? This experiment will directly affect my judgment on the novelty and necessity of the method.

Question 3: Is it possible that Conditional Expert Gain is strongly influenced by the class distribution and the existing experts' coverage capabilities? Although the paper has already analyzed different evaluation priors, in practical use, the existing expert set may have different sizes, different coverage, and different error correlations. Can we systematically change the scale and capability of the reference expert set while keeping the newly added experts fixed, in order to study the stability of the conditional gain? It is especially important to clarify whether the auxiliary experts generated by FSAL have good complementary value under different existing expert combinations, rather than being applicable only to the specific expert configuration in the paper.

Comments on the paper

1 comment

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank all reviewers for their thoughtful evaluations and constructive discussions. We are particularly encouraged by their recognition of the significance of our study of auxiliary-expert complementarity, the accuracy–gain inversion, and the role of class-frequency correspondence in long-tailed recognition.

Our work makes three connected contributions:

  • A new perspective on auxiliary-expert value. We introduce Conditional Expert Gain (CEG) to measure the additional correct predictions an auxiliary expert contributes beyond an existing expert set, providing a set-conditioned characterization of expert complementarity.
  • A finding on class-frequency correspondence. Through controlled correspondence interventions, we uncover the accuracy–gain inversion and show how class-frequency correspondence shapes residual expert specialization across frequency groups.
  • A method built on this insight. We propose Frequency-Structured Auxiliary Learning (FSAL), which uses shared class-frequency correspondence to coordinate frequency-dependent objective transformations and achieves competitive performance across three long-tailed benchmarks.

The discussion has further strengthened the empirical support and presentation of these contributions. Our additional analyses clarify the relationship between CEG and realized ensemble gains, distinguish objective formulation from correspondence coordination, and provide further evidence on component contributions. In the revised manuscript, we will make these distinctions clearer and include the requested qualitative visualizations of recovered samples.

We are grateful for the reviewers' and ACs' time, expertise, and thoughtful engagement. Their feedback has helped us better communicate the significance of the problem, the insight revealed by our analysis, and the contribution of FSAL.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.