acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Class Reweighting through Temporal Evidence for Long-Tailed Recognition

Abstract

Long-tailed audio recognition typically measures class imbalance by training sample counts per class. This view overlooks how useful information is distributed within each sample: the labeled sound may occupy only a small portion, while persistent or repeated patterns make other portions redundant. Classes with identical sample counts therefore differ in the amount of irrelevant or repeated content. To account for recording content in class reweighting, we propose Constrained Effective Temporal Evidence Learning (CETE) for long-tailed audio recognition. CETE introduces a temporal head to identify temporal features that support the class decision. The head is trained with a loss that combines classification constraints with a penalty on the total temporal weight. This loss encourages correct classification after feature weighting while discouraging large weights on unnecessary background or repeated features. A teacher model selects correctly and confidently classified examples for learning temporal weights and computing class statistics. CETE aggregates their temporal weights within each class and combines the resulting statistics with sample counts to determine class loss weights, reducing bias toward common classes. Experiments on VGG-Sound-LT, iNatSound, and BirdCLEF demonstrate CETE's consistent superiority over the baselines and its robustness across controlled imbalance ratios and naturally long-tailed data.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
long-tailed learning, audio recognition, class reweighting

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_PffUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the problem that long-tailed audio methods measure class size by counting clips, which can be misleading because a labeled sound may fill only a small part of a clip or repeat the same content multiple times. The authors propose a small network that scores each time slice of a clip: keep as few slices as possible while still getting the correct prediction. A slowly updated copy of the model decides which clips to use. While the idea is nice and intuitive, the reported gains are small and reported only for a single seed.

Critical strengths

The motivation is clear, with an easy-to-adapt solution requiring only a change to the class weights and the addition of a small head. The extra head should not hurt performance. No extra cost at test time, and the training overhead is provided.

Critical weaknesses

All results come from a single seed, while the gains over the best baseline are close to typical variation. Please report mean with std over e.g., 3 random seeds.

Clips rejected by the teacher contribute zero to v_c. As shown in Fig. 4, since the teacher accepts about 90% of Many-class clips but only about 67% of Few-class clips, hard classes get a lower v_c and therefore a higher weight, independent of their temporal content. Therefore, a method may, to some degree, act as difficulty-based reweighting. I would appreciate adding a control where all temporal weights are set to 1, so that v_c only counts accepted clips.

The ablation in Table 3 shows that count-based weights alone already match the best baseline. Please include the count-only variant in the main tables to see how much the temporal idea adds.

Table 3 removes one piece of CETE at a time to show which pieces matter, but only on BirdCLEF. I would appreciate adding one more dataset.

Take an audio clip, e.g., with one bird call, and a copy where the same call is looped five times. The theory implies that entirely repeated content adds no effective evidence. Does CETE give both clips the same contribution, or more to the looped one? I would love to see an additional analysis of v_c / n_c vs. the repetition ratio.

Minor: please consider adding "audio" to the title to make it more informative, as the method and all experiments are audio-only.

Questions for authors

Why a linear formula n~c=(1−α)×nc+α×vc\widetilde{n}_c = (1−\alpha)\times n_c + \alpha \times v_c? Is α\alpha tuned per dataset or shared? Why does it equal γ\gamma ?

Is it really temporal? The head looks at one slice at a time, and the classifier averages all slices, and an average does not care about order. So, as I understand it, given the encoder output, CETE treats a clip as a bag of slices. If the encoder's slices were shuffled before the head and pooling, would CETE's weights and statistics change?

How does CETE treat classes defined by context or acoustic scene (e.g., street, crowd, rain), where the content is the background rather than a foreground event?

Is "w/o Temporal" still training the temporal head and teacher, or are they removed?

Which dataset was used in Fig. 4?

Reviewer_yReLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Constrained Effective Temporal Evidence Learning (CETE) for long-tailed audio recognition. The motivation is that recording counts overlook differences in target occurrence and repeated content within recordings. CETE learns class-conditional temporal weights through a compactness penalty and two constraints: preserving the original classification margin and making the retained features more predictive than their complement. An EMA teacher selects correctly and confidently classified recordings for temporal-head training and classwise aggregation. The aggregated temporal weights are combined with recording counts to determine class loss weights.

The theoretical analysis relates weighted feature-mean estimation to covariance-dependent effective evidence under stated assumptions. Experiments on VGGSound-LT, iNatSound, and BirdCLEF, together with component ablations, controlled soundscape experiments, and additional backbones, evaluate the approach.

Critical strengths
  1. A relevant motivation and a practical approach to incorporating recording content into class reweighting. Differences in target occurrence and repeated content are meaningful considerations for weakly labeled audio, and recording counts alone do not describe them. CETE provides a concrete way to incorporate temporal selections without requiring event-boundary annotations. The theoretical results are stated with appropriately clear scope relative to the practical method. The auxiliary components are discarded after training, leaving the student architecture unchanged and adding no inference parameters or FLOPs.

  2. Consistent reported improvements and useful controlled evaluations. CETE improves balanced accuracy over the strongest compared baseline by 0.9–1.7 percentage points across the five main settings in Tables 1–2. Table 3 includes a matched count-only control: on BirdCLEF, the full method improves ALL accuracy from 60.0 to 61.4 and Few accuracy from 44.7 to 47.4. The synthetic experiments hold recording counts equal while varying temporal composition, directly addressing the paper’s motivation. Together with the component ablations and additional backbone results, these experiments provide meaningful evidence that the complete method is useful.

Critical weaknesses
  1. The contribution of temporal information is not sufficiently distinguished from teacher-based class difficulty. The statistic in Eq. (15) can be decomposed as vc⋆/nc=Rcsˉcv_c^\star/n_c = R_c\bar{s}_c, where RcR_c is the teacher acceptance rate and sˉc\bar{s}_c is the mean selected temporal mass among accepted recordings. Holding temporal mass and other quantities fixed, lower acceptance reduces this target statistic and therefore drives stronger relative reweighting, even without class-specific temporal information. The initialization and updates also ensure 0≤vc≤nc0 \leq v_c \leq n_c, so Eq. (17) can only lower adjusted counts and increase the corresponding unnormalized inverse-count weights. The existing removal ablations do not establish what the learned temporal weights add beyond this difficulty signal: removing reliability changes both temporal-head training and aggregation, while setting α=0\alpha=0 removes the complete correction.

    The interpretation of temporal redundancy also needs more direct support. In the reference architecture, temporal mean pooling and a linear classifier make both masked margins depend on weighted sums of per-frame contributions against competing classes. Sequences with identical per-frame contributions against every competitor therefore induce the same per-recording constrained selection problem. For example, in a binary case with identical frames, zero classifier bias, and a unit original margin, satisfying the preservation constraint with the default tolerance requires an average retained weight of at least 0.98. The same bound is attained by a sequence of distinct features with identical per-frame margin contributions, so the ideal constrained statistic cannot distinguish these two cases. The constrained objective therefore does not by itself guarantee sensitivity to temporal redundancy. The accuracy results in the repetition experiment do not establish whether the learned temporal statistic captures redundancy beyond changes in teacher confidence.

  2. The stability of the incremental improvements is insufficiently established. Appendix B.2 specifies a single seed for the main experiments, and Tables 1–4 do not report variation across training runs. The consistent direction of improvement across settings is encouraging, but the gains over the strongest baselines are 0.9–1.7 percentage points, and individual temporal-loss ablations change ALL accuracy by only 0.4–0.7 points on BirdCLEF (Table 3). Without repeated runs, it is difficult to assess how reliably these components contribute. A focused replication of the full method and its most relevant controls on one representative benchmark would materially strengthen the empirical conclusions.

Questions for authors
  1. Independent contribution of temporal weighting. Could the authors provide a matched acceptance-only control that retains the teacher construction, selection rule, running-statistic updates, and reweighting map, while replacing selected temporal mass with a scalar shared across classes, matched to the global mean selected mass among teacher-accepted recordings? This would test whether class-specific temporal information improves on acceptance-based reweighting at a comparable aggregate scale. As an optional further comparison, a simple temporal proxy without an additional learned head could help assess the value of learning the temporal weights.

  2. Response to temporal redundancy and class difficulty. In the controlled soundscape experiment, at fixed recording counts and target occupancy, how do teacher acceptance, conditional mean selected temporal mass, and the resulting class weights change with increasing repetition? Optionally, for a main benchmark, reporting these quantities for Many, Medium, and Few classes, extending the acceptance diagnostics in Figure 4(f,h), would further clarify how much of the correction is driven by lower tail-class acceptance.

  3. Reliability of the reported increments. Could the authors report repeated runs of CETE and the count-only control on one representative setting, and, if feasible, the acceptance-only control, including variation in ALL and Few accuracy? Please also clarify what the shaded bands in Figure 3 represent.

Reviewer_PSsxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CETE for long-tailed audio recognition. It argues that recording count is an imperfect proxy for class evidence because target sounds may appear briefly or repetitively. CETE learns class-conditioned temporal weights under preservation, complementarity, and compactness constraints; uses an EMA teacher to filter reliable samples; aggregates reliable temporal evidence into corrected class counts; and applies inverse-power class reweighting. The temporal branch is training-only, so inference keeps the original student encoder and classifier with zero extra parameters and FLOPs. Validation covers VGGSound-LT IR10/50/100, iNatSound, and BirdCLEF, with gains of roughly 0.9 to 1.7 points. Ablations support temporal correction and reliable filtering.

Critical strengths

The motivation is clear and audio-specific: class-conditioned temporal evidence directly targets the mismatch between recording count and effective evidence. The training-only design also makes the method deployment-friendly, with zero extra inference parameters and FLOPs.

The gradient isolation and reliability filtering are well motivated. Separating classification and temporal-head updates, together with EMA teacher filtering, makes the auxiliary branch relatively controlled. Ablations on BirdCLEF support the contributions of temporal correction and reliable filtering.

Critical weaknesses

There is a noticeable theory-implementation gap. The paper interprets temporal evidence via an equivalent observation count, but the implemented evidence mass is a reliability-filtered, length-normalized weight mass without explicit covariance estimation. Fixed-weight theory also does not directly apply to weights learned from the same features, and a finite constraint penalty does not guarantee constraint satisfaction or global optimality.

The empirical evidence is not strong enough for the claims. Gains are mostly 1 to 2 points. With several sensitive hyperparameters and limited baselines and ablations, it is difficult to rule out seed or tuning effects and to establish a stable advantage over simpler count-only or temporal-pooling alternatives.

Questions for authors

Can you empirically relate the implemented evidence mass to the theoretical equivalent observation count, for example through correlation, covariance approximation, or constraint-violation rate, to clarify whether the theory is explanatory or mainly motivational?

What are the violation rates of the preservation and complementarity constraints under a finite constraint penalty, and how sensitive are results to the constraint penalty, margin parameters, and teacher thresholds?

Can you add controls for count-only reweighting, uniform or random temporal weights, and removing the complementarity or compactness terms to isolate each design’s contribution?

Reviewer_wJ7bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CETE, a class reweighting method for long-tailed audio classification. The main idea is that the number of recordings alone may not reflect how much useful information a class contains, since target sounds can be short or repeated. CETE learns temporal weights to identify features that support classification. It uses a teacher model to select reliable samples and combines their temporal weights with class counts to adjust the training loss. The paper also provides a theoretical analysis to motivate this approach. Experiments on VGGSound-LT, iNatSound, and BirdCLEF show consistent improvements over existing methods. The authors further evaluate the method through ablation studies, different backbones, and controlled experiments on sound duration and repetition.

Critical strengths

The motivation makes sense. Simply counting recordings may not reflect how much useful audio each class contains. The DESED experiment is relevant here, as it varies sound duration and repetition while keeping class counts the same

The experimental results are fairly consistent. CETE performs best in balanced accuracy across the five main settings, and the results with different backbones also support its effectiveness. It is also useful that the method adds no extra inference cost.

Critical weaknesses

The connection between the theory and the actual method is not fully clear. The theoretical definition of effective temporal evidence considers correlations between features, while the actual method uses the sum of learned temporal weights. It is unclear how well this statistic reflects the effective evidence defined in the theory.

The role of the learned temporal weights needs further validation. The ablation with α = 0 shows that the temporal correction improves performance, but the correction also depends on which recordings are accepted by the teacher. It is therefore difficult to tell how much the temporal weights themselves contribute. Also, the paper does not directly show whether these weights actually capture non-redundant information, especially when sounds are repeated.

Questions for authors
  1. The theoretical definition of effective temporal evidence includes feature correlations, but the actual method uses the sum of temporal weights. Could the authors clarify how the sum of temporal weights relates to the covariance-based effective evidence defined in the theory?
  2. The temporal correction improves performance in the ablation study, but it also depends on teacher-based sample selection. Have the authors tried using only the teacher acceptance rate, without the learned temporal weights? This would help clarify how much the temporal weights contribute.
  3. The paper argues that repeated sounds provide less effective information. Do the learned temporal weights actually reflect this? For example, in the DESED experiments, how do the temporal weights change as the repetition level increases?
1 comment

Thank you for your time and effort in reviewing our paper. Your initial review provided valuable suggestions for improving our work. We posted our point-by-point rebuttal on November 13, providing additional theoretical analysis and experiments on the connection between temporal mass and covariance-based effective evidence, the contribution of temporal weighting beyond teacher acceptance, and the response of temporal weights to increasing sound repetition.

We would appreciate it if you could take a look at our rebuttal and share your assessment of whether the additional evidence addresses the concerns raised in your review. We would be happy to clarify any remaining questions during the discussion period. Thank you again for your time and consideration.

Comments on the paper

1 comment

We appreciate all reviewers for their valuable comments and suggestions. We are encouraged by the reviewers’ recognition of our clear motivation (wJ7b, PSsx, yReL, PffU), consistent experimental results (wJ7b, yReL), informative controlled evaluations (wJ7b, yReL), and practical training-only design (PSsx, PffU, yReL). Their feedback has helped us improve our submission and strengthen our claims. We have carefully addressed the main concerns raised by the reviewers and hope that these improvements will be taken into consideration. Our detailed point-by-point responses are provided in the corresponding review threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.