- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on multi-label classification in real-world data streams where features may simultaneously disappear and emerge, disrupting label-discriminative information and making previously learned label dependencies less reliable. To address this challenge, the authors propose AFET, a framework designed to preserve and transfer historical inter-label dependencies in a one-pass learning setting without replaying historical samples. Specifically, AFET leverages surviving features to reconstruct information associated with vanished features and employs Label-Conditioned Latent-Space Alignment to preserve label-discriminative information. It further incorporates Structure-Aware Distillation, an evolving label graph, and Adaptive Prediction Calibration to integrate historical knowledge with augmented features. The authors also provide a theoretical analysis of prediction stability and conduct comparative and ablation experiments on seven benchmark datasets. The experimental results demonstrate competitive performance across multiple evaluation metrics, while the ablation studies further illustrate the contributions of the main components.
Critical strengths
- Unified Framework for Cross-Stage Evidence Transfer. AFET uses surviving features as a bridge between learning stages, integrating vanished-feature recovery, label-conditioned latent-space alignment, structure-aware distillation, an evolving label graph, and adaptive prediction calibration into a unified framework. The method considers not only feature representations but also the adaptation of label dependencies and decision thresholds as the feature space evolves. The motivations behind the individual components are generally clear, and their integration reflects a clear approach to addressing feature evolution in multi-label classification.
- Preservation of Label-Discriminative Information. The paper investigates multi-label classification in data streams where features can simultaneously emerge and disappear. An important observation is that accurate feature reconstruction does not necessarily guarantee the preservation of label-discriminative information. Building on this observation, the authors further consider mismatches in historical label dependencies and potential shifts in prediction-score distributions under feature evolution. This perspective goes beyond feature reconstruction alone and offers a meaningful extension to existing studies on feature evolution, with practical relevance to real-world data streams. I consider this a strong strength because it connects the method’s central motivation to both a specific design choice and empirical evidence of its contribution to classification performance.
Critical weaknesses
- Insufficient Justification for the Threshold Calibration Strategy. In Online Label-Wise Threshold Calibration, the target positive ratio is primarily determined by the model's own predictions. However, the authors do not sufficiently explain why increasing this ratio should lead to better classification decisions. In particular, it remains unclear whether the reported recall improvements reflect genuinely better classification performance or mainly result from predicting more positive labels, potentially at the cost of increased false positives.
- Unclear Independent Contributions of the Core Mechanisms. AFET integrates multiple techniques into a unified framework, but the necessity and independent contributions of some core mechanisms are not sufficiently validated. Although the existing ablation studies provide useful evidence, they mainly compare broad modules or different information sources. The individual contributions of structure-aware distillation and the evolving label graph are not fully established by the current ablations. This makes it difficult to assess the novelty of the proposed evidence-transfer design and whether the additional complexity introduced by these mechanisms is justified by their performance gains.
Questions for authors
- In Section 2.4, the authors introduce Online Label-Wise Threshold Calibration to adapt the decision threshold for each label. However, the motivation for the target positive ratio remains insufficiently explained, particularly why the updated ratio should lead to better classification decisions than the previous one. Specifically, Eq. (8) first estimates the current pseudo-positive rate using the previous threshold: [ \hat{\pi}{t,l}=\frac{1}{n_t}\sum{i=1}^{n_t}\mathbb{I}[P^S_{t,il}>\tau_{t-1,l}]. ] The target positive ratio is then defined as [ r_{t,l}=\operatorname{clip}((1+\rho)\hat{\pi}{t,l}+m,r{\min},r_{\max}), ] followed by a quantile-based threshold update with exponential smoothing: [ \tilde{\tau}{t,l}=\operatorname{Quantile}({P^S{t,il}}{i=1}^{n_t},1-r{t,l}), ] [ \tau_{t,l}=\gamma\tau_{t-1,l}+(1-\gamma)\tilde{\tau}{t,l}. ] The target ratio is therefore determined primarily by the model's own predictions, without directly accounting for the true positive label prevalence or classification errors. It remains unclear why increasing the predicted positive ratio should necessarily improve classification performance. More specifically, the update rule may continuously lower the decision threshold even when the prediction score distribution remains unchanged. Consider a fixed model with a stationary score distribution whose cumulative distribution function (CDF) is continuous and strictly increasing. Following the definition in Theorem 3.5, we have [ \pi^\star{t,l}=1-F_{t,l}(\tau_{t-1,l}). ] When clipping is inactive, and assuming the adjustment terms produce a strictly positive increase, the target ratio satisfies [ r^\star_{t,l}=(1+\rho)\pi^\star_{t,l}+m>\pi^\star_{t,l}. ] Consequently, the corresponding target threshold satisfies [ \tau^\star_{t,l}=F_{t,l}^{-1}(1-r^\star_{t,l})<\tau_{t-1,l}. ] Under this population setting, the exponential smoothing update also decreases the threshold. This suggests that the calibration rule can introduce systematic threshold changes even in the absence of distribution drift, rather than responding solely to changes in the underlying data distribution. The authors should provide further justification for the target-ratio update rule and clarify how its induced threshold changes relate to actual classification performance.
- In Table 3 and Table 6, the authors mainly conduct ablation on the whole Structure-Aware Evidence Transfer(). Although these results show that can improve the performance, the individual contributions of Evolving Label Graph and Structure-Aware Distillation are still not clear. Figure 2 has already compared the performance of static graph and evolving graph, but a more systematic comparison of graph evolution and distillation mechanisms under the same experimental settings is still missing. Could the authors further compare fixed Historical Graph and Evolving Label Graph while keeping other modules, teacher/student networks, and training settings unchanged? Under each graph setting, could the authors also compare Response Distillation Only and Response + Structure Distillation? These comparisons would help clarify which mechanism mainly contributes to the performance improvement of .
Rebuttal by Authors
We sincerely appreciate the reviewer for the valuable feedback. We respond to your concerns as follows:
A1. We clarify the motivation, theoretical scope, and empirical effectiveness of our calibration strategy. 1) Motivation. Under feature evolution, the disappearance of informative features may reduce positive-label confidence, making fixed thresholds less appropriate. Our quantile-based calibration adapts label-wise thresholds to current score distributions, while ρ and m introduce a recall-oriented adjustment to mitigate missed positives. The target positive ratio is intended to control the decision operating point, rather than estimate the true label prevalence. Clipping and exponential smoothing constrain the target ratio and update speed, respectively. 2) Theoretical clarification. We agree that under a stationary continuous score distribution Fl, when rt,l>πt,l and clipping is inactive, the target threshold satisfies τt,l~=Fl−1(1−rt,l)<τt−1,l. Thus, threshold reduction may occur even without distributional drift, reflecting the intended recall-oriented adjustment. Importantly, Theorem 3.5 establishes tracking stability relative to the prescribed target quantile, rather than classification optimality or guaranteed improvements in precision. We will clarify this distinction and its limitations in the revision. 3) Empirical evidence. To isolate the effect of threshold calibration, we compare three strategies using identical prediction scores, which are Fixed (τ=0.5), Unadjusted (ρ=m=0), and Ours. All other components remain unchanged.
Compared with Unadjusted, Ours improves F1 and Recall on Enron and NUS-WIDE, respectively. These results indicate that the recall gains are accompanied by improved overall classification performance, rather than merely increasing positive predictions without improving F1. While this does not imply fewer false positives, it demonstrates a more favorable trade-off. We will also clarify that the existing M3 ablation evaluates the complete adaptive prediction module, not threshold calibration alone.
A2. The existing M2 ablation evaluates the complete structure-aware evidence transfer module without fully isolating its core mechanisms. 1) Mechanism clarification. The two mechanisms address complementary challenges. Evolving Label Graph (Eq. 4)integrates historical dependencies with current prediction-induced relationships to mitigate structural mismatch under feature evolution. Structure-Aware Distillation (Eq. 5) transfers both label-wise responses and inter-label dependencies from the frozen teacher to the student. Thus, graph evolution adapts relational representations, while structure-aware distillation preserves historical relational knowledge. 2) Controlled ablation. Following the reviewer's suggestion, we consider four configurations by varying the graph strategy (Fixed Historical vs. Evolving) and distillation objective (Response Only vs. Response + Structure), while keeping all other modules, teacher/student architectures, and training settings unchanged. 3) Results and analysis. Both mechanisms provide consistent improvements. Their combination achieves the best performance on both datasets, demonstrating complementary benefits rather than gains attributable to a single mechanism. Moreover, graph evolution and structural distillation reuse existing prediction representations and introduce no additional trainable networks, supporting a lightweight integration. We will incorporate these findings and clarify the individual contributions in the revision.