- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when concept-aligned supervision is effective and how to improve predictive performance after its benefits saturate. The results show that stronger concept alignment does not necessarily improve performance and may even reduce accuracy. Some benefits of auxiliary supervision may also be independent of concept semantics. The authors introduce ResDiag, which combines representation decomposition, directional ablation, and ensemble-contribution analysis to show that within-class concept-related residuals still contain useful predictive information. They further propose SatEnsemble, which combines an improved training recipe with prediction averaging across checkpoints to use complementary information across models, and reports competitive results on FACED.
Critical strengths
-
The research question is valuable. The authors examine whether further supervision of a concept still improves prediction once the model already encodes it. This addresses a scientifically meaningful question beyond simply pursuing higher accuracy. The paper also carefully reports many negative results rather than presenting only successful configurations.
-
The experiments cover various auxiliary supervision configurations, include random-target controls in some settings, and apply Holm–Bonferroni correction to 25 primary tests. These choices provide a useful basis for examining the relationship between valence supervision and predictive performance.
Critical weaknesses
-
The “Saturation Regularity” is currently mainly an empirical finding about additional valence supervision. Although the authors tested 25 configurations, the supervision targets remain largely focused on valence, with limited validation of other concepts such as arousal. Models may encode several concepts simultaneously, while this study primarily examines relatively prominent valence-related information. The current evidence is therefore insufficient to generalize this finding into a general regularity of concept saturation.
-
The links between “Saturation Regularity,” ResDiag, and SatEnsemble are based mainly on the research motivation and empirical associations, with limited mechanistic explanation. The useful information in the residuals may also come from concepts that were not examined, leaving the interpretation of these findings unclear.
Questions for authors
I still have several concerns about this paper and would appreciate the authors’ responses to the following questions:
-
The systematic validation of “Saturation Regularity” mainly focuses on valence, with only limited experiments on other concepts. Given that EEG may encode multiple emotional and physiological concepts simultaneously, how do the authors establish that this phenomenon generalizes beyond valence supervision, rather than being an empirical finding specific to valence?
-
The links between “Saturation Regularity,” ResDiag, and SatEnsemble currently rest mainly on the research motivation and empirical correlations, without a clear mechanistic explanation. The predictive information in within-class residuals may come from other concepts that were not analyzed, differences between subjects, or stimulus information. How do the authors interpret what these residuals represent and demonstrate a direct link to concept saturation and model complementarity?
-
Prediction averaging across checkpoints or random seeds, data augmentation, increased model depth, prototype regularization, and extended training may independently contribute to performance gains. Could the authors clarify to what extent SatEnsemble’s gains are related to concept saturation rather than these conventional training and ensemble methods? Without such a connection, SatEnsemble appears to be a relatively independent engineering improvement, lacking a clear design rationale grounded in the preceding saturation analysis. This also leaves the paper’s central line of investigation insufficiently developed.
I also have the following concerns about the presentation:
- Several passages are abstract and overly compressed, making the writing less natural and harder to follow. For example, expressions such as “An objective can change a representation as intended without improving the prediction it is meant to support” and “what the receiving model already represents” require readers to infer their specific meaning. More direct wording would improve clarity and readability.
- Figure 1 has a noticeably AI-generated appearance. Its layout could also be improved; for example, Panel B occupies more space than its content requires. I suggest reducing the overtly AI-generated aesthetic and adopting a clearer, more informative visual presentation.