- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a performance evaluation focused on active learning methods for facial age estimation. For the study, human age annotations by three annotators were solicited for the UTKFace and AAFD datasets. The annotators were asked to provide both an estimated age and an age range, allowing the authors to study differences between individual annotations. The paper additionally considers age predictions generated by several vision-language models and compares these to the original dataset labels and the collected human annotations. Based on this data, the authors evaluate 13 existing active learning methods, including approaches that can use multiple annotators. The presented experiments investigate how the performance and relative ranking of these methods change depending on the source of annotations, the underlying model, and different training settings. The reported results show that the choice of annotation source can have a considerable impact on the performance of active learning methods and that methods using multiple annotators do not necessarily outperform those relying on a single annotator. The authors refer to the collected annotations and the accompanying experimental evaluation as AgeALBench.
Critical strengths
The paper has several strengths that I can highlight:
-
The paper addresses a relevant problem in active learning, where the labels provided by human annotators may be inaccurate or inconsistent. While active learning has been widely studied in the literature, many existing evaluations assume that the annotators provide correct labels. The authors make a case that this assumption does not necessarily hold in real-world applications, especially for tasks such as facial age estimation, where it is usually difficult to accurately estimate the age of a person from an image.
-
The collection of additional human annotations for two existing datasets is a solid contribution that can also have impact down the line. The authors collect exact estimates and age ranges from three different annotators, which allows them to study annotation variability and its impact on the performance of active learning methods. The collected annotations could also be useful for future research on learning from imperfect supervision, especially if made publicly available and if adopted by the community.
-
The experimental evaluation is reasonably extensive given the studies problem. The authors consider 13 active learning approaches, including methods that work with multiple annotators, and investigate how their performance changes when using human annotations, automatically generated labels, or the original dataset labels. They also conduct additional experiments with different model architectures, training strategies, and active learning settings. The results provide some interesting observations, particularly regarding the sensitivity of active learning methods to the source of annotations and the fact that using multiple annotators does not necessarily lead to better performance, which however, are somewhat expected and not ground breaking.
Critical weaknesses
While I do find the paper interesting, I also have several concerns. Below are some that I feel are most critical:
-
The contribution of the paper is limited in my view. The paper presents a performance evaluation designed around age estimation (based on newly collected human annotations) and active learning and then reports some findings. While the findings may be interesting, they are also more or less expected. The study is overall well done, but I don’t think that some exceptional insight was reported. The contribution may therefore not justify acceptance at a venue as selective as ICLR.
-
The title is somewhat misleading. When reading the title I was expecting a new benchmark to be introduced in the paper, not a performance evaluation. This may have been intended to imply a bigger contribution but made me thing I missed something critical after reading the paper for the first time. It is quite unusual to give some research a name as the authors do when stating “We term our research AgeALBench, as it benchmarks the performance of AL algorithms for facial age estimation.”
-
The authors motivate their work by arguing that age estimation is inherently subjective and that different annotators may provide different age estimates for the same person. However, both UTKFace and AAFD are intended for chronological age estimation, rather than apparent age estimation. These are two different problems. If the objective is to estimate the actual age of a person, the disagreement between annotators is not necessarily a consequence of the subjective nature of the task, but rather reflects errors in estimating chronological age from facial appearance. Furthermore, the original age labels of these datasets are not necessarily verified chronological ages. For instance, UTKFace labels were initially estimated automatically and subsequently checked by a human annotator. I therefore find it questionable to treat these labels as exact ground truth while considering the newly collected human annotations as noisy estimates. This distinction probably needs more discussions.
-
One of the main conclusions of the paper is that rankings change considerably depending on the source of annotations and other experimental settings. However, the authors mainly quantify this through the percentage of methods that change their position in the ranking. This measure does not consider how large the actual performance differences are. Two methods with almost identical performance may exchange positions without this being practically meaningful, yet such a change is counted in the analysis.
-
Related to the previous point, I have concerns about the statistical analysis and the interpretation of the results. The authors use Friedman tests to support the significance of the obtained rankings and conclude that these are not affected by noise. However, a significant Friedman test indicates that differences exist among the evaluated methods. It does not establish that individual rankings are reliable or that changes in rankings between experimental settings are statistically significant.
Questions for authors
I would welcome responses to questions related to the weaknesses highlighted above:
-
Given that UTKFace and AAFD are intended for chronological rather than apparent age estimation, how do the authors justify the motivation based on subjective age perception and the use of potentially inaccurate original age labels as ground truth?
-
Could the authors clarify what novel insights the study provides beyond previously reported observations about the sensitivity of active learning methods to different experimental settings?
-
Could the authors provide an analysis of whether the reported changes in method rankings correspond to statistically significant or practically meaningful performance differences?
-
Could the authors clarify how the Friedman tests support the claim that the reported rankings are reliable and unaffected by noise, particularly when comparing rankings across different experimental settings?
3 comments
Author Response to Reviewer UvLq (Part 2/2)
W3 / Q1 (continued): Chronological vs. apparent age, and the reliability of the original labels.
To confirm that our conclusions do not depend on the exact reference, we re-evaluated every method in all six settings using the mean of the three human annotations as the test label (Appendix A.11, Table 22). Rankings are highly consistent with those under the original labels (Kendall's τ = 0.92-1.00 on AAFD and 0.90-0.92 on UTKFace). The best method is unchanged in five of six settings, and in the remaining one the top two methods differ by less than 0.01 MAE-AUC. Our comparisons between AL methods are therefore robust to the choice of reference.
W4 / Q3: Whether ranking changes are meaningful.
We agree that the "Moved" count can include swaps between near-identical methods. We now also report a stricter measure: a significant reversal is a pair of methods whose order flips between two settings and whose difference is significant (paired t-test, p < 0.05) in both settings, in opposite directions. Requiring significance in both settings makes chance reversals much less likely, and a swap between nearly tied methods cannot qualify.
- Label source. With everything else fixed, the annotation source produces significant reversals: 4 and 6 between human and VLM labels on AAFD and UTKFace, 2 between GT and human labels on UTKFace, and 1 between GT and VLM labels on AAFD (Appendix A.8, Table 18). For example:
- On AAFD, LearningLoss significantly outperforms BADGE with human labels (p < 10⁻⁴), while BADGE significantly outperforms LearningLoss with VLM labels (p < 0.001).
- On UTKFace, BADGE outperforms LearningLoss with GT labels (p = 0.003), while LearningLoss outperforms BADGE with human labels (p = 0.001).
- Ablation: We also report significant reversals for every ablation (Appendix A.12, Table 23). Small changes to the protocol produce none: 1% vs. 2% initial labeled data, and query size 100 vs. 200. The annotation source alone therefore changes rankings more than small budget changes do. Large changes produce more reversals (12 for 1% vs. 10% initial labeled data, 21 for full fine-tuning vs. head-only).
W5 / Q4: Interpretation of the Friedman test.
We agree. A significant Friedman test only shows that methods differ within an experimental setting. It does not show that rankings are reliable or that they change across settings. We use one sided paired t-test on per-seed AULC and significant rank reversals to draw conclusions across experimental settings. We would like to clarify the following:
- Within a setting, the Friedman test checks that differences among methods are not explained by seed variability. We have updated our paper to reflect these changes.
- To compare pairs of methods, within and across annotation sources, we use paired t-tests on per-seed AULC, following Subedi et al. (NeurIPS 2024). All pairwise p-values are reported in Section 4.1 (Table 2) and Appendix A.1 (Tables 4 and 5).
- The rank consistency table (Appendix A.8, Table 18) now reports the significant reversals described in response to W4/Q3.
We thank the reviewer again for the constructive feedback, which has helped substantially improve the paper.
References
- Cheng, J., Li, Y., Wang, J., Yu, L., Wang, S. Exploiting effective facial patches for robust gender recognition. Tsinghua Science and Technology, 24(3):333–345, 2019.
- Gong, S., Liu, X., Jain, A. K. Jointly de-biasing face recognition and demographic attribute estimation. ECCV, 2020.
- Kim, Y. H., Nam, S. H., Hong, S. B., Park, K. R. GRA-GAN: Generative adversarial network for image style transfer of gender, race, and age. Expert Systems with Applications, 2022.
- Zhang, Z., Song, Y., Qi, H. Age progression/regression by conditional adversarial autoencoder. CVPR, 2017.
- Gustafsson, F. K., Danelljan, M., Bhat, G., Schön, T. B. Energy-based models for deep probabilistic regression. ECCV, 2020.
- Shin, N.-H., Lee, S.-H., Kim, C.-S. Moving window regression: A novel approach to ordinal regression. CVPR, 2022.
- Paplhám, J., Franc, V. A call to reflect on evaluation practices for age estimation. CVPR, 2024.
- Subedi, R., Wei, L., Gao, W., Chakraborty, S., Liu, Y. Empowering active learning for 3D molecular graphs with geometric graph isomorphism. NeurIPS, 2024.
Response Follow-up
Dear Reviewer UvLq,
Thank you again for your valuable feedback and for taking the time to review our paper.
As the discussion period is approaching its end, with approximately three days remaining, we would like to kindly follow up on our response. We hope our responses have adequately addressed your concerns. If there are any remaining concerns or points that require further clarification, we would greatly appreciate your feedback and would be happy to provide additional explanations before the conclusion of the discussion period.
Thank you again for your time and consideration.
Best regards,
- Authors
Author Response to Reviewer UvLq (Part 1/2)
We thank the reviewer for the thoughtful review and for noting that the study is well done, the annotations are a solid contribution, and the evaluation is extensive. We address each concern below. The updated texts in the paper are marked in red for ease of readability.
W1 / Q2: Novelty of the insights: We believe the following findings are novel, and not evident from prior work.
We have also added an analysis of why rankings change, and experiments that use the range annotations.
Together with the release of 49,911 real human annotations, we believe these findings make the paper a useful resource for the AL community.
W2: Title and naming. Our intent was to use the term “AgeALBench” to refer to a reusable benchmark, not only to our evaluation. We will release:
We have revised the contribution statement accordingly (lines 123-125). We will be happy to revise the title, if the reviewer suggests.
W3 / Q1: Chronological vs. apparent age, and the reliability of the original labels.
We thank the reviewer for raising this point. UTKFace and AAFD are established age estimation benchmarks, and their provided age labels are the accepted reference in the literature. UTKFace has been used this way since its introduction (Zhang et al., CVPR 2017), by Gustafsson et al. (ECCV 2020), Shin et al. (CVPR 2022), and the unified age estimation benchmark of Paplhám & Franc (CVPR 2024) among others. AAFD (Cheng et al., 2019) is used in the same way. Gong et al. (ECCV 2020) train their age estimator on AAFD and UTKFace together with other benchmarks, and Kim et al. (2022) evaluate age estimation on AAFD, UTKFace, MORPH and AFAD. Both treat the provided labels as ground truth. We follow this standard practice to be consistent with the literature.
Our "subjective perception" framing refers to the human and VLM annotators, whose estimates are judgments of how old a face looks. The reference labels are the fixed target against which those judgments, and the AL methods trained on them, are evaluated. We acknowledge that no single reference is perfect: UTKFace labels, for instance, were estimated with DEX and checked by a human annotator. We now state this explicitly in Section 3.1 and Appendix A.13.