acceptodds
Under review as a conference paper at ICLR 2027

AgeALBench: Benchmarking Deep Active Learning for Facial Age Estimation with Crowd-sourced Annotations

Abstract

Age estimation from facial images is a problem of practical importance, with a variety of real-world applications. While deep neural networks have depicted commendable performance for this task, they require large amounts of hand-annotated training data, acquiring which is a time-consuming and labor-intensive process. Active Learning (AL) algorithms automatically identify the salient and exemplar samples from large amounts of unlabeled data, and tremendously reduce human annotation effort in training deep neural network models. However, facial age estimation poses a unique challenge as it is almost impossible to estimate the exact correct age of a person merely from a facial image; moreover, the perceived age is inherently subjective, and different human annotators can have very different perceptions about the age of the same subject. This necessitates a thorough analysis of the performance of AL techniques for facial age estimation, where each image is annotated by multiple annotators to capture annotation variability. In this research, we first collect crowd-sourced age annotations of two benchmark datasets: UTKFace and AAFD; each image is annotated by three annotators, who provide exact age estimates, as well as age range estimates. We then analyze the performance of 13 single annotator, as well as multi-annotator AL algorithms using the collected annotations. To the best of our knowledge, this is the first research effort to analyze the performance of deep AL algorithms (with crowd-sourced annotations) for the task of facial age estimation, where annotators can only provide an approximate label to the queried instances, rather than the exact label. We hope this research will be a step toward bridging the gap between AL and real-world applications, and our findings will be useful for the development of next generation AL algorithms under real-world annotation challenges. Our code and the collected annotations will be open-sourced upon acceptance of our paper.

Primary area:
datasets and benchmarks
Keywords:
active learning, crowd-sourced data annotations, facial age estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UvLqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a performance evaluation focused on active learning methods for facial age estimation. For the study, human age annotations by three annotators were solicited for the UTKFace and AAFD datasets. The annotators were asked to provide both an estimated age and an age range, allowing the authors to study differences between individual annotations. The paper additionally considers age predictions generated by several vision-language models and compares these to the original dataset labels and the collected human annotations. Based on this data, the authors evaluate 13 existing active learning methods, including approaches that can use multiple annotators. The presented experiments investigate how the performance and relative ranking of these methods change depending on the source of annotations, the underlying model, and different training settings. The reported results show that the choice of annotation source can have a considerable impact on the performance of active learning methods and that methods using multiple annotators do not necessarily outperform those relying on a single annotator. The authors refer to the collected annotations and the accompanying experimental evaluation as AgeALBench.

Critical strengths

The paper has several strengths that I can highlight:

  • The paper addresses a relevant problem in active learning, where the labels provided by human annotators may be inaccurate or inconsistent. While active learning has been widely studied in the literature, many existing evaluations assume that the annotators provide correct labels. The authors make a case that this assumption does not necessarily hold in real-world applications, especially for tasks such as facial age estimation, where it is usually difficult to accurately estimate the age of a person from an image.

  • The collection of additional human annotations for two existing datasets is a solid contribution that can also have impact down the line. The authors collect exact estimates and age ranges from three different annotators, which allows them to study annotation variability and its impact on the performance of active learning methods. The collected annotations could also be useful for future research on learning from imperfect supervision, especially if made publicly available and if adopted by the community.

  • The experimental evaluation is reasonably extensive given the studies problem. The authors consider 13 active learning approaches, including methods that work with multiple annotators, and investigate how their performance changes when using human annotations, automatically generated labels, or the original dataset labels. They also conduct additional experiments with different model architectures, training strategies, and active learning settings. The results provide some interesting observations, particularly regarding the sensitivity of active learning methods to the source of annotations and the fact that using multiple annotators does not necessarily lead to better performance, which however, are somewhat expected and not ground breaking.

Critical weaknesses

While I do find the paper interesting, I also have several concerns. Below are some that I feel are most critical:

  • The contribution of the paper is limited in my view. The paper presents a performance evaluation designed around age estimation (based on newly collected human annotations) and active learning and then reports some findings. While the findings may be interesting, they are also more or less expected. The study is overall well done, but I don’t think that some exceptional insight was reported. The contribution may therefore not justify acceptance at a venue as selective as ICLR.

  • The title is somewhat misleading. When reading the title I was expecting a new benchmark to be introduced in the paper, not a performance evaluation. This may have been intended to imply a bigger contribution but made me thing I missed something critical after reading the paper for the first time. It is quite unusual to give some research a name as the authors do when stating “We term our research AgeALBench, as it benchmarks the performance of AL algorithms for facial age estimation.”

  • The authors motivate their work by arguing that age estimation is inherently subjective and that different annotators may provide different age estimates for the same person. However, both UTKFace and AAFD are intended for chronological age estimation, rather than apparent age estimation. These are two different problems. If the objective is to estimate the actual age of a person, the disagreement between annotators is not necessarily a consequence of the subjective nature of the task, but rather reflects errors in estimating chronological age from facial appearance. Furthermore, the original age labels of these datasets are not necessarily verified chronological ages. For instance, UTKFace labels were initially estimated automatically and subsequently checked by a human annotator. I therefore find it questionable to treat these labels as exact ground truth while considering the newly collected human annotations as noisy estimates. This distinction probably needs more discussions.

  • One of the main conclusions of the paper is that rankings change considerably depending on the source of annotations and other experimental settings. However, the authors mainly quantify this through the percentage of methods that change their position in the ranking. This measure does not consider how large the actual performance differences are. Two methods with almost identical performance may exchange positions without this being practically meaningful, yet such a change is counted in the analysis.

  • Related to the previous point, I have concerns about the statistical analysis and the interpretation of the results. The authors use Friedman tests to support the significance of the obtained rankings and conclude that these are not affected by noise. However, a significant Friedman test indicates that differences exist among the evaluated methods. It does not establish that individual rankings are reliable or that changes in rankings between experimental settings are statistically significant.

Questions for authors

I would welcome responses to questions related to the weaknesses highlighted above:

  1. Given that UTKFace and AAFD are intended for chronological rather than apparent age estimation, how do the authors justify the motivation based on subjective age perception and the use of potentially inaccurate original age labels as ground truth?

  2. Could the authors clarify what novel insights the study provides beyond previously reported observations about the sensitivity of active learning methods to different experimental settings?

  3. Could the authors provide an analysis of whether the reported changes in method rankings correspond to statistically significant or practically meaningful performance differences?

  4. Could the authors clarify how the Friedman tests support the claim that the reported rankings are reliable and unaffected by noise, particularly when comparing rankings across different experimental settings?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer UvLq (Part 1/2)

We thank the reviewer for the thoughtful review and for noting that the study is well done, the annotations are a solid contribution, and the evaluation is extensive. We address each concern below. The updated texts in the paper are marked in red for ease of readability.

W1 / Q2: Novelty of the insights: We believe the following findings are novel, and not evident from prior work.

  1. Rankings derived with GT labels do not always carry over to real human annotators. On UTKFace, BADGE and Random are significantly better than LearningLoss with GT labels, but LearningLoss is significantly better than both with human labels. The best single-annotator method changes from BADGE to LearningLoss. Benchmarks that simulate the annotator with GT labels from the corresponding datasets can therefore recommend the wrong method for real deployments.
  2. VLM labels are not a reliable proxy for choosing an AL method. Rankings under VLM labels show 10 significant reversals relative to human labels, more than any other comparison (Appendix A.8, Table 18). Cheap VLM annotation therefore cannot replace human annotation for method selection.
  3. Multi-annotator methods help only in some settings. ActiveLab is the best method on UTKFace with both human and VLM labels. On AAFD, it trails LearningLoss with human labels and BADGE with VLM labels. Multi-annotator Uncertainty ranks in the bottom three in all six main settings. This leaves room for better multi-annotator AL methods, which our annotations can help develop.
  4. Based on new experiments from reviewer feedback, we also discovered training with a soft interval loss on range annotation can narrow the gap between using an acquisition function and using Random sampling (Appendix A.9, Table 19)

We have also added an analysis of why rankings change, and experiments that use the range annotations.

  • Why rankings change (Appendix A.10, Tables 20 and 21). Human annotation error grows with age, from about 2 years for children to as much as 10 years on AAFD and 16 years on UTKFace for older subjects (Section 3.2.2, Figure 4).
    • On AAFD, methods whose queries have noisier labels tend to gain rank under human labels (Spearman ρ = −0.80, p = 0.006), because they cover the difficult older age range (Appendix A.10, Table 20).
    • Training on noisy labels can also move a method's queries away from older subjects. For example, BALD's mean queried age on AAFD falls from 39 to 26 (Appendix A.10, Table 20).
  • Range annotations (Appendix A.9, Table 19). Training with a soft interval loss on the range annotations lowers MAE-AUC by 0.7-1.1 years for all methods except LearningLoss, and it further narrows the gap between AL methods and Random.

Together with the release of 49,911 real human annotations, we believe these findings make the paper a useful resource for the AL community.

W2: Title and naming. Our intent was to use the term “AgeALBench” to refer to a reusable benchmark, not only to our evaluation. We will release:

  • 49,911 human point and range annotations for 16,637 images, together with annotations from three VLMs;
  • fixed train/validation/test splits and initial labeled sets;
  • a standard evaluation protocol;
  • implementations of 13 AL methods adapted to regression, with code.

We have revised the contribution statement accordingly (lines 123-125). We will be happy to revise the title, if the reviewer suggests.

W3 / Q1: Chronological vs. apparent age, and the reliability of the original labels.

We thank the reviewer for raising this point. UTKFace and AAFD are established age estimation benchmarks, and their provided age labels are the accepted reference in the literature. UTKFace has been used this way since its introduction (Zhang et al., CVPR 2017), by Gustafsson et al. (ECCV 2020), Shin et al. (CVPR 2022), and the unified age estimation benchmark of Paplhám & Franc (CVPR 2024) among others. AAFD (Cheng et al., 2019) is used in the same way. Gong et al. (ECCV 2020) train their age estimator on AAFD and UTKFace together with other benchmarks, and Kim et al. (2022) evaluate age estimation on AAFD, UTKFace, MORPH and AFAD. Both treat the provided labels as ground truth. We follow this standard practice to be consistent with the literature.

Our "subjective perception" framing refers to the human and VLM annotators, whose estimates are judgments of how old a face looks. The reference labels are the fixed target against which those judgments, and the AL methods trained on them, are evaluated. We acknowledge that no single reference is perfect: UTKFace labels, for instance, were estimated with DEX and checked by a human annotator. We now state this explicitly in Section 3.1 and Appendix A.13.

AuthorsOpenReview ↗

Author Response to Reviewer UvLq (Part 2/2)

W3 / Q1 (continued): Chronological vs. apparent age, and the reliability of the original labels.

To confirm that our conclusions do not depend on the exact reference, we re-evaluated every method in all six settings using the mean of the three human annotations as the test label (Appendix A.11, Table 22). Rankings are highly consistent with those under the original labels (Kendall's τ = 0.92-1.00 on AAFD and 0.90-0.92 on UTKFace). The best method is unchanged in five of six settings, and in the remaining one the top two methods differ by less than 0.01 MAE-AUC. Our comparisons between AL methods are therefore robust to the choice of reference.

W4 / Q3: Whether ranking changes are meaningful.

We agree that the "Moved" count can include swaps between near-identical methods. We now also report a stricter measure: a significant reversal is a pair of methods whose order flips between two settings and whose difference is significant (paired t-test, p < 0.05) in both settings, in opposite directions. Requiring significance in both settings makes chance reversals much less likely, and a swap between nearly tied methods cannot qualify.

  • Label source. With everything else fixed, the annotation source produces significant reversals: 4 and 6 between human and VLM labels on AAFD and UTKFace, 2 between GT and human labels on UTKFace, and 1 between GT and VLM labels on AAFD (Appendix A.8, Table 18). For example:
    • On AAFD, LearningLoss significantly outperforms BADGE with human labels (p < 10⁻⁴), while BADGE significantly outperforms LearningLoss with VLM labels (p < 0.001).
    • On UTKFace, BADGE outperforms LearningLoss with GT labels (p = 0.003), while LearningLoss outperforms BADGE with human labels (p = 0.001).
  • Ablation: We also report significant reversals for every ablation (Appendix A.12, Table 23). Small changes to the protocol produce none: 1% vs. 2% initial labeled data, and query size 100 vs. 200. The annotation source alone therefore changes rankings more than small budget changes do. Large changes produce more reversals (12 for 1% vs. 10% initial labeled data, 21 for full fine-tuning vs. head-only).

W5 / Q4: Interpretation of the Friedman test.

We agree. A significant Friedman test only shows that methods differ within an experimental setting. It does not show that rankings are reliable or that they change across settings. We use one sided paired t-test on per-seed AULC and significant rank reversals to draw conclusions across experimental settings. We would like to clarify the following:

  • Within a setting, the Friedman test checks that differences among methods are not explained by seed variability. We have updated our paper to reflect these changes.
  • To compare pairs of methods, within and across annotation sources, we use paired t-tests on per-seed AULC, following Subedi et al. (NeurIPS 2024). All pairwise p-values are reported in Section 4.1 (Table 2) and Appendix A.1 (Tables 4 and 5).
  • The rank consistency table (Appendix A.8, Table 18) now reports the significant reversals described in response to W4/Q3.

We thank the reviewer again for the constructive feedback, which has helped substantially improve the paper.

References

  • Cheng, J., Li, Y., Wang, J., Yu, L., Wang, S. Exploiting effective facial patches for robust gender recognition. Tsinghua Science and Technology, 24(3):333–345, 2019.
  • Gong, S., Liu, X., Jain, A. K. Jointly de-biasing face recognition and demographic attribute estimation. ECCV, 2020.
  • Kim, Y. H., Nam, S. H., Hong, S. B., Park, K. R. GRA-GAN: Generative adversarial network for image style transfer of gender, race, and age. Expert Systems with Applications, 2022.
  • Zhang, Z., Song, Y., Qi, H. Age progression/regression by conditional adversarial autoencoder. CVPR, 2017.
  • Gustafsson, F. K., Danelljan, M., Bhat, G., Schön, T. B. Energy-based models for deep probabilistic regression. ECCV, 2020.
  • Shin, N.-H., Lee, S.-H., Kim, C.-S. Moving window regression: A novel approach to ordinal regression. CVPR, 2022.
  • Paplhám, J., Franc, V. A call to reflect on evaluation practices for age estimation. CVPR, 2024.
  • Subedi, R., Wei, L., Gao, W., Chakraborty, S., Liu, Y. Empowering active learning for 3D molecular graphs with geometric graph isomorphism. NeurIPS, 2024.
AuthorsOpenReview ↗

Response Follow-up

Dear Reviewer UvLq,

Thank you again for your valuable feedback and for taking the time to review our paper.

As the discussion period is approaching its end, with approximately three days remaining, we would like to kindly follow up on our response. We hope our responses have adequately addressed your concerns. If there are any remaining concerns or points that require further clarification, we would greatly appreciate your feedback and would be happy to provide additional explanations before the conclusion of the discussion period.

Thank you again for your time and consideration.

Best regards,

  • Authors
Reviewer_K5jeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a benchmark for deep active learning in facial age estimation with crowd-sourced labels. The authors collect three human annotations per image for 10,000 UTKFace and 6,637 AAFD images, each with a point age and an age range, and also collect labels from three VLMs. The ranking of AL methods changes with the label source and these choices.

Critical strengths
  • The collected annotations are a useful resource. Real labels from several human annotators per image are rare for regression tasks, and they make it possible to study active learning with imperfect oracles without simulating the noise.
  • The experiments are broad, covering three label sources, two datasets, several design choices and 10 runs per setting. If the analysis is tightened, this grid could serve as a common testbed for future work on active learning under noisy labels.
Critical weaknesses
  • The paper argues that annotators can only give approximate labels and collects range labels for this reason, but every AL method uses only point labels. The experiments therefore study AL with noisy point labels, which is a different and well-studied problem.
  • The main finding is that AL rankings change with the label source and with design choices. This is reported but not explained. The paper does not analyze why a method gains or loses under human or VLM labels, for example how label noise interacts with each acquisition criterion. As a result, the benchmark does not tell practitioners which method to use or how to choose one.
  • Several statements in the text contradict the paper's own tables. Section 4.1 says LearningLoss is significantly better than all methods on UTKFace and ActiveLab on AAFD with human labels. Table 6 shows the reverse: LearningLoss has the lowest AULC and final MAE on AAFD, and ActiveLab has the lowest AULC on UTKFace. Section 4.3 says LearningLoss and BAIT reach the lowest final GT-label MAE on UTKFace and AAFD respectively, while Table 8 gives LearningLoss on AAFD and BAIT on UTKFace.
Questions for authors

Please see the weaknesses above.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer K5je (Part 1/2)

We thank the reviewer for the careful reading, for recognizing the annotations as a useful resource, and for the concrete suggestions. We address each point below. The updated text in the revised manuscript is marked in red for ease of readability.

W1: Range labels are collected but not used.

We agree this was a gap, which we had noted as a limitation before (Appendix A.13). In response to this comment, we have added new experiments that train directly on the range labels (Appendix A.9, Table 19), using three variants:

  1. The midpoint of the range as a single label;
  2. A hard interval loss, which is zero when the prediction falls inside the annotator's range (Cheng et al., ICML 2023);
  3. A soft interval loss based on Dănăilă & Buiu (2024), which rewards predictions that are likely to fall inside the range.

We ran Random, LearningLoss, BADGE, BAIT, LCMD and Coreset on AAFD and UTKFace, with the same data splits, starting sets and budget as the point-label experiments. All results are evaluated against the GT labels and are presented in Appendix A.9, Table 19.

The new experiments show:

  • The soft interval loss lowers MAE-AUC by 0.7–1.1 years for every method except LearningLoss on both datasets (paired t-test, p < 0.05).
  • The hard interval loss gives smaller gains, and the midpoint performs close to point labels. Treating the range as an interval, rather than turning it into a single number, is what helps.
  • Range labels also change the relative rankings of the AL methods. With the soft loss, the gap between the best AL method and Random shrinks from 0.61 to 0.03 years on AAFD, and from 0.42 to 0.04 years on UTKFace, and BADGE depicts the best performance on both datasets (Appendix A.9, Table 19).

We also note that our point-label setting differs from the usual noisy-label AL setting in two ways. First, most prior work on AL with noisy annotators studies classification with simulated noise (Section 2, lines 147–168). Second, the noise in our human labels is real and depends on age: annotator error grows from about 2 years for children (ages 0–9) to as much as 10 years on AAFD and 16 years on UTKFace for older subjects (Section 3.2.2, Figure 4). This is unlike the uniform noise usually assumed. To our knowledge, this setting has not been studied for regression applications using deep neural networks.

Our responses to W2 and W3, including the new query-selection analysis and rank-reversal results, are provided in Part 2/2 below.

AuthorsOpenReview ↗

Author Response to Reviewer K5je (Part 2/2)

W2: Ranking changes are reported but not explained.

We thank the reviewer for this observation. We have added a new analysis of why methods gain or lose rank (Appendix A.10, Tables 20 and 21). For each method, we analyzed the samples it chose to label: how wrong their human labels were (|human label − GT|) and how old the subjects were. Our findings are as follows:

  • Methods whose queries have noisier labels tend to gain rank. Both annotation error and test error are highest for older subjects. On AAFD, the methods that move up under human labels (GSy, iGS, LCMD) choose samples with larger label errors than Random. The methods that move down (BALD, BAIT) choose younger samples with smaller label errors. Across methods, the change in rank is strongly linked to the label error of the chosen samples (Spearman ρ = −0.80, p = 0.006). On UTKFace the trend has the same direction but is not significant (Appendix A.10, Table 20).
  • Noisy labels can change where a method ranks. BALD drops the most on AAFD (6th → 9th). With GT labels it focuses on older faces (mean age 39; 32% aged 50+), but with human labels it shifts to young faces (mean age 26; 8% aged 50+) (Appendix A.10, Table 20).
  • Multi-annotator Uncertainty wastes budget on re-labeling. Its labels are not noisier than Random's. Instead, it spends 34–42% of its budget re-labeling images that already have a label (vs. 11–16% for Random), so it sees far fewer distinct images (Appendix A.10, Table 21).

Furthermore, to separate real ranking changes from swaps between methods with nearly identical performance, we have expanded the rank consistency analysis (Appendix A.8, Table 18) to report significant reversals: pairs of methods where one is significantly better under one label source and the other is significantly better under another.

We have also added practical guidance to the conclusion (Section 5):

  • BADGE ranks in the top three in all six settings (two datasets × three label sources) and is comparable to the best among six methods tested with the soft interval loss on both datasets, making it a robust default.
  • ActiveLab is the best choice when several annotators can be asked again (best results with human and VLM labels on UTKFace).
  • Coreset ranks in the bottom three in every setting, including with range labels. Multi-annotator Uncertainty also ranks in the bottom three in all six main settings, mainly because of re-labeling. We recommend avoiding both for regression applications with noisy labels.
  • Random sampling ranks in the top three in four of six settings and is nearly as good as the best method with range labels. It is a strong baseline that should always be reported.
  • When range annotations are available, the soft interval loss lowers MAE-AUC by 0.8–0.9 years for Random. That is more than the 0.4–0.6 years gained by choosing the best AL method over Random with point labels (Appendix A.9, Table 19).

W3: Text contradicts the tables.

The reviewer is correct, and we apologize for this oversight. In both places the dataset names were swapped in the text; all tables and figures are correct. We have corrected both statements in the revised manuscript:

  • Section 4.1 (lines 364–366): With human labels, LearningLoss is significantly better than all other methods on AAFD, and ActiveLab on UTKFace (Tables 2 and 6).
  • Section 4.3 (lines 433–434): With GT labels, LearningLoss reaches the lowest final MAE on AAFD and BAIT on UTKFace (Table 8).

We have corrected our claims in the paper against the tables, used the same VLM names (Qwen2-VL, InternVL2, LLaVA-1.6) throughout, and fixed the typos.

We hope these additions address the reviewer's concerns, and we would be glad to answer any further questions.

References

  • Cheng, X., Cao, Y., Li, X., An, B., Feng, L. Weakly supervised regression with interval targets. ICML, pp. 5428–5448, 2023.
  • Dănăilă, V. R., & Buiu, C. A deep learning approach to censored regression. Pattern Analysis and Applications, 27(1), 24, 2024.
AuthorsOpenReview ↗

Response Follow-up

Dear Reviewer K5je,

Thank you again for your valuable feedback and for taking the time to review our paper.

As the discussion period is approaching its end, with approximately three days remaining, we would like to kindly follow up on our response. We hope our responses have adequately addressed your concerns. If there are any remaining concerns or points that require further clarification, we would greatly appreciate your feedback and would be happy to provide additional explanations before the conclusion of the discussion period.

Thank you again for your time and consideration.

Best regards,

  • Authors
Reviewer_Kwg4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AgeALBench evaluates deep active learning for facial age estimation using crowd-sourced annotations. The authors collect point and interval age estimates from three annotators per image across two datasets and compare 13 acquisition strategies under human, VLM, and original dataset supervision. The problem is relevant, and the annotations could become a useful resource. However, several methodological and reporting issues currently weaken the benchmark’s conclusions.

Critical strengths

1.The paper examines the gap between idealized labeling oracles and subjective human annotations, an important limitation of conventional active learning evaluations. 2.Collecting multiple human point and interval estimates provides a more realistic basis for studying annotation variability than synthetic noise alone. 3.The evaluation includes multiple acquisition strategies, annotation sources, backbone architectures, and fine-tuning settings. Ten independent runs and supplementary variability statistics are positive aspects.

Critical weaknesses
  1. CQR uses two ConvNeXt-B encoders, but Table I does not report each baseline’s backbone, pretraining, model size, or training setup. It is unclear which results were reproduced or taken from prior papers. Thus, the gains cannot be attributed to the method alone. The ablations help, but comparisons with modality dropout and standard self-distillation should use the same architecture and training budget.
  2. CRM and RTFDNet already transfer knowledge from complete inputs to single-modality models. The main new idea appears to be using task losses to supervise query-level routing. Please clarify the differences from these methods, especially the semantic losses, and compare with standard distillation in the same framework.
  3. The paper claims that replacing either semantic loss with MSE lowers performance. However, Table VI only tests MSE at one level, without testing MSE + prototype anchoring or CRAD + query-level MSE. The gains may therefore partly come from using two levels of regularization. Please add these comparisons and explain how queries are paired for MSE when matching is done independently.
  4. Table V shows a gain, but it remains unclear whether the routing weights reflect how useful each modality-specific update is. Please show routing weights alongside candidate costs and segmentation errors. Please also examine whether supervision at the final layer is sufficient for a router shared across all decoder layers.
  5. The claim that multi-annotator active learning has not been studied for regression overlooks prior work, including A Robust UCB Scheme for Active Learning in Regression from Strategic Crowds(2016). The contribution should focus more narrowly on deep facial age estimation with real crowd annotations.
Questions for authors

The paper has useful resource potential, but the current evidence does not sufficiently support its broad conclusions. Reconsideration would require reproducible algorithm specifications, corrected statistical analysis and reporting, and additional controls for annotator choice and annotation budget. Please refer to the above specific weakness.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer Kwg4

We thank the reviewer for their time and for recognizing the value of the collected annotations, the gap between idealized oracles and real human labels, and the breadth of our evaluation. The updated texts in the paper are marked in red for ease of readability.

Weaknesses 1 - 4. We respectfully note that these points appear irrelevant to our paper. They discuss CQR, ConvNeXt-B encoders, CRM, RTFDNet, modality dropout, query-level routing, prototype anchoring, CRAD and segmentation, none of which appear in our paper, and they reference tables whose contents do not match ours. This appears to be some kind of an error. We would request the reviewer to share any concerns specific to our paper, so that we can address them.

Weakness 5 (prior work on multi-annotator AL for regression). Thank you for pointing us to Padmanabhan et al. (2016). While this paper studies AL for the regression setup with multiple annotators, the annotators have been simulated by adding zero-mean Gaussian noise to the output variables. We, on the other hand, collect annotations from real human annotators to appropriately capture the nuances of human annotation.

We agree that two sentences in our paper overstate our claim: lines 090-092 ("studied only for the classification setup") and Appendix A.5.2 (lines 1082-1084; "multi-annotator AL has not been studied for regression problems"). We now cite Padmanabhan et al. (2016) in the introduction (lines 089-095) and have revised both sentences. We have revised our claim accordingly:

To our knowledge, ours is the first study of deep active learning for vision-based regression (facial age estimation) using real crowd-sourced annotations, rather than annotators simulated using artificial noise.

We have made this framing consistent throughout the paper, including the contribution at lines 108-125.

Questions for authors.

Reproducible algorithm specifications. Appendices A.4 and A.5 give the backbone, training schedule, hyperparameters, and how each acquisition method was adapted to regression. We are happy to expand any particular specification the reviewer finds unclear.

Statistical analysis.

  • Within a setting (one dataset and annotation source), we use the Friedman test to check that differences among methods are not explained by seed variability. It is significant in every setting (all p < 10⁻¹⁰).
  • To compare pairs of methods, we use the paired t-test formulation of Subedi et al. (NeurIPS 2024) on per-seed AULC. All pairwise p-values are reported in the p-value tables of Section 4.1 (Table 2) and Appendices A.1.1-A.1.2 (Tables 4-5), A.6.1-A.6.4 (Tables 9, 11, 13 and 15).
  • In the revised rank consistency table (Table 18, Appendix A.8), we now also report significant reversals: pairs of methods where one is significantly better under one annotation source and the other is significantly better under another. This separates meaningful ranking changes from swaps between methods with nearly identical performance.

Annotation budget. Appendices A.6.1 and A.6.2 vary the initial labeled set size (1%, 2%, 5%, 10%) and the query size (100, 200, 500) on AAFD. Rankings are stable between neighbouring budgets (Kendall's τ = 0.85 for 1% vs. 2%, and 0.92 for query sizes 100 vs. 200), and change as the budget grows (τ = 0.44 for 1% vs. 10%). This shows that the best AL method depends on the labeling budget (computed from rankings in Tables 9 and 11).

Annotator choice. The single-annotator experiments in the paper use annotator A3. We have repeated them with annotators A1 and A2 on both datasets, with the same methods, data splits and 10 seeds on (Appendix A.8, "Effect of Annotator Choice", Table 17). For every annotator, the rankings of AL methods change when ground-truth labels are replaced by human annotations, and these changes include significant reversals on both AAFD and UTKFace datasets in majority experiments (Table 18, Appendix A.8). Our finding that the annotation source significantly affects the relative performance of AL methods therefore holds regardless of which annotator provides the labels.

We thank the reviewer again and would be glad to address any further concerns.

References:

Padmanabhan, D., Bhat, S., Garg, D., Shevade, S., Narahari, Y. A robust UCB scheme for active learning in regression from strategic crowds. International Joint Conference on Neural Networks (IJCNN), pp. 2212–2219, 2016.

Subedi, R., Wei, L., Gao, W., Chakraborty, S., Liu, Y. Empowering active learning for 3D molecular graphs with geometric graph isomorphism. NeurIPS, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.