- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work seemingly aims to achieve reliable race and gender inference through a transfer-learning-inspired model training strategy and the constructed demographic attribute inference datasets. This work also claims that the proposed REGI simultaneously improves accuracy, fairness, and robustness compared with widely used systems.
Critical strengths
- This work seems to contain a lot of information but is not clear. Please correct me if I misunderstand.
Critical weaknesses
- I suspect this work was generated by an LLM, but the author didn't organize it properly. Garbled color blocks always appear at the top of Figure 1, and the main text is quite strange to read. This is detrimental to the sustainable development of academia. However, please correct me if my judgment is incorrect.
Questions for authors
If possible, could the author please explain this sentence in "Statement on the Use of Large Language Models": "LLMs were also used for editing assistance on the manuscript; all technical content, proofs, experiments, and claims are the authors'." This would help clarify any misunderstandings.
4 comments
Contributions. We summarise here the list of our contributions:
- We define new taxonomy-agnostic robustness metrics, which we consider necessary (but not sufficient) for auditing any annotator.
- We show that widely used annotators do not perform well along the three evaluation axes we deem important: accuracy, fairness, and our robustness metrics.
- We provide a class of functions that comes with a robustness guarantee, and show that an SVM with an RBF kernel falls into this class.
- We show that this guarantee can be expressed as an upper bound driven by a signal-to-noise ratio: the margin of an identity along the decision direction, divided by the spread of its images in that direction. When identities are modelled as isotropic vMFs, the bound decreases exponentially with the concentration of the images of the same person. Face-recognition angular-margin losses are expressly designed to shrink this spread and model identities in this way; we made this explicit in the revision (p. 4, l. 204–205).
- We therefore present a face-recognition backbone with an SVM head as a theoretically motivated annotator.
- We note that the margin, i.e., the numerator of this ratio, is theoretically directly linked to the diversity of the training data.
- We thus create an annotator training set, WEG$^{tr}g$), but also with respect to finer ethnicities ().
- To do so, we use our own theoretically motivated sampler, which balances and simultaneously and preserves , which we consider a robust descriptor of the relation between and .
- At the same time, we create WEG$^t$, a more challenging benchmark, diversified along and .
- We empirically verify the impact of WEG$^{tr}$ and of the RBF-SVM head on accuracy, and show that both increase performance, which rules out the possibility that the face-recognition backbone alone drives the performance.
- We compare the resulting annotator with existing ones: it reaches state-of-the-art performance for ethnicity inference and competitive performance for gender inference (second only to the much larger QWEN3 models), at a limited computational cost.
- We show that the performance remains remarkably stable when changing the size and the type of the backbone, the size of the SVM training set, the size of the backbone training set, the balancing of the backbone training set, and the loss used to train the backbone.
- Illustrating the importance of taxonomy-agnostic metrics, we also use our robustness metrics to compare REGI with existing models based on the skin-color taxonomy, which highlights the instability of these approaches.
We could also add the impact of the annotator performance on the estimation of the fairness of face-recognition models: low-performance annotation leads to an underestimation of the bias of the audited models. However, as highlighted by the reviewer, this work already "seems to contain a lot of information".
Official Comment by Reviewer Zo1e
Dear Authors,
Thank you for your further responses; they really help me and readers to further understand your work.
However, I still believe that this work has problems with its language and formatting for human readers, and these problems remain unresolved. These include, but are not limited to, some results in Figure 1 being obscured, and the font sizes in the appendix figures and tables varying significantly. This demonstrates the author's weak seriousness towards their work.
These are fundamental issues concerning the high quality publication standards. I believe this work will be good and meaningful if the relevant content is carefully revised, even if it is not currently done.
Best,
Reviewer Zo1e
Dear Reviewer, We take note of your comments and regret that they are not based more on specific elements of the article. We will leave it to the AC to determine whether your comments comply with the reviewer guidelines, as noted in our initial answer.
On the structure of the paper We agree that the paper contains a lot of information. We believe that doing so is required when tackling such a sensitive matter, due to the very nature of the output of the annotators on one hand, and their wide use in the fairness literature for mitigating and auditing bias on the other hand. We are, however, surprised by the statement about our work organisation. It follows a standard structure in the literature: theoretical characterisation of the problematics tackled by the paper in Section 3, implementation of a design answering these problematics in Section 4, and experimental results with relevant ablatives in Section 5.
Figure 1. The colored blocks are a rendering bug of Chromium's built-in PDF viewer, not part of the figure (see the corresponding Chromium fix: https://chromium.googlesource.com/chromium/src/+/0942c4d6e2b66644ff5eb69e925dec86162bf60d). The figure displays correctly in any other PDF viewer. This bug is actually expected for figures exported from draw.io, i.e., it is, if anything, proof that the figure is human-made. To judge an article based on it does not look aligned with ICLR Guidelines.
Overall, we therefore remind the reviewer that the ICLR 2027 Reviewer Guidelines (https://iclr.cc/Conferences/2027/ReviewerGuidelines) ask reviewers to "read the paper carefully, check arguments in sufficient detail to convince yourself of the paper's main claims", and to "only discuss points that are actually likely to influence your final accept/reject decision", adding that even "pointing out typos [...] just add[s] to everyone's workload without changing the final conference program". A rendering artefact that depends on the PDF viewer is not even a typo, and it says nothing about the question the guidelines ask reviewers to answer: "Does the paper support the claims?" Beyond the rendering artefact, the Paper Strengths reported by the reviewer raise serious doubts about the seriousness with which they actually read the paper.
Use of LLMs. We use LLMs for editing because we are not native English speakers: they correct our grammar, spelling, and structural mistakes. The (now corrected) grammar error in the very title is a very good illustration of why we need them to correct us. All technical content, proofs, experiments, and claims are ours, as the statement says; we made it more explicit in the revision (l. 540–541).