- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors measure how much the gains from personalization in subjective tasks arise from the annotator's label distribution compared to their differing interpretation of the stimuli (here, text documents). They find that simpler models derive most of their personalization benefits from that baseline distribution, as opposed to learning complex signals from the input itself, whereas models with higher capacity can indeed learn unique annotator perspectives from the input distribution. They, therefore, caution that interpretation of perspectivist/personalization findings need to take the model's complexity into account when making claims.
Critical strengths
- The authors identify and study an often-missed baseline in the study of personalization: how much the model learns just by getting information about the marginal label distribution of each annotator.
Critical weaknesses
- The paper is, plainly put, incomprehensible, in two distinct ways. First, it is very dense. Second, it contains a lot of unexplained jargon. More details:
- For the former, I present only some concrete examples, but I want to emphasize that this is the case throughout this work: in Section 3.3, what is , , and is "sd" the standard deviation? What is a “post-thread-level split” and what is a "thread"? What is “cross-fitting”, what are "annotator-level" clusters, what is a "cluster bootstrap", and what is “cluster-bootstrap inference”? What is “calibration”? What is “annotator-item assignment” and where is it observed? Is the “synthesized problem” the above labels assigned to each item? What is a “parametric bootstrap” and how did it come about? Why is standardized across datasets? What are “annotator identifiers” and “observation’s identifier”? What is “finite-sample error”, why is it negative? etc etc etc
- For the latter, one simple example is the abstract: "nested predictors", and are mentioned, but nowhere are they defined. In fact, nested predictors are not really defined anywhere in the paper, which of course gets us back to the point above.
- The paper contains extensive evaluation of linear and "equivalent" models, whereas contemporary research in perspectivism uses the authors' most complicated models as bare-bone baselines, if at all (I want to note here that the authors don't cite exactly which models they use, but we, the readers, can make reasonable inferences from the names). The lack of any recent models, such as LLMs, is a major flaw in the paper.
- I want to emphasize that the appendix is meant for supplementary material, and not to circumvent the page limit of the main text. A proper explanation of important material needs to be contained in the main text, even with technical and intricate details missing, and not just minimally referenced in the main paper.
Questions for authors
Given that the lack of clarity and important missing models, I do not believe that a simple QnA during the rebuttal period would suffice to materially change my opinion of the paper. A wholly rewritten paper and new baselines are required.