acceptodds
Withdrawn from ICLR 2027

What Annotator-Conditioned Models Learn Depends on the Text Model: Decomposing the Gain from Personalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In subjective annotation tasks, it is now common to treat each annotator’s judgment as information to be modeled rather than noise to be aggregated away. Annotator-conditioned models improve predictive performance, but it has never been measured how much of that gain is a content-dependent component and how much is an additive, input-independent output bias specific to each annotator. We decompose this gain as a difference between nested predictors and show that its composition depends on both the shared text model and how that model is adapted. With increasingly expressive fixed text representations, the content-dependent share rises from 7.3% for a linear encoder to 30.5% for a frozen Transformer, yet remains additive-dominant. When the same Transformer is instead adapted end to end within each annotator-conditioned model, the decomposition reverses and the content-dependent share reaches 89.3%; this reversal is explained neither by hidden dimension nor by pretraining, and replicates on a second dataset. Even with the same architecture and the same data, freezing versus jointly adapting the encoder changes the content-dependent share from 30.5% to 89.3%. This content-dependent component is not an estimation artifact in which the extra capacity of M2 produces a positive gain even under the null: against labels generated from a model with no interaction, the same estimator returns zero. The composition of the personalization gain is therefore not a fixed property of the annotators but a quantity set by the representation and by how it is adapted during personalization, so there is no model-independent answer to whether a per-annotator intercept suffices or content-dependent personalization is needed. Finally, we show that the verdict itself depends decisively on how the null distribution is constructed.

Primary area:
interpretability and explainable AI
Keywords:
annotator modeling, human label variation, annotator disagreement, content-dependent interaction, representation learning, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AFVJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors measure how much the gains from personalization in subjective tasks arise from the annotator's label distribution compared to their differing interpretation of the stimuli (here, text documents). They find that simpler models derive most of their personalization benefits from that baseline distribution, as opposed to learning complex signals from the input itself, whereas models with higher capacity can indeed learn unique annotator perspectives from the input distribution. They, therefore, caution that interpretation of perspectivist/personalization findings need to take the model's complexity into account when making claims.

Critical strengths
  • The authors identify and study an often-missed baseline in the study of personalization: how much the model learns just by getting information about the marginal label distribution of each annotator.
Critical weaknesses
  • The paper is, plainly put, incomprehensible, in two distinct ways. First, it is very dense. Second, it contains a lot of unexplained jargon. More details:
    • For the former, I present only some concrete examples, but I want to emphasize that this is the case throughout this work: in Section 3.3, what is V~\tilde{V}, ua~\tilde{u_a}, and is "sd" the standard deviation? What is a “post-thread-level split” and what is a "thread"? What is “cross-fitting”, what are "annotator-level" clusters, what is a "cluster bootstrap", and what is “cluster-bootstrap inference”? What is “calibration”? What is “annotator-item assignment” and where is it observed? Is the “synthesized problem” the above yaiy_{ai} labels assigned to each item? What is a “parametric bootstrap” and how did it come about? Why is TaiT_{ai} standardized across datasets? What are “annotator identifiers” and “observation’s identifier”? What is “finite-sample error”, why is it negative? etc etc etc
    • For the latter, one simple example is the abstract: "nested predictors", and M2M_2 are mentioned, but nowhere are they defined. In fact, nested predictors are not really defined anywhere in the paper, which of course gets us back to the point above.
  • The paper contains extensive evaluation of linear and "equivalent" models, whereas contemporary research in perspectivism uses the authors' most complicated models as bare-bone baselines, if at all (I want to note here that the authors don't cite exactly which models they use, but we, the readers, can make reasonable inferences from the names). The lack of any recent models, such as LLMs, is a major flaw in the paper.
  • I want to emphasize that the appendix is meant for supplementary material, and not to circumvent the page limit of the main text. A proper explanation of important material needs to be contained in the main text, even with technical and intricate details missing, and not just minimally referenced in the main paper.
Questions for authors

Given that the lack of clarity and important missing models, I do not believe that a simple QnA during the rebuttal period would suffice to materially change my opinion of the paper. A wholly rewritten paper and new baselines are required.

Reviewer_6eRFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks a narrow but well-posed question: when a model is conditioned on annotator identity, what is the gain made of? The authors decompose the held-out log-likelihood gain from annotator conditioning into (i) a marginal/additive component (per-annotator intercepts, M1 vs. text-only M0) and (ii) a content-dependent component (low-rank annotator and text interaction, M2 vs. M1), all measured on a common scale (held-out predictive NLL on an annotator-averaged scale), with a parametric-bootstrap null and permutation controls for calibration. The central empirical claim is that the composition of the gain is not a fixed property of the annotators but is relative to the shared text representation and how it is adapted: on weak/frozen representations the additive component dominates, whereas end-to-end fine-tuning of the Transformer within each nested model reverses the composition. The authors conclude there is no model-independent answer, and that detection of the content-dependent component depends on representation expressiveness, annotations per annotator, and the construction of the calibration null.

Critical strengths

The paper has a clear, well-motivated question with a genuinely useful reframing. The paper does not dispute that annotator-conditioned models help; it asks what the gain consists of. The observation that a reported total gain "does not tell us how to design personalization" is a clean and practical framing, and the measurement-relativity conclusion is a defensible and non-trivial contribution.

The estimator is simple, uniform, and appropriately calibrated. The nested-model decomposition (M0 ⊂ M1 ⊂ M2) is applied identically from linear/TF-IDF predictors through frozen Transformers to end-to-end fine-tuned Transformers, which makes the cross-architecture comparison meaningful.

Critical weaknesses
  1. The methodology section is compressed, and key objects are used without proper introduction. M0, M1 and M2 appear rather abruptly as Equations (1)–(3); the reader is told they are nested but the motivation for why this particular parameterization (log-odds-additive intercept, rank-r bilinear interaction, centered φ(x)) is only partly explained. In particular, I can see the rank r of the interaction in M2 is central to the estimator, it only appears in Appendix N. The main text would be better to state this explicitly and report the selected ranks alongside Table 1/Table 2. Similarly, the centering of φ(x) and its role ("the interaction cannot absorb a per-annotator constant offset") is asserted but not derived. The annotator-averaged NLL scale (equal weight per annotator) is defined in one clause. Given that it drives every reported number, it deserves a clearer definition and a justification vs. item-averaged NLL.

  2. The other concern is about the number of annotators (all over 57). The four corpora have 57 (GoEmotions), 872 (CSC), 504 (MP), and 74 (EPIC) annotators, so the empirical claims live in the many-annotator regime. The synthetic identifiability study (Appendix G, Figure 4) does vary the number of annotators (92% power at 20, 100% at 200), which partially addresses this, but the composition (the share, not just detection power) is not characterized as a function of the number of annotators. How about a lower number of annotators, is your claim still valid?

Questions for authors
  1. Is r in M2 selected per fold, per condition, or globally? What are the selected ranks for each row of Tables 1 and 2, and how sensitive is the content-dependent share to fixing r = 1 or r = 4? (Appendix N reports r = 4 selected in all five folds for 28 labels. Can you justify this holds across the other corpora and splits?

  2. Given that all four corpora have over 57 annotators, how does the composition (not just detection power) behave as A is reduced?

  3. Since the frozen encoder is fine-tuned on the M0 objective, from 30.5% to 89.3% is generally understandable as it was fine-tuned. Did you try a control in which the encoder is fine-tuned on a different objective (e.g., M1 or a multi-task objective) and then frozen, to isolate geometry from joint adaptation?

  4. Given EPIC has the most annotations per annotator, what explains its non-detection? Is it low annotator-level variability (the cluster interval contains 0), a genuinely additive structure, or insufficient power at 74 annotators?

  5. The paper argues that the calibration null must preserve the empirical prediction problem. For a practitioner without the ability to fit a parametric bootstrap, is the within-item permutation a sufficient substitute, or does its exactness only under β_a = 0 limit its use?

  6. Why equal weight per annotator rather than per item? How much do the reported shares change under item-averaged NLL, particularly for CSC (872 annotators, median 29 annotations) vs. GoEmotions (median 2,649)?

Reviewer_DwZNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To model subjective tasks, one prominent strategy is to condition on the annotator: learn individual annotators instead of aggregating over multiple annotators to a single ground-truth. This paper addresses this annotator-conditioning by investigating to what extent gains in predictive performance stem from the general pattern of the labeling habit (marginal bias) of the annotator and the interaction of text and annotator representations (content-dependent).

This decomposition between marginal bias and content-dependent association is investigated by “nested predictors” and comparing the difference in performance between the three predictors:

  1. Text-only
  2. Text with the marginal bias (how often does an annotator use a specific label)
  3. Text with annotator identity (content-dependent)

To get the gains from the two different components, the negative log-likelihood (NLL) is computed. To isolate the gains from the marginal bias, the NLL of predictor 2 - predictor 1 is computed. For the gains from the content-dependence, the NLL of predictor 3 - predictor 2 is computed.

The authors use four datasets: GoEmotions, LeWiDi CSC and MP, and EPIC datasets and find that gains from both components depend on how strong the representation of the annotators are. As representations get better (e.g. from linear predictor to Transformer), the gains from content-dependency increase and overtake the gains from marginal bias. This also goes hand in hand with keeping the representations fixed or training them end-to-end.

Critical strengths
  • The paper addresses a timely topic as the field is moving away from reducing multiple annotations to a single label to actually embracing subjectivity. As such, this decomposition of sources of gains when learning annotator labels is interesting and important to study. The results can further help in understanding whether models are actually learning something substantial or not. If it is mostly marginal bias, then that is an important and interesting finding to know.
  • I like the simplicity of the approach and I feel it addresses the research question at hand properly. Using the three nested predictors to measure the gains is neat and then isolating the marginal bias and content-dependent association also makes sense.
Critical weaknesses
  • I would have liked a proper subsection dedicated to which models are used, even if brief. Also a bit more motivation for why they chose the models they use and why the frozen vs. adapted. Now, for me it is a bit difficult to gauge why we are necessarily interested in e.g. linear predictors for this specific task as usually transformers are the go to (or even autoregressive models these days). Also, for transformers, is only distilroberta used? I think model size would have been an interesting addition here.
  • I feel the paper would benefit from a round of re-writing. It took me a couple of re-reads to grasp exactly what is meant at times. For instance, I did not understand what Section 4.5 is exactly trying to convey? It seemed more like an overview, but what do these results actually mean in the larger picture? It would have been nice if it was a bit more elaborate.
  • An example figure at the beginning would improve the reading experience.
Questions for authors

N/A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.