What Annotator-Conditioned Models Learn Depends on the Text Model: Decomposing the Gain from Personalization

In subjective annotation tasks, it is now common to treat each annotator’s judgment as information to be modeled rather than noise to be aggregated away. Annotator-conditioned models improve predictive performance, but it has never been measured how much of that gain is a content-dependent component and how much is an additive, input-independent output bias specific to each annotator. We decompose this gain as a difference between nested predictors and show that its composition depends on both the shared text model and how that model is adapted. With increasingly expressive fixed text representations, the content-dependent share rises from 7.3% for a linear encoder to 30.5% for a frozen Transformer, yet remains additive-dominant. When the same Transformer is instead adapted end to end within each annotator-conditioned model, the decomposition reverses and the content-dependent share reaches 89.3%; this reversal is explained neither by hidden dimension nor by pretraining, and replicates on a second dataset. Even with the same architecture and the same data, freezing versus jointly adapting the encoder changes the content-dependent share from 30.5% to 89.3%. This content-dependent component is not an estimation artifact in which the extra capacity of M2 produces a positive gain even under the null: against labels generated from a model with no interaction, the same estimator returns zero. The composition of the personalization gain is therefore not a fixed property of the annotators but a quantity set by the representation and by how it is adapted during personalization, so there is no model-independent answer to whether a per-annotator intercept suffices or content-dependent personalization is needed. Finally, we show that the verdict itself depends decisively on how the null distribution is constructed.