- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the social choice and incentive properties of latent factor models used in Community Notes. It interprets the fitted quality of an item as its evaluation from a particular ideological viewpoint, referred to as an anchor. The analysis shows that different anchors correspond to different objectives: the mean maximizes social welfare, while the median satisfies the Condorcet criterion. The paper further examines strategic misreporting and participation incentives. It establishes strategyproofness and group strategyproofness for median-anchored mechanisms under ideological spoofing, while showing that mean-anchored mechanisms are vulnerable to manipulation. The theoretical results are supported by simulations and an observational analysis of participation patterns in Community Notes data.
Critical strengths
- The paper establishes a useful connection between latent factor estimation and social choice. The analysis shows how the choice of anchor determines the social objective of a fitted model. The results on identifiability, re-anchoring, and strategyproofness provide a coherent theoretical account of a practical rating mechanism. In particular, Theorems 5 and 6 clarify why the median anchor is resistant to ideological spoofing under the stated conditions. This connection between statistical estimation and strategic incentives is a substantive contribution.
- The paper examines both rating manipulation and participation incentives. The participation analysis adds an important dimension to the strategyproofness results by showing how the composition of the rating population can change the outcome. The simulations illustrate the differences between mean- and median-anchored mechanisms, while the Community Notes analysis provides supporting observational evidence. The effort to address estimation-related confounding in the empirical analysis is also valuable.
Critical weaknesses
- The strategyproofness guarantees rely on restrictive assumptions that limit their applicability to deployed systems. The main results assume a one-dimensional ideological structure with strict crossing and focus on ideological spoofing. The strongest guarantees are established for exact ratings, while real Community Notes data are noisy, incomplete, and subject to more general forms of manipulation. Although Appendices F.5 and F.6 extend parts of the identification analysis to noisy and partial ratings, they do not establish the same general strategyproofness guarantees for realistic fitted models. The simulations provide supporting evidence, but the gap between the theoretical mechanism and the deployed system remains substantial.
- The practical advantage of median anchoring is not sufficiently established. The paper suggests that median anchoring may be preferable to the deployed mechanism, but its protection primarily concerns misreporting within a fixed rating population. Section 4.2 shows that participation itself changes the anchor, and the turnout analysis identifies cases where equilibria do not exist or where the participating population differs substantially from the full population. Median anchoring also sacrifices social welfare relative to mean anchoring in some settings. The empirical analysis supports an association between ideological distance and rating activity, but does not establish that adopting median anchoring would improve outcomes in practice. The recommendation therefore goes beyond what the current evidence directly supports.
Questions for authors
- Which parts of the strategyproofness results remain valid when ratings are noisy and incomplete? In particular, can the authors clarify the implications of Appendices F.5 and F.6 for manipulation of a regularized, median-anchored fitted model?
- How should the strategyproofness guarantee be interpreted when agents can strategically choose whether to participate? Does the participation analysis identify conditions under which median anchoring remains robust to changes in the rating population?
- What evidence supports the recommendation to prefer median anchoring over the deployed mechanism, given the welfare trade-off and the observational nature of the Community Notes analysis?
4 comments
Response to Reviewer 3 (Additional Details)
On Incomplete Ratings and Other Manipulations (W1, Q1)
Incomplete ratings at random are covered: the remark after the framed-ratings lemma in App. F.6 gives Thm. 5 whenever the ratings stay framed once the coalition is removed. Its one exception is a note rated by the coalition alone, which the deployed minimum-rating filters discard. Since the remark was read as identification only, we will promote it to a proposition.
As for other manipulations, any deviation that is not a spoof breaks the fit and could be detected (Lemma 9). When the deviating accounts are fewer than half of a note's raters, they are also identified. This is the class the deployed screens target; the second stage of the attack in Selvam et al. is an example. Noise is treated separately in the comment "On Discrete and Noisy Reports". We would also make it clear that strategic partial ratings are outside the scope of our work but are a promising direction.
On participation (W2, Q2)
Strategyproofness is a statement about reports, which we analyze under spoofing. Turnout, however, is a second channel, which we analyze in Sec. 4.2. We first establish why a more ideologically extreme participant derives a greater benefit from rating than from abstaining, and then analyze the equilibria under different anchors. Under the median, the raters who abstain form a band of moderates, which narrows as the cost falls (Prop. 13). Under the mean, by contrast, a rater's influence shrinks as more agents rate, so only a few raters in the tails rate (Prop. 14). The turnout equilibria of Fig. 6 show the consequence: the median anchor stays within 0.13 of the population median at every cost, while the mean anchor drifts 1.2 to 2.3 into the heavier tail. Pure equilibria need not exist under either anchor, and at low cost they often do not, so these statements concern the equilibria that exist. Within them, the comparison survives turnout in a precise sense: the median anchor moves less, up to the abstaining band. We will say so in Sec. 4.2 and the Discussion.
Thank you for the detailed rebuttal and clarifications. I appreciate the authors' efforts to address my concerns, but I will maintain my initial score.
Dear reviewer,
Thank you again for your response and helpful feedback. Please let us know if you have any other questions.
Kind regards, authors
Response to Reviewer 3
We thank the reviewer for the careful reading. The two weaknesses and three questions all concern the distance between the theorems and the deployed system, so we answer them in that light. Given the character limit on responses, the points shared with the other reviews are addressed fully in the topic comments at the top of the forum, which we reference by their headers, and the rest in a separate comment in this thread.
Assumptions (W1). The body presents the results for the latent factor model because that is the model the deployed system fits, and it was also a simple case to analyze initially. The results, however, hold on a broader class, which App. D defines by two axioms: types are ordered, and quality is the utility of some type, and the utility difference between two items changes sign at most once along the order. Strict crossing is exactly what the median needs: the appendix gives instances where it fails, and the median loses identifiability or strategyproofness. One ordered dimension is likewise what the deployed model fits and the simplest setting in which the question can be posed. It also had no prior incentive analysis. Several dimensions require a condition on the factor geometry, since there is no strategyproof median unless the dimensions separate. We will state this as open in the Limitations (topic comment: On Clarifying Contributions and Model).
Noisy and incomplete ratings, and the regularized fit (W1, Q1). Incomplete ratings at random are already covered: Thm. 5 holds whenever the ratings stay framed once the coalition is removed (App. F.6). Since the remark was read as identification only, we will promote it to a proposition. For noise, the reviewer is right that App. F.5 gave the anchoring structure and not the strategyproofness, so we extended the theory (topic comment: On Discrete and Noisy Reports). We take the report to be a stochastic function of the utility whose expectation increases with it; the expected reports are then the exact reports of another instance of the class, and since a spoof is a draw at another type, Thm. 5 holds for the population fit. In sample the median's guarantee holds up to the estimation error of the fitted anchor, which we prove, and the reruns of Fig. 3 show that error does not move with the coalition. For the regularized fit the penalty leaves a residual that the same reruns show does not move with the coalition. Deviations that are not spoofs break the fit and are detected whenever the honest raters of a note are the majority (Lemma 9; detail in this thread). Strategic choice of which notes to rate is outside the scope of the paper, which the Limitations will state. The revision replaces App. F.5 with this subsection.
Participation (W2, Q2). Strategyproofness concerns reports, while turnout is a second channel, so the question is whether the comparison between anchors survives it. It does, in the sense that the median anchor moves less. Under the median only a band of moderates abstains, a band that narrows with the cost (Prop. 13), whereas under the mean only a few tail raters rate (Prop. 14). The turnout equilibria of Fig. 6 show the consequence: the median anchor stays within 0.13 of the population median at every cost, while the mean anchor drifts 1.2 to 2.3 into the heavier tail. Pure equilibria need not exist under either anchor, and at low cost they often do not; these statements concern the equilibria that exist, and we will say so. The condition the reviewer asks for is then that the median anchor is robust to turnout up to the abstaining band, which we will state in Sec. 4.2 and the Discussion (detail in this thread).
The recommendation (W2, Q3). We agree it reads as a blanket recommendation and will rewrite it as a decision with a trade-off: welfare under truthful reports (Fig. 4), which the mean does not reward, against robustness to consistent exaggeration and to turnout (topic comment: On the Recommendation of the Median Anchor). As for evidence, we can offer two kinds. On the public Community Notes data, switching the deployed anchor to the median changes the shown note on about 15% of the posts that have one, in the predicted direction, so the choice has consequences today (topic comment: On Applicability and Connection to Community Notes). Only an experiment or a deployment can show the incentive response, which is where the welfare trade-off plays out. We will state that as the boundary of the claim.
We appreciate the questions, which sharpened our understanding of where the theory needed to be extended, and would welcome any further pointers to places where the argument could be strengthened.