- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper measures how LLMs trade democratic institutions against material and social outcomes. It replays the conjoint tasks of Chu et al. (2025), answered by 4,130 respondents in the US, India, Italy and Thailand, through 23 LLMs, each prompted with the background of the respondent who faced the task, and uses a cross-fitted logistic model that answers deterministically as a baseline separating the answering rule from the model's weights. Models match individual choices above chance, but most under-penalize unfair elections, and in 19 of 23 models a high-income economy offsets a democratic loss more than low corruption does, reversing the respondents' order; LoRA fine-tuning on respondents' choices restores that order in three open models.
Critical strengths
This is the right measurement and it is new. Stated support for democracy is near universal, which is why political science measures commitment by what people are willing to trade for it, and porting that logic to LLMs on the respondents' own randomized tasks is a genuine contribution. The fitted logistic baseline is the best methodological decision in the paper and should be imitated: a temperature-0 model answers deterministically, which by itself inflates the seven penalties 1.60-1.85x, and most LLM-versus-survey work never separates that from the preference. Robustness is broad (persona variants, prompt formats, survey languages, 576 free-text narratives) and null results are reported as such.
Critical weaknesses
Mostly framing, nothing fatal. The abstract is more negative than the results: Section 4.3 shows that of the nine human orderings with intervals excluding zero, most models reverse only two, so the story is one consistent exchange-rate error (corruption vs economy) rather than broad divergence, which is a sharper and better supported claim. Relatedly, the paper itself notes the reversal is the economy weight "expressed as a ranking," so 4.3 largely re-expresses 4.2. The human corruption-economy gap is not established in every country (Thailand's interval touches zero), yet reversals are counted there as well. Finally, the main replay stored only the parsed answer, so the main analysis sits on the decision scale the paper itself shows is contaminated by the answering rule; storing next-token probabilities for the open-weight models would make 4.2-4.3 much harder to argue with. Three GPT-5 models also ran at provider defaults rather than temperature 0. A careful paper with an important result.
Questions for authors
- Do the election penalty and the corruption-economy reversal survive on the probability scale for the open-weight models?
- How should the reversal counts be read in countries where the respondents' own corruption-economy gap does not exclude zero?
- Section 4.4 finds minority heterogeneity amplified and education heterogeneity compressed in every model. Does that suggest the persona elicits a stereotype of the described person rather than the respondent's trade-off?
2 comments
Framing. Thank you for this suggestion. We will make the specificity of the finding explicit in the abstract and introduction: the median model weighs a low-income economy 1.38 times as heavily as unfair elections, against 0.46 for the respondents (20 of 23 models), and of the nine orderings the respondents establish, a majority of models reverse only two, corruption against the economy and corruption against minority treatment (Table E.2). On §4.3: as the paper notes (L373–375), the reversal is the §4.2 economy weight expressed as a ranking. §4.3 reports the ranking because it is the form in which a study using the models would draw conclusions, and it shows that the economy outranks low corruption in 16 to 21 of 23 models within each country and contrast, and in all 16 most accurate models (L378–384). §4.3 will open by stating this relation.
GPT-5 models at provider defaults. The paper's counts do not depend on these three models. When the three GPT-5 models are excluded and the BH correction is recomputed over the remaining 20 models, 17 have a negative corruption–economy gap (15 beyond the tie band with q < .05), and 16 penalize unfair elections less than the respondents (15 with q < .05). In the new no-persona runs, the smaller election penalty also appears in the three GPT-5 models at provider defaults as well as the three GPT-5.4 models at temperature 0 (all six with p < .05).
Q2. Countries where the respondents' gap does not exclude zero. We agree that "reversal" should be used only where the respondents' own ordering is established, and we will revise the text accordingly. In the US and Italy, where the respondents' interval excludes zero (Table B.1(b)), 21 and 20 of 23 models have a negative gap (point estimates). For all four countries, we also report the difference between each model and the respondents, which does not depend on the sign of the respondents' gap. It is shifted toward the economy in:
| Country | Models below respondents | Significant (q < .05) |
|---|---|---|
| US | 22 | 21 |
| Italy | 21 | 20 |
| Thailand | 20 | 16 |
| India | 18 | 12 |
L378–381 will say "reversal" for the US and Italy and "shifted toward the economy relative to respondents" for India and Thailand.
Q3. Stereotypes of the described person. These differences may reflect a stereotype the models hold of the described group. Our design cannot establish whether this is the cause, but we consider it a plausible interpretation, since each prompt gives only the respondent's background (L139–143) and the respondents' own choices barely differ by minority status (L398–400). We will note this interpretation in §4.4.
We will upload the revised PDF with these changes during the discussion period.
Thank you again for engaging so closely with the paper. Your suggestions make the main claim sharper, and the new evidence gives §4.2–§4.3 a firmer basis.
Thank you for your thoughtful and encouraging review, and for recognizing the motivation behind our measurement, the fitted logistic baseline and the range of robustness checks. We conducted new analyses to address your questions. In summary:
Line numbers refer to the submitted version.
Q1. Probability scale. As the paper notes (L251–253), the main replay stored only the final answers. The three results below show, in the four open models whose token probabilities we could score, that the differences behind §4.2–§4.3, the smaller election penalty relative to the economy and the corruption–economy gap, are not produced by deterministic answering: they appear on the probability scale, and the two scales agree.
1. New runs that record answer-token probabilities. If deterministic answering created the differences, the probability scale would not show them. We ran three open models without a persona on 1,033 respondents (design in our response to Reviewer A2jd, 3/3). We take the log-probabilities of the token carrying the answer letter (top 20 stored), merge variants of the same letter, and normalize over A and B. When one letter falls outside the stored top 20 tokens (0, 19 and 1,330 of 3,099 tasks), we set it to zero. Its probability is then at most one minus that of the chosen letter: at most 2.4×10⁻⁴ for Gemma-4-26B and 3.2×10⁻⁶ for Gemma-4-31B, too small to change the sign of any reported gap or whether it falls outside the ±0.02 tie band. As in the main replay, the decision scale is the generated answer, which matches the most probable letter in 98.8%, 99.7% and 100% of these tasks. Qwen3.5-27B's probabilities are less concentrated than the Gemma models' (median 0.93 for the chosen letter; below 0.9 in 42.8% of tasks), so for this model the agreement of the two scales is not automatic. On this sample, the respondents' gap is +0.054 and their ratio 0.38.
The two scales agree closely (gaps within 0.012, ratios within 0.02). For the two Gemma models, the election-penalty difference has the same direction but is not significant.
2. The §4.6 base models, with personas, on a separate sample. This extends item 1 to persona-conditioned prompts, the paper's main condition. Table G.6 reports only point estimates for these models. On the probability scale, the gaps of Qwen3.5-9B, Qwen3.5-27B and Gemma-4-26B-A4B fall below the respondents' by 0.10, 0.18 and 0.13 (p = .043, .001 and .011; respondents +0.098 on that sample). For Qwen3.5-9B the model's own gap is a tie (−0.002), so the difference is a shift toward the economy.
3. The penalty ratio. The §4.2 ratio changes little with the answering rule. Answering with its probabilities, the fitted logistic model reproduces the respondents' unfair-election penalty (−0.189 vs −0.190). Answering deterministically, it inflates the seven penalties by similar factors (1.60–1.85), so its ratio stays near the respondents' (0.48 vs 0.46), and its corruption–economy gap stays positive (0.066 vs 0.046; L370–371). In the new runs, the ratio is nearly identical on the two scales (table above). In the main replay, the ratio exceeds the respondents' in 20 of 23 models (19 with q < .05 in a new paired-bootstrap test; GPT-5.4-Nano's ratio cannot be tested because its unfair-election penalty is near zero), and the unfair-election penalty, as a multiple of the respondents', falls below the model's average multiple in 22 of 23 (Table D.3). Both comparisons are robust to the near-uniform inflation the answering rule produces.