- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates whether telling a VLM to answer as a young, middle-aged, or older adult changes its ratings of the 900 OASIS images the way human age groups actually differ. 17 models match pooled human valence and shift strongly under the older persona, but their per-image shifts do not correlate with human per-image age differences, while they correlate with each other. On pleasant sensitive images the models move opposite to older human raters.
Critical strengths
- The paper's framing asks the right question (same images, same direction as humans) and controls for the obvious confound (pleasantness explains most of the human age effect), with ceilings, FDR, and bootstraps.
- The negative result is clean and useful: a shared, reliable model “older persona” that is orthogonal to human data and reverses it on a specific content class.
Critical weaknesses
-
The headline result (“no model tracks human age effects”) rests on a target that is not measurable and on the wrong comparison. The human reference is thin and inconsistently reported: 25 raters aged 60+ for valence (16 for arousal). The per-image human effect is a linear age slope fit mostly on under-60 raters, while the model effect is a step at 60, and the attainable ceiling is 0.86 in the abstract/§4.1 but 0.60 in Table 1/§4.4. Moreover, the human age effect is SD ≈ 0.1 points per SD of age on a 7-point scale, in contrast to model persona shifts which are 0.2-1.5 points. The clearest results are the magnitude mismatch (caricature) and the sensitive-content gate, not r_part against a near-noise target on which the framing rests. Finally, the wrong comparison is performed. OASIS older adults rated images as themselves, where the models were asked to imagine being 60+. The right control for a role-play prompt is humans asked to role-play an older rater (the classic stereotype paradigm), who would likely also produce frailty/nostalgia and miss item-level effects. Without that baseline, the paper cannot say whether VLMs are worse than people at perspective-taking or merely as bad, and the “stricter test for personalization” demands item-level fidelity no evidence shows humans can deliver.
-
The stereotype and mitigation results are confounded or inconclusive. The prompt induces the “stereotype language.” §B.2.1 explicitly requests the description “from the perspective of the specified age group.” Asking for perspective-taking and then counting nostalgia and grandchildren as stereotype content is partly circular. Instead, the rating fields would be the cleaner evidence. Furthermore, the mitigation study gets headline status but is inconclusive: one model, 300 images, best arm chosen post hoc, gains not significant after correction, and the grounded agents’ direction agreement matches a pleasantness-only rule.
Questions for authors
- Which ceiling applies, and what are the corrected “% of attainable” numbers?
- Does a per-image 60+ vs. under-60 contrast (instead of a linear slope) change the fidelity result?
- Replicate the best mitigation arm on the held-out 600 images?
- What do young humans asked to rate “as a 60+ adult” produce on these images (even a small pilot)?
3 comments
Response to Reviewer pZ9J (Part 2/3)
-
W1c. Magnitude: a 0.1-point human effect vs 0.2 to 1.5-point model shifts
-
Response: We agree the magnitude mismatch is a finding in its own right. The two numbers are in different units, though: the human value is points per SD of age, and the model values are older-minus-young points. We put both in the same units (rating points, older at 67 vs young at 23.5 years).
Table R2. Human and model image-specific age effects in the same units (valence).
SD across images Ratio to human SD Human, image-specific effect 0.441 1 Human, after removing pleasantness 0.281 1 Models, median of 17 0.627 (range 0.327 to 0.852) 1.42 (2.20 after removing pleasantness) In matched units, model shifts are a median 1.4 times the human ones (2.2 times after removing pleasantness). The caricature is real, and the revised paper reports it in these units; the human target is not near noise (0.441 points SD; 691 of 900 images carry a permutation-supported effect).
Reported in matched units, the caricature becomes a quantified finding rather than an artifact of comparing slopes with contrasts.
-
-
W1d / Q4. Should the comparison be humans asked to role-play an older rater?
-
Response: We agree this is an informative baseline, and we did not run it. We address the reviewer's argument in three parts.
(i) The target follows from the use case. Persona simulation is proposed as a stand-in for the group's own responses (Argyle et al., 2023; Aher et al., 2023; Santurkar et al., 2023 compare with real group members). The question our paper answers is whether the persona matches the group it claims to represent. Whether VLMs are worse than people at imagining older adults is a different question, and the paper makes no claim about it.
(ii) We expect role-play humans to show the same pattern, and that supports our interpretation. Lacey et al. (2006) found that younger and older adults both believe happiness declines with age, while self-reported happiness rises with age. A role-play baseline would likely show the frailty and nostalgia pattern, which is what we mean by stereotype: the model persona matches a common belief about ageing rather than the measured responses of older raters.
(iii) Item-level fidelity is attainable from human data. Plain young-rater means predict the ratings of real 60+ raters better than any model variant (median RMSE 0.583 vs 0.756 with no age and 0.789 with the older persona), and the S4 arm, given human evidence from other images, reaches r_part = 0.105 [0.014, 0.196] on 600 held-out images. The stricter test can be met with human data; prompting does not meet it.
Revised text (Discussion):
"Human role-players may share the models' pattern, as people of all ages expect affect to decline with age (Lacey et al., 2006); our target is the represented group's own ratings."
The missing baseline would refine how the stereotype is interpreted, but it would not change the fidelity result, which is defined against the group the persona claims to represent.
-
-
W2a. The prompt requests a description from the persona's perspective, so the stereotype language is partly induced
-
Response: We agree the rating fields are the cleaner evidence. Ratings are generated before the description in the JSON field order (Appendix B.2.2), and we ran two prompts that request no description at all.
Table R3. Valence with no description requested (six models, 300 images).
Prompt Models with negative older − young Median shift on pleasant flagged (17) r_part differs from submission Submission prompt 6/6 −1.09 n/a Rating-only, third person 6/6 (largest p = 0.001) −1.39 0 of 6 Direct rating only, first person (5 models) 4/5 −0.600 0 of 5 Without a description, the downshift and the reversal on pleasant flagged images remain, and the reversal becomes larger under the third-person prompt. The perspective framing also does not produce the memory content by itself: in the same template with the age sentence replaced, a no-age "cautious person" uses memory words in 0 to 1% of descriptions, against 4 to 72% for the older persona and 7 to 78% for "retired." We agree that word counts alone are partly induced; the stereotype claim rests on the ratings and the role controls. One change: the arousal downshift holds in 2 of 6 models under the third-person prompt, so we scope the arousal claim to first-person prompts.
The cleaner evidence the reviewer asked for, ratings with no description, therefore supports the valence stereotype claim more strongly than the original prompt did.
-
Response to Reviewer pZ9J (Part 3/3)
- W2b / Q3. The mitigation study: one model, 300 images, best arm chosen post hoc, gains not significant, direction agreement matches a pleasantness-only rule
-
Response: We agree with each element and addressed them with a held-out replication. The fixed S4 arm (same prompts, model, and parameters) was run on the 600 OASIS images outside the mitigation subset (3,707 calls, 100% valid).
Table R4. Fixed S4 arm on development and held-out images.
Images System Direction agreement Mean absolute error r_part [95% CI] Held-out 600 S0 direct prompt 38.1% 0.511 −0.025 [−0.101, 0.053] Held-out 600 S4 grounded panel 85.3% 0.210 0.105 [0.014, 0.196] Held-out 600 Pleasantness-only rule (no model) 94.3% 0.133 n/a Development 300 S4 grounded panel 87.5% 0.205 0.268 [0.161, 0.373] On held-out images, S4 is above zero (p = 0.011) and above direct prompting (gain +0.130 [0.006, 0.254], p = 0.041; 400 directions fixed vs 35 broken). With one pre-specified arm, no correction applies. The held-out r_part (0.105) is lower than the development value (0.268), so the original figure was optimistic, and the revised abstract reports both. On direction and error, the pleasantness-only rule beats S4 on both splits, confirming the reviewer's point: those metrics measure what pleasantness predicts, which is why the paper's target is r_part. The revised Section 4.4 reports the rule next to S4 and presents the mitigation study as a diagnostic: reasoning alone lowers fidelity (S1 0.002, S2 −0.018 vs S0 0.081), and human evidence is what raises it.
Following the reviewer's suggestion turned the mitigation result into a held-out replication of a fixed arm, evaluated on the pleasantness-controlled target beside a no-model baseline.
-
Questions
- Q1. Which ceiling, and corrected percentages? 0.595; best model 15.6%, median 6.3% (W1b).
- Q2. Does a 60+ vs under-60 contrast change fidelity? No: median r_part 0.047, reliability near zero, and 0 of 17 models above zero once raters and images are resampled (W1a, Table R1).
- Q3. Replicate the best arm on the held-out 600? Done: r_part 0.105 [0.014, 0.196], p = 0.011 (W2b, Table R4).
- Q4. Young humans rating "as a 60+ adult"? Not run. We explain why the represented group is the correct target, why a role-play baseline would likely share the stereotype (Lacey et al., 2006), and why item-level fidelity is attainable from human data (W1d).
In summary, the human target is measurable (split-half ceiling 0.62, 92% power at fidelity 0.20), every alternative age estimator gives near-zero fidelity, and model and human magnitudes differ by a factor of 1.4 in matched units. The ceiling is corrected to 0.595. The valence stereotype and the reversal persist when no description is requested, and a no-age "retired" persona reproduces them while a "cautious" persona does not. The best mitigation arm replicates on held-out images at r_part 0.105, which we now report alongside a pleasantness-only baseline.
We hope these clarifications and the additional analyses presented above address your concerns. We respectfully ask you to consider updating your score in light of this response.
References
- Aher, Arriaga, and Kalai. 2023. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. ICML 2023.
- Argyle, Busby, Fulda, Gubler, Rytting, and Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis 31(3):337-351.
- Lacey, Smith, and Ubel. 2006. Hope I die before I get old: Mispredicting happiness across the adult lifespan. Journal of Happiness Studies 7(2):167-182.
- Santurkar, Durmus, Ladhak, Lee, Liang, and Hashimoto. 2023. Whose Opinions Do Language Models Reflect? ICML 2023.
Response to Reviewer pZ9J (Part 1/3)
We thank Reviewer pZ9J for a close technical reading, and for recognizing that the framing asks the right question (same images, same direction as humans), that the paper controls the main confound (pleasantness) with ceilings, FDR, and bootstraps, and that the negative result is clean: a shared, reliable older persona that is orthogonal to human data and reverses it on one content class. The review questions whether the human target is measurable, whether a step-shaped model response should be compared with a linear human slope, which ceiling applies, whether role-play humans are the correct comparison, and whether the stereotype and mitigation results are confounded. We ran new analyses for each; the central result holds, and we corrected the ceiling.
Weaknesses
W1a / Q2. Is the human target measurable, and does a 60+ contrast change the result?
Response: We agree these are the two conditions the headline depends on, so we tested both directly.
Measurability. Splitting raters into two disjoint halves, estimating the pleasantness-adjusted age effect in each, and applying Spearman-Brown gives a split-half reliability of 0.39 [0.26, 0.50] and a ceiling of 0.62. This is an estimate independent of the model-based ceiling (0.595) and agrees with it. A power simulation planted model effects of known fidelity into the real OASIS design (same raters, ages, lists, noise levels, integer ratings) and ran our pipeline 300 times per setting: power is 41% at a true fidelity of 0.10, 92% at 0.20, and 100% at 0.30. A true fidelity of 0.30 would produce the best model's observed 0.093 in 1.7% of runs. The target is measurable, and the design detects a modest fidelity.
Step vs slope (Q2). We re-estimated the human effect ten ways and recomputed r_part for all 17 models.
Table R1. Fidelity under alternative estimators of the human age effect (valence).
The 60+ contrast gives median r_part 0.047, so it does not change the result. It also cannot serve as a target: with 25 older raters, the 60+ vs under-60 contrast has reliability near zero (ceiling 0.00 to 0.09). Its weak positive values do not survive resampling raters and images (0 of 17 models above zero; consensus 0.126 [−0.034, 0.209]). The 50+ contrast, with 78 raters and a usable ceiling of 0.52, gives 0.003. Two further facts justify the slope. First, the human data are graded, not a step: a pure step at 60 would give the linear slope an SD of 0.016 across images, and the observed SD is 0.106. Second, the model side is already a contrast: Eq. 2 weights the three personas by (−43, −1, 44)/1893, so ΔM is a scaled older-minus-young difference, and correlation is invariant to that scale.
The step-shaped model response is therefore compared with the most reliable human target available, and every alternative target, including the contrast the reviewer suggested, gives the same near-zero fidelity.
W1b / Q1. Which ceiling applies?
Response: Thank you for catching this inconsistency. The correct ceiling for r_part is 0.595 (pleasantness-adjusted effect); 0.86 applies to the raw age effect. The split-half estimate (0.62) confirms 0.595.
Corrected values: best model (GLM-4.5V) 15.6% of the attainable value (previously reported as 11%); median model 6.3% (previously 4%); held-out S4 17.6%.
Revised text (Abstract and Section 4.1):
The correction makes the paper internally consistent and, if anything, gives the models more credit; fidelity remains a small fraction of what the data allow.