- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tests whether LLM predictions match how particular people report handling particular self-control conflicts. It uses 3,725 experience-sampling episodes from 468 participants and evaluates 22 models with four levels of personal information in English and German. The tasks predict reported strategies, strategies conditional on reported success, perceived success given strategies, and metacognitive self-control scores from conflict histories. The main results show little improvement in strategy recovery as more personal information is supplied, modest improvements in success prediction, and weak Knowledge but near-zero Regulation rank associations. The paper also includes simple reference predictors, a small human comparison, sensitivity analyses, and a replication on a separate food-desire dataset.
Critical strengths
-
Testing individual episodes is more informative than matching an average survey response. A model has to predict what this person reported doing in this situation. The separate strategy and success tasks show where extra personal information helps and where it does not. In particular, a better success estimate can coexist with almost no improvement in recovering the reported strategies.
-
The frequency comparisons make the strategy result persuasive. In the main persistence analysis, median Task 1 Jaccard overlap with the richest persona is .142, versus .204 for the held-out top-three frequency rule. Frequency predictions matched to each model's output size also win across the 44 model-language conditions. The repeated-sampling, label-order, participant-level, and external-data checks reduce the likelihood that this is just an artifact of one run or one scoring choice. For Task 4, the supervised ridge result is worth noting: the histories contain some signal about Regulation, even though the LLM rank associations are near zero.
Critical weaknesses
-
The process interpretation goes further than the prediction results support. The targets are self-reported strategies, perceived success, and questionnaire scores; the underlying decision process is not observed. Tasks 1-3 also use a fixed persona-prompting protocol without the person's earlier episode history. Failure under that protocol does not rule out other ways of modeling the person. For Task 4, adjusting for history summaries removes the Knowledge association, but does not establish how the LLM produced its ratings. The baselines and small human comparison help establish the prediction gap. I would keep that result and avoid treating it as a direct test of the underlying process.
-
Task 3 gives the LLMs a coarser output scale than the target. Appendix B.4 reports that the target averages two success items and is a half-integer in 466 of the 1,130 scored persistence episodes. Every scored LLM prediction is an integer, which forces an error of at least .5 on those episodes. The paper acknowledges this, but the mean and ridge predictions are not described as having the same restriction, and no resolution-matched comparison is shown. This concern is about the success-error comparison with reference predictors, not the separate strategy results. I would like to know how much of that error gap remains on the same output grid.
Questions for authors
-
Which conclusions apply specifically to the tested persona-prompting protocol, and which are meant as claims about the underlying self-control process? For Task 4, is there evidence beyond conditional association that the ratings rely on outcomes rather than recover metacognitive regulation? I am asking about the interpretation of the existing results, not requesting a new intervention study.
-
Were the Task 3 mean and ridge predictions constrained to the same integer grid as the LLM outputs? If not, what are their errors after imposing that restriction? A comparison allowing half-point LLM predictions would also resolve the mismatch. Does the ordering of the reference predictors and the strongest LLM conditions remain the same?
2 comments
W2 / Q2. Task 3 output scale.
We thank the reviewer for identifying this issue. The mean and ridge predictions were continuous and were not restricted to the integer grid. We have now made the comparison in both ways the reviewer proposes. With the references rounded to the integer grid, all 44 original conditions, including the strongest, remain less accurate than the population mean and the ridge; allowing half-point answers in four open-weight models changes their errors only slightly and does not close their gaps to the references.
References on the integer grid. We rounded every reference prediction to an integer from 1 to 7 (values ending in .5 rounded up) and kept the original LLM predictions. Each row uses the same episodes for the reference and the models; the history references require at least one other or earlier episode.
| Reference (episodes) | Continuous MAE | Integer MAE | LLM conditions: median (lowest) | LLM conditions with lower MAE than the integer reference |
|---|---|---|---|---|
| Population mean (1,130) | 1.138 | 1.135 | 1.306 (1.165) | 0/44 |
| Ridge (1,130) | 1.084 | 1.090 | 1.306 (1.165) | 0/44 |
| Person's other-episode mean (1,024) | 1.061 | 1.071 | 1.294 (1.151) | 0/44 |
| Person's earlier-report mean (779) | 1.121 | 1.134 | 1.313 (1.162) | 0/44 |
Rounding changes the reference errors by at most .013; the rounded population mean is the constant 5, which is also the best integer constant on the training folds. The gap between the median LLM condition and the population mean is .168 before and .171 after rounding (ridge: .222 and .216), and the strongest condition (1.165) remains above the rounded population mean and ridge by .030 and .075. Rounding values ending in .5 down leaves every count at 0/44.
Half-point LLM answers. For four open-weight models, we ran the original integer instruction and an otherwise identical instruction allowing answers in steps of 0.5 (P4, English, identical decoding). Answers that did not consist of a single rating were excluded, and each model is compared on the episodes with a valid rating in both conditions (1,079–1,130 for three models; 811 for Qwen 3.5 35B-A3B, which was served in FP8 quantization for these runs and returned more malformed answers). The models used half points in 44–100% of answers. Their MAE was 1.23–1.34 with half points and 1.26–1.34 with integers (change −.03 to +.01), against 1.14 for the population mean and 1.08 for the ridge on the same episodes (1.13–1.14 and 1.07–1.08 when these are rounded to the 0.5 grid). The finer scale thus removes at most .03 of gaps of .12–.20 to the population mean. The paper's parsing rule, applied to all 1,130 episodes, gives the same result (1.24–1.46 and 1.28–1.52). The panel's strongest conditions are closed models; for those, the comparison rests on the integer-valued references above.
Episodes with integer targets. On the 664 episodes whose target is an integer, integer answers carry no forced error. With the references again rounded, the ridge (1.100) and both history means (1.02) remain below all 44 conditions. Four closed-model conditions fall below the constant 5 (1.176), by at most .026 (six below the unrounded mean, 1.184). Integer targets include the scale endpoints and are more dispersed than half-integer targets (SD 1.44 vs. 1.28), which raises the error of a constant on this subset.
Within-person correlation. The appraisal conclusion (Section 4.3) rests on the within-person correlation, for which the same holds: rounded to integers, the ridge reaches .21 (.24 unrounded), above all 44 conditions (median .10, highest .17), and half-point answers leave the four models at .08–.11.
In the revision, we will state that the reference predictions are continuous and report these resolution-matched comparisons, including the half-point instructions and answer distributions.
We thank the reviewer for the careful review and for recognizing the value of testing individual episodes and of the frequency comparisons and robustness checks.
W1 / Q1. Process interpretation, earlier episode history, and Task 4.
Process interpretation. We agree that the targets are reports and that the underlying decision process is not observed. What the tasks measure is whether the predictions reproduce each person's reported steps and the relations between them. This bears on the process only through a necessary condition: a model that reproduced how a person handles conflicts would reproduce the person's situation–behavior contingencies [1] to the extent that the inputs carry them. Falling below simple rules that recover these contingencies from the same inputs therefore counts against such reproduction under the tested conditions, whereas agreement would not establish it. The prediction gap is thus the primary result, and the process reading is limited to this inference.
Earlier episode history. To test whether the missing history explains the results, we added the person's earlier persistence reports (activity, goal, conflict strength, and strategies) to the P4 prompt on the 661 episodes with earlier reports (Task 1, English). Own − other is the Jaccard with the person's own report minus that with other participants' reports.
History improves recovery and agreement with the person's own report: with it, every model's own − other exceeds that of the frequency top-3 on the same episodes (.022; paired Wilcoxon, Holm-adjusted p ≤ .031), whereas without history none does (p ≥ .36). Separately, in the paper's Task 3 check with four closed models, adding the person's mean success over the other scored episodes lowers the pooled error from 1.31 to 1.19 (Appendix A.3). The absence of history therefore contributes to the zero-shot results. In the new Task 1 experiment, every model nevertheless remains below a rule that lists the person's most frequent earlier strategies and ignores the focal episode (.248–.265 at the model's set size), and below the population frequency rule.
Scope (Q1). Specific to persona prompts without history are the absolute levels of recovery and error, the exact set sizes, and the small advantage of the person's own report over other participants' reports. Three results hold across the conditions tested for each task: in Task 1, recovery below the frequency rule at the same set size, also with population statistics, history, or a request for exactly three strategies; in Task 3, weak tracking of which of a person's episodes went better in all 44 conditions, after recalibration, and with half-point answers (within-person r = .10 against .24 for the ridge; W2); and in Task 4, Regulation ratings that barely relate to Regulation and follow outcome summaries. The results leave open the latent process in people, how the LLMs compute their answers, and other ways of modeling the person, such as fine-tuning on experience-sampling records (l.473–474).
Task 4. We agree that the partial correlation describes the information in the ratings and leaves open how the models compute them. Beyond that association, the existing outputs give three results. First, the ratings rank Regulation at a median ρ = .003 (|ρ| < .10 in all 44 conditions), whereas a ridge regression on the same histories reaches .158 (.125 after controlling for the history summaries), and a supervised mapping from the 12 LLM item ratings to Regulation reaches at most ρ = .038 out of fold, so a linear reweighting of the ratings recovers little Regulation information. Second, the LLM Regulation ratings follow mean success and strategy count (.34/.52), which do not rank observed Regulation (−.007/−.002) and which the ridge's Regulation predictions do not follow (−.04/.07). Third, the LLM Regulation and Knowledge ratings correlate at .75, against .46 for the observed scores and .38 for the ridge. Reliance on outcomes thus remains an inference from associations, which these comparisons consistently support.
In the revision, we will present the tasks as tests of the reported steps and their relations and use "process" only in this sense, state the scope above, report the history experiment, and state that "follow" and "track" denote associations.
References
[1] Walter Mischel and Yuichi Shoda. A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102(2):246–268, 1995.