acceptodds
Under review as a conference paper at ICLR 2027

Do LLMs Know How a Person Handles Everyday Self-Control Conflicts? A Person–Episode Test with Experience Sampling

Abstract

Self-control in daily life is a process: when people face a conflict, they select strategies that depend on who they are and on the situation, appraise how well they managed, and monitor and evaluate their strategy use. Large language models (LLMs) match human performance on several social-inference tests and are increasingly used to model and support people, yet their evaluations score outcomes, such as keyed answers, group-level rates, and questionnaire scores, leaving open whether a model reproduces the process by which a specific person handles a specific episode. We test 22 LLMs, prompted in English and German with increasingly detailed person descriptions, on 3,725 experience-sampled self-control conflicts from 468 participants, and score each predicted step (the person's strategies, success, and metacognition) against the person's own reports and reference predictors. At the selection step, the models list more strategies than people report, predict similar sets for different people, and recover the reported strategies less well than a rule that lists the same number of the most frequent strategies. At the appraisal step, person information improves their ranking of people by typical success and changes their tracking of a person's episodes only slightly. At the metacognitive step, their ratings follow reported success and strategy counts, ranking metacognitive knowledge weakly and missing regulation, which the conflict histories carry. The models thus capture what people generally do and how people differ, and miss how a person's situation shapes strategy choice, how strategies shape the appraisal of an episode, and how strategy use across episodes reflects metacognition. Because each step looks plausible on its own, evaluations of LLMs as models of individuals should test this process directly.

Primary area:
applications to neuroscience & cognitive science
Keywords:
large language models, human behavior simulation, behavioral fidelity, self-control, persona conditioning, metacognition, experience sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wpXhOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether LLM predictions match how particular people report handling particular self-control conflicts. It uses 3,725 experience-sampling episodes from 468 participants and evaluates 22 models with four levels of personal information in English and German. The tasks predict reported strategies, strategies conditional on reported success, perceived success given strategies, and metacognitive self-control scores from conflict histories. The main results show little improvement in strategy recovery as more personal information is supplied, modest improvements in success prediction, and weak Knowledge but near-zero Regulation rank associations. The paper also includes simple reference predictors, a small human comparison, sensitivity analyses, and a replication on a separate food-desire dataset.

Critical strengths
  1. Testing individual episodes is more informative than matching an average survey response. A model has to predict what this person reported doing in this situation. The separate strategy and success tasks show where extra personal information helps and where it does not. In particular, a better success estimate can coexist with almost no improvement in recovering the reported strategies.

  2. The frequency comparisons make the strategy result persuasive. In the main persistence analysis, median Task 1 Jaccard overlap with the richest persona is .142, versus .204 for the held-out top-three frequency rule. Frequency predictions matched to each model's output size also win across the 44 model-language conditions. The repeated-sampling, label-order, participant-level, and external-data checks reduce the likelihood that this is just an artifact of one run or one scoring choice. For Task 4, the supervised ridge result is worth noting: the histories contain some signal about Regulation, even though the LLM rank associations are near zero.

Critical weaknesses
  1. The process interpretation goes further than the prediction results support. The targets are self-reported strategies, perceived success, and questionnaire scores; the underlying decision process is not observed. Tasks 1-3 also use a fixed persona-prompting protocol without the person's earlier episode history. Failure under that protocol does not rule out other ways of modeling the person. For Task 4, adjusting for history summaries removes the Knowledge association, but does not establish how the LLM produced its ratings. The baselines and small human comparison help establish the prediction gap. I would keep that result and avoid treating it as a direct test of the underlying process.

  2. Task 3 gives the LLMs a coarser output scale than the target. Appendix B.4 reports that the target averages two success items and is a half-integer in 466 of the 1,130 scored persistence episodes. Every scored LLM prediction is an integer, which forces an error of at least .5 on those episodes. The paper acknowledges this, but the mean and ridge predictions are not described as having the same restriction, and no resolution-matched comparison is shown. This concern is about the success-error comparison with reference predictors, not the separate strategy results. I would like to know how much of that error gap remains on the same output grid.

Questions for authors
  1. Which conclusions apply specifically to the tested persona-prompting protocol, and which are meant as claims about the underlying self-control process? For Task 4, is there evidence beyond conditional association that the ratings rely on outcomes rather than recover metacognitive regulation? I am asking about the interpretation of the existing results, not requesting a new intervention study.

  2. Were the Task 3 mean and ridge predictions constrained to the same integer grid as the LLM outputs? If not, what are their errors after imposing that restriction? A comparison allowing half-point LLM predictions would also resolve the mismatch. Does the ordering of the reference predictors and the strongest LLM conditions remain the same?

2 comments

We thank the reviewer for the careful review and for recognizing the value of testing individual episodes and of the frequency comparisons and robustness checks.

W1 / Q1. Process interpretation, earlier episode history, and Task 4.

Process interpretation. We agree that the targets are reports and that the underlying decision process is not observed. What the tasks measure is whether the predictions reproduce each person's reported steps and the relations between them. This bears on the process only through a necessary condition: a model that reproduced how a person handles conflicts would reproduce the person's situation–behavior contingencies [1] to the extent that the inputs carry them. Falling below simple rules that recover these contingencies from the same inputs therefore counts against such reproduction under the tested conditions, whereas agreement would not establish it. The prediction gap is thus the primary result, and the process reading is limited to this inference.

Earlier episode history. To test whether the missing history explains the results, we added the person's earlier persistence reports (activity, goal, conflict strength, and strategies) to the P4 prompt on the 661 episodes with earlier reports (Task 1, English). Own − other is the Jaccard with the person's own report minus that with other participants' reports.

ModelWithout historyWith historyFrequency rule at the model's set sizeOwn − other (without → with)
Qwen 3.5 9B.160.203.229.024 → .066
Llama 3.1 8B.169.185.219.021 → .048
Qwen 3.5 27B.157.226.230.026 → .091
Qwen 3.5 35B-A3B.174.211.220.032 → .065

History improves recovery and agreement with the person's own report: with it, every model's own − other exceeds that of the frequency top-3 on the same episodes (.022; paired Wilcoxon, Holm-adjusted p ≤ .031), whereas without history none does (p ≥ .36). Separately, in the paper's Task 3 check with four closed models, adding the person's mean success over the other scored episodes lowers the pooled error from 1.31 to 1.19 (Appendix A.3). The absence of history therefore contributes to the zero-shot results. In the new Task 1 experiment, every model nevertheless remains below a rule that lists the person's most frequent earlier strategies and ignores the focal episode (.248–.265 at the model's set size), and below the population frequency rule.

Scope (Q1). Specific to persona prompts without history are the absolute levels of recovery and error, the exact set sizes, and the small advantage of the person's own report over other participants' reports. Three results hold across the conditions tested for each task: in Task 1, recovery below the frequency rule at the same set size, also with population statistics, history, or a request for exactly three strategies; in Task 3, weak tracking of which of a person's episodes went better in all 44 conditions, after recalibration, and with half-point answers (within-person r = .10 against .24 for the ridge; W2); and in Task 4, Regulation ratings that barely relate to Regulation and follow outcome summaries. The results leave open the latent process in people, how the LLMs compute their answers, and other ways of modeling the person, such as fine-tuning on experience-sampling records (l.473–474).

Task 4. We agree that the partial correlation describes the information in the ratings and leaves open how the models compute them. Beyond that association, the existing outputs give three results. First, the ratings rank Regulation at a median ρ = .003 (|ρ| < .10 in all 44 conditions), whereas a ridge regression on the same histories reaches .158 (.125 after controlling for the history summaries), and a supervised mapping from the 12 LLM item ratings to Regulation reaches at most ρ = .038 out of fold, so a linear reweighting of the ratings recovers little Regulation information. Second, the LLM Regulation ratings follow mean success and strategy count (.34/.52), which do not rank observed Regulation (−.007/−.002) and which the ridge's Regulation predictions do not follow (−.04/.07). Third, the LLM Regulation and Knowledge ratings correlate at .75, against .46 for the observed scores and .38 for the ridge. Reliance on outcomes thus remains an inference from associations, which these comparisons consistently support.

In the revision, we will present the tasks as tests of the reported steps and their relations and use "process" only in this sense, state the scope above, report the history experiment, and state that "follow" and "track" denote associations.


References

[1] Walter Mischel and Yuichi Shoda. A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102(2):246–268, 1995.

W2 / Q2. Task 3 output scale.

We thank the reviewer for identifying this issue. The mean and ridge predictions were continuous and were not restricted to the integer grid. We have now made the comparison in both ways the reviewer proposes. With the references rounded to the integer grid, all 44 original conditions, including the strongest, remain less accurate than the population mean and the ridge; allowing half-point answers in four open-weight models changes their errors only slightly and does not close their gaps to the references.

References on the integer grid. We rounded every reference prediction to an integer from 1 to 7 (values ending in .5 rounded up) and kept the original LLM predictions. Each row uses the same episodes for the reference and the models; the history references require at least one other or earlier episode.

Reference (episodes)Continuous MAEInteger MAELLM conditions: median (lowest)LLM conditions with lower MAE than the integer reference
Population mean (1,130)1.1381.1351.306 (1.165)0/44
Ridge (1,130)1.0841.0901.306 (1.165)0/44
Person's other-episode mean (1,024)1.0611.0711.294 (1.151)0/44
Person's earlier-report mean (779)1.1211.1341.313 (1.162)0/44

Rounding changes the reference errors by at most .013; the rounded population mean is the constant 5, which is also the best integer constant on the training folds. The gap between the median LLM condition and the population mean is .168 before and .171 after rounding (ridge: .222 and .216), and the strongest condition (1.165) remains above the rounded population mean and ridge by .030 and .075. Rounding values ending in .5 down leaves every count at 0/44.

Half-point LLM answers. For four open-weight models, we ran the original integer instruction and an otherwise identical instruction allowing answers in steps of 0.5 (P4, English, identical decoding). Answers that did not consist of a single rating were excluded, and each model is compared on the episodes with a valid rating in both conditions (1,079–1,130 for three models; 811 for Qwen 3.5 35B-A3B, which was served in FP8 quantization for these runs and returned more malformed answers). The models used half points in 44–100% of answers. Their MAE was 1.23–1.34 with half points and 1.26–1.34 with integers (change −.03 to +.01), against 1.14 for the population mean and 1.08 for the ridge on the same episodes (1.13–1.14 and 1.07–1.08 when these are rounded to the 0.5 grid). The finer scale thus removes at most .03 of gaps of .12–.20 to the population mean. The paper's parsing rule, applied to all 1,130 episodes, gives the same result (1.24–1.46 and 1.28–1.52). The panel's strongest conditions are closed models; for those, the comparison rests on the integer-valued references above.

Episodes with integer targets. On the 664 episodes whose target is an integer, integer answers carry no forced error. With the references again rounded, the ridge (1.100) and both history means (1.02) remain below all 44 conditions. Four closed-model conditions fall below the constant 5 (1.176), by at most .026 (six below the unrounded mean, 1.184). Integer targets include the scale endpoints and are more dispersed than half-integer targets (SD 1.44 vs. 1.28), which raises the error of a constant on this subset.

Within-person correlation. The appraisal conclusion (Section 4.3) rests on the within-person correlation, for which the same holds: rounded to integers, the ridge reaches .21 (.24 unrounded), above all 44 conditions (median .10, highest .17), and half-point answers leave the four models at .08–.11.

In the revision, we will state that the reference predictions are continuous and report these resolution-matched comparisons, including the half-point instructions and answer distributions.

Reviewer_ax7jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates whether LLMs can accurately predict how specific individuals handle everyday self-control conflicts. Using 3,725 experience-sampled episodes from 468 participants, the authors evaluate 22 LLMs across four zero-shot prediction tasks involving strategy selection, subjective success appraisal, and metacognition. They further examine the effects of increasingly detailed persona information and compare LLM predictions against several statistical and human baselines.

The results show that LLMs systematically overpredict strategy use, favor generic self-regulation strategies, and perform worse than simple frequency-based predictors in recovering reported strategies. Additional persona information improves between-person predictions of self-control success but provides little benefit for within-person predictions. LLMs also show weak associations with metacognitive knowledge and almost no ability to predict metacognitive regulation. The authors argue that conventional outcome-based evaluations may overlook important limitations in LLMs' ability to reproduce individual-level self-control processes.

Critical strengths
  1. Real-world behavioral data. The study uses experience-sampling data from everyday self-control conflicts rather than relying on hypothetical scenarios or standard psychological benchmarks. This provides an ecologically relevant setting for evaluating LLM predictions against actual human reports.

  2. Comprehensive empirical evaluation. The authors evaluate 22 LLMs across multiple model families, four levels of persona information, and two prompt languages. The additional analyses across conflict types and an independent dataset strengthen the robustness of the main observations.

  3. Informative baselines and diagnostic analyses. The comparisons with frequency-based predictors, supervised models, individual behavioral histories, and human evaluators provide useful reference points. In particular, the separation of between-person and within-person prediction helps characterize where additional persona information improves performance.

  4. Interesting and consistent empirical findings. The systematic overprediction of certain self-control strategies and the limited improvements from richer persona information reveal potentially meaningful differences between LLM-generated behavioral predictions and human reports. These observations could motivate further investigation into LLM behavioral priors and their calibration.

Critical weaknesses
  1. Unclear operationalization of process fidelity. The paper claims to move beyond outcome-based evaluation, yet the four tasks remain largely independent predictions of self-reported psychological variables. It is unclear how failures on these tasks establish limitations in modeling underlying cognitive processes. The authors should clarify what distinguishes their proposed process fidelity from conventional conditional prediction, and which aspects of the cognitive process are actually evaluated. Alternatively, the claims could be narrowed to the demonstrated limitations in behavioral prediction.

  2. The predictability of individual episodes is not established. Even a model that perfectly captures the true conditional distribution of strategy use may fail to predict a particular episode, especially given the limited persona and contextual information. Thus, low Jaccard/F1 does not necessarily indicate poor behavioral modeling. The authors could examine strategy consistency among participants or episodes with similar characteristics, and evaluate conditional behavioral distributions rather than relying primarily on single-episode overlap. This would help establish how much predictable information is actually available in the inputs.

  3. Baseline comparisons do not isolate the source of prediction failures. Frequency-based and supervised baselines learn from the target population's behavioral data, whereas LLMs make zero-shot predictions without access to this distribution. Their superior performance therefore does not necessarily demonstrate better behavioral modeling ability. A more informative comparison would provide LLMs with the same population-level frequency information or representative examples, and examine whether their prediction biases persist. This would help distinguish failures of distributional calibration from failures to use person- and context-specific information.

  4. The human evaluation is promising but insufficiently investigated. Human evaluators substantially outperform LLMs under matched information conditions, potentially offering stronger evidence of differences in behavioral prediction. However, the comparison includes only five episodes per task, and the source of the human advantage remains unclear. Expanding this evaluation and comparing predictions with and without persona/context information would help determine whether humans possess more accurate general behavioral priors or are genuinely better at person- and situation-specific inference.

Questions for authors
  1. What is the intended connection between the proposed tasks and social cognition? Section 2.2 motivates the study through theory-of-mind and social-inference research, whereas the actual tasks focus on predicting self-reported self-control strategies and psychological measures. Is the evaluation intended to assess social inference about individuals, or behavioral modeling of self-regulation more generally?

  2. How many strategies did human evaluators typically select in Task 1? Given the substantial overprediction of strategy-set size by LLMs, it would be helpful to know whether human evaluators selected sets closer to the participants' reported sizes. This would clarify how much of their advantage in Jaccard similarity may be associated with differences in predicted set size.

4 comments

We thank the reviewer for the detailed and constructive review.

W1. Operationalization of process fidelity.

We agree that the targets are self-reports, that each task is a conditional prediction, and that the submission left the operational meaning of reproducing the process implicit. The revision defines it, as the reviewer first suggests, and limits the conclusions to the process as observed in the reports. By the process we mean the reported steps of self-regulation (strategy selection, appraisal of success, and metacognition; l.038–044) and their relations to the situation and to one another for a specific person; process fidelity is the extent to which predictions reproduce these relations, beyond agreement with each reported variable. The evaluation differs from aggregate benchmarks of conditional prediction, which score each predicted variable against its target mostly across people (l.051–052), in two respects:

  • Person and episode. Each prediction is scored against the specific person's report in the specific episode (l.062), and within-person variation, which holds 82% of the variance in success (l.053–054), is evaluated separately from differences between people.
  • Relations between steps. Tasks 2–4 receive the reported values of the other steps, so each task scores one relation and a failure can be located at a step. This adds information to an error score: within a person, predicted success rises with the number of strategies at least as steeply as reported success (.159 vs. .111 per strategy; steeper in 37 of 44 conditions), yet follows which of the person's episodes went better only at r = .10 (ridge regression .24). The predictions thus apply the general relation between strategy count and success while following the person's episodes weakly.

The aspects evaluated are therefore three relations: how the situation shapes strategy choice, how strategies relate to felt success within a person, and how strategy use across conflicts relates to MISCS Knowledge and Regulation. For each, a reference predictor fit on other participants and using only fields that the prompt contains recovers more than every model condition: in matching the focal episode better than the same person's different-activity episodes (W2), in the within-person tracking above, and in ranking Regulation (ρ = .158 against .003).

These failures bear on the process because in people, too, the process is inferred from such observables: through a person's situation–behavior contingencies [1] and the experience-sampling relations among situation, strategy use, success, and metacognition [2, 3]. A model that reproduced a person's process would reproduce these relations to the extent that the inputs carry them. The tasks therefore test a necessary condition: agreement would not establish the process, whereas a systematic shortfall on relations that reference predictors recover from the same inputs is evidence against its reproduction. The revision will state this definition and use "process" in this reported sense throughout. It will also report the relation with strategy count, distinguish this general relation from the tracking of a person's episodes, and correct our statement that the relation would be attenuated in LLM-simulated data.


References

[1] Walter Mischel and Yuichi Shoda. A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102(2):246–268, 1995.

[2] Mario Wenzel, Sebastian Bürgler, Veronika Brandstätter, Antonia Kreibich, and Marie Hennecke. Self-regulatory strategy use, efficacy, and strategy-situation-fit in self-control conflicts of initiation, persistence, and inhibition. European Journal of Personality, 38(2):189–208, 2024.

[3] Sebastian Bürgler and Marie Hennecke. Metacognition and polyregulation in daily self-control conflicts. Scandinavian Journal of Psychology, 65(2):179–194, 2024.

W2. Predictability of individual episodes and conditional distributions.

We agree that a low Jaccard alone does not establish poor behavioral modeling, since a model with the correct conditional distribution may still miss a particular episode. The paper therefore interprets recovery relative to reference predictors that face the same unpredictability (Section 3.3), and the elicited outputs lack the properties of the two common ways of deriving predictions from a correct conditional distribution. Listing its m most probable strategies, such a model would in expectation include at least as many reported strategies as the population rule listing m; the models include fewer in all 44 conditions (l.311–313). Sampling sets from it would reproduce the reported label rates and set sizes; the models select self-talk in a median of 87.5% of episodes against 4.3% reported and list 5.6–17.0 strategies against 3.1 (Section 4.1), with nearly the same label rates at T = 0.7 (profile ρ = .985; Appendix A.4).

Following the reviewer's suggestions, we quantified the predictable information in the inputs and its use by the predictions:

  • Consistency of reports. Two reports overlap at .100 across people and .225 within a person (Appendix A.7). Across people, a shared activity raises the overlap only to .109 (.096 otherwise); within a person, it is .248 with a shared activity and .208 without (paired difference +.041, 95% CI [.005, .078]). Consistency thus lies mainly within the person, where it also varies with the situation.
  • Use by the predictions. A prediction matches its own episode better than the same person's other episodes by a median of +.002, and better than the person's different-activity episodes by +.009. An activity-conditional frequency rule reaches +.024 [.009, .041] and the paper's logistic ranker +.021 [.006, .036]; all 44 conditions fall below both, and at each model's own set size the rule exceeds the model in 41 of 44.
  • Conditional distributions. For groups of episodes and participants, we correlated each strategy's group-minus-overall rate between the reports and the outputs (median of 44 conditions; reliability of the reported pattern in parentheses): activity .14 (.44), goal .24 (.40), conflict strength .36 (.35), trait continuation .40 (.53), and MISCS Regulation −.05 (.50; negative in 28 of 44).

Single episodes are thus modestly predictable, the predictions use less of this information than simple rules do, and their recovery of conditional patterns varies and is weakest for Regulation, in line with the paper's conclusion (l.429–431). We will add these analyses and state that overlap is interpreted relative to the reference predictors.

W3. Providing the LLMs with population-level information.

We agree that the reference predictors draw on the target population's reports, whereas the LLMs predict zero-shot, so their higher recovery alone leaves the source of the gap open. If the gap reflected only missing population information, supplying it would close the gap, and any use of the person and the episode would appear as agreement with the person's own report.

  • Population information in the prompt (four open-weight models; Task 1, P4, 1,135 episodes). Given the training-fold rate of every strategy and the set-size distribution, the models apply it in part: their lists shorten from 7.2–10.5 to 5.7–9.7 strategies, and the over-selection of self-talk disappears in two models and shrinks in a third. Most biases persist: 68–84% of label directions are unchanged, and all four still over-select positive consequences and commitment and under-select changing feelings, changing the activity, and distraction. All four remain below the frequency rule at their own set size (.153–.175 vs. .193–.203). With ten randomly drawn training-fold reports of other participants as examples instead, they reach .139–.165 (rule .191–.203).
  • Post-hoc calibration (all 44 conditions; the three strategies most likely to be reported given the model's selection, estimated on the training folds). Median recovery rises from .142 to .203, the level of the frequency rule (.204), and the calibrated set equals the rule's set in a median of 76% of episodes, because the outputs barely discriminate reported from unreported strategies (per-strategy AUC: median .506, at most .521).
  • Agreement with the person's own report (Jaccard with the own report minus that with other participants' reports, which differ in person and episode). The models reach .009 originally, .010 after calibration, and .004–.017 with statistics in the prompt, against .006 for the frequency rule, which ignores the person; paired with the rule, the median excess is +.003 and +.004. The measure responds when person information is used: with the person's earlier reports in the prompt, it rises from .021–.032 to .048–.091 on 661 episodes and exceeds the rule (.022) for every model (Holm-adjusted p ≤ .031).

W3 (continued). The results thus separate the two explanations: a distributional mismatch, whose cost in recovery post-hoc calibration removes and information in the prompt reduces in part, and a limited use of person- and episode-specific information, which remains in either case. For the supervised references in Tasks 3 and 4, the conclusions rest on within-person and rank correlations, which recalibration leaves nearly unchanged (Task 3 within-person r .097 vs. .093; ridge .243). We will add these results and state that the frequency rule uses other participants' report rates, and that the four models given these rates remain below it.

W4. Size of the human evaluation and source of the human advantage.

We agree that five episodes per task are too few to generalize the evaluators' advantage. We could not extend the human evaluation within the discussion period, so the revision will present the comparison as a check, on these episodes, that the P1 prompt carries recoverable information, and the conclusion about the models' selection step will rest on the reference predictors evaluated on all 1,135 episodes.

On the source of the advantage, the existing responses allow one test. A prediction drawn from general priors that ignore the displayed case would fit the displayed episode no better than the other episodes, so we scored each predicted set against its own episode's report and the other four episodes' reports, with the same five episodes and P1 input for evaluators and models: the evaluators reach .492 vs. .114 (own − other +.378), and the LLMs −.014 (own above other for 6 of 22 models). The evaluators' sets are thus specific to the displayed case. Because the episodes differ in both the person and the activity, this test leaves open whether that specificity draws on the person or on the situation; the with/without-persona comparison that the reviewer proposes would separate the contribution of the person information from that of the situation and of general priors, and we will name it as the next extension. In Task 3, the evaluators' mean predictions follow the order of the reported success across the five episodes (Spearman ρ = .87; Table 5).

Q1. Connection to social cognition.

The evaluation assesses inference about specific individuals' self-regulation; it is behavioral modeling of self-regulation at the level of the person and the episode, and theory of mind as a capacity lies outside its scope. We agree that Section 2.2 leaves the connection implicit. The social-cognitive tests cited there ask what a described character believes, feels, or should do and score the answer against a key, as in false-belief vignettes [1] or the most effective way to manage an emotion [2]. The tasks ask the corresponding questions about a real person and score them against that person's own report. In a keyed item, the answer is determined by facts stipulated in the vignette; a report is not determined by the description of the person and varies across that person's episodes. High scores on keyed tests therefore leave open how well a model predicts a real person's reports across everyday episodes, which the tasks test. The results locate the limit there: the models capture what people generally do and how people differ, and miss how a specific person handled a specific episode. The revision will state this connection, and that the conclusions concern the person's reported steps and their relations (W1).

Q2. Set sizes of the human evaluators and their role in the Jaccard advantage.

The evaluators selected 4.4 strategies per episode on average, against 3.6 reported and 8.3 selected by the LLMs on the same episodes. Set-size differences alone cannot explain their advantage (.492 vs. .149). With the frequency ranking held fixed, the frequency rule reaches .110 at the models' set sizes and .115 at the evaluators', a difference of .005. The evaluators' sets contain 64% of the reported strategies and the models' sets 27%; since removing labels cannot add reported strategies, even a model set reduced to exactly its correct labels would reach .271 on average. Asked for exactly three strategies on all 1,135 episodes, the four models of W3 reach .132–.151, compared with .145–.158 under the original prompt and .204 for the frequency top-3. The advantage therefore lies mainly in which strategies are selected; the revision will report these comparisons.


References

[1] James W. A. Strachan et al. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7):1285–1295, 2024.

[2] Katja Schlegel, Nils R. Sommer, and Marcello Mortillaro. Large language models are proficient in solving and creating emotional intelligence tests. Communications Psychology, 3:80, 2025.

Thanks to the authors for their hard work on the rebuttal and clarifications. Those supplementary results largely answered my questions and addressed my empirical concerns. Therefore, I am happy to improve my score to 3.

Reviewer_zLQqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether LLMs can predict how specific individuals handle everyday self-control conflicts. Using experience-sampling data containing 3,725 episodes from 468 participants, the authors evaluate 22 LLMs in English and German on strategy selection, subjective success appraisal, and metacognitive inference. Models overpredict strategy use and recover reported strategies less accurately than simple frequency baselines. Richer persona information improves between-person success prediction but contributes little to within-person tracking, while models weakly recover metacognitive Knowledge and largely miss Regulation. Validation includes reference predictors, prompt robustness checks, a second dataset for strategy selection, and a small human comparison. The main contribution is an evaluation of person–episode fidelity demonstrating that aggregate agreement can conceal substantial limitations in predicting an individual’s reported behavior across situations.

Critical strengths
  • The study uses experience-sampling reports from real participants to evaluate strategy selection, subjective success, and metacognitive inference, making the tasks relevant to individualized behavioral modeling.

  • Separating between-person and within-person accuracy reveals that richer persona information can improve predictions of typical success while contributing little to predictions of individual episodes.

  • Frequency baselines matched to output size, cross-validated reference predictors, and checks across models, languages, and prompt settings strengthen confidence in the main findings.

Critical weaknesses
  • The experiments primarily measure agreement with self-reported behavior and questionnaire scores. Interpreting prediction failures as evidence that models cannot reproduce the underlying psychological processes requires stronger support.

  • The fitted baselines learn from labeled data from the target population, while the LLMs lack comparable calibration information. Some performance gaps may therefore reflect differences in familiarity with the population’s strategy and outcome distributions.

  • The human comparison includes only five unique episodes per task. This small sample makes it difficult to generalize the observed human advantage to the full dataset.

Questions for authors
  • In Task 4, how do you distinguish an outcome-based shortcut from legitimate use of success and strategy counts as evidence about metacognitive Knowledge?

  • Were any LLM conditions calibrated using examples or population statistics from the same training folds as the baselines? If so, did the main performance gaps persist?

  • How were the five episodes per task selected within the success strata, and how consistent was the human advantage across these episodes?

3 comments

We thank the reviewer for the careful reading and the precise questions.

W1. Whether prediction failures support conclusions about the psychological process.

We agree with the premise: the targets are self-reports and questionnaire scores, so the tasks observe neither a person's latent process nor the computation behind a model's answer. The paper notes that the design "locates the failures at the reported steps and leaves their sources open" (l.471), and the revision will state this explicitly. The conclusion concerns the process as observed in a person's reports, and three arguments support drawing it from the prediction failures.

In people, the process is inferred from the same observables. Research on daily self-control has no direct access to the process either: the relations between situation and strategy use, between strategy use and success, and between strategy use and metacognition were established from experience-sampling reports and the MISCS questionnaire [1–4], and personality theory identifies a person's processing dynamics through that person's stable situation–behavior contingencies [5]. The tasks ask whether the predictions show these contingencies for a specific person.

The tasks test a necessary condition. A model that reproduced a person's process would reproduce its observable consequences as far as the inputs carry them: strategy choices that change with the situation as the person's reports do, appraisals that follow which of the person's episodes went better, and Regulation ratings that follow reported Regulation (the implications stated at l.263–265 and l.403–405). Agreement with behavior is the criterion on which LLMs are proposed as models of human cognition [6]; agreement is compatible with several processes, whereas a systematic shortfall relative to predictors that use the same inputs counts against reproducing this one, so failure is the more informative direction of that inference.

The inputs carry these relations. Scored against the same reports, simple predictors fit on other participants, with inputs that the prompts also contain, recover each relation more fully than all 44 model–language conditions. Across a person's episodes with different activities, an activity-specific frequency rule overlaps more with the focal report than with the person's other reports (Jaccard difference .024; LLMs .009). A ridge regression tracks which of a person's episodes went better (within-person r = .24; LLMs .10) and ranks Regulation (ρ = .158; LLMs .003). All predictors are scored against the same reports, so the references show that these relations are recoverable despite reporting noise, and calibration on the same training folds leaves the within-person and Regulation contrasts nearly unchanged (W2). With the person's earlier reports in the prompt, agreement with the person's own report rises in all four models tested (W2), so the measures detect such agreement when the predictions contain it. The revision will state that the tasks test this necessary condition for the process as observed in the reports and rephrase sentences that could be read as claims about a latent mechanism.


References

[1] Marina Milyavskaya, Blair Saunders, and Michael Inzlicht. Self-control in daily life: Prevalence and effectiveness of diverse self-control strategies. Journal of Personality, 89(4):634–651, 2021.

[2] Mario Wenzel, Sebastian Bürgler, Veronika Brandstätter, Antonia Kreibich, and Marie Hennecke. Self-regulatory strategy use, efficacy, and strategy-situation-fit in self-control conflicts of initiation, persistence, and inhibition. European Journal of Personality, 38(2):189–208, 2024.

[3] Sebastian Bürgler and Marie Hennecke. Metacognition and polyregulation in daily self-control conflicts. Scandinavian Journal of Psychology, 65(2):179–194, 2024.

[4] Sebastian Bürgler, Kristian Kleinke, and Marie Hennecke. The Metacognition in Self-Control Scale (MISCS). Personality and Individual Differences, 199:111841, 2022.

[5] Walter Mischel and Yuichi Shoda. A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102(2):246–268, 1995.

[6] Marcel Binz et al. A foundation model to predict and capture human cognition. Nature, 644(8078):1002–1009, 2025.

W2 / Q2. Were any LLM conditions calibrated with information from the same training folds, and do the gaps persist?

No condition in the submission was calibrated: the LLM prompts contained the person description and the episode, and no population information. We agree that the fitted predictors hold this advantage. Following the reviewer's suggestion, we provided the same information to the LLMs, using only the training folds of the reference predictors. With population statistics or examples in the prompt (four models), the Task 1 gap to the frequency rule persists. Post-hoc calibration (all 44 conditions) removes the part of the gaps that reflects a mismatch with the population's distributions, as the reviewer anticipated: Task 1 recovery reaches the level of the frequency rule, and the Task 3 error falls below the population mean. The gaps that concern the person and the episode persist: no calibrated condition exceeds the frequency rule, which ignores the person, by more than .010; agreement with the person's own report stays small; and the tracking of a person's episodes is nearly unchanged.

Task 1: population statistics in the prompt. For each test fold, we computed from training-fold participants the report rate of every strategy and the distribution and mean of set sizes (about three), and added them to the P4 prompt (the richest person description). We ran four of the panel's twelve open-weight models (English, all 1,135 persistence episodes). Values are participant-balanced Jaccard; with the original prompt, the re-runs are within .004 of the submitted scores.

ModelOriginal prompt (re-run)+ population statisticsFrequency rule at the model's set size
Qwen 3.5 9B.146.168.203
Llama 3.1 8B.154.153.196
Qwen 3.5 27B.145.175.202
Qwen 3.5 35B-A3B.158.169.193

The models listed fewer strategies and three recovered more, yet all four remained below the frequency rule. Although the prompt stated every strategy's report rate, all four continued to over-select positive consequences and commitment and to under-select changing feelings, changing the activity, and distraction. With ten training-fold reports of other participants as examples instead, they reach .139–.165, below the rule (.191–.203). The revision will state that the frequency rule uses other participants' report rates and no information about the person, and that the gap persists when the models receive these rates.

Task 1: post-hoc calibration. For each strategy, we estimated on the training folds the probability that it was reported given that the model did or did not select it, and kept the three most probable. The calibrated sets reach .203 (.194–.214 across conditions), against .204 for the frequency top-3. They equal the frequency top-3 in a median of 76% of episodes, reflecting the outputs' weak discrimination between reported and unreported strategies (median per-strategy AUC .506; chance .5).

Task 1: agreement with the person's own report. We scored each prediction against the person's own report and against other participants' reports (own − other: Jaccard with the own report minus the mean Jaccard with others'); the contrast reflects both the person and the episode. The frequency top-3, which ignores the person, gives .006 and serves as the reference. The LLM outputs give .009 originally and .010 after calibration (medians over 44 conditions), and .004–.017 with statistics in the prompt. Paired with the rule over participants, the median difference is +.003 for the original outputs (no 95% interval excludes zero) and +.004 for the calibrated sets (unadjusted interval above zero in 14 of 44 conditions; largest estimate .019). The contrast is sensitive to information about the person: with the person's earlier reports in the prompt (661 eligible episodes; the same four models), it rises from .021–.032 to .048–.091, above the rule (.022) for every model (Holm-adjusted p ≤ .031).

Task 3. Fold-wise linear recalibration lowers the median error from 1.306 to 1.118, below the population mean (1.138) in 43 of 44 conditions; for this comparison, the reviewer's explanation holds. The error remains above the ridge regression (1.084) in all 44, and the within-person correlation remains low and nearly unchanged after fold-wise recalibration (.097 before, .093 after; ridge .243). This agrees with the paper's reading that person information enters the appraisal as a stable success level (l.396–398). The revision will specify that the paper's Task 3 comparison concerns the original outputs.

Task 4. The comparison uses rank correlations, which a strictly increasing recalibration preserves. A ridge mapping from the 12 LLM item ratings to Regulation, fit on training folds, reaches at most ρ = .038 in any condition (Q1). We will add all of these analyses to the paper.

W3 / Q3. Selection of the human episodes and consistency of the human advantage.

We agree that five episodes per task do not support generalizing the human advantage to the full dataset. The paper's conclusions rest on reference predictors evaluated on the full data, which show at scale what the comparison indicates on five episodes: the information in the prompt permits better recovery than the models achieve. The frequency rule, which ignores the person and the episode, recovers more than every LLM condition on all 1,135 episodes (.204 vs. a median of .142). Rules that read only fields of the P1 prompt also recover information about the specific episode: a frequency rule conditioned on the activity matches the focal episode better than the same person's episodes with a different activity by +.024 [.009, .041], compared with a median of +.009 for the LLMs (all 44 conditions below it). The revision will state this scope and base the conclusion about the selection step on the full-data predictors.

Selection. The ten episodes were selected by a deterministic procedure that reads only the participants' data; no LLM output or evaluator response entered it. An episode was eligible if it had a success score, at least one reported strategy, a recorded activity, and at least three scored persistence episodes from the same participant. Episodes were drawn one at a time within each success stratum (low 1.0–3.0: three episodes; middle 3.5–5.0: four; high 5.5–7.0: three), with at most one episode per participant. Each draw took the candidate whose activity, then goal domain, then set-size band was least represented among the episodes already drawn, then the candidate with the fewest missing persona items; remaining ties were broken by a seeded hash. Before any response was collected, the ten episodes were divided between the tasks in a fixed allocation, which gives Task 1 two low-, two middle-, and one high-success episode and Task 3 one, two, and two.

Consistency. In Task 1, the evaluators' mean Jaccard ranges from .421 to .605 across the five episodes and exceeds the LLM mean in each episode and every individual model in four. The margin is largest for the two episodes reporting only a change of activity, which one and two of the 22 models selected and which the models under-select on the full data (32 of 44 conditions; Section 4.1). In Task 3, the evaluators' error over the five episodes is lower than the LLMs' (1.500 vs. 1.891), and their mean predictions follow the order of the reported success across the five episodes (Spearman ρ = .87; Table 5).

Q1. Task 4: outcome-based shortcut versus legitimate use of success and strategy counts.

We distinguish the two readings through Regulation. For Knowledge itself, we agree that success and strategy counts are legitimate evidence, and the paper claims only that these summaries account for the Knowledge association (l.423): with them and repertoire size controlled, the association disappears for the LLMs and for a ridge regression trained on the same histories alike (partial ρ = −.02 and .04). In these data, the same summaries have essentially no rank association with observed Regulation, so ratings that used them as evidence about each facet need not follow them for Regulation, and the ridge predictions do not.

Spearman ρ with mean success / strategies per episodeKnowledgeRegulation
Observed MISCS score (Table 15).23 / .19−.007 / −.002
LLM ratings (median of 44 conditions).45 / .46.34 / .52
Ridge predictions.54 / .67−.04 / .07

The LLM ratings of both facets follow the summaries to a similar degree, and the predicted Knowledge and Regulation scores correlate at .75 (median), against .46 for the observed scores. The histories nevertheless contain Regulation information outside these summaries: the ridge ranks Regulation at ρ = .158 (.125 with the two summaries and repertoire size controlled) while its predictions do not follow them, whereas the LLM ratings reach .003 (maximum .096). A supervised ridge mapping from the 12 LLM item ratings to observed Regulation recovers at most ρ = .038 out of fold.

The ratings thus follow the outcome summaries irrespective of the facet rated, which is the sense in which the paper describes them as following outcomes; validation by the Knowledge correlation alone (ρ = .178) would not detect this. The revision will state that these summaries are valid evidence about Knowledge, give this criterion, and report these results.

Comments on the paper

1 comment

We thank the reviewers for their careful reviews and constructive suggestions.

We are also grateful that the reviewers recognized the strengths of the paper:

  • zLQq: real participants' experience-sampling reports make the tasks relevant to individualized behavioral modeling; separating between- and within-person accuracy shows where persona information helps.
  • ax7j: an ecologically relevant setting; 22 LLMs, four persona levels, and two languages; informative baselines.
  • wpXh: testing individual episodes is more informative than matching an average survey response; the frequency comparisons make the strategy result persuasive.

The reviewers' comments and questions led us to the following revisions. We have uploaded the revised PDF, in which the main additions are marked in blue.

1. Defining process fidelity (zLQq W1; ax7j W1; wpXh W1, Q1). Process fidelity means reproducing how reported strategies vary with the situation, how they relate to felt success, and how their use across conflicts relates to metacognition scores. We test this as a necessary condition against reference predictors with the same inputs; "follow" and "track" denote associations (Sec. 3.2). The tasks observe neither a person's latent process nor how a model computes its answers (Limitations).

2. Population information and calibration (zLQq W2, Q2; ax7j W3).

  1. Given training-fold strategy rates, four models score below the frequency rule at their own set size (Jaccard .153–.175 vs. .193–.203).
  2. Post-hoc calibration of all 44 conditions raises Task 1 recovery to the top-3 rule's level (median .203 vs. .204); no condition exceeds it by more than .010. Recalibrated Task 3 error is below the population mean's in 43 of 44 conditions and above the ridge regression's in all 44; within-person r is nearly unchanged (.097 to .093; ridge .243) (Secs. 3.3, 4.1, 4.3; App. A.8, Tables 16–17).

3. Predictability and conditional distributions (ax7j W2). Reports agree more within than between people. An activity-conditioned frequency rule matches the focal report better than the same person's reports from other activities by .024; every condition is lower (median .009). Recovery of strategy patterns conditioned on Regulation is weakest (Sec. 4.1; App. A.8, Table 19).

4. Human comparison (zLQq W3, Q3; ax7j W4, Q2). The human comparison uses five episodes per task; the strategy-selection conclusion rests on all 1,135 episodes. We describe the episode selection (App. A.2) and report that 1) in Task 1, the evaluators' mean Jaccard exceeds the LLM mean in each episode; 2) they chose 4.4 strategies per episode (LLMs 8.3; reported 3.6), and the frequency rule's Jaccard differs by .005 between the human and LLM set sizes; 3) their sets match the displayed episode's report better than the other four (.492 vs. .114) (Sec. 4.1; App. A.2; Table 5). The next extension is a human comparison with and without the persona (Limitations).

5. Earlier reports and scope of the persona prompts (wpXh W1, Q1). With the person's earlier reports in the prompt (661 episodes, four models), own − other (Jaccard with the person's report minus mean Jaccard with others' reports) rises from .021–.032 to .048–.091 (rule .022). Recovery rises but stays below the frequency rule, as in every prompt variant; recovery levels and set sizes vary with the prompt (Sec. 4.1; Table 18).

6. Task 3 output resolution (wpXh W2, Q2). The reference predictions were continuous; rounded to integers, they still have lower error than the original LLM outputs in all 44 conditions. Allowing half-point answers changes the error of four models by −.03 to +.01 (Sec. 4.3; Tables 20–22; App. B.4).

7. Task 4: Knowledge vs. Regulation (zLQq Q1; wpXh W1, Q1). Success and strategy counts are valid evidence about Knowledge but have near-zero rank correlations with observed Regulation. The LLM Regulation ratings follow both summaries; those of the ridge regression do not (ρ = −.04 and .07). A ridge regression from the LLMs' 12 item ratings to Regulation reaches at most ρ = .038 out of fold (Sec. 4.4; Table 23).

8. Connection to social cognition (ax7j Q1). In keyed vignette tests, the scenario fixes the correct answer, whereas a person's reports vary across episodes; our tasks assess inference about a specific individual's self-regulation (Sec. 2.2).

Locations in the revised PDF. Line numbers cited earlier have shifted. Tables 1–15 and all figures keep their numbers. New Appendix A.8 contains Tables 16–23: Task 1 with population statistics (16), calibration (17), and earlier reports (18); situation dependence (19); Task 3 resolution (20–22); Task 4 outcome summaries (23). Former Tables 16–66 are now Tables 24–74.

We thank the reviewers again for their time and thoughtful feedback. We would be glad to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.