- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces MindGym, a suite of 25 interactive psychological tasks, and mini-psych-agent, a framework for comparing LLM agents with different memory configurations. Unlike evaluations based on replaying human histories, agents act autonomously and experience the consequences of their own decisions. Task-specific RNNs trained on human data score the resulting trajectories. The experiments suggest that providing memory generally improves task performance and agreement with the learned human policies, although the effects vary across tasks and models.
Critical strengths
The emphasis on closed-loop evaluation is well motivated. Allowing agents to generate their own histories captures behavioral differences that teacher-forced replay can miss. The collection of interactive tasks provides a useful platform for studying these effects across different memory configurations.
Critical weaknesses
-
The main metric does not reliably establish human-likeness. Average log-likelihood under a human-policy model can favor an agent that repeatedly chooses the most probable action over one that reproduces human variability. The paper’s N-back example makes this concern concrete: an almost constant “non-match” policy receives a higher score than held-out humans despite chance-level balanced accuracy. The authors acknowledge this limitation, but it still weakens the interpretation of improved scores as improved human-likeness. More systematic trajectory-level validation is needed to support that interpretation.
-
The evaluator is not sufficiently validated on the histories it is used to score. The motivation for the RNN evaluator is to assess agent-generated histories that may be rare or absent in the human dataset. However, its predictive validation primarily uses held-out human trajectories. This does not establish that its predictions remain reliable under the distribution shift induced by autonomous agents. Consequently, differences between memory configurations could partly reflect evaluator extrapolation errors rather than differences in human behavioral compatibility.
Questions for authors
- How do simple policies, such as always selecting one action or greedily choosing the RNN’s most probable action, rank against humans and LLM agents?
- Do trajectory-level measures support the same conclusions about memory?
- Could the authors validate the RNN on human responses collected after exposure to agent-generated histories, particularly those far from the training distribution?
4 comments
Response to Reviewer HABF (2/3)
W2. The evaluator is validated mainly on held-out human trajectories, not on the agent-generated histories it scores, so differences between memory configurations could partly reflect extrapolation errors.
This is a fair concern. It cannot be fully resolved without new human data, and the limitations now say so. In the new Appendix L, we added:
(1) the RNNs reproduce behavioral signatures of held-out participants in 16 tasks, and their autonomous rollouts match held-out humans on 14 trajectory features (Fig. 9, Table 14);
(2) scoring every policy on the same histories (the human-policy RNN first plays the task on its own, and this sequence is shown to every agent; no agent-generated history is involved) gives the same advantage of memory over Stateless (Table 15; Table R3 below);
(3) simple-policy checks in closed loop (Table 12). We did not compute a re-analysis restricted to human-covered histories.
Table R3 (revised paper: Table 15). Mean log score S on the same human-like histories (teacher-forced replay, five seeds per task): a trajectory sampled from the human-policy model is imposed and every policy's choice is scored on it. The greedy row is the maximum attainable; no KL here because the replay runs recorded no token probabilities.
| Policy | Iowa gambling | Castle two-step | Social bandit | Weather | Mean |
|---|---|---|---|---|---|
| RNN greedy (maximum) | -0.431 | -0.536 | -0.301 | -0.443 | -0.428 |
| RNN sample | -0.799 | -0.763 | -0.524 | -0.686 | -0.693 |
| DeepSeek Stateless | -0.738 | -1.902 | -1.015 | -0.753 | -1.102 |
| DeepSeek Cumulative | -0.741 | -0.927 | -0.446 | -0.718 | -0.708 |
| DeepSeek Window | -0.736 | -1.058 | -0.423 | -0.708 | -0.731 |
| DeepSeek Retrieval | -0.756 | -0.930 | -0.544 | -0.707 | -0.734 |
| Qwen Stateless | -1.209 | -1.900 | -0.974 | -0.712 | -1.199 |
| Qwen Cumulative | -1.081 | -0.977 | -0.425 | -0.729 | -0.803 |
| Qwen Window | -0.809 | -1.191 | -0.434 | -0.704 | -0.785 |
| Qwen Retrieval | -0.975 | -1.021 | -0.507 | -0.722 | -0.806 |
| Best constant action | -2.042 | -1.710 | -0.626 | -0.744 | -1.281 |
| Uniform random | -2.233 | -1.864 | -0.987 | -0.772 | -1.464 |
Q1. How do simple policies, such as always selecting one action or greedily choosing the RNN’s most probable action, rank against humans and LLM agents?
We ran this on 20 tasks with the same RNNs and seeds (Table 12; Table R2 below). In the original S (the higher the better) the concern is confirmed: the greedy policy (-0.13) and the best constant action (-0.30) score above held-out humans (-0.44) and above the LLM agents (-0.38 to -0.71). In KL (the lower the better) the order is: held-out humans about 0, greedy 0.13, LLM agents 0.14 to 0.24, constant and random policies about 0.30. On shared histories, agents with memory are close to a sample from the RNN and well above constant and random policies (Table 15). We now report these results in the revised paper.
Table R2 (revised paper: Table 12). Evaluating held-out human participants, simple RNN policies and LLM agents (closed loop, same seeds, equal-task means over 20 tasks). KL is the plug-in estimate (Table 12 also gives a sample-based estimate); for held-out humans the calibration gap is shown; n/a: no token probabilities.
| Policy | The original likelihood score, S | KL | Tasks with S ≥ humans |
|---|---|---|---|
| Held-out humans | -0.438 | -0.002 (calibration gap) | - |
| Human-policy model, sampled | -0.452 | 0 (by definition) | 7/20 |
| Human-policy model, greedy | -0.130 | 0.130 | 20/20 |
| Best constant action | -0.297 | 0.297 | 17/20 |
| Worst constant action | -0.639 | 0.639 | 14/20 |
| Uniform random | -1.151 | 0.302 | 0/20 |
| DeepSeek Stateless | -0.637 | 0.200 | 5/20 |
| DeepSeek Cumulative | -0.409 | 0.146 | 12/20 |
| DeepSeek Window | -0.478 | 0.192 | 11/20 |
| DeepSeek Retrieval | -0.493 | 0.241 | 8/20 |
| DeepSeek Surprise | -0.430 | 0.186 | 11/20 |
| Qwen Stateless | -0.713 | 0.166 | 2/20 |
| Qwen Cumulative | -0.555 | 0.136 | 7/20 |
| Qwen Window | -0.596 | 0.148 | 4/20 |
| Qwen Retrieval | -0.619 | 0.186 | 3/20 |
| Qwen Surprise | -0.576 | 0.139 | 4/20 |
| GPT Luna Stateless | -0.523 | n/a | 8/20 |
| GPT Luna Cumulative | -0.380 | n/a | 14/20 |
| GPT Luna Window | -0.437 | n/a | 11/20 |
| GPT Luna Retrieval | -0.481 | n/a | 9/20 |
| GPT Luna Surprise | -0.413 | n/a | 13/20 |
Response to Reviewer HABF (3/3)
Q2. Do trajectory-level measures support the same conclusions about memory?
Only in part (Appendix M: Figs. 10 to 12, Tables 16 and 17; Tables R5 and R5b below). We developed several qualitative features for each task. Figs. 10 to 12 compare each backbone's own episodes with held-out participants on the 16 behavioral effects of Fig. 9, without any RNN: memory brings some history-dependent effects to the human level (for example repeating after a reward in the reward bandit, DeepSeek and Qwen), while in five tasks every configuration stays far from people. Features that require using past outcomes move into the human range with memory: in the two-step task the effect of the previous reward on repeating a choice is absent under Stateless and matches the RNN value under Cumulative. But memory also makes agents more likely to repeat previous decisions than humans (they switch less), most under Cumulative, so the share of 44 features inside the human-model 95 percent interval falls from 53 percent to 34 to 47 percent. In the risk tasks every configuration is far from humans (BART: 2 to 5 pumps per balloon against 37). The Discussion now states this.
Table R5 (revised paper: Table 16). Trajectory-level behavior by memory configuration: number of pre-specified behavioral features (44 features, nine tasks) inside the central 95% interval of the human-policy model's own rollouts, and paired change of log distance to the human-model median relative to Stateless (positive = farther; 95% task-bootstrap interval).
| Configuration | All backbones | GPT Luna | DeepSeek | Qwen | Distance to human model vs Stateless |
|---|---|---|---|---|---|
| Stateless | 67/126 (53%) | 21/38 (55%) | 24/44 (55%) | 22/44 (50%) | - |
| Cumulative | 43/127 (34%) | 13/39 (33%) | 11/44 (25%) | 19/44 (43%) | +0.44 (+0.23 to +0.66) |
| Window | 55/127 (43%) | 15/39 (38%) | 18/44 (41%) | 22/44 (50%) | +0.22 (-0.01 to +0.47) |
| Retrieval | 57/122 (47%) | 17/39 (44%) | 15/39 (38%) | 25/44 (57%) | +0.17 (-0.01 to +0.34) |
| Surprise | 54/126 (43%) | 15/38 (39%) | 16/44 (36%) | 23/44 (52%) | +0.30 (+0.07 to +0.55) |
Table R5b (revised paper: Table 17). Examples of behavioral features by memory configuration (range over the three backbones). Memory brings the history-dependent signature in the first row into the human range, makes agents more likely to repeat previous decisions than humans in rows two to five, and leaves risk taking far from human levels (last two rows).
| Feature | Human-policy model (95%) | Stateless | Cumulative | Window | Retrieval | Surprise |
|---|---|---|---|---|---|---|
| Two-step: reward effect on repeating | 0.050 (-0.002 to 0.105) | -0.013 to 0.007 | 0.038 to 0.071 | 0.029 to 0.058 | -0.084 to 0.029 | 0.047 to 0.103 |
| Two-step: repeats first choice | 0.49 (0.39 to 0.63) | 0.36 to 0.48 | 0.67 to 0.96 | 0.62 to 0.88 | 0.46 to 0.80 | 0.58 to 0.97 |
| Two-step: entropy of first choices | 0.89 (0.72 to 0.97) | 0.77 to 0.97 | 0.15 to 0.56 | 0.44 to 0.91 | 0.36 to 0.85 | 0.09 to 0.79 |
| Iowa: switch rate | 0.49 (0.27 to 0.72) | 0.14 to 0.33 | 0.03 to 0.20 | 0.02 to 0.17 | 0.05 to 0.42 | 0.05 to 0.22 |
| Iowa: advantageous decks | 0.51 (0.36 to 0.67) | 0.53 to 0.61 | 0.70 to 0.91 | 0.55 to 0.89 | 0.37 to 0.61 | 0.71 to 0.78 |
| BART: pumps per balloon | 37.4 (30.7 to 43.2) | 2.4 to 3.6 | 1.5 to 4.3 | 1.9 to 5.3 | 2.1 to 3.7 | 1.5 to 4.3 |
| Columbia: cards per round | 5.1 (3.7 to 6.3) | 0.9 to 1.8 | 0.6 to 0.9 | 0.9 to 1.0 | 1.0 to 1.4 | 0.7 to 0.9 |
Q3. Could the authors validate the RNN on human responses collected after exposure to agent-generated histories, particularly those far from the training distribution?
We agree that collecting additional human data after exposure to agent-generated histories would be the most direct test. We did not collect new human data in the discussion period; the limitations now state this and describe the experiment that a direct test would require. However, we provided indirect evidence in Appendix L: behavioral signatures on held-out participants, autonomous rollouts and shared histories.
Thank you for the detailed response and new analyses, which directly address some of my questions and are candidly reported.
However, the new results largely confirm my main concern. Most of the memory gain in S (76–95%) is lower entropy. No configuration differs from Stateless in KL after Holm correction. On trajectory-level features, memory moves agents away from the human-policy model (53% → 34–47% of features inside the human interval). This supports “memory makes agents more predictable and perseverative,” not “more human-like.” The KL used, KL(agent‖human), is also mode-seeking (the greedy RNN policy beats every LLM agent) and covers only two backbones. Finally, the evaluator checks in Appendix L use only human-like histories, so the off-distribution concern remains, though it is now acknowledged.
The benchmark and closed-loop framing are valuable, but the abstract, introduction, and conclusions should be reframed around predictability/compatibility rather than human-likeness, with KL and trajectory measures reported alongside S. I will keep my current rating.
Response to Reviewer HABF (1/3)
W1. Average log-likelihood under a human-policy model can favor repeatedly choosing the most probable action over reproducing human variability (e.g., an almost constant non-match policy in N-back scores above held-out humans), so higher scores alone do not establish human-likeness without trajectory-level validation.
We agree that a higher likelihood does not by itself establish human-likeness; we have also noted this in the Eq. 13 of our original submission. We now calculated an alternative KL divergence (see Table 2 and Appendix K): most of the memory gain is lower entropy, and KL falls modestly, most under Cumulative. Under KL the four memory configurations keep their original order (Cumulative, Surprise, Window, Retrieval); only Stateless moves ahead of Retrieval, and no configuration differs from Stateless after Holm correction.
The KL divergence also penalizes repetitive choices. For example, in the N-back (Table 13; Table R7 below): the always non-match policy has KL 0.249, held-out humans about 0, and the memory configurations of DeepSeek and Qwen 0.04 to 0.20; Section 4 now states this. KL has limitations, which the revised paper states (Section 4, Appendix K, Discussion). It needs the agent's action probabilities, so we could compute it only for the two backbones whose APIs return token log probabilities (DeepSeek and Qwen); many closed models, including GPT-5.6 Luna, do not return them. Even for these two it is approximate: the probabilities were recovered afterwards by re-submitting the saved requests, so they are not the probabilities the model used when it made the original choices.
Trajectory-level validation on 44 pre-specified features and on the 16 behavioral effects of Figure 9 (Appendix M, Figures 10 to 12): memory brings history-dependent features into the human range but also makes agents more likely to repeat previous decisions than humans. The Discussion now gives this interpretation.
Table R7 (revised paper: Table 13). The N-back example under the KL divergence (five seeds; KL is the plug-in estimate, and Table 13 also gives a sample-based estimate; n/a: no token probabilities).