- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how LLMs adapt to positive and negative feedback under uncertainty, comparing 29 human participants with four LLMs across different levels of feedback noise, environmental volatility, and stimulus load. The results reveal that LLMs struggle to adapt to repeated negative feedback when environmental conditions change. The authors propose Reliability-Guided Feedback Adaptation, a two-stage framework that accumulates feedback evidence and evaluates alternative strategies through prompting without updating model parameters. Experiments on Qwen3.6 show improved accuracy in volatile environments from 57.22% to 64.08%, supported by mechanism ablations and a preliminary evaluation on the Wisconsin Card Sorting Test.
Critical strengths
The controlled human-LLM comparison reveals meaningful differences in learning from negative feedback, and RGFA demonstrates improvements in adaptation to changing environments.
Critical weaknesses
The proposed controller relies on conventional evidence accumulation and hand-designed rules, with insufficient comparisons against simpler alternatives; its full version also fails to outperform the simpler Fast3 variant. Generalization evidence is limited to one primary LLM and a small cross-task evaluation.
Questions for authors
-
How does RGFA compare with simpler Bayesian or prompting baselines, including Stage 1+Fast3?
-
Does RGFA generalize to other LLMs and tasks without prior knowledge of environmental volatility?
6 comments
Response to 5zum (2/2) — Q2
Q2:Does RGFA generalize to other LLMs and tasks without prior knowledge of environmental volatility?
Probabilistic reversal learning with the environment type withheld (Appendix E.5)
- Setup. We evaluated Qwen3.7-Plus with and without RGFA on the same nine task schedules, with the environment type either announced or withheld. Reversal times and exact feedback probabilities were withheld in both settings.
- Controller. In the withheld setting, neither the model nor the controller received the environment type. The model was told that feedback could be noisy and preferences could change. The controller used a fixed evidence retention factor of 0.80 in every block and triggered Stage 2 based on observed feedback.
| Environment type | Unaided overall accuracy (%) | RGFA overall accuracy (%) |
|---|---|---|
| Announced | 53.56 | 62.29 |
| Withheld | 53.62 | 61.17 |
With the environment type withheld, RGFA improved overall accuracy by 7.55 percentage points over the unaided baseline under the same information setting.
Transfer to the Wisconsin Card Sorting Test (WCST; Section 6 and Appendix E.7)
- Task information. Qwen3.6-Plus and Kimi K2.5 were told that cards could be matched by shape, color, or number, that feedback was deterministic, and that the sorting rule could change without warning. The active rule and the timing of rule changes were withheld from both the model and the controller.
- Adaptation. Stage 1 tracked the rules consistent with observed feedback. The adapted Stage 2 used up to three subsequent informative observations to verify candidate rules. The LLM made the final card choice.
| Model | Matched schedules | Unaided (%) | Generic reminder (%) | Stage 1 only (%) | RGFA (%) |
|---|---|---|---|---|---|
| Qwen3.6-Plus | 5 | 49.17 | 49.31 | 71.11 | 75.42 |
| Kimi K2.5 | 4 | 52.08 | 53.12 | 56.60 | 57.29 |
On the tested card-sorting schedules, Stage 1 alone achieved higher overall accuracy than both controls for each model, and adding the adapted Stage 2 raised accuracy further.
Together, these results show that RGFA improved choice accuracy on the tested reversal-learning and card-sorting schedules when changes were unannounced.
Thank you again for your constructive feedback.
Best regards,
The Authors
Acknowledgments and Further Discussion
Dear Reviewer 5zum,
Thank you again for your careful reading and helpful suggestions. We would like to summarize the main points from our response.
- We adopted Stage 1 + Fast3 as the final RGFA configuration. It uses simpler Stage 2 rules and achieved higher accuracy in the volatile condition than the original extension in both model comparisons. In the Qwen3.6-27B ablation, adding Stage 2 raised volatile-condition accuracy from 61.42% to 65.69%.
- RGFA-guided Qwen3.7-Plus outperformed both prompting controls in overall and volatile-condition accuracy. Sliding-window UCB achieved similar overall accuracy and scored higher in the volatile condition. The standalone RGFA controller outperformed the guided model on both metrics.
- With the environment type withheld from both Qwen3.7-Plus and the controller, the controller used a fixed evidence retention factor and triggered Stage 2 based on observed feedback. RGFA raised overall accuracy from 53.62% to 61.17%.
- On the Wisconsin Card Sorting Test, both the active rule and the timing of rule changes were withheld. Stage 1 outperformed both controls for Qwen3.6-Plus and Kimi K2.5, and adding the adapted Stage 2 improved accuracy for both models.
The revised manuscript reports the reversal-learning ablations, baseline comparisons, and evaluation with the environment type withheld in Appendices E.3–E.5. Section 6 and Appendix E.7 report the card-sorting ablations.
We would be grateful if you could let us know whether any questions or concerns remain. We would be happy to provide further clarification or address any additional points. We look forward to hearing from you.
Best,
Authors
Response to 5zum (1/2) — Q1
Dear Reviewer 5zum,
Thank you for your careful review and for highlighting the differences in how humans and LLMs respond to negative feedback in our controlled comparison. Your suggestion to compare simpler alternatives helped us decide which RGFA configuration to adopt in the revision.
We address your concerns point by point below.
Configuration names. In the revised manuscript, RGFA refers to the configuration previously called Stage 1 + Fast3, and RGFA extension refers to the original RGFA configuration in the submitted manuscript.
Qwen3.6-27B mechanism ablation (Appendix E.3)
On ten matched task schedules, V+ accuracy was 55.87% for the unaided baseline, 61.42% with Stage 1 only, 65.69% with RGFA, and 64.65% with the RGFA extension. Stage 1 uses Beta counts to accumulate feedback and guide reconsideration. Stage 2 tracks short-term feedback on candidate options.
Additional comparisons on nine matched schedules (Appendix E.4)
Choice of the final configuration
In Stage 2, the RGFA extension uses Stage 1 contradiction levels to decide whether to retain, continue monitoring, or release a candidate. RGFA achieved higher observed V+ accuracy than the extension in both the Qwen3.6-27B ablation and the Qwen3.7-Plus guidance comparison. Given these results and its simpler Stage 2 rules, we adopted RGFA as the final configuration in the revised manuscript. Thank you again for suggesting this comparison.