acceptodds
Under review as a conference paper at ICLR 2027

RGFA: Reliability Guided Feedback Adaptation in Large Language Models via Selective Feedback Utilization

Abstract

Learning from feedback has long been studied in psychology as a fundamental mechanism underlying adaptive decision-making. In humans, this mechanism enables efficient adaptation even when feedback is noisy or uncertain. As large language models (LLMs) are increasingly deployed in interactive and decision-making tasks, a fundamental question arises: Can LLMs learn from feedback as effectively as humans, and how do their feedback-driven learning behaviors differ? Using a controlled probabilistic reversal learning paradigm, we compare how LLMs and humans learn from feedback under uncertainty. We find that LLMs perform comparably to humans with positive feedback but are much less effective with negative feedback. This gap arises mainly because LLMs struggle to distinguish informative negative feedback from environmental noise and to use it to improve their policies. To address these limitations, we propose \underline Reliability Guided Feedback Adaptation (RGFA), a lightweight approach that regulates policy updates according to the reliability of accumulated feedback evidence. Experiments show that RGFA substantially improves learning from negative feedback, and thus narrows the behavioral gap between LLMs and humans.

Primary area:
generative models
Keywords:
Feedback Learning, Large Language Models, Positive Feedback, Negative Feedback

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5zumOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how LLMs adapt to positive and negative feedback under uncertainty, comparing 29 human participants with four LLMs across different levels of feedback noise, environmental volatility, and stimulus load. The results reveal that LLMs struggle to adapt to repeated negative feedback when environmental conditions change. The authors propose Reliability-Guided Feedback Adaptation, a two-stage framework that accumulates feedback evidence and evaluates alternative strategies through prompting without updating model parameters. Experiments on Qwen3.6 show improved accuracy in volatile environments from 57.22% to 64.08%, supported by mechanism ablations and a preliminary evaluation on the Wisconsin Card Sorting Test.

Critical strengths

The controlled human-LLM comparison reveals meaningful differences in learning from negative feedback, and RGFA demonstrates improvements in adaptation to changing environments.

Critical weaknesses

The proposed controller relies on conventional evidence accumulation and hand-designed rules, with insufficient comparisons against simpler alternatives; its full version also fails to outperform the simpler Fast3 variant. Generalization evidence is limited to one primary LLM and a small cross-task evaluation.

Questions for authors
  1. How does RGFA compare with simpler Bayesian or prompting baselines, including Stage 1+Fast3?

  2. Does RGFA generalize to other LLMs and tasks without prior knowledge of environmental volatility?

6 comments
AuthorsOpenReview ↗

Response to 5zum (1/2) — Q1

Dear Reviewer 5zum,

Thank you for your careful review and for highlighting the differences in how humans and LLMs respond to negative feedback in our controlled comparison. Your suggestion to compare simpler alternatives helped us decide which RGFA configuration to adopt in the revision.

We address your concerns point by point below.

Q1. How does RGFA compare with simpler Bayesian or prompting baselines, including Stage 1+Fast3?

Configuration names. In the revised manuscript, RGFA refers to the configuration previously called Stage 1 + Fast3, and RGFA extension refers to the original RGFA configuration in the submitted manuscript.

Qwen3.6-27B mechanism ablation (Appendix E.3)

On ten matched task schedules, V+ accuracy was 55.87% for the unaided baseline, 61.42% with Stage 1 only, 65.69% with RGFA, and 64.65% with the RGFA extension. Stage 1 uses Beta counts to accumulate feedback and guide reconsideration. Stage 2 tracks short-term feedback on candidate options.

Additional comparisons on nine matched schedules (Appendix E.4)

  • Prompting controls. All LLM conditions used Qwen3.7-Plus. The generic reminder asked the model to review feedback for the current stimulus. The fixed revision rule asked the model to reconsider an option after it received negative feedback on that option on two successive encounters with the same stimulus.
  • Standalone policies. The standalone RGFA controllers and UCB policies selected actions directly and received feedback on their own choices. The RGFA controllers used the announced environment type. Discounted UCB used a discount factor of 0.95, and sliding-window UCB used the 20 most recent encounters with the same stimulus.
MethodOverall (%)Ref (%)S+ (%)V+ (%)
Qwen3.7-Plus, unaided53.5678.7062.9641.94
+ Generic reminder55.8180.0966.3945.37
+ Fixed revision rule54.4975.7451.3049.34
+ Stage 1 only58.2886.5765.4646.41
+ RGFA guidance62.2981.2062.9657.06
+ RGFA extension62.1885.0962.9654.21
Standalone RGFA75.7090.6586.1165.16
Standalone RGFA extension75.8790.2886.3065.35
Discounted UCB49.8952.6942.3151.12
Sliding-window UCB61.9571.3958.1560.07
  • RGFA guidance achieved higher overall and V+ accuracy than both prompting controls.
  • Sliding-window UCB and Qwen3.7-Plus with RGFA guidance achieved similar overall accuracy (61.95% vs. 62.29%).
  • For both configurations, the standalone controller achieved higher overall and V+ accuracy than the corresponding guided LLM.

Choice of the final configuration

In Stage 2, the RGFA extension uses Stage 1 contradiction levels to decide whether to retain, continue monitoring, or release a candidate. RGFA achieved higher observed V+ accuracy than the extension in both the Qwen3.6-27B ablation and the Qwen3.7-Plus guidance comparison. Given these results and its simpler Stage 2 rules, we adopted RGFA as the final configuration in the revised manuscript. Thank you again for suggesting this comparison.

AuthorsOpenReview ↗

Response to 5zum (2/2) — Q2

Q2:Does RGFA generalize to other LLMs and tasks without prior knowledge of environmental volatility?

Probabilistic reversal learning with the environment type withheld (Appendix E.5)

  • Setup. We evaluated Qwen3.7-Plus with and without RGFA on the same nine task schedules, with the environment type either announced or withheld. Reversal times and exact feedback probabilities were withheld in both settings.
  • Controller. In the withheld setting, neither the model nor the controller received the environment type. The model was told that feedback could be noisy and preferences could change. The controller used a fixed evidence retention factor of 0.80 in every block and triggered Stage 2 based on observed feedback.
Environment typeUnaided overall accuracy (%)RGFA overall accuracy (%)
Announced53.5662.29
Withheld53.6261.17

With the environment type withheld, RGFA improved overall accuracy by 7.55 percentage points over the unaided baseline under the same information setting.

Transfer to the Wisconsin Card Sorting Test (WCST; Section 6 and Appendix E.7)

  • Task information. Qwen3.6-Plus and Kimi K2.5 were told that cards could be matched by shape, color, or number, that feedback was deterministic, and that the sorting rule could change without warning. The active rule and the timing of rule changes were withheld from both the model and the controller.
  • Adaptation. Stage 1 tracked the rules consistent with observed feedback. The adapted Stage 2 used up to three subsequent informative observations to verify candidate rules. The LLM made the final card choice.
ModelMatched schedulesUnaided (%)Generic reminder (%)Stage 1 only (%)RGFA (%)
Qwen3.6-Plus549.1749.3171.1175.42
Kimi K2.5452.0853.1256.6057.29

On the tested card-sorting schedules, Stage 1 alone achieved higher overall accuracy than both controls for each model, and adding the adapted Stage 2 raised accuracy further.

Together, these results show that RGFA improved choice accuracy on the tested reversal-learning and card-sorting schedules when changes were unannounced.

Thank you again for your constructive feedback.

Best regards,
The Authors

AuthorsOpenReview ↗

Acknowledgments and Further Discussion

Dear Reviewer 5zum,

Thank you again for your careful reading and helpful suggestions. We would like to summarize the main points from our response.

  • We adopted Stage 1 + Fast3 as the final RGFA configuration. It uses simpler Stage 2 rules and achieved higher accuracy in the volatile condition than the original extension in both model comparisons. In the Qwen3.6-27B ablation, adding Stage 2 raised volatile-condition accuracy from 61.42% to 65.69%.
  • RGFA-guided Qwen3.7-Plus outperformed both prompting controls in overall and volatile-condition accuracy. Sliding-window UCB achieved similar overall accuracy and scored higher in the volatile condition. The standalone RGFA controller outperformed the guided model on both metrics.
  • With the environment type withheld from both Qwen3.7-Plus and the controller, the controller used a fixed evidence retention factor and triggered Stage 2 based on observed feedback. RGFA raised overall accuracy from 53.62% to 61.17%.
  • On the Wisconsin Card Sorting Test, both the active rule and the timing of rule changes were withheld. Stage 1 outperformed both controls for Qwen3.6-Plus and Kimi K2.5, and adding the adapted Stage 2 improved accuracy for both models.

The revised manuscript reports the reversal-learning ablations, baseline comparisons, and evaluation with the environment type withheld in Appendices E.3–E.5. Section 6 and Appendix E.7 report the card-sorting ablations.

We would be grateful if you could let us know whether any questions or concerns remain. We would be happy to provide further clarification or address any additional points. We look forward to hearing from you.

Best,

Authors

Reviewer_JVw6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLMs use repeated feedback when a negative outcome may reflect either noise or a change in the environment. The authors introduce a controlled task and propose RGFA, which accumulates feedback before revising a choice. RGFA improves performance on the main task for one model.

Critical strengths

The paper addresses a practical problem: an agent receiving repeated feedback must decide when to stick with a choice and when to change it.

The task separates noisy feedback from changes in the environment, and the human comparison helps identify where the models struggle.

Critical weaknesses

RGFA is mainly evaluated on one model and one task paradigm. The results do not yet establish how well the method generalizes across models or settings.

The method is simple, but the paper gives limited evidence for why the models show this behavior and why RGFA helps. It is also unclear whether a simpler feedback-tracking method would work as well.

Questions for authors

How does this work advance beyond the findings of Wang et al. (2026) on LLMs in probabilistic reversal learning?

What explains the models’ difficulty in responding to negative feedback, and what evidence supports the proposed explanation for RGFA’s gains? Would a simpler method that tracks recent feedback perform similarly?

What real-world setting motivates RGFA? How would the method work when the reliability of feedback is not known in advance?

Reference

Wang, H., Ward, T. E., and Zhang, L. (2026). Comparative reversal learning reveals rigid adaptation in LLMs under non-stationary uncertainty. Communications in Computer and Information Science, 3020, 452–466. https://doi.org/10.1007/978-3-032-28997-1_32

5 comments
AuthorsOpenReview ↗

Response to JVw6 (1/3) — Q1

Dear Reviewer JVw6,

Thank you for your constructive feedback and for highlighting the practical question of when an agent should retain or revise a choice. We appreciate your recognition that the human comparison helps identify where LLMs struggle.

We address your concerns point by point below.

Q1: How does this work advance beyond the findings of Wang et al. (2026) on LLMs in probabilistic reversal learning?

Thank you for pointing us to this work.

  • What our design adds. Wang et al. found that the tested LLMs switched less often than humans after losses and that some models persisted with formerly preferred options after reversals. We build on these findings by comparing human and LLM responses to different patterns of negative feedback on matched task schedules. Our design includes a noisy condition (S+) in which the preferred option remains fixed.

  • What we observed in the volatile condition (V+). We measured switching at the next encounter with the same stimulus after two types of feedback events: stochastic negative feedback on the currently preferred option, and two consecutive negative outcomes on the formerly preferred option after a reversal. Humans switched more often in the second case (65.43% vs. 46.25%). Qwen3.6-27B switched at similar rates in the two cases (45.92% vs. 47.89%). These results are reported in Table 2(b).

  • How this finding informed RGFA. This behavioral pattern motivated Stage 1 of RGFA, which uses corroborating negative feedback to guide when to reconsider an established choice. We now cite Wang et al. in Section 2 (Related Work) of the revised manuscript.

AuthorsOpenReview ↗

Response to JVw6 (2/3) — Q2

Q2:What explains the models’ difficulty in responding to negative feedback, and what evidence supports the proposed explanation for RGFA’s gains? Would a simpler method that tracks recent feedback perform similarly?

Response to negative feedback

In V+, we compared switching at the next encounter with the same stimulus after stochastic negative feedback on the currently preferred option and after two consecutive negative outcomes on the formerly preferred option following a reversal (Table 2(b)).

  • Humans switched more often after repeated post-reversal negative feedback (65.43% vs. 46.25%).
  • Qwen3.6-27B switched at similar rates in the two cases (45.92% vs. 47.89%). The model's switching behavior showed little differentiation between these two feedback contexts.

Evidence from the Qwen3.6-27B ablation (Appendix E.3)

In the revised manuscript, RGFA refers to the configuration previously called Stage 1 + Fast3, and RGFA extension refers to the original RGFA configuration in the submitted manuscript.

  • Stage 1 changed when the model switched. On ten matched schedules, Stage 1 alone reduced switching after stochastic negative feedback from 45.80% to 37.62% and increased switching after repeated post-reversal negative feedback from 45.58% to 80.04%, relative to the unaided baseline.
  • Adding Stage 2 increased V+ accuracy from 61.42% to 65.69%. Stage 2 tracks recent feedback on candidate options after reconsideration is triggered.

Comparison with simpler methods (Appendix E.4)

We evaluated the methods on nine matched schedules, using Qwen3.7-Plus in all LLM conditions. The table reports choice accuracy (%).

  • Prompting controls. The generic reminder asked the model to review feedback for the current stimulus. The fixed revision rule asked it to reconsider an option after it chose that option and received negative feedback on two successive encounters with the same stimulus.
  • Standalone policies. The standalone RGFA controllers used the announced environment type and selected actions directly. Discounted UCB used a discount factor of 0.95, and sliding-window UCB used the 20 most recent encounters with the same stimulus. Each method received feedback on its own choices.
MethodOverallRefS+V+
Qwen3.7-Plus, unaided53.5678.7062.9641.94
+ Generic reminder55.8180.0966.3945.37
+ Fixed revision rule54.4975.7451.3049.34
+ Stage 1 only58.2886.5765.4646.41
+ RGFA guidance62.2981.2062.9657.06
+ RGFA extension62.1885.0962.9654.21
Standalone RGFA75.7090.6586.1165.16
Standalone RGFA extension75.8790.2886.3065.35
Discounted UCB49.8952.6942.3151.12
Sliding-window UCB61.9571.3958.1560.07
  • Both guidance configurations achieved higher overall and V+ accuracy than the unaided model and both prompting controls.
  • Sliding-window UCB achieved similar overall accuracy to Qwen3.7-Plus with RGFA guidance (61.95% vs. 62.29%) and higher V+ accuracy (60.07% vs. 57.06%).
  • Both standalone controllers achieved higher overall and V+ accuracy than their corresponding guided LLM conditions.

Choice of the final configuration

In Stage 2, the RGFA extension uses Stage 1 contradiction levels to decide whether to retain, continue monitoring, or release a candidate. RGFA achieved higher observed V+ accuracy than the extension in both the Qwen3.6-27B ablation (65.69% vs. 64.65%) and the Qwen3.7-Plus guidance comparison (57.06% vs. 54.21%). Given these results and its simpler Stage 2 rules, we adopted RGFA as the final configuration in the revised manuscript.

AuthorsOpenReview ↗

Response to JVw6 (3/3) — Q3

Q3:What real-world setting motivates RGFA? How would the method work when the reliability of feedback is not known in advance?

Motivating example

A software troubleshooting assistant may recommend the same fix when an error recurs. A previously effective fix can fail during a temporary outage or become ineffective after a software update. The assistant must decide when repeated failures warrant reconsidering its recommendation.

How RGFA uses feedback

  • Stage 1 guides when to reconsider a choice. It uses accumulated feedback and treats repeated negative outcomes on the same option as corroborating evidence.
  • Stage 2 tracks feedback on candidate options. Once Stage 1 triggers reconsideration, Stage 2 tracks candidate feedback for up to three subsequent encounters with the same stimulus. The LLM makes the final choice.

Evaluation with the environment type withheld (Appendix E.5)

  • Setup. We evaluated Qwen3.7-Plus with and without RGFA on the same nine task schedules, with the environment type either announced or withheld. Exact feedback probabilities and reversal times were withheld in both settings.
  • Withheld setting. We withheld the environment type from both the model and the controller. The model was told that feedback could be noisy and preferences could change. The controller used a fixed evidence retention factor of 0.80 in every block and triggered Stage 2 based on observed feedback.
Environment typeUnaided overall accuracy (%)RGFA overall accuracy (%)
Announced53.5662.29
Withheld53.6261.17

With the environment type and exact feedback probabilities withheld, RGFA improved overall accuracy by 7.55 percentage points over the unaided baseline on the same nine schedules.On these nine task schedules, RGFA used observed feedback to improve the model’s choice accuracy even without prior knowledge of the environment type or exact feedback probabilities.

Thank you again for your constructive feedback.

Best regards,
The Authors

Reviewer_A2uTOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper compares how humans and LLMs learn from probabilistic feedback in a reversal-learning task that crosses two stimulus loads with stable, noisy and volatile environments, under feedback schedules matched between 29 participants and four LLMs. It reports that Qwen3.6 matches or exceeds humans in stable conditions and after positive feedback, but switches less than humans after repeated negative feedback following a reversal, and concludes that LLMs struggle to tell informative negative feedback from noise. The proposed method, RGFA, is a controller outside the model that maintains Beta evidence counts for each stimulus and option, which it uses to decide from the evidence gap and repeated contradictions when a revision is warranted, and then tracks new candidate options for a few presentations. The controller adds guidance to the prompt, while the LLM still makes each choice. RGFA raises Qwen3.6's accuracy in the volatile condition, and a card-sorting task is presented as evidence that RGFA can generalize beyond reversal learning.

Critical strengths
  1. The task formulation is valuable. It asks whether a model can tell noisy feedback from a real change in the environment. The finding that LLMs, while strong under stable feedback, handle these reversals poorly is a useful diagnosis.

  2. RGFA improves accuracy in the volatile condition at both stimulus loads. Accuracy rises under both loads (i.e., with 2 and 4 stimuli), the gain grows over successive reversals, and performance in the stable conditions stays close to the baseline.

Critical weaknesses
  1. The contribution is limited, and the paper is not positioned against closely related work. The task is a standard probabilistic reversal-learning setup with three options and two or four stimuli. Reflection-Bench [1], whose card-sorting task the paper uses, already evaluates LLMs on probabilistic reversal learning and reports rigid beliefs in some models after the reversal. Schubert et al. [2] also report that in-context LLM learners use positive outcomes much more than negative ones. So the new parts seem to be the human-matched schedules, the noisy condition without reversals and the split between isolated and repeated negative feedback. The setting is also related to work on LLM memory and personalization, where a model must act on a preference or fact that has changed. However, the related work covers only psychology and LLM self-correction and does not discuss work on LLM memory, personalization or preference drift. The contribution should be positioned against these lines of work.

  2. RGFA is a hand-designed bandit heuristic, and the LLM's own contribution is not tested. For each stimulus the task is a three-armed bandit with binary feedback, and in the volatile condition a switching bandit. RGFA's Beta counts with a retention factor and its window of three later presentations work like the discounted and sliding-window estimates of Garivier and Moulines [3], and revising after two same-option negatives works like change detection. Yet the paper never mentions bandits and uses the unaided LLM as its only baseline. There is no bandit baseline, no controller that chooses on its own from the same evidence, and no control that gives the model a fixed instruction stating the revision rule or a generic reminder instead of the controller's guidance. So it is unclear how much the LLM adds beyond following the controller. The constants, such as δ=0.08\delta = 0.08, W=3W = 3 and the corroboration thresholds, are set by hand, and Table 5 varies them only one at a time on three schedules. The paper does not explain how the defaults were chosen. The retention factor and the use of Stage 2 also follow the announced stable or volatile regime, so the controller never has to infer whether the environment is volatile.

  3. The evaluation of RGFA is narrow. The parameter and mechanism ablations appear only in the last appendix, Appendix E, and cover only the reversal task with Qwen3.6, so none of them is run on the card-sorting task. On that task, Table 3 compares the baseline with an "RGFA extension", but the paper does not describe how this extension changes the two stages. It is therefore hard to tell whether the gain comes from RGFA itself or from task-specific changes, and the claim that RGFA can generalize beyond reversal learning depends on this single adapted task.

[1] Li, Lingyu, et al. "Reflection-Bench: Evaluating Epistemic Agency in Large Language Models." Forty-second International Conference on Machine Learning. 2025.

[2] Schubert, Johannes A., et al. "In-Context Learning Agents Are Asymmetric Belief Updaters." Forty-first International Conference on Machine Learning. 2024.

[3] Garivier, Aurélien, and Eric Moulines. "On upper-confidence bound policies for switching bandit problems." International conference on algorithmic learning theory. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011.

Questions for authors
  1. Clarification on W1. Please position the task and the findings against Reflection-Bench [1] and Schubert et al. [2], and discuss how the setting relates to work on LLM memory and personalization.

  2. Requested comparison for W2. On the same schedules, please report the controller choosing on its own from the same evidence and a standard bandit method for switching environments, such as discounted or sliding-window UCB [3]. Please also report Qwen3.6 given a fixed instruction stating the revision rule or a generic reminder instead of the controller's guidance. This would show how much the LLM adds to the controller.

  3. Clarification on W2. Please explain how the default values of δ\delta, WW and the corroboration thresholds were chosen, and whether the three schedules in Table 5 overlap with the 31 schedules of the main comparison.

  4. Requested comparison for W3. Please describe how the "RGFA extension" changes the two stages for the card-sorting task, and run the main ablations of Table 6 on that task.

Minor Concerns

  1. Table 2(a) and Figure 4(a) report different accuracies in the volatile condition for the same participants and baseline runs, namely 57.15% and 59.04% for humans and 56.73% and 57.22% for Qwen3.6. Please clarify which values are correct or why they are not matched.

  2. The card-sorting task is attributed to Reflection-Bench with the citation "Li et al., 2025", which in the reference list is a different paper, and Reflection-Bench [1] itself is not cited. Please correct the citation.

7 comments
AuthorsOpenReview ↗

Response to A2uT (1/4) — W1/Q1

Dear Reviewer A2uT,

Thank you for your careful review and constructive feedback. We appreciate your recognition of the value of our task design for examining whether LLMs can distinguish noisy feedback from genuine environmental changes. We also appreciate your observation that RGFA improves accuracy in volatile conditions at both stimulus loads while keeping accuracy close to the baseline in stable conditions.

We address your concerns point by point below.

W1&Q1: (W1) The contribution is limited, and the paper is not positioned against closely related work. The task is a standard probabilistic reversal-learning setup with three options and two or four stimuli. Reflection-Bench [1], whose card-sorting task the paper uses, already evaluates LLMs on probabilistic reversal learning and reports rigid beliefs in some models after the reversal. Schubert et al. [2] also report that in-context LLM learners use positive outcomes much more than negative ones. So the new parts seem to be the human-matched schedules, the noisy condition without reversals and the split between isolated and repeated negative feedback. The setting is also related to work on LLM memory and personalization, where a model must act on a preference or fact that has changed. However, the related work covers only psychology and LLM self-correction and does not discuss work on LLM memory, personalization or preference drift. The contribution should be positioned against these lines of work. (Q1) Clarification on W1. Please position the task and the findings against Reflection-Bench [1] and Schubert et al. [2], and discuss how the setting relates to work on LLM memory and personalization.

  1. What our design adds. Reflection-Bench reports that some LLMs persist with earlier choices after a reversal. Schubert et al. show asymmetric belief updating after better- and worse-than-expected outcomes. We compare humans and LLMs on matched task schedules to examine how their choices respond to different patterns of negative feedback. Our design includes a noisy condition (S+) in which the optimal option stays fixed, allowing us to observe behavior under noisy feedback without reversals.

  2. What we find within the volatile condition (V+). We measure switching at the next encounter with the same stimulus after two events: (i) stochastic negative feedback on the currently optimal option and (ii) two consecutive negative outcomes on the formerly optimal option after a reversal.

    • Humans switched more often in the second case (65.43% vs. 46.25%).
    • Qwen3.6 switched at similar rates in both cases (45.92% vs. 47.89%).

    This behavioral difference (Table 2(b) and Fig. 4(d–e)) motivated RGFA’s use of corroborating feedback to decide when to reconsider an established choice.

  3. Connection to memory and personalization. LongMemEval assesses knowledge updating across conversations, and PersonaMem assesses responses to changing user preferences. The shared question is when new evidence warrants revising what the model has been relying on. We study that decision in a controlled choice task with probabilistic feedback. Section 2 of the revised manuscript discusses these connections and explains how our matched task design extends prior work.

AuthorsOpenReview ↗

Response to A2uT (2/4) — W2/Q2

W2&Q2:(W2)RGFA is a hand-designed bandit heuristic, and the LLM's own contribution is not tested. For each stimulus the task is a three-armed bandit with binary feedback, and in the volatile condition a switching bandit. RGFA's Beta counts with a retention factor and its window of three later presentations work like the discounted and sliding-window estimates of Garivier and Moulines [3], and revising after two same-option negatives works like change detection. Yet the paper never mentions bandits and uses the unaided LLM as its only baseline. There is no bandit baseline, no controller that chooses on its own from the same evidence, and no control that gives the model a fixed instruction stating the revision rule or a generic reminder instead of the controller's guidance. So it is unclear how much the LLM adds beyond following the controller. (Q2)Requested comparison for W2. On the same schedules, please report the controller choosing on its own from the same evidence and a standard bandit method for switching environments, such as discounted or sliding-window UCB [3]. Please also report Qwen3.6 given a fixed instruction stating the revision rule or a generic reminder instead of the controller's guidance. This would show how much the LLM adds to the controller.

Thank you for suggesting these comparisons. We evaluated all methods on the same nine task schedules and used Qwen3.7-Plus in every LLM condition, including the unaided baseline.

  • Prompting controls. The generic reminder asked the model to review feedback for the current stimulus. The fixed revision rule asked it to reconsider an option after choosing it and receiving negative feedback on two successive encounters with the same stimulus. The LLM made the final choice in all LLM conditions.
  • Standalone controllers and bandit baselines. Each standalone RGFA controller retained its evidence updates and candidate tracking, used the announced environment type, and selected actions directly through an explicit decision rule. Discounted UCB used a discount factor of 0.95. Sliding-window UCB used the 20 most recent encounters with the same stimulus. Each method received feedback generated by its own choices.

The table reports choice accuracy (%). Overall accuracy pools all trials. Within each environment, the two stimulus loads receive equal weight.In the revised manuscript, RGFA refers to the configuration previously called Stage 1 + Fast3, and RGFA extension refers to the original RGFA configuration in the submitted manuscript.

MethodOverallRefS+V+
Qwen3.7-Plus, unaided53.5678.7062.9641.94
+ Generic reminder55.8180.0966.3945.37
+ Fixed revision rule54.4975.7451.3049.34
+ RGFA extension62.1885.0962.9654.21
+ RGFA62.2981.2062.9657.06
RGFA extension controller only75.8790.2886.3065.35
RGFA controller only75.7090.6586.1165.16
Discounted UCB49.8952.6942.3151.12
Sliding-window UCB61.9571.3958.1560.07
  • Both RGFA guidance variants improved overall and V+ accuracy over the unaided model and both prompting controls.
  • Sliding-window UCB and Qwen3.7-Plus with RGFA guidance achieved similar overall accuracy (61.95% vs. 62.29%).
  • For both RGFA variants, the standalone controller achieved higher overall and V+ accuracy than the corresponding guided LLM. For RGFA, overall accuracy was 75.70% with direct controller decisions and 62.29% with the guided LLM.

**Why we adopted RGFA.**RGFA uses simpler Stage 2 rules and achieved higher observed V+ accuracy than the extension in both the Qwen3.6-27B ablation on ten matched schedules (Appendix E.3) and the Qwen3.7-Plus guidance comparison on nine matched schedules (Appendix E.4).We therefore adopted RGFA as the final configuration.

AuthorsOpenReview ↗

Response to A2uT (3/4) — W2/Q3

W2&Q3:(W2)The constants, such as δ = 0.08, W = 3 and the corroboration thresholds, are set by hand, and Table 5 varies them only one at a time on three schedules. The paper does not explain how the defaults were chosen. The retention factor and the use of Stage 2 also follow the announced stable or volatile regime, so the controller never has to infer whether the environment is volatile.(Q3)Clarification on W2. Please explain how the default values of δ\delta, WW and the corroboration thresholds were chosen, and whether the three schedules in Table 5 overlap with the 31 schedules of the main comparison.

  1. Stage 1 evidence margin (δ=0.08\delta = 0.08). We replayed 20 existing Qwen3.6-27B baseline trajectories offline and compared margins of 0.08, 0.10, and 0.12. We selected 0.08 based on how the margins classified stochastic negative feedback in stable environments and negative feedback after reversals. A subsequent sensitivity analysis of the RGFA compared margins of 0.04, 0.08, and 0.12. The 0.08 margin yielded the highest observed mean V+ accuracy among those settings. We retained δ=0.08\delta = 0.08 in the final RGFA configuration.

  2. Corroboration threshold and Stage 2 window (W=3W = 3).

    • Stage 1 uses two negative outcomes to corroborate weak evidence in V+. If the model chooses the same option at the next encounter with the same stimulus and again receives negative feedback, the second outcome provides the earliest corroboration of the first.
    • Stage 2 tracks candidate feedback over up to three subsequent encounters with the same stimulus. A first positive outcome provisionally supports a candidate, and a second positive outcome confirms it. Two negative outcomes on that candidate after provisional support withdraw the support. Three encounters are the minimum needed to observe one positive outcome followed by two negative outcomes. The window starts at the next encounter with the same stimulus after the Stage 1 trigger.
  3. All three sensitivity-analysis schedules were included in the original 31-schedule comparison. Table 5 of the submitted manuscript used schedules 3010, 3011, and 3012 and varied one hyperparameter at a time in the RGFA.

  4. Evaluation with the environment type withheld (Appendix E.5). Exact feedback probabilities and reversal times were withheld in both settings.

    • Announced. The RGFA controller used a retention factor of 1.00 in Ref and S+ and 0.80 in V+, with Stage 2 enabled only in V+.
    • Withheld. The environment type was withheld from both Qwen3.7-Plus and the controller. The model was told that feedback could be noisy and preferences could change. The controller used a fixed retention factor of 0.80 in every block and triggered Stage 2 based on observed feedback.

The table reports choice accuracy (%) on the same nine task schedules.

Environment typeMethodOverallRefS+V+
AnnouncedQwen3.7-Plus, unaided53.5678.7062.9641.94
Announced+ RGFA62.2981.2062.9657.06
WithheldQwen3.7-Plus, unaided53.6281.1145.9347.11
Withheld+ RGFA61.1780.0045.1960.73

With the environment type withheld, RGFA improved overall accuracy by 7.55 percentage points over the unaided baseline. Gains were concentrated in V+ (+13.62 percentage points). In Ref and S+, accuracy remained close to the unaided baseline under the same information setting.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Rebuttal Summary and Manuscript Revisions

Dear Area Chair,

Thank you and the reviewers for your time and constructive feedback. We revised the manuscript in response to the follow-up comments to clarify RGFA’s contribution and the scope of the experimental evidence.

Reviewer follow-up

  1. Reviewer A2uT found the additional baseline comparisons, the evaluation with the environment type withheld, and the WCST ablations helpful, and raised their score from 1 to 2. Their remaining concerns centered on the role of the LLM, what the WCST results establish, and the relationship to PAMU.
  2. Reviewer 5zum stated that the rebuttal and additional experiments addressed many of their original concerns, described the paper as “technically solid,” and said they would raise their rating from 2 to 3. They continued to raise questions about RGFA’s novelty and its advantages over simpler methods.

Changes in response to the follow-up

  • C1. Standalone control and LLM guidance (A2uT, 5zum). We clarify that our study compares how humans and LLMs respond to feedback on matched schedules and tests whether the observed differences can inform guidance that improves LLM choices. On the nine matched schedules, standalone RGFA achieved the highest overall accuracy among the tested methods (75.70%). When the LLM made the final choice, RGFA guidance raised overall accuracy from 53.56% to 62.29%. Sliding-window UCB achieved similar overall accuracy to the guided LLM (61.95%) and higher accuracy in the volatile condition. The revision explains what these comparisons support for each way of using RGFA.

  • C2. Scope of the WCST evidence (A2uT). We clarify how the two stages were adapted to deterministic feedback. Stage 1 accumulates constraints and prompts reconsideration when new feedback conflicts with every remaining candidate rule. Stage 2 then helps the LLM test the new candidates. In the existing Qwen3.6-Plus logs, Stage 1 rebuilt the candidate set by the third post-change trial’s feedback in 49 of 50 blocks across five schedules with two runs each. Stage 1 accounted for most of the gain for Qwen3.6-Plus, and adding Stage 2 improved accuracy by 4.31 percentage points for Qwen3.6-Plus and 0.69 points for Kimi K2.5. We present WCST as preliminary evidence that the two-stage design can be adapted to deterministic rule switching.

  • C3. Methodological contribution and prior work (A2uT, 5zum). We discuss PAMU as related work that also uses information maintained outside the LLM to guide prompting. RGFA’s specific methodological contributions are the negative-feedback corroboration rule and the tracking of candidate options. These mechanisms were motivated by the switching differences we observed between humans and LLMs. In the volatile condition, full RGFA guidance achieved 57.06% accuracy, compared with 49.34% for a fixed prompt that already instructed the LLM to reconsider after two consecutive losses on the same option.

We believe the revision addresses several of the questions raised during discussion and makes clearer what the evidence supports. Our individual replies provide further details on the comparisons and analyses. Thank you again for your consideration.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.