- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how reward-interface presentation affects multi-round in-context reinforcement learning with frozen language models. It compares numeric and structured qualitative feedback, then five rhetorical frames against a neutral interface, using fixed deterministic scorers. Experiments cover three models, FEVER and SQuAD 2.0, and 16,800 twenty-round episodes. Structured feedback helps in some model–task settings, while several framed packages reduce late reward and increase unsupported commitments. The main contribution is an empirical characterization of complete interface packages.
Critical strengths
- Fixed deterministic scorers, common evaluation items, repeated runs, and paired analyses with multiplicity correction make the package-level endpoint comparisons reasonably convincing.
- Separating late reward from adaptation gain, and examining unsupported commitments and error transitions, provides a more informative account than endpoint scores alone. The paper appropriately treats transition and attention analyses as descriptive.
Critical weaknesses
W1. The boundary with prior work and the substantive contribution are insufficiently established. Reward Is Enough [1] and OPRO [3] already use histories of candidates and scores for contextual improvement. More directly, Monea et al. [2] convert numerical rewards into natural-language correctness feedback for ICRL. Emotional and social-role prompting [4, 5], and competitive/cooperative framing in repeated-game decisions [6], already study behavioral changes induced by such language. The paper acknowledges several of these strands, but does not sufficiently establish what generalizable insight its package comparisons add beyond their combination. Comparing equally binned feedback under a fixed scorer is a meaningful distinction; the present evidence leaves its scientific significance beyond another setting of prompt sensitivity unclear.
W2. The bundled intervention limits the explanatory value of the findings. Stage 1 changes both contextual scaffolding and reward resolution. Stage 2 jointly changes the feedback contract, state vocabulary, optimization-target wording, and output restrictions. Consequently, the comparisons do not distinguish reward rendering from altered behavioral instructions. The mentioned single-run component probes do not establish replicated component-level attribution. Although the authors explicitly adopt a package-level estimand, this leaves limited evidence about which property of reward interfaces explains the observed differences.
W3. ICRL-specific effects and generality remain insufficiently supported. Framing is present before the first feedback arrives and remains in later instructions. Early/late scores and divergent trajectories therefore do not separate static instruction effects from effects mediated by accumulated reward feedback; the history-free CoT reference is not a frame-matched control. Moreover, the evaluation uses only 100 items per task, two evidence-sufficiency tasks, and one alternating exploration–exploitation protocol. The prominent over-commitment findings are informative within this setting, but provide limited support for broader conclusions about reward-interface effects in ICRL.
[1] Reward Is Enough: LLMs Are In-Context Reinforcement Learners.
[2] LLMs Are In-Context Bandit Reinforcement Learners.
[3] Large Language Models as Optimizers.
[4] Large Language Models Understand and Can be Enhanced by Emotional Stimuli.
[5] When “A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models.
[6] Strategies of cooperation and defection in five large language models.
Questions for authors
- What specific empirical conclusion distinguishes this work from existing reward verbalization, history-and-score optimization, and framing studies, beyond applying these ideas together in this evaluation setting?
- Which existing controls isolate feedback-mediated effects from the static contract and objective wording? In particular, what do the B2/BL/B4/N0/Z0 diagnostics establish about this distinction?