acceptodds
Under review as a conference paper at ICLR 2027

SAME SCORE, DIFFERENT OUTCOMES: REWARD INTERFACES SHAPE IN-CONTEXT REINFORCEMENT LEARNING

Abstract

Large language models can adapt across interaction rounds from feedback retained in context without updating their parameters. However, existing studies typically treat reward as a scalar signal, leaving open whether the language interface through which that signal enters the context affects multi-round adaptation. To study this question, we develop a two-stage evaluation framework based on reward-interface presentation packages, which map an underlying scalar reward into contextualized language feedback. Stage 1 examines semantic enrichment, asking whether adding an explicit feedback contract and qualitative state descriptors to a scalar reward changes adaptation. Stage 2 examines rhetorical presentation framing, holding the underlying reward and semantic scaffold structure fixed while expressing the feedback through different semantic frames: care-, competition-, survival-, trust-, and navigation-oriented frames. We conduct systematic multi-round comparisons across 16,800 twenty-round episodes on two deterministic tasks, FEVER fact verification and SQuAD 2.0 extractive question answering, using GPT-5.4-mini, Qwen3-8B, and Gemma-4-E4B-it. The results show that the effect of semantic enrichment depends on the task and model: it increases late FEVER reward by 0.474 points for GPT-5.4-mini and 0.909 points for Qwen3-8B, but provides limited or mixed benefit on SQuAD 2.0. Even with the underlying reward and scaffold structure held fixed, different rhetorical frames produce distinct multi-round adaptation trajectories and endpoint differences of up to 1.251 points. These findings suggest that the linguistic presentation of reward participates in shaping model behavior during ICRL, rather than serving merely as a textual format for transmitting a scalar signal.

Primary area:
reinforcement learning
Keywords:
In-Context Reinforcement Learning, Reward Interface, Semantic Enrichment, Presentation Framing, Multi-Turn Prompting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KBgEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how reward-interface presentation affects multi-round in-context reinforcement learning with frozen language models. It compares numeric and structured qualitative feedback, then five rhetorical frames against a neutral interface, using fixed deterministic scorers. Experiments cover three models, FEVER and SQuAD 2.0, and 16,800 twenty-round episodes. Structured feedback helps in some model–task settings, while several framed packages reduce late reward and increase unsupported commitments. The main contribution is an empirical characterization of complete interface packages.

Critical strengths
  1. Fixed deterministic scorers, common evaluation items, repeated runs, and paired analyses with multiplicity correction make the package-level endpoint comparisons reasonably convincing.
  2. Separating late reward from adaptation gain, and examining unsupported commitments and error transitions, provides a more informative account than endpoint scores alone. The paper appropriately treats transition and attention analyses as descriptive.
Critical weaknesses

W1. The boundary with prior work and the substantive contribution are insufficiently established. Reward Is Enough [1] and OPRO [3] already use histories of candidates and scores for contextual improvement. More directly, Monea et al. [2] convert numerical rewards into natural-language correctness feedback for ICRL. Emotional and social-role prompting [4, 5], and competitive/cooperative framing in repeated-game decisions [6], already study behavioral changes induced by such language. The paper acknowledges several of these strands, but does not sufficiently establish what generalizable insight its package comparisons add beyond their combination. Comparing equally binned feedback under a fixed scorer is a meaningful distinction; the present evidence leaves its scientific significance beyond another setting of prompt sensitivity unclear.

W2. The bundled intervention limits the explanatory value of the findings. Stage 1 changes both contextual scaffolding and reward resolution. Stage 2 jointly changes the feedback contract, state vocabulary, optimization-target wording, and output restrictions. Consequently, the comparisons do not distinguish reward rendering from altered behavioral instructions. The mentioned single-run component probes do not establish replicated component-level attribution. Although the authors explicitly adopt a package-level estimand, this leaves limited evidence about which property of reward interfaces explains the observed differences.

W3. ICRL-specific effects and generality remain insufficiently supported. Framing is present before the first feedback arrives and remains in later instructions. Early/late scores and divergent trajectories therefore do not separate static instruction effects from effects mediated by accumulated reward feedback; the history-free CoT reference is not a frame-matched control. Moreover, the evaluation uses only 100 items per task, two evidence-sufficiency tasks, and one alternating exploration–exploitation protocol. The prominent over-commitment findings are informative within this setting, but provide limited support for broader conclusions about reward-interface effects in ICRL.

[1] Reward Is Enough: LLMs Are In-Context Reinforcement Learners.

[2] LLMs Are In-Context Bandit Reinforcement Learners.

[3] Large Language Models as Optimizers.

[4] Large Language Models Understand and Can be Enhanced by Emotional Stimuli.

[5] When “A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models.

[6] Strategies of cooperation and defection in five large language models.

Questions for authors
  1. What specific empirical conclusion distinguishes this work from existing reward verbalization, history-and-score optimization, and framing studies, beyond applying these ideas together in this evaluation setting?
  2. Which existing controls isolate feedback-mediated effects from the static contract and objective wording? In particular, what do the B2/BL/B4/N0/Z0 diagnostics establish about this distinction?
Reviewer_XMhAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the wording of a deterministic reward changes multi-round in-context reinforcement learning, where a model reattempts one item with its earlier answers and feedback in context. The treatment is a complete presentation package (contract, level words, objective wording, output restriction). Stage 1 compares a numeric reward (B1) with a contract plus qualitative levels (B3); Stage 2 rewrites B3 in several rhetorical frames with the same scorer and bins. Experiments use FEVER and SQuAD 2.0, three models, three runs per condition and item-paired, Holm-corrected tests on late reward. B3 significantly helps only on FEVER for two models; no frame significantly exceeds B3, and the largest loss (1.251 points, GPT-5.4-mini on SQuAD) comes with more answers to unanswerable questions.

Critical strengths
  1. A careful, transparent package-level design. A shared deterministic scorer gives the same response the same reward under every package, every prompt block is disclosed, and contrasts are item-paired with Holm correction. This makes the large loss from changing only the package credible, and a useful caution for anyone comparing in-context RL methods.

  2. The behavioral analyses show where the losses come from. On GPT-5.4-mini SQuAD the weaker packages answer more unanswerable questions and repeat wrong decisions more often, most clearly under the fuel frame. B3 also starts below B1 there but ends above it, which justifies reporting late reward and within-episode gain together.

Critical weaknesses
  1. The evidence supports package-level effects, but not yet the claim that feedback wording changes how models adapt. The title and abstract credit the effects to how the reward is presented, "holding the underlying reward and semantic scaffold structure fixed," yet each package also changes static text the model sees every round, such as the contract, objective wording and round instructions, and that text could explain the gaps just as well. The controls the paper names for probing these components are never defined or reported. The trajectory claim runs ahead of the evidence in the same way: on FEVER most significant gaps against B3 are already present in the early rounds, more a level shift than diverging adaptation; only GPT-5.4-mini on SQuAD clearly diverges, and gain differences are never tested. As it stands, the paper shows that packages shift endpoint reward, a narrower result than its title claims.

  2. The robustness and scope of the evidence are limited. As the paper states, inference is conditional on three retained runs, so tests generalize over items rather than reruns, and no same-configuration noise estimate is given, although the two stages' GPT-5.4-mini B3 runs would supply one. Nearly all significant Stage 2 contrasts fall in three cells. Package effects carry limited practical weight: in half the cells a reference method beats every feedback package on late reward, which the paper notes only for Gemma. The discussion should engage Krishnamurthy et al. (NeurIPS 2024), who found that only GPT-4 with chain-of-thought and an externally summarized history explored satisfactorily in bandits, and Schubert et al. (ICML 2024), who showed that learning asymmetry reverses with counterfactual feedback and disappears without implied agency. Without a noise estimate, Stage 2 differences of about 0.3 points may lie within replicate noise and should be read as provisional.

Questions for authors
  1. Could you define the B2, BL, B4, N0 and Z0 controls and report their results, together with any Stage 2 variant you ran that changes only the level words?
  2. Could you report paired confidence intervals for gain differences against B3, so that real divergence can be separated from an early offset?
  3. How does the gap between the Stage 1 and Stage 2 GPT-5.4-mini B3 runs compare with the smaller significant Stage 2 effects?
  4. Qwen3-8B with CoT has identical early, late and all-round means on both tasks (Tables D.1 and D.2), although CoT is history-free and decoding is described as stochastic. How were Qwen3-8B and Gemma-4-E4B-it served, including sampling, caching and Qwen3's thinking mode?
Reviewer_YwCnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores how LLMs can adapt across multiple interaction rounds using feedback in context, without changing the parameters of the model. Specifically, the authors study the effect of presenting scalar rewards through natural language on the iterative adaptation of LLMs, building on previous work, Reward Is Enough (Song et al., 2026). They develop a two-stage evaluation framework. In Stage 1, they compare numerical rewards with structured, qualitative feedback that includes an explicit feedback contract and converts the scalar reward into five qualitative levels. In Stage 2, they change the linguistic framing, such as care, survival, competition, trust, and navigation, while keeping the underlying reward function and reward bins fixed. Through experiments, they show that the linguistic presentation of reward affects the model's behavior. However, the results are not consistent across models and tasks. For example, structured feedback significantly improves late reward in two FEVER model-task settings, but the benefit does not extend uniformly to SQuAD 2.0.

Critical strengths
  • The paper explores an interesting research question. It extends prior work that studies how models can adapt using numerical rewards by exploring whether the natural-language presentation of these rewards also matters. This could be useful in agentic workflows, where it is important for LLMs to learn from their previous attempts without updating their parameters.

  • The paper seems to use careful statistical approaches to report the results. For example, the authors do not rely on single numbers, and they report confidence intervals and statistical significance after multiple-comparison correction. The experimental setup, prompts, scoring rules, and hyperparameters are also described in detail, which helps with reproducibility.

Critical weaknesses
  • The paper does not isolate the effects of different feedback components very well. For example, in Stage 1, the authors add an explicit feedback contract and also convert the numerical reward into qualitative feedback at the same time. Therefore, it is not clear whether the improvement comes from using qualitative reward levels, adding extra instructions to the model, or their combination. Similarly, in Stage 2, they change the reward wording, framing, and optimization instructions together. The authors acknowledge that these are package-level comparisons, but I think more ablation experiments are needed, where only one component is changed at a time. Otherwise, it is hard to understand what is the main reason behind the improvements and which parts of the approach actually help the model.

  • The paper is difficult to read. The text is very number-heavy, and the tables and figure captions could benefit from a sentence explaining the main takeaway of each table or figure. There are also some other issues. For example, in Section 3.2, the authors mention the sentence "Single-replicate controls B2, BL, B4, N0, and Z0 probe interface components separately but remain diagnostic." However, these conditions are not defined beforehand. Also, the submitted PDF does not show line numbers, which makes it harder to refer to specific parts of the paper.

Questions for authors
  • Why did the authors not explore larger models? The paper mainly uses relatively small or efficient model variants. At least one larger model would help evaluate the generalizability of the conclusions, especially since the observed behavior is inconsistent across models and tasks.

  • The paper only discusses the effect of reward presentation on two tasks, FEVER and SQuAD 2.0. Do these findings generalize to other tasks, such as coding, mathematics, or more complex agentic tasks?

  • Did the authors try comparing numerical rewards and qualitative natural-language rewards while keeping the feedback contract and adaptation instructions identical? For example, could a Reflexion-style instruction be applied to both reward formats? This would help understand whether the improvement comes from the reward presentation itself or from the additional instructions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.