- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether LLMs adapt in context to a hidden evaluator from collected PASS/FAIL feedback. Models steadily improve under real pass/fail feedback but not under random feedback. The experiments indicate that the models are learning what this particular evaluator wants, not simply getting better in general. Further experiments show that the models partly adjust when the evaluator changes without notice, figure out useful information about the evaluator, and carry the learned information over to new tasks.
Critical strengths
(1) The main experiments are carefully designed and cover the NULL control, exact-yoked replay, and specificity decomposition. (2) The REAL–NULL effect is replicated across four model families.
Critical weaknesses
(1) Insufficient analysis of the results.
- The yoked results of Gemini contradict the main conclusion, but there is no discussion. For Gemini, replay of the sequence explains much of the adaptation. It weakens the claim of relying on the action-outcome.
- No analysis of the mechanism. All apply reasoning, but there is no analysis of the trace or mechanism. For example, does the model explain the assumptions for FRONT and BACK?
- In Figure 4a, there are only source/negative and reconciliation/negative (2 of 6) transfers. I think the following analysis would contribute to recognizing the impact of different rules.
(2) Limited significance. In-context policy adaptation from feedback is well-known. The new element is evaluator specificity, but the setting makes it largely expected.
Questions for authors
(1) This framework can be viewed as an in-context bandit, e.g., Krishnamurthy et al. 2024 and Binz & Schulz 2023. However, there are no references to LLM in-context exploration or bandits. Can you compare and bridge the formulation with in-context bandits? What are the new elements beyond the frameworks?
(2) The paper already includes an open-weight model (Qwen 3.8-27B). Can the authors analyze the mechanism mentioned in Weakness 1? For example, run it locally and analyze full reasoning traces or log-probabilities over FRONT- vs. BACK-optimal orderings to evaluate whether the model tracks the evaluator explicitly?
(3) All hidden evaluators are synthetic rules. Does the adaptation persist with a realistic evaluator, e.g., LLM-judge prompts that review different evidence? Regret remains available by estimating judge pass probabilities offline. If not, should the oversight implications be limited to rule-based evaluators?