acceptodds
Under review as a conference paper at ICLR 2027

Learning the Evaluator: How LLMs Adapt to Hidden Evaluation Rules

16 views

Abstract

Large language models operate in review or compliance workflows where outputs are assessed over multiple interactions while the exact decision rule remains unknown. This can make direct adjustment harder at first, yet successive outcomes may reveal which outputs are more likely to succeed. This raises the question: can an LLM use repeated results to adapt specifically to the evaluator rather than simply improve with experience? We study this in a controlled sequential setting where the active rule remains hidden from the model but known to the experimenter, enabling exact scoring against the rule-specific optimum and comparison with a matched control. Across four model families, evaluator-dependent outcomes reduce regret relative to the active evaluator's optimum more than matched controls. On GPT and Gemini, behavior re-adapts after unannounced rule changes and information-sensitive actions emerge when they can improve later decisions; on Gemini, evaluator-specific adaptation also transfers beyond position-dependent evaluation. This paper contributes a study of evaluator-specific learning and shows that evaluation rules can become behaviorally learnable over time. The results show that hiding the active evaluator identity alone does not prevent repeated outcomes from becoming informative to an adaptive model, motivating evaluation designs that consider both detection quality and what interaction reveals over time.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, evaluator learnability, adaptive evaluation, hidden evaluators, sequential feedback, AI oversight, behavioral adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_1bZnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether LLMs adapt in context to a hidden evaluator from collected PASS/FAIL feedback. Models steadily improve under real pass/fail feedback but not under random feedback. The experiments indicate that the models are learning what this particular evaluator wants, not simply getting better in general. Further experiments show that the models partly adjust when the evaluator changes without notice, figure out useful information about the evaluator, and carry the learned information over to new tasks.

Critical strengths

(1) The main experiments are carefully designed and cover the NULL control, exact-yoked replay, and specificity decomposition. (2) The REAL–NULL effect is replicated across four model families.

Critical weaknesses

(1) Insufficient analysis of the results.

  • The yoked results of Gemini contradict the main conclusion, but there is no discussion. For Gemini, replay of the sequence explains much of the adaptation. It weakens the claim of relying on the action-outcome.
  • No analysis of the mechanism. All apply reasoning, but there is no analysis of the trace or mechanism. For example, does the model explain the assumptions for FRONT and BACK?
  • In Figure 4a, there are only source/negative and reconciliation/negative (2 of 6) transfers. I think the following analysis would contribute to recognizing the impact of different rules.

(2) Limited significance. In-context policy adaptation from feedback is well-known. The new element is evaluator specificity, but the setting makes it largely expected.

Questions for authors

(1) This framework can be viewed as an in-context bandit, e.g., Krishnamurthy et al. 2024 and Binz & Schulz 2023. However, there are no references to LLM in-context exploration or bandits. Can you compare and bridge the formulation with in-context bandits? What are the new elements beyond the frameworks?

(2) The paper already includes an open-weight model (Qwen 3.8-27B). Can the authors analyze the mechanism mentioned in Weakness 1? For example, run it locally and analyze full reasoning traces or log-probabilities over FRONT- vs. BACK-optimal orderings to evaluate whether the model tracks the evaluator explicitly?

(3) All hidden evaluators are synthetic rules. Does the adaptation persist with a realistic evaluator, e.g., LLM-judge prompts that review different evidence? Regret remains available by estimating judge pass probabilities offline. If not, should the oversight implications be limited to rule-based evaluators?

Reviewer_Lq1XOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates if LLMs use repeated feedback to adapt to the evaluator judging their outputs. The family of evaluator rules are disclosed to the LLMs, but the active rule is hidden. Controlled tasks allow measuring the quality of the LLM decisions under each evaluator and its distance from the best possible decision.

The paper finds larger evaluator-specific improvement with informative feedback than with control feedback across 4 model families. Additional experiments investigate adaptation after an unannounced evaluator change, selection of a costly informative action, and if findings transfer to a different action and evaluator structure.

The main contribution is a controlled method for separating evaluator-specific adaptation from general improvement. Results support this behavioural claim within the tested method.

Critical strengths

The experimental design provides clear evidence of evaluator-specific adaptation. Scoring the same action with opposing evaluators show improvement favours the active evaluator rather than improving performance generally. The replay experiments strengthen this result: receiving the same feedback sequence produces little adaptation when that sequence is disconnected from the recipient’s own actions. These findings make the central finding more convincing than a simple increase in score over sequential feedback rounds.

Critical weaknesses

Experiments establish evaluator-specific adaptation in highly structured settings with small, disclosed rule families. In the primary experiment, models are told the possible evaluator rules, so the findings do not establish whether adaptation would occur without this guidance. In the probe experiment, models choose between prepared initial submissions. Every case has the same optimal choice pattern of probing only when the horizon is long and the evaluator persists. This supports sensitivity to the future usefulness of information, but does not test if models can construct their own informative probe. Although the author acknowledges these boundaries, they limit what the findings establish about adaptation when the possible evaluator rules and useful probing actions are not supplied.

Questions for authors
  1. How much do evaluator-specific adaptations depend on explicitly providing the evaluator rules? Does the model still improve under the active evaluator when these descriptions are omitted, while keeping the task, evaluator and feedback process the same?

  2. What is the main novelty compared with previous research on learning from feedback? Is it counterfactual scoring and controls, new findings about adaptation to hidden evaluators, or both?

Reviewer_dbGbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The core issue explored in this article is whether LLMs can gradually figure out a hidden evaluator's preferences in their behavior solely through repeatedly receiving PASS/FAIL feedback. Previous research has shown that feedback or supervision can change model behavior, but it is difficult to distinguish whether this is a general improvement in performance or adaptation to specific rules. In a controlled environment, the authors informed the model of candidate rules but hid the specific rule currently in effect, and compared real feedback with matched control conditions.

I think the main contributions of the paper are threefold: first, it clearly identifies evaluator-specific adaptation, distinguishing it from general experiential accumulation; second, it demonstrates the model's ability to readapt after an unannounced evaluator switch and its sensitivity to the subsequent informational value; third, it shows that this phenomenon can transfer to new instances governed by feature-dependent rules.

In terms of validation, the authors were relatively thorough. They mainly used the REAL/NULL contrast, precise modeling of evaluator-specific regret and counterfactual decomposition, and the YOKED (outcome replay) control that breaks action-outcome associations. The results show a trend across four model families of the model aligning with the current evaluator's preferences. Overall, the paper aims to show that repeated evaluation itself may become an information channel, allowing the model to accumulate information about hidden rules from feedback, thereby influencing subsequent decisions, without overstating internal representations or risks in real-world deployment.

Critical strengths
  1. The paper carefully distinguishes between evaluator-specific adaptation and general performance improvement. Precise regret metrics and counterfactual scoring help determine whether the model has actually become better suited to the current evaluator, rather than just performing better under both evaluators. The REAL/NULL comparison provides the main evidence for this distinction, making the paper's core conclusion clearer than just observing that model performance improves over time.

  2. Additional controls and experiments boost the credibility of the findings. In particular, the YOKED experiment shows that simply replaying the same PASS/FAIL sequences isn't enough to replicate the adaptation effects seen under real feedback conditions. The main phenomena also show up across four model families. Together, these results provide strong experimental support for the paper's key findings under controlled conditions.

Critical weaknesses
  1. Controlled experiments make it possible to measure how evaluators adapt specifically, but they also limit how widely the findings can be applied. The model already knows which rule family the candidate rules belong to, so it only needs to figure out which rule is currently active. It's still unclear whether similar adaptation would happen when the evaluation criteria are unknown or much more complex. This doesn't contradict the existing results, but it does limit how far they can be generalized to a more open evaluation setting.

  2. Feature-based transfer experiments provide evidence that the phenomenon can go beyond positional rules, but this experiment only used 48 experimental units on Gemini. That's enough to suggest that such transfer might happen, but not enough to determine how broadly it could apply to other models or tasks. So, compared to structural transfer, the paper provides stronger support for the main phenomenon of evaluator-specific adaptation.

Questions for authors
  1. The REAL/NULL comparison and the YOKED experiment address different alternative explanations. Could the authors clarify what additional conclusion the YOKED result supports beyond the primary REAL/NULL comparison, given that donor outcomes may still contain information about the evaluator?

  2. The feature-dependent transfer experiment shows an effect on Gemini with 48 experimental units. What aspects of this result do the authors expect to generalize to other models or task settings, and which aspects may be specific to the current experiment?

  3. After an unannounced evaluator switch, regret decreases but does not return to its earlier level. Do the authors attribute this mainly to the short adaptation horizon, or is there evidence that behavior shaped by the previous evaluator persists after the switch?

Reviewer_6abXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work studies whether LLMs can adapt to a hidden evaluator through repeated PASS/FAIL feedback. In the main task, the model selects and orders claims. The evaluator checks two claims and rejects the submission if either is unsupported. The model knows that the evaluator favors either earlier or later positions but does not know which rule is active. The authors measure regret against the exact evaluator-specific optimum. Informative feedback reduces regret more than action-independent feedback across four model families.

Critical strengths
  • The authors compute each output's expected utility under both evaluators and compare it with the true optimum. I find this more informative than reporting PASS rates alone. It distinguishs adaptation to the active evaluator from improvement that benefits both evaluators.
  • REAL and YOKED receive identical PASS/FAIL sequences, but only REAL improves appreciably on GPT. This supports the claim that adaptation depends on the relationship between the model's own choices and their outcomes. The outcome sequence alone does not explain the result.
  • Informative feedback produces larger regret reductions than NULL feedback on GPT, Gemini, DeepSeek and Qwen. The estimated effects range from approximately 0.14 to 0.21. This reduces concern that the result reflects one model's behavior.
Critical weaknesses
  • The model receives descriptions of the two possible rules. It therefore learns which of two known mechanisms is active, not an unknown evaluation rule. This is a valid controlled setting, but I am less convinced that the result adds enough beyond existing work on learning from sequential feedback. A larger or undisclosed rule family would clarify the contribution.
  • The probe result does not establish fine-grained sensitivity to information value. The prompt states the horizon and whether the evaluator persists. Simple rules can produce the observed pattern.
  • The paper discusses reward hacking, monitoring and strategic classification. It should compare more directly with work on LLMs learning from sequential rewards and choosing exploratory actions. I see the exact evaluator-specific measurements and controls as the main contribution. The paper should explain what they establish beyond general adaptation to feedback.
  • The task setting chosen carries several protocols which make it unnecessarily complex to understand. I had to go back and forth to the appendix mltiple times for details.
  • The writing of the paper needs improvement. Please improve the clarity. There are many instances of abruptly or inadequately mentioning concepts.
Questions for authors
  • Does adaptation persist when the candidate rules are undisclosed or the family contains more rules?
  • Does probing respond appropriately to changes in cost and information gain including cases where an informative probe is not worth the cost?
  • Does the model recover more fully after a switch with additional rounds or a shorter visible history?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.