acceptodds
Under review as a conference paper at ICLR 2027

Fine-Tuned Judgment Lives in the Weights: In-Context Examples Help Untrained Models and Override Trained Ones

Abstract

In many professional judgment tasks, answers are hidden, deferred, or counterfactual, so no automated verifier exists. We study such tasks using an environment that plants the truth and holds the answer key. We ask where a model's judgment lives: in its context or in its weights. The environment generates companies with customer cohort-level financials. Unhealthy companies hide a revenue contraction in one cohort, and most healthy companies carry a decoy cohort that looks like a contraction and is not. We evaluate an untrained Qwen 7B, Claude Fable 5 and Claude Sonnet 4.6 as of August 2026, and the same 7B fine-tuned on its own attempts filtered by the answer key, on 550 companies, 150 unhealthy and 400 healthy, under one parser and one grader. Each model runs zero-shot and with 2 in-context examples: one company solved with reasoning and a verdict, shown once with the planted flaw and once without. Fable 5 refused to emit verdicts whenever examples were present, across 3 prompt formats, leaving 7 conditions. In the zero-shot condition, every untrained model detects nearly every hidden contraction but produces false positives on most healthy companies. Fable 5 detects 0.987 with a false-positive rate of 0.812, Sonnet 4.6 detects 1.000 at 0.777, and the untrained 7B detects 0.913 at 0.833. Sampled transcripts show both frontier models compute that the decoy's revenue is intact, state it, and call the company unhealthy anyway, a failure of judgment rather than analysis. The 2 in-context examples mainly cut false positives, taking the untrained 7B to 0.807 at 0.302 and Sonnet 4.6 to 1.000 at 0.092. Fine-tuning moves the 7B to 0.960 at 0.247 zero-shot, higher detection than its untrained base with less than a third of its false positives. The same 2 examples collapse it to 0.187 at 0.030. It repeats the healthy example's reasoning on unhealthy companies. In-context examples provide judgment reasoning to models that lack it and override the model that has it. We take the reversal as evidence that the fine-tuned model's judgment lives in its weights, not its context. Limitations: one environment and one planted flaw type. A fine-tuned Mistral 7B shows the same override with the collapse in false positives rather than in detection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, fine-tuning, few-shot prompting, large language models, evaluation, LLM judgment, prompt-induced interference, synthetic environments

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_y4eXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to measure, on verifiable tasks, whether in-context learning has an effect on LLMs’ judgment. The paper employs a financial environment that mimics software subscriptions, with companies that belong to one of two categories (healthy, unhealthy). A model is asked to guess the category. The paper concludes that k-shots have a much larger influence than weights over a model prediction.

Critical strengths

I did my best to find some strengths in this paper, but I hardly found any. The paper seems clearly written by Claude and it took me a while to understand what it is about. In my opinion, the motivation and novelty are limited and experiments are far too weak to positively contribute and reach the bar of this conference.

Critical weaknesses

(major) This article is pretty much unreadable and looks like a technical report rather than a paper. The authors do not explain the task well, do not introduce any notation to explain what it means for an answer to live in the weights/context. The task is introduced very approximately and no example in the main paper is reported, making it hard to even understand what is the point of the paper. The paper immediately rushed into the results with no context. There are several issues one can spot even without understanding the task: for example, the dataset is unbalanced and no good reason justifies this choice. Also, negative results like Fable’s denial are reported as some evidence towards the thesis of the paper, but they should just be reported for what they are, i.e., that something triggered the model’s guardrails. I believe that the authors should take a step back and keep in mind that it’s their job to make an article understandable, not that of the reviewers’ to untangle the positive bits.

(major) Sentences like “The environment is the setting of this paper, not its contribution.” read very much like Claude (short, cryptic sentences with a negation of an argument no one has asked for; also, the environment is hardly a contribution as it is never properly introduced or described, see the example in the Appendix). Other examples of Claude’s beige prose appear at lines 54, 148, 200, 215, etc. Also, as an experienced Claude user, I can say that putting a lot of numerical results (with little or no context) in the abstract is something Claude does quite often, besides showing that the authors did not spend time trying to come up with a coherent narrative that embraces and summarises the results.

In the current state, I recommend a rejection.

Questions for authors

Q1. After reading the example of a task in the Appendix, I wouldn’t be able to solve it. Did the authors check that the tasks are intelligible to humans; if so, how?

1 comment
AuthorsOpenReview ↗

Feedback accepted

Thank you. We accept the main criticism. The task was not explained well enough, and a reader should not have to reconstruct it from the appendix. A future version would introduce the task with a worked example before any result, define what the central claim means, state the reason for the class split (the healthy class carries the decoy cohorts that make the task hard), and report Fable's refusals as refusals rather than as evidence.

On your question about whether the tasks are intelligible to humans, a handful of people have completed them in a demonstration setting. The difficulty you met is a fault of the paper's explanation, not of the task, and that is on us.

On the central claim, a pre-registered test reported to Reviewer 1PHP did not fully support the paper's interpretation. A future version may narrow the claim and report that test at more seeds.

AI assistance was disclosed at submission. We are not uploading a revised PDF in this cycle.

Reviewer_1PHPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how task-specific fine-tuning interacts with in-context demonstrations on a synthetic company-analysis task. The same demonstrations improve models without task-specific fine-tuning but degrade fine-tuned models. The authors interpret this as demonstrations overriding judgment learned in the weights.

Critical strengths

The contrast between the two groups of models is interesting and worth examining. The task provides explicit ground truth, and the shared grading procedure makes the comparison easier to interpret. I also appreciate the authors’ discussion of limitations and variability across training seeds.

Critical weaknesses

My main concern is the gap between the experiments and the claims in the title and abstract. The evidence comes from one synthetic, rule-governed task with a small training set and a fixed demonstration pair. These results establish a substantial performance drop in this setting. I would be more cautious about drawing general conclusions about where judgment resides or whether demonstrations override fine-tuned judgment. The prompt mismatch also leaves the proposed interpretation unresolved. Fine-tuning uses zero-shot prompts, whereas evaluation adds demonstrations that change both the input format and the available task information. I appreciate that the authors acknowledge this alternative. However, copying example-specific outputs does not by itself distinguish judgment override from sensitivity to this train–test mismatch. The current results leave both explanations plausible.

Questions for authors

Would the effect persist if fine-tuning also included demonstration-based prompts? I would find this comparison useful for separating the proposed interpretation from prompt distribution shift.

1 comment
AuthorsOpenReview ↗

Your proposed test was run. The effect does not persist.

Thank you. You asked whether the effect would persist if fine-tuning also included demonstration-based prompts. We ran that test on both architectures, Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3, pre-registered before any output was seen.

Design. For each architecture we trained 2 new adapters on the same training examples, recipe and seed as the paper's fine-tuned Qwen, both in the evaluation instruction and answer format. One adapter, which we call No-Demos, was trained without the 2 demonstrations. The other, With-Demos, was trained with the 2 demonstrations placed before the company table in all of its training examples. Each was evaluated zero-shot and with the demonstrations on the paper's 550 companies.

Result (detection minus false positives). Qwen No-Demos scored 0.985 zero-shot and 0.944 with the demonstrations. Qwen With-Demos scored 0.590 zero-shot and 0.993 with the demonstrations. Mistral No-Demos scored 0.983 zero-shot and 0.375 with the demonstrations. Mistral With-Demos scored 0.328 zero-shot and 1.000 with the demonstrations. Both With-Demos adapters beat No-Demos with the demonstrations (p 1e-5 and 1e-75). One seed per architecture.

Reading. For Qwen, matching the training instruction to the evaluation instruction removed most of the collapse on its own (0.634 in the paper's setting against 0.041 here), and training with the demonstrations removed the rest. For Mistral, matching the instruction removed little of the collapse (0.995 in the paper's setting against 0.608 here), and training with the demonstrations removed all of it. In both cases a model fine-tuned with the demonstrations is not degraded by them, and the collapse is train-test prompt mismatch, in both directions, since the With-Demos adapters fell to 0.590 and 0.328 when evaluated zero-shot. We agree that the paper's interpretation is not fully supported. The demonstrations help untrained models and override models trained without them, but they do not override models trained with them, so we will not defend the title's general claim.

We are not uploading a revised PDF in this cycle. A future version may report this experiment at more seeds under a narrowed claim.

Reviewer_yo5dOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether professional judgment in language models is supplied by in-context examples or learned through fine-tuning, using a synthetic financial-analysis task with exact ground-truth labels. It finds that demonstrations substantially improve untrained models by reducing false positives, while fine-tuning produces strong zero-shot judgment but can be disrupted by the same demonstrations, causing models to copy example verdicts rather than apply the learned decision rule. The authors interpret this interaction as evidence that fine-tuned judgment is encoded in model weights and argue that fine-tuned models should be evaluated in the prompt format used during training.

Critical strengths

The finding that the same demonstrations help untrained models but degrade fine-tuned ones is non-obvious and practically relevant for how fine-tuned models should be evaluated and prompted.

Critical weaknesses
  • This submission appears to be made with very little care and consideration in numerous aspects. For example,
    • Mistral 7B model is used for experiment, but not introduced in the model list in the introduction (L59)
    • No good introduction to the synthetic task used. There is no reason specified on why the authors had to use this task, beyond existing datasets. There is no comparison of this benchmark to existing ones
    • Evaluated models are too narrow, only 4 models
    • Fine-tuning dataset size is too small, only “28 of its own attempts filtered by the answer key” (L60)
Questions for authors

N/A. I have no specific questions that would materially affect my evaluation.

1 comment
AuthorsOpenReview ↗

Corrections accepted

Thank you. Mistral 7B should have been in the model list. The task should have been introduced before the results, with the reason for a synthetic task with planted answers. We accept both. The model count and training set size are real limits. A pre-registered test reported to Reviewer 1PHP did not fully support the paper's central interpretation. A future version may narrow the claim and report that test at more seeds. We are not uploading a revised PDF in this cycle.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.