- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes to measure, on verifiable tasks, whether in-context learning has an effect on LLMs’ judgment. The paper employs a financial environment that mimics software subscriptions, with companies that belong to one of two categories (healthy, unhealthy). A model is asked to guess the category. The paper concludes that k-shots have a much larger influence than weights over a model prediction.
Critical strengths
I did my best to find some strengths in this paper, but I hardly found any. The paper seems clearly written by Claude and it took me a while to understand what it is about. In my opinion, the motivation and novelty are limited and experiments are far too weak to positively contribute and reach the bar of this conference.
Critical weaknesses
(major) This article is pretty much unreadable and looks like a technical report rather than a paper. The authors do not explain the task well, do not introduce any notation to explain what it means for an answer to live in the weights/context. The task is introduced very approximately and no example in the main paper is reported, making it hard to even understand what is the point of the paper. The paper immediately rushed into the results with no context. There are several issues one can spot even without understanding the task: for example, the dataset is unbalanced and no good reason justifies this choice. Also, negative results like Fable’s denial are reported as some evidence towards the thesis of the paper, but they should just be reported for what they are, i.e., that something triggered the model’s guardrails. I believe that the authors should take a step back and keep in mind that it’s their job to make an article understandable, not that of the reviewers’ to untangle the positive bits.
(major) Sentences like “The environment is the setting of this paper, not its contribution.” read very much like Claude (short, cryptic sentences with a negation of an argument no one has asked for; also, the environment is hardly a contribution as it is never properly introduced or described, see the example in the Appendix). Other examples of Claude’s beige prose appear at lines 54, 148, 200, 215, etc. Also, as an experienced Claude user, I can say that putting a lot of numerical results (with little or no context) in the abstract is something Claude does quite often, besides showing that the authors did not spend time trying to come up with a coherent narrative that embraces and summarises the results.
In the current state, I recommend a rejection.
Questions for authors
Q1. After reading the example of a task in the Appendix, I wouldn’t be able to solve it. Did the authors check that the tasks are intelligible to humans; if so, how?
Feedback accepted
Thank you. We accept the main criticism. The task was not explained well enough, and a reader should not have to reconstruct it from the appendix. A future version would introduce the task with a worked example before any result, define what the central claim means, state the reason for the class split (the healthy class carries the decoy cohorts that make the task hard), and report Fable's refusals as refusals rather than as evidence.
On your question about whether the tasks are intelligible to humans, a handful of people have completed them in a demonstration setting. The difficulty you met is a fault of the paper's explanation, not of the task, and that is on us.
On the central claim, a pre-registered test reported to Reviewer 1PHP did not fully support the paper's interpretation. A future version may narrow the claim and report that test at more seeds.
AI assistance was disclosed at submission. We are not uploading a revised PDF in this cycle.