- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper seems to propose an evaluation of LLMs on physical prediction tasks. It appears to place emphasis on the ability of models to generalize from their context to other scenarios.
Critical strengths
The paper seem to deal with an important topic that is not a new question but is still relevant as more evidence can be gathered to better understand how to place LLMs on the scale of "memorization" vs "generalization".
Critical weaknesses
First weakness:
The writing in the paper is hard to follow. Many arguments lack justification or explanation, and many terms are used without being defined.
Some examples from the abstract:
- "run the full ladder" -- what is this ladder?
- "frontier arm"
Some examples from the intro:
- The second paragraph is hard to read. It packs the setup, the novelty ladder, the three hidden mechanisms, results (as numbers) and the "ideal observer" into one paragraph.
- In the intro's second paragraph, it is not clear at all what observations and explanations are referred to.
- Terms are often used before explaining what they are. For example, "arm" (first used in the abstract, defined only in Section 3) and "basic capability floor" (I couldn't understand it from the intro).
- Some results are reported in the second paragraph (under "best local patch", whose definition appears much later in the paper), but then later on results are enumerated again without it being clear if they relate to the first results.
- The task is described without any motivation for this specific task (both in the intro and in the abstract). It is not clear at all what setup was tested.
Those point are partial and are only about the abstract and the intro but the clarity of the rest of the paper is in similar or sometimes worse state.
Second weakness:
The novelty claim relative to prior work is overstated. Inferring hidden state from observed data with LLMs has been studied before. DiscoverPhysics (https://arxiv.org/abs/2605.26087) includes worlds with hidden structure that the model is not told about (dark_matter, three_species). The current setup seems to be covered by the claims in that paper.
Questions for authors
The paper is very hard to read to the point I had a difficult time assessing its contribution. The paper would benefit from professional language editing.
The authors comment did not address the critical issues I raised in a satisfactory manner and I choose to keep my score.
Arguments that lack justification and terms that are used without being defined are very common in the paper, the specific examples i gave are only a few out of many. A revision of the paper had not been uploaded at the time of writing this reply but even if such revision fixes to those specified issues I believe the paper will still be difficult to read and assess its contribution. Also, fixing only those will not bring the paper close to the level of clarity and communication expected of an ICLR paper. Given this, I choose not to raise my score.
I am not certain that I fully understand the novelty over DiscoverPhysics. Based on the explanation provided, it seems mild.