- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper asks whether LLMs solve familiar numerical optimisation problems by recalling published answers rather than reasoning from the samples given. They test released benchmarks, fresh samples, and perturbed functions. They track how often models return the canonical optimum as perturbation grows, and compare against surrogate fits. Extra experiments cover reasoning effort, model generations, sample draws, and code execution.
I find the central observation convincing: models often return familiar benchmark answers, and sometimes continue doing so after the underlying function has changed. This is a useful warning for evaluating numerical capability. My concerns lie in the stronger interpretation of this behaviour and in whether the scoring establishes that the particular answer returned was worse than the surrogate estimate.
Critical strengths
The question is worthwhile and the effort is substantial. Fresh samples, opaque identifiers, perturbation seeds, and the librarian reference give several complementary probes. The continuous perturbation ladder is a useful addition over earlier transformation-based work.
I appreciate the recognition that authors note a small perturbation may make the canonical answer still rational. Comparing against a fitted surrogate is a sensible way to separate useful priors from harmful anchoring. Testing several surrogate floors helps. The authors are open about limitations: tool-arm failures, surrogate-floor sensitivity, low power of the matched comparison.
The fresh-draw experiment is also valuable. Recall rates can vary considerably even when the function and sample budget remain fixed. That finding matters for anyone constructing small numerical benchmarks.
Critical weaknesses
"Recall" is defined to include recognising a closed form and computing its optimum. But these are different operations. Retrieving a stored constant, identifying a function family, and computing a consequence from that family are distinct mechanisms. The selected surrogates fail to hit the near-exact threshold. That separates the result from those surrogates. It does not show no fitting method could recover the answer. Identifiability depends on the hypothesis class. The evidence supports strong dependence on familiar benchmark structure. Claims about the internal mechanism, or about the absence of numerical reasoning, need more qualification. A control using random input translations or rescalings could help distinguish reuse of an answer constant from recognition followed by calculation.
Section 3 counts canonical emission if either location or value matches. Appendix A.5 labels a cell pathological if either canonical location or value loses to the GP floor. These two disjunctions don't align. A model could match the canonical value on a cell labelled pathological solely because the canonical location is poor. Counting that session does not show the reproduced component was worse. The components are being conflated. Also: the floor compares canonical minimum-value error with GP optimisation regret. Those are different quantities. Regret at a point is not the same as predicted-minimum error.
A large number of sessions describe behaviour conditional on the tested prompts. But generalisation rests on a small set of base functions, perturbation fields, and sample draws. The redraw results make this clear: emission rates go from 0% to 100%, and surrogate-floor classification flips with the draw. The authors note this, but it directly affects the uncertainty on the main claims. Also, perturbation seeds reused across ladder rungs introduce shared structure --- a violation of the independence assumption behind naive error bars. The authors should report uncertainty that clusters by perturbation field and base function, not per-cell. Replicating sample draws in the central harmful-anchoring comparison would be especially useful.
Questions for authors
Do the harmful-emission rates remain similar when location and value are scored separately against matching losses?
What happens to the surrogate near-exact rates after continuous refinement?
How are multiple global minima handled in the location metric?
Do the main conclusions survive clustering by perturbation field and replication over fresh sample draws?
Can a simple transformed-function control separate remembered constants from recognition followed by calculation?
2 comments
Thank you for the reassessment. We appreciate it and agree with your remaining point: the 40% and 33% are rates on selected harmful draws, so they do not say how often a model returns a harmful location on a randomly drawn problem. The revision states this as a limitation, and measuring the unconditional rate on unselected draws is future work.
I see that the authors have basically addressed all of my comments. I've raised my score to weak accept. The current experiments select some harmful and some favrouable draws per function from a number of candidates. To answer "across randomly sampled problems, how often does a model return a harmful canonical location?" The way they are selected may need some careful consideration in future work.