- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
It introduces four benchmarks for recovering cellular automaton rules and noise parameters, evaluates LLMs, and proposes specialised transformers. The benchmark is potentially useful, but the evaluation protocol and overstated architectural conclusions prevent me from recommending acceptance.
Critical strengths
The controlled setting provides exact ground truth and supports analysis of identifiability. The specialised models perform strongly, and the algorithmic baselines help establish task difficulty.
Critical weaknesses
-
It's such a disaster to read the whole paper, given the 117 pages. I highly suspect that most of them are generated by AI because it's not really readable for humans because of its verbose, repetitive, and defensive assertions. Especially for Figure 6, I retain high suspicion that it was completely generated by AI, which is not disclosed by the authors.
-
The claimed relevance to natural systems remains speculative. Experiments assume known noise families, fully observed binary states, and synthetic one-dimensional dynamics. No real-system validation supports the broader framing.
-
The central architectural conclusion exceeds the evidence. Limited LoRA adaptation is compared with extensive task-specific training, structured tokenisation, and mechanism-aware estimators. This does not isolate architecture as the cause of the performance gap.
-
The test set is used for model selection. The paper explicitly selects checkpoints using repeated evaluation on the final test set. Disjoint training and test rules do not eliminate selection bias. The claim that online data generation makes an independent validation set impossible is untenable.
Questions for authors
-
What justifies calling these results a held-out evaluation when the test set repeatedly determines checkpoint selection? Please report results on an untouched test set.
-
What experiment isolates architecture from training budget, input representation, and analytical assistance? Without such a control, why claim a fundamental architectural limitation?
-
What evidence supports transfer beyond this narrowly specified synthetic setting?