- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces NL-LAN, a training-free method that represents an agent's world model as natural-language text. A frozen LLM proposes candidate world models, scores each by how well it predicts observed transitions, and revises them using the prediction errors in a beam search. On ARC-AGI-3, this beats in-context learning at next-state prediction (82.2% vs. 52.4%). In online play it solves 24 levels, against 12 for the same agent without the induced world model. The gains are smaller on other benchmarks and come at 4–13× the token cost.
Critical strengths
- The method shows a large gain on ARC-AGI-3. It reaches 82.2% held-out next-state accuracy against 52.4% for the best in-context baseline, and 94.7% against at most 43.3% on transitions already in context (Table 1).
- The ablations point to search as the mechanism. One-shot induction (NL-LAN-Low) gives limited gains, while refinement drives most of the improvement (ARC 30.5 → 82.2, Sequence Hard 15.0 → 73.0; Table 2), and Fig. 3 shows that more revisions also reduce variance. This matters because it ties the gains to a specific algorithmic component rather than to prompting a strong model.
Critical weaknesses
- The control result rests largely on one game and one run. ar25 accounts for about 53% of NL-LAN’s total ARC-AGI-3 score (Table 12). Without it, the average falls from 3.10 to roughly 1.46, against 0.97 for the no-induction variant. The no-induction variant also wins on six games, and no seeds or error bars are reported.
- I think the control baselines are not comparable. The ICL scores come from the public leaderboard, so they might use a different harness, prompts, and budget. NL-LAN’s own scaffold without induction already beats every one of them. This matters because the comparison against frontier agents mixes the effect of the scaffold with the effect of induction, and only the ablation within the paper is controlled.
- There is no compute-matched baseline. NL-LAN spends about 4–13× more tokens than ICL (Table 11). Section 4.3 only takes compute away from NL-LAN and never gives ICL an equal budget, for example through self-consistency or larger reasoning budgets. This matters because the paper cannot yet rule out that much of the gain comes from extra inference compute rather than from the text latent.
Questions for authors
- Is the 1–4 cell pixel tolerance used for NL-LAN on ARC-AGI-3 also applied when scoring the in-context baselines in Table 1? If the baselines were scored by exact match, please report both methods under the same criterion.
- How does NL-LAN compare with an in-context baseline given a matched token budget, for example self-consistency or majority voting over multiple samples, or a larger reasoning budget? Without this, the gains cannot be separated from the extra inference compute.
2 comments
Thank you authors for your comments. I understanding there are substantial cost associated with additional runs. Other than that, I think most of my concerns are addressed and therefore, I would like to increase my assessment to weak acceptance.
We sincerely thank the reviewer for their constructive feedback. We are glad they appreciate the strong predictive performance and the evidence supporting iterative refinement as an important component of NL-LAN. We address their concerns below, with manuscript changes highlighted in red (or violet where shared with Reviewer 2).
W1: Robustness of control results.
We agree that the strong performance on ar25 contributes substantially to the overall ARC-AGI-3 score and that repeated runs would provide a more robust estimate of performance variability.
Nevertheless, the improvements are not limited to this environment. Excluding ar25, NL-LAN still solves 19 levels compared to 12 for the no-induction variant, with a remaining score contribution of approximately 1.46 versus 0.97 (under the original 25-game normalisation). This suggests that the induced world models provide benefits beyond a single game, although the magnitude of the overall improvement is clearly influenced by ar25.
Unfortunately, repeated evaluations across all 25 interactive environments are prohibitively expensive due to the substantial LLM inference costs associated with induction, refinement, and control. We therefore follow a single-run evaluation protocol, while reporting per-environment results (Table 12) to make the variability across tasks transparent.
We acknowledge that these results do not establish statistical robustness and have clarified this limitation in the manuscript. Nevertheless, the aggregate improvement beyond ar25 provides encouraging evidence for the practical utility of our approach.
W2: Comparability of control baselines.
We agree that the public ARC-AGI-3 leaderboard results are not controlled comparisons, as agents differ in scaffolding, prompts, and inference budgets. Indeed, our no-induction variant already outperforms these baselines, indicating that the acting scaffold itself contributes substantially to performance. We therefore treat the leaderboard results as contextual reference points for overall system performance, rather than as evidence that isolates the effect of induction. The more informative comparison for this purpose is our within-framework no-induction ablation, although we acknowledge that inevitable differences in prompts due to the absence of a world model prevent perfect isolation.
W3 & Q2: Compute-matched comparison.
We agree that inference compute is an important factor. NL-LAN deliberately allocates additional compute resources for model induction, whereas the ICL baseline already uses maximum reasoning effort and a 65,536-token limit.
To investigate compute matching, we evaluated ICL with majority voting over independent samples on Sequence Hard (Gemini 3.1 Pro, high reasoning). At k=3 samples, ICL approximately matches NL-LAN's Pro token usage; we additionally test k=5 and k=10 to account for NL-LAN's Flash calls. Accuracy improves from 42.9% (k=1) to 45.9% (k=3), 48.6% (k=5), and 52.0% (k=10), but remains below NL-LAN's 73.0%, with diminishing returns. These results suggest that NL-LAN's advantage is not merely due to increased inference compute, but also to how that compute is allocated toward explicit world-model induction and refinement.
Q1: Pixel-error tolerance.
The environment-specific pixel-error tolerance (1–4 cells) is used both during NL-LAN's candidate scoring and held-out evaluation, where it is applied consistently to NL-LAN and all direct ICL baselines.
While comparing threshold-based and strict exact-match evaluation for Reviewer 2, we identified an inconsistency in how the thresholds were applied for sc25 (4 cells) and ka59 (1 cell). We have corrected this and recomputed the ARC-AGI-3 results under a consistent protocol. We apologise for the oversight and report the corrected threshold-based accuracies below for transparency.
The main conclusions remain unchanged. We have clarified the evaluation protocol in Appendix A.2 and updated all affected results throughout the manuscript.
We thank the reviewer again for their feedback, which helped us strengthen our experimental analysis and better qualify our conclusions.