- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces LifeWorld, a 2D "game" based on the life-like ceullar automata with Conway's Game of Life to measure the reasoning capability of LLMs. Specifically, the advantage of such benchmark it can be setup to easily differentiate different type of reasoning (deduction, induction, and abduction) under the same setting (given s_t, P, and s_{t_k}) and how LLMs performs in each. The authors then tested three frontier LLMs and find abduction to be much harder for LLMs, and also find LLMs performance drops as the game/setup becomes harder (e.g., with increasing grid size L).
Critical strengths
- The constructed benchmark shows several appealing properties such as difficulty being characterized by a complexity-theoretical result and being able to scale/generate instances easily. These are beneficial for establishing a controllable study / develop training methods by future work.
- The proposed LifeAgent (space-time recursion and logical recursion) is sound, as it follows the intuition that task decomposition and sefl-verification are useful paradigms to improve LLM performance on challenging tasks.
- The authors conducted thorough evaluation with 3 LLMs under various settings (L, k, difficulty) to measure how LLMs perform in each scenario.
Critical weaknesses
-
Space-time recursion (part of LifeAgent) depends on the hyperparameters and . Based on Section 5.2, some configurations underperform a naive single-model call, while others outperform it. Finding a good configuration seems to require grid search. This limits the practical usefulness of LifeAgent and raises concerns about its generalizability to other agentic settings, such as coding or general reasoning tasks.
-
The authors conduct extensive numerical analyses, such as modeling grid accuracy in Section 5.2 and analyzing why verification rarely helps with abduction in Section 5.3. However, it is unclear what these analyses imply about the general reasoning capabilities of LLM agents. Instead, they seem to focus on model performance on this particular synthetic benchmark. Therefore, it is difficult to understand how these findings could contribute to improving the general reasoning capabilities of LLM agents.
-
Overall, the empirical findings in Section 5 seem to provide limited novel insights into LLM reasoning. For example, it is intuitive that abduction is more difficult (as already suggested by the complexity analysis in Table 2), that LLM accuracy decreases with problem complexity (e.g., increasing L), and decomposing the problem into smaller subproblems (e.g., LifeAgent) is beneficial. What are some new insights from these results for LLM reasoning?
Questions for authors
- The 2D cellular automata (especially the forward and rule induction) problem seems to be solvable via coding. Did the authors measure models performance IF they are also allowed to write and execute code to solve the given problems? I wonder how this result would contrast to the current numbers in Table 1.
4 comments
Response to Reviewer 49rL (Part II)
W2: The analysis, e.g., modeling grid accuracy in Sec. 5.2 and why verification rarely helps abduction in Sec. 5.3, focus on this benchmark. It is unclear how they contribute to improving the general reasoning of LLM agents.
Thanks for the question. We would like to argue that each analysis gives a concrete design principle for LLM agents, and we list them as hypotheses beyond LifeWorld in A.6.
- Grid-accuracy model (Sec. 5.2): measure the single-call reliability before decomposing a task. Splitting helps only when each sub-task is smaller thanthe size that the model can solve reliably, and larger pieces can even decrease (−0.17) with more calls. Besides, the errors of the sub-tasks are strongly correlated: the independence assumption predicts 0.07 where we observe 0.42. Therefore, the size of the sub-tasks, rather than their number, should guide the decomposition.
- Verification of abduction (Sec. 5.3): feedback repairs local errors but not coupled ones. For induction, a correct rule is among the candidates for 0.59 of queries, against 0.44 for the best of four independent samples. For abduction, after identifying which cells are wrong, the next predecessor matches the target on fewer cells (0.76–0.78 → 0.68–0.71). Therefore, cell-level feedback is not enough for tightly coupled inverse problems, and such problems may require search over the whole candidate.
- Checkers (Sec. 5.3): a self-verifier fails by rejecting right answers, so improve the computation it re-runs. The model checkers accept almost no wrong rule (one in about 3,700), and they reject a right rule as often as the same forward run fails as a deduction query (). Improving the forward run can improve the verifier: tiling the check brings V-split within 0.02 of the exact simulator, while the whole-grid checker's accuracy is below the exact simulator by up to 0.22. To obtain a better self-verifier, one should improve the reliablity of the computation, rather than the judging prompt.
These principles are consistent with the runs on a new model (A.4): the first explains why 5×5 windows do not help gpt-5.6-luna, and for the second, feedback raises gpt's induction by 0.18–0.28 but its abduction by at most 0.08.
W3: It is intuitive that abduction is harder, that accuracy decreases with , and that decomposition is beneficial. What are the new insights?
Thanks for the question. These three trends are what we expect to see: LifeWorld ties each form of reasoning to a known complexity result (e.g., abduction is NP-complete), and the results show that LLMs follow this computational intuition. Beyond confirming the intuition, our contributions are threefold. i) LifeWorld measures the gaps instead of assuming them: on the same non-trivial hard pairs, gemini scores 0.08 on abduction, 0.36 on deduction and 0.46 on induction. ii) LifeAgent turns the intuition about decomposition into a harness with measured gains: space–time recursion improves deduction in all nine settings (+0.12 to +0.30) and logical recursion nearly doubles induction (0.34 → 0.63). iii) The results give new insights about why models fail, which can be measured only because all three forms have exact answers on the same grids.
In addition to the three principles above, we highlight the following findings.
- Induction fails at chaining steps, not for lack of evidence. Accuracy collapses from to (gemini, hard: 0.95–0.99 → 0.47–0.48). At almost no rule bit is wrong (0%), while at , on the pairs that identify the rule, the error rate of a bit does not fall with how often the data uses it (19% when used once, 19% when used over 100 times).
- Decomposition is not always beneficial, and its failures are correlated. It helps only below a per-call size and hurts above it (−0.17), and the failures of the pieces cluster (0.42 observed against 0.07 under independence). The threshold is a property of the model that a few calls measure (A.3).
- Models know the name of a rule better than its dynamics. gemini answers Conway's B3/S23 on 28% of the induction queries for which it is not consistent, yet its deduction under B3/S23 (0.68) is no better than under other rules (Appendix F.1).
- Grading rule recovery against the generating rule is ill-posed for sampled dynamics. For 21% of hard pairs another rule reproduces the pair, and more pairs from the same trajectory hardly reduce it (21% → 19% → 18%), so such benchmarks should grade by consistency (A.1).
We believe these findings cannot be obtained from existing benchmarks, which do not provide exact answers for all three forms on the same instances.
Thanks again for your support and valuable comments. We sincerely hope that our responses address your concerns. If there are any remaining questions or clarifications needed, we would be happy to provide further details.
Response
Thank you for the detailed response. I think my concerns are mostly addressed. I encourage the authors to include insights/heuristics for future researchers to pick h and b, and the additional insights you mentioned in the final manuscript. I keep my positive score.
Thanks for
Dear Reviewer 49rL,
Thank you very much for your kind reply and your positive score. We are glad that our response has addressed your concerns. As suggested, we will include in the final manuscript practical guidance on choosing and and we will also include the additional insights discussed in our response.
Thank you again for your valuable comments, which have greatly improved our paper.
Best regards,
The Authors
Response to Reviewer 49rL (Part I)
We sincerely appreciate the reviewer for the valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.
W1: Space–time recursion depends on h and b, and finding a good configuration seems to require grid search, which limits its practical usefulness and its generalizability to coding or general reasoning.
We agree that the choice of hyber-parameters is important. We show below that it can be made from a few cheap measurements, with a simple rule from Sec. 5.2, rather than by searching over large spaces
The selection rule. Sec. 5.2 shows that the accuracy of a single call depends mainly on the number of cell updates it carries (w). With n calls, the probability that the whole grid is right is well described by p(w)nγ, so we pick the layout that this formula ranks highest. In practice, each call is kept below about 50–80 updates for gemini.
An example (L=30, k=5, gemini, hard). The whole grid is one call with 302×5=4500 updates and is right 21% of the time. With 5×5 windows, each call has only (5+2)2=49 updates and there are 36×5=180 calls. One call at w=49 is right 98.5% of the time, so the formula gives 0.9851800.77≈0.44, close to the observed 0.42 (this cell is part of the fit), twice the whole grid.
Selection without a search (A.3). Calibrated on the three k=1 cells only, the rule ranks the best layout we ran first in all six cells with k∈{3,5}, +0.23 over the whole grid on average. In a leave-one-cell-out test, it picks the best layout in 6 of 9 cells and is never worse than the whole grid. We acknowledge that we ran only two window sizes and the finest is the best in every cell, so this shows that the k=1 cells suffice to calibrate the rule, not that it separates many layouts. We discuss this in Appendix A.3.
A new model. gpt-5.6-luna solves only 61–69% of single 5×5 windows (A.4). Measured in the L=10 cell alone, this predicts (with gemini's γ) at most 0.001 for the other two cells, and we observe 0.02 and 0.00, so the rule correctly tells that this layout cannot help gpt.
On generalizability. The principle behind the rule is general: measure how reliable a single call is, and choose the size of the sub-tasks accordingly. The logical recursion (propose-then-verify) does not rely on any property of LLCA either. Only the exact window decomposition uses the locality of the dynamics, and extending it to other domains, e.g., coding, is an interesting future direction.
Q: Forward simulation and rule induction seem solvable via coding. How would the performance change if models are allowed to write and execute code, compared with Table 3?
Thanks for the question. All single-call results (Table 3 of the revised paper) are tool-free by design, as the goal of LifeWorld is to measure the reasoning of the model itself, in the same way that arithmetic benchmarks are run without a calculator. Therefore, we do not include a code-execution setting. We discuss the expected contrast below and in A.5.