acceptodds
Under review as a conference paper at ICLR 2027

Reasoning in Life: Can Language Agents Master Evolving Worlds?

Abstract

Reasoning is the core capability for large language models (LLMs) to solve complex tasks such as planning and control. Such reasoning tasks can be categorized into three forms: deduction, which predicts the future of a system from its rule and current state; induction, which recovers the hidden rule from observed transitions; and abduction, which reconstructs the antecedent that best explains an observed outcome. Existing benchmarks, however, either consider a single form of reasoning or lack the formal grounding across the three forms, leaving no principled way to measure the divergence of language agents over different reasoning forms. To narrow the gap, we introduce LifeWorld, a benchmark built on life-like cellular automata with rules generalizing the famous Conway's Game of Life (GoL), where the evolving step can be characterized by the rule , a configuration , and its -step successor . We curate the dataset for three forms of reasoning: i) deduction () is forward simulation with a unique answer, computable in polynomial time yet Turing complete in expressive power; ii) induction () is rule identification from observed transitions, corresponding to program induction from input-output examples; and iii) abduction () is predecessor reconstruction under irreversible dynamics, which is NP-complete at bounded horizon. We further propose LifeAgent, a recursive reasoning harness that operates through two forms of recursion: i) space-time recursion, which breaks a query into sub-grids and sub-horizons, and ii) logical recursion, which solves induction and abduction by a propose-then-verify loop in which candidates are checked through deduction. Extensive experiments across frontier LLMs demonstrate the effectiveness of both recursions in LifeAgent and provide a comprehensive analysis of performance across the three reasoning forms, characterizing the behaviors of frontier LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning, Language Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_49rLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces LifeWorld, a 2D "game" based on the life-like ceullar automata with Conway's Game of Life to measure the reasoning capability of LLMs. Specifically, the advantage of such benchmark it can be setup to easily differentiate different type of reasoning (deduction, induction, and abduction) under the same setting (given s_t, P, and s_{t_k}) and how LLMs performs in each. The authors then tested three frontier LLMs and find abduction to be much harder for LLMs, and also find LLMs performance drops as the game/setup becomes harder (e.g., with increasing grid size L).

Critical strengths
  • The constructed benchmark shows several appealing properties such as difficulty being characterized by a complexity-theoretical result and being able to scale/generate instances easily. These are beneficial for establishing a controllable study / develop training methods by future work.
  • The proposed LifeAgent (space-time recursion and logical recursion) is sound, as it follows the intuition that task decomposition and sefl-verification are useful paradigms to improve LLM performance on challenging tasks.
  • The authors conducted thorough evaluation with 3 LLMs under various settings (L, k, difficulty) to measure how LLMs perform in each scenario.
Critical weaknesses
  1. Space-time recursion (part of LifeAgent) depends on the hyperparameters hh and bb. Based on Section 5.2, some configurations underperform a naive single-model call, while others outperform it. Finding a good configuration seems to require grid search. This limits the practical usefulness of LifeAgent and raises concerns about its generalizability to other agentic settings, such as coding or general reasoning tasks.

  2. The authors conduct extensive numerical analyses, such as modeling grid accuracy in Section 5.2 and analyzing why verification rarely helps with abduction in Section 5.3. However, it is unclear what these analyses imply about the general reasoning capabilities of LLM agents. Instead, they seem to focus on model performance on this particular synthetic benchmark. Therefore, it is difficult to understand how these findings could contribute to improving the general reasoning capabilities of LLM agents.

  3. Overall, the empirical findings in Section 5 seem to provide limited novel insights into LLM reasoning. For example, it is intuitive that abduction is more difficult (as already suggested by the complexity analysis in Table 2), that LLM accuracy decreases with problem complexity (e.g., increasing L), and decomposing the problem into smaller subproblems (e.g., LifeAgent) is beneficial. What are some new insights from these results for LLM reasoning?

Questions for authors
  • The 2D cellular automata (especially the forward and rule induction) problem seems to be solvable via coding. Did the authors measure models performance IF they are also allowed to write and execute code to solve the given problems? I wonder how this result would contrast to the current numbers in Table 1.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 49rL (Part I)

We sincerely appreciate the reviewer for the valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.


W1: Space–time recursion depends on hh and bb, and finding a good configuration seems to require grid search, which limits its practical usefulness and its generalizability to coding or general reasoning.

We agree that the choice of hyber-parameters is important. We show below that it can be made from a few cheap measurements, with a simple rule from Sec. 5.2, rather than by searching over large spaces

  1. The selection rule. Sec. 5.2 shows that the accuracy of a single call depends mainly on the number of cell updates it carries (ww). With nn calls, the probability that the whole grid is right is well described by p(w)nγp(w)^{n^{\gamma}}, so we pick the layout that this formula ranks highest. In practice, each call is kept below about 50–80 updates for gemini.

  2. An example (L=30L=30, k=5k=5, gemini, hard). The whole grid is one call with 302×5=450030^2\times 5 = 4500 updates and is right 21% of the time. With 5×5 windows, each call has only (5+2)2=49(5+2)^2 = 49 updates and there are 36×5=18036\times5 = 180 calls. One call at w=49w=49 is right 98.5% of the time, so the formula gives 0.9851800.77≈0.440.985^{180^{0.77}} \approx 0.44, close to the observed 0.42 (this cell is part of the fit), twice the whole grid.

  3. Selection without a search (A.3). Calibrated on the three k=1k=1 cells only, the rule ranks the best layout we ran first in all six cells with k∈{3,5}k\in\{3,5\}, +0.23 over the whole grid on average. In a leave-one-cell-out test, it picks the best layout in 6 of 9 cells and is never worse than the whole grid. We acknowledge that we ran only two window sizes and the finest is the best in every cell, so this shows that the k=1k=1 cells suffice to calibrate the rule, not that it separates many layouts. We discuss this in Appendix A.3.

  4. A new model. gpt-5.6-luna solves only 61–69% of single 5×5 windows (A.4). Measured in the L=10L=10 cell alone, this predicts (with gemini's γ\gamma) at most 0.001 for the other two cells, and we observe 0.02 and 0.00, so the rule correctly tells that this layout cannot help gpt.

  5. On generalizability. The principle behind the rule is general: measure how reliable a single call is, and choose the size of the sub-tasks accordingly. The logical recursion (propose-then-verify) does not rely on any property of LLCA either. Only the exact window decomposition uses the locality of the dynamics, and extending it to other domains, e.g., coding, is an interesting future direction.


Q: Forward simulation and rule induction seem solvable via coding. How would the performance change if models are allowed to write and execute code, compared with Table 3?

Thanks for the question. All single-call results (Table 3 of the revised paper) are tool-free by design, as the goal of LifeWorld is to measure the reasoning of the model itself, in the same way that arithmetic benchmarks are run without a calculator. Therefore, we do not include a code-execution setting. We discuss the expected contrast below and in A.5.

  1. With an interpreter, deduction becomes a simple loop of O(kL2)O(kL^2) updates and induction becomes a brute-force search over 2182^{18} rules. We would expect both to be close to the ceiling, but they would then measure tool use rather than reasoning.
  2. For abduction, code changes the question rather than settling it. Trivial and sparse pairs become easy, since returning the observation can be checked by one simulation. On non-trivial hard pairs, however, a direct SAT encoding solved by MiniSat already needs a median of 17k to over 100k conflicts at k=2k=2 (A.2). The question then becomes whether a model can write an effective solver for an NP-complete inverse problem, which is an interesting direction for future work.
  3. When considering the tools, the selection of the possible tools that can be used would be another important factor, which is beyond the scope of this work. We thank the reviewers for raising this point and will include a discussion on this aspect in the final version of the paper.
AuthorsOpenReview ↗

Response to Reviewer 49rL (Part II)

W2: The analysis, e.g., modeling grid accuracy in Sec. 5.2 and why verification rarely helps abduction in Sec. 5.3, focus on this benchmark. It is unclear how they contribute to improving the general reasoning of LLM agents.

Thanks for the question. We would like to argue that each analysis gives a concrete design principle for LLM agents, and we list them as hypotheses beyond LifeWorld in A.6.

  1. Grid-accuracy model (Sec. 5.2): measure the single-call reliability before decomposing a task. Splitting helps only when each sub-task is smaller thanthe size that the model can solve reliably, and larger pieces can even decrease (−0.17) with more calls. Besides, the errors of the sub-tasks are strongly correlated: the independence assumption predicts 0.07 where we observe 0.42. Therefore, the size of the sub-tasks, rather than their number, should guide the decomposition.
  2. Verification of abduction (Sec. 5.3): feedback repairs local errors but not coupled ones. For induction, a correct rule is among the candidates for 0.59 of queries, against 0.44 for the best of four independent samples. For abduction, after identifying which cells are wrong, the next predecessor matches the target on fewer cells (0.76–0.78 → 0.68–0.71). Therefore, cell-level feedback is not enough for tightly coupled inverse problems, and such problems may require search over the whole candidate.
  3. Checkers (Sec. 5.3): a self-verifier fails by rejecting right answers, so improve the computation it re-runs. The model checkers accept almost no wrong rule (one in about 3,700), and they reject a right rule as often as the same forward run fails as a deduction query (r=0.96r=0.96). Improving the forward run can improve the verifier: tiling the check brings V-split within 0.02 of the exact simulator, while the whole-grid checker's accuracy is below the exact simulator by up to 0.22. To obtain a better self-verifier, one should improve the reliablity of the computation, rather than the judging prompt.

These principles are consistent with the runs on a new model (A.4): the first explains why 5×5 windows do not help gpt-5.6-luna, and for the second, feedback raises gpt's induction by 0.18–0.28 but its abduction by at most 0.08.


W3: It is intuitive that abduction is harder, that accuracy decreases with LL, and that decomposition is beneficial. What are the new insights?

Thanks for the question. These three trends are what we expect to see: LifeWorld ties each form of reasoning to a known complexity result (e.g., abduction is NP-complete), and the results show that LLMs follow this computational intuition. Beyond confirming the intuition, our contributions are threefold. i) LifeWorld measures the gaps instead of assuming them: on the same non-trivial hard pairs, gemini scores 0.08 on abduction, 0.36 on deduction and 0.46 on induction. ii) LifeAgent turns the intuition about decomposition into a harness with measured gains: space–time recursion improves deduction in all nine settings (+0.12 to +0.30) and logical recursion nearly doubles induction (0.34 → 0.63). iii) The results give new insights about why models fail, which can be measured only because all three forms have exact answers on the same grids.

In addition to the three principles above, we highlight the following findings.

  1. Induction fails at chaining steps, not for lack of evidence. Accuracy collapses from k=1k=1 to k=2k=2 (gemini, hard: 0.95–0.99 → 0.47–0.48). At k=1k=1 almost no rule bit is wrong (0%), while at k≥2k\ge2, on the pairs that identify the rule, the error rate of a bit does not fall with how often the data uses it (19% when used once, 19% when used over 100 times).
  2. Decomposition is not always beneficial, and its failures are correlated. It helps only below a per-call size and hurts above it (−0.17), and the failures of the pieces cluster (0.42 observed against 0.07 under independence). The threshold is a property of the model that a few k=1k=1 calls measure (A.3).
  3. Models know the name of a rule better than its dynamics. gemini answers Conway's B3/S23 on 28% of the induction queries for which it is not consistent, yet its deduction under B3/S23 (0.68) is no better than under other rules (Appendix F.1).
  4. Grading rule recovery against the generating rule is ill-posed for sampled dynamics. For 21% of hard pairs another rule reproduces the pair, and more pairs from the same trajectory hardly reduce it (21% → 19% → 18%), so such benchmarks should grade by consistency (A.1).

We believe these findings cannot be obtained from existing benchmarks, which do not provide exact answers for all three forms on the same instances.


Thanks again for your support and valuable comments. We sincerely hope that our responses address your concerns. If there are any remaining questions or clarifications needed, we would be happy to provide further details.

Reviewer_49rLOpenReview ↗

Response

Thank you for the detailed response. I think my concerns are mostly addressed. I encourage the authors to include insights/heuristics for future researchers to pick h and b, and the additional insights you mentioned in the final manuscript. I keep my positive score.

AuthorsOpenReview ↗

Thanks for

Dear Reviewer 49rL,

Thank you very much for your kind reply and your positive score. We are glad that our response has addressed your concerns. As suggested, we will include in the final manuscript practical guidance on choosing hh and bb and we will also include the additional insights discussed in our response.

Thank you again for your valuable comments, which have greatly improved our paper.

Best regards,

The Authors

Reviewer_sAYCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper first introduces LifeWorld, a benchmark that tests the deduction, induction, and abduction abilities of language models in a controlled environment. The tasks are created over the same transition (P,st,st+k)(P, s_t, s_{t+k}) by fixing two objects and solving for the third, contributing a single system that evaluates the three reasoning abilities with verifiable answers and variable difficulty levels. It also proposes LifeAgent, an agent harness featuring space-time recursion (decomposes forward simulation exactly into sub-horizons and overlapping windows), and logical recursion(solve induction and abduction with a propose-then-verify loop). Experiments on SOTA LLMs show that without additional harnessing, deduction accuracy degrades as the grid size and evolution horizon increase, induction accuracy drops sharply when step number is greater than 1, while abduction is the hardest overall. Space-time recursion improves deduction accuracy. Logical recursion improves induction accuracy but offers limited gain in abduction.

Critical strengths
  1. The paper has a clear motivation and is well-written. It explores LLMs‘ reasoning capabilities in 3 distinct dimensions, giving very structured and informative insights than scores on a general reasoning benchmark.

  2. The benchmark design is very clean and smart. It derives 3 reasoning scenarios from one underlying environment with verifiable answers and variable difficulty. The tasks are also connected to established complexity-theoretic results, providing a well-controlled playground to test LLM reasoning capabilities.

  3. The ablation of results is very comprehensive and clearly explains why each method works or fails. For example, the paper fits the per-call accuracy curve, showing that the number of cell updates carried by a single call helps explain whether tiling helps or hurts. It also separates gains from feedback from gains from resampling, and shows that a model-based checker’s acceptance rate of right rules tracks the deduction accuracy of its own layout, which are all interesting findings.

Critical weaknesses
  1. The induction evaluation criteria sounds questionable. For multi-step evolution, it's possible that a candidate rule system PP produces the two endpoints through a different intermediate trajectory, but marked incorrect by the current evaluation criteria. While this can be rare when step number is small, it can be a concern as the evolution horizon becomes larger. Reporting both endpoint consistency accuracy and ground-truth recovery accuracy would be more informative and would clarify the benchmark’s claim of verifiable answers.

  2. LifeAgent, the proposed agent harness, is tested only on one model and only the hard level. It's difficult to comprehensively tell its pros and cons over single-call methods without showing its performance across varying difficulty levels, under matched token constraints.

  3. The contribution of the paper should be phrased more carefully. The difficulty measure and space-time decomposition are tied to specific properties of LLCA, and there is no evidence that the observed reasoning characteristics or harness benefits extend to other environments. It would be great if the authors can explicitly distinguish findings specific to LLCA from hypotheses about deduction, induction, and abduction of general LLMs.

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Response to Reviewer sAYC (Part I)

We sincerely appreciate the reviewer for the positive and constructive feedback. Following your suggestions, we added a second induction score and new experiments on another model and difficulty level. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.


W1: A candidate rule may produce the two endpoints through a different intermediate trajectory. Reporting both endpoint-consistency accuracy and ground-truth recovery accuracy would be more informative.

Thanks for this valuable suggestion. We now report both, and endpoint consistency (consistency) is the metric of all induction results in the main text, while the complete results under ground-truth recovery (rule match) are reported in A.1. All conclusions hold under both gradings. The full tables are in our response to Reviewer 2qCk (W1/Q1) and in A.1.

  1. How often the two gradings differs from each other. We enumerated all 2182^{18} rules on every induction pair with k≥2k\ge2. 21% of hard pairs (40% of all pairs) have another rule that reproduces the endpoints. Regarding the concern about larger horizons, we find that this depends on the parity of kk rather than its size, so it does not grow with the horizon:
hard pairsk=2k=2k=3k=3k=4k=4k=5k=5
with a second rule26%13%32%14%

Grid size also hardly matters (24%, 20%, 19% at L=10,20,30L=10, 20, 30). On all the other pairs, the observation identifies the rule and the two gradings are identical.

  1. The results are stable. Single calls change by 0.03–0.08 for every model and level, and the model ranking and the drop from k=1k=1 to k=2k=2 are unchanged. The order of the LifeAgent variants is also unchanged (gemini, hard, mean of six cells):
one callV-wholeV-splitV-sim
rule match0.320.450.560.58
consistency0.350.500.610.63
  1. On the claim of verifiable answers. With the consistency grading, induction is graded purely by simulation. Therefore, the answers of all three forms are verifiable by simulation, and the claim holds in the strict sense.

In the revision, we updated Sec. 3, the scoring appendix, Table 3, Sec. 5 and A.1 accordingly.

AuthorsOpenReview ↗

Response to Reviewer sAYC (Part II)

W2: LifeAgent is tested only on one model and the hard level. It is difficult to tell its pros and cons over single-call methods across difficulty levels and under matched token constraints.

Thanks for the suggestion. We focused on the hard level in the submission as it is where single calls fail and a harness is needed. To address the concern, we ran both recursions on a new level and a new model, and we report the token cost for every run.

  1. Setup. gemini-3.8-flash on medium (new level) and gpt-5.6-luna on hard (new model), with three cells per recursion and task, 50 queries per cell, compared with one call on the same queries (McNemar test). Note that gemini already solves easy queries in one call, so medium is the meaningful additional level.

  2. The propose–verify loop (V-sim, R=5R=5; one call → loop; tokens relative to one call; * p<0.05p<0.05, ** p<0.01p<0.01):

settingL=10L=10L=20L=20L=30L=30tokens
gpt, hard, induction (k=3k=3)0.04 → 0.22**0.14 → 0.42**0.08 → 0.34**4.5–5.5×
gemini, medium, induction (k=3k=3)0.54 → 0.620.40 → 0.60*0.62 → 0.84**2.9–4.1×
gemini, medium, abduction (k=1k=1)0.28 → 0.46**0.32 → 0.48*0.40 → 0.504.1–4.3×
gpt, hard, abduction (k=1k=1)0.00 → 0.080.00 → 0.020.00 → 0.003.1–7.7×

The loop improves induction in all six new cells, significantly in five, and improves gemini's abduction as it does on hard queries in the paper. For gpt's abduction results,the gain is small, which is consistent with our analysis in Sec. 5.3: gpt solves only 1 of 1,328 non-trivial hard abductions in one call, so there is no near-correct proposal for the feedback to repair.

  1. Splitting into 5×5 windows (deduction):
settingL=10L=10, k=3k=3L=20L=20, k=3k=3L=30L=30, k=1k=1tokens
gemini, medium0.80 → 0.940.62 → 0.740.62 → 0.92**3.1–5.6×
gpt, hard0.02 → 0.000.02 → 0.020.00 → 0.003.9–15×

The windows improve gemini in all three cells, significantly at L=30L=30 (50 queries cannot resolve the other two). The result of gpt follows from the per-call accuracy of Sec. 5.2: gemini solves 98% of single 5×5 windows on hard cells, while gpt solves only 67%, and at L=30L=30 all 36 windows must be right. Measured in the L=10L=10 cell alone, gpt's window accuracy (with gemini's γ\gamma) predicts at most 0.001 for the other two cells, and we observe 0.02 and 0.00. Therefore, a few cheap calls tell whether splitting can help a given model (A.3, A.4).

  1. On matched tokens. For induction, we compare at the same number of calls. After four proposals, the V-sim loop commits 0.60 with 59k tokens per query, while the best of four independent calls, even picked by an oracle, reaches only 0.44 with about 70k tokens. The gain is therefore not a matter of budget. For deduction, we did not run an equal-budget control such as a vote over several whole-grid samples, and we state this in the limitations. The results show is that more calls do not help by themselves: at k=1k=1, 10×10 windows use 4–9× the calls of the whole grid and are significantly worse (−0.17).

In the revision, A.4 reports the full results and Sec. 5 adds a paragraph "Beyond One Model and One Difficulty".

AuthorsOpenReview ↗

Response to Reviewer sAYC (Part III)

W3: The difficulty measure and the space–time decomposition are tied to LLCA. Findings specific to LLCA should be distinguished from hypotheses about general LLMs.

Thanks for the suggestion, which makes the scope of the paper clearer. We have revised the paper accordingly, and we would also like to clarify why a study on LLCA is useful beyond LLCA.

  1. The revision. The contribution now reads "...how their behavior diverges across the three forms of reasoning on LifeWorld", the introduction states that the findings are established on LLCA, and the new A.6 pairs each finding with the general hypothesis it suggests. The limitations state that the difficulty level is defined through the rule bits of LLCA. The propose–verify loop of LifeAgent does not rely on any property of LLCA, and only the window decomposition uses the locality of the dynamics, which A.6 marks as specific to local dynamics.
  2. The main contributions. We would like to emphasize that our main contributions are i) one setting that unifies deduction, induction and abduction on the same instances with exact verification, and ii) guidance on how to improve each form of reasoning, rather than the specific numbers on LLCA. For example, a model-based checker almost never accepts a wrong rule and rejects right rules as often as its forward run fails (r=0.96r=0.96), which suggests that a self-verifier fails mainly by false rejection and improves with the computation it re-runs.
  3. LLCA is theoretically grounded. We choose LLCA because it is simple to state and general in theory. The Game of Life, one of the LLCA rules, is Turing complete, i.e., it can simulate any computing system, so deduction on LLCA is, in principle, the forward execution of an arbitrary computation. Deciding whether a pre-image exists is NP-complete, so every problem in NP can be reduced to abduction on LLCA. Therefore, although the experiments are run on LLCA, the tasks stand for executing a computation and inverting it in general.
  4. A similar practice. ZebraLogic (https://arxiv.org/abs/2502.01100) also focuses on one class of problems Zebra puzzle, while its insights and findings on how the reasoning of LLMs scales with complexity are general. Similarly, LifeWorld focuses on LLCA, and A.6 lists the hypotheses that its findings suggest for reasoning in general.

Thanks again for your support and valuable comments. We sincerely hope that our responses address your concerns.

Reviewer_2qCkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LifeWorld, a benchmark built on life-like cellular automata. A rule, a state and its k-step successor are tied together, so hiding any one of them gives deduction, induction or abduction over the same transition, and every answer is checked exactly by simulation. The paper also proposes LifeAgent, a harness that splits forward simulation into small space-time windows and wraps induction and abduction in a propose-then-verify loop. Three frontier models are evaluated in single-call and harness settings.

Critical strengths

Strengths:

  • Deduction, induction, and abduction are tested on the same underlying transitions. This removes several sources of variation and makes the differences in model performance more meaningful. The explicit rules and programmatic evaluation also make the results easier to inspect.

  • The experiments help explain where the models struggle, not just how often they fail. For induction, improving the model-based checker brings accuracy close to that of the simulator-based loop, 0.56 versus 0.58. For abduction, even an exact checker provides only limited gains, and proposals often get worse after feedback.

Critical weaknesses

Weaknesses:

  • Induction sees only two snapshots, and its grading does not match abduction's.
    • Induction is discovery, so a natural study is how rule recovery improves with more evidence. The definition allows one or more observation pairs, but the experiments use a single pair.
    • Abduction is graded by simulation. Induction is graded on bits used along the true trajectory, including bits used only at hidden intermediate steps, which the two endpoints may not reveal. The paper treats the resulting 2% floor as a limit of the checkers, when it follows from the grading choice.
    • Grading induction by simulation, or reporting how many rules are consistent with each query, would remove this noise. \
Questions for authors

Q1: Why is induction graded against the generating rule rather than consistency with the observed transition? How would the main conclusions change if all observation-consistent rules were accepted?

Q2: What connects the exercised-bit difficulty measure to the computational difficulty of predecessor reconstruction? Can you show that the easy, medium, and hard bins capture inverse-search difficulty, rather than primarily differences in density and the availability of trivial predecessors?

Q3: Does the gap between abduction and the other modes persist after separating structurally easy cases?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2qCk (Part I)

We sincerely appreciate the reviewer for the careful review and the insightful comments. Following your suggestions, induction is now graded by simulation in the whole paper, and we added several new analysis, which make the benchmark more rigorous. We hope our following answers will address your concerns and please let us know if there is any other clarification we can provide.


W1 / Q1: Induction is graded against the generating rule rather than by consistency with the observed transition. How would the conclusions change if all observation-consistent rules were accepted?

Thanks for this valuable suggestion. We have adopted it in the whole paper: induction is now graded by simulation in the same way as abduction, i.e., any rule that carries sts_t onto st+ks_{t+k} is accepted (consistency). Table 3 and all induction results in the main text are updated, and the complete results under the grading against the generating rule (rule match) are kept in A.1. All conclusions of the paper hold under both gradings.

  1. How many rules are consistent with each query. As suggested, for every induction pair with k≥2k\ge2 we enumerated all 2182^{18} rules. 21% of hard pairs have a consistent rule that differs from the generating one on an exercised bit (35% of medium, 64% of easy). Interestingly, this depends on the parity of kk rather than its size: 26% and 32% of hard pairs at k=2,4k=2,4, and 13% and 14% at k=3,5k=3,5. A common case at even kk is a rule that passes through the exact inverse (0↔1) of the true hidden grid and returns to the same final grid (33 of 60 sampled pairs).

  2. The two gradings are consistent whenever the rule can be inferred. At k=1k=1 and on the other 79% of hard pairs at k≥2k\ge2, every consistent rule agrees with the generating rule on the exercised bits, so the two gradings give the same score for every answer (e.g., 0.36 for gemini on these hard pairs). They differ only on the pairs where the observation does not determine the rule (gemini, hard: 0.39 under consistency, 0.26 under rule match). Therefore, consistency is the strictest grading that the observation can decide.

  3. The conclusions are unchanged. Single call, all cells with k≥2k\ge2, rule match → consistency:

modelhardmediumeasy
gemini-3.8-flash0.34 → 0.370.49 → 0.540.84 → 0.89
deepseek-flash0.21 → 0.260.33 → 0.400.75 → 0.83
gpt-5.6-luna0.13 → 0.190.18 → 0.260.48 → 0.56

LifeAgent (gemini, hard, R=5R=5, mean of six cells):

one callV-wholeV-splitV-sim
rule match0.320.450.560.58
consistency0.350.500.610.63

The scores move by only 0.03–0.08. Therefore, the model ranking, the drop from k=1k=1 to k=2k=2 (gemini, hard: 0.95–0.99 → 0.47–0.48), the order of the checkers, and the gain of the loop (+0.26 under rule match, +0.28 under consistency) all remain the same.

  1. Pairs that a fixed rule explains. Consistency also credits answers that need no inference: a pair that dies out is reproduced by B/S, and a pair that stands still by B/S012345678. The construction caps these pairs at 10% of a cell. gemini rarely gives these answers (≤0.01 of queries), while gpt gains up to 0.08 from them. Without these pairs, the two gradings agree within 0.01 on hard queries for every model (deepseek 0.20, gemini 0.33, gpt 0.11), and V-sim still rises from 0.31 (one call) to 0.60. We report this in A.1.

  2. The 2% floor. With the consistency grading, this floor disappears: V-sim accepts no wrong rule by construction, and the two model checkers accept only one wrong rule in about 3,700 candidates.

In the revision, Sec. 3 and the scoring appendix display the definitions of the grading, Table 3 and Sec. 5 report the results under consistency, and A.1 reports the comparisons and all results under rule match.

AuthorsOpenReview ↗

Response to Reviewer 2qCk (Part II)

W2: The definition allows one or more observation pairs, but the experiments use a single pair. A natural study is how rule recovery improves with more evidence.

Thanks for the suggestion. The single-pair setting is the minimal instance of the definition and keeps the three forms on exactly the same transition, which is the core design of LifeWorld. To answer the question about more evidence about more observation pairs, we provide two analyses.

  1. More pairs from the same trajectory bring little information and improvement. We added the next pair of the same trajectory, then a third, and counted the consistent rules again:
pairs given123
hard pairs with a second rule21%19%18%
all pairs40%32%30%

A rule that fits one pair usually fits the whole trajectory, e.g., the "inverse path" rule returns to the true grid at every observed step. Therefore, the single-pair setting already carries most of the evidence that the trajectory can provide.

  1. The failure of the models is not caused by the lack of evidence. The per-bit analysis (gemini, hard) shows that at k=1k=1 almost no rule bit is wrong (0%). At k≥2k\ge2, on the pairs that identify the rule (so that a differing bit is a real error), a bit used only once in the trajectory is wrong 19% of the time, and a bit used over 100 times is still wrong 19% of the time. If the evidence were the bottleneck, the second number would be near zero. The difficulty is chaining the hidden steps, rather than the amount of evidence. Due to the limited budget, we focus on the single-pair expeirments. A multi-pair evaluation with different starting grids is an interesting extension, and we will explore it in future work.

Q2: Do the easy, medium, and hard bins of the exercised-bit measure capture inverse-search difficulty, rather than density and the availability of trivial predecessors?

Yes, the exercised-bit measure correlates with inverse-search difficulty, with the link to solver effort established at short horizons (k≤2k\le2). We provide three types of evidence, where trivial predecessors are removed in all analyses.

  1. Why exercised bits. A life-like rule has 18 bits, one per (cell state, live-neighbour count). The level counts the bits that the true trajectory exercises (1–8 easy, 9–13 medium, 14–18 hard). More exercised bits mean more interacting constraints that a predecessor must satisfy.

  2. Density is not the reason. Within each density bin, accuracy still decreass from easy to hard, e.g., 0.75 → 0.32 → 0.01 at 5–10% live cells. In a regression, the number of exercised bits dominates (z=−28z=-28), while density has a far smaller effect (∣z∣≤3|z|\le3).

  3. It tracks inverse-search difficulty. We encoded each pair as a SAT problem and counted the conflicts MiniSat needs to find a predecessor:

median conflicts, L=30L=30easymediumhard
k=1k=1332,12137,035
k=2k=251,455>100k>100k

At k≤2k\le2, the solver effort grows with the level at every LL, by up to 1,000×, and within every density bin hard pairs still need 2 to over 1,000 times the conflicts of easy ones. At k≥3k\ge3, the solver exhausts its budget of 10510^5 conflicts on most pairs of every level, including easy pairs that gemini solves in one call, so we draw no conclusion from these horizons. This is consistent to the theorethical result that inverse-search is NP-complete.


Q3: Does the gap between abduction and the other modes persist after separating structurally easy cases?

Yes, and the gap becomes larger.

  1. In 11% of hard pairs, the final grid is its own predecessor, so copying it is a correct answer. We removed these pairs from all three tasks:
hard, no trivial pairsdeductioninductionabduction
gemini-3.8-flash0.360.460.08
deepseek-flash0.480.340.05
gpt-5.6-luna0.020.160.00 (1 of 1,328)
  1. This analysis also refines the trend over the horizon. The share of trivial pairs grows from 3% at k=1k=1 to 16% at k=5k=5, and without them hard abduction of gemini stays flat at 0.04–0.11 for every kk. Therefore, abduction is consistently the hardest form at every horizon. We have updated Sec. 5.1 with this result.

Thanks again for your valuable comments. With the new grading and analyses, the benchmark is more rigorous and all the conclusions are confirmed. We sincerely hope that our responses address your concerns, and we would be grateful if you could reconsider your evaluation accordingly.

AuthorsOpenReview ↗

Kind Reminder: Has Our Response Addressed Your Concerns?

Dear Reviewer 2qCk,

Thank you again for your detailed and constructive review. As the discussion period ends in three days (November 18, AoE), we would like to kindly check whether our response has addressed your concerns. Following your suggestions, we have made the following changes:

  1. W1/Q1: Induction is now graded by consistency throughout the paper, and all conclusions still hold. The full results under rule match are kept in A.1.
  2. W2: Adding more pairs from the same trajectory barely reduces the number of hard pairs with a second consistent rule (21% → 19% → 18%). The per-bit analysis also shows that the errors come from chaining the hidden steps, not from a lack of evidence.
  3. Q2: The difficulty level tracks inverse-search difficulty: within the same density, the SAT solver needs up to 1,000× more conflicts on hard pairs than on easy ones.
  4. Q3: After the trivial pairs are removed from all three tasks, the gap becomes larger (gemini, hard: abduction 0.08 vs. deduction 0.36 and induction 0.46).

If our response has addressed your concerns, we would be grateful if you could reconsider your evaluation of the paper. If you have any further concerns, we are happy to provide further results and analysis in the remaining time.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers and AC (Part I)

We thank all three reviewers for their time and valuable comments. The suggestions make the evaluation more rigorous and the experiments more complete, and all main conclusions of the paper are unchanged by the new results. Below, we respond to the common comments, summarize the revision, and highlight our contributions.


Response to the Common Comments

C1: The grading of induction. (2qCk W1/Q1, sAYC W1)

  1. Following the reviewers' suggestions, induction is now graded by simulation in the whole paper, in the same way as abduction: a rule is right if it carries sts_t onto st+ks_{t+k} (consistency). The main text uses this grading, and the complete results comparing with the generating rule (rule match) are displayed in A.1 for references.
  2. The two gradings are identical at k=1k=1 and on every pair where the observation identifies the rule (79% of hard pairs at k≥2k\ge2). Scores change by only 0.03–0.08, and all conclusions hold under both gradings, also without the pairs that die out or stand still, which a fixed rule explains (A.1).
  3. We would like thank the reviewers for pointing out the importance of this distinction, which brings a more thorough and rigorous evaluation of induction in our revised paper.

C2: Abduction versus trivial cases and grid density. (2qCk Q2, Q3)

  1. We removed the trivial pairs (the final grid is its own pre-image) from all three tasks, and the gap becomes larger: on the same hard pairs, gemini scores 0.08 on abduction, 0.36 on deduction and 0.46 on induction.
  2. At fixed density, accuracy still drops from easy to hard, and at short horizons a SAT solver needs far more search on harder levels.
  3. Without trivial pairs, hard abduction stays low and flat at every horizon, which gives a cleaner picture of its difficulty (A.2).
  4. All results are provided in the revised paper, and we thank the reviewers for this insightful suggestion.

C3: LifeAgent beyond one setting, and the choice of the window size. (sAYC W2, 49rL W1)

  1. We ran additional experiments (50 queries per cell) with gemini on medium queries and include results for gpt-5.6-luna on hard queries. The propose–verify loop works in both: gpt's induction improves by 0.18–0.28 (all p<0.005p<0.005), and gemini's medium induction and abduction by 0.08–0.22.
  2. Windows help gemini but not gpt, because gpt solves only 67% of single 5×5 windows. If the per-call accuracy of the splited widnwos is low, it is expected that the overall accuracy is low as well. The per-call accuracy of Sec. 5.2 explains the difference.
  3. The window size can be chosen from the cheapest cells: calibrated at k=1k=1, the rule ranks the best layout we conduct first in all six longer-horizon cells (A.3, A.4). So the choice of the window size is guided by the easiest measurements and then applied consistently to the harder settings.

C4: What is general and what is specific to LLCA. (sAYC W3, 49rL W1–W3)

We state explicitly that the findings are established on LifeWorld, and a new table lists the general hypothesis suggested by each finding (A.6). We also discuss solving the tasks with code (A.5).

AuthorsOpenReview ↗

General Response to Reviewers and AC (Part II)

Revision Summary of the Paper

All changed text in the revised version is highlighted in blue, and all new analysis are in a new Appendix A. The main changes are:

  1. Grading of induction (Sec. 3, Table 3, Sec. 5, A.1). Induction is graded by simulation, and Table 3 and all induction figures and appendix tables are updated. A.1 reports how often one pair leaves the rule ambiguous (21% of hard pairs), the complete results under rule match, and the results without the pairs that a fixed rule explains.
  2. Abduction and the difficulty level (Sec. 1, Sec. 5, A.2). Trivial pairs are separated in all three modes, and the introduction reports the gap without them. A.2 adds abduction at every horizon without trivial pairs, accuracy at fixed density, and the search effort of a SAT solver. The abduction gap becomes larger.
  3. LifeAgent on a second model and a second level (Sec. 5, A.4). Both recursions are run with gpt-5.6-luna on hard queries and with gemini on medium queries, with the token cost of every run. Sec. 5 adds a paragraph "Beyond One Model and One Difficulty".
  4. Choice of the window size (Sec. 5.2, A.3). Calibrated at k=1k=1, the per-call accuracy ranks the best layout we ran first in all six longer-horizon cells. A.3 also states that only 5×5 and 10×10 windows were compared.
  5. Comparison at the same number of calls (Sec. 5.3, FAQs). After four proposals, the propose–verify loop exceeds the best of four independent calls, with fewer tokens.
  6. Code and generality (A.5, A.6). A.5 discusses solving the tasks with code, and A.6 pairs each finding with the general hypothesis it suggests.
  7. Scope of the claims (Sec. 1, Sec. 6, Limitations, FAQs). The claim that abduction is the hardest is stated for medium and hard queries of the two stronger models, the contribution is stated on LifeWorld, and the limitations and FAQs are updated with the new results.

Highlight the Contributions of the Paper

We briefly highlight the main contributions and findings of our paper.

  1. LifeWorld, a benchmark. Deduction, induction and abduction are posed on the same life-like transitions, so the three forms of reasoning are compared on the same pairs and settings. Every answer is checked exactly, each form's difficulty is tied to a known complexity result, and grid size, horizon and rule are set independently. To our knowledge, no existing benchmark has all of these at once, and LifeWorld fills this gap.
  2. LifeAgent, a recursive harness. Space–time recursion splits a forward run into small exact pieces and logical recursion turns induction and abduction into propose–then–verify. The first improves deduction in all nine settings (+0.12 to +0.30), and the second nearly doubles induction (0.34 at the first proposal to 0.63), beyond what resampling gives. The loop also helps a second model and a second level, while windows help only a model whose single windows are reliable.
  3. Findings on why models fail. Because all three forms of reasoning have exact answers on the same grids, we can locate failures by analyzing the results and comparing them across settings: i) induction breaks at chaining hidden steps, not for lack of evidence, ii) a model-based checker fails by rejecting right answers, so improving its forward run brings it within 0.02 of the simulator, iii) splitting helps only below a per-call size, and the pieces fail together, and iv) feedback repairs local errors but not coupled ones.

We believe these contributions advance our understanding of reasoning in life-like environments and provide a rigorous framework for evaluating LLMs' performance across deduction, induction, and abduction. And we thank all reviewers for their valuable feedback and constructive suggestions and we believe that the revised version addresses the major concerns raised. We look forward to any further feedback and hope that our revisions meet the expectations of the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.