- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To examine whether models genuinely possess planning ability rather than rely on memorization of training data, the authors study the suitability of the Countdown arithmetic game as a planning benchmark and provide a computational complexity analysis. They then construct the CD dataset by sampling random arithmetic trajectories from a given set of input numbers and selecting the least frequently observed outcome as the target, and compare it with other datasets, including Reasoning-Gym (RG). Finally, they evaluate symbolic planning and LLM planning methods, observing two phase transitions in difficulty and finding that AutoToS and symbolic search methods perform best overall.
Critical strengths
-
The authors highlight limitations of existing benchmarks in evaluating models and LLM agents on long-horizon, sequential, multi-step tasks. They show that solving larger instances does not necessarily indicate stronger planning ability, as those instances may simply be easier to solve.
-
The paper provides a computational complexity analysis of Countdown to support its suitability as a planning benchmark, together with an instance-generation procedure that enables evaluation on fresh instances. It also reveals a non-monotonic relationship between input size and task difficulty, offering important insights for benchmark design and the interpretation of evaluation results.
Critical weaknesses
-
The paper evaluates multiple models and prompting methods, but its error categories do not clearly distinguish misunderstanding the task rules, arithmetic errors, and difficulty finding the correct sequence of operations. It therefore remains unclear to what extent Countdown accuracy reflects planning ability rather than arithmetic proficiency and rule following.
-
The authors attribute the hard-to-easy transition to the increased availability of intermediate zero values, but this explanation lacks direct experimental validation. Does the pattern persist when intermediate zero values are disallowed or the input-number range and instance-generation procedure are changed? The effects of instance distribution, search procedure, and resource limits remain unresolved, limiting the generality of the claimed difficulty pattern.
Questions for authors
-
Could the models’ better performance on the 24 Game dataset simply reflect easier instances than those in CD, rather than prior exposure to the static dataset? To distinguish these explanations, could the authors compare performance on instances matched as closely as possible in input-number range, target value, and number of solutions?
-
Could the error analysis more clearly distinguish arithmetic errors, failures to track which numbers remain available, and difficulty finding the correct sequence of operations?
3 comments
Q1. On better performance on the 24 Game dataset:
Thank you for this suggestion. We ran the comparison. We generated new size-4 instances matched to the 24 Game instances pair by pair: the same input-number range (1-13) and the same number of solutions, differing only in the target (never 24). We also evaluated Reasoning Gym instances of size 4, which are unseen and have many more solutions than the 24 Game. On the matched instances, the LLM planning methods perform as well as on the 24 Game, and on Reasoning Gym they perform better. Accuracy thus seems to be better correlated with the number of solutions of the instances than with their prior availability. The reviewer is right that the gap we reported is likely explained by instance difficulty, not by exposure during training. We will revise the corresponding claims in the contributions (Section 1), in Section 6.2 and in the conclusion accordingly. We believe this strengthens the case for our generation procedure: by selecting rarely reached targets, it produces instances with far fewer solutions than existing collections, and these instances are much harder for all tested LLM planning methods.
Q2. On error analysis:
Thank you, this is a useful suggestion. Two of the three are already separate categories: failures to track which numbers remain available are "Unknown Number Used", and difficulty finding the correct sequence of operations is "Not Target Number". Arithmetic errors were not a separate category in the submitted version, they were absorbed into Unknown Number Used and Not Target Number. We will add an "Arithmetic Error" category (a step whose stated result differs from the true value, allowing for rounding of fractions) and report it for IO, CoT and ToT. Re-running the analysis on our logs, arithmetic errors are rare compared to the other two. The table below gives, for each LLM and method, the number of errors of each category and its share of all errors (700 CD instances of sizes 4-10, 5 attempts each; an attempt can contain several errors):
| All errors | Arithmetic error | Unknown number used | Not target number | |
|---|---|---|---|---|
| Qwen2.5 IO | 9,897 | 7 (0.07%) | 3,658 (36.96%) | 791 (7.99%) |
| Qwen2.5 CoT | 6,754 | 66 (0.98%) | 1,176 (17.41%) | 455 (6.74%) |
| Qwen2.5 ToT | 3,913 | 82 (2.10%) | 274 (7.00%) | 3,358 (85.82%) |
| DeepSeek V3 IO | 8,157 | 32 (0.39%) | 3,162 (38.76%) | 535 (6.56%) |
| DeepSeek V3 CoT | 8,385 | 45 (0.54%) | 2,170 (25.88%) | 659 (7.86%) |
| DeepSeek V3 ToT | 5,072 | 8 (0.16%) | 385 (7.59%) | 3,110 (61.32%) |
| Llama 405B IO | 5,336 | 1 (0.02%) | 1,196 (22.41%) | 60 (1.12%) |
| Llama 405B CoT | 5,242 | 5 (0.10%) | 761 (14.52%) | 121 (2.31%) |
| Llama 405B ToT | 4,004 | 125 (3.12%) | 272 (6.79%) | 3,328 (83.12%) |
Arithmetic errors account for at most 3.12% of all errors. Arithmetic can only be checked where the model states the result of a step: ToT does so for 96-100% of steps and CoT for 26-83%, while IO outputs mostly list operations without results (2-38% of steps), so the IO numbers are lower bounds. For ToT, Not Target Number is by far the largest category (61-86% of all errors): ToT executes steps correctly but fails to find a sequence that reaches the target, which is a search failure rather than an arithmetic or rule-following one. The remaining errors fall into the other categories of the paper (mostly Incorrect Format for IO and CoT). In the revision, Figure 6 will show the full breakdown over all categories as stacked bars, and the appendix will give it per input size.
As the discussion period ends soon, we would like to briefly follow up on our response. Following your suggestions, we ran both experiments you proposed: (1) Disallowing intermediate 0 values and multiplication or division by 1 removes the hard-to-easy phase transition (W2). (2) On size-4 instances matched to the 24 Game in input range and number of solutions, the LLM planning methods perform as well as on the 24 Game, so the gap we reported is likely explained by instance difficulty rather than exposure (Q1). We also added a separate Arithmetic Error category to the error analysis (Q2).
We have uploaded a revised version of the paper. It reports experiment (1) in Section 8 (Table 3), with the open question of why search collapses under the restricted rules discussed as future work in Section 9. It also removes the claims attributing the gap on the 24 Game to memorization (Section 1, Section 6.2 and the Conclusions). Section 7 has the new Arithmetic Error category, with the full breakdown in Figure 6, and Appendix B gives the error counts per input size.
Your suggestions and the resulting experiments have strengthened the paper: they sharpened our understanding of the hard-to-easy transition and clarified what drives the gap on the 24 Game. Since both suggested experiments were run and the revised paper reflects their outcomes, we would be grateful if you would consider updating your assessment. We are happy to clarify anything further in the remaining time.
Helpful comments and suggestions
We thank the reviewer for the detailed comments and for the two concrete experiments suggested, both of which we ran and which sharpened our understanding of the domain. We hope that we can now alleviate your concerns and we welcome further discussion.
W2. "hard-to-easy transition":
We ran this experiment: we forbid the operations that let leftover numbers be absorbed, namely no operation may take 0 as an operand and no number may be multiplied or divided by 1 (equivalently, no intermediate 0 is created, since it could never be used afterwards). We ran the same blind DFS twice on the CD instances, under the same time and memory limits as in Figure 4: once with the standard rules and once with the restricted ones. Under the restricted rules, the hard-to-easy transition disappears:
With the standard rules, DFS shows the same pattern as AutoToS in Figure 4: easy, then hard, then easy again. With the restricted rules, small instances are barely affected (99% at sizes 4-9; the 6 unsolved instances have no solution under the restricted rules, which we verified by exhaustive enumeration), but coverage keeps falling with size instead of recovering: 0.6% (21 of 3,400) at sizes 17-50, against 88.4% with the standard rules. Consistently, 2,992 of the 3,004 standard-rule solutions at sizes 17-50 pass through an intermediate 0. This confirms that intermediate 0 values are behind the second phase transition.
The mechanism, however, may be more subtle than our original explanation suggests: we counted the solutions exhaustively for sizes 4-7 and estimated them for sizes 8-20 using Knuth's estimator of search-tree size (Knuth, 1975), and according to these estimates the restriction removes only a moderate fraction of the solutions (the median fraction removed grows from about 10% at size 8 to about 40% at size 20), which may not be enough to explain the collapse of the search on its own. Understanding why the remaining solutions become so much harder to find deserves its own investigation. The revision will report this experiment in Section 8, and will revise our explanation accordingly. We use the restricted definition as a diagnostic, not as a proposed rule change; under the standard rules, our recommendation to evaluate on sizes 4-10 stands.
On generality: the pattern is not specific to one instance distribution or one search procedure. It appears on three datasets created by different generation procedures (our CD, Reasoning Gym, and Stream-of-Search; Figure 4), and with two different search procedures (blind DFS in AutoToS and heuristic GBFS in ENHSP; Figure 3), at around the same input sizes. Regarding resource limits: all runs use the same limits (30 minutes and 3.5 GB per run, Section 6), and the two planners fail for different reasons, ENHSP by running out of time and DFS by running out of memory (Section 6.1), yet both show the transition at around the same sizes, so it is not tied to one particular limit. The experiment above was run under the same limits, and only the restriction removed the recovery.
References
D. E. Knuth. Estimating the efficiency of backtrack programs. Mathematics of Computation, 29(129):121-136, 1975.