- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a hybrid LLM-numeric search planning framework called TRACS. It essentially pairs a classical numeric planner with a fine-tuned LLM: planners run as usual, and cheap stagnation detectors flag when search is stuck. At those moments, an LLM proposer suggests trajectories from the current state. The planner replays each suggestion, keeps only the legal prefix, and feeds it into its eager/lazy heuristic search. This paper seeks to improve search performance based on coverage, solving success/plan quality (Q), and expansion reduction. The proposer (qwen3.5-9B) was trained on pieces of solver-verified solution with SFT and then with DAPO (GRPO-style) reinforcement learning. The paper tests 2 TRACS variants against 5 baselines: 2 native planners (TRACSEager against NLM-CutPlan and TRACS-Lazy against Count Downward) and 3 LLMs (kimi-k2.6, deepseek-v4.1-flash, and glm-4.7. Domains used were IPC DEPOTSNUM and ZEROTRAVEL-NUM.
Critical strengths
- The experimental performance of TRACS-Lazy is strong. It achieves 100% coverage across both evaluation domains, compare to 88.2% for the baseline Count Downward method.
- This paper fits well with the LLM-Modulo direction the planning community has been moving towards (Kambhampati et al., 2024), just in a different context. The LLM only proposes, and the planner replays each action and admits just the legal prefix. Using valid but incomplete prefixes as guidance is a smart design choice, and it applies more widely than methods that need complete LLM plans. The paper plays to complementary strengths instead of forcing an LLM into planning, which makes sense given the decades of planning tools / infrastructure we already have.
- Empirical gains are large where they should be. TRACS-Lazy solves every test problem in both domains. At Depots-scale-30 (Table 1), it solves 50/50 against 39/50 for Count Downward, Q goes from 0.71 to 0.93. At scales 30-40 (Table 2) it needs roughly ~21-27% fewer expansions at matched plan quality. The pairedexpansion analysis reduces the dependence on hardware speed. The ablations also support the design choices.
Critical weaknesses
- One of my main concerns is that the evaluation is narrower in domain count and may lack comparison on standardized public sets. The source of the Depots-num and Zenotravel-num domains have additional numeric domains in their public sets and results on several of these are reported to show generalization. The instances used in the paper are self-generated which is required in large quantity for LLM-proposer training. Per scale, the test sets are larger than the official IPC sets. However, there are distributional differences, e.g., in-solver evaluation stops at 40 entities, whereas the official sets reach 70-80 objects / entities (IPC 2002 Depots, IPC 2023 ZenoTravel). On the official ZenoTravel set, coverage is already saturated [5] to complete coverage. Thus reporting the performance of TRACS on public sets of these and additional domains could better contextualize the contribution.
- The gains may come from the intervention, i.e. detect a plateau / stagnation, then inject a validated trajectory from an assisting mechanism. Injecting trajectories at stagnation points has some existing LLM-free techniques, e.g. random-walk explorations [1,2] in propositional planning domains structurally show almost the same design. Closer to numeric planning, random walks are also shown to work on integer-based problems [3] through numeric to propositional encoding. Where propositional encoding is not practical, TRACS own LLM-based trajectory generation and replay steps can be replaced with random walks (or relaxed-plan prefixes) utilizing the planner's numeric successor generator. Similarly, LLM-based search initializations or one-time LLM-generated heuristics are shown to have strong performance on domains including Zenotravel in numeric setting [4]. Without such controls, the paper does not propoerly contextualize the role and extent of LLM’s contribution.
- TRACS and the native planner baselines ran on different CPUs and allocations (TRACS also used a GPU). The wall-clock results therefore are a bit unclear, and the expansion counts excluded inference and replay costs. The throughput and paired expansion cutoffs help, but overall, it might need more clarification.
- There are no experiments injecting non-LLM guidance (e.g., random valid rollouts) at the same trigger points. It’s unclear how much really comes from the trained proposer and how much from injecting extra states. Have the authors tried this, for example with the untrained base model in the search?
- Only 2 domains were tested, each with its own specific proposer (+ ablations are only on Depots). The scale-40 results rest on 13-15 problems, and the matched lazy comparison there has n=4. The seeded reruns vary a lot at scale-40, and the eager gains are small and mixed.
[1] Nakhost et al. Monte-Carlo Exploration for Deterministic Planning. IJCAI 2009.
[2] Xie et al. Adding Local Exploration to Greedy Best-First Search in Satisficing Planning. AAAI 2014.
[3] Nakhost et al. Resource-Constrained Planning: A Monte Carlo Random Walk Approach. ICAPS 2012.
[4] Tuisov et al. Successor-Generator Planning with LLM-generated Heuristics. AAAI 2026.
[5] Scala et al. On Using Lazy Greedy Best-First Search with Subgoaling Relaxation in Numeric Planning Problems. ICAPS 2025.
Questions for authors
- Does the gain come from the trained proposer or from injecting extra valid states at the trigger points? I did not find a control that injects non-LLM guidance. I also did not see the untrained qwen3.5-9B base used inside the search. Could you add either one? “Windowed-random” still queries the trained model.
- The proposer is trained on states from solved solution paths but queried at stagnation states. Do you have an analysis of how different these two distributions are (for example, by comparing usable-prefix rates on actual stagnation queries?)
- Could the authors explain why they specifically chose DEPOTS-NUM and ZENOTRAVEL-NUM as the 2 IPC domains for experiments/training? They partially explain in Appendix D, but it only supports the theory and not selection rationale. It can give insight on what other domains can be used for future work.
- With the same detectors, budgets (B, H, K) and priority queue, how does TRACS perform when the injected trajectories come from other exisitng trajectory proposing mechanisms like random applicable walks without state revisits or relaxed-plan lookahead prefixes. A clear advantage for the trained proposer would substantially raise my score.
- Can you report results on the official instances (IPC 2002 Depots-numeric, IPC 2023 ZenoTravel), and potentially on at least a few more numeric domains, under the IPC protocol? Where coverage is saturated, please include additional metrics that are already reported in current work e.g. plan quality.
Thank you for the feedback. We added experiments in Section 5.3 to clarify when trajectory guidance helps and how far its benefits extend.
Q1 and Q4. Transferring the DEPOTS-NUM-trained proposer to ZENOTRAVEL-NUM yielded no observed gain over Count Downward (Table 4). This suggests that the gains depend on the guidance the proposer can supply in the target domain. In the DEPOTS-NUM offline evaluation, DAPO reached the goal more often than both SFT and Qwen3.5-9B Base (Table 3(a)). In the separate seed-101 solver comparison, DAPO achieved higher average plan quality than SFT over the same search expansion intervals (Appendix C.7). The partial-only control retained most of the coverage gain after filtering out all goal-reaching proposals. Under the same per-phase sampling cap, adaptive TRACS achieved higher coverage than the initial-only control, which queried only the original initial state at the start of each search phase (Table 3(b)). Comparisons with the base model or non-LLM trajectory generators within the same search setup remain open.
Q2. On stagnation-triggered queries at scales 30–40, about 79–89% of TRACS–Lazy generations yielded a nonempty, executable prefix without state revisits across the two domains (Table 10). This shows that training on states sampled from verified solution paths can yield usable guidance at search-time stagnation states. The difference between the two state distributions remains unquantified.
Q3. The choice was pragmatic. We selected two established numeric planning domains with different demands. DEPOTS-NUM couples stack rearrangement with truck capacity constraints, and ZENOTRAVEL-NUM couples passenger routing with fuel constraints. Each TRACS variant and its native counterpart were evaluated on the same held-out instances. We have included the PDDL instance generators in the supplementary material.
Q5. Evaluation on official IPC instances and additional numeric domains remains open. On newly generated DEPOTS-NUM instances at scales 45 and 50, the full TRACS–Lazy solver improved both coverage and plan quality over Count Downward without additional proposer training (Table 3(c)).