- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
It proposes MEDAKA, a Means-Ends-Analysis-inspired prompting framework built on recursive goal decomposition and per-step state tracking, and LIQUIDWORLD, a 189-problem liquid-mixing testbed with five structural difficulty tags and a fully programmatic validator. Though I personally think that the "planning" framing is larger than what the testbed measures. The tasks this paper investigated is more like "optimization" problem rather than state space search problem.
Critical strengths
-
An apt use of Means-Ends Analysis. MEA is a domain-general control structure whose historical limitation was the need for hand-coded operators and difference tables. The paper recognizes that LLMs now supply exactly these missing pieces (proposing operators, identifying differences, solving preconditions algebraically), which makes the classical structure newly viable. This is a substantive conceptual contribution, not merely a prompting trick, and it suggests a general recipe for continuous-
-
Informative ablations. Each module helps independently (Table 3), and the per-tag breakdown (Table 5) matches each module's designed purpose, with the recursive module carrying the nonlinear tag.
Critical weaknesses
W1. The "planning" framing is larger than what the testbed measures
The paper presents LIQUIDWORLD as a continuous-state planning benchmark and argues against the search-based LLM planning literature on those terms. The combinatorial core of planning is largely absent from the benchmark. Pouring never changes a source vessel's composition, so the effect of any set of pours is additive and order-independent: for the 136 linear-tagged instances, any feasible assignment of pour volumes reaches the goal under any ordering, which is exactly what the LP ground truth in Appendix A.5 certifies. These instances are linear programs, the coupling instances are linear systems, and the nonlinear instances are small bilinear pooling programs. What the benchmark measures in practice is three things: translating a problem statement into conservation equations, solving them, and tracking state across many vessels. Only the nonlinear tag retains a plan-like structure, because the intermediate pool composition is a subgoal the solver must choose.
This gap between framing and measurement has consequences. Readers will expect MEDAKA to help where planning is actually hard, meaning irreversibility, dead ends, and ordering choices, yet no experiment tests that regime. The framing also undersells the cleaner finding the paper does support: structured prompting can turn a frontier LLM into a competent symbolic algebra engine with a state ledger.
Suggestion. Reposition the contribution around continuous constraint derivation rather than plan search, and state explicitly which components of classical planning difficulty LIQUIDWORLD contains (parameter derivation, state tracking) and which it does not (ordering combinatorics, irreversibility, dead ends). If the authors wish to keep the planning claim, one additional tag would substantiate it: instances where capacity limits or one-way routing make pour order matter, so that enumeration order and backtracking become genuinely relevant.
W2. No tool-augmented reference point, so the practical weight of the prompting gains is hard to judge
The exclusion of LLM-Modulo methods is stated and methodologically consistent: external solvers would perform the very derivations the paper wants to measure natively. Still, LIQUIDWORLD instances sit exactly in the class that coding agents handle well. The paper's own generator already uses an LP solver to verify linear instances, so the ground-truth pipeline is itself a solver. A current coding agent with a Python interpreter and scipy would very likely saturate the testbed, at lower latency and lower token cost than the configurations in Table 7. Without that number, a reader cannot tell whether the reported gaps (e.g., 33.3% to 87.4% on nonlinear for Claude-Sonnet-4.6) represent progress toward usable planning capability or progress within an artificially tool-free sandbox.
Suggestion. Add one tool-augmented reference configuration: an off-the-shelf coding agent (e.g., an opencode- or oh-my-pi-style harness) with a Python interpreter and scipy (linprog for linear and coupling instances, a small bilinear formulation for nonlinear ones), scored by the same validator. One number per tag suffices. This does not weaken the native-ability question; it gives that question a practically meaningful ceiling. If new experiments are not feasible, the discussion should at least name the deployment settings where tool-free planning ability is the operative quantity, so the claims are bounded accordingly.
W3. No scaling analysis, and there is a principled reason to expect prompting-based strategies to degrade faster than solver-based ones
Difficulty in LIQUIDWORLD varies by tag, but scale is never varied as an independent axis. This matters because the paper's central claim is structural: MEA-style decomposition is said to fit continuous domains better than enumerate-and-rank. A structural claim is implicitly a scaling claim, and the evidence here is thin. LLM reasoning accuracy is well documented to degrade as instances grow in objects, constraints, and context length, while a solver pipeline degrades in runtime rather than in exactness. The paper's own results point in the same direction: every method drops sharply on the multi-goal and distractor tags (Table 2), and the 100-vessel trace in Table 6 shows state tracking failing at that scale even though the underlying algebra stays simple. If the framework's edge over CoT narrows or inverts as instances grow, while an LLM + interpreter + scipy pipeline remains exact, the case for the framework weakens accordingly. This asymmetry is also the standard argument for neuro-symbolic designs over purely neural ones. I recognize the authors have scoped the paper to native ability, and I do not lower my score for that choice; I raise the concern because it determines how durable the conclusions are as coding-agent pipelines become the default way such problems are solved.
Suggestion. Run a scaling sweep on the linear tag, which the parametric generator already supports: increase the number of sources, targets, and liquids (from a handful to 100+ vessels) and report pass@1 as a function of scale for all three methods, ideally alongside the tool-augmented reference from W2. A degradation curve, even without the tool baseline, would delimit the framework's operating range. Separately, the weaker-model slots currently go to GPT-4o, LLaMA-3.3-70B, and the two small Gemma models. Adding a recent mid-size open-weight model that performs well in coding-agent settings, such as Qwen3.8-27B, would make both the native-ability claims and any tool-augmented comparison more representative of current practice.
W4. A uniform temperature=0 configuration is no longer a safe decoding default
All rollouts use temperature=0 (Appendix B.1). Greedy decoding used to be the standard reproducibility choice, but current providers publish model-specific sampling guidance and warn against deviating from it. The Qwen3.8-27B model card, for example, recommends temperature=1.0, top_p=0.95, top_k=20 for thinking mode and temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5 for non-thinking mode, and explicitly suggests a nonzero presence_penalty to suppress endless repetition. In my own engineering practice with recent instruction-tuned models, greedy decoding measurably increases repetition loops and degeneration on long structured outputs, which is precisely the output format that all three methods in this paper require.
Three consequences follow. First, the cross-model comparison is uneven: seven models from four providers are each evaluated away from their tuned operating point, and the resulting penalty is unlikely to be uniform across families. Second, the paper's strongest capability claim, that the tested open-weight models "inherently lack the reasoning capabilities" for nonlinear problems (Section 4.2), was measured under a decoding regime that model providers actively discourage, so part of the observed capability floor could be a decoding artifact rather than a property of the weights. Third, temperature=0 makes the pass@3 metric nearly degenerate, since the three rollouts differ only through serving-side nondeterminism. The oracle best-of-3 comparison used to argue compute efficiency in Section 4.2 is therefore weaker than it appears.
Suggestion. Rerun at least one frontier model and one open-weight model under the providers' recommended sampling parameters and report both configurations, or else justify temperature=0 with provider documentation for each model. Reporting pass@3 under sampled decoding would also make the compute-matched comparison meaningful. This interacts with W3: if Qwen3.8-27B is added to the model set, its published sampling guidance should be followed rather than overridden.
W5. The nonlinear results would benefit from the same failure-case transparency the paper applies to linear instances
Section 4.3 is one of the paper's strengths: it inspects failed trajectories and names concrete failure modes. That analysis, however, covers only linear-tagged problems from GPT-4o and LLaMA-3.3-70B. The nonlinear tag, which carries the paper's headline improvement, receives no comparable treatment. This leaves several things a reader cannot tell. Claude-Sonnet-4.6 still fails 12.6% of nonlinear instances, Gemini-3-Flash fails about half, and the open-weight models fail most of them even under MEDAKA. Are these failures decomposition failures (no productive subgoal found), leaf-level algebra errors, or state-tracking drift? Each diagnosis has a different implication: the first bounds the framework, the second bounds the model, the third bounds the prompting format.
There is a related reporting gap on the instance side. The nonlinear problems are hand-designed and hand-solved (Section 3.2), which is substantial work, and the validator pre-specifies the routing. What remains unreported is how the 53 instances distribute over difficulty-relevant structure: how many actually require the recursive subgoal mechanism, and how many would yield to sequential single-gap derivation. The tag criterion in Table 1 certifies bilinearity, not decomposition depth, so readers currently cannot calibrate how far the 33.3% to 87.4% result generalizes beyond the instances that were sampled.
Suggestion. Extend the Section 4.3 failure study to the nonlinear tag: categorize MEDAKA's remaining failures, and the open-weight models' dominant failures, into decomposition, derivation, and state-tracking causes, with example traces in the style of Table 6. Reporting a simple decomposability characterization of the 53 instances alongside would further help readers calibrate the result. If the failures turn out to concentrate on instances that resist decomposition, stating that explicitly would give the structural-fit claim a well-defined boundary, which I see as strengthening the paper rather than weakening it. Optionally, a small stress set of solver-verified harder instances (rather than hand-verified ones) would quantify this boundary, but the failure analysis alone would already address the main gap. This complements W3, which covers the size axis, with the structure axis.
Questions for authors
S1. Attribution of the gains: a method-primed baseline. The paper's ablations show that the "algebra-aware CoT" configuration (−Recursive Module) already reaches 91.0/75.1 on Claude-Sonnet-4.6/Gemini-3-Flash, suggesting that a large share of MEDAKA's gains comes from instructing explicit algebraic derivation rather than from the MEA control structure itself. Mixture and mass-balance problems are standard textbook material, and the failure traces in Table 6 suggest models possess the relevant knowledge but do not invoke it under vanilla CoT. I suggest adding a method-primed CoT baseline that names the problem class and method up front (e.g., "this is a mass-balance problem; set up the component conservation equations and solve them before committing to any pour"), without any MEA machinery. This condition would isolate what the MEA structure adds beyond schema activation, and would sharpen the paper's central claim that means-ends decomposition is the load-bearing ingredient rather than generic algebraic prompting.