- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The submission combines two studies. The first revisits three earlier comparisons between autoregressive and iterative systems, on code repair, GF(2) reasoning and a minimal planning task, each of which failed for a different reason, and distills from them six admission gates intended to indicate when a performance difference can be interpreted. The second introduces VMGR for Blocksworld with two-action macros. Verified plans from two constructors are merged into a budget-indexed graph, completion costs are obtained by dynamic programming, and Qwen2.5-0.5B is fine-tuned with LoRA on cost-weighted soft targets, with a control that uses uniform targets over exactly the same candidates. The weighted policy is about 3.6 points better on 768 new tasks. A fallback search added later brings both policies close to 100 percent on the 600 public tasks and nearly removes the difference, while on the harder 110-task configuration the weighted policy leads by roughly ten points and uses less inference compute.
Critical strengths
- The statistical reporting is exceptionally careful. Paired fits, task-level bootstrap intervals, leave-one-fit-out analyses and seed sensitivity checks are all provided, and negative or inconclusive results are retained rather than omitted. Holding the candidate set fixed while changing only the target weights also isolates the effect of cost weighting from that of coverage, a confound that genuinely arises in this setting.
Critical weaknesses
- The scope is very narrow and the main effect is fragile. All experiments use a single 0.5B model released in 2024, one toy domain, two-action macros and three training runs. The authors acknowledge that exponentiated cost weighting is well established, citing RAML and AWR, so the novelty lies only in applying it to a verified, fixed support. The headline improvement of 3.6 points has a seed-level interval that includes zero, and one of the three fits shows no gain at all. The permutation control changes sign once the strongest fit is removed, and the fallback search all but eliminates the difference on the public tasks. No classical planner is included, although Blocksworld is easily solved by one, and no stronger language model is evaluated. As the authors concede, the SFT and feedback baselines that solve almost none of the hard tasks are not competent references, so the large system-level gaps carry little information. Beyond this specific configuration, the paper offers limited insight into generative reasoning.
- The manuscript is very difficult to read, and its two parts are only loosely related. The title and abstract foreground the admission gates, yet the gates were derived after the fact from unsuccessful studies, are never validated, and play no essential role in VMGR. The main text is dominated by resource accounting, seed identifiers and qualifications about what each result does not show, while essential methodological details are divided between Section 3 and the appendix. Even the abstract is hard to follow without prior knowledge of the project. In its current form the paper reads as an internal research record rather than a conference submission, and it would be much stronger if split into two papers or refocused on a single contribution.
Questions for authors
- Given that the seed-level interval includes zero, could the authors train a larger number of fits, for example ten, to establish whether the effect holds across training seeds? This is the result most likely to change my assessment.
- How sensitive are the results to the floor and temperature parameters, which Section 7 notes were not varied?
- Does the improvement persist with a larger or more recent backbone, or in a different planning domain with longer macros?
- Which conclusions about VMGR, if any, depend on the admission gates introduced in Section 2?