- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper's main claim, as framed, is that training an LLM on diverse linearizations of a partial-order plan gives it high entropy over the plans it generates. Framed this way, the finding is trivial.
I personally feel the authors' choice of storyline is a pity. Pls see W4
Critical strengths
Good problem setup and solid experiments, code is provided
Critical weaknesses
W1. The likelihood metrics are relative within the valid set; the paper never measures how much probability mass the valid set actually carries.
Both headline likelihood metrics are defined relative to the valid linearizations. renormalizes likelihoods inside before computing entropy, and compares a valid order only against an invalid one produced by swapping two adjacent, directly connected nodes. Neither measures the total probability the model assigns to valid sequences as opposed to invalid or malformed ones. The main interpretation of the key result, that increasing "mainly redistributes probability among valid alternatives" (Section 5.1), is therefore an inference from relative scores rather than a measurement. A model could assign high, evenly spread likelihood within while placing most of its mass elsewhere, and Table 4 would look exactly the same.
The generation results partially close this gap on Blocks World, where the seven models with high generation validity show invalid-order rates of 0.1% to 4.2% at = all (Table 9). But that check is behavioral: two samples per plan, temperature 1, unshuffling format only. On proScript the same check fails outright, with as low as .067 and malformed outputs near 30% (Table 12). So for the natural-language domain, where the paper still claims likelihood-diversity gains and cross-domain transfer, the evidence rests almost entirely on the relative metrics above.
Two additions would make the claim direct. First, score a broader invalid set (multi-edge violations, long-range reorderings) so that tests more than single-edge sensitivity. Second, estimate the total mass on valid sequences, either from a larger sampled set with exact validity checking or by summing likelihoods over a bounded candidate set. This matters most for proScript and the recipe-domain transfer, where generation validity is weak and the entropy increase is the main positive result.
W2. The intermediate- evidence behind the paper's most actionable claim covers only the two smallest models, and for generation effectively one.
The claim that much of the diversity gain "already appears with only a few valid training orders per plan" is the result a practitioner would act on, since = 2 is the cheapest possible intervention. Its support is narrow. Only GPT-2 (124M) and SmolLM-135M cover (Table 3).
Extending the = 2 condition to the remaining seven models, for both likelihood and generation, would either solidify the claim or delineate where it stops. To the authors' credit, the abstract scopes the few-orders claim to the two tested models; the issue is that the paper's practical significance rests on exactly this claim, so its evidence base deserves the same nine-model coverage as the = 1 vs = all comparison.
W3. The related work misses the closest methodological previous work literature, which changes how the contribution should be framed.
He et al., "Order Doesn't Matter, But Reasoning Does: Training LLMs with Order-Centric Augmentation" (EMNLP 2025). That work constructs a DAG over reasoning steps, generates valid reorderings via topological sorting, and trains LLMs on them as SFT augmentation. This is the same training mechanism the present paper varies with , applied to logical reasoning with accuracy rather than diversity as the goal. The introduction's framing, "We do this by varying the number of valid linearizations shown to a model," reads as if the training scheme itself is new. It is not. What is new is the diversity and likelihood-invariance objective, the dose-response analysis over , and the probing and steering experiments. The contribution should be framed accordingly, and the two works should be compared directly.
W4. The headline contrast is the least informative result in the paper, and the one genuinely surprising observation is reported without any mechanistic analysis.
The central narrative of the paper is the contrast between collapse at = 1 and recovery at = all. The first direction is close to a tautology: maximum-likelihood training on a target distribution that places all mass on one order per plan will, given sufficient capacity, reproduce that concentration. The second direction follows from mass covering over the now-broadened target. Both are what standard supervised training predicts given how the data pipeline is constructed, and the paper's framing gives them the abstract, the first table, and the section lead.
The observation that does not follow from these considerations is the saturation at = 2 (Table 4b). Fitting a training distribution supported on two orders per plan yields per-plan entropy of at most ; for Blocks World, where the median is 3 (Table 2), that bound is 0.63, yet the measured is 0.937. The model therefore assigns substantial likelihood to valid orders it never saw in training, which means the intervention works through generalization of an order-symmetry rather than through coverage of the shown orders. The paper neither performs this ceiling analysis nor offers or tests any mechanism for the saturation. Natural hypotheses are available and testable with the paper's own tools: the incomparable-swap bias in Tables 8 and 11 suggests the model learns local pairwise interchangeability rather than global uniformity, and stratifying the sweep by would show whether saturation requires larger on plans with wider partial orders. As written, the most theoretically interesting number in the paper is presented as a side remark.
The missing mechanism question has a sharp edge that the paper is well positioned to address but does not. Wang et al. (2026), cited by the authors, show that policy-gradient training collapses solution diversity on Blocks World even when the reward covers all valid orders, a regime in which the data-degeneracy explanation offered here cannot apply. The supervised collapse studied in this paper and the RL collapse reported there therefore have different sufficient causes, and contrasting the two regimes would isolate what actually drives order concentration: whether a model trained with resists RL-stage collapse, and whether the same saturation at small appears when the objective is mode-seeking rather than maximum likelihood, are questions that would turn this paper's empirical observation into a mechanistic claim about training objectives (see also S1).
Questions for authors
S1. Test whether multi-linearization training protects against diversity collapse under RL. The paper attributes the SFT-stage collapse to the data side: a degenerate per-plan target fitted by maximum likelihood. Wang et al. (2026), cited in the paper, show that policy-gradient training collapses solution diversity on Blocks World even after training accuracy saturates, which is a mode-seeking effect of the RL objective itself and cannot be fixed by data augmentation alone. The natural joint question is whether a model first trained with retains its entropy over valid orders when subsequently optimized with RL, or whether the RL stage re-collapses the distribution onto a single order. A minimal version needs no new infrastructure: take the = 1 and = all BW checkpoints and apply the same policy-gradient recipe with validity as the reward, tracking and over training. The outcome is informative either way: if multi-linearization SFT confers lasting resistance, it becomes a pre-RL intervention worth adopting; if not, the paper's remedy is bounded to supervised stages and that boundary is worth stating.
S2. Ground the mechanism discussion in the forward-KL/reverse-KL framing of post-training. The paper's two collapse regimes map cleanly onto an existing analysis: SFT is maximum likelihood, i.e. forward KL against the empirical target, which is mass-covering over whatever support the data provides, while RL fine-tuning behaves as reverse KL against a reward-defined target, which is mode-seeking. Chen et al., "Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting" (arXiv:2510.18874), develop exactly this SFT-as-forward-KL / RL-as-reverse-KL model of post-training and would give the discussion section a vocabulary for the contrast raised in W4: the collapse studied here is a property of the data the objective is applied to, whereas the collapse in Wang et al. (2026) is a property of the objective itself. Citing this line of work would also connect the = 2 saturation result to a concrete open question that the framing makes precise: why generalization, rather than coverage, does the work once the empirical support contains more than one order.