- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why LLM accuracy depends on the order in which related facts appear in a prompt. It frames an ordering by three properties: proximity (how far apart dependent facts sit), precedence (whether a fact appears after the facts it depends on), and position (where a fact sits in the context). Proximity is modeled as a minimum linear arrangement (MinLA) cost over a dependency DAG, with precedence as a hard legality constraint. Several order families are compared on two synthetic domains: arithmetic problems in GSM8K's step format and narrative word problems. On Llama-3.2-3B-Instruct, the legal cost-optimal order beats the natural order by 34.5 points (arithmetic) and 16.3 points (narrative). The authors decompose this gain into clustering and direction components with domain-dependent shares. Further checks cover Qwen3 from 4B to 32B, Llama-3.1-8B.
Critical strengths
- It tackles an important, known but unexplained problem.
- Order sensitivity in prompts is widely observed but not well explained.
- Previous work has attempted explanations, but an additional, structural view of the phenomenon is useful to the community.
- A good experimental design for separating factors.
- Building prompts from dependency-annotated facts lets proximity and precedence be manipulated as formally separate properties of an ordering, rather than searching over unstructured permutations as prior ordering work does.
- This makes it possible to ask which property of an arrangement drives the effect.
Critical weaknesses
[W1] The main findings rest on a single small model, and the effect seems to largely disappear on other models
- The central analyses all come from Llama-3.2-3B-Instruct.
- The authors acknowledge this but the cross-model check covers only the order-family comparison, and there the effect is much smaller.
- For example,
- On Qwen3-8B, free outperforms natural (0.915 vs. 0.897, Table 4), contradicting the statement at l.322 that pi_free does not outperform pi_natural.
[W2] The paper is framed as analysis but argued with method-style evidence
- When I read the introduction, the paper positions constrained as an oracle for analysis, not as an ordering method.
- Much of its supporting evidence, however, is method-style accuracy rankings.
- It does not show that proximity and precedence explain the observed variation in order effects, which is the paper's stated goal (Line 054–056).
[W3] The introduction frames the paper as ICL, but the experimental setup is CoT-like reasoning. Therefore, a reasoning-required benchmark seems a poor fit
- The introduction frames the paper as a study of in-context learning, and its related work and baselines are ICL demonstration-ordering methods. ICL is generally based on input–output demonstration pairs.
- The experiments, however, and especially the experimental setup, are not ICL but closer to chain-of-thought (CoT) style reasoning
- Each prompt contains the solution steps of a single problem ("one fact per step, each step's own computed value masked out"), and the model works through them to compute one answer.
- The paper acknowledges that this differs from the few-shot setting: "Unlike the classical few-shot setting, where each demonstration is independent and self-contained, the facts in F are the constituent steps of a single reasoning chain" (Line146–149).
- The findings therefore concern the ordering of "reasoning steps" within a single problem, and say little about the ICL demonstration ordering that the introduction sets out to explain.
- This concern is also related to the choice of a GSM8K-style, reasoning-required task.
- Fact-linked benchmarks, in which the facts have explicit dependency structure and come with distractors, would fit the paper's idea better and would separate fact use from computation.
[W4] The synthetic datasets lack detailed statistics and verification
- The generators carry the paper's entire empirical weight, yet they are described in a few sentences.
- The authors should report, for example:
- distributions of critical-chain length, number of distractor chains, edges per instance, and answer range
- how narrative sentences are produced (templates or an LLM), and how lexical or entity overlap between distractors and the critical chain is controlled
- how gold answers and dependency annotations were verified for correctness and unambiguity;
[W5] The arithmetic-domain weighting w_ji is not justified
- In the arithmetic domain, the cost uses w_ji.
- The constants 1.0 and 0.3, the max operator, and the character-4-gram similarity are neither justified nor ablated.
[Minor] There are too many mathematical formulations, which make the paper hard to read
- The formalism could be simpler.
Questions for authors
See weaknesses