acceptodds
Withdrawn from ICLR 2027

Proximity, Precendence and Position: A Minimum Linear Arrangement Account of Ordering Effects in LLM Prompts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In-context learning is highly sensitive to the ordering of demonstrations, but the root factors of context ordering remain undiscovered. Existing work treats a demonstration set as an unordered bag and searches over permutations based on a certain metric, without modeling the underlying relational structure. We give a structural account of order sensitivity built on three key properties of an arrangement: proximity (how far apart dependent facts sit), precedence (whether each fact appears after the facts it depends on), and position (where a fact sits in the context). We model proximity and precedence using minimum linear arrangement (MinLA), with precedence as a hard constraint and isolate position through controlled experiments. We treat the MinLA-constrained order as an oracle and measure how much of the total order effect it recovers relative to the worst achievable order. This provides a direct measure of how well our account explains observed behavior. On small-scale instances with known ground-truth ordering accuracy, it recovers 81% of the order effect in the referential domain and 63% in the associative domain. Across two domains with different dependency semantics — referential (GSM8K arithmetic, where a step reuses an unresolved prior value) and associative (synthetic narratives, where a sentence names a concept introduced earlier) — orders that are both MinLA-optimal and precedence-legal outperform the natural order by +34.5% accuracy in arithmetic and +16.3% in narrative on Llama-3.2-3B-Instruct, while MinLA optimization without the legality constraint does not reliably help. Decomposing the order effect into clustering (proximity) and direction (precedence), direction accounts for 61% of it in arithmetic and clustering for 56% in narrative, consistent with a referential-versus-associative account of how each domain encodes dependency. A third, independent, U-shaped position effect, isolated under experimental control, connects our results to lost-in-the-middle phenomena reported in previous work only observationally.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, demonstration ordering, minimum linear arrangement, dependency graphs, precedence constraints, position bias

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aQGuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why LLM accuracy depends on the order in which related facts appear in a prompt. It frames an ordering by three properties: proximity (how far apart dependent facts sit), precedence (whether a fact appears after the facts it depends on), and position (where a fact sits in the context). Proximity is modeled as a minimum linear arrangement (MinLA) cost over a dependency DAG, with precedence as a hard legality constraint. Several order families are compared on two synthetic domains: arithmetic problems in GSM8K's step format and narrative word problems. On Llama-3.2-3B-Instruct, the legal cost-optimal order beats the natural order by 34.5 points (arithmetic) and 16.3 points (narrative). The authors decompose this gain into clustering and direction components with domain-dependent shares. Further checks cover Qwen3 from 4B to 32B, Llama-3.1-8B.

Critical strengths
  • It tackles an important, known but unexplained problem.
    • Order sensitivity in prompts is widely observed but not well explained.
    • Previous work has attempted explanations, but an additional, structural view of the phenomenon is useful to the community.
  • A good experimental design for separating factors.
    • Building prompts from dependency-annotated facts lets proximity and precedence be manipulated as formally separate properties of an ordering, rather than searching over unstructured permutations as prior ordering work does.
    • This makes it possible to ask which property of an arrangement drives the effect.
Critical weaknesses

[W1] The main findings rest on a single small model, and the effect seems to largely disappear on other models

  • The central analyses all come from Llama-3.2-3B-Instruct.
  • The authors acknowledge this but the cross-model check covers only the order-family comparison, and there the effect is much smaller.
  • For example,
    • On Qwen3-8B, free outperforms natural (0.915 vs. 0.897, Table 4), contradicting the statement at l.322 that pi_free does not outperform pi_natural.

[W2] The paper is framed as analysis but argued with method-style evidence

  • When I read the introduction, the paper positions constrained as an oracle for analysis, not as an ordering method.
    • Much of its supporting evidence, however, is method-style accuracy rankings.
  • It does not show that proximity and precedence explain the observed variation in order effects, which is the paper's stated goal (Line 054–056).

[W3] The introduction frames the paper as ICL, but the experimental setup is CoT-like reasoning. Therefore, a reasoning-required benchmark seems a poor fit

  • The introduction frames the paper as a study of in-context learning, and its related work and baselines are ICL demonstration-ordering methods. ICL is generally based on input–output demonstration pairs.
  • The experiments, however, and especially the experimental setup, are not ICL but closer to chain-of-thought (CoT) style reasoning
    • Each prompt contains the solution steps of a single problem ("one fact per step, each step's own computed value masked out"), and the model works through them to compute one answer.
    • The paper acknowledges that this differs from the few-shot setting: "Unlike the classical few-shot setting, where each demonstration is independent and self-contained, the facts in F are the constituent steps of a single reasoning chain" (Line146–149).
    • The findings therefore concern the ordering of "reasoning steps" within a single problem, and say little about the ICL demonstration ordering that the introduction sets out to explain.
  • This concern is also related to the choice of a GSM8K-style, reasoning-required task.
  • Fact-linked benchmarks, in which the facts have explicit dependency structure and come with distractors, would fit the paper's idea better and would separate fact use from computation.

[W4] The synthetic datasets lack detailed statistics and verification

  • The generators carry the paper's entire empirical weight, yet they are described in a few sentences.
  • The authors should report, for example:
    • distributions of critical-chain length, number of distractor chains, edges per instance, and answer range
    • how narrative sentences are produced (templates or an LLM), and how lexical or entity overlap between distractors and the critical chain is controlled
    • how gold answers and dependency annotations were verified for correctness and unambiguity;

[W5] The arithmetic-domain weighting w_ji is not justified

  • In the arithmetic domain, the cost uses w_ji.
    • The constants 1.0 and 0.3, the max operator, and the character-4-gram similarity are neither justified nor ablated.

[Minor] There are too many mathematical formulations, which make the paper hard to read

  • The formalism could be simpler.
Questions for authors

See weaknesses

Reviewer_xKQcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the ordering of dependency-related facts in LLM prompts affects reasoning performance. To this end, the authors represent the relationships between facts using dependency graphs and introduce Minimum Linear Arrangement (MinLA) to minimize the distance between dependent facts. Meanwhile, precedence constraints are incorporated to preserve the logical ordering of these dependencies. The proposed approach is then evaluated on synthetic arithmetic and narrative tasks, where different ordering strategies are compared. The experimental results show that combining MinLA with precedence constraints can improve reasoning performance. Furthermore, the authors analyze how proximity, precedence, and position contribute to ordering effects in LLM prompts.

Critical strengths

1.The authors introduce Minimum Linear Arrangement (MinLA) with precedence constraints to model the proximity and ordering of dependent facts. Although MinLA itself is a well-established optimization problem, applying it to dependency-aware prompt ordering provides a clear and structured way to analyze and improve the arrangement of facts.

2.The experimental results show that the precedence-constrained MinLA ordering can significantly improve accuracy compared with natural ordering, especially on smaller models. The authors also conduct additional experiments across different model sizes and dependency structures, which further support the observed ordering effects.

Critical weaknesses

1.The experiments mainly focus on two task settings, and both datasets are synthetically generated. Although the authors mention GSM8K, the arithmetic problems are generated based on its format rather than directly using the original benchmark. For the Narrative dataset, the dependency chains and distractors are also artificially constructed. In my opinion, it is difficult to know whether the observed findings can generalize to real-world tasks.

2.Although MinLA is a well-established optimization problem, I feel that its connection to LLM ordering effects is not clearly explained. The authors define the cost function based on the sum of positional distances between dependent facts, but why should this particular distance measure reflect how LLMs process dependencies? For example, why use the absolute distance rather than other distance functions or simply grouping related facts together? In my opinion, the current formulation looks more like a heuristic choice than a theoretically justified explanation of LLM behavior.

Questions for authors
  1. The mathematical notation in Sections 3.1 and 3.2 could be clearer. For example, FF is represented as a set, although the formulation relies on the original ordering of facts. Moreover, Section 3.2 indicates that references within facts are updated according to the permutation, but it is not entirely clear what remains fixed across different ordering conditions. I suggest the authors provide a more intuitive explanation, perhaps with a simple example, to improve readability.

  2. Some figure captions are not so clear. For example, Figure 1 introduces the solid green line and the dashed red line, but does not introduce the meaning of the solid red line. Although the reader can infer the meaning, for readability, I suggest the authors polish the captions.

  3. Some unnecessary ref. For example, on Line 267, the authors refer to (Section 3.2), for the paper writing perspective, this is not necessary, because "Narrative dependencies" is the key concept of this paper experiment setting, and the reader surely would pay attention to it. So, in my opinion, the authors should save the space for the most important content in the main text. Same suggestion on Line 371, Line 399, etc.

  4. The authors mainly compare Natural, Free, and Constrained ordering to demonstrate the effectiveness of the proposed method. However, I feel that these comparisons are somewhat limited. For example, the Natural order is constructed by interleaving different dependency chains, while the Constrained order is specifically optimized to preserve dependencies and reduce distances. Therefore, it is not surprising that the Constrained order performs better. Although the authors also consider Legal Random, I wonder whether simpler strategies, such as topological sorting or directly grouping dependent facts, could achieve similar results.

Reviewer_cZVMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper attempts to explain ordering effects in in-context learning (ICL) through three factors: proximity, precedence, and position. However, rather than studying the ordering of input-output demonstrations in conventional ICL, the authors investigate a different task in which an LLM must answer a question using interdependent facts presented in varying orders. To this end, they construct two synthetic datasets consisting of arithmetic and narrative reasoning problems, where short chains of facts required to compute an answer are interleaved with independent distractor chains. The authors formulate fact ordering as a minimum linear arrangement (MinLA) problem with precedence constraints, aiming to place dependent facts close together while ensuring that prerequisite facts appear before the facts that depend on them. Although the paper is motivated by ordering effects in ICL, its experiments evaluate the ordering of interdependent facts within individual synthetic reasoning problems, rather than the ordering of worked input-output demonstrations as studied in conventional ICL

Critical strengths

The paper provides a systematic experimental analysis of how proximity, precedence, and position affect the performance of LLMs operating without explicit reasoning mode on synthetic multi-step reasoning tasks

Critical weaknesses
  1. The paper is motivated by ordering effects in ICL but the experiments do not study the ordering of worked input-output demonstrations. Instead, they investigate how to arrange interdependent facts within individual synthetic reasoning problems. It is unclear how the findings explain the ordering effects observed in conventional ICL, and no experiments establish this connection.
  2. The datasets deliberately interleave short dependency chains with irrelevant facts, and the proposed method rearranges these facts according to their known dependencies. This effectively removes a source of difficulty introduced by the task construction itself. The accuracy improvements for non reasoning models may therefore reflect simplifying the representation of these particular problems rather than demonstrating a broadly applicable ordering principle.
  3. The Qwen3 experiments disable thinking mode due to unjustified limited generation budget of 50n+200 tokens. The paper does not justify why this constraint is added. Consequently, it remains unclear whether lower performance is because depth is insufficient for the task and explicit reasoning is needed. Thus the proposed reordering may be necessary when models can't reason through the dependencies themselves. In fact on larger models the benefits diminish which may suggest that implicit reasoning via depth of the model helps. This leaves little evidence that the proposed method offers meaningful advantages over explicit reasoning or or model depth.
  4. The Qwen3 evaluations use greedy decoding despite the official model card recommending sampling-based inference settings and specifically advising against greedy decoding in thinking mode. It's unclear whether the results hold under the recommended inference settings, and for explicit reasoning models.
  5. The paper over claims beyond what their two dataset establish. Additionally, the method is not benchmarked on non synthentic tasks
Questions for authors

Why did you use greedy decoding for Qwen3 despite the official model card providing recommended inference parameters and advising against greedy decoding in thinking mode? Additionally, including more capable models such as Qwen3.6-27B (released April 21, 2026) would strengthen the paper

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.