- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OrderCover, a structure-aware method for budgeted prompt-order optimization that preserves the original prompt content. The central motivation is that language models can be sensitive to the ordering of prompt components, while existing prompt optimization methods primarily modify instructions or demonstrations. OrderCover first decomposes prompts into validated, movable components and then selects candidate permutations using precedence, directed-adjacency, coarse-position coverage, and Kendall-distance separation. Candidate construction does not require model outputs or task labels, although selecting the best candidate requires labeled development evaluations. The empirical evaluation covers 70 controlled task–model landscapes, 15 held-out application-prompt settings, and 24 post-content-optimization settings. In the controlled suite, OrderCover improves Anchor accuracy by 3.35 percentage points, captures 82.99% of the available in-sample headroom, and exceeds budget-matched Random search by 0.46 points. On application prompts, the mean improvement over Anchor is 3.32 points, but its confidence interval includes zero. After GEPA or MIPROv2 optimization, additional order search improves held-out accuracy by 1.48 points on average. The evidence supports prompt ordering as a useful optimization dimension, while the incremental benefit of the proposed selector and its generalizability remain more limited.
Critical strengths
-
Sections 3.1–3.2 establish a clear distinction between modifying prompt content and modifying component order. The decomposition and grouping procedure preserves the exact text and within-component order, with deterministic reconstruction checks described in Appendix A. This provides a useful experimental intervention for investigating order sensitivity without conflating it with textual rewriting. The explicit restriction to independently movable components also makes the method's intended scope relatively clear. The formulation is valuable because it isolates an optimization dimension that can be investigated independently of conventional content optimization.
-
Section 6.1, Table 1, and Appendix D provide complete permutation landscapes across ten tasks and seven models, enabling direct measurement of attainable in-sample order headroom. OrderCover achieves 67.38% accuracy compared with 64.03% for Anchor, 66.92% for budget-matched Random, and 68.00% for Oracle. Its 0.46-point advantage over Random has a reported 95% bootstrap interval of [0.21, 0.83]. Random search is averaged over 10,000 independently sampled candidate sets, and both methods include Anchor under equal evaluation budgets. These controls make the comparison informative and distinguish the substantial benefit of searching over orders from the smaller incremental benefit of structure-aware candidate selection.
-
Section 6.3, Table 2(b), and Appendix F investigate whether optimized prompts retain useful order headroom. After freezing prompts produced by GEPA or MIPROv2, OrderCover improves held-out accuracy in 19 of 24 cells, with four ties and one loss. The aggregate improvement is 1.48 percentage points, with a reported 95% interval of [0.94, 2.06]. Both GEPA and MIPROv2 show improvements, suggesting that content optimization does not necessarily exhaust the potential benefits of reordering instructions. The use of disjoint test sets provides stronger evidence of practical value than the controlled landscapes alone.
Critical weaknesses
-
The principal algorithmic contribution is the combination of structural feature coverage, Kendall separation, and anchor-distance balancing in Equation 6. However, Table 1 shows that budget-matched Random already recovers much of the available order headroom. More importantly, Appendix B.5, Table 5, reports 67.38% accuracy for the full selector versus 67.30% for Kendall farthest-first and 67.27% for the unbucketed hybrid. These differences of 0.08 and 0.11 percentage points are small, with many pair-level ties. Although the ablations support a modest aggregate advantage, they do not establish that the complete structural design yields a practically substantial improvement over simpler diversity-based selection. This distinction matters because the stronger empirical contribution may be the usefulness of budgeted order search itself rather than the specific OrderCover scoring rule.
-
The controlled evaluation in Section 6.1 selects and scores permutations on the same evaluation set. Consequently, its Oracle headroom and recovered-headroom measurements are descriptive in-sample quantities rather than estimates of deployable test improvements. The paper appropriately includes a separate held-out application suite, but Table 2(a) reports a mean improvement over Anchor of 3.32 points with a 95% interval of [−0.43, 8.69]. The advantage over Random is 0.84 points with an interval of [−0.02, 1.13]. Neither interval excludes zero. Furthermore, Appendix E, Table 16, reports a particularly large 22.00-point improvement for LlamaIndex with Ministral-3-8B, illustrating substantial variation across cells. These results suggest that beneficial orders frequently exist, but they do not establish a consistently positive average improvement across unseen applications or a reliably superior selector in that regime. The uncertainty should not be interpreted as evidence of equivalence or ineffectiveness.
-
Sections 3.1–3.2 and Appendix A validate component types, grouping contiguity, and exact-text reconstruction. However, these checks do not establish that every resulting permutation preserves the intended semantics or instruction dependencies. This is particularly relevant when reordering directives, constraints, label definitions, and instance inputs. The authors acknowledge this limitation in Section 8. Furthermore, Appendix G.1, Table 21, reports exact component-count agreement of only 41.3% and 42.1% for two alternative decomposition models, despite much stronger agreement on whether prompts have at most six components. The structuralization sensitivity analysis does not directly evaluate whether alternative decompositions preserve OrderCover's downstream accuracy gains or its advantage over matched Random search. Therefore, stability of the component-count regime should not be interpreted as robustness of the complete optimization pipeline.
-
Section 5.3 and Appendix F demonstrate residual order headroom after content optimization, but the experiment uses one saved artifact from one seed per optimizer configuration. Appendix F.2 also specifies an instruction-only MIPROv2 configuration with demonstration optimization disabled. Thus, the reported results characterize these particular optimizer outputs rather than the variability of order-search benefits across independently optimized prompts. Moreover, the post-optimization suite has no budget-matched Random arm, so we cannot attribute its 1.48-point improvement specifically to the proposed structural selector. Appendix C.2 additionally computes its confidence interval by resampling 24 cell-level differences, although these cells reuse tasks and models. This interval describes variability across the observed cells but does not fully establish uncertainty across independent tasks, models, or optimizer seeds. These limitations do not negate the observed improvements, but they constrain the interpretation of complementarity and robustness.
-
Section 8 and Appendix D.3, Table 14, show that all controlled experiments involve at most six components. For four components, the evaluation budget covers 16 of 24 possible orders, making the search relatively close to exhaustive. For six components, only 36 of 720 orders are evaluated, but the seven corresponding task–model pairs all share one task. Appendix B.3 describes extending the selector to larger spaces through sampled candidate pools, without evaluating this extension. Consequently, the experiments demonstrate effectiveness in small enumerable spaces but do not establish how selection quality or computational cost behaves for larger prompts requiring genuinely sparse exploration. The paper explicitly acknowledges this boundary, but it remains important for assessing the practical reach of a budgeted search method.
Questions for authors
-
Given the small differences from Kendall farthest-first and the unbucketed hybrid in Appendix B.5, could the authors report paired uncertainty estimates for these comparisons? If feasible, evaluating Kendall farthest-first under the same development/test protocol as the application suite, with identical componentization, candidate budgets, and selection data, would help determine whether the full structural objective provides benefits beyond generic permutation diversity.
-
Could the authors provide application-stratified paired differences and a leave-one-application-out analysis for Table 2(a), particularly accounting for the large LlamaIndex–Ministral improvement in Table 16? This would clarify whether the observed average gain reflects recurring improvements across applications or substantial dependence on particular settings. An explanation of which prompt characteristics are associated with stronger gains would also improve the practical interpretation.
-
Appendix G.1 establishes component-count agreement but not performance stability. Could the authors evaluate a small representative subset using the alternative decompositions already generated, while holding prompt text, target model, development/test examples, and evaluation budget fixed? Reporting differences in selected orders, held-out performance, and comparisons with Random would directly assess whether decomposition variability materially affects the method's benefit. A brief audit of cross-component semantic dependencies would additionally clarify the conditions under which arbitrary permutation is appropriate.
-
Could the authors provide a budget-matched Random comparison for the existing frozen GEPA and MIPROv2 prompts, using the same development splits and test metrics? Additionally, is there evidence from alternative optimizer seeds, or an analysis indicating how artifact variability might affect the results? A task-clustered uncertainty analysis would help distinguish variation across genuinely different tasks from repeated measurements involving the same tasks and models. If additional runs are infeasible, explicitly limiting the conclusions to the evaluated optimizer artifacts would resolve part of this concern.
-
Appendix B.3 proposes candidate-pool sampling for larger component counts. Could the authors clarify how they would choose pool size, global versus local sampling, and greedy-selection cost? A controlled comparison using the existing six-component landscapes, where the candidate pool is artificially restricted while holding the evaluation budget fixed, could provide preliminary evidence about sensitivity to pool construction without requiring substantially larger experiments. This would help assess whether the proposed method's benefits extend beyond direct enumeration of small permutation spaces.
Thank you again for your detailed reviews. As the discussion phase closes on November 18, we would like to check whether our response and revised Appendix H have addressed your concerns (about paired uncertainty, application-level influence, semantic dependencies, the post-optimization Random arm, and restricted candidate pools). We would be glad to clarify any remaining points or provide additional analysis.