acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Demonstration Ordering for In-Context Learning via Performance-Oriented Ordering Predictor

15 views

Abstract

In-context learning (ICL) is highly sensitive to demonstration ordering: even with the same input query, retrieved demonstrations, and target LLM, different orderings can induce markedly different inference performance. However, existing demonstration ordering methods primarily rely on static proxy scores, such as similarity or entropy, which are often poorly aligned with downstream performance and ignore how ordering interacts with the input query, retrieved demonstrations, and target LLM. Therefore, we formulate demonstration ordering as a performance prediction task and propose Adaptive Demonstration Ordering for In-Context Learning (ADOICL), a performance-oriented and adaptive framework that selects demonstration orderings conditioned on the input query, the retrieved demonstrations, and the target LLM. Specifically, ADOICL learns an ordering predictor from offline performance supervision to estimate the downstream performance of candidate orderings and select the most promising ordering before a single target-LLM inference. We instantiate ADOICL in two forms: ADOICL-Seq, a sequence-level version that directly scores serialized query–ordering pairs, and ADOICL-Emb, an embedding-first version that embeds each query and retrieved demonstration once. ADOICL-Emb reuses these representations across candidate orderings, thereby avoiding repeated re-embedding and substantially reducing embedding time, so that the dominant embedding-time cost scales approximately linearly with the number of demonstrations. Empirically, ADOICL achieves state-of-the-art performance across 9 target LLMs, 5 tasks, and 8 datasets, covering classification, generation, and regression. It is plug-and-play with existing ICL pipelines, generalizes across LLMs, tasks, datasets, and different demonstration numbers, further supports visual ICL and few-shot training-data scenarios, and provides a clustering-based group-level scalability variant for many-shot settings.enhancer for existing ICL pipelines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
in-context learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uPm4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ADOICL, which formulates demonstration ordering for in-context learning as a performance-prediction problem. Rather than relying on task-agnostic proxy scores, it learns a lightweight ordering predictor from offline performance supervision: the predictor estimates the downstream task performance of candidate demonstration orderings conditioned on the input query, the retrieved demonstrations, and the target LLM, and selects the best-predicted ordering before a single target-LLM inference. The framework is instantiated in two forms: ADOICLS (a sequence-level scorer over serialized query-ordering pairs) and ADOICLE (an embedding-first scorer that reuses query and demonstration embeddings to reduce encoding from k! to k). The method is validated across 9 target LLMs (1B–70B, including GPT-4o), 5 tasks, and 8 datasets spanning classification, regression, and generation, with ablations (removing the predictor costs 8.57% accuracy; the dual-input architecture adds 4.58%), generalization studies (cross-LLM, cross-dataset, cross-task, cross-demonstration-number), and extensions to plug-and-play, visual ICL, few-shot, and many-shot settings.

Critical strengths
  1. A clean, well-motivated problem reframing backed by direct evidence. The paper treats demonstration ordering as a performance-prediction problem, directly regressing the target LLM's realized task performance instead of relying on task-agnostic proxy scores.

  2. Thorough and well-ablated validation. The evaluation spans 9 target LLMs, 5 tasks, and 8 datasets covering classification, regression, and generation, with clean ablations , broad generalization studies, and extensions to plug-and-play, visual ICL, few-shot, and many-shot settings.

Critical weaknesses
  1. Key results are left unanalyzed, especially the coarse binary supervision for classification. The per-instance supervision for classification is binary (0/1), so the predictor learns "correct or not" rather than "how good", which may prevent it from distinguishing orderings that are both correct (or both wrong) — yet classification is where the method shows its largest gains, and this tension is not discussed.

  2. The evaluation does not cover the current frontier of LLMs. The tested open-source models cap at 70B, and the only commercial model evaluated is GPT-4o, which is no longer the newest frontier model. Robustness on the latest open- and closed-source models is therefore unverified.

Questions for authors
  1. The oracle gap remains about 4.6 points on Emotion+Llama. What limits the predictor from reaching the oracle, can you analyze this?

  2. For classification, the per-instance supervision is binary (0/1), so the predictor learns "correct or not" rather than "how good." Does this limit its ability to distinguish between orderings that are both correct (or both wrong)? Would a smoother (e.g., probability- or margin-based) supervision improve it?

  3. The evaluation's open-source models cap at 70B and the only commercial model is GPT-4o. Can you extend the evaluation to the current frontier LLMs (GPT-5.6 / DeepSeek-v4 / Claude-4.8) to confirm the gains hold at the frontier?

if concerns can be adequately addressed, I would be willing to raise my score to 4.

6 comments
AuthorsOpenReview ↗

Response to Weakness1 and Question2

Thank you for your thoughtful review. We appreciate your recognition of the motivation behind our formulation and the breadth of our experimental validation. Your constructive comments on supervision granularity, the remaining oracle gap, and evaluation on frontier LLMs have helped us strengthen the analysis. We address these points below with additional experiments and references to the corresponding sections and tables in the revised manuscript.

W1 and Q2: Binary classification supervision and finer-grained alternatives

Thank you for this insightful suggestion. We agree that finer-grained supervision can improve learning when it captures meaningful differences in downstream performance. We have added two complementary experiments to investigate this issue (Appendix Section W, Tables 39 and 40).

Why binary supervision can still yield substantial classification gains. Binary correctness does not distinguish two correct orderings or two incorrect orderings. However, these orderings have equal utility under the accuracy metric. Improving accuracy requires selecting a correct ordering when one is available, rather than distinguishing among all correct orderings. Thus, the observed classification gains are compatible with binary supervision, although richer signals could further improve learning and generalization.

1. Classification: output log-likelihood supervision.

We evaluated complete-output log-likelihood as a continuous alternative (Appendix Section W, Table 39). A practical limitation of our current scoring pipeline is that it provides scores for generated responses, rather than separate scores for the ground-truth label and competing classes. These scores alone cannot determine label-specific probabilities or class margins; obtaining those targets requires an additional class-conditional scoring procedure. Accordingly, this experiment partially explores the suggestion rather than directly testing ground-truth label probabilities or margins.

Each cell below reports GLM / Llama / Qwen.

Table 39.

DatasetMetricOutput log-likelihood supervisionBinary correctness supervision
EmotionAccuracy (%) ↑70.1 / 55.6 / 69.371.4 / 58.1 / 68.6
SST5Accuracy (%) ↑52.3 / 39.9 / 54.152.0 / 40.3 / 53.8

Output log-likelihood improves three settings and underperforms in three, with mean accuracy of 56.88%, compared with 57.37% for binary correctness. One possible explanation is that likelihood measures confidence in the generated answer, which may be incorrect. These results therefore do not imply that finer-grained supervision is unhelpful; they show that a continuous confidence score is not necessarily better aligned with classification accuracy.

2. Generation and regression: coarsening continuous task scores.

To examine the value of fine-grained task-performance information, we normalized the training scores using training-set minimum and maximum values, then mapped the intervals [0, 0.33), [0.33, 0.66), and [0.66, 1] to 0, 1, and 2, respectively. We retained the predictor architecture and MSE training objective. Higher targets indicate better generation quality; lower targets indicate better STS performance because STS supervision uses squared prediction error (Appendix Section W, Table 40).

Table 40.

DatasetMetricThree-level supervisionOriginal continuous supervision
GigawordROUGE-1 ↑0.062 / 0.061 / 0.15490.069 / 0.078 / 0.183
GigatinyROUGE-1 ↑0.212 / 0.134 / 0.3160.239 / 0.136 / 0.347
STS14MSE ↓0.791 / 5.951 / 0.6830.752 / 5.830 / 0.671
STSBMSE ↓0.775 / 1.598 / 0.6840.739 / 1.510 / 0.641
WMT19 En–ZhBLEU ↑0.224 / 0.026 / 0.1320.231 / 0.029 / 0.139
WMT19 Zh–EnBLEU ↑0.479 / 0.094 / 0.5450.512 / 0.098 / 0.582

Original continuous supervision performs better in all 18 settings. This supports the reviewer’s intuition that preserving fine-grained performance distinctions benefits ordering prediction. Since discretization also changes the target scale, we interpret this as supporting evidence rather than a complete isolation of granularity effects.

Together, these experiments support the importance of both supervision granularity and alignment with task performance. We have included the results and discussion in the appendix, while explicitly acknowledging that ground-truth label probability and margin supervision remain untested.

AuthorsOpenReview ↗

Response to Weaknesses 2, Question 1, and Question 3

Q1: What limits the predictor from reaching the oracle?

Thank you for suggesting this analysis. Our additional experiment identifies insufficient training data as an important contributor to the remaining oracle gap.

We increase the sampled orderings per training query from M = 5 to M = 10, doubling the number of query–ordering–score training examples while keeping the training queries, predictor architecture, and supervision definition unchanged. Both settings score all 24 permutations at test time, so this change increases only the training data, without expanding the inference-time search space.

The results are shown below (Appendix Section X, Table 41). Each cell reports GLM / Llama / Qwen.

Table 41.

DatasetMetricADOICLS-5ADOICLS-10Oracle
EmotionAccuracy (%) ↑71.4 / 58.1 / 68.673.1 / 60.4 / 73.675.6 / 62.7 / 76.1
SST5Accuracy (%) ↑52.0 / 40.3 / 53.855.1 / 42.5 / 57.458.2 / 46.4 / 61.2
GigawordROUGE-1 ↑0.069 / 0.078 / 0.1830.074 / 0.080 / 0.1870.078 / 0.087 / 0.199
GigatinyROUGE-1 ↑0.239 / 0.136 / 0.3470.258 / 0.152 / 0.3620.261 / 0.163 / 0.394

On Emotion + Llama, accuracy increases from 58.1% to 60.4%, reducing the gap to the 62.7% oracle from 4.6 to 2.3 percentage points—a 50% reduction. Improvements occur in all 12 model–dataset settings.

These results support the explanation that sampling only five orderings per query provides insufficient training supervision to fully learn ordering-dependent performance. Additional training data substantially narrows the gap, even with unchanged binary supervision for classification. Although the residual gap cannot be attributed entirely to data scarcity, this experiment demonstrates that limited training data is an important bottleneck. The additional cost is incurred offline; test-time scoring and the single final target-LLM inference remain unchanged. % TODO: Add response and supporting evidence.

W2 and Q3: Evaluation on frontier LLMs

Thank you for highlighting this limitation. We have extended our evaluation to GPT-5.6 and DeepSeek-v4, covering classification on Emotion and text expansion on Gigatiny (Appendix Section Y, Table 42). These experiments complement our original GPT-4o results (Section 4, Table 6).

Table 42.

MethodGPT-5.6: Emotion Acc. (%) ↑GPT-5.6: Gigatiny ROUGE-1 ↑DeepSeek-v4: Emotion Acc. (%) ↑DeepSeek-v4: Gigatiny ROUGE-1 ↑
BM2567.60.16565.60.194
CR56.10.27551.80.310
CD67.40.30369.90.312
Kate69.70.35270.10.320
PLR70.00.31670.60.323
CDS68.90.34165.00.323
TTF71.60.33068.70.310
GlobalE59.00.25459.00.228
ADOICLE78.80.39776.90.369
ADOICLS80.10.40476.60.384

Both ADOICL variants outperform all compared baselines in all four settings. Specifically:

  • GPT-5.6: ADOICLS improves Emotion accuracy by 8.5 percentage points over the strongest baseline, TTF (80.1% vs. 71.6%), and Gigatiny ROUGE-1 by 0.052 over Kate (0.404 vs. 0.352).
  • DeepSeek-v4: ADOICLE improves Emotion accuracy by 6.3 percentage points over PLR (76.9% vs. 70.6%). ADOICLS improves Gigatiny ROUGE-1 by 0.061 over the strongest baselines, PLR and CDS (0.384 vs. 0.323).

These results provide additional evidence that ADOICL’s benefits extend to the two newly evaluated models across classification and generation. We have included the full results and discussion in the appendix.

Unfortunately, API access restrictions prevented us from evaluating Claude. We acknowledge this remaining limitation and restrict our conclusions to the evaluated models and tasks.

Thank you again for your time and constructive feedback. We would appreciate it if you could let us know whether our response has addressed your concerns and whether there are any remaining issues we could clarify.

Reviewer_oN2HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ADOICL to order retrieved in-context demonstrations. Given the input query and the retrieved demonstrations, ADOICL trains a predictor to select the ordering before inference, so the LLM is called only once at test time. Detailed speaking, ADOICL samples a few candidate permutations of each training query's retrieved demonstrations and score them offline with the target LLM. A lightweight predictor is trained to regress these per-instance scores. At inference, the highest-scoring permutation is used to construct the prompt. The authors develop two variants of ADOICL: ADOICL$_S$ encodes serialized query–ordering pairs, and ADOICL$_E$ reuses cached demonstration embeddings to reduce repeated encoding. They perform ablation studies and a fixed-demonstration-set analysis to show that query-dependent ordering outperforms query-independent ordering strategies. Lastly, empirical experiments across 9 LLM backbones and 5 tasks demonstrate the superior performance of ADOICL compared to existing ICL baselines.

Critical strengths
  1. This paper is generally well-written and easy to follow.
  2. The experiments cover a broad range of tasks and backbone types.
Critical weaknesses
  1. The positioning relative to previous works remains unclear. As mentioned by the author, many works already "dynamically order demonstrates under the objective of downstream performance". In my understanding, ADOICL trains an order scorer to reduce LLM API calls during inference, which might achieve better efficiency. However, the author does not provide the running time for dynamic ordering baselines (DEmO, OptiSeq, Self-Adaptive ICL, etc.). Meanwhile, many other works (e.g., LLM-R [1], Se2 [2]) already use a scorer to retrieve demos for ICL; this paper should also incorporate discussions and experiments with them.
  2. In the main exp table (table 1), the author only compared with fixed-order baselines, while the DEMOs are different across baselines, making it hard to tell whether the performance gain comes from Demo ordering or selection. In table 24, the author only compares with some trivial baselines in table 24. Meanwhile, in table 28, the dynamic ordering baseline OptiSeq even underperforms random ordering in table 24, which makes it hard to interpret.

[1] Wang, Liang, Nan Yang, and Furu Wei. "Learning to retrieve in-context examples for large language models." EACL 2024.

[2] Liu et al., "Se^2: Sequential Example Selection for In-Context Learning",Findings of ACL 2024

Questions for authors
  1. Could the author provide more details for the BLEU computation? (e.g., what type of tokenizer? is it sentence-average or corpus average?)
  2. Could the author provide the ROUGE-1/2/L (F1) as the current ROUGE-1 in Eq.13 might favor long generation?
7 comments
AuthorsOpenReview ↗

Response to Weaknesses 1

Thank you for your thoughtful review and for recognizing the clarity of our presentation and the breadth of our experiments. We appreciate your constructive suggestions on methodological positioning, controlled comparisons, efficiency, and evaluation metrics. Below, we respond to each concern and indicate the corresponding evidence in the revised manuscript.

W1: Methodological positioning and comparisons with prior methods

Thank you for highlighting these relevant methods and the missing runtime comparison. We have clarified the methodological positioning and added comparisons with LLM-R and Se², together with an online efficiency evaluation against DEmO, OptiSeq, and Self-Adaptive ICL. These additions are presented in the appendix section, Comparison with Learned Selection and Dynamic Ordering Methods (Appendix Section T, Tables 35 and 36).

1. Clarified methodological positioning.

We agree that adaptive demonstration ordering has been studied previously. LLM-R learns retrieval from LLM feedback, while Se² learns context-dependent sequential selection and explicitly considers example order. Our specific contribution is learning a query-conditioned predictor of downstream task scores for complete permutations of a fixed retrieved demonstration set. DEmO, OptiSeq, and Self-Adaptive ICL assess candidate contexts online using criteria derived from target-LLM output probabilities. ADOICL uses offline task-score supervision to train a lightweight predictor that selects an ordering before the final target-LLM inference.

2. Added comparisons with LLM-R and Se².

We evaluate these methods on Emotion and SST5 (classification), Gigaword (summarization), and Gigatiny (text expansion), using GLM-4-9B, Llama-3.1-8B, and Qwen2.5-7B (Appendix Section T, Table 35). Each cell below lists GLM / Llama / Qwen.

Table 35.

MethodEmotion: Accuracy (%) ↑SST5: Accuracy (%) ↑Gigaword: ROUGE-1 ↑Gigatiny: ROUGE-1 ↑
LLM-R54.7 / 39.4 / 57.341.0 / 33.2 / 48.10.043 / 0.036 / 0.0940.161 / 0.126 / 0.225
Se²54.6 / 45.2 / 56.946.0 / 35.7 / 47.30.053 / 0.055 / 0.1290.186 / 0.109 / 0.281
ADOICL-E70.8 / 57.5 / 69.051.6 / 39.9 / 52.30.065 / 0.081 / 0.1790.235 / 0.133 / 0.341
ADOICL-S71.4 / 58.1 / 68.652.0 / 40.3 / 53.80.069 / 0.078 / 0.1830.239 / 0.136 / 0.347

Both ADOICL-E and ADOICL-S outperform both baselines in all 12 model–dataset settings. Relative to the stronger of these two baselines in each setting, ADOICL-S improves classification accuracy by 4.6–16.7 percentage points and ROUGE-1 by 0.010–0.066. These full-pipeline comparisons complement our fixed-set analysis and plug-and-play experiments, which isolate ordering gains by preserving demonstration membership.

The original LLM-R and Se² studies do not report continuous-valued regression evaluations. Our added comparison covers classification and generation, while ADOICL's STS14/STSB regression results are reported separately (Section 3, Table 1).

3. Added online efficiency comparison.

We have additionally rerun Self-Adaptive ICL on SST5 using GLM-4-9B, Llama-3.1-8B, and Qwen2.5-7B, obtaining an average accuracy of 42.8% and an average testing time of 1.46 seconds per query. These newly obtained results are included in the comparison below, where both accuracy and testing time are averaged over the three target LLMs (Appendix Section T, Table 36).

Table 36.

MethodAccuracy (%) ↑Time (s/query) ↓
DEmO41.10.73
OptiSeq41.24.27
Self-Adaptive ICL42.81.46
ADOICL-E47.90.24

Compared with DEmO, OptiSeq, and Self-Adaptive ICL, respectively, ADOICL-E improves accuracy by 6.8, 6.7, and 5.1 percentage points, with approximately 3.0×, 17.8×, and 6.1× speedups, calculated as ratios of the reported mean testing times.

These dynamic ordering baselines evaluate multiple candidate contexts using the target LLM during inference, introducing additional online computation. ADOICL-E evaluates candidate permutations with its lightweight predictor, reuses demonstration embeddings, and invokes the target LLM only for the final prediction. Its offline supervision and predictor-training costs are reported separately in the main text (Section 4, “Cost and time”).

AuthorsOpenReview ↗

Response to Weaknesses 2 and Questions

W2: Isolating the contribution of demonstration ordering

Thank you for this helpful comment. Demonstration selection should be controlled when assessing the contribution of ordering. Table 1 (Section 3.2) reports the overall performance of each method. To clarify the source of the improvements, we discuss the controlled experiments included in the original manuscript and the newly added OptiSeq comparison below.

1. Original plug-and-play experiments: keeping demonstrations fixed and changing only their order.

Table 10 of the original manuscript (see Section 4, “Plug-and-play”) applies ADOICL to different retrieval methods while keeping the retrieved demonstrations unchanged. Ordering improves performance in all 12 retriever–dataset–LLM settings, showing that its benefits extend to demonstration sets obtained from different retrievers.

Each cell below reports GLM / Llama / Qwen results. The metrics are accuracy (%) for Emotion and ROUGE-1 for Gigaword.

Table 10.

DatasetRetrieverOriginal+ ADOICL ordering
EmotionBM2554.3 / 28.3 / 60.361.3 / 31.1 / 65.2
EmotionCD55.8 / 37.0 / 53.865.1 / 40.4 / 60.6
GigawordBM250.028 / 0.020 / 0.0230.031 / 0.023 / 0.025
GigawordCD0.032 / 0.056 / 0.1510.037 / 0.064 / 0.155

2. Original ablations and fixed-set experiments provide further evidence.

Tables 2 and 17 of the original manuscript (Section 3.2 and Appendix Section H, respectively) examine the contribution of the ordering predictor. Removing this predictor while retaining the retrieval component reduces ADOICL-S accuracy by an average of 9.08 percentage points across Emotion and SST5 with the three LLMs.

Table 24 (Appendix Section L) further compares orderings of the same four demonstrations for each query, including Best-fixed, the best fixed ordering selected on the validation set. ADOICL-S outperforms Best-fixed in all 12 settings. For example, on Emotion with Llama, Random, Best-fixed, and ADOICL-S achieve 42.2%, 47.4%, and 58.1%, respectively. These results support the benefit of query-dependent ordering over a validation-optimized fixed order when demonstration selection is held constant.

3. Clarifying the comparison between OptiSeq and Random across tables.

OptiSeq in Table 28 (Appendix Section M) and Random in Table 24 (Appendix Section L) use different demonstration sets. Their performance difference may therefore reflect both selection and ordering and cannot be attributed to ordering alone. We apologize that this distinction was not sufficiently clear in the original presentation.

4. New controlled experiment: applying ADOICL-S to the demonstrations used by OptiSeq.

We also added an experiment that retains the exact demonstrations used by OptiSeq, including their number, and changes only their order using ADOICL-S. Accuracy (%) is reported below (Appendix Section U, Table 37).

Table 37.

DatasetLLMOptiSeqOptiSeq + ADOICL-SGain (percentage points)
EmotionGLM50.361.2+10.9
EmotionLlama43.652.9+9.3
EmotionQwen49.955.2+5.3
SST5GLM43.648.2+4.6
SST5Llama33.837.3+3.5
SST5Qwen46.348.7+2.4

Performance improves in all six settings, with an average gain of 6.0%. Because demonstration selection is unchanged, this experiment further supports the contribution of ADOICL-S ordering and provides a direct comparison against a dynamic ordering baseline. We hope this additional evidence, together with the original controlled experiments, helps address your concern about the source of the gains.

Q1: Clarifying BLEU tokenization and score aggregation

For WMT19 Zh→En, we lowercase both generated translations and references and apply whitespace tokenization (str.lower().split()). Punctuation is retained, without additional punctuation splitting.

Our evaluation script uses corpus-level BLEU-4 as the primary metric, with uniform weights of 0.25 for 1–4-grams and a corpus-level brevity penalty (for the general BLEU definition, see Appendix Section D.1, Eqs. (11)–(12)).

Q2: Additional evaluation with ROUGE-1/2/L F1

We reran the evaluation experiments for ADOICL-S and computed ROUGE-1, ROUGE-2, and ROUGE-L F1 scores on Gigaword (summarization) and Gigatiny (text expansion), using GLM, Llama, and Qwen (Appendix Section V, Table 38).

Table 38.

DatasetLLMROUGE-1 F1ROUGE-2 F1ROUGE-L F1
GigawordGLM0.0700.0220.061
GigawordLlama0.0760.0210.065
GigawordQwen0.1870.0600.162
GigatinyGLM0.2340.0710.208
GigatinyLlama0.1390.0400.120
GigatinyQwen0.3540.1040.303

Thank you for the response, which largely addresses my concerns. One remaining concern is about the generation evaluation. Table 38 reports ROUGE-1/2/L F1 only for ADOICL-S. Could you also provide these metrics and average generation lengths for ADOICL and the main baselines on Gigaword and Gigatiny? I will raise my score if these comparisons support the reported advantages.

Reviewer_jBwMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ADOICL, a performance-oriented framework that predicts the downstream utility of demonstration orderings for in-context learning. Experiments show improvements across multiple LLMs and downstream tasks.

Critical strengths
  • The paper provides a broad empirical evaluation across multiple models and tasks, complemented by ablation studies, generalization experiments, and efficiency analysis. The results are generally competitive.
Critical weaknesses

W1: Both PDDS and ADOICL train predictors using downstream task performance as supervision, but optimize demonstration selection and ordering, respectively. Without a direct experimental comparison, ADOICL’s practical advantages over PDDS remain unclear.

W2: In Table 4, the transferred predictor falls below TTF in ROUGE-1 on GLM and only slightly outperforms the strongest baselines on Llama and Qwen. Moreover, the evaluation covers only one cross-task transfer direction, Emotion → Gigaword, providing limited support for broader cross-task generalization.

W3: ADOICL uses MSE to predict candidate orderings’ performance scores, whereas its ultimate goal is to select the best ordering. The paper does not compare MSE with ranking losses that directly optimize relative preferences among candidate orderings, leaving the choice of training objective insufficiently justified.

W4: In the 100-shot setting, the paper does not compare learned group ordering against random group ordering while keeping group assignments and within-group orderings fixed. This makes it difficult to isolate the ordering predictor’s contribution in this setting.

Questions for authors

Please address the weaknesses mentioned above.

5 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 and Weaknesses 2

Thank you for your thoughtful review and for recognizing the breadth of our empirical evaluation. We appreciate your constructive comments on the comparison with PDDS, cross-task generalization, the training objective, and the contribution of learned group ordering in the 100-shot setting. Below, we address each concern with additional experiments and clarifications, together with references to the corresponding sections and tables in the revised manuscript.

W1: Direct comparison with PDDS

We added a direct comparison with PDDS across all eight benchmarks and three LLMs (Appendix Section P, Table 30). Both ADOICLE and ADOICLS outperform PDDS in all 24 model–dataset settings, providing direct evidence of their downstream performance advantages.

Each numerical cell below reports GLM / Llama / Qwen.

Table 30.

DatasetMetricPDDSADOICLEADOICLS
EmotionAccuracy (%) ↑60.2 / 50.2 / 60.170.8 / 57.5 / 69.071.4 / 58.1 / 68.6
SST5Accuracy (%) ↑47.3 / 36.5 / 49.951.6 / 39.9 / 52.352.0 / 40.3 / 53.8
GigawordROUGE-1 ↑0.058 / 0.066 / 0.1460.065 / 0.081 / 0.1790.069 / 0.078 / 0.183
GigatinyROUGE-1 ↑0.209 / 0.126 / 0.3090.235 / 0.133 / 0.3410.239 / 0.136 / 0.347
STS14MSE ↓0.823 / 5.944 / 0.7050.768 / 5.871 / 0.6450.752 / 5.830 / 0.671
STSBMSE ↓0.796 / 1.676 / 0.7210.752 / 1.486 / 0.6540.739 / 1.510 / 0.641
WMT19 En–ZhBLEU ↑0.223 / 0.024 / 0.1240.232 / 0.026 / 0.1360.231 / 0.029 / 0.139
WMT19 Zh–EnBLEU ↑0.460 / 0.091 / 0.4900.495 / 0.095 / 0.5650.512 / 0.098 / 0.582

We further evaluate PDDS + ADOICLS on STS14 and WMT19 En–Zh (Appendix Section P, Table 31). For each query, we retain exactly the same demonstrations selected by PDDS and change only their order using ADOICLS. The combination outperforms both standalone methods in all six settings. Compared with PDDS, STS14 MSE decreases from 5.944 to 5.499 on Llama, and En–Zh BLEU increases from 0.223 to 0.247 on GLM.

Table 31.

DatasetMetricPDDSADOICLSPDDS + ADOICLS
STS14MSE ↓0.823 / 5.944 / 0.7050.752 / 5.830 / 0.6710.697 / 5.499 / 0.646
WMT19 En–ZhBLEU ↑0.223 / 0.024 / 0.1240.231 / 0.029 / 0.1390.247 / 0.030 / 0.149

These controlled results show that ordering optimization provides additional gains even after PDDS-based selection, demonstrating their complementarity. Both comparisons and protocol details are included in new Appendix Section P, Tables 30 and 31.

W2: Cross-task transfer from classification to generation

We have expanded the evaluation with two additional cross-task directions: classification → text expansion and classification → machine translation, using ADOICLS (Appendix Section Q, Table 32). These complement the original classification → summarization experiment.

Thank you for highlighting the limited scope of the original evaluation. We agree that a single task-transfer direction provides limited evidence for broader generalization. We also acknowledge that the transferred ADOICLE predictor in Table 4 (Section 4) performs below TTF on GLM.

In the new experiments, we train the ADOICLS ordering predictor on classification data (Emotion) and evaluate it on text expansion (Gigatiny) and machine translation (WMT19 Zh–En). Thus, both the task type and the dataset change between training and evaluation.

Each numerical cell below reports GLM / Llama / Qwen. Best baseline includes PDDS and denotes the highest score among the compared non-ADOICL baselines for each target dataset and LLM.

Table 32.

Task transferDataset transferMetricBest baselineTransferred ADOICLS
Classification → Text expansionEmotion → GigatinyROUGE-1 ↑0.209 / 0.129 / 0.3090.225 / 0.132 / 0.326
Classification → Machine translationEmotion → WMT19 Zh–EnBLEU ↑0.469 / 0.092 / 0.5280.485 / 0.094 / 0.537

Transferred ADOICLS outperforms the corresponding best baseline in all six newly evaluated settings. The gain magnitude varies across target LLMs: Llama shows smaller improvements than GLM and Qwen in both directions, with gains of 0.003 ROUGE-1 for classification-to-expansion transfer and 0.002 BLEU for classification-to-translation transfer. We explicitly discuss these modest gains in the revised manuscript.

Together with the original classification-to-summarization experiment, the reported evaluation now covers three transfer directions from classification to three generation task types: summarization, text expansion, and machine translation. These results broaden the evidence for the evaluated classification-to-generation transfers.

We have added the experimental setup, results, and discussion to the Appendix Section Q, Table 32.

AuthorsOpenReview ↗

Response to Weaknesses 3 and Weaknesses 4

W3: Choice of supervision for the ordering predictor

Thank you for raising this concern. We agree that using MSE to regress task performance requires empirical justification, even though the ultimate goal is to select the best demonstration ordering. Our rationale is that observed task scores provide both relative preferences among candidate orderings and the magnitude of their performance differences, which can inform predictor training.

To investigate this, we added a variant supervised by candidates’ ordinal ranks within each query, denoted Rank-based, and examined whether rank supervision improves downstream performance in our implementation (Appendix Section R, Table 33). Each cell below reports GLM / Llama / Qwen; ROUGE-1 and BLEU follow the 0–1 scale used in the paper.

Table 33.

BenchmarkMetricRank-basedADOICLEADOICLS
GigawordROUGE-1 ↑0.060 / 0.061 / 0.1400.065 / 0.081 / 0.1790.069 / 0.078 / 0.183
GigatinyROUGE-1 ↑0.208 / 0.126 / 0.3060.235 / 0.133 / 0.3410.239 / 0.136 / 0.347
STS14MSE ↓0.814 / 6.159 / 0.7020.768 / 5.871 / 0.6450.752 / 5.830 / 0.671
STSBMSE ↓0.798 / 1.636 / 0.6990.752 / 1.486 / 0.6540.739 / 1.510 / 0.641
WMT19 En–ZhBLEU ↑0.218 / 0.025 / 0.1310.232 / 0.026 / 0.1360.231 / 0.029 / 0.139
WMT19 Zh–EnBLEU ↑0.479 / 0.093 / 0.5340.495 / 0.095 / 0.5650.512 / 0.098 / 0.582

Both ADOICLE and ADOICLS outperform the rank-supervised variant in all 18 settings across six benchmarks and three LLMs. Thus, rank supervision does not improve downstream performance in this evaluation, providing empirical support for retaining the original performance-regression approach. This additional experiment specifically examines the choice of supervision targets.

One possible explanation is that ordinal ranks do not fully represent actual performance differences. For a higher-is-better metric, the top-ranked candidate for one query may still score below the fourth-ranked candidate for another query. Rank labels retain the ordering within each query, while task scores additionally preserve score magnitudes and gaps. We hypothesize that this additional information benefits a predictor shared across queries; the revised manuscript presents this as a possible explanation rather than an established causal finding.

We focus this experiment on tasks with graded performance feedback. For classification, candidate correctness is binary, yielding at most two tied relevance levels. Although correct-versus-incorrect preferences remain learnable, the available fine-grained rank information is limited; classification is therefore excluded from this comparison.

We have added these results and clarified the motivation for performance-score supervision in the revised section “Performance-Score versus Rank Supervision” (Appendix Section R, Table 33).

W4: Isolating the contribution of learned group ordering

Thank you for pointing out this missing control. We have added the requested ablation in the 100-shot setting. For each query, we keep the same 100 demonstrations, the same group assignments, and the same within-group orderings. The random baseline changes only the order of the groups, while ADOICLE uses its learned predictor to select the group ordering.

The results are shown below (Appendix Section S, Table 34). Each cell reports GLM / Llama / Qwen.

Table 34.

DatasetMetricRandom group orderingADOICLEGain
GigawordROUGE-1 ↑0.40 / 0.40 / 0.420.44 / 0.54 / 0.48+0.04 / +0.14 / +0.06
EmotionAccuracy (%) ↑56.8 / 65.9 / 58.678.1 / 73.0 / 70.5+21.3 / +7.1 / +11.9
SST5Accuracy (%) ↑59.2 / 48.6 / 39.459.4 / 60.3 / 55.9+0.2 / +11.7 / +16.5

Accuracy gains are in percentage points; ROUGE-1 gains use the reported 0–1 scale.

ADOICLE outperforms random group ordering in all nine model–dataset settings. Because demonstration selection, group assignments, and within-group orderings are held fixed, this comparison isolates the additional contribution of learned group ordering. The results directly support the ordering predictor’s effectiveness within the grouping-based 100-shot framework.

We have added this ablation and its protocol to the revised manuscript in “Effect of Learned Group Ordering in the 100-Shot Setting” (Appendix Section S, Table 34).

Thank you again for your time and constructive feedback. We would appreciate it if you could let us know whether our response has addressed your concerns and whether there are any remaining issues we could clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank Reviewers jBwM, oN2H, and uPm4 for their thoughtful feedback. Even when the demonstrations are fixed, changing their order can substantially affect in-context learning, and the most effective order depends on the query, the examples, and the target LLM. Our contribution is to formulate demonstration ordering as a prediction problem grounded in downstream task performance. ADOICL learns from the scores a target LLM achieves under different orderings offline. It can then reorder the demonstrations returned by an existing retriever for each new query, with only one target-LLM call at inference. Our original evaluation covered 9 target LLMs, 5 tasks, and 8 datasets. In the main comparison across three LLMs and eight datasets, an ADOICL variant achieved the best reported result in 23 of 24 settings (Table 1). The learned ordering predictor transfers across models, across datasets, and even across tasks. The discussion and additional experiments strengthened four findings. Additional evaluations demonstrate ADOICL’s plug-and-play compatibility with different retrieval pipelines and show that its predictor transfers across LLMs, datasets, and tasks. ADOICL also supports visual ICL, few-shot predictor training with only 30 queries, and many-shot demonstration settings with 100 demonstrations.

1. Query-dependent ordering improves performance with the demonstrations held fixed. On Emotion with Llama, the same four examples per query yielded 42.2% accuracy in random order, 47.4% with the best fixed order chosen on validation data, and 58.1% with ADOICL (Table 24). Reordering examples retrieved by BM25 or CD also improved all 12 settings tested (Table 10). During rebuttal, we extended this control to OptiSeq and PDDS. Keeping OptiSeq’s exact examples and changing only their order improved accuracy in all six settings, by 6.0 percentage points on average (Table 37). Reordering the examples selected by PDDS likewise improved all six regression and translation settings (Table 31).

2. ADOICL improves accuracy while reducing online inference time. ADOICL outperformed PDDS in all 24 direct comparisons and both LLM-R and Se² in all 12 settings evaluated (Tables 30 and 35). On SST5, averaged over three LLMs, ADOICL achieved 47.9% accuracy at 0.24 seconds per query. DEmO, OptiSeq, and Self-Adaptive ICL achieved 41.1–42.8% at 0.73–4.27 seconds per query (Table 36).

3. Task-score supervision and broader coverage of training orders improve prediction. On generation and regression tasks, training on actual task scores outperformed training on within-query ordinal ranks in all 18 settings (Table 33). Increasing the sampled training orders per query from five to ten improved all 12 settings while keeping the same 24 candidates at inference. On Emotion with Llama, accuracy rose from 58.1% to 60.4%, halving the gap to the 62.7% oracle (Table 41).

4. The benefits extend across tasks, models, and prompt sizes. A predictor trained on Emotion classification outperformed the strongest compared baseline on both text expansion and machine translation across three LLMs (Table 32). In the 100-shot setting, changing only the order of fixed groups improved all nine comparisons (Table 34). On GPT-5.6 and DeepSeek-v4, ADOICL exceeded the strongest baseline on Emotion by 8.5 and 6.3 accuracy points, respectively, and also led on Gigatiny (Table 42). On Gigatiny, the advantage held across three LLMs under ROUGE-1, ROUGE-2, and ROUGE-L F1. On GLM and Qwen, ADOICL-S achieved this with shorter outputs than every compared baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.