- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ADOICL, which formulates demonstration ordering for in-context learning as a performance-prediction problem. Rather than relying on task-agnostic proxy scores, it learns a lightweight ordering predictor from offline performance supervision: the predictor estimates the downstream task performance of candidate demonstration orderings conditioned on the input query, the retrieved demonstrations, and the target LLM, and selects the best-predicted ordering before a single target-LLM inference. The framework is instantiated in two forms: ADOICLS (a sequence-level scorer over serialized query-ordering pairs) and ADOICLE (an embedding-first scorer that reuses query and demonstration embeddings to reduce encoding from k! to k). The method is validated across 9 target LLMs (1B–70B, including GPT-4o), 5 tasks, and 8 datasets spanning classification, regression, and generation, with ablations (removing the predictor costs 8.57% accuracy; the dual-input architecture adds 4.58%), generalization studies (cross-LLM, cross-dataset, cross-task, cross-demonstration-number), and extensions to plug-and-play, visual ICL, few-shot, and many-shot settings.
Critical strengths
-
A clean, well-motivated problem reframing backed by direct evidence. The paper treats demonstration ordering as a performance-prediction problem, directly regressing the target LLM's realized task performance instead of relying on task-agnostic proxy scores.
-
Thorough and well-ablated validation. The evaluation spans 9 target LLMs, 5 tasks, and 8 datasets covering classification, regression, and generation, with clean ablations , broad generalization studies, and extensions to plug-and-play, visual ICL, few-shot, and many-shot settings.
Critical weaknesses
-
Key results are left unanalyzed, especially the coarse binary supervision for classification. The per-instance supervision for classification is binary (0/1), so the predictor learns "correct or not" rather than "how good", which may prevent it from distinguishing orderings that are both correct (or both wrong) — yet classification is where the method shows its largest gains, and this tension is not discussed.
-
The evaluation does not cover the current frontier of LLMs. The tested open-source models cap at 70B, and the only commercial model evaluated is GPT-4o, which is no longer the newest frontier model. Robustness on the latest open- and closed-source models is therefore unverified.
Questions for authors
-
The oracle gap remains about 4.6 points on Emotion+Llama. What limits the predictor from reaching the oracle, can you analyze this?
-
For classification, the per-instance supervision is binary (0/1), so the predictor learns "correct or not" rather than "how good." Does this limit its ability to distinguish between orderings that are both correct (or both wrong)? Would a smoother (e.g., probability- or margin-based) supervision improve it?
-
The evaluation's open-source models cap at 70B and the only commercial model is GPT-4o. Can you extend the evaluation to the current frontier LLMs (GPT-5.6 / DeepSeek-v4 / Claude-4.8) to confirm the gains hold at the frontier?
if concerns can be adequately addressed, I would be willing to raise my score to 4.
6 comments
Response to Weaknesses 2, Question 1, and Question 3
Q1: What limits the predictor from reaching the oracle?
Thank you for suggesting this analysis. Our additional experiment identifies insufficient training data as an important contributor to the remaining oracle gap.
We increase the sampled orderings per training query from M = 5 to M = 10, doubling the number of query–ordering–score training examples while keeping the training queries, predictor architecture, and supervision definition unchanged. Both settings score all 24 permutations at test time, so this change increases only the training data, without expanding the inference-time search space.
The results are shown below (Appendix Section X, Table 41). Each cell reports GLM / Llama / Qwen.
Table 41.
| Dataset | Metric | ADOICLS-5 | ADOICLS-10 | Oracle |
|---|---|---|---|---|
| Emotion | Accuracy (%) ↑ | 71.4 / 58.1 / 68.6 | 73.1 / 60.4 / 73.6 | 75.6 / 62.7 / 76.1 |
| SST5 | Accuracy (%) ↑ | 52.0 / 40.3 / 53.8 | 55.1 / 42.5 / 57.4 | 58.2 / 46.4 / 61.2 |
| Gigaword | ROUGE-1 ↑ | 0.069 / 0.078 / 0.183 | 0.074 / 0.080 / 0.187 | 0.078 / 0.087 / 0.199 |
| Gigatiny | ROUGE-1 ↑ | 0.239 / 0.136 / 0.347 | 0.258 / 0.152 / 0.362 | 0.261 / 0.163 / 0.394 |
On Emotion + Llama, accuracy increases from 58.1% to 60.4%, reducing the gap to the 62.7% oracle from 4.6 to 2.3 percentage points—a 50% reduction. Improvements occur in all 12 model–dataset settings.
These results support the explanation that sampling only five orderings per query provides insufficient training supervision to fully learn ordering-dependent performance. Additional training data substantially narrows the gap, even with unchanged binary supervision for classification. Although the residual gap cannot be attributed entirely to data scarcity, this experiment demonstrates that limited training data is an important bottleneck. The additional cost is incurred offline; test-time scoring and the single final target-LLM inference remain unchanged. % TODO: Add response and supporting evidence.
W2 and Q3: Evaluation on frontier LLMs
Thank you for highlighting this limitation. We have extended our evaluation to GPT-5.6 and DeepSeek-v4, covering classification on Emotion and text expansion on Gigatiny (Appendix Section Y, Table 42). These experiments complement our original GPT-4o results (Section 4, Table 6).
Table 42.
| Method | GPT-5.6: Emotion Acc. (%) ↑ | GPT-5.6: Gigatiny ROUGE-1 ↑ | DeepSeek-v4: Emotion Acc. (%) ↑ | DeepSeek-v4: Gigatiny ROUGE-1 ↑ |
|---|---|---|---|---|
| BM25 | 67.6 | 0.165 | 65.6 | 0.194 |
| CR | 56.1 | 0.275 | 51.8 | 0.310 |
| CD | 67.4 | 0.303 | 69.9 | 0.312 |
| Kate | 69.7 | 0.352 | 70.1 | 0.320 |
| PLR | 70.0 | 0.316 | 70.6 | 0.323 |
| CDS | 68.9 | 0.341 | 65.0 | 0.323 |
| TTF | 71.6 | 0.330 | 68.7 | 0.310 |
| GlobalE | 59.0 | 0.254 | 59.0 | 0.228 |
| ADOICLE | 78.8 | 0.397 | 76.9 | 0.369 |
| ADOICLS | 80.1 | 0.404 | 76.6 | 0.384 |
Both ADOICL variants outperform all compared baselines in all four settings. Specifically:
- GPT-5.6: ADOICLS improves Emotion accuracy by 8.5 percentage points over the strongest baseline, TTF (80.1% vs. 71.6%), and Gigatiny ROUGE-1 by 0.052 over Kate (0.404 vs. 0.352).
- DeepSeek-v4: ADOICLE improves Emotion accuracy by 6.3 percentage points over PLR (76.9% vs. 70.6%). ADOICLS improves Gigatiny ROUGE-1 by 0.061 over the strongest baselines, PLR and CDS (0.384 vs. 0.323).
These results provide additional evidence that ADOICL’s benefits extend to the two newly evaluated models across classification and generation. We have included the full results and discussion in the appendix.
Unfortunately, API access restrictions prevented us from evaluating Claude. We acknowledge this remaining limitation and restrict our conclusions to the evaluated models and tasks.
Thank you again for your time and constructive feedback. We would appreciate it if you could let us know whether our response has addressed your concerns and whether there are any remaining issues we could clarify.
Response to Weakness1 and Question2
Thank you for your thoughtful review. We appreciate your recognition of the motivation behind our formulation and the breadth of our experimental validation. Your constructive comments on supervision granularity, the remaining oracle gap, and evaluation on frontier LLMs have helped us strengthen the analysis. We address these points below with additional experiments and references to the corresponding sections and tables in the revised manuscript.
W1 and Q2: Binary classification supervision and finer-grained alternatives
Thank you for this insightful suggestion. We agree that finer-grained supervision can improve learning when it captures meaningful differences in downstream performance. We have added two complementary experiments to investigate this issue (Appendix Section W, Tables 39 and 40).
Why binary supervision can still yield substantial classification gains. Binary correctness does not distinguish two correct orderings or two incorrect orderings. However, these orderings have equal utility under the accuracy metric. Improving accuracy requires selecting a correct ordering when one is available, rather than distinguishing among all correct orderings. Thus, the observed classification gains are compatible with binary supervision, although richer signals could further improve learning and generalization.
1. Classification: output log-likelihood supervision.
We evaluated complete-output log-likelihood as a continuous alternative (Appendix Section W, Table 39). A practical limitation of our current scoring pipeline is that it provides scores for generated responses, rather than separate scores for the ground-truth label and competing classes. These scores alone cannot determine label-specific probabilities or class margins; obtaining those targets requires an additional class-conditional scoring procedure. Accordingly, this experiment partially explores the suggestion rather than directly testing ground-truth label probabilities or margins.
Each cell below reports GLM / Llama / Qwen.
Table 39.
Output log-likelihood improves three settings and underperforms in three, with mean accuracy of 56.88%, compared with 57.37% for binary correctness. One possible explanation is that likelihood measures confidence in the generated answer, which may be incorrect. These results therefore do not imply that finer-grained supervision is unhelpful; they show that a continuous confidence score is not necessarily better aligned with classification accuracy.
2. Generation and regression: coarsening continuous task scores.
To examine the value of fine-grained task-performance information, we normalized the training scores using training-set minimum and maximum values, then mapped the intervals [0, 0.33), [0.33, 0.66), and [0.66, 1] to 0, 1, and 2, respectively. We retained the predictor architecture and MSE training objective. Higher targets indicate better generation quality; lower targets indicate better STS performance because STS supervision uses squared prediction error (Appendix Section W, Table 40).
Table 40.
Original continuous supervision performs better in all 18 settings. This supports the reviewer’s intuition that preserving fine-grained performance distinctions benefits ordering prediction. Since discretization also changes the target scale, we interpret this as supporting evidence rather than a complete isolation of granularity effects.
Together, these experiments support the importance of both supervision granularity and alignment with task performance. We have included the results and discussion in the appendix, while explicitly acknowledging that ground-truth label probability and margin supervision remain untested.