- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how to select in-context demonstrations for classification tasks. It proposes LADR, a simple method that combines semantic similarity with auxiliary label-distribution compatibility to rerank candidate demonstrations. The paper also proposes LADR-GA, which further incorporates observed candidate labels. The authors demonstrate the effectiveness of the proposed methods on nine classification tasks.
Critical strengths
[S1] The method is simple and easy to understand, combining semantic similarity with auxiliary label-distribution compatibility to rerank demonstrations.
Critical weaknesses
[W1] Key advantages of ICL are that it requires no parameter updates and uses only a few demonstrations. However, the proposed method requires training models on large labeled datasets. This requirement runs counter to the practical advantages of training-free, few-shot adaptation offered by ICL.
Questions for authors
[Q1] The evaluation is limited to classification tasks. Can the proposed method be extended to open-ended generation tasks, such as code generation or mathematical reasoning?
[Q2] Please provide ablation experiments to isolate the effects of demonstration membership and ordering, as well as gold anchoring and query-posterior concentration gating.
[Q3] The evaluation relies on outdated LLMs, such as Qwen2.5 and Gemma2-2B-it. Please include more recent LLMs.
[Q4] The indicator function in line 404 is not rendered correctly.
[Q5] Since the method requires labeled data to train a LM, please compare it with SFT using the same labeled data.
[Q6] The baseline comparison is not sufficiently comprehensive. Please compare with recent demonstration selection methods [1-6].
[1] Demonstration Selection for In-Context Learning via Reinforcement Learning, ICML 2025.
[2] Sample Efficient Demonstration Selection for In-Context Learning, ICML 2025.
[3] Efficient and effective in-context demonstration selection with coreset, AAAI 2026.
[4] Unified demonstration retriever for in-context learning, ACL 2023.
[5] Strategic demonstration selection for improved fairness in llm in-context learning, ACL 2024.
[6] Performance-Driven Demonstration Selection for In-Context Learning, IJCAI.
2 comments
Thank you for your rebuttal, it has addressed the bulk of my concerns.
W1 and Q5. Auxiliary training, frozen-target ICL, and SFT
The APM is trained on labeled task data; the target LLM is frozen. Our setting therefore uses a supervised selector with a small final prompt, rather than assuming that the entire system learns from only eight labels. The memory size N and prompt size L=8 serve different roles. Training a selector is compatible with frozen-target ICL, as also illustrated by UDR.
SFT is a useful alternative when target updates are permitted, but it answers a different deployment question. We have not run matched SFT and do not claim an accuracy or cost advantage over it. Improving selection under a frozen-target constraint does not require outperforming a system allowed to update that target.
We did evaluate the classifier already trained by our method. In a separate nine-task, three-seed supplementary study, APM-only reaches 87.10%, versus 85.41% for GA with Qwen and 86.44% with Mistral. Direct classification is stronger on macro accuracy in this panel. Our claim concerns selection for the frozen target, not universal superiority of the complete pipeline. These supplementary results do not replace the submitted 85.69% and 87.49% scores.
Q1. Open-ended generation
LADR compares distributions over a shared finite label space. Extending it to code generation or mathematical reasoning requires an appropriate output-compatibility representation and new validation. We claim results for classification; generation is an extension, not an assumption needed for the current method.
Q2. Membership, ordering, anchoring and gating
The supplementary study uses grouped cross-fitting and training-side development splits. Holding each selected set fixed, changing GA from native order to semantic order changes accuracy by -0.246 pp on Qwen, CI [−0.458,−0.021], and -0.054 pp on Mistral, CI [−0.222,0.120]. All intervals are paired 95% bootstrap intervals.
With common semantic ordering, GA still exceeds TopK by 3.963 pp on Mistral, CI [3.536,4.392]. Its benefit therefore includes a membership effect and cannot be explained entirely by presentation order.
We also crossed unanchored/adaptively anchored candidates with fixed-0.5/entropy-dependent mixtures. The components do not help uniformly. The original GA shows no measurable advantage over an exact one-hot replacement. With a fixed anchor pi=0.5pi+0.5eyi, however, retaining the posterior gains 0.172 pp on Qwen and 0.180 pp on Mistral over matched one-hot candidates; both intervals exclude zero and pass Holm correction. This is a benefit of that ablation, not a result attributed to the submitted adaptive rule.
Q3. A recent target model
On Qwen3.5-4B, a frozen screen of 256 queries per task across all nine tasks gives: zero-shot 74.48%, TopK-8 83.25%, GA-8 83.51%, one-hot-8 83.46%, TopK-30 85.07%, and APM-only 86.68%. GA minus TopK-8 is 0.26 pp, CI [−0.95,1.43]. The secondary comparison with zero-shot gives +9.03 pp, CI [7.51,10.59]. This uses one APM seed, greedy decoding and thinking disabled. It addresses recency while leaving the GA advantage over TopK unresolved.
Q4. Indicator notation
For clarity, the indicator is written as 1{A}: it equals 1 if event A holds and 0 otherwise.
Q6. Recent selection methods
RDES, CASE, UDR and PDDS are relevant comparisons. Their training/search feedback and target-model calls should be included in resource accounting; equal label counts and final shots do not equalize those resources. PDDS also avoids extra target calls at selection time after training.
CoDR targets vision-language ICL and needs a specified text-only adaptation here. Strategic Demonstration Selection optimizes fairness in tabular classification, which an accuracy-only ranking would not evaluate.
For PDDS, the proceedings date is August 2026, within the contemporaneous-work window in the ICLR reviewer FAQ. We discuss its relevance, but the absence of this comparison should not itself be a rejection basis under that policy.
We additionally evaluated full-memory TopK-SD with a shared GTE backbone. At its fixed default interpolation, GA gains 0.497 pp on Qwen and 3.375 pp on Mistral, with positive paired intervals. This supplies a close label-aware control; it is an adaptation, not a reproduction of published scores or a claim to outperform all six suggested methods.