acceptodds
Under review as a conference paper at ICLR 2027

Label Alignment Divergence Reranking for In-Context Demonstration Selection

Abstract

Semantic retrieval identifies demonstrations that resemble a query, but input similarity need not capture their relationship to the task's label distinctions. We study how auxiliary task supervision can improve demonstration selection for classification in-context learning. We propose Label-Alignment Divergence Reranking, which combines semantic similarity with Jensen–Shannon compatibility between label distributions estimated by an auxiliary predictor. The method reranks a fixed semantic candidate pool without scoring candidates with the target language model. Because candidate labels are observed whereas the query label is unknown, we also introduce a gold-anchored extension that anchors candidate posteriors to their observed labels and balances distribution compatibility with label matching according to query posterior concentration. In a controlled evaluation across nine classification tasks with cross-fitted posteriors and three selection seeds, the proposed method reaches macro accuracies of 85.69% and 87.49% on Qwen and Mistral, compared with 84.81% and 82.56% for semantic TopK. Comparisons with direct label-probability matching show that the incremental benefit of the proposed method varies across target models. Selection diagnostics characterize changes within semantic neighborhoods, while reliability analyses identify sensitivity to posterior quality and domain shift. These findings support auxiliary label information as a useful complement to semantic relevance in classification ICL. The code is released here: https://anonymous.4open.science/r/L2D-401B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, large language model, reranking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ELUUOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how to select in-context demonstrations for classification tasks. It proposes LADR, a simple method that combines semantic similarity with auxiliary label-distribution compatibility to rerank candidate demonstrations. The paper also proposes LADR-GA, which further incorporates observed candidate labels. The authors demonstrate the effectiveness of the proposed methods on nine classification tasks.

Critical strengths

[S1] The method is simple and easy to understand, combining semantic similarity with auxiliary label-distribution compatibility to rerank demonstrations.

Critical weaknesses

[W1] Key advantages of ICL are that it requires no parameter updates and uses only a few demonstrations. However, the proposed method requires training models on large labeled datasets. This requirement runs counter to the practical advantages of training-free, few-shot adaptation offered by ICL.

Questions for authors

[Q1] The evaluation is limited to classification tasks. Can the proposed method be extended to open-ended generation tasks, such as code generation or mathematical reasoning?

[Q2] Please provide ablation experiments to isolate the effects of demonstration membership and ordering, as well as gold anchoring and query-posterior concentration gating.

[Q3] The evaluation relies on outdated LLMs, such as Qwen2.5 and Gemma2-2B-it. Please include more recent LLMs.

[Q4] The indicator function in line 404 is not rendered correctly.

[Q5] Since the method requires labeled data to train a LM, please compare it with SFT using the same labeled data.

[Q6] The baseline comparison is not sufficiently comprehensive. Please compare with recent demonstration selection methods [1-6].

[1] Demonstration Selection for In-Context Learning via Reinforcement Learning, ICML 2025.

[2] Sample Efficient Demonstration Selection for In-Context Learning, ICML 2025.

[3] Efficient and effective in-context demonstration selection with coreset, AAAI 2026.

[4] Unified demonstration retriever for in-context learning, ACL 2023.

[5] Strategic demonstration selection for improved fairness in llm in-context learning, ACL 2024.

[6] Performance-Driven Demonstration Selection for In-Context Learning, IJCAI.

2 comments

W1 and Q5. Auxiliary training, frozen-target ICL, and SFT

The APM is trained on labeled task data; the target LLM is frozen. Our setting therefore uses a supervised selector with a small final prompt, rather than assuming that the entire system learns from only eight labels. The memory size NN and prompt size L=8L=8 serve different roles. Training a selector is compatible with frozen-target ICL, as also illustrated by UDR.

SFT is a useful alternative when target updates are permitted, but it answers a different deployment question. We have not run matched SFT and do not claim an accuracy or cost advantage over it. Improving selection under a frozen-target constraint does not require outperforming a system allowed to update that target.

We did evaluate the classifier already trained by our method. In a separate nine-task, three-seed supplementary study, APM-only reaches 87.10%, versus 85.41% for GA with Qwen and 86.44% with Mistral. Direct classification is stronger on macro accuracy in this panel. Our claim concerns selection for the frozen target, not universal superiority of the complete pipeline. These supplementary results do not replace the submitted 85.69% and 87.49% scores.

Q1. Open-ended generation

LADR compares distributions over a shared finite label space. Extending it to code generation or mathematical reasoning requires an appropriate output-compatibility representation and new validation. We claim results for classification; generation is an extension, not an assumption needed for the current method.

Q2. Membership, ordering, anchoring and gating

The supplementary study uses grouped cross-fitting and training-side development splits. Holding each selected set fixed, changing GA from native order to semantic order changes accuracy by -0.246 pp on Qwen, CI [−0.458,−0.021][-0.458,-0.021], and -0.054 pp on Mistral, CI [−0.222,0.120][-0.222,0.120]. All intervals are paired 95% bootstrap intervals.

With common semantic ordering, GA still exceeds TopK by 3.963 pp on Mistral, CI [3.536,4.392][3.536,4.392]. Its benefit therefore includes a membership effect and cannot be explained entirely by presentation order.

We also crossed unanchored/adaptively anchored candidates with fixed-0.5/entropy-dependent mixtures. The components do not help uniformly. The original GA shows no measurable advantage over an exact one-hot replacement. With a fixed anchor p~i=0.5pi+0.5eyi\widetilde p_i=0.5p_i+0.5e_{y_i}, however, retaining the posterior gains 0.172 pp on Qwen and 0.180 pp on Mistral over matched one-hot candidates; both intervals exclude zero and pass Holm correction. This is a benefit of that ablation, not a result attributed to the submitted adaptive rule.

Q3. A recent target model

On Qwen3.5-4B, a frozen screen of 256 queries per task across all nine tasks gives: zero-shot 74.48%, TopK-8 83.25%, GA-8 83.51%, one-hot-8 83.46%, TopK-30 85.07%, and APM-only 86.68%. GA minus TopK-8 is 0.26 pp, CI [−0.95,1.43][-0.95,1.43]. The secondary comparison with zero-shot gives +9.03 pp, CI [7.51,10.59][7.51,10.59]. This uses one APM seed, greedy decoding and thinking disabled. It addresses recency while leaving the GA advantage over TopK unresolved.

Q4. Indicator notation

For clarity, the indicator is written as 1{A}\mathbf{1}\{A\}: it equals 1 if event AA holds and 0 otherwise.

Q6. Recent selection methods

RDES, CASE, UDR and PDDS are relevant comparisons. Their training/search feedback and target-model calls should be included in resource accounting; equal label counts and final shots do not equalize those resources. PDDS also avoids extra target calls at selection time after training.

CoDR targets vision-language ICL and needs a specified text-only adaptation here. Strategic Demonstration Selection optimizes fairness in tabular classification, which an accuracy-only ranking would not evaluate.

For PDDS, the proceedings date is August 2026, within the contemporaneous-work window in the ICLR reviewer FAQ. We discuss its relevance, but the absence of this comparison should not itself be a rejection basis under that policy.

We additionally evaluated full-memory TopK-SD with a shared GTE backbone. At its fixed default interpolation, GA gains 0.497 pp on Qwen and 3.375 pp on Mistral, with positive paired intervals. This supplies a close label-aware control; it is an adaptation, not a reproduction of published scores or a claim to outperform all six suggested methods.

Thank you for your rebuttal, it has addressed the bulk of my concerns.

Reviewer_c3c1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method for improving demonstration selection in in-context learning. The motivation is that semantic similarity between a query and a demonstration does not necessarily imply that their labels agree. To address this, the authors propose label-alignment divergence reranking (LADR), which compares the label distributions of the query and each candidate (estimated with techniques such as APM, whose details are not adequately specified) and uses the gap between them as an additional signal of how useful the candidate is. In experiments, LADR marginally outperforms existing baselines such as GenICL, but the overall results remain unsatisfactory given recent advances in LLMs (since each dataset likely has a performance ceiling due to label noise, reporting this ceiling would help readers judge how much room for improvement remains).

Critical strengths

None

Critical weaknesses
  • Please justify why the research problem discussed in the paper is crucial. The paper aims to solve demonstration selection for in-context learning in 2026, when most current LLMs can solve many tasks reasonably well even without demonstrations or with many examples (many-shot in-context learning). However, the authors still adhere to a configuration that was acceptable several years ago, using only LLMs at the 7B level (relatively small) with just 8 examples, evaluated on classical tasks such as SST-2.
  • Please introduce concepts and models properly before using them. For instance, APM appears in Section 3.2 without a proper definition, with only a brief mention ("An auxiliary posterior model (APM) estimates a label distribution for the query and each candidate.") in the Introduction.
  • The related work section is superficial, considering the large body of papers that investigate the mechanism of in-context learning. There is no justification for what new value this work offers compared to this extensive literature.
Questions for authors
  • Why should the classical setting still deserve attention, given that the proposed method offers only marginal gains over existing baselines that are simple yet effective?
  • In the Introduction, the paper argues that "The prompt budget limits the examples shown to the target, not the labeled data available to train an auxiliary predictor. The question is how to turn that supervision into a useful selection signal without target-LLM candidate scoring." Do the authors believe this still holds in 2026, when we routinely upload a PDF of a thousand pages as context to an LLM? I am concerned that the setting discussed in this paper may be too outdated to be practical.
2 comments

Long-context capability does not by itself make fixed-budget demonstration selection obsolete. We have added an experiment on Qwen3.5-4B to address the concern about current models and the value of demonstrations.

  1. Why study a fixed demonstration budget?

Our setting is a frozen target LLM with access to a labeled memory. The memory size NN and the number of examples placed in a prompt, LL, are separate quantities. Setting L=8L=8 holds the demonstration budget constant across selectors; it does not assume that the model cannot accept a longer prompt.

The thousand-page-PDF example establishes context capacity. It does not establish which demonstrations are useful, what accuracy a shorter prompt can achieve, or whether processing the entire memory is preferable for a particular application. Many-shot ICL provides another operating point. Comparing accuracy at a fixed budget remains a well-defined experimental question.

  1. Evidence on a recent model

We evaluated Qwen3.5-4B on 256 uniformly sampled queries from each of all nine tasks, using one frozen APM seed, greedy decoding and thinking disabled. All conditions share query IDs, task templates, verbalizers and parsing; invalid outputs count as errors.

MethodMacro accuracy (%)
Zero-shot74.48
TopK, 8 shots83.25
LADR-GA, 8 shots83.51
GA-one-hot, 8 shots83.46
TopK, 30 shots85.07
APM alone86.68

In a secondary comparison, GA improves over zero-shot by 9.03 pp, paired 95% query-bootstrap CI [7.51,10.59][7.51,10.59]. Thus demonstrations still have measurable value on this recent model. TopK-30 scores higher than TopK-8, with mean input length increasing from 982 to 3,138 tokens. The results show an accuracy/budget tradeoff; we do not argue that eight shots are always preferable.

GA's increment over TopK-8 is 0.26 pp, CI [−0.95,1.43][-0.95,1.43], and remains unresolved. These intervals condition on one APM seed and nine fixed tasks. This experiment addresses model recency and a modest budget increase; it does not test frontier models or thousand-shot ICL. It also does not establish that GA should replace direct APM classification.

  1. Clarifying APM and supervision

APM means auxiliary posterior model, a supervised classifier estimating pϕ(y∣x)p_\phi(y\mid x) over the task's finite label set. LADR uses these distributions to select demonstrations, after which the frozen target LLM predicts the query label. In cross-fitted evaluation, the candidate's posterior comes from a model that excluded that candidate, and the compatibility term uses the query prediction from the same fold model.

We agree that the definition should precede its first use. The eight-shot constraint applies to the target prompt, not to auxiliary training data. The supplementary controls use a RoBERTa-base APM and report its direct accuracy, so the contribution of the subsequent selection and LLM steps can be assessed explicitly.

  1. Contribution and the size of the gains

Our contribution is an explicit posterior-compatibility reranker and an empirical account of when that signal helps selection. We do not claim to introduce label-aware retrieval. TTF learns task-supervised retrieval embeddings; TopK-SD interpolates representations with class centers. LADR preserves semantic retrieval and uses the label distributions directly in reranking. These are the relevant design differences to assess alongside the broader ICL literature.

The submitted GA scores are 85.69% on Qwen and 87.49% on Mistral. Against Gold-Soft, the 0.07 pp Qwen increment is unresolved, while the 0.83 pp Mistral increment has a positive paired interval. In a separate nine-task, three-seed supplementary study, GA retains a 3.963 pp Mistral advantage over semantic TopK under a common presentation order, CI [3.536,4.392][3.536,4.392]. This establishes a selection-set effect in that protocol, although the original GA does not measurably outperform its exact one-hot counterpart there.

These findings support a specific, model-dependent selection result. We ask that its value be assessed from those comparisons, rather than inferred from benchmark age or maximum context length.

An independently estimated label-noise ceiling would help contextualize absolute accuracy. We do not have such an estimate and cannot infer it from dataset age or published scores. Its absence does not prevent paired comparisons from measuring the effect of a selection policy on the same evaluation examples.

Reviewer_c3c1OpenReview ↗

Response

Thank you for your work.

Still, I believe that the experimental configurations must be properly adapted to reflect the real advances happening in the field every day, rather than rigidly following the previous ones, e.g., by including a wider range of models and more diverse and challenging tasks.

For this reason, I will maintain my current score.

Reviewer_n817OpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LADR, which reranks semantically retrieved demonstrations using Jensen–Shannon compatibility between auxiliary label distributions. LADR-GA additionally incorporates observed candidate labels through posterior anchoring and an entropy-based gate. Experiments cover nine classification tasks, including a controlled study with cross-fitted posteriors, three selection seeds, and two target LLMs.

Critical strengths
  1. Clear and practical motivation: semantic similarity may not reflect task-label compatibility. The method is modular and avoids target-LLM candidate scoring.
  2. Useful controlled evaluation: cross-fitting reduces candidate-training exposure, while Gold-Soft and Gold-Hard provide meaningful label-aware baselines.
  3. Appropriately qualified conclusions: the paper acknowledges task-level regressions, posterior-quality dependence, domain-shift limitations, and target-dependent gains.
Critical weaknesses
  1. The contribution beyond simple label matching is not firmly established. Gold-Soft outperforms base LADR on both target models. LADR-GA improves over Gold-Soft by only 0.07 pp on Qwen, with an interval containing zero, although the 0.83 pp improvement on Mistral is encouraging.

  2. The auxiliary classifier’s standalone accuracy is missing. Since it uses substantial task supervision, its direct predictions are an essential baseline for understanding whether the additional retrieval and LLM inference provide meaningful value.

  3. Key components lack ablations. Anchoring and entropy gating are evaluated only jointly. Comparisons against anchoring-only, gating-only, fixed mixtures, and one-hot candidate labels would clarify their contributions. The anchoring rule also warrants justification: its gold-label mass, is non-monotonic in the original gold-label probability and always at least 0.75.

  4. Protocol differences need clearer documentation. Deterministic TopK results differ substantially between the original and controlled panels—for example, AG News changes from 75.71% to 91.89%. Cross-fitting cannot explain this. Differences in evaluation data, memory, prompts, and ordering should be explicitly listed.

  5. Selection and ordering effects are confounded. Demonstrations are presented in reranker order. Fixed-order comparisons are needed to isolate improvements in set membership.

Questions for authors

NAN

2 comments

We now report the direct APM baseline and controls for candidate labels, anchoring, entropy gating and demonstration order. The Mistral improvement over semantic TopK persists when demonstration order is held to a common rule.

The supplementary study covers nine tasks and three seeds, with grouped cross-fitting and a 256-example training-side development split per task. It uses a separate data preparation protocol. The submitted GA results remain 85.69% on Qwen and 87.49% on Mistral; all intervals below are paired 95% bootstrap intervals.

  1. Direct APM prediction

We evaluated y^APM=arg⁡max⁡ypˉx(y)\widehat y_{\mathrm{APM}}=\arg\max_y\bar p_x(y) on the same query IDs and fold predictions used by the supplementary ICL runs:

MethodQwen accuracy (%)Mistral accuracy (%)
APM alone87.1087.10
Semantic TopK84.7182.42
Gold-Soft85.4886.64
GA-one-hot85.4986.51
LADR-GA85.4186.44

APM-only has a three-seed SD of 0.13 pp and is independent of the target LLM. It is the stronger choice on macro accuracy in this panel if the application can use the classifier directly. These results show an improvement in demonstration selection for a frozen target; they do not establish that the full ICL pipeline is generally preferable to direct classification.

  1. Separating membership and ordering

We held each selected set fixed and replaced native reranker order with the original semantic order. GA changes by -0.246 pp on Qwen, CI [−0.458,−0.021][-0.458,-0.021], and -0.054 pp on Mistral, CI [−0.222,0.120][-0.222,0.120].

Under this common ordering rule, GA still exceeds TopK by 3.963 pp on Mistral, CI [3.536,4.392][3.536,4.392]. Thus its Mistral benefit survives the ordering control and includes a set-membership effect. Qwen's corresponding difference is 0.460 pp, CI [0.085,0.836][0.085,0.836]; its Holm-adjusted p-value is 0.052, so it does not pass the adjusted criterion for this comparison family. We do not make the same significance claim for Qwen.

  1. One-hot, anchoring and gating controls

The exact one-hot replacement gives GA-minus-one-hot differences of -0.079 pp on Qwen, CI [−0.263,0.112][-0.263,0.112], and -0.077 pp on Mistral, CI [−0.244,0.096][-0.244,0.096]. The original adaptive candidate representation therefore has no measurable advantage in this panel. Gold-Soft also remains a strong comparator: the submitted +0.83 pp Mistral gain is specific to that evaluation protocol.

To separate anchoring from gating, we crossed the candidate representation with a fixed 0.5 mixture or the entropy-dependent mixture, keeping semantic presentation order:

Candidate representation; gateQwen (%)Mistral (%)
Unanchored posterior; fixed 0.585.32986.529
Adaptive anchor; fixed 0.585.23286.358
Unanchored posterior; entropy gate85.40686.536
Adaptive anchor; entropy gate85.16886.384

These controls isolate anchoring at each gate setting and compare entropy gating with a fixed mixture. Neither component helps uniformly. The anchor-by-gate interaction is -0.141 pp on Qwen, CI [−0.284,−0.005][-0.284,-0.005], and unresolved on Mistral.

A further ablation fixes the candidate anchor at p~i=0.5pi+0.5eyi\widetilde p_i=0.5p_i+0.5e_{y_i}. Relative to matched one-hot candidates, it gains 0.172 pp on Qwen, CI [0.018,0.320][0.018,0.320], and 0.180 pp on Mistral, CI [0.007,0.351][0.007,0.351]; both pass Holm correction. This shows a small benefit from residual posterior information under that anchor, while leaving the original adaptive rule open to improvement.

For the original rule, p~i(yi)=1−ti+ti2\widetilde p_i(y_i)=1-t_i+t_i^2 with ti=pi(yi)t_i=p_i(y_i). We agree that it is non-monotonic. It is a construction using the observed label, not a calibrated confidence estimate: residual mass is largest at ti=0.5t_i=0.5 and vanishes at either endpoint. The ablations provide an empirical test of that heuristic. Similarly, low query entropy alone does not guarantee correctness.

  1. Protocol differences

We agree that deterministic TopK changes cannot be attributed to cross-fitting. A verified account requires the original memory, query, retrieval, prompt, verbalizer, model and parsing configurations. We have not reconstructed that mapping, and do not treat the two submitted panels as a matched cross-fitting ablation. The new controls establish the effects above within their explicitly shared protocol.

Thank you for addressing my concerns.

I improved my scores because the first concern has been solved.

Reason for the weak rejection: the utilized dataset and models are too old. These datasets and models cannot reflect real-world use, while this direction is purely driven by application. You may give an on-device story if you want to use the same dataset or use SWE, hle and RR benchmakr instead.

And demonstration-based examples do not work well for advanced models.

Reviewer_9xPKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies classification in-context learning with labeled task data and a frozen target language model. LADR retrieves 30 semantic neighbors, then selects eight demonstrations using a weighted combination of semantic similarity and Jensen–Shannon compatibility between label distributions predicted by an auxiliary classifier. LADR-GA additionally anchors candidate posteriors to their observed labels and uses query-posterior entropy to mix direct label matching with distribution compatibility. The eight-example limit applies to the final prompt, not to the total supervision available for auxiliary training. Validation includes a nine-task benchmark and a separate controlled study with three-fold cross-fitting, three selection seeds and two target LLMs. LADR-GA reaches macro accuracies of 85.69% on Qwen and 87.49% on Mistral. It improves over semantic TopK on average; its additional gain over Gold-Soft is supported by the reported paired interval on Mistral but unresolved on Qwen. Selection and reliability analyses examine prompt composition, posterior quality, label mappings, candidate exposure, overlap and domain shift.

Critical strengths
  1. A reusable selection strategy with a meaningful gain over a strong simple comparator.
  2. Informative controls and explicit boundary conditions strengthen the empirical contribution.
Critical weaknesses
  1. The distinctive value of candidate-posterior scoring is not sufficiently established.

(1) Closely related work already uses task labels to improve retrieval: TTF fine-tunes a retriever on the demonstration data (Liu et al., “Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning,” ACL 2025, Section 4.2), while TopK-SD incorporates label information through class-center interpolation (Chen et al., “Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective,” AAAI 2026). LADR is not equivalent to these methods, but its contribution must be established through its particular design and findings, rather than the general usefulness of label information.

(2) In Table 2, base LADR is below Gold-Soft by 0.15 points on Qwen and 1.88 points on Mistral. GA provides a meaningful Mistral gain, but its Qwen margin is only 0.07 points, with a reported paired interval of [−0.14, +0.27]. Equation (6) implies an anchored observed-label mass of 1 − t + t² ≥ 0.75, where t is the original probability of that label; Table 3(c) reports an overlap of 7.27/8 demonstrations with Gold-Soft. Neither observation establishes equivalence, but both motivate a control replacing the anchored candidate posterior with the observed-label one-hot vector while preserving the rest of GA.

(3) Without that control, or a closely matched comparison to task-supervised embedding retrieval, the value of the richer candidate representation remains difficult to assess. I do not require every component to be indispensable or the method to win on every model. The issue is whether the demonstrated increment is sufficiently distinct and substantial to support acceptance.

  1. Important differences between the two evaluation protocols remain under-specified.

(1) The deterministic Qwen TopK baseline changes substantially between Tables 1 and 5: 75.71% versus 91.89% on AG News, 92.45% versus 85.95% on Subj, and 94.29% versus 88.96% on Banking77. The authors appropriately separate the protocols, but TopK does not use auxiliary posteriors; cross-fitting alone therefore cannot explain these changes.

(2) Section 4 needs a more explicit account of changes in memory construction, evaluation examples, prompts, verbalizers, model configuration and output processing. This is not evidence of leakage or invalid within-protocol comparisons. It nevertheless limits reproducibility and interpretation: the Qwen LADR–TopK gain changes from 3.01 to 0.66 points, and this cannot be attributed to cross-fitting without explaining the other differences. The relationship between the broad and controlled evidence consequently remains unclear.

Questions for authors

1.What are the specific differences between the original benchmark and the controlled evaluation in terms of demonstration memory construction, evaluation examples, retrieval configuration, prompt templates, verbalizers, target-model settings, and output processing? In particular, which differences account for the substantial changes in deterministic TopK accuracy between Tables 1 and 5? Additionally, which data partitions were used to develop the mixing weight and LADR-GA formulation on CR, SST-2, and Subj, and how were these partitions separated from the reported evaluation data? Clarifying these details would materially affect my assessment of reproducibility and the interpretation of the reported improvements.

2.Under the same cross-fitted evaluation protocol, how would LADR-GA perform if the gold-anchored candidate posterior were replaced by a one-hot vector of its observed label, while keeping the fold-matched query predictions, entropy gating, semantic weighting, candidate pool, and demonstration ordering unchanged? Does retaining the full candidate posterior provide a measurable advantage over this simpler alternative, particularly on Mistral? This comparison would help establish whether the proposed method benefits from residual information in the candidate posterior or primarily from a more elaborate form of observed-label matching.

3.How does the proposed approach compare with TTF-style task-supervised retrieval or retrieval using task-supervised embeddings from the same auxiliary backbone, under matched labeled training data and an identical eight-demonstration budget? Such alternatives need not be restricted to LADR's fixed semantic candidate pool, since modifying the retrieval representation is part of their design. Do the authors have evidence of advantages in predictive accuracy, cross-model reuse, or computational efficiency that would establish the distinctive value of explicit posterior-based reranking relative to these closely related alternatives?

1 comment

We have evaluated the requested one-hot control, a fixed-anchor ablation, and a full-memory label-aware retrieval baseline. These controls distinguish the contribution of candidate-posterior information from the effect of the particular adaptive anchoring rule.

  1. What is new relative to label-aware retrieval?

We agree that using task labels for selection is not itself new. TTF learns task-supervised retrieval representations; TopK-SD interpolates embeddings with class centers. LADR instead keeps the semantic retriever fixed and makes posterior compatibility an explicit reranking signal. Our contribution should be judged by this design and its empirical behavior, including the comparison with observed-label matching.

The submitted GA scores are 85.69% on Qwen and 87.49% on Mistral. The gains over Gold-Soft are 0.07 and 0.83 pp, respectively. We agree that the Qwen interval [−0.14,+0.27][-0.14,+0.27] does not establish an improvement.

  1. Does the candidate posterior add information beyond one-hot labels?

We evaluated the exact replacement requested: only the anchored candidate vector changes to eyie_{y_i}. Fold-matched query predictions, entropy gating, semantic weighting, the Top-30 pool and the native ordering rule remain fixed.

This supplementary study covers all nine tasks and three seeds, using grouped cross-fitting and a 256-example training-side development split per task. Because its data preparation differs, we report it separately from the submitted evaluation. Paired macro differences are:

ContrastQwen, pp [95% CI]Mistral, pp [95% CI]
GA minus TopK+0.707 [0.308, 1.100]+4.017 [3.606, 4.430]
GA minus Gold-Soft-0.064 [-0.270, 0.133]-0.201 [-0.368, -0.031]
GA minus one-hot-0.079 [-0.263, 0.112]-0.077 [-0.244, 0.096]

The original adaptive GA does not show a measurable advantage over one-hot here. The submitted Mistral gain over Gold-Soft also does not persist in this supplementary protocol.

However, a matched ablation with p~i=0.5pi+0.5eyi\widetilde p_i=0.5p_i+0.5e_{y_i} and common semantic presentation order does benefit from the candidate posterior: +0.172 pp on Qwen, CI [0.018,0.320][0.018,0.320], and +0.180 pp on Mistral, CI [0.007,0.351][0.007,0.351], relative to one-hot. Holm-adjusted p-values are 0.0043 and 0.0131. This provides evidence that residual posterior information can help. It supports that specific variant, rather than establishing the superiority of the submitted adaptive anchor.

Your anchoring calculation is correct: for ti=pi(yi)t_i=p_i(y_i), p~i(yi)=1−ti+ti2≥0.75\widetilde p_i(y_i)=1-t_i+t_i^2\geq0.75. The bound and 7.27/8 overlap motivate the ablation, but cannot determine its outcome. Gold-Soft is also not the exact replacement: for u=p(yi)u=p(y_i), 1−JS2(p,eyi)=((1+u)log⁡2(1+u)−ulog⁡2u)/21-\mathrm{JS}_2(p,e_{y_i})=((1+u)\log_2(1+u)-u\log_2u)/2. The one-hot GA control retains this nonlinear compatibility term and entropy mixing.

  1. Comparison with task-supervised retrieval

We allowed TopK-SD to retrieve from the full memory. With a shared GTE backbone and its fixed default interpolation, GA exceeds this adaptation by 0.497 pp on Qwen, CI [0.095,0.893][0.095,0.893], and 3.375 pp on Mistral, CI [2.968,3.784][2.968,3.784]. This is a controlled implementation comparison, not a reproduction of its published scores.

Our TTF implementation uses BERT while the APM uses RoBERTa, so it does not answer the exact same-backbone comparison. We therefore make no claim of superiority over matched TTF. LADR's selections can be reused across targets without target-model candidate scoring, but we do not claim that this property is unique or that it proves a computational advantage over TTF.

  1. Why does deterministic TopK change between the original panels?

We agree that cross-fitting alone cannot explain these changes. We cannot provide a verified reconstruction of the original memory/query IDs, prompts, verbalizers, model settings and parsing changes from the available records. Nor can we verify the original CR/SST-2/Subj development partitions. We therefore do not interpret the +3.01 versus +0.66 pp gains as an effect of cross-fitting.

The supplementary comparisons use shared memory and evaluation IDs, retrieval configuration, task templates, verbalizers, target settings and parsing, with the selector or ordering intervention specified explicitly. They answer the component questions under a controlled protocol; the original panel mapping remains a separate reproducibility issue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.