acceptodds
Under review as a conference paper at ICLR 2027

When Does Posterior Geometry Matter for In-Context Demonstration Selection?

Abstract

Posterior-based demonstration selection uses label-distribution compatibility to refine semantic retrieval. Understanding these selections requires identifying which posterior components can change the selected examples. We analyze candidate posterior tails, comprising probabilities assigned to labels other than the observed candidate label, at fixed query posteriors and observed-label probabilities. For Jensen–Shannon alignment, we derive exact attainable score intervals and establish a necessary and sufficient condition for selection invariance. Every non-invariant observed set admits a constructive counterexample involving at most two candidate tails. This characterization determines when posterior variation can reverse the ordering at the selection boundary. A five-task audit certifies 32.71% of selected sets under equal task and seed weighting, with rates ranging from 0.05% to 89.64%. In controlled nine-task comparisons, full posterior alignment improves macro accuracy over a query-label control by 0.83 percentage points on Mistral-7B, while the two selectors achieve similar means on Qwen2.5-7B. The certificate permits recovery of the full selector's demonstration set without candidate-tail access on certified queries. Code is available at https://anonymous.4open.science/r/posterior-geometry-5376.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, large language models, demonstration selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_58pAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when the posterior geometry of candidate demonstrations actually matters for selecting demonstrations in in-context learning based on posterior alignment. It focuses on a score called Full Posterior Alignment (FPA), and breaks the Jensen–Shannon divergence between the query posterior and a candidate posterior into two parts: how much probability mass goes to the observed label, and how the remaining probability is spread over competing labels—the tail geometry. The authors derive the exact range that the tail part can take when the observed-label probability is fixed, and they characterize when the Top-L selected set stays the same no matter how that tail is reallocated. They also give rank bounds and build counterexamples. In addition, they introduce a certification procedure that tells when candidate-tail information is not needed for selection. Finally, they test FPA and related selectors on several classification benchmarks and language models.

Critical strengths
  1. The paper provides a coherent analysis of posterior-tail geometry under fixed observed-label mass, including an exact attainable interval for the JS contribution and a necessary-and-sufficient condition for Top-L membership invariance. The constructive counterexamples and rank bounds further strengthen the theoretical characterization, making the result more than a qualitative observation.

  2. The authors conduct extensive numerical checks of the theoretical claims and evaluate the proposed FPA selector across multiple tasks and models. In particular, the results show that FPA can provide measurable gains over the label-mass baseline and that the proposed certificate can identify a substantial fraction of cases where candidate-tail information is unnecessary.

Critical weaknesses
  1. FPA differs from the Label-Mass baseline in several components beyond the competing-label tail, including observed-label probability, Bernoulli-level JS divergence, entropy-based weighting, and candidate-specific query posteriors. Therefore, the reported accuracy improvements do not establish that posterior-tail geometry itself is responsible for the gains. This is consequential because demonstrating the practical importance of posterior geometry is the central motivation of the paper.

  2. The proposed uncertainty set allows arbitrary reallocations of the candidate posterior tail while fixing only the observed-label mass, which may be substantially broader than realistic posterior variations. Moreover, certification coverage varies dramatically across tasks (from 89.64% on AG News to only 0.05% on Banking77). Thus, the theoretical non-invariance results establish worst-case sensitivity, but the paper provides limited evidence that such tail variations occur in practice or materially affect demonstration selection.

Questions for authors
  1. Eq. (11) and (12) show that FPA and the Label Mass baseline differ in more than just the competing label tail geometry. They also differ in the observed label mass term, the Bernoulli level JS divergence, and the entropy based weighting c. Could the authors check whether the gains in Table 1 still hold when they compare Label Mass to the coarse score M_i, and then to full FPA? This would isolate how much the tail geometry term G_i actually contributes.

  2. Appendix C.2 says the scoring mixture and anchor were "developed" on CR, SST-2, and Subj. But these three tasks are also in Suite A in Table 1. What does "developed" mean here exactly? Did the authors use these three tasks to choose hyperparameters, design choices, or models? If so, how do the FPA results change under a strict held out evaluation setup?

  3. Eq. (10) lets the candidate posterior tail be reallocated in any way as long as b_i is fixed. So Theorem 3 gives worst case invariance or non invariance under a very broad uncertainty set. Do the authors have evidence that these allowed tail reallocations match realistic posterior variation, for example across APM seeds or other sources of model variation? How often do such realistic perturbations actually change the Top L set?

  4. Theorem 3 only guarantees that Top L membership does not change. Appendix A.6 says the order of the selected demonstrations can still change. In the experiments, is the demonstration order fixed after selection, independent of the FPA score? Or can a change in the score change the order shown to the target LLM? If the order can change, how do the authors separate the reported gains from possible order effects instead of from membership selection effects?

1 comment

Thank you for the careful assessment. We agree that the original comparison with Label-Mass did not isolate posterior-tail geometry. We therefore added controlled scoring comparisons, a new held-out task, and an audit of actual APM seed variation and tail transfers.

1. Isolating posterior-tail information

The submitted +0.83 pp Mistral improvement compares complete FPA with Label-Mass and cannot be attributed solely to posterior tails. From Si=Mi−κGiS_i=M_i-\kappa G_i, where κ=(1−λ)(1−c)\kappa=(1-\lambda)(1-c), MiM_i retains the observed-label mass, Bernoulli JS term, entropy mixture, and candidate-specific query posteriors. Comparing FPA with MiM_i therefore isolates the tail residual.

We evaluated these three methods on five tasks, using 512 queries per task, three APM seeds, K=30K=30, L=8L=8, and λ=0.5\lambda=0.5. All methods use a common fixed semantic demonstration order, so equal selected sets produce identical prompts.

TargetLabel-MassMiM_iFPA
Qwen2.5-7B81.2281.2581.21
Mistral-7B83.0682.2082.29

FPA minus MiM_i is -0.04 pp (95% CI [-0.23, +0.16]) for Qwen and +0.09 pp (95% CI [-0.21, +0.39]) for Mistral. Membership differs in 21.50% of the 7,680 query-seed cases, while all 2,558 certified cases agree. Notably, FPA is 0.77 pp below Label-Mass on Mistral in this supplementary panel. We retain this result rather than attributing the original gain to tail geometry.

2. Development history and held-out evaluation

We clarify that historical development included a Qwen2.5-7B mixture-weight sweep on CR, SST-2, and Subj, and SST-2 pilot results informed the anchor/confidence-gate design. No separate historical tuning split was documented. We therefore do not claim that the original nine-task evaluation was strictly held out.

To provide additional evidence outside these development tasks, we evaluated the frozen rule on TREC-6. We fixed the scoring rule, hyperparameters, target-model snapshots, and three APM seeds before target inference. We used all 500 official test queries, removed 11 text-overlapping training records, and performed no hyperparameter tuning or checkpoint selection.

TargetLabel-MassMiM_iFPA
Qwen2.5-7B95.8095.8795.87
Mistral-7B93.4093.5393.47

The selectors are nearly tied on this task. This provides additional evaluation outside the disclosed development tasks, but does not retrospectively make the original evaluation strictly held out.

3. Real posterior variation and theoretical scope

Theorem 3 fixes query posteriors, observed-label masses, and other scoring inputs while varying candidate tails. It is a conditional robustness guarantee, not a probability model of retraining outcomes.

We compared three clean APM seeds across six directed seed pairs on five tasks. We distinguish:

  • Joint seed variation: actual predictions from another APM seed, potentially changing query posteriors, label masses, and tails.
  • Tail transfer: importing another seed's normalized candidate-tail directions while retaining the base query posteriors, label masses, and other inputs. These are controlled hybrid posteriors, not literal outputs of a newly trained classifier.
TaskJoint change (%)Tail-transfer change (%)
AG News10.920.79
MNLI34.414.74
MASSIVE48.3119.64
Banking7783.2527.17
SST-583.8843.73
Equal-task mean52.1519.21

No certified tail-transfer comparison changed membership (0/117,958). The six directed pairs reuse the same seeds and queries, so these rates are descriptive rather than independent statistical observations. Actual joint seed changes can alter certified membership because they change inputs that Theorem 3 holds fixed.

4. Membership versus ordering invariance

We agree that Theorem 3 guarantees membership, not ordering. The tail-transfer experiment produced 28,992 order-only changes, including 4,411 certified comparisons. This is consistent with the theorem: selected membership can remain unchanged while ranking selected candidates by variable FPA scores changes the prompt.

Accordingly, our controlled membership comparisons use a common fixed semantic order, ensuring identical prompt tokens when selected sets agree. Original Suite A comparisons retain selector-score ordering and therefore include both membership and ordering effects. We do not attribute their full accuracy differences exclusively to membership selection.

Our contribution is an exact characterization of when posterior-tail information affects selected-set membership, not a guarantee of invariant prompt ordering or universally improved prediction accuracy.

Reviewer_mQ3vOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors analyze the impact of the posterior tail, which is the probability distribution over class labels other than the correct label, on the final demonstration selection results, and mathematically derive a threshold at which the posterior tail can be ignored. The proposed FPA scoring method achieves higher performance than other baselines, such as D.Va and ConE, as well as Label-Mass, a comparison method proposed by the authors.

Critical strengths

This paper analyzes the impact of the posterior tail on demonstration selection and makes a theoretical contribution.

Critical weaknesses

Minor 1. In Lines 39–41 of the Introduction, the authors state that they investigate how the components of the posterior affect individual demonstration selection decisions. However, the subsequent discussion of FPA, APM, cross-fitting, anchoring, JS divergence, and other concepts appears to be a list of methodological components and theoretical details, making the content difficult to understand and obscuring the main message the authors intend to convey.

Minor 2. The Method section is overall too descriptive and list-like. What exactly are the authors trying to achieve? Why is the scoring function (S_i(v)) defined? While presenting mathematical derivations is valuable, the authors should explain the motivation and purpose behind each step as the equations are developed. Overall, the readability of this section is poor.

Major 1. In FPA Scoring, I understand that three terms—(1) Semantic Relevance, (2) Label Evidence, and (3) Posterior Alignment—are incorporated into a single formulation. However, it is unclear why the second term, Label Evidence, is separately included in addition to the third term, Posterior Alignment. Furthermore, the justification for defining the coefficient (c), which determines the relative contributions of the two terms, using entropy-based weighting is insufficient. The authors should explain why the current three-term formulation is necessary, rather than a two-term formulation using only Semantic Relevance and Posterior Alignment, which are central to the paper, or validate its necessity through an ablation study.

Major 2. Furthermore, this three-term formulation makes it difficult to clearly evaluate the independent contribution of posterior-tail information in the comparison with the authors' proposed Label-Mass baseline. Specifically, entropy-based weighting is applied to the second term of FPA, whereas such weighting is not applied to the Label-Mass baseline. Therefore, it is difficult to directly attribute the performance difference between the two methods to the contribution of Posterior Alignment, particularly the posterior tail.

Major 3. This paper analyzes the invariance of demonstration selection under changes in the posterior tail and, based on this analysis, presents conditions under which the same set of demonstrations can be selected without accessing the posterior tail. However, Section 3.1 explains that candidate posteriors are computed and cached offline, making it unclear what practical computational cost benefits can be obtained by skipping access to the posterior tail through an additional certification process. Therefore, experiments comparing the inference latency and computational cost of conventional FPA scoring and the certification-based approach are needed.

Major 4. The performance difference between Label-Mass and FPA varies depending on the target model, and the magnitude of improvement is not large. Therefore, it is difficult to conclude that Posterior Alignment leads to a significant performance improvement. This diminishes the significance of the analysis of demonstration selection invariance with respect to the posterior tail. It would be helpful to validate the results on other, more recent models.

Major 5. The composition of the evaluation datasets used in Table 7 is unclear. Which datasets were used? Also, what is the performance when evaluated using the full query set? The evaluation sets in Appendix C are divided into four categories, but what exactly is the Target-aware panel corresponding to the last category?

Questions for authors

The questions are summarized below. Please refer to the Critical Weaknesses section for detailed explanations.

Major 1–2: Why is the three-term FPA formulation necessary, and can the independent contribution of posterior-tail information be validated through ablation studies?

Major 3: Does the certification framework provide measurable computational benefits over standard FPA scoring, considering that candidate posteriors are already cached?

Major 4: Does Posterior Alignment consistently improve performance across different LLMs, including more recent models?

Major 5: What are the exact datasets and query sampling procedures used for the Target-aware panel in Table 7, and what are the results on the full query set?

Please refer to the corresponding weaknesses above for further details.

2 comments

Thank you for pointing out that the original presentation did not sufficiently connect the scoring components to the central research question. Our central question is: which posterior information is actually needed to determine the selected demonstrations? We have clarified the component roles and added controlled ablations, newer-model evaluations, and further dataset details.

1. Why are three scoring components included?

The three components serve distinct roles:

  • Semantic relevance measures input-level compatibility.
  • Label evidence measures the query probability assigned to the candidate's observed memory label.
  • Posterior alignment compares the full query and candidate distributions, including competing-label structure.

Label evidence provides direct information about the observed label, whereas JS alignment captures distributional compatibility beyond that coordinate. Cross-fitting estimates candidate posteriors out of fold, and anchoring incorporates the known memory label.

The entropy-based coefficient c=1−H(qˉ)/log⁡2Cc=1-H(\bar q)/\log_2 C increases the contribution of direct label evidence when the query posterior is concentrated. It is a design choice, not a theoretically optimal weighting. Importantly, our exact interval and invariance results hold for any fixed c∈[0,1]c\in[0,1].

We added controls using c=0c=0 and constant c=0.5c=0.5, retaining anchoring. On the five-task supplementary panel (512 queries per task, three APM seeds, K=30K=30, L=8L=8, and λ=0.5\lambda=0.5), we obtained:

TargetFPA (%)c=0c=0 (%)c=0.5c=0.5 (%)
Qwen2.5-7B81.2181.4681.46
Mistral-7B82.2982.5482.84

These results do not establish that three terms or entropy weighting are necessary. We therefore present entropy weighting as a design choice rather than a theoretical requirement.

2. Isolating posterior-tail contributions and evaluating newer models

The submitted +0.83 pp Mistral result compares FPA with Label-Mass and does not isolate tail information. We therefore directly compared FPA with MiM_i, where Si=Mi−κGiS_i=M_i-\kappa G_i. This retains the entropy mixture and coarse Bernoulli JS term while removing only the candidate-tail residual.

TargetFPA minus MiM_i (pp), 95% CI
Qwen2.5-7B-0.04 [-0.23, +0.16]
Mistral-7B+0.09 [-0.21, +0.39]
Qwen3-4B+0.01 [-0.22, +0.25]
Phi-4-mini-0.03 [-0.30, +0.25]

Each target was evaluated on 7,680 query-seed cases. The newer models used label-constrained greedy decoding, whereas the original models used unconstrained greedy decoding. These results do not establish a consistent accuracy gain from candidate-tail information. However, membership differs in 21.50% of cases in the controlled comparison, showing that tail information can affect selection even when average prediction accuracy remains similar.

3. Computational benefits

We measured direct scoring against certification plus fallback on the same 7,680 cases:

OrderingDirect (ms/query)Certificate + fallback (ms/query)
Semantic0.0680.161
FPA score0.0650.172

The measurements include endpoint computation, sorting, certificate checking, and fallback, using cached posteriors, NumPy, one BLAS thread, and seven repetitions on an AMD EPYC 7453 CPU. The certificate path is slower at K=30K=30, and full posteriors remain cached. We therefore do not claim runtime acceleration or memory savings. Its contribution is identifying exactly when candidate-tail information is unnecessary for membership recovery.

4. Table 7: datasets and full-query evaluation

The Target-aware panel compares FPA with TopK+ConE and D.Va, which use target-LLM likelihood information. It comprises SST-5, MNLI, Banking77, SemEval-D, and FEVER-NLI. Each task contributes 512 queries sampled without replacement through proportional label-stratified sampling (seed 2701), with identical query IDs across methods.

SemEval-D uses a 10-class label space with relation direction removed; FEVER-NLI uses supplied evidence and excludes evidence retrieval.

We additionally evaluated FPA on the complete query sets:

TaskQueriesQwen (%)Mistral (%)
SST-52,21060.9459.43
MNLI19,64777.8581.88
Banking773,08088.0489.66
SemEval-D2,71777.7373.33
FEVER-NLI19,54068.4966.76
Macro47,19474.6174.21

The corresponding 512-query panel means are 75.23% for Qwen and 74.51% for Mistral. Full-query ConE and D.Va results are unavailable; therefore, comparative gains in Table 7 apply only to the sampled panels and are not extrapolated to the complete datasets.

Overall, these additions distinguish the design of the complete FPA score, the independent effect of candidate-tail information, and the theoretical conditions under which that information can be omitted without changing selected-set membership.

In the response to Major 1, I see very little difference in performance across the tested values of the entropy-based coefficient cc. I therefore view this as a flaw in the design of the method rather than simply a design choice.

I interpret the experimental results in the response to Major 2 as showing that, when the effect of tail information is isolated, performance remains similar even though the set of selected demonstrations changes. This suggests to me that tail information does not help select more useful demonstrations, which leaves me questioning the need for this research.

In Major 4, I asked the authors to evaluate FPA on more recent models. The authors responded with an ablation study, but they did not report the absolute accuracy of FPA on either Qwen3-4B or Phi-4-mini.

Overall, my concerns remain unresolved, and the responses have raised further questions for me. I will therefore lower my rating to 1.

Reviewer_ntxxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when candidate posterior information affects demonstration selection in in-context learning. The proposed FPA selector combines semantic similarity, query label probabilities, and Jensen–Shannon alignment between query and candidate posteriors. The analysis fixes the query posteriors and each candidate’s probability mass on its observed label, while allowing the remaining probability mass to be redistributed independently across competing labels. The authors derive exact attainable score intervals and a necessary and sufficient condition for the selected Top-L set to remain unchanged. When this condition fails, they construct a change in selection by modifying at most two candidate tails. The paper includes synthetic numerical checks and an audit on five classification tasks, reporting an average certification rate of 32.71%, with substantial variation across tasks. In a separate nine-task comparison, FPA improves macro accuracy over LABEL-MASS by 0.83 percentage points on Mistral-7B, while the two methods perform similarly on Qwen2.5-7B.

Critical strengths

The strongest part of the paper is its precise account of when candidate-tail probabilities affect selection. Theorem 2 gives the full attainable range of the tail contribution and identifies the distributions that reach both endpoints. This leads to a necessary and sufficient condition for the Top-L set to stay unchanged, along with a concrete counterexample when the condition fails. The authors also handle ties and degenerate cases carefully. For me, the value of these results is that they spell out what information the selector actually needs: under the stated assumptions, we can tell when candidate-tail probabilities can be left out without changing the selected set. This is a useful theoretical contribution, although more evidence is needed to establish its practical significance.

Critical weaknesses
  1. I am not sure how much of the reported accuracy gain comes from the candidate tails. The comparison with LABEL-MASS changes several parts of the scoring rule at once. On SST-2, for example, FPA improves accuracy by 0.98 points, although the tail residual is zero in binary classification. The full scoring rule can therefore help even when there is no tail allocation to analyze. To understand what the tails contribute, I would like to see Si0S_i^0 compared directly with MiM_i, keeping the demonstration order fixed. How often do they select different sets, and does that difference help or hurt accuracy? This seems to be the missing experiment for connecting the paper’s theoretical question to its prediction results.

  2. The benefit of the exact certificate is also hard to assess from the current experiments. The reported coverage should be compared with coverage under the generic entropy bound; otherwise, we cannot tell how much the sharper bound adds on real data. I would also like to understand why coverage is 89.64% on AG News but only 0.05% on Banking77. Showing the interval widths alongside the selection margins would help explain this gap. The proposed execution path needs a timing comparison as well. Section 5.3 says that the experiments use the original reranker, so they do not establish whether checking the certificate is worthwhile when candidate posteriors are already cached. The theoretical guarantee is clear, but its usefulness in the evaluated setting is still not well established.

Questions for authors
  1. With demonstration order held fixed, how do Si0S_i^0 and MiM_i compare in selected-set agreement and downstream accuracy? This would help isolate the contribution of the tail residual. Please also clarify the ordering used in Suite A.

  2. On the same five-task audit, how much additional certification coverage does the sharp bound provide over the generic entropy bound? A per-task comparison would help assess the empirical value of the tighter result.

  3. Does the certificate-based selection path save runtime compared with directly computing Si0S_i^0 from cached posteriors? Please include the cost of checking the certificate and falling back to full scoring, using the same demonstration ordering.

3 comments

Thank you for identifying the missing empirical connections between our theoretical characterization and the selection procedure. We conducted additional experiments to isolate candidate-tail effects, quantify the sharp bound's contribution, and measure certification with cached posteriors.

1. Comparing FPA with MiM_i under fixed demonstration ordering

The submitted +0.83 pp Mistral result compares complete FPA with Label-Mass and does not isolate candidate-tail information. From Si=Mi−κGiS_i=M_i-\kappa G_i, where κ=(1−λ)(1−c)\kappa=(1-\lambda)(1-c), MiM_i retains the Bernoulli JS term and entropy mixture, while FPA additionally uses the candidate-tail residual GiG_i. In binary classification, Gi=0G_i=0; thus, improvements over Label-Mass on binary tasks cannot be attributed to tail allocation.

We directly compared FPA and MiM_i on five tasks, using 512 queries per task, three APM seeds, K=30K=30, L=8L=8, and λ=0.5\lambda=0.5. Both methods use the same fixed semantic ordering, so equal selected sets produce identical prompts. This supplementary panel differs from Suite A, which uses selector-score ordering.

TargetFPA (%)MiM_i (%)Difference (pp), 95% CI
Qwen2.5-7B81.2181.25-0.04 [-0.23, +0.16]
Mistral-7B82.2982.20+0.09 [-0.21, +0.39]

Membership agrees in 6,029/7,680 cases (78.50%) and differs in 1,651 (21.50%). Among changed-membership cases, FPA corrects 29 errors made by MiM_i but loses 32 correct predictions on Qwen; the corresponding Mistral counts are 63 and 56. Neither paired accuracy contrast establishes a mean improvement. These results show that tails affect selection, but their downstream prediction effects can be positive or negative.

2. Sharp versus generic certification bounds

We compared the sharp bound with the generic entropy bound Gi≤wiH2(ηi)G_i\leq w_iH_2(\eta_i) on the same 100,455 query-seed rows.

TaskGeneric (%)Sharp (%)
AG News89.4089.64
MNLI63.5368.04
MASSIVE4.454.61
SST-51.131.21
Banking770.050.05

The sharp bound retains all generic certifications and adds 2,596 cases. Equal-task/equal-seed coverage rises from 31.71% to 32.71%; query-weighted coverage rises from 56.13% to 58.71%. Importantly, failure of the sharp test admits an attainable membership-changing allocation, whereas failure of the generic bound may simply reflect bound looseness.

3. Explaining the AG News–Banking77 coverage gap

We added interval/margin diagnostics. For each query, ww is the mean of its 30 interval widths, gg is the observed Top-8/Top-9 score gap, and h=min⁡i∈Aℓi−max⁡j∉Aujh=\min_{i\in A}\ell_i-\max_{j\notin A}u_j is the robust boundary margin.

TaskMedian interval widthMedian observed gapMedian robust margin
AG News0.00002030.002190.00179
Banking770.111880.00240-0.11614

The observed selection gaps are similar, whereas Banking77 has substantially wider attainable intervals. Its low coverage therefore reflects greater possible score variation relative to the selection boundary, rather than merely a smaller observed margin.

4. Runtime with cached posteriors

We benchmarked direct scoring against certification plus fallback on the same 7,680 cases. The latter includes endpoint computation, sorting, certificate checking, and fallback when necessary. Measurements used warm cached posteriors, NumPy, one BLAS thread, and seven repetitions on an AMD EPYC 7453 CPU.

OrderingDirect (ms/query)Certificate + fallback (ms/query)
Semantic0.0680.161
FPA score0.0650.172

The certificate path is slower at K=30K=30. With FPA ordering, certified cases still require full scores to preserve ordering. Both paths reproduced the same membership and required ordering. These selection-only measurements exclude retrieval, APM inference, and target generation; full posteriors remain cached.

We therefore do not claim measured runtime or memory savings. The certificate's contribution is an exact, query-specific determination of whether candidate-tail information is necessary for recovering FPA membership.

Thanks for running the additional experiments. The fixed-order comparison addresses my main question, though it does not show a clear accuracy gain from the tail term. Certification is also slower than direct scoring with cached posteriors. I appreciate the theoretical contribution, but I still have reservations about its practical significance. I will keep my rating at 2.

The practical use of our analysis is to determine when a simpler selector can reproduce the demonstrations supplied by FPA. In the fixed-order experiment, FPA and MiM_i select different sets in 1,651 of 7,680 query-seed cases (21.50%). Similar average accuracy therefore does not establish that candidate-tail information is irrelevant to selection.

Our certificate establishes more than agreement at the observed posteriors. With query posteriors, observed-label masses and all other scoring inputs fixed, it gives a necessary and sufficient condition for the selected set to remain unchanged under every admissible independent tail reallocation. For certified queries, MiM_i recovers the same demonstrations, and the common semantic ordering preserves the prompt. When the condition fails, our construction gives an admissible perturbation involving at most two candidate tails that changes the selected set.

This distinction affects the conclusions we can draw on real data. In the five-task audit, the sharp bound certifies 2,596 additional cases among 100,455 query-seed cases while retaining every certification from the generic bound. A failed generic test may reflect looseness in the bound; failure of the sharp condition establishes that a selection change is possible within the stated uncertainty set. These results provide a way to audit which parts of a candidate posterior actually determine an ICL selector's decisions.

Reviewer_zfGMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates when candidate posterior distributions affect in-context demonstration selection. The authors propose Full Posterior Alignment (FPA), combining semantic relevance, query-label probabilities, and Jensen–Shannon (JS) alignment. They derive exact score bounds under candidate-tail variation and establish a necessary and sufficient condition for Top-L selection invariance, along with constructive counterexamples and rank bounds.

Critical strengths
  • The paper derives exact JS divergence bounds and a necessary and sufficient condition for Top-L selection invariance, together with constructive counterexamples. These results precisely characterize when candidate-tail information affects demonstration selection.
  • The experiments verify the theoretical results across multiple settings and evaluate FPA on diverse classification tasks and two LLMs. The analysis also reveals substantial task-dependent differences in certification coverage.
Critical weaknesses
  • The analysis assumes arbitrary redistribution of candidate-tail probabilities while keeping other posterior components fixed. Although mathematically valid, such perturbations may not reflect realistic changes in classifier predictions, limiting the practical relevance of the theoretical guarantees.
  • The certificate guarantees selection membership but not demonstration ordering, which may still require full posterior information. Moreover, no latency or memory measurements are provided to demonstrate actual efficiency gains, leaving the practical value of the certificate unclear.
  • FPA provides limited, model-dependent improvements over Label-Mass, with negligible gains on Qwen2.5-7B.
Questions for authors
  • How much does the proposed sharp JS bound improve certification coverage compared with the generic bound?
  • Could you compare FPA with the tail-independent score (M_i) to isolate the contribution of candidate-tail information?
  • How do different posterior anchoring strengths affect certification coverage and classification accuracy, and were the datasets used for hyperparameter development strictly separated from evaluation?
2 comments

Thank you for recognizing the exact bounds and invariance characterization. We conducted additional experiments to clarify the practical scope of our uncertainty set, isolate the contribution of candidate-tail information, and assess certification.

1. Realistic posterior variation and the scope of the guarantee

Our uncertainty set fixes query posteriors, candidate observed-label masses, semantic scores, and other scoring inputs, while allowing candidate-tail allocations to vary independently. This is a conditional robustness analysis, not a probabilistic model of retraining outcomes. Certification guarantees invariance over every admissible tail allocation. Failure yields an attainable membership-changing witness involving at most two candidate tails, but does not imply that retraining will produce that perturbation.

We additionally evaluated actual APM retraining. Certified/uncertified set-change rates were 67.52%/88.80% under undertraining and 21.13%/81.92% under 20% APM training-label noise. These interventions also change query posteriors and observed-label masses, so they lie outside Theorem 3's fixed-input guarantee.

To examine tail variation more directly, we compared three clean APM seeds on five tasks. Across six directed seed pairs, actual joint seed changes produced a 52.15% equal-task mean membership-change rate. A controlled tail-transfer experiment, importing another seed's normalized candidate-tail directions while retaining the base query posteriors and observed-label masses, produced a 19.21% rate. No certified tail-transfer comparison changed membership (0/117,958). These are controlled hybrid posteriors, not literal outputs of a newly trained classifier.

2. Contribution of the sharp bound

We compared the sharp bound with the generic bound Gi≤wiH2(ηi)G_i\leq w_iH_2(\eta_i) on the same 100,455 query-seed rows.

TaskGeneric (%)Sharp (%)
AG News89.4089.64
MNLI63.5368.04
MASSIVE4.454.61
SST-51.131.21
Banking770.050.05

The sharp bound retains all generic certifications and adds 2,596 cases. Equal-task/equal-seed coverage increases from 31.71% to 32.71%, and query-weighted coverage from 56.13% to 58.71%. Beyond the coverage gain, the sharp bound gives failure a precise interpretation: a failed sharp certificate admits a membership-changing allocation, whereas failure under the generic bound may only reflect bound looseness.

3. Isolating candidate-tail information

Since Si=Mi−κGiS_i=M_i-\kappa G_i, comparing FPA with MiM_i isolates the candidate-tail residual while retaining the Bernoulli JS term and entropy mixture. This is more informative than comparing FPA with Label-Mass, which changes several scoring components.

On five tasks (512 queries per task, three APM seeds, K=30K=30, L=8L=8, and λ=0.5\lambda=0.5), we obtained:

TargetFPA (%)MiM_i (%)Difference (pp), 95% CI
Qwen2.5-7B81.2181.25-0.04 [-0.23, +0.16]
Mistral-7B82.2982.20+0.09 [-0.21, +0.39]

Membership differs in 21.50% of the 7,680 query-seed cases. Neither accuracy contrast establishes a mean improvement. Thus, tail information can affect selection, but its downstream prediction effect is not consistently positive.

4. Anchoring and development/evaluation separation

We evaluated different anchoring strengths on the same supplementary panel. Constant anchoring uses p~i=(1−γ)pi+γeyi\widetilde p_i=(1-\gamma)p_i+\gamma e_{y_i}.

AnchorCoverage (%)Qwen (%)Mistral (%)
Adaptive33.3181.2182.29
γ=0.25\gamma=0.2533.0581.9383.09
γ=0.5\gamma=0.533.8481.6782.90
γ=0.75\gamma=0.7535.5981.5882.89
γ=1\gamma=1100.0081.3882.89

At γ=1\gamma=1, tail uncertainty vanishes and coverage reaches 100%. Coverage and predictive accuracy are therefore distinct objectives. These sensitivity results were not used to select a new parameter.

We also clarify that CR, SST-2, and Subj informed historical method development and appear in Suite A. No separate historical tuning split is documented; we therefore do not claim that the original evaluation was strictly held out.

5. Ordering and computational cost

Theorem 3 guarantees membership, not ordering. A fixed semantic order preserves identical prompts when membership is unchanged; FPA-score ordering may still change.

At K=30K=30, CPU selection-only timings were 0.068 ms/query for direct scoring and 0.161 ms/query for certification plus fallback under semantic ordering (0.065 vs. 0.172 ms under FPA ordering). Full posteriors remain cached, and we do not claim measured speed or memory savings.

Our contribution is an exact, query-specific characterization of when candidate-tail information can be omitted without changing FPA membership, rather than a claim of universal accuracy or runtime improvement.

I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.