- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks when the posterior geometry of candidate demonstrations actually matters for selecting demonstrations in in-context learning based on posterior alignment. It focuses on a score called Full Posterior Alignment (FPA), and breaks the Jensen–Shannon divergence between the query posterior and a candidate posterior into two parts: how much probability mass goes to the observed label, and how the remaining probability is spread over competing labels—the tail geometry. The authors derive the exact range that the tail part can take when the observed-label probability is fixed, and they characterize when the Top-L selected set stays the same no matter how that tail is reallocated. They also give rank bounds and build counterexamples. In addition, they introduce a certification procedure that tells when candidate-tail information is not needed for selection. Finally, they test FPA and related selectors on several classification benchmarks and language models.
Critical strengths
-
The paper provides a coherent analysis of posterior-tail geometry under fixed observed-label mass, including an exact attainable interval for the JS contribution and a necessary-and-sufficient condition for Top-L membership invariance. The constructive counterexamples and rank bounds further strengthen the theoretical characterization, making the result more than a qualitative observation.
-
The authors conduct extensive numerical checks of the theoretical claims and evaluate the proposed FPA selector across multiple tasks and models. In particular, the results show that FPA can provide measurable gains over the label-mass baseline and that the proposed certificate can identify a substantial fraction of cases where candidate-tail information is unnecessary.
Critical weaknesses
-
FPA differs from the Label-Mass baseline in several components beyond the competing-label tail, including observed-label probability, Bernoulli-level JS divergence, entropy-based weighting, and candidate-specific query posteriors. Therefore, the reported accuracy improvements do not establish that posterior-tail geometry itself is responsible for the gains. This is consequential because demonstrating the practical importance of posterior geometry is the central motivation of the paper.
-
The proposed uncertainty set allows arbitrary reallocations of the candidate posterior tail while fixing only the observed-label mass, which may be substantially broader than realistic posterior variations. Moreover, certification coverage varies dramatically across tasks (from 89.64% on AG News to only 0.05% on Banking77). Thus, the theoretical non-invariance results establish worst-case sensitivity, but the paper provides limited evidence that such tail variations occur in practice or materially affect demonstration selection.
Questions for authors
-
Eq. (11) and (12) show that FPA and the Label Mass baseline differ in more than just the competing label tail geometry. They also differ in the observed label mass term, the Bernoulli level JS divergence, and the entropy based weighting c. Could the authors check whether the gains in Table 1 still hold when they compare Label Mass to the coarse score M_i, and then to full FPA? This would isolate how much the tail geometry term G_i actually contributes.
-
Appendix C.2 says the scoring mixture and anchor were "developed" on CR, SST-2, and Subj. But these three tasks are also in Suite A in Table 1. What does "developed" mean here exactly? Did the authors use these three tasks to choose hyperparameters, design choices, or models? If so, how do the FPA results change under a strict held out evaluation setup?
-
Eq. (10) lets the candidate posterior tail be reallocated in any way as long as b_i is fixed. So Theorem 3 gives worst case invariance or non invariance under a very broad uncertainty set. Do the authors have evidence that these allowed tail reallocations match realistic posterior variation, for example across APM seeds or other sources of model variation? How often do such realistic perturbations actually change the Top L set?
-
Theorem 3 only guarantees that Top L membership does not change. Appendix A.6 says the order of the selected demonstrations can still change. In the experiments, is the demonstration order fixed after selection, independent of the FPA score? Or can a change in the score change the order shown to the target LLM? If the order can change, how do the authors separate the reported gains from possible order effects instead of from membership selection effects?
Thank you for the careful assessment. We agree that the original comparison with Label-Mass did not isolate posterior-tail geometry. We therefore added controlled scoring comparisons, a new held-out task, and an audit of actual APM seed variation and tail transfers.
1. Isolating posterior-tail information
The submitted +0.83 pp Mistral improvement compares complete FPA with Label-Mass and cannot be attributed solely to posterior tails. From Si=Mi−κGi, where κ=(1−λ)(1−c), Mi retains the observed-label mass, Bernoulli JS term, entropy mixture, and candidate-specific query posteriors. Comparing FPA with Mi therefore isolates the tail residual.
We evaluated these three methods on five tasks, using 512 queries per task, three APM seeds, K=30, L=8, and λ=0.5. All methods use a common fixed semantic demonstration order, so equal selected sets produce identical prompts.
FPA minus Mi is -0.04 pp (95% CI [-0.23, +0.16]) for Qwen and +0.09 pp (95% CI [-0.21, +0.39]) for Mistral. Membership differs in 21.50% of the 7,680 query-seed cases, while all 2,558 certified cases agree. Notably, FPA is 0.77 pp below Label-Mass on Mistral in this supplementary panel. We retain this result rather than attributing the original gain to tail geometry.
2. Development history and held-out evaluation
We clarify that historical development included a Qwen2.5-7B mixture-weight sweep on CR, SST-2, and Subj, and SST-2 pilot results informed the anchor/confidence-gate design. No separate historical tuning split was documented. We therefore do not claim that the original nine-task evaluation was strictly held out.
To provide additional evidence outside these development tasks, we evaluated the frozen rule on TREC-6. We fixed the scoring rule, hyperparameters, target-model snapshots, and three APM seeds before target inference. We used all 500 official test queries, removed 11 text-overlapping training records, and performed no hyperparameter tuning or checkpoint selection.
The selectors are nearly tied on this task. This provides additional evaluation outside the disclosed development tasks, but does not retrospectively make the original evaluation strictly held out.
3. Real posterior variation and theoretical scope
Theorem 3 fixes query posteriors, observed-label masses, and other scoring inputs while varying candidate tails. It is a conditional robustness guarantee, not a probability model of retraining outcomes.
We compared three clean APM seeds across six directed seed pairs on five tasks. We distinguish:
No certified tail-transfer comparison changed membership (0/117,958). The six directed pairs reuse the same seeds and queries, so these rates are descriptive rather than independent statistical observations. Actual joint seed changes can alter certified membership because they change inputs that Theorem 3 holds fixed.
4. Membership versus ordering invariance
We agree that Theorem 3 guarantees membership, not ordering. The tail-transfer experiment produced 28,992 order-only changes, including 4,411 certified comparisons. This is consistent with the theorem: selected membership can remain unchanged while ranking selected candidates by variable FPA scores changes the prompt.
Accordingly, our controlled membership comparisons use a common fixed semantic order, ensuring identical prompt tokens when selected sets agree. Original Suite A comparisons retain selector-score ordering and therefore include both membership and ordering effects. We do not attribute their full accuracy differences exclusively to membership selection.
Our contribution is an exact characterization of when posterior-tail information affects selected-set membership, not a guarantee of invariant prompt ordering or universally improved prediction accuracy.