Label Alignment Divergence Reranking for In-Context Demonstration Selection

Semantic retrieval identifies demonstrations that resemble a query, but input similarity need not capture their relationship to the task's label distinctions. We study how auxiliary task supervision can improve demonstration selection for classification in-context learning. We propose Label-Alignment Divergence Reranking, which combines semantic similarity with Jensen–Shannon compatibility between label distributions estimated by an auxiliary predictor. The method reranks a fixed semantic candidate pool without scoring candidates with the target language model. Because candidate labels are observed whereas the query label is unknown, we also introduce a gold-anchored extension that anchors candidate posteriors to their observed labels and balances distribution compatibility with label matching according to query posterior concentration. In a controlled evaluation across nine classification tasks with cross-fitted posteriors and three selection seeds, the proposed method reaches macro accuracies of 85.69% and 87.49% on Qwen and Mistral, compared with 84.81% and 82.56% for semantic TopK. Comparisons with direct label-probability matching show that the incremental benefit of the proposed method varies across target models. Selection diagnostics characterize changes within semantic neighborhoods, while reliability analyses identify sensitivity to posterior quality and domain shift. These findings support auxiliary label information as a useful complement to semantic relevance in classification ICL. The code is released here: https://anonymous.4open.science/r/L2D-401B.