CROSS: Conservative Routing over Complementary Demonstration Selectors for In-Context Learning

In-context learning (ICL) enables large language models (LLMs) to adapt to novel tasks from a small number of demonstrations, but its performance depends heavily on which demonstrations are selected. Although existing approaches often produce query-specific rankings or subsets, they typically apply a fixed selection strategy, with a predetermined bias toward relevance, diversity, or their trade-off, across different queries. This formulation overlooks query-level heterogeneity: some queries may benefit more from tightly localized relevance, whereas others may benefit from broader diversity at the demonstration-set level. To address this challenge, we propose a novel framework named Conservative Routing over Complementary Demonstration Selectors (CROSS) for demonstration selection in in-context learning. At its core, CROSS treats the prediction pathway associated with each demonstration selector as an expert and learns from calibration data which expert is more reliable for a given query. Specifically, one branch of CROSS learns a label-free projection into a Poincaré ball that preserves local neighborhood structures defined in the target LLM's representation space, and then retrieves demonstrations based on hyperbolic distance. The other branch employs a determinantal point process (DPP) to select a demonstration set that balances query-aware relevance with set-level diversity. Building on these complementary branches, CROSS trains a lightweight, prediction-aware router using calibration instances where the two branches disagree and exactly one prediction is correct, thereby learning which selector is more reliable for a given query. Additionally, we conduct extensive experiments across multiple benchmarks and LLM backbones, where CROSS consistently outperforms diverse demonstration-selection baselines.