acceptodds
Under review as a conference paper at ICLR 2027

CROSS: Conservative Routing over Complementary Demonstration Selectors for In-Context Learning

Abstract

In-context learning (ICL) enables large language models (LLMs) to adapt to novel tasks from a small number of demonstrations, but its performance depends heavily on which demonstrations are selected. Although existing approaches often produce query-specific rankings or subsets, they typically apply a fixed selection strategy, with a predetermined bias toward relevance, diversity, or their trade-off, across different queries. This formulation overlooks query-level heterogeneity: some queries may benefit more from tightly localized relevance, whereas others may benefit from broader diversity at the demonstration-set level. To address this challenge, we propose a novel framework named Conservative Routing over Complementary Demonstration Selectors (CROSS) for demonstration selection in in-context learning. At its core, CROSS treats the prediction pathway associated with each demonstration selector as an expert and learns from calibration data which expert is more reliable for a given query. Specifically, one branch of CROSS learns a label-free projection into a Poincaré ball that preserves local neighborhood structures defined in the target LLM's representation space, and then retrieves demonstrations based on hyperbolic distance. The other branch employs a determinantal point process (DPP) to select a demonstration set that balances query-aware relevance with set-level diversity. Building on these complementary branches, CROSS trains a lightweight, prediction-aware router using calibration instances where the two branches disagree and exactly one prediction is correct, thereby learning which selector is more reliable for a given query. Additionally, we conduct extensive experiments across multiple benchmarks and LLM backbones, where CROSS consistently outperforms diverse demonstration-selection baselines.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
in context learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cRqiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CROSS, which is a routing framework for in-context learning that combines a target-aligned hyperbolic nearest-neighbor selector with a diversity-aware determinantal point process (DPP) selector.
A logistic regression router uses 12 features derived from the query, predictions, and selected demonstrations to decide whether to replace the default DPP prediction with the Hyperbolic prediction.

Critical strengths
  1. Effective use of complementary selectors
  • CROSS combines query-local relevance and set-level diversity in a lightweight framework, achieving consistent improvements over the DPP baseline across multiple datasets and LLM backbones without additional model fine-tuning.
Critical weaknesses
  1. Limited validation of the routing mechanism
  • The router compresses 12 features into a single score and applies a global switching threshold, but the paper does not sufficiently discuss whether the scalar score is sufficient to distinguish the retrievers and whether the global threshold is sufficient for different types of queries.
  • Beyond final accuracy, the evaluation would benefit from held-out oracle-agreement and routing-regret analyses, threshold-sensitivity experiments, and comparisons with more expressive routers.
  1. Outdated models and baselines
  • Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 are somewhat old.
  • Topic Coverage-based Demonstration Retrieval for In-Context Learning, EMNLP'25 also discusses the trade-off between embedding similarity and diversity. The authors need to compare the proposed method with this paper in a theoretical and empirical manner.
Questions for authors
  1. Is a scalar score enough to route the retriever?
  2. Is a global threshold enough for all queries?
  3. Why not use an ensemble of the retrievers?
  4. Does the paper evaluate the calibration of the router by comparing its output scores with oracle routing decisions?
Reviewer_YoBQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CROSS is a framework for in-context demonstration selection that combines a target-model-aligned hyperbolic retriever with a diversity-aware DPP selector. Both branches generate predictions, and a lightweight router uses a calibrated threshold to selectively override the default DPP prediction. Experiments on five intent classification datasets and three LLM backbones show improved average accuracy, supported by ablations and analyses of routing behavior and computational cost.

Critical strengths
  1. Simple and practical method. CROSS combines two demonstration selectors with a lightweight logistic router while keeping the target LLM frozen. This straightforward design makes it easy to implement and integrate into existing ICL pipelines.

  2. Thorough validation of the routing design. Ablations examine the calibrated threshold and routing features, while rescued-versus-damaged analyses directly assess beneficial and harmful switches. Together, these experiments substantiate the routing mechanism’s effectiveness.

Critical weaknesses
  1. Limited task coverage. Experiments cover only intent classification, leaving the method’s applicability to other ICL tasks unverified.

  2. Unclear necessity of key components. The method introduces both target-model representations and hyperbolic projection, but their individual contributions and necessity are not sufficiently isolated.

  3. Additional computational cost. Although performance improves, the method requires two LLM predictions per query, increasing inference cost. The accuracy–cost trade-off needs stronger justification.

Questions for authors

see weaknesses please.

Reviewer_crNQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CROSS combines hyperbolic retrieval and DPP-based demonstration selection through a calibrated logistic router. Experiments on five intent classification datasets and three language models show improved average accuracy. The main concern is whether the proposed components provide benefits beyond simpler combinations with comparable inference and calibration resources.

Critical strengths
  1. The paper shows that a weaker selector can complement a stronger default through query-dependent routing, with improvements across three backbones.

  2. The evaluation includes several competitive baselines. The rescued/damaged prediction analysis and runtime measurements help explain the practical trade-offs.

Critical weaknesses
  1. Insufficiently controlled ablations: The ablations mainly examine the router, leaving the contribution of hyperbolic geometry confounded with encoder and retrieval differences. Matched Euclidean controls and simpler routed selector pairs are missing, while the threshold ablation only compares calibration against a fixed value of 0.9.

  2. The comparisons do not control for additional resources. CROSS uses two predictions and labeled calibration, but lacks comparable two-call baselines, making it difficult to separate the proposed design’s contribution from extra inference and supervision.

  3. All evaluated backbones were released in 2024. Testing a suitable 2026 model available before submission would strengthen the evidence that CROSS remains effective on newer model generations.

  4. All three models have only 7B–8B parameters. Evaluation on a substantially larger backbone, such as a 30B-class model, would clarify whether the routing gains persist across model scales.

Questions for authors

Q1: Can you provide results on a newer backbone and a substantially larger model, reporting both accuracy gains and inference costs?

Q2: Please reconcile the BANKING77 accuracy of 84.47 in Table 1 with 84.42 in Table 10, and the conflicting dataset attribution for the shopping-list example in Figure 5 and Appendix D.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.