acceptodds
Under review as a conference paper at ICLR 2027

SG-XICL: Coordinating Evidence and Reasoning Scopes in Large-Label In-Context Learning

Abstract

In-context learning (ICL) enables large language models (LLMs) to perform new tasks from a small set of demonstrations without task-specific parameter updates. With a large label space, however, a separate design question is which labels should receive dedicated demonstration evidence and explicit reasoning for a given query. We propose Subspace-Guided X-ICL (SG-XICL), which constructs a query-conditioned candidate label subspace to coordinate label-aligned demonstration evidence and candidate-wise reasoning while retaining the full task label space for final prediction. Across six language models and eight classification datasets, SG-XICL achieves higher average Accuracy than the four evaluated ICL baselines on their supported comparisons. Compared with full-label X²-ICL, SG-XICL keeps dedicated reasoning local and incurs substantially lower online token burden over common-supported conditions. These results support separating the scope of dedicated explicit processing from the full prediction space in large-label ICL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, large-label-space classification, large language models, demonstration selection, natural language explanations, label-wise reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pKPzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes in-context learning with large label spaces. The proposed method first builds a candidate set, specific to the query, by scoring each label with cosine similarity between the query and nearest references for the corresponding label. Then, it selects the nearest reference for each candidate label as a demonstration, along with an LLM-generated explanation of why the reference is paired with a particular label. At the end, the method prompts the model to write reasoning sentences for candidates and select a label from the full label space.

The method achieves improvements in accuracy over standard ICL and three other ICL baselines across a broad set of models and datasets. In the analyses, the authors show that their method's references include the gold label more often than retrieval-based ICL and that more references or broader explanations come with a substantially larger number of tokens, while delivering little or no improvement.

Critical strengths
  • The evaluation is broad, covering eight datasets, ranging from 18 to 165 labels across four task types, and six models (including API-based and local ones). The method shows the best averaged performance, compared to other baselines.

  • The idea seems simple and practical with a clear efficiency benefit. Concretely, the authors propose that although the final prediction could be any label from the full label space, reasoning and explicit evidence are tied to a query-specific candidate set. Based on this, the cost of explicit reasoning doesn't need to grow with the number of labels. The rationale for this is that even very detailed reasoning about obviously incorrect labels wastes the budget, and that a better way to use the budget is to focus on a plausible candidate set. The results show that SG-XICL uses about 95% fewer input tokens compared to X^2-ICL.

Critical weaknesses
  • The baselines are quite weak and do not properly isolate the contribution of SG-XICL. Three baselines (ICL, X-ICL, X^2-ICL) use fixed random demonstrations, which seems like a poor choice for large label spaces. Their substantially weaker performance could largely come from lack of retrieval (compared to the proposed method and retrieval-based ICL that use some form of retrieval). The only retrieval-based baseline (retrieval ICL) selects from a pool of only seven references for each label, while SG-XICL selects from the entire pool. Against this baseline, the average gains are much more modest. Moreover, some of the methods that look more closely related are not evaluated (e.g., CLL-RetICL and ICXML), and none of the used baselines combine retrieval with explanations.

  • The main new component of the proposed method is restricting reasoning in the style of X^2-ICL to top candidate labels, which seems very similar to approaches such as ICXML. The other components, namely label-aware retrieval, explanations for labels, and label-wise reasoning, already exist in prior methods. The paper frames separating the local reasoning scope from the full prediction space as its main contribution, but it never shows that keeping the full label space really matters. It seems that the method boils down to a restricted-label approach if correct predictions beyond the candidates are rare, and the proposed design doesn't add much beyond existing work.

Questions for authors
  1. How does SG-XICL compare to the baselines when the settings are matched? For instance, using the same full reference pool, retrieved demonstrations with explanations for the labels, or an embedding-only classifier.

  2. Is the full label space really important? How often is the predicted label outside the candidate set, and how accurate are those predictions?

Reviewer_UM9pOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper designed a method called Subspace-Guided X-ICL (SG-XICL) which coordinates label-aligned demonstration evidence and candidate-wise reasoning within a query-conditioned label subspace while keeping full-label prediction. Experiments across six language models and eight classification benchmarks demonstrate comparable results.

Critical strengths
  1. Interesting Research Problem: Coordinating evidence and reasoning in large-label in-context learning is an important and practically relevant research problem.

  2. Extensive Evaluation: The paper evaluates multiple language models and datasets, with supporting ablation and sensitivity analyses.

Critical weaknesses
  1. Limited Methodological Novelty:

The main ideas of query-dependent candidate-label selection, label-space refinement, and adaptive demonstration retrieval have already been extensively studied in previous literature [1–7]. The proposed method appears to combine several established methods, including label-aware retrieval [3], candidate-label refinement [4–6], explanation augmentation [8], and candidate-wise reasoning [9]. Especially, previous studies have investigated query-adaptive retrieval for large-label classification [1, 2], label-aware demonstration selection [3], candidate-label generation and refinement [4–6], and explanation-guided reasoning [8, 9]. Furthermore, existing work has explored optimizing demonstration composition and coverage [7, 10]. While SG-XICL introduces a unified framework for coordinating these components, the fundamental methodological novelty beyond combining existing techniques remains unclear.

  1. Mismatch Between Baseline Evaluation Settings:

Most importantly, key baseline, X²-ICL [9], originally tested on natural language inference and paraphrase identification datasets, such as SNLI, HANS, NaN, PISP, ST, ANLI R1–R3, QQP, and PAWS. These tasks primarily involve two or three labels and emphasize out-of-distribution robustness. In addition, SG-XICL’s evaluation design is significantly different classification tasks involving 18 to 165 labels. It extended existing baselines to new tasks, however, this also raises question regarding the suitability and implementation fidelity in large-label settings. Therefore, It is vital show the comparison on SNLI, HANS, NaN, PISP, ST, ANLI R1–R3, QQP, and PAWS as well for fair comparision.

  1. Comparative Scope is too narrow:

The empirical comparisons focus primarily on conventional ICL, retrieval-based ICL, and explanation-based approaches. These are insufficient. Therefore, broader comparisons with existing label-aware retrieval, candidate-selection, and adaptive label-space refinement methods would improve the evaluation.

  1. Unfair Retrieval Baseline:

Retrieval ICL searches a restricted pool of seven references per label, whereas SG-XICL uses the full reference pool. Why are the retrieval resources different? For fair comparison it should be the same.

  1. Performance Improvements is not consistent:

SG-XICL demonsrate improvement over Retrieval ICL by +2.42 percentage points on average and underperforms on several model configurations. The authors need to justify inconsistent model-dependent behavior and also provide statistical significance tests as well.

Reference List:

1] In-Context Learning for Text Classification with Many Labels. GenBench 2023.

[2] Demonstrations in In-Context Learning for LLMs with Large Label Space. ICML 2024 Workshop on Long-Context Foundation Models.

[3] CLL-RetICL: Contrastive Linguistic Label Retrieval-Based In-Context Learning for Text Classification via Large Language Models. IJCNLP-AACL 2025.

[4] ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification. Findings of NAACL 2024.

[5] Unlearning Misalignment for Personalized LLM Adaptation via Instance-Response-Dependent Discrepancies. Transactions on Machine Learning Research.

[6] Bottom-Up Competition Based on Label Hierarchy Trees for In-Context Learning with Large Label Space. Information Fusion, 138, 104727.

[7] Compositional Exemplars for In-Context Learning. ICML 2023.

[8] Using Natural Language Explanations to Improve Robustness of In-Context Learning. ACL 2024.

[9] Exploring Explanations Improves the Robustness of In-Context Learning. ACL 2025.

[10] UCS: Estimating Unseen Coverage for Improved In-Context Learning. Findings of ACL 2026.

Questions for authors

I think the authors need to put more effort into conducting a thorough literature review on the topic they are working on.

Firstly, it is vital to include all relevant works and honestly compare them with the proposed methods, highlighting their differences.

Secondly, the author should conduct comparisons on the same datasets as those used in related works to demonstrate genuine improvements and the effectiveness of proposed algorithm.

Thirdly, the authors should ensure that the reported results are consistent and explain any inconsistencie.

Reviewer_ez2LOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SG-XICL, which selects candidate labels for each query by aggregating retrieval scores by label. It organizes demonstrations, explanations, and label-wise reasoning around these candidates while retaining the full label space for final prediction.

Critical strengths
  1. Aligning demonstration selection and reasoning scope through a shared candidate label set is intuitive and has practical value.

  2. The experiments cover six models and eight datasets, with fairly detailed prompt templates, token usage measurements, ablation studies, and failure cases.

Critical weaknesses
  1. The paper primarily targets methods that reason over the full label space. However, Retrieval ICL does not require label-wise reasoning and even outperforms the proposed method on some models.

  2. The paper emphasizes that final predictions are not restricted to the candidate set, but it neither reports accuracy when the gold label falls outside this set nor compares against a variant that restricts predictions to candidate labels. Consequently, the results do not sufficiently establish the contribution of this core design choice.

  3. The main experiments fix m=8m=8, while the sensitivity analyses primarily vary the number of demonstrations per label and the scope of explanations. However, the number of candidate labels directly determines label coverage, reasoning scope, and token usage. Performance and cost should therefore be evaluated across different values of mm.

Questions for authors

Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.