- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes in-context learning with large label spaces. The proposed method first builds a candidate set, specific to the query, by scoring each label with cosine similarity between the query and nearest references for the corresponding label. Then, it selects the nearest reference for each candidate label as a demonstration, along with an LLM-generated explanation of why the reference is paired with a particular label. At the end, the method prompts the model to write reasoning sentences for candidates and select a label from the full label space.
The method achieves improvements in accuracy over standard ICL and three other ICL baselines across a broad set of models and datasets. In the analyses, the authors show that their method's references include the gold label more often than retrieval-based ICL and that more references or broader explanations come with a substantially larger number of tokens, while delivering little or no improvement.
Critical strengths
-
The evaluation is broad, covering eight datasets, ranging from 18 to 165 labels across four task types, and six models (including API-based and local ones). The method shows the best averaged performance, compared to other baselines.
-
The idea seems simple and practical with a clear efficiency benefit. Concretely, the authors propose that although the final prediction could be any label from the full label space, reasoning and explicit evidence are tied to a query-specific candidate set. Based on this, the cost of explicit reasoning doesn't need to grow with the number of labels. The rationale for this is that even very detailed reasoning about obviously incorrect labels wastes the budget, and that a better way to use the budget is to focus on a plausible candidate set. The results show that SG-XICL uses about 95% fewer input tokens compared to X^2-ICL.
Critical weaknesses
-
The baselines are quite weak and do not properly isolate the contribution of SG-XICL. Three baselines (ICL, X-ICL, X^2-ICL) use fixed random demonstrations, which seems like a poor choice for large label spaces. Their substantially weaker performance could largely come from lack of retrieval (compared to the proposed method and retrieval-based ICL that use some form of retrieval). The only retrieval-based baseline (retrieval ICL) selects from a pool of only seven references for each label, while SG-XICL selects from the entire pool. Against this baseline, the average gains are much more modest. Moreover, some of the methods that look more closely related are not evaluated (e.g., CLL-RetICL and ICXML), and none of the used baselines combine retrieval with explanations.
-
The main new component of the proposed method is restricting reasoning in the style of X^2-ICL to top candidate labels, which seems very similar to approaches such as ICXML. The other components, namely label-aware retrieval, explanations for labels, and label-wise reasoning, already exist in prior methods. The paper frames separating the local reasoning scope from the full prediction space as its main contribution, but it never shows that keeping the full label space really matters. It seems that the method boils down to a restricted-label approach if correct predictions beyond the candidates are rare, and the proposed design doesn't add much beyond existing work.
Questions for authors
-
How does SG-XICL compare to the baselines when the settings are matched? For instance, using the same full reference pool, retrieved demonstrations with explanations for the labels, or an embedding-only classifier.
-
Is the full label space really important? How often is the predicted label outside the candidate set, and how accurate are those predictions?