acceptodds
Under review as a conference paper at ICLR 2027

Skill-RAG: Retrieval-augmented LLMs for Extreme Multilabel Skill Classification of Unstructured Job Ad Texts

Abstract

Skill Classification, the process of identifying skill-related information from job ads and mapping it to a fine-grained set of skill labels, opens up new possibilities for social science research on labor market dynamics. Mainstream skill taxonomies are generally large-scale, such as the ESCO framework with 13,939 labels, making this an extreme multi-label classification (XMLC) task. While Large Language Models (LLMs) prompted with job ads and skill ontologies offer robust automated classification capabilities, deploying them on massive, fine-grained taxonomies introduces excessive inputs, thereby triggering issues of context window constraints, long-context performance degradation, and positional bias. To overcome these challenges, we propose Skill-RAG, a novel pipeline that integrates a Retrieval-Augmented Generation (RAG) framework into the inference process for XMLC. Specifically, we systematically explore various knowledge base construction paradigms, query granularity strategies (raw vs. span-based retrieval), and multi-stage retrieval pipelines to optimize retrieval accuracy. Furthermore, for downstream LLM-based classification, we comprehensively evaluate prompting and reasoning strategies across three core dimensions: label space cardinality, label ordering, and thought-guided reasoning. Our empirical results demonstrate that all aforementioned components are critical to final classification quality, and our proposed pipeline substantially outperforms previous state-of-the-art methods on standard benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-label text classification, Skill classification, Large language model, Retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_53bNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Skill-RAG maps job-ad sentences to ESCO skills, an extreme multi-label task over 13,939 labels. The pipeline:

  • a local LLM extracts skill spans;
  • dense retrieval (mxbai-embed-large-v1) finds the top-100 labels from a knowledge base, optionally expanded with LLM-written skill descriptions;
  • a cross-encoder (bge-reranker-v2-m3) reranks to the top-60;
  • an LLM (Gemini-2.5-Flash or DeepSeek-V4.1-Flash) selects the final labels.

Component studies cover knowledge-base variants, span vs raw queries, candidate-pool size, label ordering, and zero-shot/few-shot/CoT prompting with and without native reasoning. The final system is compared with four published systems on HOUSE, TECH and TECHWOLF and claims a new state of the art.

Critical strengths

Clear practical task and large gains over the cited systems without fine-tuning:

  • "+All" improves RP@5 over the best cited baseline by +8.47 (HOUSE), +5.99 (TECH) and +6.97 (TECHWOLF) (Table 3).
  • If the gains hold under a fair comparison, this is a useful system for labour-market research.

A consistent practitioner finding:

  • explicit CoT helps when native reasoning is off and hurts when it is on, in the same direction for both model families (Table 2).
Critical weaknesses

The gains cannot be credited to the pipeline rather than to the newer LLM backbone, and the comparison set is incomplete.

  • The Table 3 LLM baselines use GPT-3.5/GPT-4 or Llama-2, while Skill-RAG uses 2025-era models.
  • The paper does not say which LLM or reasoning mode produced Table 3 (L431–450).

The component studies don't support the design choices well. The pool-size and ordering experiments use candidate pools built to contain every correct label, so they measure ranking within an easy pool, not retrieval from 14,000 labels; the real pipeline's recall is never reported. The pool-size curve appears to peak around 40–50 candidates, yet the system uses 60. No seeds, confidence intervals or significance tests are given, although several effects are called "significant" on test sets of only ~300 sentences. Finally, the released prompts use two HOUSE test sentences as few-shot examples. The effect is small, but it raises the question of which split the design was tuned on.

Questions for authors

Which LLM and reasoning mode produced each row of the main results table? Do the gains hold if earlier pipelines are re-run with your backbone, and against ConTeXT-match?

What is the recall of the real pipeline over the full label set, after retrieval and after reranking?

Which data was used to choose the pool size, knowledge base and prompt? Can you report means and confidence intervals over several runs, with exemplars taken from outside the test sets?

Reviewer_3QEJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Skill-RAG, a retrieval-augmented LLM framework for extreme multi-label skill classification using the ESCO ontology. The pipeline combines dense retrieval, cross-encoder reranking, LLM-generated skill descriptions, span-based query extraction, and downstream LLM classification. The authors investigate the effects of knowledge-base construction, retrieval strategy, candidate-set size, label ordering, and prompting/reasoning configurations. Experiments on the HOUSE, TECH, and TECHWOLF benchmarks report improvements over previously published results.

Critical strengths
  • Mapping unstructured skill mentions to large, standardized taxonomies is an important problem.

  • The proposed system reports improvements over previously published results across the three evaluation datasets.

  • The experiments settings differing effects of synthetic skill descriptions on dense retrieval versus reranking, and the interaction between explicit CoT prompting and native reasoning modes, are interesting observations.

Critical weaknesses
  • My primary concern is that the paper does not clearly establish what new knowledge it contributes beyond existing work on retrieval-augmented extreme multi-label classification. The main architecture combines well-established techniques, including dense retrieval, cross-encoder reranking, synthetic text generation, span extraction, and LLM-based classification.

  • the motivation for the proposed architecture relies heavily on the limitations of presenting thousands of candidate labels directly to an LLM. Retrieval and candidate shortlisting are already established approaches for avoiding it. The paper does not sufficiently explain which limitations of these existing approaches remain unresolved and why the particular combination of components in Skill-RAG is necessary.

  • the paper attributes performance degradation with larger candidate pools to attention dilution and interprets the label-ordering experiments through positional bias. But, the experiments do not isolate the proposed mechanisms from alternative factors such as semantic ambiguity, distractor similarity, or prompt configuration.

  • section 4.2 describes the creation of candidate pools of 100 labels containing all ground-truth labels, supplemented by hard and easy negatives. Section 4.3 similarly evaluates classification using candidate pools containing all gold labels.Such experiments seem to eliminate candidate-generation failures and therefore do not fully represent extreme multi-label retrieval from the complete ontology.

Questions for authors
  1. What are the actual Recall@100 and Recall@60 values when retrieving from the complete ESCO taxonomy without oracle inclusion of gold labels? How much does each pipeline component improve final performance under this setting?

  2. Which implementation of RP@K was used? Does it match the metric used by the cited baselines, and were all comparisons performed on identical benchmark splits and evaluation protocols?

  3. How much improvement remains when comparing against a strong baseline using the same LLMs and computational budget? What are the end-to-end latency and inference-cost implications of the additional components?

Reviewer_jxqxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Skill-RAG, a pipeline for mapping job-ad text to ESCO skill labels (13,939 labels). It has three steps:

  1. Optionally, a local LLM (not named in the paper) extracts skill-related spans from the input.
  2. The spans are matched against a knowledge base of ESCO labels, optionally expanded with LLM-synthesized skill descriptions or synthetic job-ad sentences. Matching uses dense retrieval (mxbai-embed-large-v1, top 100) followed by a cross-encoder reranker (bge-reranker-v2-m3, top 60).
  3. An LLM (Gemini-2.5-Flash or DeepSeek-V4.1-Flash) selects the final labels from the candidate pool.

The paper runs component-wise studies on controlled candidate pools. These cover KB expansion, span-based querying and reranking versus dense retrieval (Section 4.2), plus candidate pool size, candidate ordering (including gold-label position), and zero-shot/few-shot/CoT prompting with and without native reasoning (Section 4.3). The full pipeline is then evaluated on the HOUSE, TECH, and TECHWOLF ESCO benchmarks (the split is not stated), and it reports higher RP@5/RP@10 and MRR than the prior results it compares against.

The claimed findings are:

  • candidate reranking outperforms dense retrieval;
  • synthetic descriptions and synthetic job-ad sentences help dense retrieval, but degrade reranking;
  • span-based queries help dense retrieval, but have only a marginal effect on reranking;
  • performance peaks at an intermediate pool size;
  • labels in the middle of the prompt are neglected;
  • CoT helps in non-reasoning mode, but hurts in native reasoning mode.
Critical strengths

The paper is easy to follow at the narrative level. It studies each stage of the pipeline separately in a controlled setting and states that code and prompts are released. For practitioners mapping job-ad text to ESCO, it is a very useful record of which engineering choices matter, and it reports higher RP@5/10 and MRR than the previously published baselines it compares against, on all three benchmarks.

Critical weaknesses

1. Limited novelty: the pipeline and its findings largely reproduce existing work.

The architecture closely follows Clavié & Soulié (2023): synthetic sentences per ESCO skill, embedding retrieval, and an LLM that selects from the shortlist. The LLM step before retrieval closely mirrors the "Infer" step of IReRa (D'Oosterlinck et al., 2024). Compared to IReRa, Skill-RAG mainly adds an intermediate cross-encoder, and it is not surprising that an extra reranking stage and a newer LLM improve results. The component findings are interesting, but they have precedents too:

  • KB expansion. Document-side expansion with generated text is doc2query (Nogueira et al., 2019). In the skill domain itself, Clavié & Soulié (2023) already index LLM-generated job-ad sentences per ESCO skill for candidate retrieval. The paper cites both, yet still calls the efficacy of synthetic expansion "an open question" (lines 249 to 250).
  • Span-based retrieval. This closely mirrors IReRa's Infer step and the extract-then-match pipelines the paper itself cites (SkillGPT, SkiLLMo). Span-level linking to ESCO is also studied in Zhang et al. (Findings of EACL 2024).
  • Prompting. Lost-in-the-middle and the cost of distractors are the cited results of Liu et al. (2024) and Cuconasu et al. (2024). That few-shot prompts hurt native reasoning models is reported in the DeepSeek-R1 report, and that CoT prompts can hurt them is reported by Meincke et al. (2025) for Gemini 2.5 Flash.
  • SOTA. ConTeXT-match (Decorte et al., IEEE Access 2025), a 109M-parameter bi-encoder, is not cited. It scores higher on TECH RP@10 (73.99 vs. 72.27) at a noticeably higher efficiency.

Taken together, the contribution is an empirical recombination of known components. While these empirical insights for the skill extraction domain are interesting and valuable to the community, they are unsurprising, and their novelty is low.

2. The "computationally efficient" claim is unsupported.

The paper calls Skill-RAG "computationally efficient" (lines 450 and 469), but it reports no latency, token counts or cost. Its main implicit comparison is against prompting an LLM with the full ontology.

A rough estimate gives a sense of scale. For 10,000 vacancies per day at about 20 sentences each, the pipeline makes about 200,000 Gemini-2.5-Flash calls per day, each with roughly 3,000 input tokens (60 candidates with definitions, plus demonstrations) and a few hundred output tokens for the CoT. At current list prices, that is roughly 60k to 250k dollars per year for the API alone, depending on batching and whether thinking is enabled, before counting the span-extraction LLM and the cross-encoder. At its reported throughput, ConTeXT-match handles the same volume on a 2-vCPU CPU machine for under 1 dollar per day, which puts Skill-RAG two to three orders of magnitude higher in cost, for gains of 3.5 to 14 RP@5 points over ConTeXT-match. The trade-off may well be worth it, but it should be reported rather than labelled efficient.

3. The setup is underspecified.

It is not clear on which data the controlled experiments were run. If these are the same test sets, this leaks into the final headline numbers. Hard negatives are "the most semantically similar false labels", but the similarity model is not given, which makes it hard to understand what these hard negatives are and whether they are real negatives. Baselines are copied from prior papers (whose LLM components are GPT-3.5, GPT-4 and Llama-2), and none is re-run with the same LLM.

Questions for authors

Clarifications (could change my assessment):

  1. Which split are the Table 3 results on? Which data were the Section 4.2 and 4.3 analyses and the choices of N, ordering and prompting made on? TECHWOLF has no validation split, so how was it handled?
  2. How exactly were the controlled pools built (the similarity model for hard negatives and whether similarity is to the input or to the gold labels, the definition of a "false label", random seeds)?
  3. For Table 3: which LLM was used, and was native reasoning on?
  4. What does the full pipeline cost per 1,000 sentences, counting span extraction, cross-encoder and LLM calls? Please give input and output tokens (including thinking tokens), wall-clock latency and dollar cost. How does this compare to current SOTA?

Toward a more principled approach (suggestions):

  1. Rather than relying on the reranker to push gold labels to the front of the prompt, how does the pipeline compare with established position-debiasing methods?
  2. The pool-size result is a trade-off between gaining recall and adding distractors (and cost). Can that trade-off be modelled explicitly, so that the optimum is predicted rather than observed?
  3. How does the same pipeline perform with much smaller final LLMs?

Minor issues:

  • Lines 366 to 368 repeat a clause ("dense retrieval relies heavily on localized, fine-grained text spans/segments").
  • "label market dynamics" should be "labor market dynamics" (line 44).
  • Notation is overloaded:
    • N denotes the number of evaluation instances (Eqs. 2 to 4), the number of synthetic sentences per skill (Section 4.2) and the candidate pool size (Figure 3).
    • K denotes both the retrieval cutoff and the pool size, and its subscript in R_K sits next to the knowledge base K in Eq. 1.
  • Figure 2's caption says K in [10,100], but its axes span 10 to 50. Figure 3 uses N on the axis but K in the caption.
  • The rank_i definition says "set to 0", which makes 1/rank_i undefined; the reciprocal rank should be set to 0.
  • Table 2's caption says "candidate reranking performance", but the table reports classification.
  • Lines 369 and 370 attribute the reranker's robustness to "full-context attention or generative reasoning", but bge-reranker is not generative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.