- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Skill-RAG maps job-ad sentences to ESCO skills, an extreme multi-label task over 13,939 labels. The pipeline:
- a local LLM extracts skill spans;
- dense retrieval (mxbai-embed-large-v1) finds the top-100 labels from a knowledge base, optionally expanded with LLM-written skill descriptions;
- a cross-encoder (bge-reranker-v2-m3) reranks to the top-60;
- an LLM (Gemini-2.5-Flash or DeepSeek-V4.1-Flash) selects the final labels.
Component studies cover knowledge-base variants, span vs raw queries, candidate-pool size, label ordering, and zero-shot/few-shot/CoT prompting with and without native reasoning. The final system is compared with four published systems on HOUSE, TECH and TECHWOLF and claims a new state of the art.
Critical strengths
Clear practical task and large gains over the cited systems without fine-tuning:
- "+All" improves RP@5 over the best cited baseline by +8.47 (HOUSE), +5.99 (TECH) and +6.97 (TECHWOLF) (Table 3).
- If the gains hold under a fair comparison, this is a useful system for labour-market research.
A consistent practitioner finding:
- explicit CoT helps when native reasoning is off and hurts when it is on, in the same direction for both model families (Table 2).
Critical weaknesses
The gains cannot be credited to the pipeline rather than to the newer LLM backbone, and the comparison set is incomplete.
- The Table 3 LLM baselines use GPT-3.5/GPT-4 or Llama-2, while Skill-RAG uses 2025-era models.
- The paper does not say which LLM or reasoning mode produced Table 3 (L431–450).
The component studies don't support the design choices well. The pool-size and ordering experiments use candidate pools built to contain every correct label, so they measure ranking within an easy pool, not retrieval from 14,000 labels; the real pipeline's recall is never reported. The pool-size curve appears to peak around 40–50 candidates, yet the system uses 60. No seeds, confidence intervals or significance tests are given, although several effects are called "significant" on test sets of only ~300 sentences. Finally, the released prompts use two HOUSE test sentences as few-shot examples. The effect is small, but it raises the question of which split the design was tuned on.
Questions for authors
Which LLM and reasoning mode produced each row of the main results table? Do the gains hold if earlier pipelines are re-run with your backbone, and against ConTeXT-match?
What is the recall of the real pipeline over the full label set, after retrieval and after reranking?
Which data was used to choose the pool size, knowledge base and prompt? Can you report means and confidence intervals over several runs, with exemplars taken from outside the test sets?