- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies skill retrieval when a user's requirements change during a conversation, motivated by skill-based recursive self-improvement (RSI). It argues that encoding the whole request as one vector compresses several needs into a single direction, while per-requirement MaxSim keeps only the strongest individual match. SkillProj encodes each active requirement separately with a frozen SkillRouter encoder, builds an orthonormal basis of their span by SVD, and scores every skill by projection membership, the squared norm of its projection onto that subspace. The top 50 candidates are then reranked with a frozen SkillRouter reranker, taking the maximum relevance over requirements. Finally, a DPP-style greedy step selects ten skills, using a residual kernel that down-weights the shared requirement component. To evaluate this setting, the authors build three trajectory benchmarks from SkillRouter, SkillRet and R3-Skill resources: 143 base tasks, each with three five-turn paths of additions, withdrawals and replacements, giving 429 terminal requests over libraries of up to 79,141 skills.
The paper claims the following contributions: (1) A requirement-subspace retrieval pipeline with projection scoring, requirement-level reranking and residual set selection. (2) Three trajectory benchmarks with evolving requirements: SkillRouter-Trajectory, SkillRet-Trajectory and R3-Skill-Trajectory. (3) Evidence that SkillProj improves mean Recall@10 under three language models, with matched ablations naming subspace retrieval and requirement-level reranking as the main contributors.
Retrieval is evaluated at the terminal state with eight top-10 metrics against seven baselines (Dense, SkillRouter, DSR, SkillSight, SkillDreamer, Field-Aware and SkillFlow), all using the same frozen encoder. With GPT-5.6 Luna, SkillProj leads Recall@10 on all three benchmarks: 80.61 on SR, 93.25 on Ret and 79.56 on R3. Averaged equally over the benchmarks, its Recall@10 exceeds the strongest baseline by 8.16, 5.30 and 5.95 points under Luna, DeepSeek V4.1 Flash and GLM-5.3-Flash. Confidence intervals come from a bootstrap over 42 to 45 shared-skill task families. An end-to-end study runs 1,128 executions on 47 SR tasks, where SkillProj answers 92 of 141 requests correctly against 89 for the next best method. Ablations on SR replace projection with whole-request or MaxSim retrieval, replace per-requirement reranking with whole-request reranking, and remove the DPP step.
Critical strengths
The clearest strength is the matched ablation on SR. With encoder, budgets, reranker and selector fixed, replacing projection retrieval with whole-request retrieval lowers Recall@10 by 8.12 points, and replacing per-requirement reranking with whole-request reranking lowers it by 7.63 points. Both intervals exclude zero. This shows that representing active requirements separately, rather than as one merged request, is the main source of the gain. It is a simple and practical lesson for skill routing in multi-need requests. The ablation is run only on SR with Luna, but the mean recall gain itself appears under all three language models.
The second strength is the benchmark design. Requests change through additions, withdrawals, replacements and restorations, and retrieval is scored against the requirements that remain active at the end. This tests something that static skill retrieval benchmarks do not: whether a method drops skills for withdrawn work and keeps skills for retained work. The paper also reports full metric tables for every model and benchmark, with family-level intervals, so the reader can see where SkillProj does not help.
Critical weaknesses
The main weakness is the comparison protocol behind the headline numbers. In the Luna setting on SR, SkillProj uses requirement lists maintained deterministically from the benchmark's public edits, while the baselines work from LLM-generated complete requests. This gives SkillProj a cleaner view of the task structure. When the requirements are extracted by an LLM, as in the DeepSeek and GLM settings, the gain drops by about a third, and SkillProj loses to SkillDreamer on R3. The end-to-end study adds to the doubt. SkillProj and five baselines ran in separate batches, and most paired accuracy intervals include zero. The abstract's claims of the "highest" recall and content accuracy are therefore stronger than the evidence.
The second weakness is limited novelty relative to the evidence. Projection membership is taken from Ishibashi et al. (2024), the reranker and encoder are SkillRouter's, and the residual DPP step adds nothing measurable. The remaining new element, scoring candidates per requirement and taking the maximum, is close to MaxSim at the reranking stage. The benchmarks are small, LLM-assisted, and drawn from three existing resources, and the RSI motivation is never tested. Taken together, I see a reasonable engineering pipeline evaluated on a small new benchmark, which is not enough for acceptance at a top venue.
Questions for authors
N/A
Follow-up on our response
Thank you again for your thoughtful feedback. We have added a shared-input comparison to address your fairness concern and would be grateful for your thoughts on our response when convenient. Best regards,