- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SkillFM, a two-stage framework capable of generating task-relevant textual skills for frozen LLM agents without requiring skill retrieval at test time. First, a skill codec is trained to encode textual skills into a continuous, decodable latent space; subsequently, a conditional one-step flow matcher learns a velocity field that maps task descriptions to latent skill codes and decodes these codes back into readable instructions. SkillFM achieves superior performance compared to vectorized skill baselines on the ALFWorld, Search-QA, and WebShop datasets, demonstrates portability across model families, and exhibits robustness against retrieval issues in the presence of noisy skill libraries.
Critical strengths
The experiments are comprehensive, featuring multi-domain evaluations and employing consistent frozen-encoder settings and ablation studies to isolate the impact of the codec training and pipeline. The work demonstrates a viable alternative to retrieval-intensive pipelines, thereby simplifying deployment and reducing reliance on complex memory management (a point I particularly value).
Critical weaknesses
1: Baseline comparability: Much of the baseline data originates from prior studies where the actuators, prompting rules, or operational budgets may not have been identical. Retrieval baselines under noise conditions lack specific details (regarding retrieval models, re-ranking, or top-k strategies), potentially leading to an underestimation of robust retrieval pipelines (a point I particularly value).
2: Jointly learned latent language generators offer an alternative to the two-stage codec-plus-stream approach; however, the relevant discussion is confined to the appendix, and a more explicit comparison in the main text would have been beneficial.
Questions for authors
1: Previous studies have reported several baselines. Could key baselines (e.g., LatentSkill, SHINE, strong retrieval methods) be re-run using the same actuators, prompts, and operational budgets to ensure a fair comparison?
2: How robust is the decoding process to latent perturbations at inference time (e.g., small Gaussian noise or geodesic noise around )? Does the decoding of skills degrade gracefully, and does this correlate with success?
3: For retrieval baselines with noise levels below 25%, what retrievers and settings were used (embedding models, values, re-ranking, hybrid sparse-dense approaches)? Could a stronger retrieval-plus-re-ranking baseline be added to facilitate a better comparison?
8 comments
Response to Reviewer 8Mtk (W1, Q1, Q3): Retrieval Details and Additional Experiments under Noise
Retriever and settings. All retrieval baselines in the noisy-bank experiments use one dense retriever, fixed before any test rollout: Qwen3-Embedding-0.6B, with last-token pooling, L2-normalised vectors and cosine similarity. The query is the task text. Each skill is embedded as it is. We retrieve once per episode and execute the top-1 skill. We use no re-ranking and no sparse component.
Stronger retrieval. Following the suggestion, we ran every domain with re-ranking. The top-5 distinct skills from the same index are re-scored by the cross-encoder Qwen3-Reranker-4B, and the highest-scoring skill is executed. Executor, prompts and budgets are unchanged.
| Domain (metric) | Dense top-1 | Dense top-5 + reranker | Ours |
|---|---|---|---|
| ALFWorld (SR) | 34.67 | 34.31 | 75.91 |
| WebShop (SR) | 15.20 | 17.20 | 18.40 |
| Single-hop QA (EM) | 42.20 | 40.73 | 45.73 |
| Multi-hop QA (EM) | 27.45 | 29.42 | 31.82 |
Scaling the encoder to 4B helps, but the retriever still stays far below our method.
| ALFWorld retriever | Seen SR (%) | Unseen SR (%) | Overall SR (%) |
|---|---|---|---|
| 0.6B encoder, top-1 | 38.6 | 30.6 | 34.7 |
| 0.6B encoder, top-5 + reranker | 36.4 | 32.1 | 34.3 |
| 4B encoder, top-1 | 45.7 | 39.6 | 42.7 |
| 4B encoder, top-5 + reranker | 47.9 | 39.6 | 43.8 |
| Ours | 74.3 | 77.6 | 75.9 |
Response to Reviewer 8Mtk (W2, Q2): Clarifying the Model Architecture and Robustness to Noise
As for W2, the jointly learned latent language generators mentioned in Weakness 2 are not the approach we adopt. We train the codec and the flow in separate stages, so that the codec first establishes a stable latent space and the flow is then learned on top of it. Following your suggestion, we added Section 5.4 ("Two-stage versus joint latent generation") to the main text. It compares SkillFM with jointly learned latent language generators (text VAEs and embedding diffusion) and with two-stage approaches that fit a generator to frozen autoencoder latents, and explains why we adopt the two-stage design: the flow target stays fixed, which avoids latent collapse; the frozen latent space lets generated codes be checked against the decoder's training range; and flow training does not back-propagate through the codec. Our ablations, together with the additional direct-mapping and retrieval baselines, support the necessity of flow matching.
Robustness to Noise
The codec decoder is trained with spherical perturbations of up to 0.35 rad (Section 3.2 and Appendix B.1). Following your suggestion, we tested whether this robustness holds at inference time. Details are provided in Appendix I.
Setup. For each of the 274 ALFWorld test tasks, we took the generated skill latent from the main run and perturbed it in two ways:
- Geodesic noise on the unit hypersphere: , where is a random unit vector orthogonal to .
- Gaussian noise with scale relative to , so that the realized angle approximately matches .
We swept from 0.05 to and decoded every perturbed latent with the frozen decoder, greedily and one latent at a time (6,028 decodes in total). A decode is task-consistent if it names the correct task type, object, receptacle, and tool, and its route acts on them. We also record whether its text is identical to the decode of the unperturbed . Decoding the unperturbed twice gives identical text in 274/274 tasks.
Degrades gracefully. Geodesic and Gaussian noise give nearly identical results:
| θ (rad) | Task-consistent (geodesic) | Identical (geodesic) | Task-consistent (Gaussian) | Identical (Gaussian) |
|---|---|---|---|---|
| 0 | 99.6% | 100% | 99.6% | 100% |
| 0.05 | 99.8% | 76.7% | 99.6% | 78.8% |
| 0.1 | 100% | 73.4% | 100% | 74.0% |
| 0.2 | 99.3% | 58.2% | 99.6% | 61.2% |
| 0.3 | 96.9% | 38.6% | 96.7% | 41.8% |
| 0.5 | 73.4% | 4.4% | 73.4% | 2.9% |
| 0.8 | 0% | 0% | 0% | 0% |
| π/2 | 0% | 0% | 0% | 0% |
Up to rad, within the codec's training perturbation range, at least 96.7% of decoded skills remain task-consistent. This range covers the generated endpoints: on the flow-stage holdout, the 95th-percentile angle from generated endpoints to their nearest reference codes is 0.28 rad (Appendix F.4). Exact text identity falls smoothly as grows, so small perturbations change only surface details, such as object instance indices, the order in which locations are searched, and wording, while the skill still describes the same task. Consistency reaches zero at about 0.8 rad, the median angle between two training skills, where the decoder switches to a different skill.
Response to Reviewer 8Mtk (W1): Baseline Settings
We are grateful for your thorough review and helpful questions, which have helped us improve the paper. Our responses to each point are provided below.
We would like to add that our ALFWorld protocol strictly follows that of LatentSkill and related work: the same executor (Qwen3-8B in thinking mode, frozen), nearly identical prompts (the only difference being that our skills are injected via the prompt, whereas their methods use LoRA), the same interaction budget (ALFWorld's default maximum of 50 steps), and a shared evaluation pipeline. Moreover, our vanilla baseline results closely match their published numbers which means our pipeline are idetical. When we reproduced the key baselines in our own environment, they fell short of the results reported in the original papers; to keep the comparison conservative, we therefore report their (higher) published numbers. The results are as follows:
In addition, our WebShop and Search-QA experiments follow the same settings as the baselines; please refer to Appendix B.1 (Table 14) of the revised paper for details.
Reference
[1] Aofan Yu, Chenyu Zhou, Tianyi Xu, Zihan Guo, Rong Shan, Zhihui Fu, Jun Wang, Weiwen Liu, Yong Yu, Weinan Zhang, and Jianghao Lin. LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents. arXiv preprint arXiv:2606.06087, 2026.