- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PROSE learns a skill document for a frozen LLM agent. It compares repeated attempts of the same task, collects lessons supported by multiple tasks, and applies structured edits after checking their supporting evidence. The paper reports results on three benchmarks, with higher mean success rates and shorter documents than the evaluated baselines. It also evaluates a setup using one open-weight model for all roles.
Critical strengths
S1. The learned skills are substantially shorter, using 43–88% fewer characters than the competing methods while achieving higher mean success rates. This reduces the context needed each time the skill is used.
S2. The method also works with one open-weight model serving all roles. With Qwen3.6-27B on AppWorld test-normal, TGC improves from 79.2% without a skill to 91.9%, and SGC from 62.5% to 83.3%.
Critical weaknesses
W1. The methodological contribution is incremental. PROSE combines grouped sampling, trajectory reflection, cross-task consolidation, and constrained document editing. Its main addition is an explicit protocol linking evidence to individual edits and checking whether each edit meets predefined requirements. This improves control and inspectability, but the contribution remains primarily a system-level integration of existing skill-learning ideas.
W2. The ablations do not isolate the contribution of evidence licensing. The free-editing variant removes structured editing and licensing together, while the no-mixed-group-contrast variant excludes entire mixed groups from the evidence available to the optimizer. These comparisons support the complete design, but cannot separate better evidence use from more conservative editing or access to additional useful trajectories.
W3. Comparisons with additional skill extraction and optimization methods, such as SkillCAT and SkillAdam, are needed to establish the method's relative advantages.
Questions for authors
Q1. The free-editing ablation removes structured editing and evidence licensing together. The performance drop therefore cannot be attributed to evidence licensing alone, leaving its independent contribution unclear.
Q2. Additional evaluation on a web-interaction or code-maintenance benchmark would help establish whether the compactness and performance gains extend to other agent workflows.
Q3. Please include additional skill extraction and optimization baselines.
2 comments
Thank you for reading the response and the new ablation. One clarification for the record.
We agree, and stated in our response, that removing evidence licensing alone does not change accuracy significantly. We do not claim an accuracy gain for licensing. Its measured effect is on the document: without it the document is 38% larger (3,894 against 2,830 characters) and holds 13 entries instead of 8, because single-task lessons are written in instead of being held pending.
This matches the paper's claim, which is accuracy on par with the strongest baseline with a document 1.8–8.6× smaller in which every entry is traceable to verified evidence. Licensing contributes the compactness and the traceability, at no cost in accuracy.
We appreciate your engagement throughout the discussion.
Thank you for the detailed response and additional ablation. My remaining concern is that the methodological contribution is still incremental, as the framework mainly integrates existing contrast and structured-editing ideas under a rule-based evidence check whose isolated accuracy gain is not statistically significant in the new ablation, so I will maintain my current assessment.