- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Skill Tree uses Markdown hierarchy to consolidate trajectory-derived skill edits without rewriting retained content. It combines scoped deduplication, core/reference allocation, constrained reordering, and task-specific reference retrieval. Evaluation covers spreadsheet tasks, frozen-skill transfer to WikiTableQuestions, cross-model reuse, and separately evolved DocVQA skills.
Critical strengths
- Using the same document hierarchy to consolidate, organize, and retrieve experience is a coherent response to detail loss during skill rewriting. Eight component and structural ablations directly support the design, beyond the auxiliary similarity metrics.
- With the same trajectory pool as several strong baselines, Skill Tree leads on all four held-out Vrf settings, including gains above eight points in both Creation settings. Frozen WikiTQ evaluation and cross-family reuse provide additional evidence that the skills transfer beyond their construction tasks.
Critical weaknesses
- Appendix A.9 says domain priorities, workflow constraints, routing weights, and intent rules are fixed, but does not list them. These choices affect the actual algorithm, so their omission limits reproducibility and makes the effort needed for a new domain unclear.
- Full SpreadsheetBench Soft/Hard scores include the 200 evolution tasks. They mix seen-task improvement with generalization and should be separated when discussing robustness. This limits those particular claims, while the independent Vrf and frozen WikiTQ results remain meaningful.
- Actual context and construction costs are missing. A 1,500-line core and 40,000-character reference allowance do not reveal token exposure across methods. The existing structural controls support effectiveness, but users still need the performance–cost tradeoff to judge practical value.
- Embedding-based retention favors unchanged wording and cannot establish preservation of every condition or formula. NLI deletion errors and conflicting edits also need discussion. These gaps limit the mechanism explanation; downstream accuracy and the deduplication ablations already provide stronger evidence of usefulness.
- The cited ACE already uses localized edits and deterministic merging, and SkillReducer separates core guidance from references. Positioning should emphasize the shared hierarchy and its tested benefits, rather than imply that non-rewriting consolidation or core/reference separation is itself new.
- Minor corrections: resolve “Section ??” in Appendix A.6 and change “Subtrees form Reference File” to “Subtrees from Reference File” in Appendix A.12. Add uncertainty to the three-seed means, especially for the small DocVQA margins.
Questions for authors
- Can you provide the exact prompts and domain-specific rules, explain how they were selected, and identify what changed between spreadsheets and DocVQA?
- What are Soft/Hard scores excluding evolution tasks, and is the DocVQA split disjoint by document identity as well as question–image pair?
- What are the actual construction costs and execution-time prompt tokens for Skill Tree and the main baselines?
- How are contradictory edits or competing replacements handled, and what deletion errors appear in a small audit of existing NLI decisions?
2 comments
Thank you very much for taking the time to revisit our rebuttal and for your support. We sincerely appreciate your thoughtful feedback and consideration!
Thank you for the additional experiments. The held-out SpreadsheetBench results and document-disjoint DocVQA evaluation address my main concern about generalization. I am raising my rating from 2 to 3 (Weak acceptance).