- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MechaSkill, a framework that takes three existing Text-to-SQL workflows (APEX-SQL, DeepEye-SQL, AV-SQL), recovers "modules" from their code, documentation and execution traces, aligns module interfaces along a shared task timeline, composes a subset of modules into one skill package (a SKILL.md plus tools) for a single agent, and then evolves the package by attributing failures to modules and applying small local edits. Each candidate package is accepted only if full-skill execution accuracy on an internal dev set does not drop (§4.2–4.3, Appendix A). Two public packages are built, one for BIRD-147 and one for SPIDER2-135, plus a private BIZ-64 package. With Qwen3.6-27B, MechaSkill reports the highest execution accuracy on all three setsand an average 86.4% inference-cost reduction relative to the most accurate full workflow on each set.
Critical strengths
- The decision-level analysis identifies where multi-candidate Text-to-SQL fails, and the finding transfers. On the 16-question panel, "Eight high-temperature candidates contain a correct answer for 13 of the 16 questions but produce only seven correct final answers" (D.4): the candidate pool is rarely the problem, the selector is. Figure 5 shows one selection-rule edit flipping a wrong answer to a right one with generation and execution held fixed, and Section 5.4 shows planning and selection edits interacting ("each add two correct answers alone; together they add five"). This is an interesting and useful lesson.
Critical weaknesses
-
The headline accuracy claim is inside the paper's own measured noise and is based on single runs. The abstract says MechaSkill "achieves the highest execution accuracy among the compared systems". Table 5 shows the margins are 105 vs 104 (BIRD-147), 75 vs 70 (SPIDER2-135) and 46 vs 44 (BIZ-64). Table 10 reports that the same SPIDER2-135 package scored 76/135 and 71/135 on two runs, a 5-question spread equal to the headline gain; other pairs move 70→67 and 75→71. Table 1 reports no seeds, no standard deviation and no run count. On BIRD-147 the +1 question over DeepEye-SQL is not distinguishable from noise, and the +10 over the best distilled skill is bought at 11× its cost. Without ≥3 seeded runs per system, the "highest accuracy" claim and the "accuracy–cost frontier" claim are not supported.
-
The evaluation sets were used to engineer the skills. Appendix B.5 says SPIDER2-135 "is not used for skill development", yet Table 10 scores twelve skill edits on SPIDER2-135 and the final skill carries exactly those rule families (C.2), while on BIRD-147 the rules in Table 4 were written for three failures on a BIRD-147 panel and the final skill matches the best of 18 archived BIRD-147 runs (D.4, D.8). The development pool also includes official BIRD dev questions that share databases with BIRD-147 (Table 2). The reported accuracies therefore measure fit to the evaluation subsets, not generalization, and the accuracy–cost frontier inherits the problem.
-
The step from workflow to skill is essentially skill distillation, which has become a common practice with agent skill files. The novelty is questionable. And the other components - automatic multi-source module recovery and attribution-guided evolution - were not demonstrated as a method. The paper gives no count of module candidates kept, merged or split without human decisions, no effort figures, and no statement of whether the SKILL.md in Appendix C.4 was generated or written; the evolution examples (calendar-year rule, percentage contract, "Do not reconstruct Amount * Price") read as hand-authored prompt rules. Until automation is quantified and the compiler is run on workflows the authors did not pick, the framework reads as careful engineering on three codebases rather than a transferable method.
Questions for authors
- Were any rules from Table 4 (temporal precision, scalar output, arithmetic) or Table 10 (percentage, calendar-year, session-endpoint, full-name, set-union contracts) present in the frozen packages scored in Table 1? Please give the content hashes and dates of the frozen packages and of each archived BIRD-147 / SPIDER2-135 run, so the order "frozen before diagnostics" can be verified.
- How many distinct package versions were evaluated on BIRD-147 and on SPIDER2-135 in total before the Table 1 numbers were produced? Figure 6 shows 18 and 12; were there others?
- Please report ≥3 seeded runs (mean ± sd) for MechaSkill, the strongest workflow and the strongest distilled skill on each dataset. Does the 1-question BIRD-147 margin over DeepEye-SQL survive?
- What concurrency was used for each method in the resource runs? Can you report summed model-compute time or token counts as the cost axis instead of batch wall-clock? Why does MechaSkill cost 61.50 on BIRD-147 but 9.96 on SPIDER2-135?
- How automatic was compilation? Of the 30 raw candidates (D.3), how many were accepted, merged or split without human edits, and was the SKILL.md in C.4 produced by the LLM or by hand? Which of V4–V18 / R14–R21 were human-authored?
- Can you ablate the stages separately: composition without evolution, evolution without attribution-based targeting, and composition without coverage guidance? Without this the gain cannot be assigned to any part of the framework rather than to the extra compute spent.
- Of the attribution-identified boundaries, how many produced an edit, how many edits were accepted on the dev set, and how many accepted edits regressed previously solved questions in a repeat run? Table 10 shows the data exist.