acceptodds
Withdrawn from ICLR 2027

MECHASKILL: Automatic Compilation and Evolution of Modular Skills for Text-to-SQL Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Strong Text-to-SQL workflows offer complementary expertise, but their capabilities remain tied to workflow-specific execution logic, making them difficult to understand, reuse, and improve. Combining these strengths requires preserving the conditions under which each capability works and assessing how capabilities affect one another. We introduce MECHASKILL, a framework that compiles workflow expertise into a modular skill for a single agent and supports module-level evolution. It recovers modules with compatible interfaces from code, documentation, and execution traces, preserving the instructions, tools, and dependencies needed for reuse. We use complementary question coverage to guide composition and evaluate the complete skill to select modules and determine when they should run. We trace failures to individual modules to guide targeted revisions, then evaluate their effect on the complete skill. We evaluate MECHASKILL alongside models, agents, full workflows, and distilled skills on BIRD-147, SPIDER2-135, and BIZ-64. In our primary evaluation, MECHASKILL achieves the highest execution accuracy among the compared systems while substantially reducing inference cost relative to the strongest workflow baseline on each dataset.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Skills, Skill Evolution, Text-to-SQL Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HxaqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MechaSkill, a framework that takes three existing Text-to-SQL workflows (APEX-SQL, DeepEye-SQL, AV-SQL), recovers "modules" from their code, documentation and execution traces, aligns module interfaces along a shared task timeline, composes a subset of modules into one skill package (a SKILL.md plus tools) for a single agent, and then evolves the package by attributing failures to modules and applying small local edits. Each candidate package is accepted only if full-skill execution accuracy on an internal dev set does not drop (§4.2–4.3, Appendix A). Two public packages are built, one for BIRD-147 and one for SPIDER2-135, plus a private BIZ-64 package. With Qwen3.6-27B, MechaSkill reports the highest execution accuracy on all three setsand an average 86.4% inference-cost reduction relative to the most accurate full workflow on each set.

Critical strengths
  • The decision-level analysis identifies where multi-candidate Text-to-SQL fails, and the finding transfers. On the 16-question panel, "Eight high-temperature candidates contain a correct answer for 13 of the 16 questions but produce only seven correct final answers" (D.4): the candidate pool is rarely the problem, the selector is. Figure 5 shows one selection-rule edit flipping a wrong answer to a right one with generation and execution held fixed, and Section 5.4 shows planning and selection edits interacting ("each add two correct answers alone; together they add five"). This is an interesting and useful lesson.
Critical weaknesses
  • The headline accuracy claim is inside the paper's own measured noise and is based on single runs. The abstract says MechaSkill "achieves the highest execution accuracy among the compared systems". Table 5 shows the margins are 105 vs 104 (BIRD-147), 75 vs 70 (SPIDER2-135) and 46 vs 44 (BIZ-64). Table 10 reports that the same SPIDER2-135 package scored 76/135 and 71/135 on two runs, a 5-question spread equal to the headline gain; other pairs move 70→67 and 75→71. Table 1 reports no seeds, no standard deviation and no run count. On BIRD-147 the +1 question over DeepEye-SQL is not distinguishable from noise, and the +10 over the best distilled skill is bought at 11× its cost. Without ≥3 seeded runs per system, the "highest accuracy" claim and the "accuracy–cost frontier" claim are not supported.

  • The evaluation sets were used to engineer the skills. Appendix B.5 says SPIDER2-135 "is not used for skill development", yet Table 10 scores twelve skill edits on SPIDER2-135 and the final skill carries exactly those rule families (C.2), while on BIRD-147 the rules in Table 4 were written for three failures on a BIRD-147 panel and the final skill matches the best of 18 archived BIRD-147 runs (D.4, D.8). The development pool also includes official BIRD dev questions that share databases with BIRD-147 (Table 2). The reported accuracies therefore measure fit to the evaluation subsets, not generalization, and the accuracy–cost frontier inherits the problem.

  • The step from workflow to skill is essentially skill distillation, which has become a common practice with agent skill files. The novelty is questionable. And the other components - automatic multi-source module recovery and attribution-guided evolution - were not demonstrated as a method. The paper gives no count of module candidates kept, merged or split without human decisions, no effort figures, and no statement of whether the SKILL.md in Appendix C.4 was generated or written; the evolution examples (calendar-year rule, percentage contract, "Do not reconstruct Amount * Price") read as hand-authored prompt rules. Until automation is quantified and the compiler is run on workflows the authors did not pick, the framework reads as careful engineering on three codebases rather than a transferable method.

Questions for authors
  • Were any rules from Table 4 (temporal precision, scalar output, arithmetic) or Table 10 (percentage, calendar-year, session-endpoint, full-name, set-union contracts) present in the frozen packages scored in Table 1? Please give the content hashes and dates of the frozen packages and of each archived BIRD-147 / SPIDER2-135 run, so the order "frozen before diagnostics" can be verified.
  • How many distinct package versions were evaluated on BIRD-147 and on SPIDER2-135 in total before the Table 1 numbers were produced? Figure 6 shows 18 and 12; were there others?
  • Please report ≥3 seeded runs (mean ± sd) for MechaSkill, the strongest workflow and the strongest distilled skill on each dataset. Does the 1-question BIRD-147 margin over DeepEye-SQL survive?
  • What concurrency was used for each method in the resource runs? Can you report summed model-compute time or token counts as the cost axis instead of batch wall-clock? Why does MechaSkill cost 61.50 on BIRD-147 but 9.96 on SPIDER2-135?
  • How automatic was compilation? Of the 30 raw candidates (D.3), how many were accepted, merged or split without human edits, and was the SKILL.md in C.4 produced by the LLM or by hand? Which of V4–V18 / R14–R21 were human-authored?
  • Can you ablate the stages separately: composition without evolution, evolution without attribution-based targeting, and composition without coverage guidance? Without this the gain cannot be assigned to any part of the framework rather than to the extra compute spent.
  • Of the attribution-identified boundaries, how many produced an edit, how many edits were accepted on the dev set, and how many accepted edits regressed previously solved questions in a repeat run? Table 10 shows the data exist.
Reviewer_uj2ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MECHASKILL recovers reusable modules from the code, documentation, and execution traces of APEX-SQL, DeepEye-SQL, and AV-SQL. It composes modules using complementary question coverage and supports targeted evolution through failure attribution. The resulting single-agent skills achieve execution accuracies of 71.4%, 55.6%, and 71.9% on three evaluation sets, with an average inference-cost reduction of 86.4% relative to the strongest complete workflow on each dataset. The paper also presents cross-model transfer and diagnostic analyses of module edits.

Critical strengths
  1. The modular representation has practical value. Beyond transferring prompts, the framework preserves states, dependencies, and invocation conditions, enabling capabilities from heterogeneous workflows to be combined and revised selectively. This is a meaningful engineering contribution beyond simply concatenating workflow instructions.
  2. The paper demonstrates an attractive accuracy–cost tradeoff. In the primary evaluation, MECHASKILL exceeds individual workflows and distilled skills in accuracy while substantially reducing costs relative to complete workflows. Selection-rule replay with a fixed candidate pool also provides concrete explanations of how particular decisions improve.
Critical weaknesses
  1. The central automation claim is insufficiently validated. The appendix provides module inventories and cached boundary checks, but not a sufficiently detailed automatic decision trail from raw source code to the final skill. Boundary-recovery quality and human intervention are also not quantified. Consequently, it remains unclear how much of the contribution comes from automatic discovery rather than careful reconstruction of existing systems. An end-to-end construction audit and a predefined-module comparison would strengthen this claim.
  2. Improvements are not adequately disentangled from additional computation. On BIRD-147, the gain over the strongest distilled skill comes at approximately 11.14 times its cost (Appendix D.1). Without compute-matched single-source baselines and controlled comparisons isolating discovery, coverage-guided composition, and local evolution, the results do not establish superiority over additional sampling or conventional prompt optimization.
  3. Evidence for generality remains limited. The two public benchmarks use different skill packages, and improvements over the strongest complete workflows correspond to only one, five, and two questions. Cross-model transfer preserves some advantages, but covers only another Qwen model and trails DeepEye on all three datasets. These results support the usefulness of particular configurations more strongly than broadly reusable compilation across heterogeneous workflows.
Questions for authors
  1. Which final module boundaries, tool adaptations, invocation rules, and patches were generated automatically, and which required human specification or modification? Can you provide a complete construction trace?
  2. Does MECHASKILL retain its advantage over the strongest single-source skill under matched candidate-generation and inference budgets? Which controlled comparisons establish an independent contribution from automatic discovery and evolution?
  3. Are development and evaluation separated at the database level? Do you have results using one frozen skill across datasets or incorporating a workflow not involved in framework design?
Reviewer_exV1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MechaSkill takes three existing Text-to-SQL workflows (APEX, DeepEye, AV-SQL), automatically "compiles" them into modules, composes the modules into one skill for a single agent, and then evolves the skill module by module (Discover, Align, Compose, Evolve). It claims the highest execution accuracy on BIRD-147, SPIDER2-135 and BIZ-64 at more than 80% lower cost than the strongest workflow (86.4% on average), and that a skill built with Qwen3.6-27B transfers unchanged to Qwen3.5-35B-A3B.

Critical strengths
  1. The appendix is unusually honest. Appendix D.1 spells out that the gains are 1, 5 and 2 questions and does not claim statistical significance, Table 5 gives exact counts, and Table 10 reports repeat runs for several edits, which most papers in this space skip.

  2. The full SKILL.md text for the composed skill and the three distilled skills is reproduced (Appendix C), so someone could actually rerun this, and Figures 1 and 2 give a clear picture of what was built and which source each module came from.

Critical weaknesses
  1. The headline gains are inside the noise. On BIRD-147 the margin is one question over DeepEye (105 versus 104 of 147); on SPIDER2-135 it is five over APEX. Table 10 shows the same package hash scoring 76 and then 71 of 135 on repeat, a five-question swing. Table 1 is a single run with seed 42 and no confidence intervals. "Highest execution accuracy" in the abstract is not supported.

  2. The evolution loop appears to touch the evaluation set. Appendix B.5 says train and dev only, but Section 5.4 and Appendix D.4 compare 18 BIRD-147 and 12 SPIDER2-135 "historical runs" of skill edits scored on the test sets, and the final 105/147 is one of these archive runs. The Figure 5 fix is BIRD-147 question 1527, and the Table 4 rules "address residual failures" with "all three targeted reruns" producing correct answers on test questions. Appendix D.4 states the diagnostics "did not guide the selection", but the timeline is not convincing and needs to be clarified.

Questions for authors
  1. Were any of the 18 BIRD-147 or 12 SPIDER2-135 historical runs (Figure 6, Appendix D.4) used to pick or edit the skill that produced Table 1? Please give the timeline.
  2. Can you run Table 1 with three or more seeds and report confidence intervals? Does the +1 on BIRD survive?
  3. Why compare cost to the workflow rather than to the distilled skill, which is the closer system?
  4. How much of the Figure 2 composition (shared logical plan, 4-of-8 consensus guard) was proposed by the search versus written by the authors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.