- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper optimizes reusable textual skills for frozen LLM executors on multi-stage agentic tasks. It argues that natural-language "textual gradients" implicitly fuse two decisions — where in a skill to apply an update and whether a proposed revision actually reduces task loss. SkillBackprop separates them: a skill is compiled into a fixed operator DAG; Stage 1 (failure localization) uses an LLM Analyzer to attribute each failed execution to exactly one responsible operator and aggregates loss-weighted blame across a mini-batch; Stage 2 (loss-reduction estimation) has an LLM Optimizer propose typed candidate edits at the selected operator, each measured on a validation subset, accepting only candidates with positive measured loss reduction (ε = 0), with an epoch-level consolidation ledger. Evaluation: SpreadsheetBench, OfficeQA, DeepPlanning-Shopping, SearchQA × three frozen backbones (DeepSeek-v4-Flash, MiniMax-M3, GLM-5.3) vs No Skill / Compiled Skill / GEPA / SkillOpt, plus ablations, a blinded human re-audit of 103 failure attributions (93.2% supported), and a cross-model transfer study. Reported average: 74.24 vs 65.46 for the strongest baseline (SkillOpt).
Critical strengths
- A principled decomposition with a measured (not LLM-judged) acceptance rule. The where/whether separation is a real conceptual delta over GEPA/SkillOpt, candidate promotion is deterministic code rather than LLM preference, and the ablations (random/first-failure attribution, accept-all, monolithic skill, no clustering, no consolidation) support that the components are complementary. The blinded human audit of failure attributions is a welcome validation step that most textual-gradient work skips.
- Unusually transparent engineering. Identical task-exposure and update schedules vs SkillOpt, benchmark-native metrics (no LLM-as-judge), validation Q/A exclusion from the policy, full initial+final skill artifacts, prompt contracts, split manifests, and qualitative traces (including a rejected candidate) are all provided. This substantially raises confidence that the pipeline does what is described.
Critical weaknesses
- The central comparative claim is statistically unsupported. Every model×benchmark cell is a single archived run — no variance, CIs, or significance tests anywhere. On DeepPlanning the margin over the strongest baseline equals exactly one test case (1.19 pts of 84; 75.00 vs 73.81); on SearchQA it is ≈6/840 cases. The transfer study itself provides internal evidence of large run-to-run variance: a GLM-5.3-learned skill transferred to DeepSeek-v4-Flash beats the skill trained on DeepSeek itself by +22.9 pts on SpreadsheetBench (84.29 vs 61.43, Table 3), an unexplained inversion. The abstract's headline numbers are the best model's (GLM-5.3) row, not cross-model averages. Multi-seed results could resolve this, but as submitted the claim "matches or outperforms the strongest baseline across all settings" is not established.
- Thin baseline coverage and unverified budget parity. The paper's named foils — TextGrad and ProTeGi — are never evaluated, which matters because ProTeGi already couples textual "gradients" with validation-based beam selection, undercutting the novelty framing of Stage 2; the closest work (SkillGrad) is cited but not benchmarked. GEPA is capped at 100 metric calls while SkillBackprop consumes ~160 training rollouts plus per-candidate validation evaluations whose total rollout/token cost is never reported, so compute parity is unestablished. Additionally, the ε = 0 acceptance threshold on 8-task validation sets (one task = 12.5 pts, ~20 sequential accept decisions) is fragile against adaptive overfitting, and the emphasized relative gains are computed against Compiled Skill, which is worse than No Skill in 8 of 12 cells.
Questions for authors
- Can you report ≥3 independent optimization runs per model–benchmark cell with standard deviations, and explain the Table 3 transfer inversion (transferred skill beating the target-trained skill by 22.9 pts)? This single question most affects my assessment.
- What is SkillBackprop's total rollout/token cost per benchmark, including candidate evaluations on V, versus GEPA's 100 metric calls? Can you add a compute-matched GEPA configuration and TextGrad/ProTeGi baselines?
- The human audit uses a single annotator with no inter-rater reliability, and OfficeQA agreement is only 79.3% (11/29 ambiguous). How sensitive are the downstream gains to localization errors?
- (Clarification) Algorithm 1's EC(v) = f_contract(v) + α f_lineage^p(v) + β f_downstream(v) and U(v) use undefined f-functions, α, β, p, T. Please define them and provide sensitivity analysis.