- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SKILLZIP, an engineering method that converts recurring phases in successful agent trajectories into two replay-checked macros, INSPECT and CHECK, which handle pre-edit inspection and post-edit validation. The macros are author-provided templates with typed parameters and observable contracts; source traces only determine whether a template is supported and admitted through replay. The evaluation uses ten Go repositories with temporally separated source and held-out tasks, two models, and four conditions: no memory, text workflow, retrieved actions, and SKILLZIP. The main metric is model-mediated tool decisions, with completion measured by applying the agent patch and running the full suite with developer tests. Results show fewer tool decisions under SKILLZIP across conditions, with cost and wall-time savings against some baselines, while the macros internally execute many deterministic local stages that are logged separately.
Critical strengths
-
The work is motivated by a reasonable engineering observation: successful agents tend to repeat the same pre-edit inspection and post-edit validation phases. Turning these phases into reusable, replay-checked macros is a coherent way to reduce repeated tool interaction, and the macro design follows standard software-engineering practice.
-
The experimental setup is controlled. The temporally separated benchmark, frozen revisions, hidden developer tests, and repository-clustered inference give the evaluation more structure than is common in this area. The authors also include a fresh repeat and a third generation, and report model-level results alongside pooled numbers.
Critical weaknesses
-
The paper claims efficiency gains, but the primary metric, model-mediated tool decisions, counts a macro call as one decision while treating its internal subprocesses separately. This makes the reduction partly a matter of accounting. The paper also does not report token overhead, even though context length and workflow differ across conditions. On DeepSeek, cost is higher under SKILLZIP and the decision count does not decrease. Pooling dollar costs across models with different pricing makes the cost-neutral conclusion difficult to interpret. As a result, the efficiency claim is not fully supported by the current evidence.
-
The contribution is framed as a experience-learning mechanism, but the macros, typed slots, contracts, and executable bodies are all author-provided. The compiler does not synthesize macro bodies or discover new template families; source traces only determine support and admission. The two admitted macros reflect standard software-engineering practice, and there is no baseline with the same hand-written macros but without experience gating. Since macros and primitive tools remain freely mixed at inference, the method can also be read as a consolidated tool with coarser action granularity. The authors note that the primary matrix measures the author-specified interface rather than the causal effect of gating, which makes the broader framing harder to sustain.
Questions for authors
-
Could you report token usage and normalized cost per task for each condition? The current decision metric counts a macro call as one decision but does not account for internal subprocesses, so it is unclear whether the efficiency gain is real or a change in accounting.
-
According to Table~3, On DeepSeek, SKILLZIP shows higher cost and no reduction in decisions. Could you explain this model difference and clarify whether the pooled result is the right basis for the efficiency claim?
-
The macros, slots, contracts, and bodies are all author-provided. What does experience gating contribute beyond admission?