acceptodds
Under review as a conference paper at ICLR 2027

SkillZip: Experience-Gated Macro Actions for Tool-Using Agents

Abstract

Language agents routinely rediscover the same multi-step procedures: search a repository, open the relevant implementation and tests, then format, test, and inspect a patch. Existing experience memories preserve such procedures as text that the model must interpret again. We introduce SkillZip, a training-free compiler that uses successful-trace support to select parameterized phase templates and admits each executable action only after sandbox replay establishes its observable contract. We evaluate the resulting action interface on 20 temporally held-out software tasks from 10 public repositories with two agent models and four matched conditions. Across 40 model–task pairs, SkillZip matches observed no-memory completion at 24/40 (paired 95% CI: to points) while reducing model-mediated tool decisions by 23.1% (repository-clustered 95% CI: 14.1–31.2% fewer; exact sign-flip ). It removes 31.2% of decisions versus an equally sourced textual workflow and 27.6% versus retrieved action sequences; against retrieval it also reduces API cost by 10.9% and wall time by 29.1%. Fresh regeneration reduces decisions by 29.6%, model cost by 18.4%, and raises success from 14/20 to 16/20. Across three paired Luna generations, the decision reduction is 35.4% (two-way repository–generation 95% CI: 19.6–45.0% fewer). Successful experience can therefore become a smaller interface for future decisions, not another block of context.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, tool use, experience memory, macro actions, temporal abstraction, software agents, action interfaces

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mpKhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SKILLZIP, an engineering method that converts recurring phases in successful agent trajectories into two replay-checked macros, INSPECT and CHECK, which handle pre-edit inspection and post-edit validation. The macros are author-provided templates with typed parameters and observable contracts; source traces only determine whether a template is supported and admitted through replay. The evaluation uses ten Go repositories with temporally separated source and held-out tasks, two models, and four conditions: no memory, text workflow, retrieved actions, and SKILLZIP. The main metric is model-mediated tool decisions, with completion measured by applying the agent patch and running the full suite with developer tests. Results show fewer tool decisions under SKILLZIP across conditions, with cost and wall-time savings against some baselines, while the macros internally execute many deterministic local stages that are logged separately.

Critical strengths
  1. The work is motivated by a reasonable engineering observation: successful agents tend to repeat the same pre-edit inspection and post-edit validation phases. Turning these phases into reusable, replay-checked macros is a coherent way to reduce repeated tool interaction, and the macro design follows standard software-engineering practice.

  2. The experimental setup is controlled. The temporally separated benchmark, frozen revisions, hidden developer tests, and repository-clustered inference give the evaluation more structure than is common in this area. The authors also include a fresh repeat and a third generation, and report model-level results alongside pooled numbers.

Critical weaknesses
  1. The paper claims efficiency gains, but the primary metric, model-mediated tool decisions, counts a macro call as one decision while treating its internal subprocesses separately. This makes the reduction partly a matter of accounting. The paper also does not report token overhead, even though context length and workflow differ across conditions. On DeepSeek, cost is higher under SKILLZIP and the decision count does not decrease. Pooling dollar costs across models with different pricing makes the cost-neutral conclusion difficult to interpret. As a result, the efficiency claim is not fully supported by the current evidence.

  2. The contribution is framed as a experience-learning mechanism, but the macros, typed slots, contracts, and executable bodies are all author-provided. The compiler does not synthesize macro bodies or discover new template families; source traces only determine support and admission. The two admitted macros reflect standard software-engineering practice, and there is no baseline with the same hand-written macros but without experience gating. Since macros and primitive tools remain freely mixed at inference, the method can also be read as a consolidated tool with coarser action granularity. The authors note that the primary matrix measures the author-specified interface rather than the causal effect of gating, which makes the broader framing harder to sustain.

Questions for authors
  1. Could you report token usage and normalized cost per task for each condition? The current decision metric counts a macro call as one decision but does not account for internal subprocesses, so it is unclear whether the efficiency gain is real or a change in accounting.

  2. According to Table~3, On DeepSeek, SKILLZIP shows higher cost and no reduction in decisions. Could you explain this model difference and clarify whether the pooled result is the right basis for the efficiency claim?

  3. The macros, slots, contracts, and bodies are all author-provided. What does experience gating contribute beyond admission?

Reviewer_DngcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper contributes an interesting idea that can improve the efficiency of LLM agents. It identifies the repetitive tool call sequences for coding tasks, and abstracts those tool call seqs as a 'macro' action. Next time the agent can call the sequence by directly specifying typed args in the 'Marco' action. It aims to reduce the number of steps of the agents.

Critical strengths

The paper targets a very important problem in LLM agents, and the solution is smart and neat. It demonstrates a simple yet generalizable way to summarize procedure memories.

Critical weaknesses

However, I have a few concerns about the empirical results.

  1. In the abstract, there are discussions like "Across 40 model–task pairs, S KILL Z IP matches observed no-memory completion at 24/40 (paired 95% CI: −7.5 to +7.5 points) while reducing model-mediated tool decisions by 23.1% (repository-clustered 95% CI: 14.1–31.2% fewer; exact signflip p = .0059). The pooled effect combines a 40.6% Luna reduction with a neutral 1.1% DeepSeek increase; primary cost is likewise neutral (+0.4%).". I am not sure I understand how to interpret the results. Do the results show that the approach is not working on DeepSeek? If that is the case, how to justify the effectiveness/generalization of the proposed technique? Why it's not working on DeepSeek? For table 3, why is the reduction in steps only less than 3 steps? Is this important saving?
  2. The tools are evaluated on author-curated benchmarks. Why not use more standard coding benchmarks such as SWE-Bench and its harder variants?
  3. I think some important baselines are missing. For example, does the step reduction consider parallel tool calls where a step can contain multiple tool calls? What about CodeAct that let agents write code instead of individual tool calls?
Questions for authors

Please see the above discussion.

Reviewer_FX54OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SKILLZIP, a training‑free compiler for tool‑using agents, which compiles past successful execution traces into parameterized typed macro‑actions. Different from prior work that stores past experience as plain text in context windows, SKILLZIP mines recurring subsequences from trajectories, instantiates macro candidates from author‑provided phase templates, and admits macros to the agent’s action menu only after sandbox replay verifies their observable contract. The authors build a temporally‑separated benchmark suite with 10 source tasks and 20 held‑out tasks collected from 10 public Go repositories. A full factorial evaluation is conducted with two agent models under four conditions: no‑memory, textual workflow, retrieved raw action sequences, and SKILLZIP.

Critical strengths

This paper treats experience as executable interface instead of textual context. SKILLZIP compiles recurring workflows into executable macros collapsing multiple primitive choices into one model‑mediated decision. This conceptual distinction is valuable for tool‑agent research. In experiments, temporally‑split benchmark, equally‑sourced baselines, repository‑aware statistical tests and multiple audit studies mitigate confounders. Empirically, SKILLZIP reduces model‑mediated tool decisions by 23.1% while retaining the same task success rate as the no‑memory baseline. All research artifacts are released for reproducibility.

Critical weaknesses
  1. While the paper validates the value of replay‑gated macro interfaces, the actual macro bodies (INSPECT / CHECK) are hand‑written by authors. Real‑world deployment requires domain‑specific human template engineering, limiting generality. In addition, the system cannot automatically discover brand‑new operation templates purely from traces.

  2. Large decision reduction is observed on Luna while DeepSeek shows marginal benefit. Only empirical outcomes are reported, and there is no ablation or diagnosis whether this stems from prompt‑following differences or intrinsic model bias toward macro usage.

  3. Lack of direct quantitative comparison against peer skill‑learning agent papers. Related‑work discussion remains conceptual without benchmark‑aligned comparisons.

Questions for authors
  1. Macro templates are fully hand‑authored in this submission. Could you clarify: without manually written templates, is the SKILLZIP framework itself capable of synthesizing complete executable macro bodies purely from raw traces? How can it reduce dependency on human‑written templates?

  2. There exists large performance gap between Luna and DeepSeek. Could you perform further ablation to diagnose?

  3. Why do you not evaluate on established public agent benchmarks such as SWE‑bench?

  4. When a macro execution fails mid‑run and agent falls back to primitive tools, have you analyzed agent behaviors under macro‑failure scenarios?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.