acceptodds
Under review as a conference paper at ICLR 2027

Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity

Abstract

Modern language-model agents are built around the *agent loop*, where the LLM is placed in an environment exposing a set of tools, and the LLM has full control over the workflow by alternating between tool calls and observing their output. However, certain workflows currently require additional engineering beyond the agent loop itself, such as memory systems and self-improving systems. We built an LLM agent framework, JAZ, to explore the extent to which a minimal harness that is little more than the agent loop itself can accomplish tasks these specialized systems are built for. JAZ exposes a single LLM-based primitive `invoke` and provides a set of built-in hooks that allow the programmer to apply constraints and monitoring. Generalizing existing code-mode agent loops, `invoke` is the simplest loop that satisfies two defining properties: (1) the LLM can write arbitrary executable code that can include recursive `invoke`; (2) everything visible to the LLM — all inputs to `invoke` as well as its interaction history with the code environment — are variables in the code environment. We motivate our design from first principles, viewing `invoke` as a language primitive representing a function whose implementation is provided at runtime by an LLM every time it is called. To validate the design of our core `invoke` primitive, we evaluate `invoke` — with only prompting, no manually designed tools, harness, or external systems (e.g., memory or the file system) — on workflows traditionally implemented through specialized external harnesses. On long-horizon workflows requiring recall beyond the context window, JAZ `invoke` outperforms Letta (MemGPT) by 8% at half its cost on the recall-heavy portion of StuLife. On continual self-improvement, JAZ `invoke` outperforms ACE by 4% at a lower cost on AppWorld.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agent frameworks, self-improvement, long-horizon, multi-agent systems, meta-agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_mpYPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces JAZ, a minimalist agent framework built around a unified language-model primitive. The model can generate and execute programs in a code environment and recursively call invoke to create subagents. The key design choice is to expose inputs, prompts, and interaction histories as programmatically accessible variables, allowing agents to retrieve, combine, and pass these objects directly. The framework is evaluated through two case studies, i.e., StuLife and AppWorld. The authors report performance or cost advantages over CodeAct, CodeAct+subagents, Letta, and ACE, arguing that a minimal agent loop with prompting can realize capabilities commonly implemented through specialized external harnesses.

Critical strengths
  1. Exposing prompts and interaction histories is a good design, since program objects can reduce information loss caused by asking the model to regenerate their contents. It also makes it easier for a parent agent to inspect a subagent's execution.

  2. The long-range recall results are promising. In Table 1, JAZ achieves a 69.9% pass rate on the far-recall subset of StuLife, compared with 61.8% for Letta, while reducing the cost per complete run from $42.1 to $18.3. These results suggest that programmatic access to complete histories is worth investigating under the evaluated setting.

Critical weaknesses
  1. It is hard to tell the difference from existing general-purpose agent harnesses and the resulting research contribution are not sufficiently established. Code execution, subagent invocation, and context handoff are not unique to this work. The paper should clarify what its design adds to CodeAct/RLM and modern general-purpose harnesses such as Claude Code and Codex, particularly in history access, context inheritance, and workflow orchestration. The clearest addition is exposing prompts and histories as code variables, but the paper does not establish whether this interface difference enables capabilities that existing approaches cannot readily support, or yields consistent efficiency advantages.

  2. Although the tables contain multiple methods, several are variants of the same framework family, leaving limited coverage of independent, complete general-purpose harnesses. The paper neither directly evaluates modern harnesses such as Claude Code and Codex nor provides a representative baseline with comparable subagent, history-access, and context-management capabilities. It therefore remains unclear whether JAZ offers performance or cost advantages over existing complete systems.

  3. This work shows that automatically exposing coding related objects is helpful, but leaves a simpler question unanswered: could a conventional agent achieve similar results if equipped with automatic logging and log search? Adding this baseline, together with separate ablations that expose only prompts or only histories, would clarify the contribution of each design choice.

Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Rebuttal 1/3

We appreciate that the reviewer likes our design and finds our results promising. We respond to your individual comments below.

W1: contribution/novelty

The paper should clarify what its design adds to CodeAct/RLM and modern general-purpose harnesses such as Claude Code and Codex, particularly in history access, context inheritance, and workflow orchestration.

As an agent loop primitive, the difference between JAZ invoke and CodeAct+subagents is that everything the LLM sees can be accessed as a programmatic variable, including the user prompt and the REPL history. In CodeAct+subagents implementations (e.g. smolagents, RLM), there's no variable containing the REPL history, and the user prompt isn't available as a variable either (task in smolagents and root_prompt in RLM are only shown to the agent in its context). The paper currently describes this at L22 in the abstract, L61-64 in the introduction, and re-emphasize it in L146-149 when introducing the invoke primitive. We are happy to revise the presentation to improve the clarity.

The difference from complete systems like Claude Code and Codex is that JAZ invoke is an agent loop primitive instead of a complete system, yet exhibits capabilities that traditionally only complete systems do. In our experiments, invoke doesn't get external engineering beyond the agent loop itself and a prompt ("prompt-only setup" as defined in Appendix D.1), whereas complete general-purpose harnesses like Claude Code and Codex are terminal agents with a file system and engineered context-management strategies (compaction, dynamic skill/tool loading, etc.). JAZ invoke doesn't have any of this and can thus be a bare agent primitive inside an agent framework while still offering capabilities like long-range recall and self-improvement without needing any of this extra setup and engineering. (Also see our response to W2 below for an empirical comparison with Codex.)

In terms of workflow orchestration, JAZ invoke provides two novel uses of subagents: a) tail-recursive delegation (L258-293, Section 4.1) emulates compaction while preserving history, and b) continual self-improvement (L373-406, Section 4.2). Both uses are enabled by the presence of history and user prompt as programmatic variables.

The clearest addition is exposing prompts and histories as code variables, but the paper does not establish whether this interface difference enables capabilities that existing approaches cannot readily support, or yields consistent efficiency advantages.

Our paper does in fact establish capabilities of the invoke primitive that previous agent loop primitives do not support, as well as performance and cost advantages over specialized systems specifically built with these capabilities. In particular, Tables 1 and 2 (Section 4) report comparisons with CodeAct+subagents, which isolates the capabilities and advantages from exposing prompts and histories as code variables in the agent loop. JAZ invoke outperforms CodeAct+subagents at a similar cost since the history variable is needed for recall (StuLife long-horizon) and for root meta-agent's observability into solver subagents (AppWorld self-improvement). The tables also report comparisons with existing approaches for long-horizon tasks and self-improvement, namely Letta (MemGPT) and ACE, respectively. JAZ invoke outperforms both Letta and ACE at a substantially lower cost, and the analysis that follows identifies the reason to be the flexibility the agent has due to the expressivity of invoke.

We additionally compare with Codex and find that JAZ invoke outperforms it at a lower cost due to invoke's minimal interface -- see our response to W2 below.

Code execution, subagent invocation, and context handoff are not unique to this work.

The reviewer is right that code execution and subagent invocation are not claimed novelties of JAZ invoke. The novelty that we do claim is an agent framework whose core agent primitive can perform long-horizon and self-improvement workflows with prompting only -- there's no need to build the extra harnessing traditionally needed for such workflows (e.g. compaction, tools for accessing a file system or memory system).

We'd also like to point out that the empirical finding that such a minimalist agent can perform such tasks can be considered novel on its own. There has been prior work that was published where the main novelty was the finding that an existing method was able to perform tasks people did not know it could. An example is RLMs. An RLM is a code-mode agent loop with subagents and allowed programmatic inputs -- all three existed at the time and were present in frameworks like smolagents. Its novelty was its finding that that was all that was needed for many long-context tasks.

AuthorsOpenReview ↗

Rebuttal 2/3

W2: comparison with a general-purpose complete system

Although the tables contain multiple methods, several are variants of the same framework family.

Our experiments compare JAZ invoke with two general agent loops CodeAct and CodeAct+subagents under a prompt-only setting, as well as an independent, complete specialized harnesses Letta (Table 1) and ACE (Table 2). Two variants of CodeAct+subagents (Table 1) and CodeAct (Table 2) were provided for better control of confounds, as the existing implementations (smolagents and AppWorld official) differed from JAZ in additional ways that unfairly reduced their performance, so we included results of JAZ reimplementations for a controlled comparison. Appendices D.2.2 and D.3.2 provide further details.

The paper neither directly evaluates modern harnesses such as Claude Code and Codex nor provides a representative baseline with comparable subagent, history-access, and context-management capabilities.

We provide comparisons with Codex below, since our experiments used OpenAI models. Codex is a complete system serving as a representative baseline with subagents and compaction. It supports either traditional tool calling or code-mode, so we set it to use code-mode to reduce confounds. To give it comparable history access, we put the path to its session log in its prompt.

Under this setup, the main difference between Codex and JAZ invoke is that the former is a considerably more complex harness than JAZ invoke, which is a bare agent loop derived from functional invoke (Section 2). For example, the Codex setup requires registering tools on an MCP server and generating MCP specs, setting up a Docker container for a sandboxed file system workspace, finding the path to the session log to tell the agent, etc. On the other hand, the JAZ setup simply involves passing the Python functions defining the tools as arguments directly to invoke, and writing a prompt.

Despite JAZ invoke's simplicity, it outperforms Codex at a lower cost:

StuLifePass, all (%)Score, all (%)Pass, far recall (%)Score, far recall (%)Cost ($)
Codex72.6 ±\pm 0.979.9 ±\pm 0.871.2 ±\pm 1.373.0 ±\pm 1.218.2 ±\pm 0.6
JAZ invoke (ours)*74.1 ±\pm 0.683.3 ±\pm 0.772.5 ±\pm 1.376.0 ±\pm 1.415.6 ±\pm 0.6

*Results differ slightly from paper b/c we ran more attempts totaling 6 after the submission deadline, and a prompt bug in JAZ invoke was fixed.

AppWorldTGC (%)SGC (%)Cost ($)Meta $Solver $
Codex59.0 ±\pm 3.932.0 ±\pm 3.423.8 ±\pm 1.12.9 ±\pm 0.320.9 ±\pm 0.9
JAZ invoke (ours)74.2 ±\pm 2.151.1 ±\pm 3.720.9 ±\pm 3.79.9 ±\pm 3.810.9 ±\pm 0.6

In analyzing traces, we found that JAZ invoke's minimal interface was the reason it outperformed Codex. Here we present two failure modes of Codex that JAZ invoke doesn't run into due to its design.

One is silent tool failures, caused by the fact that tools are translated to MCP for Codex before being placed in its JavaScript REPL. MCP tools return an object with an isError field denoting whether the tool failed, rather than actually throwing inside Codex's JavaScript REPL. As a result, the agent sometimes mistakenly thought it has succeeded in when it hasn't. In JAZ, tools are just regular Python functions (Section 2, L104-106). As any other input object into JAZ invoke, it simply becomes an object in the agent's REPL, with no transformation in between like registering as an MCP tool. Thus, the tool raises when it fails so the agent sees the failure early.

Another failure mode is noise in history. Codex's logs are less structured and more complicated, and the entire session log includes both the compacted and current history. In StuLife long-range recall, Codex retrieves more irrelevant results such as the task it is currently working on. JAZ's subagent delegation workflow automatically separates prev_history -- the history that came before the current session -- and __history__, the current session, allowing the agent to only search the history that has exited its context, avoiding noise such as the current task text. In AppWorld self-improvement, logs are truncated more often, making it harder for the meta-agent to identify root cause of subagent failures. On the other hand, JAZ invoke's __history__ by definition contains only what the LLM actually sees -- the inputs and outputs to the REPL (Property 2, L140-144, Section 2).

The reason Codex is more costly than StuLife on AppWorld is the fact that AppWorld's apis object that hides individual tool specs can't be directly put into Codex agent's environment. In JAZ, it's just a regular object with a docstring. In Codex, it has to become individual MCP tools, blowing GPT-5.4-nano's context as the OpenAI API for GPT-5.4-nano doesn't support tool search.

AuthorsOpenReview ↗

Rebuttal 3/3

W3: ablations

could a conventional agent achieve similar results if equipped with automatic logging and log search?

See our Codex results above.

separate ablations that expose only prompts or only histories

Ablation results for StuLife long-horizon are provided below.

MethodPass, all (%)Score, all (%)Pass, far recall (%)Score, far recall (%)Cost ($)
JAZ invoke (ours)*74.1 ±\pm 0.683.3 ±\pm 0.772.5 ±\pm 1.376.0 ±\pm 1.415.6 ±\pm 0.6
− prompt vars.65.4 ±\pm 1.374.4 ±\pm 1.539.8 ±\pm 4.142.6 ±\pm 4.313.0 ±\pm 0.8
− history72.2 ±\pm 0.781.4 ±\pm 0.668.2 ±\pm 2.771.7 ±\pm 2.616.1 ±\pm 0.3
CodeAct+subagents*63.3 ±\pm 2.071.8 ±\pm 2.238.5 ±\pm 4.741.1 ±\pm 5.213.8 ±\pm 0.7

*Results differ slightly from paper b/c we ran more attempts totalling 6 after the submission deadline, and a prompt bug in JAZ invoke was fixed.

We find that removing prompt variables alone can reduce the far recall performance to the level of CodeAct+subagents due to the loss of instructions across delegation boundaries. Removing the history variable has a lesser effect under our prompt, which makes up for the lack of a history variable by telling the agent to maintain its own. Despite that, the agent was inconsistent and sometimes forgot to append to it, causing later tasks to be unsolvable as they depended on information that the agent forgot to store to the history it was maintaining by itself.

Ablation results for AppWorld self-improvement are provided below.

MethodTGC (%)SGC (%)Cost ($)
JAZ invoke (ours)74.2 ±\pm 2.151.1 ±\pm 3.720.9 ±\pm 3.7
− prompt vars.74.8 ±\pm 0.653.7 ±\pm 0.219.1 ±\pm 2.1
− history72.3 ±\pm 1.448.1 ±\pm 2.823.1 ±\pm 2.4
CodeAct+subagents71.1 ±\pm 1.347.6 ±\pm 2.121.6 ±\pm 1.8

As expected, "− prompt vars." performs level with JAZ invoke as the self-improvement workflow doesn't rely on programmatic reference to the user prompt. For the same reason, "− prompt vars." performs level with CodeAct+subagents. And the two arms without the history variable ("− history" and "CodeAct+subagents") perform worse than the two arms with it ("JAZ invoke" and "− history"), for the reason discussed in the paper (Section 4.2, L441-446), i.e., the subagent hand-building its history for the meta-agent to inspect builds it unreliably and lossily.


Again, we greatly appreciate the reviewer for their time and constructive feedback, and we look forward to hearing back from you.

Reviewer_cgXtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JAZ, a minimalist agent framework built around invoke, an LLM-based primitive supporting recursive agent calls and programmatic access to model inputs and interaction histories. The authors argue that this design enables long-horizon memory management and continual self-improvement without specialized external harnesses. JAZ also provides hooks for monitoring and resource control. Experiments on StuLife and AppWorld show that JAZ outperforms CodeAct-based agents and specialized systems such as Letta and ACE at lower costs than these specialized systems.

Critical strengths
  1. A Simple and Generalizable Agent Abstraction. By exposing model inputs and interaction histories as programmable variables, JAZ provides a unified mechanism for context management, recursive delegation, and self-improvement. This offers a potentially useful alternative to task-specific harness designs.
  2. Strong Empirical Results on Long-Horizon Tasks. JAZ demonstrates substantial improvements in long-range recall over both CodeAct with subagents and Letta, while maintaining lower costs than the latter. The controlled implementation of CodeAct-based baselines further strengthens the evidence for the proposed abstraction's practical value.
Critical weaknesses
  1. Limited Methodological Novelty. JAZ builds upon established capabilities in CodeAct and RLMs, with its main extension being programmatic access to prompts and interaction histories. The paper does not sufficiently demonstrate why this abstraction offers fundamental advantages over simpler history-access mechanisms or constitutes a substantial advance in expressivity.
  2. Insufficient Support for the Central Claims. The paper argues that specialized harnesses can be replaced by a minimal agent loop. However, its memory and self-improvement workflows still rely on carefully designed prompts, hooks, and explicit workflow guidance. This raises the question of whether JAZ eliminates specialized harness logic or merely relocates it.
  3. Experimental Limitations and Unclear Attribution. The ablations remove prompt and history access simultaneously, obscuring their individual contributions. In AppWorld, ACE learns from only 42 tasks while JAZ continuously adapts across 417 tasks. Moreover, the gains over CodeAct+subagents are modest and variable, with limited analysis of self-improvement dynamics and generalization to unseen tasks.
Questions for authors
  1. What fundamental advantages does invoke offer over CodeAct or RLMs equipped with a simple history-access interface? Could such a baseline help clarify the methodological contribution?
  2. To what extent do the demonstrated capabilities emerge from the invoke abstraction rather than the detailed workflow instructions and hooks? How does performance change with less prescriptive guidance?
  3. Could the authors provide separate ablations for programmatic access to prompts and interaction histories to identify their individual contributions?
  4. Can the AppWorld comparisons be conducted under matched learning opportunities and computational budgets, particularly for ACE? Could learning curves and evaluation on unseen tasks further substantiate the claimed continual self-improvement?
8 comments
AuthorsOpenReview ↗

Rebuttal 1/5

We're glad the reviewer liked the idea and empirical results of our paper. We respond to your comments and questions below.

W1/Q1: contribution/novelty

Limited Methodological Novelty. JAZ builds upon established capabilities in CodeAct and RLMs, with its main extension being programmatic access to prompts and interaction histories.

JAZ provides the following novel contributions:

  1. (conceptual) The minimalist programming model invoke (Section 2). Practical contributions that arise from this programming model include:

1a. (methodological) Programmatic access to everything the LLM sees. In particular, programmatic access to the user prompt and history distinguishes it from existing implementations of CodeAct+subagents like smolagents and RLMs. The reviewer correctly identifies this contribution.

1b. (methodological) Uniformly treating all agent inputs (prompt, tools, etc.) as objects seeding the REPL, along with an LLM-friendly version of Python’s str() for displaying any Python object to the LLM. This lets JAZ invoke outperform Codex even when Codex is given its session logs to match history access: 76% vs 73% far-recall score on StuLife, and 74% vs 59% on AppWorld. The MCP layer that stands between agent inputs and the agent loop is to blame, since it removes a lot of the structure and flexibility you get from arbitrary Python objects as tools. Our response to Reviewer mpYP's W2 explains this in further detail.

1c. (programming framework) An agent framework, JAZ, where the core out-of-the-box primitive is more capable than the core primitives (e.g. CodeAct+subagents) of existing frameworks. With prompting only and no additional setup (file system, compaction, etc.), the primitive can perform long-horizon and self-improvement workflows that have traditionally required these external mechanisms.

  1. (empirical) The finding that such a minimalist primitive without external mechanisms (e.g. compaction, file system, etc.) can outperform systems with specialized components on tasks those systems are designed for: long-horizon requiring long-range recall and continual self-improvement.

We will revise our introduction section to clarify these contributions.

We'd also like to point out that similar prior work has been published without methodological novelty. For example, the RLM (accepted to NeurIPS 2026) is a code-mode agent with subagents supporting programmatic inputs to the agent -- frameworks at the time already provided all three (e.g., LangChain's PythonREPL tool, smolagents). However, the work was accepted due to its novel finding that a minimalist agent (CodeAct+subagents) can outperform long-context LLMs and coding agents on long-context tasks. JAZ's empirical contribution is a similar one, with "long-context" changed to "long-horizon requiring long-range recall" and "continual self-improvement", and with a focus on agentic tasks.

What fundamental advantages does invoke offer over CodeAct or RLMs equipped with a simple history-access interface? Could such a baseline help clarify the methodological contribution?

Could the reviewer clarify what they mean exactly by a “simple history-access interface”? Our existing experimental results include the following results that may be close to what you’re asking for:

  • "− prompt vars." ablation, aka RLM/CodeAct+subagents with programmatic access to history but no programmatic access to user prompt. Results are given in our response to Q3 below, showing that programmatic access to the user prompt is important for long-horizon under the prompt-only setup, but not for self-improvement. (History access is what's important for self-improvement.)
  • Letta (MemGPT) is a parallel tool calling agent with a memory system, which exposes a single method conversation_search in its interface. (It also has a core memory interface for reusable facts, but that's not useful for StuLife.) Its far-recall pass rate is 61.8%, which underperforms the reported 69.9% for JAZ (Table 1). The reason is attributed to the limited search interface conversation_search -- our analysis in the paper (L354-364, Section 4.1) provides further details.
AuthorsOpenReview ↗

Rebuttal 2/5

W2/Q2: role of prompting

The paper argues that specialized harnesses can be replaced by a minimal agent loop.

Our claim is that specialized harnesses (long-horizon, self-improvement) can be replaced by a minimal agent loop with prompting. We tried to make this clear in the paper (Abstract L26-27 "with only prompting", Introduction L71 "with little more than pure prompting", Introduction L80 "prompting a minimal agent loop", Introduction L90 "with prompting"), but if the reivewer finds there remain wordings in the paper that are misleading/overclaiming, we're happy to revise.

However, its memory and self-improvement workflows still rely on carefully designed prompts, hooks, and explicit workflow guidance.

Prompting is standard practice in LLM agent work, so we are following the established standard. For example, the RLM -- a similar work to ours accepted to NeurIPS 2026) -- uses a prompt with 3 in-context examples of the code the agent should write to decompose the context. Our long-horizon search prompt and delegation prompt each show one in-context example (Appendix F.1), and our self-improvement prompt has no example (Appendix F.2).

While our prompts do guide the agent, a decent portion is the task and the information needed to perform it, rather than instructions on how to perform it. For example, in our self-improvement prompt (Appendix F.2), the "Subagent's context" section describes what the subagent sees. Of course, there's also guidance, a lot of which is often general principles. For example, the "Subagent prompt" and "Subagent tools" sections describe general principles for what kind of prompt/tool addition is appropriate (e.g. it should be generalizable).

Regarding hooks, they were mainly used for evaluation infrastructure (logging, cost tracking, etc.). The only exception is that the ContextWindowWarning hook was used to deliver the delegation prompt as a message when the context window is close to full.

This raises the question of whether JAZ eliminates specialized harness logic or merely relocates it.

We believe that JAZ does not merely relocate specialized harness logic -- we saw concrete gains in performance and cost relative to specialized harnesses: +8% far-recall pass rate at half the cost than Letta on StuLife, and +4% at lower cost than ACE on AppWorld (Tables 1 and 2). Examining the traces we find that this performance gain can be directly attributed to the flexibility that JAZ offers the agent, and the prompt does not merely constrain the agent to follow the logic of existing harnesses. While memory systems provide a restrictive interface like "search", JAZ lets the agent interact with its history with full programmatic expressivity. While existing self-improvement methods like ACE and GEPA have a fixed batch size, we saw our agent adaptively decide batch size based on how well it's performing. Without explicit prompting to do so, the agent has halved its batch size after a change in order to better test it, and it often dispatches all remaining tasks (sometimes over 100 of them) to a single batch after it decided by itself that it has converged. This directly contributed to fewer meta-agent LLM calls and hence the lower cost that we observe compared to human-engineered self-improvement methods like ACE.

AuthorsOpenReview ↗

Rebuttal 3/5

(W2/Q2 continued)

To what extent do the demonstrated capabilities emerge from the invoke abstraction rather than the detailed workflow instructions and hooks?

In our main experiments that compare invoke with CodeAct+subagents, we made sure that CodeAct+subagents receives no less prompting than JAZ invoke to isolate the effect of the invoke abstraction (see our prompts in our anonymous code repo). Appendix D.1 describes our approach to prompting to ensure that comparisons isolate harness differences rather than being confounded by differences in prompting. Thus, our results where JAZ invoke outperforms CodeAct+subagents (69.9% vs 32.0% far-recall pass rate on StuLife and 74.2% vs 71.1% on AppWorld) are due to the invoke abstraction rather than the workflow instructions. (As mentioned above, hooks were used for eval infrastructure and prompting only.)

How does performance change with less prescriptive guidance?

We simplified our history search and self-improvement prompts to remove most of the guidance but kept the "what" and the information the agent needed to perform the task. The simplified history search prompt is 950 characters long compared to the original 2227 (57% reduction); it removed the code example and simplified guidance. The simplified self-improvement prompt is 1835 characters compared to the original 4561 (60% reduction); it removed mentions of prompt/tools and their corresponding sections (they're just "inputs") and mentions of batching, and only the "Subagent's context" and "Observability" sections were kept to provide information on what the subagent sees and the format of the subagent's __history__ that it should return to the root agent.

The results are as follows (the "less guidance" rows).

StuLifePass, all (%)Score, all (%)Pass, far recall (%)Score, far recall (%)Cost ($)
JAZ invoke (ours)*74.1 ±\pm 0.683.3 ±\pm 0.772.5 ±\pm 1.376.0 ±\pm 1.415.6 ±\pm 0.6
less guidance73.4 ±\pm 0.583.8 ±\pm 0.668.6 ±\pm 1.772.8 ±\pm 1.815.4 ±\pm 0.6
CodeAct+subagents*63.3 ±\pm 2.071.8 ±\pm 2.238.5 ±\pm 4.741.1 ±\pm 5.213.8 ±\pm 0.7
CodeAct (per-task)52.5 ±\pm 0.459.2 ±\pm 0.324.8 ±\pm 0.925.8 ±\pm 0.94.4 ±\pm 0.1

*Results differ slightly from paper b/c we ran more attempts totalling 6 after the submission deadline, and a prompt bug in JAZ invoke was fixed.

AppWorldTGC (%)SGC (%)Cost ($)Meta $Solver $
JAZ invoke (ours)74.2 ±\pm 2.151.1 ±\pm 3.720.9 ±\pm 3.79.9 ±\pm 3.810.9 ±\pm 0.6
less guidance71.2 ±\pm 1.948.4 ±\pm 2.337.8 ±\pm 13.325.6 ±\pm 12.712.2 ±\pm 0.6
CodeAct+subagents71.1 ±\pm 1.347.6 ±\pm 2.121.6 ±\pm 1.87.3 ±\pm 1.714.3 ±\pm 0.9
CodeAct (per-task)67.5 ±\pm 1.043.4 ±\pm 1.710.2 ±\pm 0.1---10.2 ±\pm 0.1

With less guidance, the agent suffers just a 3% drop. In StuLife long-range recall, this is negligible compared to the gap with CodeAct+subagents. In AppWorld self-improvement, JAZ invoke with less guidance does become similar to CodeAct+subagents with the original more prescriptive guidance; however, it is still meaningfully above the no-improvement baseline, so self-improvement still works. We found that, in self-improvement, the main effect of having more guidance was to reduce variance, especially low outliers. (3 of our "less guidance" self-improvement numbers were around 74% whereas the other 3 were lower (two around 70 and one 64). 3 cost similarly to original JAZ invoke but the other 3 cost significantly more.)

Reviewer_MG7yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an agent framework JAZ built upon a recursive invoke primitive to explore the minimal harness capability compared with agent loop itself. The main idea is that agents can inspect prompts, inputs, and execution histories as program objects, and pass to subagents. Therefore, this interface allows memory management and continual self-improvement to be implemented by prompting rather than orchestration. The authors demonstrate memory management on StuLife and continual self-improvement on AppWorld.

Critical strengths
  • The proposed interface is practical and effective. I like the idea of automatic recording of historical trajectories, which avoids asking the model to reconstruct them from observations. The full history remains stored outside the LLM context, and the cost is encouraging ($18.3/run, versus $42.1 for Letta and $13.1 for CodeAct plus subagents.)

  • The manuscript provides detailed descriptions of how prompt copying, history retrieval, and interpreter behavior affects the performance.

Critical weaknesses
  • My major concern is what does JAZ builds make it beyond ordinary agent infrastructure. Memory management and self-improvement can already be achieved by prompting when agents have the necessary signals such as execution logs and persistent state. What JAZ does is making the workflow easier by treating prompts and histories as program objects. The large performance gap with baseline (CodeAct) comes from reliable logging and state transfer rather than recursive invoke itself. Suppose we provide the prompt access, history access, etc. to CodeAct, rather than asking it to reconstruct the info by itself. What would be the performance? Ablation study is needed here.

  • The comparison on self-improvement does not show a clear advantage. For example, JAZ achieves 74.2 ± 2.1% TGC versus 71.1 ± 1.3% for CodeAct plus subagents. The 3% is modest considering ACE adapts only on 42 tasks but JAZ adapt on all 417 tasks.

Questions for authors
  • As I mentioned above, how does JAZ compare with CodeAct equipped with automatic transcript logging, programmatic history access, and persistent instructions? What benefit exactly come from recursive invoke?

  • Can JAZ and ACE be compared under matched adaptation opportunities?

3 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We appreciate that the reviewer liked our proposed primitive and our detailed analyses. We thank the reviewer for their constructive feedback, and we provide responses to your points below.

W1/Q1: role of recursion

The large performance gap with baseline (CodeAct) comes from reliable logging and state transfer rather than recursive invoke itself. Suppose we provide the prompt access, history access, etc. to CodeAct, rather than asking it to reconstruct the info by itself. What would be the performance? Ablation study is needed here.

As I mentioned above, how does JAZ compare with CodeAct equipped with automatic transcript logging, programmatic history access, and persistent instructions? What benefit exactly come from recursive invoke?

We'd first like to clarify that recursion itself is not a claimed contribution of our work -- in L145 we state "Property 1 [arbitrary code allowing recursive invoke] makes invoke a variant of CodeAct+subagents or RLMs", which are existing methods, and our main experiments directly compare JAZ invoke against CodeAct+subagents/RLMs, controlling for the presence of recursion. Nevertheless, the reviewer's question is a good one, since subagents are not being used in the typical way they're used (aka subtasks) -- in a sense we're proposing new ways of using subagents that now make sense given that JAZ invoke provides programmatic prompt and history access.

Our main research question focuses on the capabilities of an agent loop under a minimal "prompt-only" setup (defined in Section D.1). Under this setup, mechanisms and systems external to the loop (e.g., compaction and file systems) are not available, representing the default of agent primitives in existing agent frameworks. As a result, an agent without recursion/subagents hits its context window limit, and for long task sequences like StuLife and AppWorld, the agent is simply crippled and can't even finish all tasks:

StuLifePass, all (%)Score, all (%)Pass, far recall (%)Score, far recall (%)Cost ($)Tasks attempted
JAZ invoke, 6 runs*74.1 ±\pm 0.683.3 ±\pm 0.772.5 ±\pm 1.376.0 ±\pm 1.415.6 ±\pm 0.61284/1284
− recursion, 6 runs9.4 ±\pm 0.210.2 ±\pm 0.31.2 ±\pm 0.21.7 ±\pm 0.31.0 ±\pm 0.1141–166/1284

*The numbers differ slightly from those in the paper because it was re-run after the submission deadline upon fixing a prompt bug.

AppWorldTGC (%)SGC (%)Cost ($)Tasks attempted
JAZ invoke, 6 runs74.2 ±\pm 2.151.1 ±\pm 3.720.9 ±\pm 3.7417
− recursion, 6 runs1.8 ±\pm 0.20.0 ±\pm 0.00.7 ±\pm 0.015–20

JAZ's key novel insight regarding subagents is that subagents/recursion, while traditionally used for well-defined subtasks, can also be used for long-horizon tasks requiring long-range recall and learning across a sequence of tasks, provided that everything the LLM sees (including user prompt and history) is a programmatic variable. Recursive invoke itself is not as effective, as seen in the comparisons with CodeAct+subagents in Tables 1 and 2 and the analyses that follow. On the other hand, removing recursive invoke while keeping the programmatic variables removes the agent's sole means of context management under a prompt-only setup.

  • In StuLife long-horizon, a subagent is launched near the context window limit to continue the remaining tasks by emulating what could be thought of as "lossless compaction", aka compaction while preserving the full uncompacted history behind an interface. The presence of everything as programmatic variables enables this use of subagents, as the agent can losslessly pass its user prompt and history into the subagent for it to continue its work. More discussion can be found at L251-293 and L432-451 of the submission -- L261 shows the tail-recursive call to invoke.
  • In AppWorld self-improvement, the root agent acts as the meta-agent that optimizes prompts and tools for solver subagents. The presence of history as a variable enables this use of subagents, as the subagent can losslessly pass its history to the parent meta-agent for observability. More discussion can be found at L373-405 and L432-450 of the paper -- L398 shows a recursive call to invoke.
AuthorsOpenReview ↗

Rebuttal 2/2

W2/Q2: self-improvement comparisons

The comparison on self-improvement does not show a clear advantage. For example, JAZ achieves 74.2 ± 2.1% TGC versus 71.1 ± 1.3% for CodeAct plus subagents.

If we take into account CodeAct's baseline score of (67.5 ±\pm 1.2)%, JAZ's improvement over the baseline (6.7%) almost doubles that of CodeAct+subagents (3.6%).

This picture where __history__ substantially improves the self-improvement gain holds as we take into account the ablations that we ran (remove only programmatic access to user prompt or remove only __history__). Then our 12 runs with __history__ average (74.5 ±\pm 1.0)% (gain over baseline is 7.0%) and our 12 runs without __history__ average (71.7 ±\pm 0.9)% (gain over baseline is 4.2%). Having __history__ gives an advantage of (2.8 ±\pm 1.4)%, and the rank-sum test gives a one-sided p-value of 0.009. If requested, we can provide the raw data points so that the reviewer can reproduce those results.

The 3% is modest considering ACE adapts only on 42 tasks but JAZ adapt on all 417 tasks.

Can JAZ and ACE be compared under matched adaptation opportunities?

The reviewer appears to have accidentally quoted the number 3% from the comparison with CodeAct+subagents. ACE is (69.9 ±\pm 1.4)%, which JAZ outperforms by 4.3% at a lower cost (\20.9vs20.9 vs \\30.6). Nevertheless, we agree with the reviewer that, although this comparison is more cost-controlled, it raises the question of whether ACE potentially does much better than JAZ when allowed to adapt on all tasks. As a result, we also ran two variants of ACE on all tasks: the original where reflector and curator calls are made for every single task, and a cheaper variant that supports batching (not part of the original method). The results are given in the following table, extending Table 2:

MethodPrompt-only?TGC (%)SGC (%)Cost ($)Meta $Solver $
ACE, train 10%✗69.9 ±\pm 1.447.1 ±\pm 1.330.6 ±\pm 0.517.1 ±\pm 0.313.5 ±\pm 0.3
ACE, train all, unbatched (1 run)✗74.348.2323.5309.314.2
ACE, train all, batched✗74.9 ±\pm 0.952.2 ±\pm 1.4116.1 ±\pm 0.9102.2 ±\pm 0.713.9 ±\pm 0.3
JAZ invoke (ours)✓74.2 ±\pm 2.151.1 ±\pm 3.720.9 ±\pm 3.79.9 ±\pm 3.810.9 ±\pm 0.6

Here, we find that JAZ performs similarly while being 5x to 16x cheaper. Thus, the advantage of JAZ over ACE on AppWorld largely shifts to a cost advantage once we allow ACE to adapt on all tasks.


Again we thank the reviewer for their time helping us improve our work.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer MG7y,

Given that the discussion period is nearing its end, we'd greatly appreciated it if you could take the time to go through our rebuttal. We'd be happy to answer any further feedback and questions you may have. Thank you!

Best,

Authors

Reviewer_1cM2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes JAZ, a minimal agent framework built around a single language primitive called invoke. In addition to running Python code and calling recursive subagents in a REPL, invoke exposes all prompt inputs and the REPL interaction history as variables inside the code environment. With this interface and prompting, the agent can handle long-horizon recall and continual self-improvement without specialized external memory or self-improvement harnesses.

Critical strengths

Simple and practical framework design: Making prompt inputs and history directly accessible as REPL variables is a very clean design. It solves two real issues in existing CodeAct/RLM setups: passing long prompts to subagents without manual string copying, and returning subagent trajectories to a meta-agent directly.

Solid baseline implementation: The authors reimplement baselines inside JAZ for controlled comparisons, fix prompt bugs in the official AppWorld baseline, control cache hit rates for cost calculation and report all individual runs in the appendix.

Critical weaknesses

Prompt variable forwarding and history are confounded in the ablation: Property 2 bundles two distinct mechanisms: exposing static prompt inputs (instructions and guidance) as variables, and exposing the dynamic REPL execution history (history) as a variable. Although the paper repeatedly highlights history as the key novelty of invoke (Section 2.2, Appendix A), the CodeAct+subagents ablation removes both history and the prompt variables simultaneously. As shown in Section 4.1 and Appendix D.2.2, CodeAct+subagents can maintain its own output_history list in Python, and primarily fails on StuLife because the nano model loses task instructions while manually copying literal prompt strings across 69 recursive calls. Allowing subagents to pass prompt inputs by reference (or inherit fixed task instructions automatically) is much simpler than exposing dynamic REPL history, yet the ablation does not separate their contributions.

Marginal and noisy gains from history in continual self-improvement: On AppWorld, where recursion depth is capped at 2 (eliminating deep prompt-copying degradation), CodeAct+subagents already achieves most of the self-improvement gain over per-task CodeAct (climbing from 67.5% to 71.1%), while JAZ invoke reaches 74.2 ± 2.1% vs. 71.1 ± 1.3% (Table 2). Across the 6 runs in Table 4, this 3.1pp difference is not statistically significant and exhibits high variance (Run 5 drops to 64.5% at $38.4 cost, below the 67.5% per-task baseline). Furthermore, the ACE baseline is frozen after adapting on only the first 10% (42/417) of tasks (Appendix D.3.2), whereas JAZ invoke relies on a detailed hand-crafted batching and validation meta-prompt (Appendix F.2) across all 417 tasks.

Questions for authors

On StuLife, how does CodeAct+subagents perform if only history is removed (so the agent maintains output_history in Python), while instructions and guidance remain passable by reference (or are automatically inherited by subagents)?

4 comments
AuthorsOpenReview ↗

Rebuttal 1/3

We appreciate the reviewer's thoughtful comments, and we are glad that the reviewer liked our framework and rigorous baseline comparisons. We address your comments below.

W1/Q1: history vs prompt vars. ablations

Prompt variable forwarding and history are confounded in the ablation...

On StuLife, how does CodeAct+subagents perform if only history is removed (so the agent maintains output_history in Python), while instructions and guidance remain passable by reference (or are automatically inherited by subagents)?

The results from ablating the history are shown below in the second row.

MethodPass, all (%)Score, all (%)Pass, far recall (%)Score, far recall (%)Cost ($)
JAZ invoke (ours)*74.1 ±\pm 0.683.3 ±\pm 0.772.5 ±\pm 1.376.0 ±\pm 1.415.6 ±\pm 0.6
− history72.2 ±\pm 0.781.4 ±\pm 0.668.2 ±\pm 2.771.7 ±\pm 2.616.1 ±\pm 0.3
− prompt vars.65.4 ±\pm 1.374.4 ±\pm 1.539.8 ±\pm 4.142.6 ±\pm 4.313.0 ±\pm 0.8
CodeAct+subagents*63.3 ±\pm 2.071.8 ±\pm 2.238.5 ±\pm 4.741.1 ±\pm 5.213.8 ±\pm 0.7

*Results differ slightly from paper b/c we ran more attempts totalling 6 after the submission deadline, and a prompt bug in JAZ invoke was fixed.

As the reviewer correctly points out, the underperformance of CodeAct+subagents is mainly due to removing prompt vars. But removing the history variable has a cost as well, even when we tell the agent to maintain its own. By analyzing the trajectories, we find that the mild decrease in far recall performance is caused by the agent occasionally forgetting to append something it printed to the output_history it is asked to maintain, so a few far-recall tasks became unsolvable because they relied on information that was dropped by the agent. Thus, the __history__ variable makes up for LLM unreliability.

Asking the agent to maintain history by itself also requires it to write more boilerplate code, resulting in token overhead. However this cost increase appears too small to be statistically significant. (The error bars around the 15.6 and 16.1 cost figures overlap.)

AuthorsOpenReview ↗

Rebuttal 2/3

W2: self-improvement comparisons

Marginal and noisy gains from history in continual self-improvement: ... CodeAct+subagents already achieves most of the self-improvement gain over per-task CodeAct (climbing from 67.5% to 71.1%), while JAZ invoke reaches 74.2 ± 2.1% vs. 71.1 ± 1.3% (Table 2).

From these numbers, the self-improvement gain from CodeAct+subagents is 3.6%, and JAZ invoke gains 3.1% compared to CodeAct+subagents (6.7% over the baseline), so CodeAct+subagents actually only achieves 3.6/6.7 = 54% of the self-improvement gain that JAZ invoke achieves.

Marginal and noisy gains from history in continual self-improvement: ... Across the 6 runs in Table 4, this 3.1pp difference is not statistically significant

The 3.1pp increase has a one-sided p-value of 0.090 with the rank-sum test (standard distribution-free comparison), with U=27 (out of 36 pairs). Although not significant at 0.05, it is significant at the less stringent 0.1 level. Also, since we have ablation results above that only ablate prompt vars. or history, we can get two comparisons that measure the effect of __history__: JAZ invoke (74.2 ±\pm 2.1%) over −history (72.3 ±\pm 1.4%) has a p-value of 0.090 (U=27), and −prompt vars. (74.8 ±\pm 0.6%) over CodeAct+subagents (71.1 ±\pm 1.3%) has a p-value of 0.033 (U=30), and the combined p-value using the standard Fisher's method is 0.020, which is significant at the 0.05 level. If requested, we can provide the raw data points so that the reviewer can reproduce those results.

... and exhibits high variance (Run 5 drops to 64.5% at $38.4 cost, below the 67.5% per-task baseline)

High variance and low outliers are not specific to JAZ invoke, but a common feature of current models performing continual/test-time self-improvement where each task is only attempted once. Specialized self-improvement methods like ACE exhibit the same behavior: ACE run 3 is 64.7%. These are often due to a harmful addition the meta-agent makes where it doesn’t realize it was harmful since the agent is unable to A/B-test on a validation set (that would require more than one attempt on the same task). Perhaps future stronger models will become better at attributing failures to mistakes they made themselves. Either way, the low outliers appear to be a more general model reasoning issue and not specific to JAZ invoke. The key observation here -- that JAZ invoke performs better than CodeAct+subagents and ACE on average -- holds. Variance is generally high (also in specialized self-improvement methods), and it is reduced by running 6 independent runs of each method.

Furthermore, the ACE baseline is frozen after adapting on only the first 10% (42/417) of tasks (Appendix D.3.2)...

While the 10% was chosen in an effort for a more cost-controlled comparison, we agree with the reviewer that this raises the question of whether ACE actually does much better than JAZ when allowed to adapt on all tasks. So we also ran two variants of ACE on all tasks: the original ACE (unbatched), and a batched variant (not part of the original method). As shown below, JAZ's performance advantage becomes a cost advantage once ACE adapts on all tasks, where the two methods perform within noise of each other but ACE is 6x to 16x more expensive.

MethodPrompt-only?TGC (%)SGC (%)Cost ($)Meta $Solver $
ACE, adapt 10%✗69.9 ±\pm 1.447.1 ±\pm 1.330.6 ±\pm 0.517.1 ±\pm 0.313.5 ±\pm 0.3
ACE, adapt all, unbatched (1 run)✗74.348.2323.5309.314.2
ACE, adapt all, batched✗74.9 ±\pm 0.952.2 ±\pm 1.4116.1 ±\pm 0.9102.2 ±\pm 0.713.9 ±\pm 0.3
JAZ invoke (ours)✓74.2 ±\pm 2.151.1 ±\pm 3.720.9 ±\pm 3.79.9 ±\pm 3.810.9 ±\pm 0.6
AuthorsOpenReview ↗

Rebuttal 3/3

... whereas JAZ invoke relies on a detailed hand-crafted batching and validation meta-prompt (Appendix F.2) across all 417 tasks.

Prompting is standard practice in LLM agent work, so we are following the established standard. For example, the RLM -- a similar work to ours accepted to NeurIPS 2026) -- uses a prompt with 3 in-context examples of the code the agent should write to decompose the context. Our long-horizon search prompt and delegation prompt each show one in-context example (Appendix F.1), and our self-improvement prompt has no example (Appendix F.2).

While our prompts do guide the agent, a decent portion is the task and the information needed to perform it, rather than instructions on how to perform it. In our self-improvement prompt (Appendix F.2), the "Subagent's context" section describes what the subagent sees. Of course, there's also guidance, a lot of which is often general principles. For example, the "Subagent prompt" and "Subagent tools" sections describe general principles for what kind of prompt/tool addition is appropriate (e.g. it should be generalizable).

Although the reviewer points out batching guidance in the meta-prompt, the lower cost of JAZ is exactly due to the meta-prompt providing only high-level guidance to give the agent flexibility (L458-461, Section 4.2). While existing self-improvement methods typically have a fixed batch size, we saw our agent adaptively decide batch size based on how well it's performing. The agent has halved its batch size after a change in order to better test it, and it often dispatches all remaining tasks (sometimes over 100 of them) to a single batch after it decided by itself that it has converged. Neither was mentioned in the prompt. This directly contributed to more efficient use of meta-agent LLM calls and hence the lower cost that we observe compared to human-engineered self-improvement methods like ACE.


Again we greatly thank the reviewer for their time and constructive feedback.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer 1cM2,

Given that the discussion period is nearing its end, we'd greatly appreciated it if you could take the time to go through our rebuttal. We'd be happy to answer any further feedback and questions you may have. Thank you!

Best,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reviewer-author discussion summary

We thank the AC and reviewers for their time and effort in helping us improve our paper. We summarize the discussion from the discussion period for your convenience.

Reviewer responses to rebuttal

ReviewerRatingResponse
1cM23No reply yet
MG7y3No reply yet
cgXt2 -> 2Three follow-ups, which we addressed by clarifying the relevant parts of our rebuttal. There is no reply to our response yet.
mpYP1 -> 2Score raise. No new concerns were raised.

Strengths identified by the reviewers.

  • Exposing prompts and histories as variables is a clean, practical design. "Making prompt inputs and history directly accessible as REPL variables is a very clean design" (1cM2). "Exposing prompts and interaction histories is a good design, since program objects can reduce information loss caused by asking the model to regenerate their contents" (mpYP). "The proposed interface is practical and effective" (MG7y).
  • Strong long-horizon results at lower cost. "JAZ demonstrates substantial improvements in long-range recall over both CodeAct with subagents and Letta, while maintaining lower costs than the latter" (cgXt). "The cost is encouraging ($18.3/run, versus $42.1 for Letta and $13.1 for CodeAct plus subagents.)" (MG7y).
  • The comparisons are carefully controlled. "The authors reimplement baselines inside JAZ for controlled comparisons, fix prompt bugs in the official AppWorld baseline, control cache hit rates for cost calculation and report all individual runs in the appendix" (1cM2). "The controlled implementation of CodeAct-based baselines further strengthens the evidence for the proposed abstraction's practical value" (cgXt).
  • The analysis is detailed. "The manuscript provides detailed descriptions of how prompt copying, history retrieval, and interpreter behavior affects the performance" (MG7y).

Concerns raised and how we addressed them. StuLife numbers are far-recall pass rates and AppWorld numbers are task goal completion (TGC), unless stated otherwise.

Concern (reviewer)Rebuttal summary
What do prompt access and history access each contribute? (1cM2, cgXt, mpYP)Ran ablations. On StuLife, removing prompt variables drops 72.5% to 39.8% and removing history drops it to 68.2%. On AppWorld, history is what matters (combined p = 0.020).
Can JAZ invoke and ACE be compared with matched adaptation? (1cM2, MG7y, cgXt)Ran ACE adapting on all 417 tasks. ACE then matches JAZ (74.3% and 74.9% vs. 74.2%) but costs 6x to 16x more.
How does JAZ invoke compare with a complete harness (e.g. Codex) with logging and log search? (mpYP)Explained the difference in complexity -- invoke is much simpler; ran Codex evaluations on Codex -- JAZ is slightly ahead on StuLife (72.5% vs. 71.2%, $15.6 vs. $18.2) and well ahead on AppWorld (74.2% vs. 59.0%).
Is the self-improvement gain over CodeAct+subagents real, given the variance? (1cM2, MG7y, cgXt)The 3.1-point gain over CodeAct+subagents has one-sided p = 0.090; the 12 runs with the history variable beat the 12 without by 2.8 points (one-sided p = 0.009).
What does recursive invoke itself add? (MG7y)Clarified that recursion is not a claimed contribution, but our work can be viewed as showcasing a new way of using subagents under a prompt-only setting (aka for long-horizon recall and self-improvement); ablation showing recursion is the means of context management for the long task queues in our evaluations
Do the results come from invoke or from prompts? (cgXt, 1cM2)Noted that prompting guidance is standard practice (gave RLM's prompts as an example); summarized analysis from paper showing that outperforming or costing less than specialized methods is attributed to the flexibility our prompts provide; ran prompt ablations removing all guidance -- only 3-point score decrease (68.6% on StuLife, 71.2% on AppWorld)
Learning curves and eval on unseen tasks? (cgXt)Clarified that the setup of continual self-improvement is already an evaluation on unseen tasks, so Table 2 showing JAZ scoring 74.2% vs CodeAct+subagents' 71.1% is what the reviewer is looking for; added a learning curve showing JAZ learns over the first half of tasks before converging.
What does JAZ add over CodeAct, RLMs and existing harnesses? (cgXt, mpYP)Clarified contributions: compared to the most similar agent loop (CodeAct+subagents), everything the model sees (incl. user prompt and history) are variables; compared to existing methods for long-horizon recall and self-improvement, JAZ invoke is much simpler -- simple enough to be the core agent loop of a general agent framework

We thank the reviewers, whose suggestions strengthened the paper, and the AC for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.