- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces JAZ, a minimalist agent framework built around a unified language-model primitive. The model can generate and execute programs in a code environment and recursively call invoke to create subagents. The key design choice is to expose inputs, prompts, and interaction histories as programmatically accessible variables, allowing agents to retrieve, combine, and pass these objects directly. The framework is evaluated through two case studies, i.e., StuLife and AppWorld. The authors report performance or cost advantages over CodeAct, CodeAct+subagents, Letta, and ACE, arguing that a minimal agent loop with prompting can realize capabilities commonly implemented through specialized external harnesses.
Critical strengths
-
Exposing prompts and interaction histories is a good design, since program objects can reduce information loss caused by asking the model to regenerate their contents. It also makes it easier for a parent agent to inspect a subagent's execution.
-
The long-range recall results are promising. In Table 1, JAZ achieves a 69.9% pass rate on the far-recall subset of StuLife, compared with 61.8% for Letta, while reducing the cost per complete run from $42.1 to $18.3. These results suggest that programmatic access to complete histories is worth investigating under the evaluated setting.
Critical weaknesses
-
It is hard to tell the difference from existing general-purpose agent harnesses and the resulting research contribution are not sufficiently established. Code execution, subagent invocation, and context handoff are not unique to this work. The paper should clarify what its design adds to CodeAct/RLM and modern general-purpose harnesses such as Claude Code and Codex, particularly in history access, context inheritance, and workflow orchestration. The clearest addition is exposing prompts and histories as code variables, but the paper does not establish whether this interface difference enables capabilities that existing approaches cannot readily support, or yields consistent efficiency advantages.
-
Although the tables contain multiple methods, several are variants of the same framework family, leaving limited coverage of independent, complete general-purpose harnesses. The paper neither directly evaluates modern harnesses such as Claude Code and Codex nor provides a representative baseline with comparable subagent, history-access, and context-management capabilities. It therefore remains unclear whether JAZ offers performance or cost advantages over existing complete systems.
-
This work shows that automatically exposing coding related objects is helpful, but leaves a simpler question unanswered: could a conventional agent achieve similar results if equipped with automatic logging and log search? Adding this baseline, together with separate ablations that expose only prompts or only histories, would clarify the contribution of each design choice.
Questions for authors
See weakness.
6 comments
Rebuttal 2/3
W2: comparison with a general-purpose complete system
Although the tables contain multiple methods, several are variants of the same framework family.
Our experiments compare JAZ invoke with two general agent loops CodeAct and CodeAct+subagents under a prompt-only setting, as well as an independent, complete specialized harnesses Letta (Table 1) and ACE (Table 2). Two variants of CodeAct+subagents (Table 1) and CodeAct (Table 2) were provided for better control of confounds, as the existing implementations (smolagents and AppWorld official) differed from JAZ in additional ways that unfairly reduced their performance, so we included results of JAZ reimplementations for a controlled comparison. Appendices D.2.2 and D.3.2 provide further details.
The paper neither directly evaluates modern harnesses such as Claude Code and Codex nor provides a representative baseline with comparable subagent, history-access, and context-management capabilities.
We provide comparisons with Codex below, since our experiments used OpenAI models. Codex is a complete system serving as a representative baseline with subagents and compaction. It supports either traditional tool calling or code-mode, so we set it to use code-mode to reduce confounds. To give it comparable history access, we put the path to its session log in its prompt.
Under this setup, the main difference between Codex and JAZ invoke is that the former is a considerably more complex harness than JAZ invoke, which is a bare agent loop derived from functional invoke (Section 2). For example, the Codex setup requires registering tools on an MCP server and generating MCP specs, setting up a Docker container for a sandboxed file system workspace, finding the path to the session log to tell the agent, etc. On the other hand, the JAZ setup simply involves passing the Python functions defining the tools as arguments directly to invoke, and writing a prompt.
Despite JAZ invoke's simplicity, it outperforms Codex at a lower cost:
| StuLife | Pass, all (%) | Score, all (%) | Pass, far recall (%) | Score, far recall (%) | Cost ($) |
|---|---|---|---|---|---|
| Codex | 72.6 0.9 | 79.9 0.8 | 71.2 1.3 | 73.0 1.2 | 18.2 0.6 |
JAZ invoke (ours)* | 74.1 0.6 | 83.3 0.7 | 72.5 1.3 | 76.0 1.4 | 15.6 0.6 |
*Results differ slightly from paper b/c we ran more attempts totaling 6 after the submission deadline, and a prompt bug in JAZ invoke was fixed.
| AppWorld | TGC (%) | SGC (%) | Cost ($) | Meta $ | Solver $ |
|---|---|---|---|---|---|
| Codex | 59.0 3.9 | 32.0 3.4 | 23.8 1.1 | 2.9 0.3 | 20.9 0.9 |
JAZ invoke (ours) | 74.2 2.1 | 51.1 3.7 | 20.9 3.7 | 9.9 3.8 | 10.9 0.6 |
In analyzing traces, we found that JAZ invoke's minimal interface was the reason it outperformed Codex. Here we present two failure modes of Codex that JAZ invoke doesn't run into due to its design.
One is silent tool failures, caused by the fact that tools are translated to MCP for Codex before being placed in its JavaScript REPL. MCP tools return an object with an isError field denoting whether the tool failed, rather than actually throwing inside Codex's JavaScript REPL. As a result, the agent sometimes mistakenly thought it has succeeded in when it hasn't. In JAZ, tools are just regular Python functions (Section 2, L104-106). As any other input object into JAZ invoke, it simply becomes an object in the agent's REPL, with no transformation in between like registering as an MCP tool. Thus, the tool raises when it fails so the agent sees the failure early.
Another failure mode is noise in history. Codex's logs are less structured and more complicated, and the entire session log includes both the compacted and current history. In StuLife long-range recall, Codex retrieves more irrelevant results such as the task it is currently working on. JAZ's subagent delegation workflow automatically separates prev_history -- the history that came before the current session -- and __history__, the current session, allowing the agent to only search the history that has exited its context, avoiding noise such as the current task text.
In AppWorld self-improvement, logs are truncated more often, making it harder for the meta-agent to identify root cause of subagent failures. On the other hand, JAZ invoke's __history__ by definition contains only what the LLM actually sees -- the inputs and outputs to the REPL (Property 2, L140-144, Section 2).
The reason Codex is more costly than StuLife on AppWorld is the fact that AppWorld's apis object that hides individual tool specs can't be directly put into Codex agent's environment. In JAZ, it's just a regular object with a docstring. In Codex, it has to become individual MCP tools, blowing GPT-5.4-nano's context as the OpenAI API for GPT-5.4-nano doesn't support tool search.
Rebuttal 3/3
W3: ablations
could a conventional agent achieve similar results if equipped with automatic logging and log search?
See our Codex results above.
separate ablations that expose only prompts or only histories
Ablation results for StuLife long-horizon are provided below.
| Method | Pass, all (%) | Score, all (%) | Pass, far recall (%) | Score, far recall (%) | Cost ($) |
|---|---|---|---|---|---|
JAZ invoke (ours)* | 74.1 0.6 | 83.3 0.7 | 72.5 1.3 | 76.0 1.4 | 15.6 0.6 |
| − prompt vars. | 65.4 1.3 | 74.4 1.5 | 39.8 4.1 | 42.6 4.3 | 13.0 0.8 |
| − history | 72.2 0.7 | 81.4 0.6 | 68.2 2.7 | 71.7 2.6 | 16.1 0.3 |
| CodeAct+subagents* | 63.3 2.0 | 71.8 2.2 | 38.5 4.7 | 41.1 5.2 | 13.8 0.7 |
*Results differ slightly from paper b/c we ran more attempts totalling 6 after the submission deadline, and a prompt bug in JAZ invoke was fixed.
We find that removing prompt variables alone can reduce the far recall performance to the level of CodeAct+subagents due to the loss of instructions across delegation boundaries. Removing the history variable has a lesser effect under our prompt, which makes up for the lack of a history variable by telling the agent to maintain its own. Despite that, the agent was inconsistent and sometimes forgot to append to it, causing later tasks to be unsolvable as they depended on information that the agent forgot to store to the history it was maintaining by itself.
Ablation results for AppWorld self-improvement are provided below.
| Method | TGC (%) | SGC (%) | Cost ($) |
|---|---|---|---|
JAZ invoke (ours) | 74.2 2.1 | 51.1 3.7 | 20.9 3.7 |
| − prompt vars. | 74.8 0.6 | 53.7 0.2 | 19.1 2.1 |
| − history | 72.3 1.4 | 48.1 2.8 | 23.1 2.4 |
| CodeAct+subagents | 71.1 1.3 | 47.6 2.1 | 21.6 1.8 |
As expected, "− prompt vars." performs level with JAZ invoke as the self-improvement workflow doesn't rely on programmatic reference to the user prompt. For the same reason, "− prompt vars." performs level with CodeAct+subagents. And the two arms without the history variable ("− history" and "CodeAct+subagents") perform worse than the two arms with it ("JAZ invoke" and "− history"), for the reason discussed in the paper (Section 4.2, L441-446), i.e., the subagent hand-building its history for the meta-agent to inspect builds it unreliably and lossily.
Again, we greatly appreciate the reviewer for their time and constructive feedback, and we look forward to hearing back from you.
Rebuttal 1/3
We appreciate that the reviewer likes our design and finds our results promising. We respond to your individual comments below.
W1: contribution/novelty
As an agent loop primitive, the difference between JAZ
invokeand CodeAct+subagents is that everything the LLM sees can be accessed as a programmatic variable, including the user prompt and the REPL history. In CodeAct+subagents implementations (e.g. smolagents, RLM), there's no variable containing the REPL history, and the user prompt isn't available as a variable either (taskin smolagents androot_promptin RLM are only shown to the agent in its context). The paper currently describes this at L22 in the abstract, L61-64 in the introduction, and re-emphasize it in L146-149 when introducing theinvokeprimitive. We are happy to revise the presentation to improve the clarity.The difference from complete systems like Claude Code and Codex is that JAZ
invokeis an agent loop primitive instead of a complete system, yet exhibits capabilities that traditionally only complete systems do. In our experiments,invokedoesn't get external engineering beyond the agent loop itself and a prompt ("prompt-only setup" as defined in Appendix D.1), whereas complete general-purpose harnesses like Claude Code and Codex are terminal agents with a file system and engineered context-management strategies (compaction, dynamic skill/tool loading, etc.). JAZinvokedoesn't have any of this and can thus be a bare agent primitive inside an agent framework while still offering capabilities like long-range recall and self-improvement without needing any of this extra setup and engineering. (Also see our response to W2 below for an empirical comparison with Codex.)In terms of workflow orchestration, JAZ
invokeprovides two novel uses of subagents: a) tail-recursive delegation (L258-293, Section 4.1) emulates compaction while preserving history, and b) continual self-improvement (L373-406, Section 4.2). Both uses are enabled by the presence of history and user prompt as programmatic variables.Our paper does in fact establish capabilities of the
invokeprimitive that previous agent loop primitives do not support, as well as performance and cost advantages over specialized systems specifically built with these capabilities. In particular, Tables 1 and 2 (Section 4) report comparisons with CodeAct+subagents, which isolates the capabilities and advantages from exposing prompts and histories as code variables in the agent loop. JAZinvokeoutperforms CodeAct+subagents at a similar cost since the history variable is needed for recall (StuLife long-horizon) and for root meta-agent's observability into solver subagents (AppWorld self-improvement). The tables also report comparisons with existing approaches for long-horizon tasks and self-improvement, namely Letta (MemGPT) and ACE, respectively. JAZinvokeoutperforms both Letta and ACE at a substantially lower cost, and the analysis that follows identifies the reason to be the flexibility the agent has due to the expressivity ofinvoke.We additionally compare with Codex and find that JAZ
invokeoutperforms it at a lower cost due toinvoke's minimal interface -- see our response to W2 below.The reviewer is right that code execution and subagent invocation are not claimed novelties of JAZ
invoke. The novelty that we do claim is an agent framework whose core agent primitive can perform long-horizon and self-improvement workflows with prompting only -- there's no need to build the extra harnessing traditionally needed for such workflows (e.g. compaction, tools for accessing a file system or memory system).We'd also like to point out that the empirical finding that such a minimalist agent can perform such tasks can be considered novel on its own. There has been prior work that was published where the main novelty was the finding that an existing method was able to perform tasks people did not know it could. An example is RLMs. An RLM is a code-mode agent loop with subagents and allowed programmatic inputs -- all three existed at the time and were present in frameworks like smolagents. Its novelty was its finding that that was all that was needed for many long-context tasks.