- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose ReCommit, a framework for repairing trajectories of tool calls in a simulated environment.
The task, as proposed, defines success as a trajectory which contains no failed tool calls, and which also satisfies a given success criterion given a request. Given a set of trajectories which fail at least one of these conditions, a repair method attempts to recover a sequence of tool-calls that passes both conditions.
As described in the paper, the method consists of three steps. The first is to use a proposer, instantiated by a diffusion LM, to score all possible operation types so as to select a subset of operations which might be feasible to use; these scores are used to produce "supports," which are subsets of all potential operations.
For each support, ReCommit produces a canonical realization based on "service-specific schema-based expansion rules," and failing that attempts an autoregressive sequence conditioned on the support. Alternating between canonical and expressive realizations, ReCommit proceeds from support to support until it recovers a correct sequence with no failing tool calls, or until a budget is reached.
After evaluating a sequence consisting of at least one failing tool call (in addition to other content that would not be pruned), ReCommit also re-attempts the task by pruning the rejected tool calls.
This method is tested in a custom evaluation framework based on Agent-Diff, which is an external agentic benchmarking framework. The authors report improved sequence recovery with less wall-clock time for ReCommit as compared to other baselines.
Critical strengths
None
Critical weaknesses
1. Experimental Soundness and Reproducibility
The framework contribution of the paper is minimal, essentially rehashing a standard MDP construction. Thus, it ought to be evaluated on the soundness of its empirical claims, and of how well it demonstrates a convincing, general advantage over baseline methods.
Under this framework, there are many problems with experimental soundness and reproducibility as it stands. I detail a few below:
- The proposed method, ReCommit, is provided with hard-coded, benchmark-specific grounding templates (via the canonical expansion step). Baselines do not have access to these templates, and their provenance is not documented in the paper.
- Code for baselines is absent from the attached repository; while not all code must be provided, its absence and the use of a custom simulator / environment based on but not identical to Agent-Diff means that there is no way to verify whether baseline methods are reasonably and fairly implemented
- The actual evaluator is not released and differs from Agent-Diff; these differences are not specified in the paper
- Success criteria mix two quantities (all tool calls valid & passing) without reporting the split of either; canonical expansion and pruning may inflate success for ReCommit simply by increasing the ERR without improving relative task performance, which is supported by Table 7
2. Clarity and Technical Presentation
This work omits crucial details about the pipeline, some of which could materially alter the conclusion of the paper.
For instance, the description of "support-conditioned realization search" includes the following passage:
The canonical mode expands the permitted operation types in S into a call structure using service-specific schema-based expansion rules and read dependencies, then grounds this structure to construct . Grounding resolves entities and task-specified values from the request and available public state. When public evidence suffices to fill the arguments, ReCommit adopts the grounded sequence directly. Otherwise, an autoregressive model fills unresolved arguments while preserving the grounded fields and call structure. This mode preserves choices established by public evidence and concentrates generation on the remaining arguments.
This contains many vague and undefined terms, and elides the fact that the canonical step consists of benchmark-specific templates which are only made available to ReCommit, and not to other baselines (as mentioned above).
Such jargon persists throughout the paper, decreasing its readability while at the same time often omitting key details.
In short, the paper lacks precision in novelty in its framework proposal (differences from DiffuAgent, Tool-Planner, etc are not specified), and its experiments are both narrow (single-benchmark, one-model for ReCommit) and unsound.
Questions for authors
3.1
- What is “public information”, versus the full state?
- What are the “execution observations” x, and how are they used?
3.2
- How does this differ from a straightforward MDP formulation, in which T is the single-step action space and pi is a trajectory through multiple steps in that action space?
- This is an important question to answer if one of the key contributions is “introducing operation supports as an abstraction”
4.2
- What are the “schema-based expansion rules and read dependencies”? It is unclear from the paper, and the section does not elaborate about how these are established; further, they seem to correspond to hard-coded benchmark-specific templates in the code, a serious concern that I elaborate on below.
5.1
- Is there a reason why ReCommit is not benchmarked on 32B, if it is less costly than Feedback-Reflective and Direct baselines? Where do the time gains come from, and how are they measured?
- How many LLM calls and tokens per trial?
- Is timing wall-clock, process time, or another measure?
- How much time in each modality is spent on tool calls, generation, model loading, etc.
5.2
- At a high level, why do we expect an 8B dLLM model to yield better affordances than an 8B AR model? Why do we expect restricting the action space via dLLM to yield better results than letting the proposer decide, especially as models scale?
5.3
- Why do more slots (M growing) harm performance? The text suggests that it flatlines, but it seems to actively decline.
- Are there any numbers in the paper that can be verified against external baselines / findings, or are all gains and baselines internal to the paper?
- What is the failure composition of the pool? What proportion of trajectories received y=0 by having at least one invalid tool call, versus not satisfying the objective? How does this split vary according to repair types -- that is, does repair tend to satisfy the objective more, or does it simply prevent invalid tool calls on sequences that already succeed?
General
- Is there a reason why the simulator and framework is hand-spun instead of using AgentDiff’s framework? For example, issueLabelDelete acts differently in the shipped code versus in Agent-Diff.
- Is there code for how baselines were executed in the repository, to ensure that they are fair and accurate?
- There appear to be hand-written templates for each task that are used in the canonicalization step, which are not mentioned explicitly in the paper, not defined except in the code, and not provided to any baseline methods; additionally, given that the canonical expansion seems to give most of the gains (versus expressive-only), it seems that the current set of experiments does not rule out that the gains may simply come from the benchmark-specific hard-coded templates made available only to the proposed method.