acceptodds
Under review as a conference paper at ICLR 2027

EffectSet: A Batch Tool Call Is More Than One Outcome

Abstract

A batch tool call can apply some requested effects, reject others, and leave still others uncertain. Treating that call as one success or failure forces a language agent to reconstruct item-level state from overlapping responses, which can abandon unfinished work or repeat completed effects. We introduce EffectSet, a deterministic runtime representation that folds tool events by target item into settled, verify, retry, and escalate sets, then exposes the residual obligations to the agent. We also introduce PartialCallBench, an executable benchmark with 288 fresh tasks, twelve response grammars, 6,912 target items, and 1,496 model-task decisions from Claude Haiku and GPT-5.6 Luna. On the 120-task main split, exact residual-action accuracy rises from 29.2% with raw transcripts and 72.5% with raw-plus-tagged transcripts to 86.7% with item-grouped canonical histories, then 100% when each item receives a derived state. A fully crossed, tags-only control sharpens the mechanism: grouping changes Haiku and Luna by -8.3 and +4.2 points, whereas derived state reaches 100% for both models, gaining 20.8 and 8.3 points over call-ordered tags. An independently generated 48-task Luna split and four unseen synthetic grammars also reach 100% with EffectSet. More decisively, a 32-task replay grounded in four independently documented public batch protocols raises exact accuracy from 71.9% raw and 75.0% tagged to 100% with the state ledger, a paired gain of 25.0 points over tags. The representation is linear-time, processes 120,000 events in 78.2 ms, and reduces prompt characters by 25.2%. EffectSet turns partial batch execution into explicit state that agents can act on exactly and efficiently.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, tool use, partial success, batch operations, execution state, residual obligations, agent reliability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tJvyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EffectSet, a deterministic state representation for partially successful batch tool calls. It reconciles execution outcomes across multiple calls into item-level states and residual action sets, reducing the reasoning burden on language agents when interpreting complex execution histories. The authors further introduce PartialCallBench to evaluate its effectiveness in accurately determining the remaining actions.

Critical strengths
  1. Practical Problem and Effective Solution. The paper addresses the problem of execution state recovery in tool calls. EffectSet provides a simple, efficient, and auditable solution by using deterministic item-level state management to reconcile complex execution histories across multiple calls into explicit residual action sets, helping agents avoid redundant executions and overlooked operations.

  2. Well-Designed Experiments and Clear Empirical Findings. The paper includes replay experiments grounded in four real-world tool protocols and systematically investigates the sources of performance improvements. The results demonstrate that explicit state derivation consistently improves the accuracy of residual-action decisions.

Critical weaknesses
  1. Strong Coupling Between the Benchmark and the Proposed Method. PartialCallBench first assigns a target state to each item and then generates event histories consistent with that state, while EffectSet derives these states using predefined deterministic rules. As a result, the model is already provided with the key intermediate information needed to solve the task. Therefore, the reported 100% accuracy primarily demonstrates the effectiveness of precomputed state representations in controlled settings, rather than establishing their effectiveness for more general agent state recovery.

  2. Lack of End-to-End Execution Evaluation. The paper primarily evaluates the accuracy of predicted residual action sets, without examining whether these predictions actually reduce redundant executions or prevent overlooked operations in real tool interactions. Moreover, the execution traces are synthetically generated based on public protocol specifications, with only eight tasks per protocol. Consequently, the current experiments provide limited evidence of EffectSet's reliability and end-to-end benefits in practical agent workflows.

  3. Insufficient Comparison with Existing State Management Approaches. The baselines primarily focus on different representations of execution events, without direct comparisons against conventional structured state stores, explicit state-tracking prompts, or agent-maintained state ledgers. As a result, the current experiments do not sufficiently establish whether EffectSet offers substantial advantages over existing state management mechanisms.

Questions for authors
  1. How can the authors ensure that the strong alignment between PartialCallBench's task construction and EffectSet's deterministic state rules does not overestimate the method's effectiveness?

  2. Can the authors provide end-to-end evaluations demonstrating that EffectSet reduces redundant executions and missed operations in real agent workflows?

  3. How does EffectSet compare with conventional structured state stores or agent-maintained state ledgers under the same evaluation settings?

Reviewer_uqTvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EffectSet, a per-item ledger for partial batch tool calls. It maps protocol-specific outcomes into seven canonical semantics, folds events into monotone item states, and outputs residual sets for verify/retry/escalate plus a done flag. It also introduces PartialCallBench, with synthetic grammars plus a small public-protocol replay. The main claim is that derived item state, not grouping or raw transcripts, is what makes exact residual planning work across models and response surfaces.

Critical strengths
  • The representation staircase is cleanly controlled. The matched same-task, same-packing cross-model comparison helps separate raw text, grouping, and derived state, and the token-length check rules out a simple context-volume explanation.

  • The label pipeline is stronger than in many agent benchmarks: an independent table-driven oracle, exact set scoring, and a public-protocol replay with fixture checks. This matters because the headline numbers depend on exact labels.

Critical weaknesses
  • The main 100% result is close to tautological. In the ledger-only condition the model is already given the derived per-item state s_i; producing V/R/E/done is then mostly a state-to-action mapping. A trivial script could likely hit 100%. So the experiment shows that supplying the answer state makes the answer easy, not that a model can reconcile partial outcomes across calls. The paper’s “state derivation is load-bearing” claim is therefore doing double duty: the derivation is performed by EffectSet, not by the model under test.

  • The scope is narrow and the external evidence is thin. The benchmark assumes monotone per-item effects, explicit identities, deterministic adapters, and no ordered dependencies or compensation. The public replay has only 32 tasks, 8 per protocol, and is schema-faithful rather than production-like. With one main model and a second model on selected splits, the generalization claim is weaker than stated.

Questions for authors
  1. In ledger-only, is the derived state s_i explicitly shown for every item? If so, what happens when the model must infer state from grouped events and receipts but is not given s_i ? That is the condition that would actually test reconciliation.

  2. Can you add a scripted baseline that maps s_i directly to V/R/E/ done? If it also scores 100%, what does the model result establish beyond format compliance?

  3. Do the public replay gains survive larger traces with missing/ambiguous IDs and non-monotone updates, or are they limited to the 32 clean schema-faithful cases?

Reviewer_Wr4TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EffectSet represents partially successful batch calls as per-item event histories, then folds those events into four states: settled, verify, retry, and escalate. It also introduces PartialCallBench, with 288 tasks across synthetic response grammars and a 32-task replay based on four public batch protocols. In the reported model experiments, providing the derived state ledger yields 100% exact residual-action accuracy across the main split, confirmation split, grammar transfer, and protocol replay. Grouping events alone is less consistent, and adding explicit residual-action lists does not improve on the ledger. The implementation processes 120,000 events in 78.2 ms. Overall, the paper presents a clear and potentially useful runtime abstraction, but its evaluation leaves questions about the novelty and practical generality of the result.

Critical strengths
  • The problem is practically relevant. Treating a partially completed batch as item-level effects can help prevent repeated side effects while preserving unfinished work. The proposed ledger is deterministic, auditable, and linear-time.

  • The ablations are thoughtfully designed. The paper separates call-ordered tags, grouped histories, and derived state, and includes a matched-task, two-model control. The independent oracle and protocol-fixture checks are useful safeguards for benchmark scoring.

Critical weaknesses
  • The headline model result largely tests whether a model can translate an answer-bearing representation. The ledger supplies a per-item state, and the shared legend maps each state directly to the requested action. Thus, 100% accuracy mainly shows that models can copy this mapping; it does not establish that EffectSet reliably derives correct state from realistic tool interactions. The public replay is small (32 generated tasks, eight per protocol) and schema-faithful rather than production traffic, and the paper does not report end-to-end execution of the residual actions. The matched cross-model control is also limited to 24 tasks per model; the reported 8.3-point Luna gain has an interval whose lower bound is zero. These results make the broad practical and cross-model claims less convincing than the headline suggests.

  • The stated “monotone” reconciliation rule is underspecified. In Equation 1, only settled is protected from later events. A later retryable, unknown, or absent event can overwrite an earlier permanent/escalate state, even though the paper describes permanent failures as irrecoverable. The paper should clarify whether events are guaranteed to be ordered and tied to the current attempt, and how stale or reordered receipts are handled. Without that contract, the exact residual plan may depend on an assumption not represented in the ledger.

Questions for authors
  • What ordering and freshness guarantees are assumed for events in Equation 1? Can a later event refer to an older or different attempt, and if so, how is an earlier permanent rejection prevented from being replaced by a retry or verify state?

  • Is the intended interpretation of the 100% ledger result that the runtime supplies a sufficient per-item answer for the model to format, or that the benchmark demonstrates improved agent reconciliation? Clarifying this distinction, especially given the absence of production traces or executed recovery actions, would affect how I assess the contribution’s empirical scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.