acceptodds
Withdrawn from ICLR 2027

The Serialization Bottleneck: Separating State Preservation from Property Derivation in Tool-Using Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-using agents often return intermediate structured objects, such as polygons, graphs, and tables, to the language model as serialized text. The next step then asks the model for two different operations: derive a property that selects the branch, and re-emit the object as an argument of its next tool call. End-to-end benchmarks cannot tell which of the two fails. We isolate them with a controlled diagnostic benchmark over geometry, graph, and tabular pipelines and seven language models. Under standard non-thinking inference, models read surface facts far better (51.3%–99.5%) than they derive global properties (20.8%–49.0%). In multi-step pipelines, calls that must re-emit the full object abort in up to 99.7% of tabular runs. Trace audits show early semantic corruption of the payload, not truncation, and schema-constrained decoding does not prevent it. In the most abort-prone setting (Qwen3-32B, tabular), keeping the full observation but passing an immutable reference as the argument cuts aborts from 99.7% to 1.2%, so the aborts come from re-emitting the payload, not from seeing it. For recurring branch predicates, runtime-computed summaries turn derivation into lookup; predicates outside the schema still need on-demand inspection or code execution. Deriving a property from a structured state and re-emitting that state in the next action are separate failure points, and each needs its own interface fix.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, tool-use agents, compositional reasoning, structured data, intermediate representations, agent evaluation, multi-step reasoning, runtime state management

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7eEgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies tool-using agents that pass intermediate objects between tool calls as serialized text. It separates two sources of failure. Across 3 domains, 7 models, local properties are answered at 51.3–99.5% accuracy while global properties reach only 20.8–49.0%. In multi-step pipelines, raw re-emission aborts 16.2–73.5% of graph pipelines and up to 99.7% of tabular pipelines.

Critical strengths
  1. A decomposition with a controlled design. Separating "cannot derive" from "cannot re-emit", and crossing decision exposure with state preservation in a factorial design, isolates the two failure sources more cleanly than end-to-end agent benchmarks do.
Critical weaknesses
  1. The writing raises concerns, including suspected excessive AI involvement.
    • The 9-page main text contains only one table but more than 60 pointers to the appendix, and it cites about 20 appendix tables (Tables 2–43) for its key numbers. Most claims cannot be checked from the main text.
    • The prose is dominated by coined notation (Π-Oracle, Π-Agnostic-Hit/Miss, Πfixed-A/B\Pi_{\text{fixed-A/B}}, Casc(All)/Casc(Clean), OHG, AG, βpres\beta_{\text{pres}}, ArtifactRef) and by long strings of numeric ranges, with little explanation of why each analysis is needed (e.g., the five RQ findings on p.2).
Questions for authors
  1. Which deployed agent frameworks or APIs actually require full-object re-emission as tool arguments? Can you provide evidence, such as framework analysis or real traces, that this interface is common?
  2. What completion cap was used in the Experiment 2 pipelines? Why do non-semantic aborts increase from about 35 to about 130 under constrained decoding in the tabular cell?
  3. How were the ArtifactRef cohorts selected?
Reviewer_dgKQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

When a tool returns a structured object as text, the next step asks the model both to derive a branch property and to re-emit the object. End-to-end benchmarks cannot tell which fails, and the fixes difference.

The paper proposed a controlled benchmark over geometry, graph, and tabular pipelines with 25 properties in four serial-derivation classes and seven models. Conditions cross state preservation (text vs. handle) with decision exposure (no summary vs. runtime summary), plus two controls: JSON-schema constrained decoding and ArtifactRef, which keeps the full object in view but passes an immutable reference as the argument.

The conclusion is local properties 51–99.5% vs. global 20.8–49.0%. Raw tabular pipelines abort in up to 99.7% of runs. 88–94% of aborts are schema-valid but semantically corrupted payloads, so constrained decoding barely helps, while ArtifactRef cuts aborts from 99.7% to 1.2%. Summaries make covered predicates near-perfect; uncovered ones fall to 70–82%. Models skip optional inspection on up to 54% of failure gates. Thinking helps arithmetic but not invariants, at up to 32.7× cost.

Critical strengths
  1. The ArtifactRef control preserves the full observation while changing how the next action passes the object. This is a strong design for investigating the proposed distinction. The empirical investigation is also considerately thoroughly defined.
  2. Thorough, honest, actionable: large N with intervals, dual denominators, format and schema ablations, REPL arm, cost accounting, and recommendations that map onto real interface decisions.
Critical weaknesses
  1. Moderate novelty and a possibly dated model set. Reference passing already exists in production systems. The 99.7% headline is Qwen3-32B tabular, and the "scale and reasoning don't fix it" claim rests on DeepSeek-V4-Pro in non-thinking mode and reasoning sweeps on two small models.

  2. The 512-token completion cap vs. payloads up to 1,648 tokens limitation may be insufficient to reproduce some arguments. Early semantic failures do not eliminate the need to establish that correct baseline outputs are feasible.

  3. ArtifactRef and raw results use unequal cohorts, particularly for graphs. Matched-pipeline analysis is needed to support precise causal effect-size interpretations.

Questions for authors
  1. Did you rerun the Qwen3-32B tabular Raw cell with a cap larger than the longest serialization?

  2. With native function calling and thinking on a frontier model, what are the tabular abort and gate numbers?

  3. How is the summary schema chosen, and does the oracle schema leak the gate predicate name?

Reviewer_ENeYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates failures in tool-using LLM agents that operate on serialized structured objects. The authors distinguish two sources of failure: deriving properties from an intermediate object and preserving that object when passing it to subsequent tool calls. They introduce a diagnostic benchmark spanning geometry, graph theory, and tabular data, evaluating seven language models on single-turn property recovery and multi-step tool-use pipelines. Their experiments compare raw serialization, deterministic property summaries, in-memory object handles, and additional controls such as schema-constrained decoding and artifact references. The results show substantial gaps between recovering explicitly exposed information and deriving global properties, as well as severe execution failures when models must regenerate large structured payloads. The authors further demonstrate that runtime-computed summaries and reference-based state passing can mitigate these failures. The paper argues that state preservation and property derivation should be treated as separate design concerns in tool-using agent systems.

Critical strengths
  1. A useful decomposition of two distinct failure mechanisms. Separating property derivation from payload re-emission provides a meaningful diagnostic perspective on structured tool-use pipelines. In particular, the Raw + ArtifactRef control is informative because it retains the full serialized object in the model’s observation while replacing the re-emitted payload with an immutable reference. The substantial reduction in tabular aborts provides evidence that payload regeneration, rather than merely observing the object, is a major source of failure in the evaluated setting.

  2. Extensive controlled experiments with informative interface ablations. The paper evaluates multiple domains, models, pipeline types, and state-passing conditions. The factorial comparisons between state preservation and decision exposure, together with the predicate-held-out experiments, help clarify when each intervention is beneficial. The analysis of aborted executions is also valuable, since excluding aborted runs could otherwise substantially distort estimates of end-to-end agent performance.

Critical weaknesses
  1. The generality of the proposed bottleneck is not sufficiently established by the evaluated task distribution. The benchmark consists primarily of procedurally generated polygons, relatively small graphs, and small tabular datasets, with tool operations and intermediate states controlled by the authors. This setup is useful for causal isolation, but it provides limited evidence that the observed failure patterns characterize a broader class of tool-using agents. The paper’s central claims would be more convincing with evidence from more diverse object types or realistic agent workflows, where intermediate state representations, tool interfaces, and failure modes differ substantially from the benchmark’s canonical serialization-and-re-emission setup. As presented, it remains unclear how broadly the measured bottleneck and the relative benefits of the proposed interventions generalize beyond these controlled pipelines.

  2. The headline improvements from Handle + Summary partly reflect providing the answer to the decision problem. This condition combines reference-based state preservation with runtime-computed properties that can directly determine the next action. The paper does include useful factorial and held-out-predicate experiments, so this is not an absence of ablation. Nevertheless, several of the strongest performance comparisons are between an unassisted model that must derive a property and regenerate the object, and a system that receives the relevant property from a trusted runtime. Such comparisons establish the practical value of the combined system, but provide weaker evidence for claims about the underlying reasoning limitations of LLM agents or the independent contribution of the proposed architecture. The paper should more carefully distinguish improvements due to supplying additional task-relevant information from improvements due to preserving object state.

  3. The novelty and scope of the contribution relative to established stateful tool interfaces remain insufficiently clear. The proposed interventions are intuitive systems-design choices, which retain objects in memory, pass references instead of full payloads, and expose deterministic summaries. The ArtifactRef control offers a useful diagnosis of why these choices matter in the tested setting, but the paper does not fully establish how much of its contribution lies in a new generalizable characterization of agent failures, rather than in demonstrating the benefits of conventional stateful interfaces on a carefully controlled benchmark. A clearer comparison with existing state-management and tool-execution designs, together with a sharper statement of the new insight that would not follow directly from the interface design, would strengthen the case for the paper’s research contribution.

Questions for authors
  1. How sensitive are the reported effects to the choice of serialization and action interface? The paper studies several serialization formats and includes an ArtifactRef control, but the main benchmark still relies on a particular set of tool signatures and object-passing conventions. Can the authors clarify which conclusions they expect to hold when tools accept native structured arguments, file references, or code-based transformations without requiring the model to reproduce the complete intermediate object?

  2. How should a practical agent runtime select its summary schema when the downstream predicates are not known in advance? The held-out-predicate experiments show that coverage of the required property is important, while uncovered predicates require additional inspection or derivation. What principled strategy would the authors recommend for selecting or dynamically extending the summary schema in open-ended tasks, and what evidence supports its effectiveness beyond the fixed schemas evaluated here?

  3. How robust are the reported conclusions under alternative agent execution policies? The primary pipeline experiments use a single-shot termination policy, while the appendices explore limited retry and interactive execution variants. Could the authors clarify whether the principal comparisons remain stable under standard recovery mechanisms, such as error-feedback retries or execution-based validation, and which failure categories persist after such recovery?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.