acceptodds
Under review as a conference paper at ICLR 2027

The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

Abstract

LLM agent benchmarks measure task completion, reliability, and inference cost, but not the persistent data an agent run leaves on local disk, including logs, context snapshots, checkpoints, and debug traces. These bytes are absent from the leaderboards we survey, yet bear on whether an agent system can be deployed on desktops, under data-retention compliance regimes, or at fleet scale. To our knowledge, AgentFootprint is the first systematic cross-framework benchmark of post-run agent storage footprint. Its serialization-aware metric suite covers total retention, channel composition, duplication, growth exponent, compressibility, and a conversation-history reconstructability score. It addresses a measurement trap: naïve byte-level measurement understates duplication by an order of magnitude, because database paging and JSON escaping obscure repeated content. The footprint has two determinants: the logical volume generated by the agent's behavior and the amplification added by the persistence layer. A fixed-trace control compares the latter: the same scripted content replayed through each persisting framework's adapter yields retained sizes differing by 6.7×. Among persisting defaults achieving 100% accuracy under identical models, tools, and tasks, retained bytes differ by 15.7×. The defaults bundle different recovery and audit capabilities, so the suite reports storage jointly with accuracy and reconstructability. Three full-history configurations grow superlinearly on a repeated-observation stress task, and on a deliberately minimal write task framework residue exceeds the delivered output files by orders of magnitude. Exported trajectories from 108 instance-normalized SWE-bench Verified submissions span three orders of magnitude in per-instance volume with no detectable correlation with resolve rate. A content-addressed store reduces retention 4.8–32.7× while preserving all properties checked by our restoration tests, including every reconstructability score. Reconstructing recorded conversation content does not require megabytes, and these storage metrics can be reported alongside inference cost.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, storage footprint, benchmarking, persistence, reproducibility

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gsbqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AgentFootprint is a benchmark that measures different aspects of storage, including total size, duplication, growth, compressibility, and history reproducibility. The authors evaluate eight agent frameworks using the same LLM backend and controlled tasks. Their results show that frameworks with similar accuracy and recovery capabilities can differ more than 10 fold in storage usage, and some frameworks exhibit nearly quadratic storage growth.

Critical strengths

The proposed benchmark covers several useful aspects of storage that other benchmark miss, and the evaluation covers eight popular agent frameworks, and the fixed trace experiment helps separate storage overhead from agent behavior. The results provide useful evidence that similar task performance does not necessarily require similar storage costs.

Critical weaknesses
  1. The results depend heavily on storage settings. The paper shows that changing these settings can greatly reduce storage, so it is unclear whether the differences come from the frameworks or their configurations.

  2. The R score mainly checks whether conversation history can be recovered. However, it does not fully measure whether agents can recover their internal states or debugging information.

  3. The proposed compression method is only tested after agent execution. The paper does not measure its runtime or memory overhead during actual agent tasks.

Questions for authors

How would the framework rankings change if all frameworks used similar storage settings? Would the reported gain still hold?

Why does the R score focus mainly on conversation history? Could the authors also test whether frameworks can recover complete agent states?

What would be the runtime and computational overhead of using content-addressed storage during agent execution? Would the storage savings still be significant?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gsbq (1/3): rankings under similar settings

Thank you for the concise review and for identifying the three issues that matter most. The three parts below answer your questions in order.

Q1. How would the framework rankings change if all frameworks used similar storage settings? Would the reported gain still hold?

The ranking changes substantially, and the storage reduction from a minimal setting holds on every task family and grows with task length. Section 5.1 now reports this next to the headline number. On the ten file-QA tasks, we ran each framework's default and, for three frameworks, a minimal configuration that retains conversation history: LangGraph keeps the newest checkpoint, AutoGen omits its event log, and LlamaIndex keeps the latest context.

Table R1. File-QA ranking under default and minimal settings (mean MiB; 20 runs each; AutoGen's test counts are default / minimal)

\hline \text{Framework} & \text{Default MiB} & \text{Rank} & \text{Minimal MiB} & \text{Rank} & \text{History and request tests} \\\\ \hline \text{LangGraph} & 5.324 & 1 & 0.326 & 5 & \text{20, 20} \\\\ \text{AutoGen} & 2.596 & 2 & 0.904 & 3 & \text{19, 19 / 20, 20} \\\\ \text{InfiAgent} & 2.114 & 3 & 2.114 & 1 & \text{20, 20} \\\\ \text{Agno} & 1.141 & 4 & 1.141 & 2 & \text{20, 20} \\\\ \text{LlamaIndex} & 0.924 & 5 & 0.306 & 7 & \text{20, 20} \\\\ \text{OpenAI Agents} & 0.329 & 6 & 0.329 & 4 & \text{20, 20} \\\\ \text{CrewAI} & 0.325 & 7 & 0.325 & 6 & \text{20, 20} \\\\ \hline \end{array}$$ LangGraph moves from largest to fifth of seven. Kendall's $\tau$ between the two orderings is 0.24, and the spread falls from $16.4\times$ to $6.9\times$. LangGraph's, LlamaIndex's, and OpenAI Agents' minimal forms converge at 0.31–0.33 MiB. So your concern is correct: much of the difference comes from configuration. The paper now reports each result with the retention setting attached, and we read a framework's position as a property of that setting. Table 37 defines every measured configuration. A default is a documented durable-session recipe: for LangGraph, AutoGen, LlamaIndex, OpenAI Agents, and Agno, the adapter attaches the persistence component from the framework's tutorial, and without it these agents keep nothing on disk. LangGraph's and LlamaIndex's minimal configurations are adapter code that prunes at user-turn boundaries, and AutoGen's omits the event logger. The differences we report are therefore differences between recipes, which is the reading your question pointed to. The gain from a retention setting also grows with task length. **Table R2. Default-to-minimal ratio of retained bytes, $r = \sum S^{\mathrm{default}} / \sum S^{\mathrm{minimal}}$ over the same tasks and repeats** $$\begin{array}{lrr} \hline \text{Workload} & \text{LangGraph} & \text{AutoGen} \\\\ \hline \text{File-QA} & 16.3\times & 2.9\times \\\\ \text{HTTP retrieval} & 4.9\times & 5.0\times \\\\ \text{Airline API} & 10.2\times & 2.8\times \\\\ \text{SWE-bench repository tasks} & 31.9\times & 8.3\times \\\\ \text{Enterprise, all pairs} & 202.6\times & 44.3\times \\\\ \text{Enterprise, both end earlier} & 32.5\times & 4.3\times \\\\ \text{Enterprise, both use all 300 calls} & 217.0\times & 105.5\times \\\\ \hline \end{array}$$ Retention settings leave the model's input unchanged, so delivery agrees within run-to-run variation: 26 vs. 23 enterprise deliveries for LangGraph and 25 vs. 22 for AutoGen. An enterprise task is one user turn, and LangGraph's pruning runs when the turn returns, so the enterprise ratios describe what a run leaves behind (part 3).
AuthorsOpenReview ↗

Response to Reviewer gsbq (2/3): beyond conversation history

Q2. Why does the R score focus mainly on conversation history? Could the authors also test whether frameworks can recover complete agent states?

We used conversation history as the shared reference because every persisting framework can be scored against it in the same way. RR grades the structure of the retained record: whether the store keeps a complete record per model call (Section 3.2). CrewAI's per-task records score R=1R=1 and pass the history and request tests on finished runs. To cover agent state, the revision adds direct tests: self-contained reconstruction of full historical requests, event and token-usage audit queries, and restart after a process kill.

Recovery test. On nine public tasks we kill the agent process at three points, then restart a fresh process from native state and let it finish the task. F1 follows a committed native state, F2 follows a tool return before persistence, and F3 follows a side effect before acknowledgement. The study covers 398 fault conditions and reaches 391 kills.

Table R3. Airline service: tasks completed after restart / process kills

\hline \text{Configuration} & \text{F1} & \text{F2} & \text{F3} \\\\ \hline \text{LangGraph default} & \text{4/6} & \text{5/6} & \text{2/4} \\\\ \text{LangGraph minimal} & \text{5/6} & \text{5/6} & \text{2/4} \\\\ \text{OpenAI Agents default} & \text{4/6} & \text{4/6} & \text{2/4} \\\\ \text{AutoGen default} & \text{6/6} & \text{0/6} & \text{0/3} \\\\ \text{AutoGen minimal} & \text{4/6} & \text{0/6} & \text{0/4} \\\\ \text{InfiAgent default} & \text{4/6} & \text{0/6} & \text{0/3} \\\\ \text{Agno default} & \text{5/6} & \text{0/6} & \text{0/3} \\\\ \text{LlamaIndex default} & \text{4/6} & \text{0/6} & \text{0/4} \\\\ \text{LlamaIndex minimal} & \text{4/6} & \text{0/6} & \text{0/4} \\\\ \text{CrewAI default} & \text{5/6} & \text{0/6} & \text{0/4} \\\\ \text{SmolAgents default} & \text{–} & \text{0/6} & \text{0/4} \\\\ \hline \end{array}$$ On the airline service, nearly every framework continues after a committed state (F1). Mid-tool faults separate them: only LangGraph and OpenAI Agents complete tasks after F2 or F3, on the airline service and on repository tasks alike, where each completes 3–4 tasks per fault type. AutoGen, Agno, InfiAgent, and LlamaIndex retain native history at the crash point and complete no airline task after F2 or F3. LangGraph's two rows are close because at a kill its minimal store is as large as the default's (Appendix U.6), so these tests do not separate its two configurations. So a framework can score the top $R$ and pass every history test and still be unable to resume interrupted work. This is exactly the gap you identified, and it is why these are now separate tests. **Table R4. Native reconstruction across workloads (passes / runs)** $$\begin{array}{lrr} \hline \text{Workload} & \text{History} & \text{Full request} \\\\ \hline \text{File-QA} & \text{199/220} & \text{199/220} \\\\ \text{SWE-bench} & \text{174/200} & \text{173/200} \\\\ \text{HTTP} & \text{180/200} & \text{180/200} \\\\ \text{Airline API} & \text{180/220} & \text{180/220} \\\\ \text{Enterprise} & \text{672/800} & \text{672/800} \\\\ \hline \end{array}$$ The request test regenerates system instructions, tool schemas, generation settings, and request order from native state and the common program, with no proxy-supplied task data. A retained store can also differ on audit: AutoGen's default and minimal settings both pass history and request tests on all 80 enterprise runs, and only the default retains token-usage receipts. Debugging and audit needs can therefore be matched against specific channels (Section 6, Appendices U.3 and U.6).
AuthorsOpenReview ↗

Response to Reviewer gsbq (3/3): runtime overhead

Q3. What would be the runtime and computational overhead of using content-addressed storage during agent execution? Would the storage savings still be significant?

CAS in this paper runs after a task finishes, and we did not build a write-path integration, so we have no measurement of its overhead during execution. Section 7 now presents CAS as a post-run reference, and we report three things that bear directly on your question.

1. The cost of the post-run pass. We measured packing, restoration, peak memory, and temporary disk use on 94 stores, 940 timing trials in total, against a complete tar+zstd archive at the same compression level.

Table R5. Post-run cost on tool-workload stores (Linux): tar+zstd / CAS, medians

\hline \text{Framework} & \text{Size KiB} & \text{Pack ms} & \text{Restore ms} & \text{Peak RSS MiB} \\\\ \hline \text{LangGraph} & \text{11.5 / 157.5} & \text{63.2 / 123.1} & \text{1.8 / 7.7} & \text{105 / 28} \\\\ \text{AutoGen} & \text{9.4 / 124.1} & \text{56.3 / 27.1} & \text{2.3 / 5.7} & \text{104 / 26} \\\\ \text{InfiAgent} & \text{62.9 / 795.6} & \text{165.3 / 194.2} & \text{11.9 / 84.0} & \text{109 / 43} \\\\ \text{Agno} & \text{11.4 / 42.2} & \text{104.9 / 131.6} & \text{3.3 / 23.9} & \text{104 / 30} \\\\ \text{LlamaIndex} & \text{4.3 / 15.6} & \text{46.9 / 31.8} & \text{1.6 / 1.6} & \text{104 / 25} \\\\ \text{OpenAI Agents} & \text{4.8 / 43.0} & \text{47.2 / 23.9} & \text{1.3 / 4.2} & \text{104 / 25} \\\\ \text{CrewAI} & \text{7.3 / 19.2} & \text{76.5 / 54.2} & \text{6.1 / 9.5} & \text{104 / 25} \\\\ \hline \end{array}$$ A post-run pass costs tens to hundreds of milliseconds per store. Complete archives are smaller on 92 of 94 stores and restore all 94 byte-for-byte, while CAS uses far less memory (24–43 MiB against 103–109 MiB). **2. The storage reduction from retention policy.** We measured default and minimal configurations on the same tasks, under the same model and budgets. **Table R6. Enterprise tasks: retention after the run under default and minimal settings (mean MiB per run)** $$\begin{array}{lrrrr} \hline \text{Configuration} & \text{Default} & \text{Minimal} & \text{Ratio} & \text{Tasks delivered, default / minimal} \\\\ \hline \text{LangGraph} & 606.7 & 3.00 & 202.6\times & \text{23 / 26} \\\\ \text{AutoGen} & 137.9 & 3.11 & 44.3\times & \text{22 / 25} \\\\ \hline \end{array}$$ AutoGen's minimal configuration omits the event logger and acts during the run. LangGraph's and LlamaIndex's are adapter code that prunes at user-turn boundaries. An enterprise task is one user turn, so Table R6 reports what a run leaves behind, and LangGraph's store during the run equals the default's. Where each round is a user turn, pruning also lowers the live store: in the 400-round fixed-trace study, LangGraph's store before its end-of-run VACUUM is 22.9 MiB under the minimal setting and 1,973 MiB under the default; AutoGen's is 196 MiB and 570 MiB. Reducing storage inside one long turn therefore needs a write-path policy, which neither CAS nor these configurations provide (Sections 4.2, 6, and 8). **3. What an in-execution CAS would need to deliver.** A write-path CAS would have to handle the same content that drives these defaults: repeated serialization of a growing history. After pruning, little exact repetition remains to deduplicate. LangGraph's minimal store has a duplication factor of 1.52 and LlamaIndex's 1.54, so the additional saving from write-path deduplication on top of a minimal configuration would be modest. It would also need its own deletion and recovery design for shared objects (Appendix A).
Reviewer_urwBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper focuses on the continuous accumulation of local data such as logs, conversation history, and checkpoints of LLM agents, and proposes AgentFootprint, which incorporates storage overhead and its supported conversation reconstruction capabilities into the evaluation. This method identifies duplicate content through serialization-aware analysis, and combines storage volume, growth rate, and history reconstructability to characterize the cost and capability trade-offs of different persistence strategies. Controlled experiments of eight frameworks and SWE-bench public trace analysis show that the amount of retained data varies significantly between different configurations or systems; content-addressed deduplication and compression experiments further demonstrate the reduction of storage space while maintaining the measured conversation reconstruction capability.

Critical strengths
  1. The paper focuses on the storage overhead that is less involved in existing Agent evaluation. Compared with task success rate and inference cost, the logs and conversation history retained after execution are usually not included in the comparison. The author conducted systematic measurements and analyzed the storage differences under different frameworks and persistence configurations.
  2. The author analyzes the source of storage differences through comparative experiments and ablation. Fixed-trace experiments fix the observation and response content to compare the persistence overhead of different configurations; history-only ablates the impact of retention strategies. Recovery checks after deduplication and compression verified that the measured conversation reconstruction capabilities were retained even with reduced storage.
Critical weaknesses
  1. The paper has not fully explained why storage needs to be used as an independent agent evaluation dimension. Experiments have shown large differences in the amount of retained data across configurations, but have not fully established the connection between these differences and operating costs, system reliability, or data management burden. Therefore, the current results indicate more that there are differences in storage behavior rather than that the amount of storage can reflect important system attributes not covered by existing metrics. This makes it still unclear in what scenarios this indicator should be used and how to assist in evaluating the Agent system.
  2. The paper uses history reconstructability to measure the ability of storage to retain, but this dimension is not enough to cover the goals of different persistence strategies. Conversation reconstruction, auditing, execution replay and failure recovery rely on different information, so storage differences under the same RR do not necessarily represent avoidable overhead. The authors do not adequately explain why conversation reconstruction is used as the primary reference, and how additional storage that serves other goals is evaluated. This limits the explanatory power of comparing storage efficiency based on RR.
Questions for authors
  1. In which practical scenarios will storage footprint become an important factor affecting Agent deployment or system selection? Can existing experimental or production data illustrate how observed storage differences translate into operating costs, capacity constraints, or data management burdens, and explain how these impacts relate to workload and retention policy?
  2. Different persistence strategies may serve goals such as auditing, execution replay, or failure recovery. Why choose history reconstructability as the main capability reference? Taking into account the differences in capabilities in Appendix R, which storage overhead in the existing results can be classified as redundant, and which might be used to support functionality not covered by R?
  3. The controlled experiment is mainly based on the synthetic file task, while the SWE-bench part measures the exported trajectory volume. To what extent does the existing evidence support that the storage differences and growth patterns in controlled experiments apply to real workloads? Can the impact of workload and logging/export policy be distinguished?
4 comments
AuthorsOpenReview ↗

Response to Reviewer urwB (1/3): practical importance

Thank you for the positive assessment and for pressing on why storage deserves its own evaluation dimension. The three parts below follow your three questions.

Q1. In which practical scenarios will storage footprint become an important factor affecting Agent deployment or system selection? Can existing experimental or production data illustrate how observed storage differences translate into operating costs, capacity constraints, or data management burdens, and explain how these impacts relate to workload and retention policy?

We collected operator-reported inventories from three production deployments.

Table R1. Production inventories (operator-reported, anonymized)

\hline \text{Deployment} & \text{Framework} & \text{Window} & \text{Tasks} & \text{Primary storage} & \text{Per task} \\\\ \hline \text{Cold-chain quotation} & \text{InfiAgent} & \text{60 days} & \text{4,530} & \text{171 GB added} & \text{38 MB} \\\\ \text{Retail sales ChatBI} & \text{In-house} & \text{about 5 days} & \text{about 7,900} & \text{about 90 GB held} & \text{11 MB} \\\\ \text{Logistics price inquiry} & \text{LangGraph} & \text{30 days} & \text{2,440} & \text{25 GB added} & \text{10 MB} \\\\ \hline \end{array}$$ Per-task figures are primary storage divided by task count. These cases show three scenarios where storage matters. 1. **Long-lived services that retain by default.** The cold-chain deployment adds about 86 GB a month even after staff raised storage concerns and optimized request caching. Data older than one year will move to slower storage, and the final retention period is an open decision. 2. **Multi-user systems with trajectory reuse.** In the retail ChatBI inventory, 55 GB is session workspaces, 20 GB is training trajectories, and 15 GB is conversations. A further 120 GB of backups exceeds the primary data. 3. **Capacity that is comfortable today.** The logistics operator retains everything and has no concern yet. Knowing what a run leaves behind lets such a team plan before it becomes one. Hardware prices add to this. TrendForce reported an approximately 80% quarter-on-quarter rise in enterprise SSD contract prices in Q1 2026 [1]. The controlled data show how the differences scale with retention policy and workload. **Table R2. Enterprise tasks: retained storage implied per 1,000 runs** $$\text{GiB per 1,000 runs} = \frac{1000 \times \text{mean MiB per run}}{1024}$$ $$\begin{array}{lrrr} \hline \text{Configuration} & \text{Mean MiB per run} & \text{GiB per 1,000 runs} & \text{Tasks delivered (of 80)} \\\\ \hline \text{LangGraph default} & 606.7 & 592 & 23 \\\\ \text{LangGraph minimal} & 3.0 & 2.9 & 26 \\\\ \text{AutoGen default} & 137.9 & 135 & 22 \\\\ \text{AutoGen minimal} & 3.1 & 3.0 & 25 \\\\ \hline \end{array}$$ A retention setting alone moves LangGraph from 592 GiB to 2.9 GiB per 1,000 such runs, with similar delivery counts. Workload matters as well. The same LangGraph default retains 0.14 MiB on HTTP retrieval, 9.6 MiB on repository tasks, and 607 MiB on enterprise tasks, because checkpoint growth depends on interaction length. Unfinished work carries most of this burden: 87% of enterprise bytes sit in runs that never delivered, so task success alone would hide them. Of 288 failed quotation runs, 283 never complete profile validation, the second of seven workflow stages, and these runs alone hold 60% of enterprise bytes (Table 36). In 195 of the 288, one identical command recurs at least 20 times in a row, 244 times at the median, and full-history persistence stores each repetition (Table 38). Under full-history persistence, the storage bill of this suite is mostly paid by runs that stall early and spend the whole call budget. After history reduction, delivered quotations still keep about 20 MiB of business files. That is a separate optimization target, which workflow organization, delivery format, and reuse of intermediate artifacts can address. Task success and inference cost say nothing about any of these continuing requirements, which is the case for reporting storage as its own dimension. [1] TrendForce, 2026 (cited in Section 8).
AuthorsOpenReview ↗

Response to Reviewer urwB (2/3): what retained bytes support

Q2. Different persistence strategies may serve goals such as auditing, execution replay, or failure recovery. Why choose history reconstructability as the main capability reference? Taking into account the differences in capabilities in Appendix R, which storage overhead in the existing results can be classified as redundant, and which might be used to support functionality not covered by R?

We chose conversation history as the common reference because it is the one capability nearly every framework claims to persist and can be checked the same way across all eight. It is one use among several, so the revision adds tests for the others: complete historical requests (BB), event and token-usage audit queries, and recovery after process kills. RR itself grades the structure of the retained record: whether the store keeps a complete record per model call (Section 3.2). CrewAI's per-task records score R=1R=1 and pass the content tests HH and BB on finished runs, so RR is read together with those tests.

Table R3. What each configuration supports: mean MiB, history and request tests, event and usage queries on 80 enterprise runs; tasks completed after process kills across all public tasks

\hline \text{Configuration} & \text{MiB} & H\text{, }B & \text{Event} & \text{Usage} & \text{Recovered / kills} \\\\ \hline \text{LangGraph default} & 606.7 & \text{80, 80} & 80 & 80 & \text{20 / 45} \\\\ \text{LangGraph minimal} & 3.0 & \text{80, 80} & 80 & 80 & \text{23 / 45} \\\\ \text{AutoGen default} & 137.9 & \text{80, 80} & 80 & 80 & \text{6 / 33} \\\\ \text{AutoGen minimal} & 3.1 & \text{80, 80} & 80 & 0 & \text{4 / 33} \\\\ \text{InfiAgent default} & 31.0 & \text{80, 80} & 80 & 80 & \text{4 / 45} \\\\ \text{Agno default} & 5.0 & \text{80, 80} & 80 & 80 & \text{5 / 33} \\\\ \text{LlamaIndex default} & 11.1 & \text{80, 80} & 80 & 0 & \text{4 / 34} \\\\ \text{OpenAI Agents default} & 4.3 & \text{80, 80} & 80 & 0 & \text{19 / 46} \\\\ \text{CrewAI default} & 1.8 & \text{32, 32} & 32 & 0 & \text{5 / 33} \\\\ \text{SmolAgents default} & 3.9 & \text{0, 0} & 0 & 0 & \text{0 / 28} \\\\ \hline \end{array}$$ **Which overhead is redundant, and which serves a function:** - **Removable with every tested capability intact.** LangGraph's superseded checkpoints are 94% of its default file-QA bytes. Its minimal setting answers all history, request, event, and usage queries. The last column does not separate LangGraph's two configurations, because at a kill both hold the same checkpoints (Appendix U.6). These older checkpoints do support time travel to earlier graph states, which none of our tests exercise, so a team that needs it should keep them. The same applies to LlamaIndex's older context snapshots (67%) and Agno's free database pages. - **Serving a measured function.** AutoGen's event log is 65% of its default file-QA bytes. It is the only place AutoGen's token-usage receipts are kept: the usage query passes on all 80 enterprise runs by default and on none with the log disabled. Those bytes buy auditing. - **Duplicated inside required state.** AutoGen's per-turn snapshots hold the history the tests read, and they repeat it ($D = 4.37$), so they still grow near-quadratically in the minimal setting (196 MiB at 400 rounds). - **Recovery is a separate question.** OpenAI Agents retains 4.3 MiB on enterprise tasks and completes 19 of 46 tasks after process kills, close to LangGraph. InfiAgent keeps native history at all 45 crash points and completes 4, and its pinned configuration can reject a restart over a stale running-task marker. History at the crash does not by itself allow the work to continue. Appendix R now separates documented from measured capabilities: after F2 or F3 kills on airline tasks, LangGraph completes 7 of 10 and OpenAI Agents 6 of 10, and the other six complete none, including frameworks whose documentation lists workflow recovery. Mutable services add one more use for saved observations: 196 of 802 repeated read-only airline calls returned different content from the original. Those observations cannot be regenerated by re-querying. So the extra bytes in the current results fall into identifiable classes, and each class can be judged against a stated requirement (Section 6, Appendices U.3, U.4, and U.6).
AuthorsOpenReview ↗

Response to Reviewer urwB (3/3): applicability to real workloads

Q3. The controlled experiment is mainly based on the synthetic file task, while the SWE-bench part measures the exported trajectory volume. To what extent does the existing evidence support that the storage differences and growth patterns in controlled experiments apply to real workloads? Can the impact of workload and logging/export policy be distinguished?

We now measure native stores on real workloads directly. The revision adds 620 public tool-task runs and 800 runs of 40 mixed business tasks from multiple enterprises, with the same storage boundary as the controlled study: new and modified workspace and native-state files. The earlier SWE-bench export survey stays in Appendix T as a separate view of developer-chosen logging.

The controlled ordering carries over. Across the seven persisting defaults, the rank correlation between file-QA retention and each realistic workload is high.

Table R4. Spearman rank correlation of default retention with file-QA (seven frameworks)

\hline \text{Workload} & \text{Spearman }\rho \\\\ \hline \text{SWE-bench repository tasks} & 0.75 \\\\ \text{HTTP retrieval} & 0.71 \\\\ \text{Airline API} & 0.79 \\\\ \text{Enterprise tasks} & 0.96 \\\\ \hline \end{array}$$ **Workload and policy can be separated** using three comparisons, each holding different factors fixed. **Table R5. Separating workload from retention policy (all 36 fixed-trace pairs have identical request sequences)** $$\begin{array}{lll} \hline \text{Comparison} & \text{Held fixed} & \text{Result for LangGraph} \\\\ \hline \text{Across task families} & \text{Configuration} & \text{0.14, 9.6, and 607 MiB on HTTP, code, enterprise} \\\\ \text{Default vs. minimal} & \text{Task and repeat} & 4.9\text{–}31.9\times\text{ public, }202.6\times\text{ enterprise} \\\\ \text{Fixed trace} & \text{Exact request sequence} & \text{1,969 vs. 1.42 MiB at 400 rounds} \\\\ \hline \end{array}$$ **Growth patterns apply on real tasks, with interaction length as the mediating factor.** $$\begin{array}{lrr} \hline \text{Enterprise pairs} & \text{LangGraph default / minimal} & \text{AutoGen default / minimal} \\\\ \hline \text{Both use all 300 model calls} & 217.0\times\text{ (39 pairs)} & 105.5\times\text{ (35 pairs)} \\\\ \text{Both end earlier} & 32.5\times\text{ (20 pairs)} & 4.3\times\text{ (21 pairs)} \\\\ \hline \end{array}$$ Runs ending earlier take a median of 14–16 model requests. The rise from short to long runs agrees with the fitted exponent near 2 from the fixed-trace study, so the controlled growth pattern matches what real workloads show. Per model request on enterprise tasks, LangGraph's default retains 2.9 MiB after the run and its minimal configuration 15 KiB. An enterprise task is one user turn, and LangGraph's minimal configuration prunes when the turn returns, so during the run its store equals the default's. AutoGen's minimal configuration, which omits the event logger, acts during the run. **Business data is a workload effect that retention cannot remove.** Delivered quotations retain about 20 MiB under both minimal configurations, about 95% of it workspace files. In the controlled file-QA tasks workspace files are negligible, so this source appears only on realistic work. The benchmark's separate workspace and native-state accounting exposes it.

Thank you for your response, which has addressed most of my concerns. I will keep my positive score.

Reviewer_EJc7OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces AgentFootprint, a benchmark for measuring the persistent local storage left behind by LLM agent executions. It combines filesystem accounting, serialization-aware duplication analysis, and conversation-history reconstructability to evaluate eight agent frameworks. Controlled experiments reveal substantial configuration-dependent storage differences, while an analysis of public SWE-bench submissions documents wide variation in exported trajectory sizes. A reference content-addressed compactor demonstrates substantial post-run savings while preserving the tested reconstruction contract.

Critical strengths
  1. Persistent storage is a useful addition to agent evaluation. Distinguishing retained state from inference expenditure draws attention to a resource that matters for long-lived agent deployments.
  2. The measurement methodology contains several careful controls: isolated workspaces, logical extraction from serialized stores, known-duplication calibration, backend replications, and a fixed-trace experiment separating generated content from persistence behavior.
  3. The manuscript documents its assumptions and limitations in detail. The reconstruction checks, restoration tests, and alternative retention configurations make the results interpretable.
Critical weaknesses
  1. The experimental evidence provides limited support for the benchmark’s generalizability and practical value. The controlled evaluation relies heavily on ten synthetic file-QA tasks and several small additional suites. Repetitions and backend replications increase the execution count but provide limited workload diversity. The SWE-bench study measures exported trajectories rather than comparable end-to-end storage footprints and therefore provides only indirect corroboration. Broader, realistic workloads are needed to support the paper’s conclusions.
  2. The headline comparison does not match retained capabilities. Equal task accuracy, R=3, and between-turn continuation do not establish equivalent workflow recovery or debugging capabilities. Consequently, the 15.7× storage difference cannot be attributed entirely to avoidable storage amplification. The appendix shows that LangGraph and LlamaIndex approach OpenAI Agents’ footprint after changing retention policies while preserving the measured reconstruction capability. These results should be central to the main evaluation and should qualify conclusions about framework-level efficiency.
  3. Existing reduction strategies and compression costs are incompletely evaluated. Although Appendix tests selected retention variants, existing framework-level reduction options are not systematically compared across workloads. Whole-store compression is evaluated mainly by compressed size, without measuring compression/decompression overhead or access and recovery latency. A practical comparison between CAS and a restorable compressed archive of the original store is missing. The tradeoff between storage savings and operational performance therefore remains unclear.
  4. The controlled evaluation uses only file-listing, reading, and writing tools, omitting browsing, code execution, external APIs, and multi-agent communication. Observations come from static local files that can be reread, leaving transient or non-reproducible observations untested. This limits the evidence for when persistent retention is necessary and how its storage costs should be weighed against recoverability. The approximately 39-byte write task also makes the reported residue-to-output ratio particularly sensitive to an unusually small denominator.
  5. The reconstructability metric provides limited capability discrimination. R=3 relies on externally supplied call boundaries and a reconstruction contract that excludes system messages. Detailed validation covers three proxied tasks per qualifying framework, while the continuation probe tests nonce recall after an exit between turns. These checks are useful but do not establish self-contained reconstruction of historical requests or recovery of an interrupted workflow. Although the manuscript acknowledges these boundaries, assigning the same score can obscure meaningful capability differences when interpreting storage efficiency.
  6. The paper insufficiently separates agent-specific storage issues from established storage lifecycle management problems, including garbage collection, checkpoint pruning, log rotation, and database compaction. These mechanisms address different sources of accumulation: obsolete retained state, redundant historical versions, and physically unreclaimed space. The appendix partially explores this issue through checkpoint pruning and VACUUM, but the evaluation does not systematically distinguish footprint before and after routine maintenance under matched recovery requirements. Consequently, some reported storage differences may reflect retention and maintenance policies rather than inherent framework inefficiency. The paper should clarify what is specific to LLM agents and evaluate whether conventional lifecycle management substantially changes its conclusions.
Questions for authors
  1. Can the authors evaluate realistic workloads involving code execution, browsing, external APIs, and non-reproducible observations, reporting storage jointly with task success and recovery capability?
  2. How much of the headline storage dispersion remains under matched retention requirements? Why are Appendix’s minimal history-retaining configurations not the primary comparison for history-storage efficiency?
  3. How does CAS compare with a restorable compressed archive in storage savings, compression/decompression overhead, selective-access latency, and restoration time? Can restored stores successfully continue execution?
  4. Can retained stores recover historical call boundaries without the validation proxy? Does the reconstruction contract hold on the long-horizon and heterogeneous task suites?
  5. Do the observed growth patterns persist with changing observations and alternative checkpoint policies? How should configuration-dependent growth be distinguished from a framework-level property?
  6. How much of the measured footprint consists of necessary live state, obsolete historical state, duplicate content, and physically unreclaimed space? Can the authors compare storage before and after checkpoint pruning, log rotation, and database compaction while verifying that the same explicitly defined recovery capabilities remain available?
6 comments
AuthorsOpenReview ↗

Response to Reviewer EJc7 (1/6): realistic workloads

Thank you for the detailed review and for recognizing the measurement controls. Your comments prompted new experiments on each point. The six parts below follow your six questions.

Q1. Can the authors evaluate realistic workloads involving code execution, browsing, external APIs, and non-reproducible observations, reporting storage jointly with task success and recovery capability?

Yes. We added 620 runs on 30 public tasks and 800 runs of 40 mixed business tasks from multiple enterprises, all across eight frameworks with shared tools and budgets. Repository agents edit code and run the official SWE-bench tests [1]. HTTP agents retrieve pages from a frozen Wikipedia corpus to answer HotpotQA questions [2]. Airline agents act on the mutable τ2\tau^2-bench service [3]. The business tasks derive from real cases at multiple enterprises, with names and business values de-identified. Their inputs, skills, tools, and checkers are in the supplement, with identifying names replaced by placeholders.

Table R1. Mean retained MiB over all runs (runs completing the task)

\hline \text{Configuration} & \text{Code (of 20)} & \text{HTTP (of 20)} & \text{API (of 20)} & \text{Enterprise (of 80)} \\\\ \hline \text{LangGraph default} & \text{9.586 (13)} & \text{0.137 (4)} & \text{0.378 (17)} & \text{606.699 (23)} \\\\ \text{LangGraph minimal} & \text{0.301 (11)} & \text{0.028 (4)} & \text{0.037 (20)} & \text{2.995 (26)} \\\\ \text{AutoGen default} & \text{2.423 (12)} & \text{0.035 (4)} & \text{0.264 (18)} & \text{137.900 (22)} \\\\ \text{AutoGen minimal} & \text{0.292 (12)} & \text{0.007 (4)} & \text{0.095 (15)} & \text{3.109 (25)} \\\\ \text{InfiAgent default} & \text{2.710 (17)} & \text{0.214 (4)} & \text{2.058 (17)} & \text{31.031 (75)} \\\\ \text{Agno default} & \text{0.450 (13)} & \text{0.052 (5)} & \text{0.287 (18)} & \text{5.000 (26)} \\\\ \text{LlamaIndex default} & \text{0.377 (10)} & \text{0.010 (3)} & \text{0.061 (20)} & \text{11.094 (23)} \\\\ \text{OpenAI Agents default} & \text{0.326 (12)} & \text{0.034 (2)} & \text{0.042 (17)} & \text{4.258 (24)} \\\\ \text{CrewAI default} & \text{0.826 (14)} & \text{0.019 (2)} & \text{0.051 (17)} & \text{1.771 (32)} \\\\ \text{SmolAgents default} & \text{0.093 (11)} & \text{0.000 (3)} & \text{0.000 (18)} & \text{3.859 (43)} \\\\ \hline \end{array}$$ Storage differences persist on every workload family, and they widen as interactions lengthen, reaching $343\times$ between LangGraph's default and CrewAI's on enterprise tasks. Retention settings leave the model's input unchanged, so a default and its minimal configuration deliver the same up to run-to-run variation (23 and 26 enterprise tasks for LangGraph). Recovery capability is reported in part 4. **Reading the completion column.** Of 288 failed quotation runs, 283 never complete profile validation, the second of seven workflow stages, and hold 60% of enterprise bytes; 195 repeat one identical command 20 or more times in a row (Tables 36 and 38). InfiAgent rebuilds its context every 30 tool actions: its longest such streak is 24, and its quotation requests carry about 72K prompt tokens against 172–307K for the others, whose token-based context limits the adapters raise or disable (Table 37). The enterprise tooling and workflow text were adapted from production agents built on one of the evaluated frameworks, so we compare delivery within a framework and do not rank frameworks by it. **Non-reproducible observations.** We repeated 802 successful read-only calls from the first four frameworks' 140 airline runs against each run's final database. The same arguments returned different content for 196 calls, across 87 runs and seven tasks. On a mutable service, the saved observation is the only record of what the agent saw, which gives retained observations a concrete use beyond rereading static files. **Multi-agent communication.** Our experiments measure single-agent workflows, for two reasons. Among the three production deployments inventoried here and more than ten other enterprises we work with, single-agent workflows are the dominant and growing pattern. Multi-agent systems also typically exchange messages or summaries and rarely the whole trajectory, so each agent persists its own trajectory through the same mechanisms we measure. **The 39-byte denominator.** We agree that this ratio is sensitive by construction. The paper now reports the residue itself, 0.13–0.94 MB per task for the five frameworks that deliver into the workspace, next to the ratio of $3{,}400$–$24{,}000\times$. Realistic deliverables give the other end of the range. Under LangGraph's and AutoGen's minimal settings, delivered quotations retain 21.2 and 20.2 MiB, about 95% of it workspace files (Table 23). [1] Jimenez et al., SWE-bench, ICLR 2024. [2] Yang et al., HotpotQA, EMNLP 2018. [3] Barres et al., $\tau^2$-bench, 2025.
AuthorsOpenReview ↗

Response to Reviewer EJc7 (2/6): matched retention requirements

Q2. How much of the headline storage dispersion remains under matched retention requirements? Why are Appendix's minimal history-retaining configurations not the primary comparison for history-storage efficiency?

You are right that these configurations belong in the main evaluation, and Section 5.1 now reports them next to the headline number. On the same ten file-QA tasks, the spread among seven persisting frameworks falls from 16.4×16.4\times to 6.9×6.9\times when LangGraph, AutoGen, and LlamaIndex use their minimal settings. All 140 runs in that set pass the history and request tests.

Table R2. File-QA: default and minimal configurations in MiB (20 runs each; usage query passes listed as default, minimal)

\hline \text{Framework} & \text{Default} & \text{Minimal} & \text{Ratio} & \text{Rank, default} & \text{Rank, minimal} & \text{Usage query} \\\\ \hline \text{LangGraph} & 5.324 & 0.326 & 16.3\times & 1 & 5 & \text{20, 20} \\\\ \text{AutoGen} & 2.596 & 0.904 & 2.9\times & 2 & 3 & \text{20, 0} \\\\ \text{InfiAgent} & 2.114 & \text{same} & \text{–} & 3 & 1 & 20 \\\\ \text{Agno} & 1.141 & \text{same} & \text{–} & 4 & 2 & 20 \\\\ \text{LlamaIndex} & 0.924 & 0.306 & 3.0\times & 5 & 7 & \text{0, 0} \\\\ \text{OpenAI Agents} & 0.329 & \text{same} & \text{–} & 6 & 4 & 0 \\\\ \text{CrewAI} & 0.325 & \text{same} & \text{–} & 7 & 6 & \text{–} \\\\ \hline \end{array}$$ The ordering changes substantially (Kendall's $\tau = 0.24$ between the two rankings), and LangGraph's minimal setting at 0.326 MiB sits next to OpenAI Agents at 0.329 MiB. The $15.7\times$ default gap therefore reflects retention policy far more than anything intrinsic to LangGraph. The last column shows why a capability check belongs beside the byte count: AutoGen's event log costs 1.69 MiB and holds token-usage receipts that its minimal setting loses, while LangGraph's minimal setting keeps event and usage answers intact. The same comparison on long tasks shows how the gap depends on interaction length. For a set $P$ of task–repeat pairs, we report the ratio of summed retained bytes: $$r = \frac{\sum_{(t,k) \in P} S^{\mathrm{default}}_{t,k}}{\sum_{(t,k) \in P} S^{\mathrm{minimal}}_{t,k}}$$ **Table R3. Enterprise runs paired by task and repeat: mean retained MiB (LG = LangGraph, first four columns; AG = AutoGen, last four)** $$\begin{array}{lrrrrrrrr} \hline \text{Paired runs} & \text{LG }n & \text{Default} & \text{Minimal} & \text{Ratio} & \text{AG }n & \text{Default} & \text{Minimal} & \text{Ratio} \\\\ \hline \text{All pairs} & 80 & 606.7 & 3.00 & 202.6\times & 80 & 137.9 & 3.11 & 44.3\times \\\\ \text{Both use 300 calls} & 39 & 863.9 & 3.98 & 217.0\times & 35 & 204.4 & 1.94 & 105.5\times \\\\ \text{Both end earlier} & 20 & 35.0 & 1.08 & 32.5\times & 21 & 4.13 & 0.96 & 4.3\times \\\\ \text{Both deliver} & 15 & 40.2 & 0.96 & 41.6\times & 13 & 5.23 & 1.24 & 4.2\times \\\\ \hline \end{array}$$ In the second row, every pair issues the same number of model requests (39/39 and 35/35), so the ratio measures persistence at matched interaction length. AutoGen's ratio drops to about $4\times$ on short runs and LangGraph's holds at $32$–$42\times$, which is why we report these rows together. Of the pairs in that row, 23 of LangGraph's 39 and 24 of AutoGen's 35 are quotation runs that never complete profile validation, the second of seven workflow stages, so the long interactions of this suite are mostly agents that spend their budget before that stage (Table 36). The fixed-trace study (part 5) confirms the dependence with identical request sequences. **What the configurations are.** A default is a framework's documented durable-session recipe. CrewAI, SmolAgents, and InfiAgent need no added persistence code. For LangGraph, AutoGen, LlamaIndex, OpenAI Agents, and Agno, the adapter attaches the persistence component from the framework's tutorial, and without it these agents keep nothing on disk (Table 37). AutoGen's minimal configuration omits the event logger and acts during the run. LangGraph's and LlamaIndex's are adapter code that prunes at user-turn boundaries. An enterprise task is one user turn, so the ratios in Table R3 describe what a run leaves behind, and LangGraph's store on disk during the run equals its default's. The defaults remain in the paper because they are the documented starting point and what operators report in production. The minimal configurations show how much of that storage a stated requirement actually needs.
AuthorsOpenReview ↗

Response to Reviewer EJc7 (3/6): CAS and restorable archives

Q3. How does CAS compare with a restorable compressed archive in storage savings, compression/decompression overhead, selective-access latency, and restoration time? Can restored stores successfully continue execution?

We added a complete tar+zstd archive at the same compression level (zstd 19) and compared it with the unchanged CAS implementation on 94 stores: 70 file-QA stores and 24 stores from public tool tasks, with five timing repeats each.

Table R4. Tool-workload stores on Linux: tar+zstd / CAS (medians of per-store medians)

\hline \text{Framework} & \text{Size KiB} & \text{Pack ms} & \text{Restore ms} & \text{Peak RSS MiB} & \text{Logical restore} \\\\ \hline \text{LangGraph} & \text{11.5 / 157.5} & \text{63.2 / 123.1} & \text{1.8 / 7.7} & \text{105 / 28} & \text{3/3, 3/3} \\\\ \text{AutoGen} & \text{9.4 / 124.1} & \text{56.3 / 27.1} & \text{2.3 / 5.7} & \text{104 / 26} & \text{3/3, 3/3} \\\\ \text{InfiAgent} & \text{62.9 / 795.6} & \text{165.3 / 194.2} & \text{11.9 / 84.0} & \text{109 / 43} & \text{3/3, 0/3} \\\\ \text{Agno} & \text{11.4 / 42.2} & \text{104.9 / 131.6} & \text{3.3 / 23.9} & \text{104 / 30} & \text{3/3, 3/3} \\\\ \text{LlamaIndex} & \text{4.3 / 15.6} & \text{46.9 / 31.8} & \text{1.6 / 1.6} & \text{104 / 25} & \text{3/3, 3/3} \\\\ \text{OpenAI Agents} & \text{4.8 / 43.0} & \text{47.2 / 23.9} & \text{1.3 / 4.2} & \text{104 / 25} & \text{3/3, 3/3} \\\\ \text{CrewAI} & \text{7.3 / 19.2} & \text{76.5 / 54.2} & \text{6.1 / 9.5} & \text{104 / 25} & \text{3/3, 3/3} \\\\ \text{SmolAgents} & \text{0.5 / 0.5} & \text{57.5 / 31.0} & \text{1.4 / 0.7} & \text{103 / 24} & \text{3/3, 3/3} \\\\ \hline \end{array}$$ **Storage savings.** Summed over stores, tar+zstd produces 2.2 MB against 12.7 MB for CAS on the file-QA stores, and 0.50 MB against 4.40 MB on the tool stores. It is smaller on 92 of 94 stores and restores all 94 byte-for-byte, which makes it the right baseline for cold retention. This result fits expectations, since whole-store compression exploits similarity across near-duplicate checkpoints that exact content addressing cannot. **Overhead.** Packing a store takes tens to hundreds of milliseconds for both methods. CAS uses less memory, at 24–43 MiB against 103–109 MiB, and packs faster on five of the eight frameworks here. Tables 33 and 34 give the full costs on both hosts, including disk high-water use. **Restoration.** Tar+zstd restores in 1.3–11.9 ms and CAS in 0.7–84 ms on these stores. CAS restores logical content and schema for 81 of 94 stores. The 13 failures are all InfiAgent stores, whose full-text-index tables the compactor omits. **Selective access.** Framework SDKs open their native SQLite or JSONL files directly, so both methods restore a store before use, and full restoration time bounds the cost of reading any part of it. We did not build an indexed selective reader, so we have no separate latency figure for one. **Continued execution.** Native SDK loading matches the raw store on all 21 tested stores under both methods. Scripted next-turn continuations agree between raw and archived stores on the 12 public stores and 3 Agno stores (next request and persisted history). Testing also exposed one CAS defect: a dropped trailing newline makes the next append to InfiAgent's JSONL log concatenate records. Tar-restored InfiAgent stores continue with intact logs. Details are in Appendix U.8.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: overview of the revision

We thank the three reviewers for their careful reading. The reviews converge on three requests: evidence from realistic workloads, comparisons under matched retention settings, and a clearer account of what retained bytes are for. We ran new experiments on all eight frameworks and revised the paper around them. AgentFootprint remains a benchmark, and the new results extend its original measurements.

Table G1. Evidence added in this revision

\hline \text{Study} & \text{Scale} & \text{Main result} \\\\ \hline \text{File-QA settings} & \text{220 runs} & \text{Default spread }16.4\times\text{, minimal }6.9\times \\\\ \text{Public tool tasks} & \text{620 runs, 30 tasks} & \text{LangGraph default/minimal }4.9\text{–}31.9\times \\\\ \text{Enterprise tasks} & \text{800 runs, 40 tasks} & \text{Equal requests: }217\times\text{ LangGraph, }105\times\text{ AutoGen} \\\\ \text{Capability tests} & \text{all runs above} & \text{Request reconstruction 199/220, 533/620, 672/800} \\\\ \text{Fault recovery} & \text{398 conditions, 391 kills} & \text{94 kills end in a completed task} \\\\ \text{Fixed-trace growth} & \text{108 trajectories} & \text{LangGraph 1,969 MiB default, 1.42 MiB minimal} \\\\ \text{Complete archives} & \text{94 stores, 940 trials} & \text{tar+zstd smaller on 92/94, exact on 94/94} \\\\ \text{Production} & \text{3 deployments} & \text{171 GB; 90 GB plus 120 GB backups; 25 GB} \\\\ \hline \end{array}$$ **The main new finding.** Long interactions amplify persistence. On enterprise tasks where a default and its minimal setting both spend the full 300-call budget, request counts match exactly and the default retains $217\times$ (LangGraph) and $105\times$ (AutoGen) as many bytes. On short runs the ratios are $32\times$ and $4.3\times$. Storage is incurred whether or not the work succeeds: undelivered runs hold 87% of all enterprise bytes. Of 288 failed quotation runs, 283 never complete profile validation, the second of seven workflow stages, and 195 repeat one identical command 20 or more times in a row, which full-history persistence stores each time (Tables 36 and 38). After history reduction, business files remain, at about 20 MiB per delivered quotation. **How to read the configurations** - **Default and minimal.** A *default* is a framework's documented durable-session recipe. CrewAI, SmolAgents, and InfiAgent need no added persistence code; for the other five, the adapter attaches the persistence component from the framework's tutorial. Table 37 lists every configuration (Section 4.2). - **When minimal configurations act.** AutoGen's acts during the run. LangGraph's and LlamaIndex's prune at user-turn boundaries. Enterprise, repository, and HTTP tasks are one user turn, so there the ratios describe what a run leaves behind. At a process kill LangGraph's minimal store is as large as its default's, so its fault-recovery counts do not test recovery from a pruned store. - **Metrics.** $\alpha$ in Table 1 comes from the scripted protocol, with two workload requests per round. $R$ grades the structure of the retained record, and content is checked by the history and request tests. - **Enterprise delivery.** The enterprise tooling and workflow text were adapted from production agents built on one of the evaluated frameworks, so delivery counts are not a framework ranking. **Where to find the changes** - Section 5.1 now reports minimal history-retaining configurations next to the headline comparison. - Section 5.6 adds the 108-trajectory growth study. - Section 6 and Tables 2–3 are new: realistic workloads, retained capabilities, and fault recovery. - Section 7 adds the complete-archive baseline. - Section 8 adds the production and single-agent discussion. - Appendix U (U.1–U.10) gives protocols and full tables, and Appendix C accounts for all 2,691 executions. - The supplement now includes the inputs, skills, tools, and checkers of the 40 enterprise tasks, per-run records for every run, and scripts that recompute the reported numbers. Retention settings are labelled *default* and *minimal* throughout. The main text stays within 10 pages. Each reviewer's thread below answers every question with the relevant tables in place.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.