- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AgentFootprint is a benchmark that measures different aspects of storage, including total size, duplication, growth, compressibility, and history reproducibility. The authors evaluate eight agent frameworks using the same LLM backend and controlled tasks. Their results show that frameworks with similar accuracy and recovery capabilities can differ more than 10 fold in storage usage, and some frameworks exhibit nearly quadratic storage growth.
Critical strengths
The proposed benchmark covers several useful aspects of storage that other benchmark miss, and the evaluation covers eight popular agent frameworks, and the fixed trace experiment helps separate storage overhead from agent behavior. The results provide useful evidence that similar task performance does not necessarily require similar storage costs.
Critical weaknesses
-
The results depend heavily on storage settings. The paper shows that changing these settings can greatly reduce storage, so it is unclear whether the differences come from the frameworks or their configurations.
-
The R score mainly checks whether conversation history can be recovered. However, it does not fully measure whether agents can recover their internal states or debugging information.
-
The proposed compression method is only tested after agent execution. The paper does not measure its runtime or memory overhead during actual agent tasks.
Questions for authors
How would the framework rankings change if all frameworks used similar storage settings? Would the reported gain still hold?
Why does the R score focus mainly on conversation history? Could the authors also test whether frameworks can recover complete agent states?
What would be the runtime and computational overhead of using content-addressed storage during agent execution? Would the storage savings still be significant?
3 comments
Response to Reviewer gsbq (2/3): beyond conversation history
Q2. Why does the R score focus mainly on conversation history? Could the authors also test whether frameworks can recover complete agent states?
We used conversation history as the shared reference because every persisting framework can be scored against it in the same way. grades the structure of the retained record: whether the store keeps a complete record per model call (Section 3.2). CrewAI's per-task records score and pass the history and request tests on finished runs. To cover agent state, the revision adds direct tests: self-contained reconstruction of full historical requests, event and token-usage audit queries, and restart after a process kill.
Recovery test. On nine public tasks we kill the agent process at three points, then restart a fresh process from native state and let it finish the task. F1 follows a committed native state, F2 follows a tool return before persistence, and F3 follows a side effect before acknowledgement. The study covers 398 fault conditions and reaches 391 kills.
Table R3. Airline service: tasks completed after restart / process kills
\hline \text{Configuration} & \text{F1} & \text{F2} & \text{F3} \\\\ \hline \text{LangGraph default} & \text{4/6} & \text{5/6} & \text{2/4} \\\\ \text{LangGraph minimal} & \text{5/6} & \text{5/6} & \text{2/4} \\\\ \text{OpenAI Agents default} & \text{4/6} & \text{4/6} & \text{2/4} \\\\ \text{AutoGen default} & \text{6/6} & \text{0/6} & \text{0/3} \\\\ \text{AutoGen minimal} & \text{4/6} & \text{0/6} & \text{0/4} \\\\ \text{InfiAgent default} & \text{4/6} & \text{0/6} & \text{0/3} \\\\ \text{Agno default} & \text{5/6} & \text{0/6} & \text{0/3} \\\\ \text{LlamaIndex default} & \text{4/6} & \text{0/6} & \text{0/4} \\\\ \text{LlamaIndex minimal} & \text{4/6} & \text{0/6} & \text{0/4} \\\\ \text{CrewAI default} & \text{5/6} & \text{0/6} & \text{0/4} \\\\ \text{SmolAgents default} & \text{–} & \text{0/6} & \text{0/4} \\\\ \hline \end{array}$$ On the airline service, nearly every framework continues after a committed state (F1). Mid-tool faults separate them: only LangGraph and OpenAI Agents complete tasks after F2 or F3, on the airline service and on repository tasks alike, where each completes 3–4 tasks per fault type. AutoGen, Agno, InfiAgent, and LlamaIndex retain native history at the crash point and complete no airline task after F2 or F3. LangGraph's two rows are close because at a kill its minimal store is as large as the default's (Appendix U.6), so these tests do not separate its two configurations. So a framework can score the top $R$ and pass every history test and still be unable to resume interrupted work. This is exactly the gap you identified, and it is why these are now separate tests. **Table R4. Native reconstruction across workloads (passes / runs)** $$\begin{array}{lrr} \hline \text{Workload} & \text{History} & \text{Full request} \\\\ \hline \text{File-QA} & \text{199/220} & \text{199/220} \\\\ \text{SWE-bench} & \text{174/200} & \text{173/200} \\\\ \text{HTTP} & \text{180/200} & \text{180/200} \\\\ \text{Airline API} & \text{180/220} & \text{180/220} \\\\ \text{Enterprise} & \text{672/800} & \text{672/800} \\\\ \hline \end{array}$$ The request test regenerates system instructions, tool schemas, generation settings, and request order from native state and the common program, with no proxy-supplied task data. A retained store can also differ on audit: AutoGen's default and minimal settings both pass history and request tests on all 80 enterprise runs, and only the default retains token-usage receipts. Debugging and audit needs can therefore be matched against specific channels (Section 6, Appendices U.3 and U.6).Response to Reviewer gsbq (3/3): runtime overhead
Q3. What would be the runtime and computational overhead of using content-addressed storage during agent execution? Would the storage savings still be significant?
CAS in this paper runs after a task finishes, and we did not build a write-path integration, so we have no measurement of its overhead during execution. Section 7 now presents CAS as a post-run reference, and we report three things that bear directly on your question.
1. The cost of the post-run pass. We measured packing, restoration, peak memory, and temporary disk use on 94 stores, 940 timing trials in total, against a complete tar+zstd archive at the same compression level.
Table R5. Post-run cost on tool-workload stores (Linux): tar+zstd / CAS, medians
\hline \text{Framework} & \text{Size KiB} & \text{Pack ms} & \text{Restore ms} & \text{Peak RSS MiB} \\\\ \hline \text{LangGraph} & \text{11.5 / 157.5} & \text{63.2 / 123.1} & \text{1.8 / 7.7} & \text{105 / 28} \\\\ \text{AutoGen} & \text{9.4 / 124.1} & \text{56.3 / 27.1} & \text{2.3 / 5.7} & \text{104 / 26} \\\\ \text{InfiAgent} & \text{62.9 / 795.6} & \text{165.3 / 194.2} & \text{11.9 / 84.0} & \text{109 / 43} \\\\ \text{Agno} & \text{11.4 / 42.2} & \text{104.9 / 131.6} & \text{3.3 / 23.9} & \text{104 / 30} \\\\ \text{LlamaIndex} & \text{4.3 / 15.6} & \text{46.9 / 31.8} & \text{1.6 / 1.6} & \text{104 / 25} \\\\ \text{OpenAI Agents} & \text{4.8 / 43.0} & \text{47.2 / 23.9} & \text{1.3 / 4.2} & \text{104 / 25} \\\\ \text{CrewAI} & \text{7.3 / 19.2} & \text{76.5 / 54.2} & \text{6.1 / 9.5} & \text{104 / 25} \\\\ \hline \end{array}$$ A post-run pass costs tens to hundreds of milliseconds per store. Complete archives are smaller on 92 of 94 stores and restore all 94 byte-for-byte, while CAS uses far less memory (24–43 MiB against 103–109 MiB). **2. The storage reduction from retention policy.** We measured default and minimal configurations on the same tasks, under the same model and budgets. **Table R6. Enterprise tasks: retention after the run under default and minimal settings (mean MiB per run)** $$\begin{array}{lrrrr} \hline \text{Configuration} & \text{Default} & \text{Minimal} & \text{Ratio} & \text{Tasks delivered, default / minimal} \\\\ \hline \text{LangGraph} & 606.7 & 3.00 & 202.6\times & \text{23 / 26} \\\\ \text{AutoGen} & 137.9 & 3.11 & 44.3\times & \text{22 / 25} \\\\ \hline \end{array}$$ AutoGen's minimal configuration omits the event logger and acts during the run. LangGraph's and LlamaIndex's are adapter code that prunes at user-turn boundaries. An enterprise task is one user turn, so Table R6 reports what a run leaves behind, and LangGraph's store during the run equals the default's. Where each round is a user turn, pruning also lowers the live store: in the 400-round fixed-trace study, LangGraph's store before its end-of-run VACUUM is 22.9 MiB under the minimal setting and 1,973 MiB under the default; AutoGen's is 196 MiB and 570 MiB. Reducing storage inside one long turn therefore needs a write-path policy, which neither CAS nor these configurations provide (Sections 4.2, 6, and 8). **3. What an in-execution CAS would need to deliver.** A write-path CAS would have to handle the same content that drives these defaults: repeated serialization of a growing history. After pruning, little exact repetition remains to deduplicate. LangGraph's minimal store has a duplication factor of 1.52 and LlamaIndex's 1.54, so the additional saving from write-path deduplication on top of a minimal configuration would be modest. It would also need its own deletion and recovery design for shared objects (Appendix A).
Response to Reviewer gsbq (1/3): rankings under similar settings
Thank you for the concise review and for identifying the three issues that matter most. The three parts below answer your questions in order.
The ranking changes substantially, and the storage reduction from a minimal setting holds on every task family and grows with task length. Section 5.1 now reports this next to the headline number. On the ten file-QA tasks, we ran each framework's default and, for three frameworks, a minimal configuration that retains conversation history: LangGraph keeps the newest checkpoint, AutoGen omits its event log, and LlamaIndex keeps the latest context.
Table R1. File-QA ranking under default and minimal settings (mean MiB; 20 runs each; AutoGen's test counts are default / minimal)
\hline \text{Framework} & \text{Default MiB} & \text{Rank} & \text{Minimal MiB} & \text{Rank} & \text{History and request tests} \\\\ \hline \text{LangGraph} & 5.324 & 1 & 0.326 & 5 & \text{20, 20} \\\\ \text{AutoGen} & 2.596 & 2 & 0.904 & 3 & \text{19, 19 / 20, 20} \\\\ \text{InfiAgent} & 2.114 & 3 & 2.114 & 1 & \text{20, 20} \\\\ \text{Agno} & 1.141 & 4 & 1.141 & 2 & \text{20, 20} \\\\ \text{LlamaIndex} & 0.924 & 5 & 0.306 & 7 & \text{20, 20} \\\\ \text{OpenAI Agents} & 0.329 & 6 & 0.329 & 4 & \text{20, 20} \\\\ \text{CrewAI} & 0.325 & 7 & 0.325 & 6 & \text{20, 20} \\\\ \hline \end{array}$$ LangGraph moves from largest to fifth of seven. Kendall's $\tau$ between the two orderings is 0.24, and the spread falls from $16.4\times$ to $6.9\times$. LangGraph's, LlamaIndex's, and OpenAI Agents' minimal forms converge at 0.31–0.33 MiB. So your concern is correct: much of the difference comes from configuration. The paper now reports each result with the retention setting attached, and we read a framework's position as a property of that setting. Table 37 defines every measured configuration. A default is a documented durable-session recipe: for LangGraph, AutoGen, LlamaIndex, OpenAI Agents, and Agno, the adapter attaches the persistence component from the framework's tutorial, and without it these agents keep nothing on disk. LangGraph's and LlamaIndex's minimal configurations are adapter code that prunes at user-turn boundaries, and AutoGen's omits the event logger. The differences we report are therefore differences between recipes, which is the reading your question pointed to. The gain from a retention setting also grows with task length. **Table R2. Default-to-minimal ratio of retained bytes, $r = \sum S^{\mathrm{default}} / \sum S^{\mathrm{minimal}}$ over the same tasks and repeats** $$\begin{array}{lrr} \hline \text{Workload} & \text{LangGraph} & \text{AutoGen} \\\\ \hline \text{File-QA} & 16.3\times & 2.9\times \\\\ \text{HTTP retrieval} & 4.9\times & 5.0\times \\\\ \text{Airline API} & 10.2\times & 2.8\times \\\\ \text{SWE-bench repository tasks} & 31.9\times & 8.3\times \\\\ \text{Enterprise, all pairs} & 202.6\times & 44.3\times \\\\ \text{Enterprise, both end earlier} & 32.5\times & 4.3\times \\\\ \text{Enterprise, both use all 300 calls} & 217.0\times & 105.5\times \\\\ \hline \end{array}$$ Retention settings leave the model's input unchanged, so delivery agrees within run-to-run variation: 26 vs. 23 enterprise deliveries for LangGraph and 25 vs. 22 for AutoGen. An enterprise task is one user turn, and LangGraph's pruning runs when the turn returns, so the enterprise ratios describe what a run leaves behind (part 3).