- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets two failure modes of experience-reusing tool-use agents: (i) copying stale, episode-specific literals (IDs, phone numbers) from retrieved demonstrations into live API calls, and (ii) issuing state-changing calls before the required argument has been observed in the current episode. The method splits memory into a Static Procedural Bank of abstract workflow graphs (literals replaced by typed slots) and an online Dynamic Fact Ledger that records typed values from the current task with timestamp and provenance, encoded as a temporal graph. A trained dual-head readout then, per slot, estimates whether supporting evidence is present (Presence Head, threshold 0.75), emitting an UNRESOLVED "sense first" directive if not, and otherwise ranks type-compatible ledger candidates (Source Ranking Head); the result is injected into the frozen LLM's prompt. On AppWorld test-normal (168 tasks) the method solves 82 tasks vs. 73 for ReMe and 70 for ExpeL, with invalid-call rate 0.21% vs. 1.10%/0.72%; it also reports gains on ALFWorld, a 90-episode ScienceWorld panel and five AppWorld backbones (vs. ExpeL), plus ablations, mechanism-level error counts (e.g., old-value copies 14 → 2) and a threshold sweep.
Critical strengths
- The paper identifies a concrete and practically relevant gap in experience reuse for tool-use agents, and states it precisely. The prefix-causal grounding criterion (§3.2) turns "reuse the procedure, not the values" into a checkable requirement on each dynamic argument, and the split into missing evidence (Presence Head) and choice among observed candidates (Source Ranking Head) is a clean decomposition of the problem. This framing is useful independently of the specific architecture, and it is the main reason I see potential in the paper.
Critical weaknesses
-
The paper does not show that its gains come from the mechanism it proposes. The evidence for the motivating failures is thin: all examples are schematic (Figs. 1–2, Appendices C and H), and over 168 tasks ExpeL makes only 14 old-value copies and 18 state changes without evidence. These errors also cannot account for the readout's gain: from the matched static + dynamic control to the full method, success rises by 14 tasks while the two error types fall by 4 and 6, at most 10 tasks. Simpler remedies are not compared (Question 3): the control without any directive already cuts copies from 14 to 6, and neither a non-learned readout nor a rule-based pre-dispatch check is evaluated. The central claim, that the learned readout is what makes grounding work, is therefore unsupported.
-
The gains may be within run-to-run variation. The direction is consistent across environments and backbones, but each condition is a single run, and in the paper's own two-attempt protocol (Table 16) ExpeL's outcome changes on 34 of 168 tasks, more than the method's 12-task gain over ExpeL and 9-task gain over ReMe (Question 4). Whether the gains hold with the official AppWorld agent and faithfully reproduced baselines is also unclear (Question 6).
Questions for authors
- Is the directive in Listings 2–3 the prompt used in the reported experiments? Listing 2 shows a fully specified lookup (
apis.venmo.search_transactions(query="Mark")), and Listing 3 fills{sensing_api}and{search_key}. I could not find this template in the supplementary code. For an unresolved slot,experience_instantiation_runtime.py(around l. 414–429) lists only the GET routes from which the value was read in the historical trajectory; these routes come from a fixed dataflow rule inexperience_instantiation_data.py(around l. 303–311), not from the readout, and the emitted text states "No query parameters are inferred". Which code path and prompt variant produced the main tables? - Can you show recorded (not schematic) traces of stale-value copying and premature state changes by the evaluated backbones? For tasks solved by the full method but not by ExpeL, what caused ExpeL's failures?
- What success rate does a non-learned readout achieve with the same prompt and directive? How does ExpeL perform with a rule-based pre-dispatch check that blocks calls whose ID-typed arguments do not appear in the current goal or observations? What exactly does the "Static + dynamic, no final readout" arm place in the prompt? The base prompt (Listing 1, rule 4) already tells the agent not to copy IDs and to query unobserved entities first; do all arms use it, and what fraction of UNRESOLVED directives is followed by a lookup before the state-changing call?
- Can you report several independent runs with confidence intervals? Is attempt 2 in Table 16 independent of attempt 1?
- How were "confirmed old-value copies" and "state changes without evidence" identified: automatically or manually?
- Is the AppWorld agent the official ReAct implementation, and why was a 40-step horizon chosen when the official agent allows 100 LLM calls? The no-memory agent solves 28.6% of test-normal, below the 48.8% TGC the AppWorld paper reports for GPT-4o; do the gains persist with the official setup? Relatedly, ExpGraph gains +11.3 pp (AppWorld) and +5.2 pp (ALFWorld) over no-memory here, versus roughly +23 to +27 pp and +23 to +40 pp in its own paper (arXiv 2605.30712, listed there as "ExpHarness"). Was its full system, including the trained retrieval copilot, implemented?
- By which criterion, and on which held-out set, was τ_pres = 0.75 chosen (§4.4)? In the test-set sweep (Table 11), 0.75 coincides with the best success count; please confirm that the selection did not use test outcomes.
Suggestions: Figures 1–2 are very dense and would benefit from simplification, and the "causal" terminology should either be justified or dropped, since the paper itself states that its graph edges carry no causal meaning.