- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a dual-layer, training-free memory system for LLM agents together with a stability analysis of its memory statistics. Retrieval over a case store is governed by four statistics (median utility, execution-time variance, mean similarity, success rate). A shared layer aggregates global experience and a private layer tracks user-specific patterns, each discretizing a linear time-varying ODE, with coarse-step RK4 for the shared layer, fine-step Euler for the private layer, and the private-layer regularization anchored to the shared state. Retrieval proceeds through semantic pre-filtering, utility gating with a fused threshold, temporal re-ranking, and top-k selection, all with frozen encoders. Theoretically, the paper proves convergence of the single-layer statistics via Lyapunov and Grönwall arguments (Theorem 1) and an ISS bound for the coupled system (Theorem 2).
Critical strengths
- The system itself is a reasonable, reproducible, training-free retrieval pipeline.
- Disclosing all hyperparameters and pseudocode reflect a careful empirical setup.
- If the cross-task token savings (51-54%) hold up, they are relevant for cost-sensitive deployments.
Critical weaknesses
- The theoretical contribution does not hold in the form claimed. First, the analyzed system is a four-dimensional linear time-varying ODE, whose convergence is textbook material; deriving it via Lyapunov functions and Grönwall's inequality is not a substantive contribution.
- The paper provides no evidence that any existing system's statistics diverge or fail due to instability, and no experiment links statistic convergence to end-task gains. The guarantees also cover only four scalars, not the memory contents themselves.
- The empirical support is thin and mismatched to the claims. LoCoMo contains only 10 dialogues, the margin over O-Mem is 1.17 F1 with per-category margins of 0.27 to 1.96, and no variance, repeated runs, or significance tests are reported. The component ablation is run entirely on a different domain, two Kaggle tabular tasks with single runs and several variants failing outright, so the central claim that ODE dynamics improve conversational memory is never ablated on the conversational benchmark. A-Mem's re-run score of 13.83, below Full Context at 27.22 and far below its published numbers, suggests a reimplementation problem. The case study's average F1 (33.36 vs. 10.65 for RAG) is inconsistent in magnitude with the main table, raising concerns about selective presentation.
Questions for authors
- Corollary 1 claims an global truncation error for the shared layer, but Algorithm 1 evaluates k1 through k4 on four different historical snapshots rather than classical RK4 substeps. Please provide a proof of the error bound for this modified scheme or withdraw the claim.
- Please demonstrate that the failure mode the theory guards against actually exists: show an existing system whose memory statistics drift over long horizons and degrade retrieval, and report the trajectories of your statistics versus case count together with their correlation to retrieval quality.
- Please provide the missing ablations on LoCoMo itself: performance without the shared layer, with plain EMA updates in place of RK4, and with the ODE removed entirely (the four-stage retrieval with static thresholds only).
3 comments
Response regarding the experimental section
Thank you for your comments.
(1) Regarding LoCoMo ablations and repeated experiments, our response is as follows:
We compared the full configuration with three ablation configurations, running each three times. w/o RK4 replaces the shared-layer RK4 update with EMA; w/o Euler disables the private-layer Euler update; and w/o Shared & Euler removes the shared layer and also disables the private-layer Euler update.
The F1 scores, means, and sample standard deviations are:
| Configuration | Run 1 | Run 2 | Run 3 | Mean ± SD |
|---|---|---|---|---|
| Full | 52.3653 | 52.2789 | 52.4863 | 52.3768 ± 0.1042 |
| w/o RK4 (replaced with EMA) | 51.3572 | 51.3377 | 50.8629 | 51.1859 ± 0.2799 |
| w/o Euler | 51.3018 | 51.0617 | 51.0527 | 51.1387 ± 0.1413 |
| w/o Shared & Euler | 50.2604 | 50.2476 | 50.2699 | 50.2593 ± 0.0112 |
The full configuration scores highest in all three runs, exceeding the EMA variant by 1.1909 F1 percentage points on average. Disabling the private-layer Euler update reduces mean F1 by 1.2381 percentage points. Removing the shared layer and also disabling Euler updates reduces it by 2.1175 percentage points.
(2) Regarding the A-Mem reproduction score, our response is as follows:
The F1 score of 13.83 comes from our local evaluation using gpt-4o-mini. The revised manuscript identifies this source and removes the explanation that directly attributed the difference from published results to backbone-model capability.
(3) Regarding the difference between the case-study and main-table scores, our response is as follows:
The main table reports results over all 1,540 valid LoCoMo questions. The case study selects two examples from each of the four question categories, giving eight examples in total. The scores of 33.36 and 10.65 are the mean F1 scores over those eight examples. The case-study table illustrates specific answers and error types, and the revised text and caption explicitly state this scope.
Regarding Some Supplementary Replies
Thank you for your comment. Regarding the actual role of statistical updates and the contribution of each component in cross-task ablations, our response is as follows:
(1) Under the same retrieval pipeline, updating statistics improves answer accuracy
The controlled synthetic experiments provide the same fact writing and question sequences for the three configurations, and keep the semantic pre-screening, utility gating, temporal reordering, and top-3 selection pipeline consistent. The frozen configuration fixes the four statistics while retaining memory writing and subsequent retrieval, in order to compare the impact of statistical updates on the final answer.
Under the initial condition, the EM of decay update and frozen statistics are 95% and 60%, respectively; after 20,000 statistical updates per user, they are 94% and 59%, respectively. Updating statistics brings a 35 percentage point improvement under both conditions.
The constant-gain control further tests the choice of update strength. It is 95% under the initial condition, the same as decay update, and under the pre-update condition the two are 88% and 94%, respectively. This result supports that the decay configuration better maintains answer performance after long-term updates in this task.
(2) The shared layer and update method improve accuracy, while adaptive triggering mainly saves tokens
Table 3 sets independent controls with the full configuration as the reference: removing the shared layer tests the role of shared experience, changing the shared layer to Euler tests the role of the update rule, and disabling adaptive triggering tests the role of the triggering mechanism. Vanilla R&D-Agent is listed separately as a system baseline, and the rows are not adding components row by row from the previous row.
After removing the shared layer, the average accuracy decreases by 1.40 percentage points; after changing the shared layer update to Euler, it decreases by 1.90 percentage points. The benefit of adaptive triggering is mainly token saving: after it is enabled, the combined consumption across the two tasks decreases from 127.2K to 60.7K, a reduction of 52.3%, while the average accuracy remains comparable. Therefore, the shared layer and the update method help the agent perform the task better, and adaptive triggering helps it complete the task with less consumption.
Response Regarding the Theoretical Part
Thank you for your comments.
(1) Regarding the theoretical contribution, our response is as follows:
The revised manuscript provides a discrete stability proof for Algorithm 1. It accounts for two features of the update: the four stages use different historical inputs, and the private-state update depends on the shared state. For the step sizes used in the paper, we derive bounds on the four retrieval statistics and conditions for their convergence. Lyapunov, Grönwall, and ISS are the standard tools used in the analysis.
(2) Regarding the fourth-order error bound in Corollary 1, our response is as follows:
We have rewritten the proof by expanding the four-stage update actually performed by the algorithm, replacing the original appeal to classical RK4 fourth-order error bounds. Let m denote the number of shared-layer updates and em=Ss,m−S∗. At step size Hs=1, the update satisfies the exact recurrence:
em+1=Amem+∑j=14wm,jλm,j(Xm,j−S∗).
The coefficients satisfy 0.4≤Am≤0.93, wm,j≥0, and ∑jwm,j≤1. If the historical inputs satisfy the uniform bound ∥Xm,j−S∗∥≤Ms, then:
∥em+1∥≤0.93∥em∥+1+m/1000.5Ms.
This inequality ensures that the statistics remain bounded. Expanding the recurrence gives a geometrically decaying initial-state term and an accumulated input contribution that also tends to zero. Thus, if shared updates continue indefinitely, the statistics converge to S∗ at the fixed step size, with a general O(1/m) upper bound.
The private layer uses Euler updates with hp=0.05, giving a state coefficient in [0.97,0.995]. Bounded observations and shared states ensure bounded private states. If both the user's update count and the shared-update index it observes tend to infinity, the private state also converges to S∗. The full coefficient derivation and proof are provided in the appendix.