- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes OnDevMem, a long-term memory framework for fully local LLM agents. It combines four components: topic-aware segmentation into Evidence Memory (EM) records that preserve source text and provenance; locally generated Semantic Signatures (SS); dual-channel TF-IDF retrieval that combines source-text and signature relevance; and a maintenance controller that schedules semantic enrichment using foreground activity, resource pressure, and maintenance debt.
On LoCoMo, the paper evaluates Jetson AGX Orin with Qwen3.5-4B and Jetson Thor with Qwen3.5-35B. It reports judge-accuracy improvements of 11.6–14.5 percentage points over the strongest evaluated baseline in each deployment and memory-construction time reductions of 88.4–92.5% relative to the fastest baseline. Additional experiments examine architectural choices, streaming maintenance, constrained context budgets, and a small LongMemEval subset.
Critical strengths
-
The deployment setting is practically relevant. Fully local memory systems must preserve useful information while sharing limited compute with foreground inference. Evaluating complete pipelines on two real Jetson deployments provides useful evidence about their behavior under these constraints.
-
The evaluation extends beyond aggregate accuracy and construction time. It includes architecture ablations, streaming experiments that directly measure maintenance interference, and a context-budget study. In particular, the source-context ablation supports preserving original evidence, and the streaming results demonstrate the importance of foreground-aware maintenance admission.
Critical weaknesses
- The motivation does not sufficiently justify the specific architectural choices or their trade-offs.
Sections 2–3 provide plausible reasons to preserve source evidence, generate additional retrieval cues, and defer expensive maintenance. However, the connection between these general motivations and the particular combination of topic-aware segmentation, a five-field signature, separate retrieval indices, and a multi-signal controller remains underdeveloped. The paper does not sufficiently characterize which limitations of existing systems require these choices, or when their additional complexity and processing costs are warranted over simpler alternatives.
For example, the claim that small local models lose useful information during memory construction is not supported by a direct analysis of what evidence is lost in the evaluated baselines. Table 2B supports retaining original source context within OnDevMem, but does not diagnose the causes of the baseline systems’ failures. For semantic enrichment, Table 2C compares summary-only and full signatures, but does not establish which additional fields provide useful information or whether their benefits justify their generation and maintenance costs. Likewise, comparing fused retrieval with each channel individually does not establish the advantage of maintaining separate indices over indexing the same source text and signature together.
The controller results also require a clearer account of the intended operating trade-off. Table 6 shows that debt control reduces enrichment delay and pending work, while Table 5 reports lower foreground latency and higher observed accuracy when debt control is removed. This demonstrates a maintenance-progress trade-off, but the evaluation does not establish under what workload or freshness requirement the improved maintenance progress warrants the foreground cost. Overall, the reader is given plausible reasons for the components, but insufficient evidence explaining why this particular architecture is the appropriate design.
- The evaluation does not establish robust incremental benefits over a simpler source-preserving retrieval pipeline.
In Table 2C, accuracy increases from 52.87% with source-only retrieval to 53.52% with summary-based enrichment and 54.30% with full signatures. Thus, full signatures improve accuracy by 1.43 percentage points over source-only retrieval and 0.78 percentage points over summary-based enrichment. Token F1 also improves, but Tables 1–2 do not report uncertainty estimates or clearly specify repetition counts. The variability reported for the separate streaming experiment in Table 5 does not establish the stability of these architecture-ablation differences. Construction costs for the source-only, summary-only, and full-signature variants are also missing, leaving their incremental quality–cost trade-offs unclear.
The comparison with existing memory systems demonstrates useful end-to-end gains, but does not resolve whether the proposed architecture offers sufficient advantages over straightforward chunked retrieval. Table 2A compares topic-aware segmentation with turn-pair and session-level units, but omits fixed-size chunks of comparable length. Table 2C already shows that source-only retrieval is a strong internal alternative, making a matched fixed-window, source-only TF-IDF baseline particularly important.
Without this comparison and uncertainty estimates for the key ablations, it remains unclear how much of the benefit requires topic-aware boundaries, semantic enrichment, or separate-index fusion, rather than simply preserving and retrieving source text. These gaps concern the core architectural contribution and require additional controlled evaluation to resolve.
Although the paper addresses a relevant deployment setting and reports useful empirical results, I do not consider the central architectural contribution sufficiently established. The missing comparisons and insufficient characterization of the design trade-offs leave its added value over simpler alternatives unresolved. These concerns affect the main contribution and require substantial additional validation.
Questions for authors
Clarification and additional evidence that could change my assessment:
-
Can you provide uncertainty estimates for the accuracy and Token F1 differences in Table 2, and clarify the repetition counts and generation settings used for Tables 1–2? Please distinguish uncertainty across evaluation questions from variability introduced by memory construction, answer generation, and judging. If using paired bootstrap confidence intervals, please explain how dependence among questions from the same conversation is handled.
-
Can you compare topic-aware segmentation with fixed-window segmentation at comparable segment lengths, using source-only retrieval and SS-based fusion under the same TF-IDF retriever, top-10 limit, and 4,096-token evidence budget? Reporting construction cost alongside answer quality would help determine whether the additional processing is justified.
-
What advantage do separate source and signature indices provide over a single TF-IDF index built from concatenated source text and signatures? A controlled comparison, supplemented by evidence-retrieval metrics such as Recall@10 or MRR and an analysis of where the two channels succeed or fail differently, would help establish the value of the proposed separation.
7 comments
Architectural Comparisons and Construction Cost (2/3)
Segmentation. With source-only retrieval, topic segmentation and fixed windows have similar QA scores, and the confidence intervals for their differences cross zero. With full SS, topic segmentation achieves 1.16 pp higher accuracy and 1.13 pp higher F1. These results support using topic segments as units for semantic enrichment and show that the value of segmentation should be considered together with the subsequent retrieval representation.
Score fusion and signature fields. Reusing identical EM and SS, separate-channel fusion improves Recall@10 from 70.33% to 75.58% over concatenated single-index retrieval, with accuracy and F1 gains of 3.32 and 2.49 pp. Evidence recall and answer quality improve together, supporting separate source/signature scoring followed by fusion. Full SS also has higher QA point estimates than the summary-field configuration, but their difference intervals cross zero. The evidence for separate-channel fusion is therefore stronger than the evidence establishing the necessity of all five signature fields.
2. Category-level analysis: which queries benefit from SS
To examine the distribution of SS gains across question types, we compare No-SS and Full SS by category using identical EM records.
Table R2. Category-level quality and paired gains
Both deployments evaluate 1,986 questions: 282 multi-hop, 321 temporal, 96 open-domain, 841 single-hop, and 446 adversarial. Differences are calculated before rounding.
Thor / Qwen3.5-35B
| Category | No-SS Acc. (%) | SS Acc. (%) | ΔAcc. pp [95% CI] | No-SS F1 (%) | SS F1 (%) | ΔF1 pp [95% CI] |
|---|---|---|---|---|---|---|
| Multi-hop | 25.53 | 30.50 | +4.96 [+1.80,+8.63] | 33.07 | 34.55 | +1.48 [+0.28,+3.63] |
| Temporal | 31.15 | 31.78 | +0.62 [−1.89,+2.81] | 22.25 | 21.82 | −0.43 [−1.88,+1.08] |
| Open-domain | 37.50 | 40.62 | +3.12 [−1.23,+6.36] | 20.61 | 23.34 | +2.72 [+0.95,+4.34] |
| Single-hop | 83.47 | 81.45 | −2.02[-3.28,-0.22] | 68.14 | 67.85 | −0.29[-1.12,-0.08] |
| Adversarial | 27.35 | 30.72 | +3.36[-1.75,+5.38] | 29.06 | 30.44 | +1.38[+0.12,+3.25] |
AGX 203 / Qwen3.5-4B
| Category | No-SS Acc. (%) | SS Acc. (%) | ΔAcc. pp [95% CI] | No-SS F1 (%) | SS F1 (%) | ΔF1 pp [95% CI] |
|---|---|---|---|---|---|---|
| Multi-hop | 25.18 | 27.30 | +2.13 [+0.40,+3.86] | 27.74 | 28.22 | +0.48 [+0.13,+1.08] |
| Temporal | 30.53 | 33.02 | +2.49 [+1.17,+6.60] | 26.43 | 28.13 | +1.70 [+0.10,+3.86] |
| Open-domain | 35.42 | 35.42 | 0.00 [−5.94,+6.67] | 16.26 | 17.29 | +1.03 [−1.16,+3.87] |
| Single-hop | 74.32 | 76.69 | +2.38[-0.12,+4.27] | 56.90 | 59.71 | +2.80[-0.11,+5.21] |
| Adversarial | 47.76 | 48.65 | +0.90[-1.33,+2.98] | 45.45 | 47.90 | +2.45[+0.11,+3.37] |
The category results show benefits for complex queries requiring multiple facts to be connected. Adding SS improves multi-hop accuracy by 4.96 pp on Thor and 2.13 pp on AGX, with F1 gains of 1.48 and 0.48 pp, respectively. Both deployments show improvements in the same direction. On AGX, temporal accuracy and F1 also improve by 2.49 and 1.70 pp. These results support the retrieval benefits of SS for complex questions: while EM preserves original evidence, explicit semantic cues help locate related information needed for multi-hop and some temporal queries.
3. From semantic computation to background execution: connecting quality and foreground cost
The quality gains require local LLM computation. OnDevMem generates and reuses SS at segment granularity, reducing construction time by 88.4%–92.5% relative to the four complete memory-system baselines in the original Table 1. The No-SS comparison further identifies the additional construction work introduced by SS.
Table R1d. No-SS and Full-SS construction cost
| Configuration | EM / SS | Timing scope | Construction time |
|---|---|---|---|
| B: No-SS | 964 / 0 | In-memory raw objects → EM persistence and source TF-IDF ready | 0.371471 ± 0.000622 s |
| D: Full SS | 964 / 964 | In-memory raw objects → EM/SS persistence and source/SS retrieval structures ready | 8,617.779 s (2.393828 h) |
SS generation uses batches of four EM records, a nominal per-record output budget of 420 tokens, a batch limit of 1,800 tokens, and an actual request limit of 1,680 tokens; temperature=0, repeat_penalty=1.1, timeout=120 s, retries=2, and profile=full.
The construction-cost difference shows that the incremental quality gains require additional local generation. We therefore also evaluate how this computation affects foreground waiting. New EM is first made available through the source channel, and Adaptive schedules SS generation according to foreground activity, resource pressure, and maintenance backlog. The streaming comparison measures the additional foreground cost while completing enrichment tasks.
Streaming Performance and Maintenance Trade-offs (3/3)
Table R3a. Streaming foreground performance
AGX 152, Qwen3.5-4B-Q8, the same trace of 135 QA requests and 330 EM arrivals, at most seven evidence records, and a 4,096-token evidence budget. Each configuration is run once in each of two execution orders; summaries are arithmetic means of run-level metrics. “Forward” and “reverse” refer to configuration execution order.
| Order / pairing | Policy | QA mean (ms) | QA P95 (ms) | QA P99 (ms) | F1 (%) |
|---|---|---|---|---|---|
| Forward | No-SS | 1,746.16 | 2,133.32 | 2,839.54 | 34.12 |
| Forward | Adaptive, preemptive | 1,518.53 | 1,930.11 | 2,185.44 | 35.18 |
| Forward | Adaptive, non-preemptive | 1,753.91 | 2,153.44 | 2,548.81 | 34.15 |
| Reverse, preemptive pair | Adaptive, preemptive | 1,511.85 | 1,889.90 | 2,189.89 | 35.24 |
| Reverse, preemptive pair | No-SS | 1,736.80 | 2,126.38 | 2,542.15 | 34.12 |
| Reverse, non-preemptive pair | Adaptive, non-preemptive | 1,753.87 | 2,183.15 | 2,546.88 | 35.24 |
| Reverse, non-preemptive pair | No-SS | 1,741.31 | 2,120.38 | 2,572.49 | 34.50 |
Table R3b. Additional foreground metrics and maintenance completion (same runs)
| Order / pairing | Policy | TTFT P95 (ms) | Evidence tokens | SS completed | EM without SS at endpoint |
|---|---|---|---|---|---|
| Forward | No-SS | 1,775.45 | 1,225.67 | 0 | 330 |
| Forward | Adaptive, preemptive | 1,453.55 | 1,210.81 | 330 | 0 |
| Forward | Adaptive, non-preemptive | 1,800.60 | 1,214.22 | 330 | 0 |
| Reverse, preemptive pair | Adaptive, preemptive | 1,447.01 | 1,210.11 | 330 | 0 |
| Reverse, preemptive pair | No-SS | 1,761.80 | 1,225.67 | 0 | 330 |
| Reverse, non-preemptive pair | Adaptive, non-preemptive | 1,766.05 | 1,214.22 | 330 | 0 |
| Reverse, non-preemptive pair | No-SS | 1,765.54 | 1,225.67 | 0 | 330 |
The forward-order comparisons share a No-SS control; reverse-order comparisons use the corresponding paired runs. For No-SS, “EM without SS” denotes records for which no signature is generated.
On the same matched-budget trace, non-preemptive Adaptive increases mean QA latency by 0.44% and 0.72% relative to its paired No-SS controls in the two execution orders. The increase in the average run-level mean latency is approximately 0.58%, and the increase in average run-level P95 is approximately 1.95%. Mean F1 across runs is 34.70% for Adaptive and 34.31% for its paired No-SS controls. Both Adaptive runs complete all 330 SS enrichment tasks. Under this supplementary workload, maintenance therefore progresses while foreground response time remains close to No-SS.
The No-SS comparison measures the foreground overhead of adding SS; the original Eager comparison further examines the role of scheduling. On the original streaming workload, Adaptive reduces mean QA latency from 4.928 to 2.418 s and P95 from 11.713 to 8.472 s. Together, the experiments support scheduling semantic enrichment in the background to limit foreground interference.
Regarding the maintenance trade-off, we agree that debt control does not achieve the lowest foreground latency or highest observed accuracy under the current workload. It trades some foreground responsiveness for more timely enrichment and a smaller backlog: mean enrichment delay decreases from 420.6 to 208.0 s and final pending work from 59.7 to 23.0 items, while mean QA latency increases by 0.333 s. We therefore describe it as a policy choice for applications that prioritize timely maintenance.
Thank you again for the careful scrutiny of our architectural choices. Your feedback motivated the controlled comparisons that now clarify both the incremental benefits and the limitations of the design. We hope this additional evidence provides a stronger basis for assessing the contribution beyond source-preserving retrieval. If these results and clarifications address your main concerns, we would be grateful if you would reconsider your overall assessment and rating.
Controlled Ablations and Incremental SS Gains (1/3)
Thank you for recognizing the practical relevance of the setting and the value of evaluation on real devices. Following your suggestions on architectural justification and incremental benefits, we added fixed-window, No-SS, and concatenated single-index comparisons, paired confidence intervals, category-level analysis, and streaming experiments. We first examine which designs contribute to the gains, then assess their computational and foreground costs.
OnDevMem targets deployments where memory processing and foreground inference share limited local resources. EM preserves source text and makes it available for retrieval before enrichment. SS converts local LLM processing into reusable retrieval cues, while Adaptive schedules enrichment tasks. This separates source-evidence availability from semantic processing and its scheduling.
1. From raw-text retrieval to semantic enrichment: identifying the sources of improvement
We construct six controlled configurations, A–F, to compare segmentation, the inclusion of SS, separate-channel fusion versus concatenated single-index retrieval, and full signatures versus the summary field. All configurations use the same QA model, prompt, top-10 limit, and 4,096-token evidence budget. E and D reuse the same EM and SS; F uses only the summary field from the full SS for signature retrieval.
Table R1a. Controlled comparison of answer quality and evidence retrieval
AGX 203, Qwen3.5-4B-Q8, all 1,986 questions, top-10 retrieval, a 4,096-token evidence budget, and the same QA prompt and GPT-4o-mini judging protocol. F is a field ablation reusing the full SS.
Table R1b. Query costs for the same configurations
Table R1c. Paired differences and 95% confidence intervals
Each additional ablation configuration was evaluated in three independent end-to-end runs, rebuilding memory from the original conversations before retrieval and answer generation; the reported results are means across runs. The original main streaming policies were each run three times, while the additional matched-budget streaming comparisons were each run once in each of two execution orders. Paired confidence intervals use conversation-cluster bootstrap with 10,000 resamples, preserving questions within conversations and pairing configurations. They quantify sampling uncertainty conditional on the recorded outputs. Differences are the first configuration minus the second; pp denotes percentage points.
Raw-text baseline and SS gains. Fixed-window raw-text retrieval achieves 52.52% accuracy and 43.86% F1; the complete system improves these by 1.81 and 1.57 pp, respectively. With topic EM held fixed, adding SS improves accuracy by 1.91 pp and F1 by 2.13 pp, while reducing mean evidence length from 2,917 to 2,828 tokens. Together, these comparisons show additional gains from SS on top of a strong source-preserving retriever.