acceptodds
Under review as a conference paper at ICLR 2027

OnDevMem: Efficient Long-Term Memory for On-Device LLM Agents

Abstract

Long-term memory is essential for LLM-based agents to preserve and reuse information accumulated over extended interactions. Agent memory has advanced substantially under cloud-based settings, whereas fully on-device long-term memory remains at an early stage. Deploying existing memory pipelines unchanged on resource-constrained devices can compromise memory quality and incur substantial construction costs from frequent LLM-based semantic processing. Memory maintenance can further increase foreground inference latency by competing for limited local resources. To address these challenges, this paper introduces OnDevMem, a cloud-independent memory framework that separates evidence preservation, semantic indexing, and maintenance scheduling. Topic-aware Evidence Memory (EM) preserves source interactions and provenance, while locally generated Semantic Signatures (SS) provide complementary semantic signals for retrieval without replacing the original evidence. Semantic enrichment at topic granularity reduces memory-construction cost by processing coherent interaction segments jointly. A resource-adaptive controller coordinates signature generation and index updates with foreground demand and device conditions to reduce foreground latency. Experiments and ablations across two local deployments on LoCoMo show that OnDevMem improves judge accuracy by 11.6%–14.5% over the strongest evaluated baseline and reduces memory-construction time by 88.4%–92.5% compared with the fastest baseline across deployments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Long-Term Memory, On-Device AI, Memory-Augmented Generation, Resource-Aware Computing, Efficient Inference, Edge Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qcuYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes OnDevMem, a long-term memory framework for fully local LLM agents. It combines four components: topic-aware segmentation into Evidence Memory (EM) records that preserve source text and provenance; locally generated Semantic Signatures (SS); dual-channel TF-IDF retrieval that combines source-text and signature relevance; and a maintenance controller that schedules semantic enrichment using foreground activity, resource pressure, and maintenance debt.

On LoCoMo, the paper evaluates Jetson AGX Orin with Qwen3.5-4B and Jetson Thor with Qwen3.5-35B. It reports judge-accuracy improvements of 11.6–14.5 percentage points over the strongest evaluated baseline in each deployment and memory-construction time reductions of 88.4–92.5% relative to the fastest baseline. Additional experiments examine architectural choices, streaming maintenance, constrained context budgets, and a small LongMemEval subset.

Critical strengths
  1. The deployment setting is practically relevant. Fully local memory systems must preserve useful information while sharing limited compute with foreground inference. Evaluating complete pipelines on two real Jetson deployments provides useful evidence about their behavior under these constraints.

  2. The evaluation extends beyond aggregate accuracy and construction time. It includes architecture ablations, streaming experiments that directly measure maintenance interference, and a context-budget study. In particular, the source-context ablation supports preserving original evidence, and the streaming results demonstrate the importance of foreground-aware maintenance admission.

Critical weaknesses
  1. The motivation does not sufficiently justify the specific architectural choices or their trade-offs.

Sections 2–3 provide plausible reasons to preserve source evidence, generate additional retrieval cues, and defer expensive maintenance. However, the connection between these general motivations and the particular combination of topic-aware segmentation, a five-field signature, separate retrieval indices, and a multi-signal controller remains underdeveloped. The paper does not sufficiently characterize which limitations of existing systems require these choices, or when their additional complexity and processing costs are warranted over simpler alternatives.

For example, the claim that small local models lose useful information during memory construction is not supported by a direct analysis of what evidence is lost in the evaluated baselines. Table 2B supports retaining original source context within OnDevMem, but does not diagnose the causes of the baseline systems’ failures. For semantic enrichment, Table 2C compares summary-only and full signatures, but does not establish which additional fields provide useful information or whether their benefits justify their generation and maintenance costs. Likewise, comparing fused retrieval with each channel individually does not establish the advantage of maintaining separate indices over indexing the same source text and signature together.

The controller results also require a clearer account of the intended operating trade-off. Table 6 shows that debt control reduces enrichment delay and pending work, while Table 5 reports lower foreground latency and higher observed accuracy when debt control is removed. This demonstrates a maintenance-progress trade-off, but the evaluation does not establish under what workload or freshness requirement the improved maintenance progress warrants the foreground cost. Overall, the reader is given plausible reasons for the components, but insufficient evidence explaining why this particular architecture is the appropriate design.

  1. The evaluation does not establish robust incremental benefits over a simpler source-preserving retrieval pipeline.

In Table 2C, accuracy increases from 52.87% with source-only retrieval to 53.52% with summary-based enrichment and 54.30% with full signatures. Thus, full signatures improve accuracy by 1.43 percentage points over source-only retrieval and 0.78 percentage points over summary-based enrichment. Token F1 also improves, but Tables 1–2 do not report uncertainty estimates or clearly specify repetition counts. The variability reported for the separate streaming experiment in Table 5 does not establish the stability of these architecture-ablation differences. Construction costs for the source-only, summary-only, and full-signature variants are also missing, leaving their incremental quality–cost trade-offs unclear.

The comparison with existing memory systems demonstrates useful end-to-end gains, but does not resolve whether the proposed architecture offers sufficient advantages over straightforward chunked retrieval. Table 2A compares topic-aware segmentation with turn-pair and session-level units, but omits fixed-size chunks of comparable length. Table 2C already shows that source-only retrieval is a strong internal alternative, making a matched fixed-window, source-only TF-IDF baseline particularly important.

Without this comparison and uncertainty estimates for the key ablations, it remains unclear how much of the benefit requires topic-aware boundaries, semantic enrichment, or separate-index fusion, rather than simply preserving and retrieving source text. These gaps concern the core architectural contribution and require additional controlled evaluation to resolve.

Although the paper addresses a relevant deployment setting and reports useful empirical results, I do not consider the central architectural contribution sufficiently established. The missing comparisons and insufficient characterization of the design trade-offs leave its added value over simpler alternatives unresolved. These concerns affect the main contribution and require substantial additional validation.

Questions for authors

Clarification and additional evidence that could change my assessment:

  1. Can you provide uncertainty estimates for the accuracy and Token F1 differences in Table 2, and clarify the repetition counts and generation settings used for Tables 1–2? Please distinguish uncertainty across evaluation questions from variability introduced by memory construction, answer generation, and judging. If using paired bootstrap confidence intervals, please explain how dependence among questions from the same conversation is handled.

  2. Can you compare topic-aware segmentation with fixed-window segmentation at comparable segment lengths, using source-only retrieval and SS-based fusion under the same TF-IDF retriever, top-10 limit, and 4,096-token evidence budget? Reporting construction cost alongside answer quality would help determine whether the additional processing is justified.

  3. What advantage do separate source and signature indices provide over a single TF-IDF index built from concatenated source text and signatures? A controlled comparison, supplemented by evidence-retrieval metrics such as Recall@10 or MRR and an analysis of where the two channels succeed or fail differently, would help establish the value of the proposed separation.

7 comments
AuthorsOpenReview ↗

Controlled Ablations and Incremental SS Gains (1/3)

Thank you for recognizing the practical relevance of the setting and the value of evaluation on real devices. Following your suggestions on architectural justification and incremental benefits, we added fixed-window, No-SS, and concatenated single-index comparisons, paired confidence intervals, category-level analysis, and streaming experiments. We first examine which designs contribute to the gains, then assess their computational and foreground costs.

OnDevMem targets deployments where memory processing and foreground inference share limited local resources. EM preserves source text and makes it available for retrieval before enrichment. SS converts local LLM processing into reusable retrieval cues, while Adaptive schedules enrichment tasks. This separates source-evidence availability from semantic processing and its scheduling.

1. From raw-text retrieval to semantic enrichment: identifying the sources of improvement

We construct six controlled configurations, A–F, to compare segmentation, the inclusion of SS, separate-channel fusion versus concatenated single-index retrieval, and full signatures versus the summary field. All configurations use the same QA model, prompt, top-10 limit, and 4,096-token evidence budget. E and D reuse the same EM and SS; F uses only the summary field from the full SS for signature retrieval.

Table R1a. Controlled comparison of answer quality and evidence retrieval

AGX 203, Qwen3.5-4B-Q8, all 1,986 questions, top-10 retrieval, a 4,096-token evidence budget, and the same QA prompt and GPT-4o-mini judging protocol. F is a field ablation reusing the full SS.

IDSegmentationRetrieval configurationAcc. (%)F1 (%)Recall@10 (%)
AFixed windowSource-only, single index, no SS52.5243.8657.51
BTopicSource-only, single index, no SS52.4243.3075.22
CFixed windowSource + full SS, dual index53.1744.3059.06
DTopicSource + full SS, dual index54.3345.4375.58
ETopicSource and full SS concatenated, single index51.0142.9470.33
FTopicSource + summary-field-only, dual index53.1744.1975.94

Table R1b. Query costs for the same configurations

IDRetrieval (ms)Evidence tokensStatic QA mean/P95 (s)
A0.882,6752.88 / 3.36
B0.772,9173.08 / 3.84
C1.712,6522.86 / 3.36
D1.532,8282.98 / 3.82
E0.832,3802.61 / 3.65
F1.422,8793.02 / 3.83

Table R1c. Paired differences and 95% confidence intervals

ComparisonΔAcc. pp [95% CI]ΔF1 pp [95% CI]
B − A−0.10 [−1.37, +1.21]−0.56 [−1.31, +0.17]
D − C+1.16 [+0.78, +2.61]+1.13 [+0.60, +1.93]
D − B+1.91 [+0.76, +2.22]+2.13 [+0.36, +3.74]
D − E+3.32 [+0.39, +3.80]+2.49 [+0.40, +2.56]
D − F+1.16 [−1.36, +1.38]+1.24 [−0.83, +1.43]
D − A+1.81 [+0.37, +2.82]+1.57 [+0.49, +1.73]

Each additional ablation configuration was evaluated in three independent end-to-end runs, rebuilding memory from the original conversations before retrieval and answer generation; the reported results are means across runs. The original main streaming policies were each run three times, while the additional matched-budget streaming comparisons were each run once in each of two execution orders. Paired confidence intervals use conversation-cluster bootstrap with 10,000 resamples, preserving questions within conversations and pairing configurations. They quantify sampling uncertainty conditional on the recorded outputs. Differences are the first configuration minus the second; pp denotes percentage points.

Raw-text baseline and SS gains. Fixed-window raw-text retrieval achieves 52.52% accuracy and 43.86% F1; the complete system improves these by 1.81 and 1.57 pp, respectively. With topic EM held fixed, adding SS improves accuracy by 1.91 pp and F1 by 2.13 pp, while reducing mean evidence length from 2,917 to 2,828 tokens. Together, these comparisons show additional gains from SS on top of a strong source-preserving retriever.

AuthorsOpenReview ↗

Architectural Comparisons and Construction Cost (2/3)

Segmentation. With source-only retrieval, topic segmentation and fixed windows have similar QA scores, and the confidence intervals for their differences cross zero. With full SS, topic segmentation achieves 1.16 pp higher accuracy and 1.13 pp higher F1. These results support using topic segments as units for semantic enrichment and show that the value of segmentation should be considered together with the subsequent retrieval representation.

Score fusion and signature fields. Reusing identical EM and SS, separate-channel fusion improves Recall@10 from 70.33% to 75.58% over concatenated single-index retrieval, with accuracy and F1 gains of 3.32 and 2.49 pp. Evidence recall and answer quality improve together, supporting separate source/signature scoring followed by fusion. Full SS also has higher QA point estimates than the summary-field configuration, but their difference intervals cross zero. The evidence for separate-channel fusion is therefore stronger than the evidence establishing the necessity of all five signature fields.

2. Category-level analysis: which queries benefit from SS

To examine the distribution of SS gains across question types, we compare No-SS and Full SS by category using identical EM records.

Table R2. Category-level quality and paired gains

Both deployments evaluate 1,986 questions: 282 multi-hop, 321 temporal, 96 open-domain, 841 single-hop, and 446 adversarial. Differences are calculated before rounding.

Thor / Qwen3.5-35B

CategoryNo-SS Acc. (%)SS Acc. (%)ΔAcc. pp [95% CI]No-SS F1 (%)SS F1 (%)ΔF1 pp [95% CI]
Multi-hop25.5330.50+4.96 [+1.80,+8.63]33.0734.55+1.48 [+0.28,+3.63]
Temporal31.1531.78+0.62 [−1.89,+2.81]22.2521.82−0.43 [−1.88,+1.08]
Open-domain37.5040.62+3.12 [−1.23,+6.36]20.6123.34+2.72 [+0.95,+4.34]
Single-hop83.4781.45−2.02[-3.28,-0.22]68.1467.85−0.29[-1.12,-0.08]
Adversarial27.3530.72+3.36[-1.75,+5.38]29.0630.44+1.38[+0.12,+3.25]

AGX 203 / Qwen3.5-4B

CategoryNo-SS Acc. (%)SS Acc. (%)ΔAcc. pp [95% CI]No-SS F1 (%)SS F1 (%)ΔF1 pp [95% CI]
Multi-hop25.1827.30+2.13 [+0.40,+3.86]27.7428.22+0.48 [+0.13,+1.08]
Temporal30.5333.02+2.49 [+1.17,+6.60]26.4328.13+1.70 [+0.10,+3.86]
Open-domain35.4235.420.00 [−5.94,+6.67]16.2617.29+1.03 [−1.16,+3.87]
Single-hop74.3276.69+2.38[-0.12,+4.27]56.9059.71+2.80[-0.11,+5.21]
Adversarial47.7648.65+0.90[-1.33,+2.98]45.4547.90+2.45[+0.11,+3.37]

The category results show benefits for complex queries requiring multiple facts to be connected. Adding SS improves multi-hop accuracy by 4.96 pp on Thor and 2.13 pp on AGX, with F1 gains of 1.48 and 0.48 pp, respectively. Both deployments show improvements in the same direction. On AGX, temporal accuracy and F1 also improve by 2.49 and 1.70 pp. These results support the retrieval benefits of SS for complex questions: while EM preserves original evidence, explicit semantic cues help locate related information needed for multi-hop and some temporal queries.

3. From semantic computation to background execution: connecting quality and foreground cost

The quality gains require local LLM computation. OnDevMem generates and reuses SS at segment granularity, reducing construction time by 88.4%–92.5% relative to the four complete memory-system baselines in the original Table 1. The No-SS comparison further identifies the additional construction work introduced by SS.

Table R1d. No-SS and Full-SS construction cost

ConfigurationEM / SSTiming scopeConstruction time
B: No-SS964 / 0In-memory raw objects → EM persistence and source TF-IDF ready0.371471 ± 0.000622 s
D: Full SS964 / 964In-memory raw objects → EM/SS persistence and source/SS retrieval structures ready8,617.779 s (2.393828 h)

SS generation uses batches of four EM records, a nominal per-record output budget of 420 tokens, a batch limit of 1,800 tokens, and an actual request limit of 1,680 tokens; temperature=0, repeat_penalty=1.1, timeout=120 s, retries=2, and profile=full.

The construction-cost difference shows that the incremental quality gains require additional local generation. We therefore also evaluate how this computation affects foreground waiting. New EM is first made available through the source channel, and Adaptive schedules SS generation according to foreground activity, resource pressure, and maintenance backlog. The streaming comparison measures the additional foreground cost while completing enrichment tasks.

AuthorsOpenReview ↗

Streaming Performance and Maintenance Trade-offs (3/3)

Table R3a. Streaming foreground performance

AGX 152, Qwen3.5-4B-Q8, the same trace of 135 QA requests and 330 EM arrivals, at most seven evidence records, and a 4,096-token evidence budget. Each configuration is run once in each of two execution orders; summaries are arithmetic means of run-level metrics. “Forward” and “reverse” refer to configuration execution order.

Order / pairingPolicyQA mean (ms)QA P95 (ms)QA P99 (ms)F1 (%)
ForwardNo-SS1,746.162,133.322,839.5434.12
ForwardAdaptive, preemptive1,518.531,930.112,185.4435.18
ForwardAdaptive, non-preemptive1,753.912,153.442,548.8134.15
Reverse, preemptive pairAdaptive, preemptive1,511.851,889.902,189.8935.24
Reverse, preemptive pairNo-SS1,736.802,126.382,542.1534.12
Reverse, non-preemptive pairAdaptive, non-preemptive1,753.872,183.152,546.8835.24
Reverse, non-preemptive pairNo-SS1,741.312,120.382,572.4934.50

Table R3b. Additional foreground metrics and maintenance completion (same runs)

Order / pairingPolicyTTFT P95 (ms)Evidence tokensSS completedEM without SS at endpoint
ForwardNo-SS1,775.451,225.670330
ForwardAdaptive, preemptive1,453.551,210.813300
ForwardAdaptive, non-preemptive1,800.601,214.223300
Reverse, preemptive pairAdaptive, preemptive1,447.011,210.113300
Reverse, preemptive pairNo-SS1,761.801,225.670330
Reverse, non-preemptive pairAdaptive, non-preemptive1,766.051,214.223300
Reverse, non-preemptive pairNo-SS1,765.541,225.670330

The forward-order comparisons share a No-SS control; reverse-order comparisons use the corresponding paired runs. For No-SS, “EM without SS” denotes records for which no signature is generated.

On the same matched-budget trace, non-preemptive Adaptive increases mean QA latency by 0.44% and 0.72% relative to its paired No-SS controls in the two execution orders. The increase in the average run-level mean latency is approximately 0.58%, and the increase in average run-level P95 is approximately 1.95%. Mean F1 across runs is 34.70% for Adaptive and 34.31% for its paired No-SS controls. Both Adaptive runs complete all 330 SS enrichment tasks. Under this supplementary workload, maintenance therefore progresses while foreground response time remains close to No-SS.

The No-SS comparison measures the foreground overhead of adding SS; the original Eager comparison further examines the role of scheduling. On the original streaming workload, Adaptive reduces mean QA latency from 4.928 to 2.418 s and P95 from 11.713 to 8.472 s. Together, the experiments support scheduling semantic enrichment in the background to limit foreground interference.

Regarding the maintenance trade-off, we agree that debt control does not achieve the lowest foreground latency or highest observed accuracy under the current workload. It trades some foreground responsiveness for more timely enrichment and a smaller backlog: mean enrichment delay decreases from 420.6 to 208.0 s and final pending work from 59.7 to 23.0 items, while mean QA latency increases by 0.333 s. We therefore describe it as a policy choice for applications that prioritize timely maintenance.

Thank you again for the careful scrutiny of our architectural choices. Your feedback motivated the controlled comparisons that now clarify both the incremental benefits and the limitations of the design. We hope this additional evidence provides a stronger basis for assessing the contribution beyond source-preserving retrieval. If these results and clarifications address your main concerns, we would be grateful if you would reconsider your overall assessment and rating.

Reviewer_HkJBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies long-term conversational memory for LLM agents running on resource-constrained devices and proposes OnDevMem, a framework that separates source-evidence preservation, semantic indexing, and background maintenance scheduling. The system first segments conversations into topically coherent units to form Evidence Memory (EM), preserving the original text and provenance. A local language model then generates Semantic Signatures (SS). Retrieval combines relevance scores from source text and SS, while the original EM is supplied to the answer model. A maintenance controller determines when to perform semantic enrichment based on foreground demand, device resource pressure, and pending work.

The authors evaluate OnDevMem on LoCoMo using two local device/model configurations, reporting improvements in answer quality and memory construction efficiency over the evaluated baselines. Architecture ablations and streaming experiments examine the system’s components and maintenance policies. Supplementary evaluations cover 24 LongMemEval queries and different evidence-context budgets.

Critical strengths

Strength 1: A practical memory organization design supported by experiments on real devices.

Separating source evidence from generated retrieval cues reduces reliance on compressed memory as the answer context. With retrieved record identifiers held fixed, Table 2(B) reports an accuracy of 54.30% when using source EM, compared with 38.67% when using SS as the context. This supports the value of retaining source evidence in the tested configurations. Table 1 also reports advantages in accuracy, Token F1, and memory construction time across two local deployments, supporting the practical value of the design.

Strength 2: An informative analysis of the trade-off between background maintenance and foreground responsiveness.

Compared with Eager, Adaptive reduces mean foreground QA latency from 4.928 to 2.418 seconds and P95 latency from 11.713 to 8.472 seconds. The authors also report semantic enrichment delay and pending maintenance work, and examine the contributions of individual control signals through ablations. These results expose both the responsiveness benefits and the associated maintenance costs, providing useful evidence for designing memory systems that share local resources with foreground inference.

Critical weaknesses

Weakness 1: The quality improvement from SS ranking signals is supported, but the associated system cost is not sufficiently quantified.

In Table 2(C), source-only ranking achieves an accuracy of 52.87% and an F1 of 0.4332, compared with 54.30% and 0.4540 for Full SS—gains of 1.43 percentage points and 0.0208 F1. The table caption and Appendix A.2 describe this as a ranking-signal ablation, leaving it unclear whether the Source EM-only configuration also disables SS construction and maintenance. Uncertainty estimates are not reported for this comparison, and the context length and QA latency of the source-only ranking configuration are omitted. Since SS construction and index updates consume additional local resources, the absence of a corresponding cost comparison with these operations disabled limits assessment of the incremental value of semantic enrichment.

Weakness 2: Missing scheduling parameters limit the reproducibility of a central comparison.

Appendix C describes the maintenance admission logic, including foreground requests blocking new maintenance operations, maintenance debt overriding resource-based deferral, and sequential, non-preemptive execution. However, the supplied PDF does not specify the resource-pressure thresholds, the override conditions based on pending-work count and age, or the scheduling period, number of maintenance opportunities, and matching target for Fixed-Matched. Although Table 6 describes Org. as semantic-organization availability at query time, its numerator, denominator, and aggregation procedure are not specified. These details are necessary to independently reproduce the reported foreground latency and background maintenance outcomes.

Questions for authors

SS ranking ablation and cost: Does the Source EM-only configuration in Table 2(C) disable SS only during ranking while still generating and maintaining it? If disabling SS construction and maintenance has not been evaluated, could you provide a limited quality–cost comparison under the existing Orin setup, keeping segmentation and evidence budgets unchanged? Please include memory construction cost, foreground QA latency, and realized context length. Per-conversation results or a paired uncertainty analysis would also help interpret the observed quality improvement.

Scheduling parameters and metrics: Building on the admission logic in Appendix C, please specify the resource-pressure thresholds and debt override conditions based on pending-work count and age used in Tables 5–6. Please also provide the scheduling period, number of maintenance opportunities, and matching target for Fixed-Matched. Table 6 describes Org. as semantic-organization availability at query time; please clarify its numerator, denominator, and aggregation procedure.

Segmentation and evidence availability: Are segment boundaries predetermined inputs to the replay, or are they computed online as interactions arrive? When do the right-hand windows and session-relative thresholds become available, can unfinished segments be retrieved, and how are EM arrival timestamps and QA injection conditions defined? I understand that the QA latency in Appendix A.4 includes queueing and model service time; this question concerns when evidence becomes available and which stages of the pipeline the replay evaluates.

5 comments
AuthorsOpenReview ↗

SS Quality Gains and Category Analysis (1/3)

Thank you for recognizing the practical value of the system, the real-device evaluation, and the analysis of maintenance trade-offs. Following your suggestions, we added No-SS quality–cost comparisons, paired confidence intervals, and streaming experiments, and clarified the scheduling parameters, Org. metric, and evidence-availability timeline.

1. The role of SS and its quality–cost trade-off

OnDevMem combines source-preserving EM, reusable SS, and adaptive maintenance to coordinate local memory quality and foreground responsiveness. The original Table 2(C) retained SS generation and set its ranking weight to zero; the new No-SS configuration disables SS generation, the signature index, and maintenance. We compare quality, construction cost, and foreground latency using the same segmentation, QA model, prompt, top-10 limit, and 4,096-token evidence budget.

Each additional ablation configuration was evaluated in three independent end-to-end runs, rebuilding memory from the original conversations before retrieval and answer generation. Table R1a reports the mean quality and query-cost results. Paired 95% confidence intervals use conversation-cluster bootstrap with 10,000 resamples, retaining all questions within each sampled conversation and pairing the configurations.

Table R1a. Controlled quality and query-cost comparison

AGX 203, Qwen3.5-4B-Q8, all 1,986 questions, top-10 retrieval, a 4,096-token evidence budget, and the same QA prompt and GPT-4o-mini judging protocol. Both configurations use identical topic EM.

Retrieval configurationAcc. (%)F1 (%)Retrieval (ms)Evidence tokensStatic QA mean/P95 (s)
Source-only, single index, no SS52.4243.300.772,9173.08 / 3.84
Source + full SS, dual index54.3345.431.532,8282.98 / 3.82

Adding SS improves accuracy by 1.91 pp and F1 by 2.13 pp, with paired 95% CIs of [0.76, 2.22] and [0.36, 3.74], respectively. Meanwhile, mean evidence length decreases from 2,917 to 2,828 tokens. SS therefore improves evidence selection within the limited context budget; the quality gain does not rely on supplying more input tokens.

To examine which queries benefit, we further compare No-SS and Full SS by question category using the same EM records.

Table R2. Category-level quality and paired gains

Both deployments evaluate 1,986 questions: 282 multi-hop, 321 temporal, 96 open-domain, 841 single-hop, and 446 adversarial. Differences are calculated before rounding; pp denotes percentage points.

Thor / Qwen3.5-35B

CategoryNo-SS Acc. (%)SS Acc. (%)ΔAcc. pp [95% CI]No-SS F1 (%)SS F1 (%)ΔF1 pp [95% CI]
Multi-hop25.5330.50+4.96 [+1.80,+8.63]33.0734.55+1.48 [+0.28,+3.63]
Temporal31.1531.78+0.62 [−1.89,+2.81]22.2521.82−0.43 [−1.88,+1.08]
Open-domain37.5040.62+3.12 [−1.23,+6.36]20.6123.34+2.72 [+0.95,+4.34]
Single-hop83.4781.45−2.02[-3.28,-0.22]68.1467.85−0.29[-1.12,-0.08]
Adversarial27.3530.72+3.36[-1.75,+5.38]29.0630.44+1.38[+0.12,+3.25]

AGX 203 / Qwen3.5-4B

CategoryNo-SS Acc. (%)SS Acc. (%)ΔAcc. pp [95% CI]No-SS F1 (%)SS F1 (%)ΔF1 pp [95% CI]
Multi-hop25.1827.30+2.13 [+0.40,+3.86]27.7428.22+0.48 [+0.13,+1.08]
Temporal30.5333.02+2.49 [+1.17,+6.60]26.4328.13+1.70 [+0.10,+3.86]
Open-domain35.4235.420.00 [−5.94,+6.67]16.2617.29+1.03 [−1.16,+3.87]
Single-hop74.3276.69+2.38[-0.12,+4.27]56.9059.71+2.80[-0.11,+5.21]
Adversarial47.7648.65+0.90[-1.33,+2.98]45.4547.90+2.45[+0.11,+3.37]

The category results show benefits for complex queries requiring multiple facts to be connected. Adding SS improves multi-hop accuracy by 4.96 pp on Thor and 2.13 pp on AGX, with F1 gains of 1.48 and 0.48 pp, respectively. Both deployments show improvements in the same direction. On AGX, temporal accuracy and F1 also improve by 2.49 and 1.70 pp. These results support the retrieval benefits of SS for complex questions: while EM preserves original evidence, explicit semantic cues help locate related information needed for multi-hop and some temporal queries.

AuthorsOpenReview ↗

Construction Cost and Streaming Performance (2/3)

Table R1b reports construction times and their timing scopes for No-SS and Full SS. Relative to the four complete memory-system baselines in the original Table 1, OnDevMem reduces construction time by 88.4%–92.5%.

Table R1b. No-SS and Full-SS construction cost

ConfigurationEM / SSTiming scopeConstruction time
B: No-SS964 / 0In-memory raw objects → EM persistence and source TF-IDF ready0.371471 ± 0.000622 s
D: Full SS964 / 964In-memory raw objects → EM/SS persistence and source/SS retrieval structures ready8,617.779 s (2.393828 h)

Construction timing excludes QA and judging. The main difference is local model generation: No-SS only stores source text and builds retrieval structures, whereas Full SS additionally generates and stores signatures, introducing substantial construction cost. OnDevMem makes EM available through the source-text channel before SS enrichment, so new evidence need not wait for signature generation. Generated SS are reused across queries, without additional LLM-based query rewriting or reranking at retrieval time.

The streaming comparison measures how much foreground waiting this enrichment work adds. Adaptive maintenance admits tasks according to foreground activity and resource pressure, and uses maintenance debt to advance pending work.

Table R3a. Streaming foreground performance

AGX 152, Qwen3.5-4B-Q8, the same trace of 135 QA requests and 330 EM arrivals, at most seven evidence records, and a 4,096-token evidence budget. Each configuration is run once in each of two execution orders. “Forward” and “reverse” refer to configuration execution order.

Order / pairingPolicyQA mean (ms)QA P95 (ms)QA P99 (ms)F1 (%)
ForwardNo-SS1,746.162,133.322,839.5434.12
ForwardAdaptive, preemptive1,518.531,930.112,185.4435.18
ForwardAdaptive, non-preemptive1,753.912,153.442,548.8134.15
Reverse, preemptive pairAdaptive, preemptive1,511.851,889.902,189.8935.24
Reverse, preemptive pairNo-SS1,736.802,126.382,542.1534.12
Reverse, non-preemptive pairAdaptive, non-preemptive1,753.872,183.152,546.8835.24
Reverse, non-preemptive pairNo-SS1,741.312,120.382,572.4934.50

Table R3b. Additional foreground metrics and maintenance completion (same runs)

Order / pairingPolicyTTFT P95 (ms)Evidence tokensSS completedEM without SS at endpoint
ForwardNo-SS1,775.451,225.670330
ForwardAdaptive, preemptive1,453.551,210.813300
ForwardAdaptive, non-preemptive1,800.601,214.223300
Reverse, preemptive pairAdaptive, preemptive1,447.011,210.113300
Reverse, preemptive pairNo-SS1,761.801,225.670330
Reverse, non-preemptive pairAdaptive, non-preemptive1,766.051,214.223300
Reverse, non-preemptive pairNo-SS1,765.541,225.670330

Across the two execution orders, the original non-preemptive Adaptive implementation increases mean QA latency by 0.44% and 0.72% relative to its paired No-SS controls. The increase in the average run-level mean latency is approximately 0.58%, and the increase in average run-level P95 is approximately 1.95%. Both Adaptive runs complete all 330 SS enrichment tasks. Mean F1 across runs is 34.70% for Adaptive and 34.31% for its paired No-SS controls. Under this workload, the system completes semantic enrichment while adding little foreground mean latency.

The new preemptive extension achieves mean QA latencies of 1,518.53 and 1,511.85 ms, also completing all 330 enrichment tasks in both runs. This extension further reduces foreground mean latency and is reported separately from the original non-preemptive implementation.

AuthorsOpenReview ↗

Scheduling Details and Evidence Availability (3/3)

2. Scheduling parameters, Fixed-Matched, and the Org. definition

The original Tables 5–6 use the following scheduling parameters:

ParameterSetting
CPU / GPU / RAM high-pressure thresholds85% / 80% / 85%; any threshold reached indicates HIGH pressure
Resource sampling period250 ms
Scheduling decision period0.25 s
Foreground idle guard2 s
Critical debtPending count ≥ 8, or oldest waiting time ≥ 60 s
Debt overrideOverrides idle guard and resource-pressure restrictions, but not active/queued QA protection

Given pending work and an available worker, active or queued QA requests block admission. Otherwise, admission is governed by the idle guard and resource-pressure checks, subject to the critical-debt override. Debt can override those two restrictions but not foreground protection. The continuous debt score is:

d = 0.5 × min(n_pending / 8, 1) + 0.5 × min(a_oldest / 60 s, 1)

Fixed-Matched starts at most one EM enrichment and index-update task every 9.15 s, without reading foreground activity, resource pressure, or debt. Its period is calibrated using sustainable maintenance capacity on a development workload; nominal capacity is approximately 1.20 times the input arrival rate.

For question q, Org. is defined as:

Org_q = (number of EM records enriched and committed to the index at the QA retrieval snapshot) / (total number of EM records released at that snapshot)

Within each run, we average Org. over the 1,791 QA requests, and then average over three runs. Pending and in-flight EM records count toward the denominator; they enter the numerator only after enrichment and index commitment are complete.

3. Segment boundaries, evidence availability, and QA injection

Segment boundaries are computed and frozen before replay. At each candidate boundary, the segmenter uses up to two turns on either side. A complete right-hand window requires the following two turns; truncated windows are used at the session end. Session-relative thresholds are computed from boundary-difference scores over the complete session:

c = median({δ_t}), s = 1.4826 × MAD({δ_t})

τ_low = c + 0.50s, τ_high = c + 1.25s

If s < 10⁻⁶, we fall back to the standard deviation; if it remains below this value, we use 10⁻⁶. The default path accepts boundaries satisfying δ_t > δ_(t−1), δ_t ≥ δ_(t+1), and δ_t ≥ τ_low. The high threshold is used in the cascade path.

Replay schedules each complete EM record for release after its final source turn and before the next turn. If its source turn cannot be resolved, release is scheduled after the session’s last turn. Unfinished segments do not have a separately retrievable state. Upon EM arrival, the source text immediately enters the retrievable pool and the record enters the maintenance queue. SS participates in retrieval only after enrichment and index updates are complete.

QA requests are injected only after all required EM records are scheduled to have been released; the test stream contains 1,791 questions. Gold evidence is used solely to determine injection eligibility, not for retrieval or answering.

Non-QA events are mapped to replay time in logical order at 0.20 s intervals. The inter-session gap is 20 s, and QA requests within a session are separated by 2 s.

In Tables 5–6, QA latency is measured from the client issuing the QA HTTP request until the response ends. It includes service queueing, prompt processing, and generation, but excludes retrieval, evidence packing, and segmentation waiting. Org. and retrieval snapshots are recorded at QA dispatch.

Thank you again for your thoughtful and detailed feedback. Your questions helped us clarify the incremental costs of SS, the scheduling rules, and the scope of the streaming evaluation. We hope these additions provide a clearer basis for assessing the system’s practical value and quality–cost trade-offs. If you find that they address your main concerns, we would sincerely appreciate your reconsideration of the overall assessment and rating.

Reviewer_CtCJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OnDevMem is a fully local long-term memory system for LLM agents. It splits sessions into topic segments with a lexical method and stores them verbatim, uses a local LLM to write a Semantic Signature for each segment, and ranks records by fusing two TF-IDF scores, si(q)=(1−α)sˉiE(q)+αsˉiS(q)s_i(q) = (1-\alpha)\bar{s}^E_i(q) + \alpha \bar{s}^S_i(q). A rule-based controller defers signature generation when foreground demand is high. On LoCoMo with two Jetson devices, it reports 11.6 to 14.5 points higher judge accuracy and about 90% lower build time than Mem0, A-Mem, LightMem, and LangMem.

Critical strengths
  1. All methods are evaluated on real edge hardware with the same local models, and build time, retrieval latency, and streaming latency are measured end to end.
  2. The ablation that replaces raw records with signatures (Token F1 drops from 0.454 to 0.349) clearly shows why raw evidence should be kept.
Critical weaknesses
  1. Raw-text retrieval baselines are missing, and the paper's own ablation suggests they explain most of the gain. "Source EM only" (TF-IDF over raw segments, no LLM) already reaches 52.87%, 13 points above the best baseline, and the signature adds only 1.4 points.
  2. Several claims rest on weak evidence. LongMemEval uses only 24 queries, main results are single runs, and Mem0 is worse with the 35B model than with the 4B model. The judge also sees the retrieved evidence. Streaming accuracy (about 37%) is far below the offline result (54.3%) without explanation, and the controller rules are not specified.
Questions for authors
  1. How do BM25 and dense retrieval over raw turns or sessions perform under the same budget?
  2. Why does accuracy drop from 54.3% offline to about 37% in streaming?
  3. Does the LoCoMo set include the adversarial category, and what are the per-category results?
  4. Why is Mem0 worse with Qwen3.5-35B than with Qwen3.5-4B?
  5. What are the exact segmentation thresholds, controller rules, and signature prompt?

Suggestions for future work:

  1. Evaluate on the full LongMemEval and on phone-class devices.
5 comments
AuthorsOpenReview ↗

Additional Retrieval Experiments and SS Gains (1/3)

Thank you for recognizing the real-device evaluation and the value of preserving source evidence. Following your suggestions, we added BM25/dense retrieval baselines, No-SS comparisons, and per-category results, and examined the Mem0 performance difference, evaluation protocol, and implementation details.

OnDevMem targets local deployments where memory processing and foreground inference share limited resources. EM preserves original evidence, SS provides reusable retrieval cues generated by the local model, and adaptive maintenance coordinates semantic processing with foreground demand.

1. Raw-text retrieval baselines and category-level SS gains (Questions 1 and 3)

The additional experiments compare raw-text retrievers and isolate the contribution of SS on the same EM records. Source text and signatures are scored separately and fused, while the answer model still receives the original evidence.

All additional retrieval configurations use all 1,986 LoCoMo questions on AGX 203/Qwen3.5-4B-Q8, with the same QA prompt, temperature 0, a top-10 retrieval limit, and a 4,096-token evidence budget. Retrieval is restricted to the current user. Turn-pair records combine adjacent pairs of messages within each session, retaining an unpaired final message separately; session records preserve the entire session. These produce 3,011 and 272 records, respectively.

Each additional ablation configuration was evaluated in three independent end-to-end runs, rebuilding memory from the original conversations before retrieval and answer generation. We report their mean results, which therefore differ slightly from the original single-run results. Corresponding gains are calculated from these means.

Table R1 compares fixed-window TF-IDF, BM25, local dense retrieval, and the complete system, including a No-SS/Full-SS comparison using identical topic EM.

Table R1. Raw-text retrieval baselines and OnDevMem

Memory unitRetrieval configurationAcc.%F1%Mean retrieval (ms)Evidence tokens
Fixed windowSource-only TF-IDF, single index, no SS52.5243.860.882,675
TopicSource-only TF-IDF, single index, no SS52.4243.300.772,917
TopicSource + Full SS, dual TF-IDF indices54.3345.431.532,828
Turn-pairBM25, no SS43.1039.124.121,458
Turn-pairLocal Dense, no SS40.7935.9122.901,384
SessionBM25, no SS47.8342.630.604,096
SessionLocal Dense, no SS30.9726.3322.634,096

BM25 uses k1=1.2 and b=0.75. Local dense retrieval uses all-MiniLM-L6-v2 on CPU, with 384-dimensional mean-token pooling, L2 normalization, and cosine similarity, without lexical fallback. Its encoding limit is 256 WordPiece tokens; longer records retain their beginning for encoding. After retrieval, the original record text is supplied to QA subject to the 4,096-token evidence budget.

We estimate uncertainty using paired, conversation-cluster bootstrap with 10,000 resamples. Each resample draws conversations with replacement and retains all questions within each selected conversation. The same sample is applied across configurations and all three runs. We then calculate each configuration’s mean score across runs and the between-configuration difference, obtaining its 95% confidence interval. This preserves within-conversation dependence.

Table R2. Paired performance differences and 95% confidence intervals (percentage points)

ComparisonΔAcc. pp [95% CI]ΔF1 pp [95% CI]
Topic Full SS − Topic No-SS+1.91 [+0.76, +2.22]+2.13 [+0.36, +3.74]
Topic Full SS − Fixed-window No-SS+1.81 [+0.37, +2.82]+1.57 [+0.49, +1.73]

Raw-text retrieval is already a strong alternative: fixed-window TF-IDF achieves 52.52% accuracy and 43.86% F1. The complete system improves these by 1.81 and 1.57 pp, respectively. Thus, preserving source text accounts for a substantial part of the observed gains, while SS provides additional improvement.

With topic EM held fixed, adding SS increases accuracy from 52.42% to 54.33% and F1 from 43.30% to 45.43%; both paired confidence intervals are above zero. Meanwhile, mean evidence length decreases from 2,917 to 2,828 tokens. The improvement therefore does not rely on providing more context, supporting the complementary role of SS in evidence selection. Compared with session-level BM25, the complete system also achieves higher accuracy and F1 while using fewer evidence tokens.

AuthorsOpenReview ↗

Category Results and Evaluation Clarifications (2/3)

We further compare No-SS and Full SS by question category using the same EM. The evaluation includes all categories: 282 multi-hop, 321 temporal, 96 open-domain, 841 single-hop, and 446 adversarial questions.

Table R3a. Category-level results on Thor/Qwen3.5-35B

CategoryNo-SS Acc.%SS Acc.%ΔAcc. ppNo-SS F1%SS F1%ΔF1 pp
Multi-hop25.5330.50+4.9633.0734.55+1.48
Temporal31.1531.78+0.6222.2521.82−0.43
Open-domain37.5040.62+3.1220.6123.34+2.72
Single-hop83.4781.45−2.0268.1467.85−0.29
Adversarial27.3530.72+3.3629.0630.44+1.38

Table R3b. Category-level results on AGX/Qwen3.5-4B

CategoryNo-SS Acc.%SS Acc.%ΔAcc. ppNo-SS F1%SS F1%ΔF1 pp
Multi-hop25.1827.30+2.1327.7428.22+0.48
Temporal30.5333.02+2.4926.4328.13+1.70
Open-domain35.4235.420.0016.2617.29+1.03
Single-hop74.3276.69+2.3856.9059.71+2.80
Adversarial47.7648.65+0.9045.4547.90+2.45

Differences are calculated before rounding.

Adding SS improves multi-hop accuracy by 4.96 pp on Thor and 2.13 pp on AGX, with F1 gains of 1.48 and 0.48 pp, respectively. Both deployments therefore show improvements in the same direction. On AGX, temporal accuracy and F1 also improve by 2.49 and 1.70 pp. These results support the retrieval benefits of SS for complex questions: while EM retains original evidence, explicit semantic cues help locate related information needed for multi-hop and some temporal queries.

2. Clarification of the judging protocol and correction to Appendix A.3

Thank you for raising this concern. We identified an error in Appendix A.3: the judge did not receive retrieved evidence in our main experiments.

We checked the frozen evaluation artifacts from both devices, including the judge-input files, prompt template, cache-key construction, and API request code. These confirm that the judge received only the question, reference answer, and generated response. Retrieved evidence was supplied only to the local answer model during response generation. All methods used this same judging protocol.

We will correct Appendix A.3. The reported results remain unchanged because this correction fixes the protocol description, not the evaluation actually performed.

3. Offline versus streaming accuracy (Question 2)

The offline experiment evaluates 1,986 questions after memory construction is complete. The streaming experiment evaluates 1,791 questions while EM records are progressively released and SS enrichment proceeds. The question sets, available history, and enrichment states therefore differ, so the two accuracy values are not directly comparable under identical conditions.

The streaming experiment replays frozen EM records, releasing each record after its final source turn and injecting QA requests only after all required EM records are scheduled to have arrived. Gold evidence is used solely to determine injection eligibility, not for retrieval or answering. All maintenance policies use the same trace and injection rules to compare foreground responsiveness and maintenance progress.

To separate the effects of question selection, available history, and enrichment state, we are conducting a controlled comparison on the same 1,791 questions. This experiment is not yet complete.

4. Mem0 performance on the 35B versus 4B deployment (Question 4)

We performed a paired examination of Mem0’s 1,986 predictions from the original Table 1.

Mem0 performance and explicit Unknown responses

MetricMem0 / Qwen3.5-4BMem0 / Qwen3.5-35B
Questions1,9861,986
Correct answers790640
Accuracy39.78%32.23%
Explicit Unknown responses370635
Explicit Unknown rate18.63%31.97%

The 35B deployment produces “Unknown” more frequently. Among questions answered correctly by 4B but incorrectly by 35B, 151 involve a 35B “Unknown” response. The reverse case occurs for 31 questions, yielding a net difference of 120 questions—80% of the overall 150-question difference in correct answers.

This associates Unknown-related errors with much of the observed accuracy difference. Because each deployment independently constructs memory using its own model, the difference may reflect both the retrieved evidence and answering behavior.

AuthorsOpenReview ↗

Implementation Details and Remaining Evaluation (3/3)

Part 3/3 of our response to this review

5. Segmentation parameters, controller rules, and SS prompt (Question 5)

Segmentation

At each candidate boundary, the segmenter uses up to two turns on each side and computes: δ_t = 1 − cosine(v_left, v_right) For each complete session: c = median({δ_t}) s = 1.4826 × MAD({δ_t}) τ_low = c + 0.50s τ_high = c + 1.25s

If s < 10⁻⁶, we fall back to the standard deviation; if it remains below this value, we use 10⁻⁶. The default path accepts boundaries satisfying δ_t > δ_(t−1), δ_t ≥ δ_(t+1), and δ_t ≥ τ_low. The high threshold is used in the cascade path. The original streaming experiment replays pre-segmented EM.

Maintenance controller

The debt score is: d = 0.5 × min(n_pending / 8, 1) + 0.5 × min(a_oldest / 60 s, 1)

Debt becomes critical when n_pending ≥ 8 or the oldest task has waited at least 60 s. Given pending work and an available worker, the controller checks foreground activity, the idle guard, and resource pressure. Critical debt can override the idle guard and resource-pressure restrictions, but cannot override foreground protection.

Maintenance settings

ControlOriginal setting
CPU/GPU/RAM pressure thresholds85% / 80% / 85%; any threshold reached indicates HIGH pressure
Sampling / scheduling period250 ms / 0.25 s
Idle guard2 s
Critical debtPending count >= 8, or oldest waiting time >= 60 s
Foreground protectionActive or queued QA blocks new maintenance admission
Debt overrideOverrides idle guard and resource-pressure restrictions, but not foreground protection
ExecutionSequential and non-preemptive; admitted tasks run to completion
Fixed-MatchedAt most one item every 9.15 s, calibrated using development-workload capacity; does not read foreground, pressure, or debt signals

SS generation

Full SS uses five semantic fields and processes four EM records per batch. Its input contains source text and metadata; query prototypes are generated from the corresponding source record.

SS generation settings

ParameterOriginal main-experiment setting
batch_size4
max_tokens420 nominal output tokens per EM
batch_max_tokens1,800 per batch request
Actual request max_tokens1,680 = min(420 x 4, 1,800)
temperature0.0
repeat_penalty1.1
timeout120 s
retries2
qos_profilefull
prompt_versionroutemem-qos-v1-qwen-local
Logical batches964 EM / 4 = 241, excluding retries

System message

Generate compact grounded routing JSON for each evidence item. Use only facts explicitly present in each item.

User message

Evidence items:
{evidence_items}

Return one JSON object:
{"items":[{"item_index":0,"em_id":"...","topic":...}]}
Each item must contain exactly these semantic keys:
topic, entities, event_types, grounded_summary, query_prototypes.
event_types must be chosen from ["identity","preference","activity","temporal","relationship","health","work","education","location","object","emotion","other"].
Use 2 to 4 query_prototypes per item, answerable by that exact item text.
grounded_summary must be compact and grounded in the exact item text.
Do not infer dates or facts not in the item text.

Each evidence item is formatted as

item_index: {index}
em_id: {em_id}
user_id: {user_id}
session_id: {session_id}
speakers: {speakers_json}
text:
{raw_text}

Items are separated by blank lines and ---. The ellipses in the output example are illustrative placeholders.

Finally, thank you for suggesting full LongMemEval evaluation and phone-class deployment. Given the substantial local memory-construction time required by the baselines, the current LongMemEval evaluation uses a fixed 24-query subset shared by all methods. Full evaluation is underway but is not expected to finish before the rebuttal deadline. Phone-class deployment remains a direction for future work.

Thank you again for your careful reading and constructive feedback. Your comments prompted us to strengthen the raw-text retrieval comparisons and clarify the evaluation protocol. We hope the additional evidence and clarifications help address your concerns and make the contribution clearer. If you find that they resolve your main concerns, we would be grateful if you would reconsider your overall assessment and rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.