acceptodds
Under review as a conference paper at ICLR 2027

DuplexLedger: Benchmarking Transactional Memory in Full-Duplex Speech Agents

Abstract

Full-duplex speech agents must update persistent memory while listening and speaking, yet it remains unclear which events should be stored, revised, or excluded. Existing benchmarks isolate interaction timing from memory outcomes, obscuring the state transitions behind a correct answer. We introduce , a bilingual benchmark of 1,094 persistent sessions aligning four full-duplex scenarios with event-level supervision, explicit pre/post memory states, and 6,564 same-session questions and checks. Evaluating five full-duplex models reveals an interaction–memory gap: no system combines strong boundary control with coherent final-state memory, and the best final-state fact coverage is 2.01%. We propose , an 8.92M-parameter sidecar comprising an event controller, a versioned transactional ledger, and a typed retriever coupled with a frozen -o 4.5 backbone. On 219 held-out sessions, increases latest-value accuracy from 4.11% to 8.68% while reducing stale assertions and distractor adoption. It improves interaction boundaries but slows interruption stopping. These results motivate selective, auditable full-duplex memory rather than indiscriminate transcript retention. Code is available at https://anonymous.4open.science/r/DuplexLedger.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-Duplex agent, Memory, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oU37OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DuplexLedger, a synthetic English–Chinese benchmark of 1,094 full-duplex sessions evaluating whether speech agents correctly store, revise, and exclude information through event-level memory operations, state transitions, and spoken QA. The proposed ACT-Mem combines an admission controller, a versioned ledger, and a typed retriever for external memory management. Retrieved evidence is supplied through text prefill to a frozen MiniCPM-o 4.5 only during QA. The evaluation covers five full-duplex systems; on 219 held-out sessions, ACT-Mem increases latest-value accuracy from 4.11% to 8.68% over the unaugmented backbone.

Critical strengths
  1. The benchmark separates interaction timing from memory updates by aligning adaptive challenges with state transitions and supporting/forbidden evidence. Stopping or continuing appropriately does not establish correct storage or revision. Evaluating which information should be admitted or excluded, beyond ordinary recall, is a useful contribution.
  2. Transaction traces help localize failures that final answers alone cannot reveal. In Table 21, UPDATE is predicted correctly, but inconsistent entity keys leave the original values active without supersession. This distinguishes failure to recognize a correction from failure to apply a recognized correction to the memory state.
Critical weaknesses

The stated final-state coverage metric can penalize complete, correct summaries. Appendix C.7, Eq. (29), compares each reference fact against the entire answer, requiring token F1 ≥ 0.5. For four equal-length facts with non-overlapping tokens, an answer correctly concatenating all four obtains F1=2/(4+1)=0.4F_1=2/(4+1)=0.4 per fact and therefore 0% coverage. Answering only one fact instead receives 25% coverage. Thus, adding correct information can reduce the score.

This counterexample concerns the stated definition; the actual implementation and its impact on the reported results remain unverified. Since Section 4.4 uses the maximum coverage of 2.01% as evidence of state-consolidation failure, scorer validation is necessary before interpreting this value as the extent of that failure.

Questions for authors
  1. Does the implemented scorer compare each reference fact with the entire answer or a matching span/fact? What coverage is obtained when reference summaries themselves are used as predictions? If implemented as described in Eq. (29), does corrected scoring preserve the conclusion in Section 4.4?
  2. Did Figure 4 and Table 4 use annotated query types or ASR-text routing? Appendix E.2 describes a speaker/addressee metadata gate that can block admission independently of controller predictions. Please distinguish evidence for learned admission itself from results for the metadata-assisted system.
  3. Appendix B.6 retains planned reference targets even when a challenge is not delivered, meaning that memory scores also reflect missed evidence delivery. For latest-value questions with sufficient supporting evidence delivered in both the backbone and ACT-Mem runs, what are the common-subset size and each system’s accuracy? This would clarify whether the gain persists in subsequent memory and answer processing.
8 comments
AuthorsOpenReview ↗

Response for issues 1

Q1:Does whole-answer matching underestimate fact coverage? How does the scorer perform when reference summaries are used as predictions, and do the conclusions in Section 4.4 hold after correcting the scoring?

A1:Thank you for identifying this scoring issue. The original scorer indeed computed token F1 between each reference fact and the entire answer, using a coverage threshold of 0.5 without local-span matching. Consequently, adding correct facts could reduce the score for an individual fact.

We revised the metric to atomic-fact semantic coverage (SFC). Reference facts are first verified, and Qwen2.5-32B and Mistral-Small-3.1-24B assess whether each fact is correctly expressed in the answer, with bounds reported for unresolved judgments. This revision removes the whole-answer lexical-F1 length penalty. The scoring logic passed 17 deterministic tests. When reference summaries were used as predictions, all 1,081 adjudicable summaries out of 1,094 achieved 100% coverage; 13 retained unresolved reference-verification judgments, yielding overall coverage of 99.13–100%.

We then uniformly rescored the frozen outputs of all five systems across the same 1,094 sessions:

ModelOriginal summary coverageRevised SFC ↑Revised LVA ↑
MiniCPM-o 4.52.01%36.91–49.48%32.08–40.63%
Freeze-Omni0.00%1.56–6.15%26.10–32.82%
Moshi0.04%0.39–1.43%5.80–9.46%
Lychee-FD1.06%19.44–31.85%54.89–60.47%
PersonaPlex0.24%5.60–10.69%9.83–15.04%

Ranges represent bounds from unresolved semantic judgments, not confidence intervals. Model outputs and interaction trajectories are unchanged.

These results require updating the absolute-coverage argument in Section 4.4 and support a more precise conclusion: latest-fact recovery and final-summary coverage exhibit distinct performance profiles following full-duplex interaction. For example, Lychee-FD achieves the highest latest-value accuracy, whereas MiniCPM achieves higher summary coverage, supporting separate evaluation of these capabilities.

We will revise the manuscript accordingly: update Eq. (29) and Appendix C.7 with the revised scoring definition and treatment of unresolved judgments, replace Table 3, and rewrite Section 4.4 to replace the “maximum coverage of only 2.01%” statement and its interpretation. We will also include the reference-summary self-scoring results in the main text, directly connecting the revised metric, its validation, and the updated conclusions.

​

AuthorsOpenReview ↗

Response for issues 2

Q2:Did Figure 4 and Table 4 use annotated query types or ASR-text routing? How can the contributions of metadata gating and learned admission be distinguished?

A2:Thank you for suggesting clearer attribution of input information and component contributions. Typed retrieval in Figure 4 and Table 4 used annotation-prioritized routing: annotated query types were used when available, with rule-based routing from ASR text as the fallback. The full system also used scheduler-provided speaker/addressee metadata as an independent veto. The “w/o addressee gating” variant removed both metadata rejection and learned addressee gating, making it a combined ablation.

We added two complementary evaluations:

(1) Actual-ASR routing and separate gate interventions. Across the same 24 sessions and 144 questions, we independently toggled metadata and learned addressee gates, holding operation, completeness, and confidence mechanisms fixed, and added an annotated-routing control. Event-level auditing showed that all four gate configurations committed the same 71 of 167 events, with 90.14% admission precision and 56.14% recall. Thus, the gates did not produce different committed-event sets on this subset, and answer differences are not attributed to independent admission-gate benefits.

(2) Direct evaluation of the learned module. With matched training budgets and initialization across three seeds, we separately removed addressee supervision and all auxiliary supervision:

Training configurationAddressee accuracy ↑Learned admission recall ↑
Full model79.01±0.24%53.23±1.09%
Without addressee supervision43.44±14.11%16.49±16.64%
Without auxiliary supervision43.91±12.28%2.21±2.89%

Mean±SD across three seeds on the module-validation set using synthetic noisy transcripts. Removing training supervision and disabling an inference-time gate are distinct interventions.

These results support the module-level contribution of auxiliary supervision to learned admission, separately from the performance of the metadata-assisted system.

We will revise the manuscript accordingly: specify routing sources and metadata access in the Figure 4 and Table 4 captions and main text, relabel the original gate ablation as jointly removing both gates, and present the gate audit, ASR-routing comparison, and learned-module ablations separately in Appendices E.2/E.4, with explicit attribution of what each evaluation establishes.

​

AuthorsOpenReview ↗

Response for issues 3

Q3:Among questions whose supporting evidence was fully delivered to both systems, what are the common-subset size and each system’s accuracy? Does ACT-Mem’s positive gain persist?

A3:Thank you for suggesting that we distinguish evidence delivery from subsequent processing. The fixed-denominator metrics in Appendix B.6 measure end-to-end task completion. To evaluate performance conditional on delivery, we audited all 438 latest-value questions across 219 paired sessions, retaining questions whose supporting audio existed, was fully transmitted, and completed delivery before QA in both systems. Existing answers were evaluated using the same revised scorer.

The common-delivery subset contains 430 questions across 215 sessions, covering 98.17% of all latest-value questions. Accuracy is 33.95–45.12% for MiniCPM-o and 35.12–45.81% for ACT-Mem. On the 343 jointly adjudicable questions, the paired difference is +2.16 percentage points, retaining the direction and a similar magnitude to the full-set estimate of +2.36 points. Its 95% CI is [−4.57, +8.65] points: the positive point estimate persists, while a statistically significant improvement is not established.

MetricFull setCommon-delivery subset
Sessions / questions219 / 438215 / 430
MiniCPM-o latest-value accuracy34.02–44.98%33.95–45.12%
ACT-Mem latest-value accuracy35.39–45.89%35.12–45.81%
Jointly adjudicable questions / represented sessions351 / 212343 / 208
Adjudicable paired difference: ACT-Mem−MiniCPM-o+2.36 pp+2.16 pp
95% CI for the paired difference[−4.01, +8.73][−4.57, +8.65]

Accuracy is session-macro-averaged. Bounds reflect unresolved semantic judgments, not confidence intervals. Paired differences and session-level bootstrap CIs use jointly adjudicable questions. Verified audio delivery does not establish successful comprehension.

We will update the main text, Figure 4 description, and Appendix B.6 to report fixed-denominator and common-delivery results together, specifying selection criteria, effective denominators, and statistical support. This will distinguish end-to-end task completion from perception, memory, and answering performance conditional on evidence delivery.

​

Reviewer_6mNdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DuplexLedger, a bilingual benchmark for evaluating persistent memory in full-duplex speech agents, covering four interaction scenarios with event-level memory operations, explicit state transitions, and same-session QA. It also proposes ACT-Mem, a lightweight memory module combining event admission control, a versioned transactional ledger, and typed retrieval, integrated with a frozen speech backbone. Experiments on five speech models reveal substantial gaps between interaction performance and coherent memory maintenance. On 219 held-out sessions, ACT-Mem improves latest-value accuracy from 4.11% to 8.68% and reduces stale assertions and distractor adoption, but final-state coverage remains very low and interruption stopping becomes slower. While the benchmark and problem formulation are promising, the evidence for methodological novelty, generalization beyond synthetic data, and robust end-to-end memory improvement remains limited.

Critical strengths
  1. A useful benchmark formulation for transactional memory in full-duplex speech. DuplexLedger jointly evaluates interaction scenarios, event-level memory operations, explicit state transitions, and same-session QA. This design enables more fine-grained diagnosis of memory admission, updating, and distractor handling than answer-level evaluation alone, providing a potentially valuable resource for future research.

  2. A lightweight and interpretable memory-management architecture. ACT-Mem combines event admission control, a versioned ledger, and typed retrieval without modifying the speech backbone. Its evaluation shows improved latest-value accuracy and fewer stale assertions and distractor adoptions, offering preliminary evidence that explicit memory-state management can improve selected aspects of memory reliability.

Critical weaknesses
  1. Limited methodological novelty: ACT-Mem primarily combines existing components, including event gating, versioned storage, and retrieval, without sufficiently demonstrating its advantages over strong memory-management baselines. This weakens the paper’s methodological contribution.

  2. Insufficient experimental validation: The benchmark relies on synthetic speech, while absolute memory performance remains poor (8.68% latest-value accuracy and 2.00% final-state coverage). In addition, inconsistencies in reported baseline results and the adaptive challenge protocol raise concerns about evaluation reliability and generalizability.

Questions for authors
  1. Why do the reported MiniCPM-o 4.5 results differ between Sections 4 and 6 (latest-value accuracy: 4.30% vs. 4.11%; final-state coverage: 2.01% vs. 1.27%)? Are the evaluation subsets, denominators, and configurations identical? These discrepancies need clarification to establish the reliability of the main comparison.

  2. How does ACT-Mem substantively differ from existing memory systems combining event gating, versioned storage, and retrieval? Can the authors provide direct comparisons with strong baselines to demonstrate that the proposed design offers meaningful advantages beyond component integration?

  3. Since challenges are triggered adaptively, do different models face different challenge distributions? Would the main conclusions hold under fixed challenge sequences and matched evaluation conditions?

  4. Given the low absolute performance (8.68% latest-value accuracy and 2.00% final-state coverage), what evidence supports the claim that ACT-Mem meaningfully improves transactional memory? Can the authors provide an error breakdown and clarify which improvements are statistically supported?

  5. Since the benchmark uses synthetic dialogues and speech, what evidence supports the applicability of the findings to natural conversations? Without real-world validation, how should the scope of the paper's claims be interpreted?

6 comments
AuthorsOpenReview ↗

Response for issues 1

Q1:The experimental results of MiniCPM-o 4.5 are inconsistent, undermining the reliability of the main conclusions.

A1:Thank you for highlighting the need to clarify the comparison. Sections 4 and 6 use different evaluation sets and denominators: the 4.30%/2.01% results in Section 4 refer to all 1,094 sessions, whereas the 4.11%/1.27% results in Section 6 refer to the 219 held-out sessions. Both use session-level macro-averaging, but aggregate over different session sets.

To make the main comparison directly auditable, we added same-archive subset rescoring. We extracted baseline outputs for the 219 held-out session IDs from the same historical archive used for Section 4, retaining the original answers and execution traces. This requires neither a separately configured baseline run nor answer regeneration. ACT-Mem uses the same frozen backbone in a separate memory-augmented run on those same 219 sessions, and both conditions are evaluated with the revised scorer.

Table: Full-set results and the matched main comparison under unified scoring

Method and evaluation setSessionsLatest-value questionsSummary questionsLatest-value accuracy ↑Summary fact coverage ↑
MiniCPM, Section 4 full set1,0942,1881,09432.08–40.63%36.91–49.48%
MiniCPM, Section 6 subset21943821934.02–44.98%34.54–41.91%
ACT-Mem, identical subset21943821935.39–45.89%39.81–49.28%

Latest-value accuracy is calculated over each session’s latest-value questions; summary coverage is calculated over reference atomic facts, followed by session-level macro-averaging. Ranges represent semantic uncertainty bounds.

The revised scores are computed from frozen outputs, so changes from the original scores reflect the updated scoring definitions. We will explicitly identify the evaluation set, denominator, run provenance, and scorer version in the captions, while reporting new fixed-protocol experiments with different configurations separately.

AuthorsOpenReview ↗

Response for issues 2

Q2:What substantively distinguishes ACT-Mem from existing memory systems, and do direct comparisons with strong baselines demonstrate benefits beyond component integration?

A2:Thank you for the suggestions regarding methodological contribution and direct comparisons. ACT-Mem’s design centers on explicit transactional constraints connecting event eligibility, state updates, and answer evidence during full-duplex interaction. Events must satisfy admission conditions before commitment; only a later, admissible UPDATE for the same entity can supersede an existing value; and retrieval selects active facts according to the question type while preserving provenance and version links. This provides a common, auditable workflow for corrections delivered through interruptions, incomplete utterances, and side conversations.

We added two complementary comparisons. First, we uniformly rescored the historical 219 sessions. Full ACT-Mem achieved 39.81–49.28% summary fact coverage, exceeding the corresponding ranges for FIFO memory, removal of addressee gating, and untyped retrieval.

Table 1: Historical structural ablations, 219 sessions

ConfigurationSummary fact coverage ↑
FIFO memory24.51–33.52%
Without addressee gating23.85–31.52%
Untyped retrieval21.56–30.19%
Full ACT-Mem39.81–49.28%

Second, we compared running summarization, hybrid dense-plus-BM25 RAG, and ACT-Mem using the same MiniCPM backbone, 40 sessions, audio and ASR inputs, and an 800-token evidence cap. On 27 jointly adjudicable sessions, ACT-Mem improved summary coverage over RAG by 19.34 percentage points, with a 95% CI of [9.77, 28.86] , while achieving a higher response rate and using shorter evidence.

Table 2: Controlled comparison of external-memory approaches, 40 sessions

MethodSummary fact coverage ↑Confirmed response rate ↑Median evidence tokens ↓
Running summarization8.97–23.31%46.67%356.5
Hybrid-retrieval RAG3.75–12.50%54.58%684
ACT-Mem21.67–27.88%77.08%84

Ranges reflect unresolved semantic judgments, not confidence intervals. Comparisons are made within each experiment. Tokens are counted using a shared tokenizer.

These results support the complete transactional pipeline’s value for summary coverage and evidence compactness under the evaluated conditions.

We additionally provide an event-level case study from the same conversation (FDMBV2EN_TO2_0103, shown in the revised main text Figure. 5). The user initially schedules IELTS practice for “Friday night, two recording rounds,” corrects this to “Thursday morning, four rounds,” and confirms the revision in a final recap. Four intervening side-talk turns concern webcam backgrounds and deleting old recordings.

This case makes the connection between event admission, correction storage with provenance, question-conditioned evidence selection, and the resulting answer directly inspectable. On this summary question, ACT-Mem provides more compact evidence and higher fact coverage than both external-memory baselines.

​

AuthorsOpenReview ↗

Response for issues 3

Q3:Does adaptive triggering expose different models to different realized challenges, and do the main conclusions hold under fixed challenge sequences and matched evaluation conditions?

A3:Thank you for raising the issue of challenge distributions and evaluation conditions. Adaptive triggering in the original protocol ensures that full-duplex challenges are behaviorally meaningful: interruptions and backchannels require ongoing assistant speech. Models share the planned audio, event order, questions, and triggering rules, while dispatch timing follows observed speech activity. Realized challenge sets can therefore vary with model behavior, which we document through challenge realization rate (RCR), eligible denominators, and conditional behavioral metrics.

To additionally examine performance under a fixed input plan, we evaluated five systems on the same 24 sessions, 144 questions, and 167 planned audio events, including 12 interruptions and 23 backchannels. Challenges were dispatched at predefined positions without waiting for assistant speech eligibility. We separately recorded complete delivery, actual dispatch deviations, and VAD-based eligibility. All systems fully transmitted 167/167 audio events.

Table: Eligible opportunities and memory performance under the fixed challenge plan

ModelEligible/planned interruptionsEligible/planned backchannelsDispatch deviation p95 (s)Summary fact coverage ↑
Moshi0/120/237.8470.00%
Freeze-Omni4/129/235.3632.03–3.90%
PersonaPlex6/122/237.1840.52%
Lychee-FD0/120/234.0530.00%
MiniCPM-o 4.50/121/238.95438.47–54.04%

Dispatch deviations are measured relative to planned times. Stopping metrics are NA without eligible interruptions. Coverage bounds reflect unresolved semantic judgments, not confidence intervals.

Interaction opportunities and memory outcomes remain distinct after complete delivery. For example, Freeze-Omni stopped successfully in all four eligible interruptions, while its summary coverage was 2.03–3.90%, illustrating the complementary dimensions of interaction response and memory retention. Furthermore, in the fixed-plan comparison with a shared QA-endpoint adaptation, Freeze-Omni+ACT-Mem achieved 68.75–72.92% latest-value accuracy, versus 31.25–37.50% without external memory and 50.00–58.33% with RAG; all three conditions maintained a 100% response rate.

These findings support joint interaction–memory diagnosis and provide evidence of memory benefits under the evaluated fixed conditions. The revision will present both protocols, interpreting specific rankings and conclusions using their respective delivery records, eligible denominators, and observed timing.

​

Reviewer_Gn6vOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce a synthetic benchmark for evaluating retrieval accuracy of full-duplex speech assistants. The benchmark explicitly models key-value storage of intermediate facts and the sequence of actions for updating the memory states. This is a form of dialog modeling with slot filling without pre-defined slots.

They also introduce ACT-Mem, an auxiliary model to improve interaction with the external memory that can be used to augment the main assistant model inputs with data from memory. The model can be trained in combination with any frozen LLM, though they only evaluate it on one. The results are difficult to interpret.

Critical strengths
  1. It can be useful to have an explicitly auditable memory so that failures can be attributed and mitigated more effectively. Handling of the ever-growing conversation context can be difficult to inspect and the retrieval accuracy can be low in very long conversations.
  2. Formalization of the memory operations can be a useful tool for automatic evaluation and tracking of models' failures. Without it, the model's internal state evolution is difficult to interrogate.
Critical weaknesses

Main weaknesses

  1. How do we know the value of the proposed metrics? Does the final-state accuracy predict the quality in e.g. spoken Q&A or needle-retrieval benchmarks? I don't think the authors have established sufficient evidence for the predictive power of their framework. Many of the metrics seem to be good at describing the internal state of the models, but for practical purposes - who cares? If the explicit memory tracking is supposed to help the models to give better answers, where is the eval showing that the answers are better?
  2. The framework is very narrow in scope: (a) not everything worth remembering can be expressed as key-value pairs; it seems very difficult to make it universal enough to handle inquires like "do you remember how I felt about this idea at the beginning of our conversation?" (b) All results come from a single, custom synthetic dataset and (in case of ACT-mem) a single LLM backbone. How can we trust its generalizability? I understand that human evals are expensive, but there must be ways of evaluating the models on existing benchmarks or to employ LLM-as-a-judge raters?

Other issues

  1. It is not clear if the modeling decisions leading to heuristics in Eq. 3 and Eq. 5 are well justified. They are not derived from anything fundamental and appear to be the result of manual tweaking.
  2. The framework introduces tons of very specific terms and concepts that are not explicitly defined and make it difficult to follow.
  3. The huge amount of appendices is overwhelming. Maybe some answers to my questions can be found there, I can't tell.
  4. If I have only a limited amount of time to spend on improving my spoken LLM, I am not convinced the time is well spent tuning the many meta-parameters of the proposed approach, instead of exploring general LLM techniques like compacting of context (i.e. memory compression), background thinker module, lightweight summarization etc. Models improve constantly, so patching the deficiency of any particular model is difficult to justify.
Questions for authors

n/a

6 comments
AuthorsOpenReview ↗

Response for Weakness 1

Q1: Do these metrics reflect actual answer quality? Does ACT-Mem improve the quality of the model’s answers?

A1:Thank you for highlighting metric utility and answer quality. LVA and the revised summary fact coverage (SFC) directly evaluate generated answers: LVA assesses recovery of updated facts, while SFC measures coverage of active reference facts in a summary. In full-duplex speech, responding promptly to an interruption and correctly using the revised information in subsequent answers are distinct capabilities. Our memory metrics assess the latter, alongside stale-value errors and distractor adoption.

We added two complementary evaluations.

(1) Independent blinded assessment of metric validity. An InternLM judge from an independent model family assessed answer correctness with system identities hidden and both answer orders presented. For jointly adjudicable sessions from the historical 219-session outputs and new runs on 40 held-out sessions, we correlated per-session ACT-minus-baseline metric changes with correctness changes. Both LVA and SFC showed positive associations across the two output sets, supporting within-task validity for answer evaluation. This analysis assesses association with answer quality rather than cross-benchmark predictive validity.

Table 1: Correlations between metric changes and independently assessed correctness changes

Output setMetricAdjudicable paired sessionsSpearman’s ρ
Historical 219 sessionsLVA1390.486
Historical 219 sessionsSFC1090.368
New runs on 40 sessionsLVA220.463
New runs on 40 sessionsSFC240.332

(2) Controlled evaluation of spoken-answer improvements. On a second speech backbone, Freeze-Omni, we compared no external memory, RAG, and ACT-Mem using the same 24 sessions, fixed challenges, and a shared QA-endpoint adaptation. ACT-Mem achieved 68.75–72.92% LVA, with fewer stale-value errors and less distractor adoption; all three conditions maintained a 100% response rate. Its summary coverage exceeded the no-memory baseline, while its bounds overlapped with RAG’s.

Table 2: Final-answer performance on Freeze-Omni, 24 sessions

MetricNo external memoryRAGACT-Mem
Latest-value accuracy ↑31.25–37.50%50.00–58.33%68.75–72.92%
Summary fact coverage ↑2.03–3.90%23.18–31.87%25.87–31.90%
Stale-value error ↓6.25–10.42%4.17%2.08%
Distractor adoption ↓20.83–25.00%20.83–25.00%0.00–4.17%
Confirmed response rate ↑100%100%100%

Ranges reflect unresolved scoring judgments, not confidence intervals.

These results support the metrics’ association with answer correctness and ACT-Mem’s benefits for updated-fact recovery and exclusion of incorrect content on the evaluated speech backbone, respectively. We will foreground these complementary findings and report interaction timing and answer-content quality separately.

AuthorsOpenReview ↗

Response for Weakness 2

Q2(a): What types of information can key–value memory represent, and how does it relate to emotional and episodic memory? (b): What evidence supports generalization beyond a single synthetic dataset and speech backbone, and can LLM-as-a-judge evaluation provide additional validation?

A2:

Thank you for the suggestions regarding research scope and generalization.

(a) Emotional and episodic memory. This paper studies memory management in complex full-duplex speech scenarios: incorporating corrections delivered through interruptions, retaining valid facts, and preventing backchannels and side conversations from contaminating memory. The key–value representation makes facts and their changes explicit, allowing admission, updating, and exclusion to be verified and traced. Emotional and episodic memory are beyond the primary focus of this work. We plan to explore their integration with full-duplex memory management, including evolving emotional states and recollection of past experiences.

(b) Cross-backbone validation and LLM-based assessment. We added controlled experiments on a second speech backbone, Freeze-Omni, using the same 24 sessions, fixed challenges, and a shared QA-endpoint adaptation. ACT-Mem achieved 68.75–72.92% latest-value accuracy, alongside fewer stale-value errors and less distractor adoption, while all three conditions maintained a 100% response rate. These results support factual-updating and selective-memory benefits beyond the original MiniCPM backbone.

Table 1: Answer performance on Freeze-Omni, 24 sessions

MetricNo external memoryRAGACT-Mem
Latest-value accuracy ↑31.25–37.50%50.00–58.33%68.75–72.92%
Summary fact coverage ↑2.03–3.90%23.18–31.87%25.87–31.90%
Stale-value error ↓6.25–10.42%4.17%2.08%
Distractor adoption ↓20.83–25.00%20.83–25.00%0.00–4.17%
Confirmed response rate ↑100%100%100%

Ranges reflect unresolved scoring judgments, not confidence intervals.

Following the reviewer’s suggestion, we also introduced InternLM and Qwen32 for identity-blinded, order-balanced evaluation of the historical 219-session outputs and new runs on 40 sessions. Both judges showed positive correlations between changes in memory metrics and changes in answer correctness, providing complementary evidence of within-task metric validity.

Table 2: Spearman correlations between metric changes and blinded correctness changes on historical outputs

JudgeLVA: 139 adjudicable paired sessionsSFC: 109 adjudicable paired sessions
InternLM0.4860.368
Qwen320.5100.417

Correlations use per-session ACT-minus-baseline differences. InternLM is an independent model family; Qwen32 is shared with the primary scorer.

(c) Natural conversations. We are conducting additional natural-conversation experiments to examine whether these findings extend to natural full-duplex spoken interactions. We will report the results upon completion and refine the scope of our claims accordingly. The completed experiments currently support cross-backbone benefits under controlled conditions and the association between memory metrics and answer quality.

AuthorsOpenReview ↗

Response for issues 1, 2, 3

Q3:What motivates the designs of Eqs. 3 and 5, and how have their coefficients been validated?

A3:Thank you for requesting clarification of the modeling rationale. Our design addresses specific memory risks in full-duplex speech.

Equation 3 implements a task-driven soft conjunction. Addressee prediction filters side conversations; completeness prediction discourages premature commitment of unfinished utterances; WRITE/UPDATE probabilities identify facts or corrections requiring admission; and confidence constrains uncertain proposals. Since all factors lie in [0,1][0,1], their product cannot exceed any individual factor. Consequently, high scores on other prerequisites cannot compensate for weak evidence on one prerequisite. We interpret this as a conservative admission score.

Equation 5 turns these decisions into supervised learning objectives. Classification losses supervise operation, addressee, and completeness predictions; the confidence loss supervises reliability estimates; and the ranking term prioritizes valid supporting evidence over superseded facts and distractors.

We added ablations with a common training budget, matched initialization, and fixed final checkpoints: six configurations across three seeds, plus four single-seed weight-sensitivity settings. The results support the contributions of auxiliary supervision to admission and confidence prediction, and of the ranking term to mean retrieval performance.

Table: Module-validation results, mean ± SD across three seeds

ConfigurationAdmission precision (%) ↑Admission recall (%) ↑Confidence Brier ↓Retrieval Top-1 (%) ↑
Full model85.61 ± 0.9853.23 ± 1.090.1292 ± 0.001546.48 ± 2.55
Without auxiliary losses76.44 ± 25.122.21 ± 2.890.2588 ± 0.018546.48 ± 2.55
Without ranking loss85.61 ± 0.9853.23 ± 1.090.1292 ± 0.001543.28 ± 1.97

Module experiments use synthetic noisy transcripts with reference-transcript fallback; retrieval is evaluated on annotated candidate sets.

In the local sensitivity study, halving/doubling the auxiliary weights yielded admission precision of 85.61%/85.88% and recall of 54.39%/54.87%. Changing the ranking weight from 0.5 to 0.25/1.0 yielded retrieval Top-1 scores of 45.14%/44.34%. Separately, in the original 40-session metadata-assisted system calibration, thresholds of 0.35, 0.45, and 0.50 yielded F1 scores of 71.19%, 71.19%, and 70.75%.

These findings provide empirical support for the task-driven design, the objectives’ module-level contributions, and local stability around the tested settings. We will incorporate this rationale into the main text and explicitly distinguish module validation from system-level calibration.

Q4:How can the terminology be clearly defined and the method made easier to understand?

A4:Thank you for highlighting the presentation issues. We will reduce unnecessary terminology, standardize naming and notation, and provide concise definitions when core concepts first appear. We will also use a running example to explain how event admission, memory updating, and retrieval connect.

Q5:How can the appendices be streamlined and key evidence made easier to locate?

A5:Thank you for the suggestion on organization. We will foreground essential definitions and key evidence in the main text, with explicit references to the relevant appendix sections, tables, or equations alongside methodological explanations and experimental conclusions. We will also remove repetitive material and consolidate supplementary experiments and reproducibility details by topic, making supporting evidence easier to locate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.