- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DuplexLedger, a synthetic English–Chinese benchmark of 1,094 full-duplex sessions evaluating whether speech agents correctly store, revise, and exclude information through event-level memory operations, state transitions, and spoken QA. The proposed ACT-Mem combines an admission controller, a versioned ledger, and a typed retriever for external memory management. Retrieved evidence is supplied through text prefill to a frozen MiniCPM-o 4.5 only during QA. The evaluation covers five full-duplex systems; on 219 held-out sessions, ACT-Mem increases latest-value accuracy from 4.11% to 8.68% over the unaugmented backbone.
Critical strengths
- The benchmark separates interaction timing from memory updates by aligning adaptive challenges with state transitions and supporting/forbidden evidence. Stopping or continuing appropriately does not establish correct storage or revision. Evaluating which information should be admitted or excluded, beyond ordinary recall, is a useful contribution.
- Transaction traces help localize failures that final answers alone cannot reveal. In Table 21, UPDATE is predicted correctly, but inconsistent entity keys leave the original values active without supersession. This distinguishes failure to recognize a correction from failure to apply a recognized correction to the memory state.
Critical weaknesses
The stated final-state coverage metric can penalize complete, correct summaries. Appendix C.7, Eq. (29), compares each reference fact against the entire answer, requiring token F1 ≥ 0.5. For four equal-length facts with non-overlapping tokens, an answer correctly concatenating all four obtains per fact and therefore 0% coverage. Answering only one fact instead receives 25% coverage. Thus, adding correct information can reduce the score.
This counterexample concerns the stated definition; the actual implementation and its impact on the reported results remain unverified. Since Section 4.4 uses the maximum coverage of 2.01% as evidence of state-consolidation failure, scorer validation is necessary before interpreting this value as the extent of that failure.
Questions for authors
- Does the implemented scorer compare each reference fact with the entire answer or a matching span/fact? What coverage is obtained when reference summaries themselves are used as predictions? If implemented as described in Eq. (29), does corrected scoring preserve the conclusion in Section 4.4?
- Did Figure 4 and Table 4 use annotated query types or ASR-text routing? Appendix E.2 describes a speaker/addressee metadata gate that can block admission independently of controller predictions. Please distinguish evidence for learned admission itself from results for the metadata-assisted system.
- Appendix B.6 retains planned reference targets even when a challenge is not delivered, meaning that memory scores also reflect missed evidence delivery. For latest-value questions with sufficient supporting evidence delivered in both the backbone and ACT-Mem runs, what are the common-subset size and each system’s accuracy? This would clarify whether the gain persists in subsequent memory and answer processing.
8 comments
Response for issues 2
Q2:Did Figure 4 and Table 4 use annotated query types or ASR-text routing? How can the contributions of metadata gating and learned admission be distinguished?
A2:Thank you for suggesting clearer attribution of input information and component contributions. Typed retrieval in Figure 4 and Table 4 used annotation-prioritized routing: annotated query types were used when available, with rule-based routing from ASR text as the fallback. The full system also used scheduler-provided speaker/addressee metadata as an independent veto. The “w/o addressee gating” variant removed both metadata rejection and learned addressee gating, making it a combined ablation.
We added two complementary evaluations:
(1) Actual-ASR routing and separate gate interventions. Across the same 24 sessions and 144 questions, we independently toggled metadata and learned addressee gates, holding operation, completeness, and confidence mechanisms fixed, and added an annotated-routing control. Event-level auditing showed that all four gate configurations committed the same 71 of 167 events, with 90.14% admission precision and 56.14% recall. Thus, the gates did not produce different committed-event sets on this subset, and answer differences are not attributed to independent admission-gate benefits.
(2) Direct evaluation of the learned module. With matched training budgets and initialization across three seeds, we separately removed addressee supervision and all auxiliary supervision:
| Training configuration | Addressee accuracy ↑ | Learned admission recall ↑ |
|---|---|---|
| Full model | 79.01±0.24% | 53.23±1.09% |
| Without addressee supervision | 43.44±14.11% | 16.49±16.64% |
| Without auxiliary supervision | 43.91±12.28% | 2.21±2.89% |
Mean±SD across three seeds on the module-validation set using synthetic noisy transcripts. Removing training supervision and disabling an inference-time gate are distinct interventions.
These results support the module-level contribution of auxiliary supervision to learned admission, separately from the performance of the metadata-assisted system.
We will revise the manuscript accordingly: specify routing sources and metadata access in the Figure 4 and Table 4 captions and main text, relabel the original gate ablation as jointly removing both gates, and present the gate audit, ASR-routing comparison, and learned-module ablations separately in Appendices E.2/E.4, with explicit attribution of what each evaluation establishes.
Response for issues 3
Q3:Among questions whose supporting evidence was fully delivered to both systems, what are the common-subset size and each system’s accuracy? Does ACT-Mem’s positive gain persist?
A3:Thank you for suggesting that we distinguish evidence delivery from subsequent processing. The fixed-denominator metrics in Appendix B.6 measure end-to-end task completion. To evaluate performance conditional on delivery, we audited all 438 latest-value questions across 219 paired sessions, retaining questions whose supporting audio existed, was fully transmitted, and completed delivery before QA in both systems. Existing answers were evaluated using the same revised scorer.
The common-delivery subset contains 430 questions across 215 sessions, covering 98.17% of all latest-value questions. Accuracy is 33.95–45.12% for MiniCPM-o and 35.12–45.81% for ACT-Mem. On the 343 jointly adjudicable questions, the paired difference is +2.16 percentage points, retaining the direction and a similar magnitude to the full-set estimate of +2.36 points. Its 95% CI is [−4.57, +8.65] points: the positive point estimate persists, while a statistically significant improvement is not established.
| Metric | Full set | Common-delivery subset |
|---|---|---|
| Sessions / questions | 219 / 438 | 215 / 430 |
| MiniCPM-o latest-value accuracy | 34.02–44.98% | 33.95–45.12% |
| ACT-Mem latest-value accuracy | 35.39–45.89% | 35.12–45.81% |
| Jointly adjudicable questions / represented sessions | 351 / 212 | 343 / 208 |
| Adjudicable paired difference: ACT-Mem−MiniCPM-o | +2.36 pp | +2.16 pp |
| 95% CI for the paired difference | [−4.01, +8.73] | [−4.57, +8.65] |
Accuracy is session-macro-averaged. Bounds reflect unresolved semantic judgments, not confidence intervals. Paired differences and session-level bootstrap CIs use jointly adjudicable questions. Verified audio delivery does not establish successful comprehension.
We will update the main text, Figure 4 description, and Appendix B.6 to report fixed-denominator and common-delivery results together, specifying selection criteria, effective denominators, and statistical support. This will distinguish end-to-end task completion from perception, memory, and answering performance conditional on evidence delivery.
Response for issues 1
Q1:Does whole-answer matching underestimate fact coverage? How does the scorer perform when reference summaries are used as predictions, and do the conclusions in Section 4.4 hold after correcting the scoring?
A1:Thank you for identifying this scoring issue. The original scorer indeed computed token F1 between each reference fact and the entire answer, using a coverage threshold of 0.5 without local-span matching. Consequently, adding correct facts could reduce the score for an individual fact.
We revised the metric to atomic-fact semantic coverage (SFC). Reference facts are first verified, and Qwen2.5-32B and Mistral-Small-3.1-24B assess whether each fact is correctly expressed in the answer, with bounds reported for unresolved judgments. This revision removes the whole-answer lexical-F1 length penalty. The scoring logic passed 17 deterministic tests. When reference summaries were used as predictions, all 1,081 adjudicable summaries out of 1,094 achieved 100% coverage; 13 retained unresolved reference-verification judgments, yielding overall coverage of 99.13–100%.
We then uniformly rescored the frozen outputs of all five systems across the same 1,094 sessions:
Ranges represent bounds from unresolved semantic judgments, not confidence intervals. Model outputs and interaction trajectories are unchanged.
These results require updating the absolute-coverage argument in Section 4.4 and support a more precise conclusion: latest-fact recovery and final-summary coverage exhibit distinct performance profiles following full-duplex interaction. For example, Lychee-FD achieves the highest latest-value accuracy, whereas MiniCPM achieves higher summary coverage, supporting separate evaluation of these capabilities.
We will revise the manuscript accordingly: update Eq. (29) and Appendix C.7 with the revised scoring definition and treatment of unresolved judgments, replace Table 3, and rewrite Section 4.4 to replace the “maximum coverage of only 2.01%” statement and its interpretation. We will also include the reference-summary self-scoring results in the main text, directly connecting the revised metric, its validation, and the updated conclusions.