- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper first analyzes the contents of KV caches in full-duplex speech models and the effects of removing different token types. The results show that the cost of losing a token also depends on content and on whether the token belongs to the user or to the agent, rather than just the token’s recency and accumulated attention. They also show the fact that eviction under a budget can leave the model taking the turn while the reply loses its content. This paper then proposes DuplexKV, a lightweight gate to be combined with attention and recency at inference. It also proposes Judged Eviction Tuning (JET), which tunes the gate on an LLM judge’s verdicts about the replies the model generates. Those combined scored highest on the interruption task of Full-Duplex-Bench.
Critical strengths
- Strong response quality while maintaining high takeover rates.
- Good analysis of cache contents.
Critical weaknesses
- Methodological advances need clearer isolation.
- Potential problems with full fine-tuning need to be addressed.
Questions for authors
- See critical weaknesses.
- Can the authors additionally fine-tune the backbone on the same data without learned eviction, then evaluate with existing eviction policies under the same cache budgets? This can better separate the benefits of the proposed gate from the backbone adaptation.
- Can the authors explain why there are performance degradations for Raon-SpeechChat when M = 32 in Table 1?
- Furthermore, does joint fine-tuning affect the backbone’s performance beyond interruption responses?
- Can the authors provide measured VRAM usage, inference latency, and throughput during inference to better demonstrate the practical benefits beyond the calculated KV cache reduction?
6 comments
Response to Reviewer QR8n (Cont.)
- To cover smaller budgets and the budgets matched in seconds of input that [Q2] below uses, we repeat the comparison at , 19 and 24 on MiniCPM-o 4.5 and on Raon-SpeechChat, with as in Tables R3.1 and R3.2 and the levels and TOR in Figure 22.
Table R3.3: Policies on the Full KV fine-tune at smaller budgets against DuplexKV + JET, MiniCPM-o 4.5.
| Policy | , | 95% CI | , | 95% CI | , | 95% CI |
|---|---|---|---|---|---|---|
| AudioKV | +1.734 | [+1.457, +2.000] | +1.350 | [+1.075, +1.615] | +0.920 | [+0.685, +1.150] |
| R-KV | +2.455 | [+2.195, +2.710] | +2.680 | [+2.415, +2.935] | +1.665 | [+1.405, +1.920] |
| StreamingLLM | +2.710 | [+2.465, +2.950] | +3.135 | [+2.900, +3.365] | +3.080 | [+2.850, +3.295] |
| DefensiveKV | +2.520 | [+2.275, +2.760] | +3.145 | [+2.905, +3.375] | +2.195 | [+1.925, +2.455] |
| Ada-KV | +2.655 | [+2.400, +2.900] | +3.010 | [+2.760, +3.245] | +2.440 | [+2.170, +2.705] |
| Attn + recency | +1.980 | [+1.700, +2.250] | +1.785 | [+1.520, +2.045] | +0.530 | [+0.330, +0.730] |
- On MiniCPM-o 4.5 DuplexKV + JET is above every policy at these budgets, by 0.530 to 3.145, with a TOR of at least 0.995 for every method.
Table R3.4: Policies on the Full KV fine-tune at against DuplexKV + JET, Raon-SpeechChat.
| Policy | , | 95% CI | n |
|---|---|---|---|
| AudioKV | +0.892 | [+0.640, +1.151] | 186 |
| R-KV | +0.478 | [+0.266, +0.690] | 184 |
| StreamingLLM | +2.201 | [+1.862, +2.541] | 159 |
| DefensiveKV | +0.175 | [-0.021, +0.378] | 143 |
| Ada-KV | +0.651 | [+0.417, +0.896] | 192 |
| Attn + recency | +0.673 | [+0.442, +0.909] | 165 |
- On Raon-SpeechChat DuplexKV + JET is above every policy, by 0.175 to 2.201. The smallest gap, DefensiveKV at , comes with a TOR of 0.720 against 0.985 for DuplexKV + JET. Appendix L gives both tables.
- Since these policies run on a backbone adapted with the same data and recipe, DuplexKV + JET keeps a margin beyond the adaptation of the backbone.
- On the jointly trained backbone itself, DuplexKV + JET obtains a higher judge score than all 15 baselines run on this backbone, at both budgets. At the closest are DefensiveKV on Raon-SpeechChat, by +0.389, and AudioKV on MiniCPM-o 4.5, by +0.095 (Table R1.5 in our response to Reviewer nEe8).
- With the backbone and the other two channels fixed, removing the gate channel lowers the judge score of DuplexKV in all eight combinations of backbone, evaluation set and budget (Figure 17 of the paper).
- Against fine-tunes that replace our gate objective with that of TrimKV or LKV under the same joint training, DuplexKV before JET obtains a significantly higher judge score than the TrimKV objective at on MiniCPM-o 4.5 and at both budgets on Raon-SpeechChat, and on Raon-SpeechChat DuplexKV takes the turn more often than both objectives, by 0.250 to 0.765 in TOR (Table R1.8 in our response to Reviewer nEe8).
Response to Reviewer QR8n (Cont.)
[W2/Q3] Effect of joint fine-tuning beyond interruption responses
- To look beyond interruption responses, we evaluate three further FDB v1.0 tasks on MiniCPM-o 4.5, pause handling on two sets (synthetic, n = 137, and CANDOR, n = 216), turn-taking on CANDOR (n = 119) and backchannel (n = 55), for the released weights, both fine-tunes with the Full KV cache and DuplexKV + JET at and 32. In a pause the model should keep listening (lower TOR is better), at the end of a turn the model should take the turn (higher TOR is better), and its backchannels should follow the human timing (lower JSD is better). JSD is the Jensen-Shannon divergence of the model's backchannel timing distribution against the human ground truth of the benchmark, and frequency is backchannels per second.
Table R3.5: MiniCPM-o 4.5 on three further FDB v1.0 tasks.
| Setting | Synthetic pause TOR | CANDOR pause TOR | CANDOR turn-taking TOR | Backchannel JSD | Backchannel frequency |
|---|---|---|---|---|---|
| Full KV, released weights | 0.146 | 0.352 | 0.874 | 1.0000 | 0.0000 |
| Full KV, Full KV fine-tune | 0.051 | 0.144 | 0.983 | 0.7739 | 0.0637 |
| Full KV, jointly trained backbone | 0.044 | 0.088 | 0.966 | 0.7633 | 0.0388 |
| DuplexKV + JET, | 0.044 | 0.097 | 1.000 | 0.7632 | 0.0388 |
| DuplexKV + JET, | 0.015 | 0.088 | 1.000 | 0.7636 | 0.0366 |
- Every fine-tuned setting, with the Full KV cache or with DuplexKV + JET, takes the turn less often in pauses, by 0.095 to 0.264 in TOR, and more often at the end of a turn, by 0.092 to 0.126, than the released weights. The released weights produce no backchannel in the 55 episodes (frequency 0.0000, JSD 1.0000), and every fine-tuned setting backchannels at 0.0366 to 0.0637 per second with a JSD of 0.7632 to 0.7739.
- Against the Full KV cache on the same jointly trained backbone, the backchannel JSD of DuplexKV + JET differs by at most 0.0003 at and 64 (n = 55). In Table R3.5 fine-tuning on our data improves pause handling, turn-taking and backchannel, and DuplexKV + JET keeps these improvements at and 64.
- Please refer to Appendix M for the full results and confidence intervals.
Response to Reviewer QR8n
We thank the reviewer for pointing to the reply quality that DuplexKV keeps at high takeover rates. The reviewer asks whether this gain comes from the gate or from adapting the backbone, and we answer this first by comparing existing policies on a backbone fine-tuned with our recipe and the gate removed, as the reviewer suggested. For the remaining questions we add three further tasks of Full-Duplex-Bench v1.0 (FDB v1.0), budgets matched in seconds of input, and the real-time serving cost.
[W1/Q1] Isolating the gate from backbone adaptation
Table R3.1: DuplexKV + JET against each policy on the Full KV fine-tune, Raon-SpeechChat, same budget.
Table R3.2: DuplexKV + JET against each policy on the Full KV fine-tune, MiniCPM-o 4.5, same budget.