acceptodds
Under review as a conference paper at ICLR 2027

DuplexKV: Understanding and Learning KV Cache Eviction in Full-Duplex Speech Models

Abstract

Full-duplex spoken dialogue models listen and speak at once, and every audio frame of both speakers appends key-value states to the cache, so its memory grows with elapsed session time, even during silence, rather than only with the amount of speech exchanged. We first analyze what this cache holds on two full-duplex models, MiniCPM-o 4.5 and Raon-SpeechChat. The cost of losing a token depends on its content and on whether the token belongs to the user or to the agent, which neither recency nor accumulated attention measures. Eviction under a budget can also leave the model taking the turn while the reply loses its content, a failure that the next-token loss tracks poorly because the loss scores a given reply token by token rather than the reply that the model generates under eviction. We propose DuplexKV, a lightweight -Gate trained jointly with the backbone under a fixed cache budget with an -regularized objective and combined with attention and recency at inference, together with Judged Eviction Tuning (JET), which tunes the gate on an LLM judge's verdicts about the replies the model generates. On the user interruption task of Full-Duplex-Bench, DuplexKV with JET attains the highest judge score among prior eviction policies on both backbones even at small budgets, responding to almost every interruption, and its advantage grows as conversations become longer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-duplex speech language models, KV cache compression, Learned cache eviction, Memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QR8nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper first analyzes the contents of KV caches in full-duplex speech models and the effects of removing different token types. The results show that the cost of losing a token also depends on content and on whether the token belongs to the user or to the agent, rather than just the token’s recency and accumulated attention. They also show the fact that eviction under a budget can leave the model taking the turn while the reply loses its content. This paper then proposes DuplexKV, a lightweight gate to be combined with attention and recency at inference. It also proposes Judged Eviction Tuning (JET), which tunes the gate on an LLM judge’s verdicts about the replies the model generates. Those combined scored highest on the interruption task of Full-Duplex-Bench.

Critical strengths
  • Strong response quality while maintaining high takeover rates.
  • Good analysis of cache contents.
Critical weaknesses
  • Methodological advances need clearer isolation.
  • Potential problems with full fine-tuning need to be addressed.
Questions for authors
  • See critical weaknesses.
    • Can the authors additionally fine-tune the backbone on the same data without learned eviction, then evaluate with existing eviction policies under the same cache budgets? This can better separate the benefits of the proposed gate from the backbone adaptation.
    • Can the authors explain why there are performance degradations for Raon-SpeechChat when M = 32 in Table 1?
    • Furthermore, does joint fine-tuning affect the backbone’s performance beyond interruption responses?
  • Can the authors provide measured VRAM usage, inference latency, and throughput during inference to better demonstrate the practical benefits beyond the calculated KV cache reduction?
6 comments
AuthorsOpenReview ↗

Response to Reviewer QR8n

We thank the reviewer for pointing to the reply quality that DuplexKV keeps at high takeover rates. The reviewer asks whether this gain comes from the gate or from adapting the backbone, and we answer this first by comparing existing policies on a backbone fine-tuned with our recipe and the gate removed, as the reviewer suggested. For the remaining questions we add three further tasks of Full-Duplex-Bench v1.0 (FDB v1.0), budgets matched in seconds of input, and the real-time serving cost.

  • As in Table 1, Judge / TOR gives the interruption Judge and the takeover rate (TOR). Unless stated otherwise, each ΔJ\Delta J is paired over the n episodes in which both settings take the turn, with a 95% bootstrap CI.

[W1/Q1] Isolating the gate from backbone adaptation

  • For each backbone we trained a Full KV fine-tune with the data, 300 steps, learning rates, batch and seed of DuplexKV, with the gate removed and the Full KV cache during training. On this fine-tune we run the existing policies StreamingLLM, R-KV, Ada-KV, DefensiveKV, SnapKV, AudioKV and, on MiniCPM-o 4.5, TOVA, as well as the attention and recency channels of DuplexKV with the gate channel off (Attn + recency), and compare each with DuplexKV + JET on the jointly trained backbone at the same budget.
  • Figure 22 in Appendix L of the revised paper shows the interruption Judge (upper panels) and TOR (lower panels) of every method by budget, with Full KV on the released weights (dotted) and on the Full KV fine-tune (dashed).
  • DuplexKV + JET obtains a higher judge score than every policy on the Full KV fine-tune at both budgets on both backbones, and each ΔJ\Delta J compares the replies of episodes in which both methods take the turn. On Raon-SpeechChat at M=32M=32 DuplexKV + JET also takes the turn more often, in 0.985 of the episodes against 0.540 to 0.885 for the policies, and on MiniCPM-o 4.5 every method takes the turn in every episode.
  • In Tables R3.1 and R3.2, ΔJ=JDuplexKV + JET−Jpolicy\Delta J = J_{\text{DuplexKV + JET}} - J_{\text{policy}}, and TOR is in the lower panels of Figure 22.

Table R3.1: DuplexKV + JET against each policy on the Full KV fine-tune, Raon-SpeechChat, same budget.

PolicyM=32M=32, ΔJ\Delta J95% CInM=64M=64, ΔJ\Delta J95% CIn
StreamingLLM+3.082[+2.789, +3.361]147+1.692[+1.385, +2.006]169
Ada-KV+1.695[+1.383, +2.024]167+0.551[+0.348, +0.763]198
DefensiveKV+0.523[+0.196, +0.879]107+0.351[+0.179, +0.536]168
R-KV+0.549[+0.297, +0.800]175+0.516[+0.335, +0.697]188
AudioKV+1.760[+1.440, +2.080]175+0.570[+0.378, +0.772]193
SnapKV+1.636[+1.335, +1.938]176+0.288[+0.094, +0.482]191
Attn + recency+1.951[+1.622, +2.294]143+0.739[+0.497, +0.988]161
  • On Raon-SpeechChat DuplexKV + JET is above every policy, by 0.523 to 3.082 at M=32M=32 and 0.288 to 1.692 at M=64M=64.

Table R3.2: DuplexKV + JET against each policy on the Full KV fine-tune, MiniCPM-o 4.5, same budget.

PolicyM=32M=32, ΔJ\Delta J95% CInM=64M=64, ΔJ\Delta J95% CIn
StreamingLLM+2.765[+2.530, +2.990]200+0.775[+0.585, +0.970]200
R-KV+0.480[+0.340, +0.625]200+0.160[+0.050, +0.265]200
AudioKV+0.260[+0.145, +0.380]200+0.075[-0.045, +0.205]200
DefensiveKV+0.295[+0.185, +0.410]200+0.080[-0.055, +0.210]200
TOVA+0.785[+0.595, +0.980]200+0.105[-0.040, +0.250]200
Ada-KV+1.025[+0.815, +1.235]200+0.100[-0.010, +0.210]200
SnapKV+0.840[+0.650, +1.040]200+0.170[+0.035, +0.300]200
Attn + recency+0.260[+0.125, +0.395]200+0.095[-0.035, +0.220]200
  • On MiniCPM-o 4.5 DuplexKV + JET is above every policy, by 0.260 to 2.765 at M=32M=32 and 0.075 to 0.775 at M=64M=64, where five policies are within 0.105.
AuthorsOpenReview ↗

Response to Reviewer QR8n (Cont.)

  • To cover smaller budgets and the budgets matched in seconds of input that [Q2] below uses, we repeat the comparison at M=16M=16, 19 and 24 on MiniCPM-o 4.5 and M=53M=53 on Raon-SpeechChat, with ΔJ\Delta J as in Tables R3.1 and R3.2 and the levels and TOR in Figure 22.

Table R3.3: Policies on the Full KV fine-tune at smaller budgets against DuplexKV + JET, MiniCPM-o 4.5.

PolicyM=16M=16, ΔJ\Delta J95% CIM=19M=19, ΔJ\Delta J95% CIM=24M=24, ΔJ\Delta J95% CI
AudioKV+1.734[+1.457, +2.000]+1.350[+1.075, +1.615]+0.920[+0.685, +1.150]
R-KV+2.455[+2.195, +2.710]+2.680[+2.415, +2.935]+1.665[+1.405, +1.920]
StreamingLLM+2.710[+2.465, +2.950]+3.135[+2.900, +3.365]+3.080[+2.850, +3.295]
DefensiveKV+2.520[+2.275, +2.760]+3.145[+2.905, +3.375]+2.195[+1.925, +2.455]
Ada-KV+2.655[+2.400, +2.900]+3.010[+2.760, +3.245]+2.440[+2.170, +2.705]
Attn + recency+1.980[+1.700, +2.250]+1.785[+1.520, +2.045]+0.530[+0.330, +0.730]
  • On MiniCPM-o 4.5 DuplexKV + JET is above every policy at these budgets, by 0.530 to 3.145, with a TOR of at least 0.995 for every method.

Table R3.4: Policies on the Full KV fine-tune at M=53M=53 against DuplexKV + JET, Raon-SpeechChat.

PolicyM=53M=53, ΔJ\Delta J95% CIn
AudioKV+0.892[+0.640, +1.151]186
R-KV+0.478[+0.266, +0.690]184
StreamingLLM+2.201[+1.862, +2.541]159
DefensiveKV+0.175[-0.021, +0.378]143
Ada-KV+0.651[+0.417, +0.896]192
Attn + recency+0.673[+0.442, +0.909]165
  • On Raon-SpeechChat DuplexKV + JET is above every policy, by 0.175 to 2.201. The smallest gap, DefensiveKV at M=53M=53, comes with a TOR of 0.720 against 0.985 for DuplexKV + JET. Appendix L gives both tables.
  • Since these policies run on a backbone adapted with the same data and recipe, DuplexKV + JET keeps a margin beyond the adaptation of the backbone.
  • On the jointly trained backbone itself, DuplexKV + JET obtains a higher judge score than all 15 baselines run on this backbone, at both budgets. At M=32M=32 the closest are DefensiveKV on Raon-SpeechChat, by +0.389, and AudioKV on MiniCPM-o 4.5, by +0.095 (Table R1.5 in our response to Reviewer nEe8).
  • With the backbone and the other two channels fixed, removing the gate channel lowers the judge score of DuplexKV in all eight combinations of backbone, evaluation set and budget (Figure 17 of the paper).
  • Against fine-tunes that replace our gate objective with that of TrimKV or LKV under the same joint training, DuplexKV before JET obtains a significantly higher judge score than the TrimKV objective at M=32M=32 on MiniCPM-o 4.5 and at both budgets on Raon-SpeechChat, and on Raon-SpeechChat DuplexKV takes the turn more often than both objectives, by 0.250 to 0.765 in TOR (Table R1.8 in our response to Reviewer nEe8).
AuthorsOpenReview ↗

Response to Reviewer QR8n (Cont.)

[W2/Q3] Effect of joint fine-tuning beyond interruption responses

  • To look beyond interruption responses, we evaluate three further FDB v1.0 tasks on MiniCPM-o 4.5, pause handling on two sets (synthetic, n = 137, and CANDOR, n = 216), turn-taking on CANDOR (n = 119) and backchannel (n = 55), for the released weights, both fine-tunes with the Full KV cache and DuplexKV + JET at M=64M=64 and 32. In a pause the model should keep listening (lower TOR is better), at the end of a turn the model should take the turn (higher TOR is better), and its backchannels should follow the human timing (lower JSD is better). JSD is the Jensen-Shannon divergence of the model's backchannel timing distribution against the human ground truth of the benchmark, and frequency is backchannels per second.

Table R3.5: MiniCPM-o 4.5 on three further FDB v1.0 tasks.

SettingSynthetic pause TOR ↓\downarrowCANDOR pause TOR ↓\downarrowCANDOR turn-taking TOR ↑\uparrowBackchannel JSD ↓\downarrowBackchannel frequency
Full KV, released weights0.1460.3520.8741.00000.0000
Full KV, Full KV fine-tune0.0510.1440.9830.77390.0637
Full KV, jointly trained backbone0.0440.0880.9660.76330.0388
DuplexKV + JET, M=64M=640.0440.0971.0000.76320.0388
DuplexKV + JET, M=32M=320.0150.0881.0000.76360.0366
  • Every fine-tuned setting, with the Full KV cache or with DuplexKV + JET, takes the turn less often in pauses, by 0.095 to 0.264 in TOR, and more often at the end of a turn, by 0.092 to 0.126, than the released weights. The released weights produce no backchannel in the 55 episodes (frequency 0.0000, JSD 1.0000), and every fine-tuned setting backchannels at 0.0366 to 0.0637 per second with a JSD of 0.7632 to 0.7739.
  • Against the Full KV cache on the same jointly trained backbone, the backchannel JSD of DuplexKV + JET differs by at most 0.0003 at M=32M=32 and 64 (n = 55). In Table R3.5 fine-tuning on our data improves pause handling, turn-taking and backchannel, and DuplexKV + JET keeps these improvements at M=32M=32 and 64.
  • Please refer to Appendix M for the full results and confidence intervals.
Reviewer_q1YROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes KV cache eviction in full duplex spoken dialogue models. DuplexKV trains a light-weight L0-gate jointly with the backbone through a rollout-free two-pass objective. It combines the gate score with accumulated attention and recency at inference. To overcome the poor correlation between next token loss and generated reply quality under eviction, DuplexKV further introduce JET which fine-tunes the gate on a LLM judge’s verdicts about the replies generated under eviction.

Critical strengths
  • Training and tuning L0-gate with the rollout-free two-pass training and JET is a computationally efficient solution for training under hard cache budgets.
  • Every design choice in the paper is grounded in quantitative diagnostics which leads to clear logical flow rather than ad-hoc heuristics.
Critical weaknesses
  1. DuplexKV performs eviction independently per layer and KV head by combining a shared gate score with head-specific accumulated attention. Consequently, when a token satisfies the boundary condition, its partial retention across heads does not necessarily indicate gate uncertainty or suboptimal eviction. It may reflect natural head specialization where certain heads need to attend to a token while others can safely discard it to save their budget. However, JET assigns a single binary label to each boundary token and updates the globally shared gate accordingly. Forcing the shared gate to push boundary tokens toward entire retention or entire eviction across all heads may suppress head diversity and reduce the effective capacity of per-head KV cache under a budget.

  2. Even though Section 3.2 shows that accumulated attention favors old tokens (the opposite of recency), the inference score still use raw accumulated attention. Then is the current score also biased toward selecting old tokens?

  3. How much is the latency per step on GPU? Can per-step percentile ranking across all heads runs within real-time frame budget?

Questions for authors

Please see section Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer q1YR

We thank the reviewer for noting that rollout-free two-pass training with Judged Eviction Tuning (JET) trains the gate efficiently under a hard cache budget. We answer each of the three questions below with a direct measurement: head diversity with and without JET, the age of the kept tokens, and the step time against the frame budget.

[W1] Does one binary label per boundary token reduce head diversity?

  • If a single verdict per boundary token pushed all heads to keep or drop that token together, head diversity and the capacity of each head would fall, so we measured head diversity directly. At the end of each of the 200 Full-Duplex-Bench v1.0 (FDB v1.0) user interruption episodes we count per layer the tokens held by at least one of the 8 key-value (KV) heads, the overlap between the tokens that two heads hold, and the share of tokens held by all 8 heads, with and without JET on the same episodes. If the shared label suppressed diversity, the first would fall and the other two would rise.
  • In Table R2.1 head overlap is the mean Jaccard of the token sets held by two KV heads of a layer, over all pairs. Δ=XDuplexKV + JET−XDuplexKV\Delta = X_{\text{DuplexKV + JET}} - X_{\text{DuplexKV}} for each measure XX, paired over the 200 episodes, with a 95% bootstrap CI.

Table R2.1: Head diversity per layer at the end of the episode, DuplexKV and DuplexKV + JET.

MeasureMMDuplexKV / + JETΔ\Delta95% CI
MiniCPM-o 4.5
Tokens held by ≥1 head3282.6 / 82.9+0.3[-0.0, +0.7]
64152.0 / 152.8+0.8[+0.4, +1.2]
Head overlap320.403 / 0.402-0.002[-0.004, +0.001]
640.430 / 0.427-0.003[-0.005, -0.001]
Share held by all 8 heads320.110 / 0.109-0.000[-0.002, +0.001]
640.116 / 0.114-0.002[-0.003, -0.001]
Raon-SpeechChat
Tokens held by ≥1 head3268.1 / 72.8+4.7[+4.1, +5.4]
64133.5 / 139.5+6.0[+5.2, +6.8]
Head overlap320.512 / 0.502-0.010[-0.016, -0.004]
640.518 / 0.503-0.015[-0.019, -0.011]
Share held by all 8 heads320.184 / 0.189+0.005[-0.000, +0.011]
640.185 / 0.186+0.001[-0.002, +0.005]
  • With JET, the tokens held by at least one head increase by 0.3 to 0.8 per layer on MiniCPM-o 4.5 and by 4.7 to 6.0 on Raon-SpeechChat, the overlap between two heads decreases by 0.002 to 0.015, and the share held by all 8 heads changes by 0.005 or less, so head diversity is preserved on both backbones.
  • JET judges the tokens whose retention differs across heads because a change of the shared gate term can change their retention (Section 4.5), and trains only that term. In the score wg Π(log⁡α)+wa Π(A)+wd Π(d)w_g\,\Pi(\log\alpha) + w_a\,\Pi(A) + w_d\,\Pi(d), each head takes the percentile rank Π\Pi within its own candidates and has its own attention channel AA, so heads still choose differently after JET moves the gate term, and the partial retention that reflects head specialization remains.
AuthorsOpenReview ↗

Response to Reviewer q1YR (Cont.)

[W2] Is the inference score biased toward old tokens?

  • Section 3.2 shows that accumulated attention favors old tokens, so we measured which tokens the inference score keeps, since the score contains the accumulator. On 50 FDB v1.0 episodes per setting we read, every 5 s, the tokens that each KV head keeps, and report the share of kept tokens among the newest MM tokens and their median age. StreamingLLM keeps exactly the newest MM tokens and serves as the reference. A score biased toward old tokens would keep few of its tokens among the newest MM and hold them much longer than StreamingLLM.
  • In Table R2.2 shares and ages are pooled over layers and KV heads. StreamingLLM keeps the newest MM tokens by construction, so its share is 1. On MiniCPM-o 4.5 ages resolve to 1 s chunks.

Table R2.2: Share of kept tokens among the newest MM and their median age.

MMShare among the newest MM, DuplexKV / + JETMedian kept age (s), DuplexKV / + JETMedian kept age (s), StreamingLLM
MiniCPM-o 4.5
320.560 / 0.5602.00 / 2.001.32
640.625 / 0.6273.04 / 3.022.14
Raon-SpeechChat
320.633 / 0.6600.64 / 0.590.53
640.652 / 0.6871.35 / 1.201.13
  • DuplexKV and DuplexKV + JET keep 56 to 69% of their tokens among the newest MM, and their median kept age exceeds that of StreamingLLM by 0.68 to 0.90 s on MiniCPM-o 4.5 and by 0.06 to 0.22 s on Raon-SpeechChat. JET leaves the share within 0.002 of DuplexKV on MiniCPM-o 4.5 and raises the share by 0.03 to 0.04 on Raon-SpeechChat.
  • The inference score adds the recency channel Π(d)\Pi(d) to the accumulator with wd≥waw_d \ge w_a on both backbones (Appendix G), and Π(d)\Pi(d) favors the newest tokens while the accumulator favors old ones (Section 3.2), so DuplexKV keeps most of its tokens among the newest.
  • To test whether the older tokens kept through the attention channel matter for the reply, we set wa=0w_a = 0 with everything else fixed on the jointly trained backbones (Table R2.3).
  • In Table R2.3 the interruption Judge and the takeover rate (TOR) are on the 200 FDB v1.0 episodes. ΔJ=Jwa=0−JDuplexKV\Delta J = J_{w_a=0} - J_{\text{DuplexKV}} is paired over the n episodes that both settings rate, with a 95% bootstrap CI. The share among the newest MM is read as in Table R2.2 on the first 50 episodes.

Table R2.3: DuplexKV without the attention channel (wa=0w_a=0) on the jointly trained backbone.

MMDuplexKV, Judge / TORwa=0w_a=0, Judge / TORΔJ\Delta J95% CIn
MiniCPM-o 4.5
324.370 / 1.0002.050 / 1.000-2.320[-2.560, -2.075]200
644.395 / 1.0004.045 / 1.000-0.350[-0.505, -0.190]200
Raon-SpeechChat
323.531 / 0.9701.533 / 0.985-2.016[-2.328, -1.688]192
644.071 / 0.9803.611 / 0.965-0.444[-0.683, -0.196]189
  • Without the attention channel the kept set moves onto the newest tokens (share among the newest MM 0.896 to 0.953, against 0.560 to 0.652 for DuplexKV) and the interruption Judge falls at every budget, most at M=32M=32 (by 2.320 on MiniCPM-o 4.5 and 2.016 on Raon-SpeechChat). The older tokens that the attention channel keeps matter for the reply. Removing the gate channel instead (wg=0w_g=0) also lowers the judge score in all eight combinations of backbone, evaluation set and MM (Section 5.2), so DuplexKV relies on all three channels.

[W3] Latency per step against the frame budget

  • To give the step latency on the GPU and to check the per-step ranking against the frame budget, we time each part of the step at M=64M=64 with CUDA events over five 20-minute sessions per backbone on H200 GPUs, in which the model takes its own turns, for DuplexKV and for H2O, which runs in the same inference code with accumulated attention as the eviction rule (Table G.2 of our general comment).

Table R2.4: Step time at M=64M=64 on H200 GPUs, p95 (ms).

Backbone (unit)Ranking, DuplexKV / H2OFull step, DuplexKV / H2OBudget
MiniCPM-o 4.5 (1 s chunk)5.17 / 3.1897.42 / 95.721000
Raon-SpeechChat (80 ms frame)0.38 / 0.2310.27 / 9.9780
  • Ranking across all heads takes about 0.5% of the chunk and of the frame, and the full step of DuplexKV takes 10% of the chunk and 13% of the frame. The step has one shape for the whole session because the cache width is fixed at the prefix plus MM.
  • Over the 20-minute sessions, DuplexKV, H2O and StreamingLLM stay within the frame and the chunk and grow GPU memory by at most 42 MB, whereas with the Full KV cache GPU memory grows by 10.30 GB on Raon-SpeechChat and 3.51 GB on MiniCPM-o 4.5 and the latency leaves the frame at minute 10 and the chunk from minute 13 (Table G.1 of our general comment).

We hope these results answer the reviewer's questions, and we are glad to clarify anything that remains.

Thank you for your answers. I am keeping my current score, which I feel is reasonable.

We thank the reviewer for the follow-up and for taking the time to review our rebuttal. We appreciate that the additional experiments and analyses were taken into account, and we remain glad to clarify anything further if needed.

Reviewer_nEe8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV cache eviction in full-duplex speech models. Because audio frames from both speakers continually enter the model, the cache grows with elapsed session time, including during silence. On MiniCPM-o 4.5 and Raon-SpeechChat, the authors analyze the roles of different speakers, modalities, and control tokens. They show that high attention does not necessarily imply high retention value, and that token-level prediction loss on a given reply does not fully reflect the quality of a reply freely generated after eviction.

DuplexKV jointly trains a lightweight gate and the language backbone, then combines the gate with attention and recency for eviction under a fixed budget at inference. Judged Eviction Tuning (JET) further trains the gate using a teacher model's judgments of actual generated replies and candidate words. Experiments cover two backbones, multiple cache budgets, user interruption tasks, and recorded long histories, with controls on the same backbone, gate ablations, label-stability tests, and comparisons between masked computation and actual cache deletion.

Critical strengths
  1. The problem matters and the analysis addresses the specific structure of full-duplex speech. The authors distinguish retention value by speaker, modality, and control-token type, and separately examine turn-taking and reply content. The importance of a small number of control tokens for taking the turn, together with the inability of token-level loss to fully track declining reply quality, provides useful evidence for designing eviction policies for full-duplex speech.
  2. The method has substantial experimental support and key contributions have been isolated. The experiments cover two backbones and multiple cache policies. Table 7 compares several baselines on the same jointly trained backbone, and Figure 17 removes the gate while keeping the backbone and other score channels fixed, supporting an independent contribution from the learned selection strategy. On Raon-SpeechChat, JET increases the reply score from 3.53 to 3.90 at M=32 and from 4.07 to 4.27 at M=64 while maintaining a high takeover rate. Label-stability tests, a second judge, and numerical consistency checks in the appendix further strengthen the evidence.
Critical weaknesses
  1. The meaning of JET labels and the independent value of conditioning on the reply need more direct validation. The teacher sees the conversation, the actual generated reply, and candidate words, but does not see paired replies obtained by retaining or deleting the same candidate while holding the rest of the cache fixed. The labels can therefore provide useful semantic weak supervision, but cannot yet be interpreted directly as the marginal contribution of an individual token to the reply. Agreement under repeated judging and candidate shuffling establishes some stability, but does not replace validity testing. A control with the same labeling budget that judges only the conversation without the generated reply, or paired retention and deletion tests on a small candidate sample, would clarify what reply conditioning contributes.
  2. Attribution and statistical precision of some advantages need clarification. Table 7 and the gate ablation already control for backbone differences, which is important positive evidence. However, on MiniCPM, the JET advantage over the strongest baseline on the same backbone is only 0.10 at M=32 and 0.04 at M=64, warranting paired confidence intervals for these comparisons. Learned baselines also freeze the backbone during training, whereas DuplexKV updates it jointly, leaving unequal training freedom. I recommend emphasizing the same-backbone results and prioritizing a strong baseline under matched training conditions, or narrowing attribution of the overall gains in the main table. This concern affects the precision of the claimed advantage and its attribution; it does not negate the clearer gains on Raon or the existing independent evidence for the gate.
Questions for authors
  1. Training uses representations from a full-cache forward pass and gate-based ranking, whereas inference operates on states produced by continual eviction and ranks jointly using attention and recency. How consistent are the gate rankings or retained sets between these states? Appendix I.6 verifies computational consistency when the same mask is given, but does not establish consistency of the actual selected sets.
  2. Appendix I.2 derives KV memory usage from the model architecture. In deployment, how much overhead comes from gate computation, sorting, and cache movement, and what is peak cache usage including the prefix and current-step temporary states? These details affect the assessment of the method's value for real-time deployment.
5 comments
AuthorsOpenReview ↗

Response to Reviewer nEe8

We thank the reviewer for recognizing that the problem matters and that our analysis of the cache gives useful evidence for designing eviction policies for full-duplex speech. Following the reviewer's suggestions, we run the two proposed tests on the meaning of the Judged Eviction Tuning (JET) labels, and add measurements on attribution, on the gate at inference and on overhead.

  • As in Table 1, Judge / TOR gives the interruption Judge and the takeover rate (TOR). Unless stated otherwise, each ΔJ\Delta J is paired over the n episodes in which both settings take the turn, with a 95% bootstrap CI, and the table caption gives its direction.

[W1] Meaning of JET labels and value of reply conditioning

  • We ran both tests that the reviewer suggested. The reply-free control measures what conditioning on the generated reply adds to the labels. The paired retention and deletion test measures whether the verdict on a set of candidates agrees with the effect of those tokens on the reply. JET labels come from Qwen3.8-27B (Section 5.3), and every judge score in this response comes from a different judge, gpt-5.6-luna (Appendix E).
  • Reply-free control. The same judge labels the same word items of the same documents again with the generated reply removed, so the two label sets differ only in the verdicts (Table R1.1), and we retrain JET on the reply-free labels.

Table R1.1: Agreement of the reply-conditioned and reply-free labels on the same items.

BackboneItemsKeep share, reply-conditioned / reply-freeKeep overlap (Dice)Dice by chanceCohen's κ\kappa
Raon-SpeechChat159,89020.6% / 45.1%0.5240.2830.336
MiniCPM-o 4.5128,80825.1% / 49.2%0.5810.3320.372
  • Without the reply, the judge marks about twice as many items keep, and 83% (Raon-SpeechChat) and 86% (MiniCPM-o 4.5) of the reply-conditioned keep items are among them. The overlap is above chance but the two label sets differ (κ\kappa 0.336 and 0.372), so seeing the reply makes the judge keep fewer and partly different items. Table R1.2 tests whether this difference changes the trained gate.

Table R1.2: Reply-conditioned against reply-free JET labels, ΔJ\Delta J = reply-conditioned minus reply-free.

MMReply-conditioned, Judge / TORReply-free, Judge / TORΔJ\Delta J95% CIn
Raon-SpeechChat
323.904 / 0.9853.647 / 0.950+0.266[+0.032, +0.511]188
644.271 / 0.9954.077 / 0.975+0.191[+0.052, +0.335]194
MiniCPM-o 4.5
324.455 / 1.0004.290 / 1.000+0.165[+0.050, +0.285]200
644.440 / 1.0004.320 / 1.000+0.120[+0.015, +0.225]200
  • JET trained on the reply-conditioned labels obtains the higher interruption Judge at both budgets on both backbones, so conditioning the judge on the generated reply improves the trained gate.
  • Paired retention and deletion test. On up to 30 judged turns per backbone from the dialogues in which the JET verdicts were collected, we take a set of n candidate items that received the same verdict (n = 1, 2, 4). From the same point we generate the reply twice, once with the tokens of the set in the cache and once without them. ΔJ=Jwith−Jwithout\Delta J = J_{\text{with}} - J_{\text{without}} is the contribution of the set, as in Appendix K, averaged over the sets of a turn and over turns, with a 95% bootstrap CI over turns. Following the verdict means keeping the set for a keep verdict and removing the set for a discard verdict, so a keep verdict predicts a positive ΔJ\Delta J and a discard verdict predicts ΔJ=0\Delta J = 0.
  • To control every other condition, we first run the test with the Full KV cache, where the two generations differ only in the tokens of the set (Table R1.3).

Table R1.3: Contribution ΔJ\Delta J of a set of n candidates, Full KV cache.

nKeep verdicts, ΔJ\Delta J95% CIDiscard verdicts, ΔJ\Delta J95% CI
Raon-SpeechChat
1+0.068[-0.250, +0.361]+0.014[-0.342, +0.314]
2+0.131[-0.209, +0.428]-0.010[-0.420, +0.347]
4+0.251[-0.124, +0.605]+0.009[-0.369, +0.317]
MiniCPM-o 4.5
1+0.031[-0.338, +0.382]+0.025[-0.360, +0.400]
2+0.154[-0.241, +0.536]+0.019[-0.367, +0.378]
4+0.181[-0.264, +0.635]+0.026[-0.420, +0.511]
  • With the Full KV cache, the gain of following a keep verdict grows with n, to +0.251 on Raon-SpeechChat and +0.181 on MiniCPM-o 4.5 at n = 4. Following a discard verdict changes the judge score by at most 0.026.
AuthorsOpenReview ↗

Response to Reviewer nEe8 (Cont.)

  • JET trains the gate to select tokens within a budget, so we repeat the test at M=32M=32, the budget at which the verdicts were collected. To keep the rest of the cache unchanged, the generation without the set follows the record of which tokens each KV head held at every step in the generation with the set, and the slots of the set stay empty (Table R1.4).

Table R1.4: Contribution ΔJ\Delta J of a set of n candidates at M=32M=32.

nKeep verdicts, ΔJ\Delta J95% CIDiscard verdicts, ΔJ\Delta J95% CI
Raon-SpeechChat
1+0.098[-0.001, +0.202]-0.085[-0.246, +0.080]
2+0.189[+0.019, +0.359]+0.009[-0.259, +0.271]
4+0.415[+0.155, +0.699]+0.043[-0.210, +0.334]
MiniCPM-o 4.5
1+0.065[-0.078, +0.207]-0.044[-0.203, +0.123]
2+0.225[-0.007, +0.474]+0.039[-0.194, +0.283]
4+0.331[+0.091, +0.578]+0.031[-0.242, +0.297]
  • At M=32M=32 the gain of following a keep verdict also grows with n, to +0.415 on Raon-SpeechChat and +0.331 on MiniCPM-o 4.5 at n = 4, with intervals above 0. For discard verdicts the two replies differ by at most 0.085 in either direction, the result a discard verdict predicts. Appendix K (Table 12) gives both settings.
  • Together, the two tests show that the JET labels gain from seeing the generated reply and that keep verdicts match the contribution of candidate sets to the judged reply, more clearly as the set grows. This indicates that JET trains the gate to keep the tokens that the judged reply depends on.

[W2] Attribution and statistical precision of the advantage

  • To address both points directly, we first give paired intervals for the margins of 0.10 and 0.04 in Table 7 and for the closest of the 15 baselines that we ran on the jointly trained backbone, which our two-pass training fine-tunes together with the gate (Section 4.3), and extend this comparison to budgets below M=32M=32. Since the learned baselines freeze the backbone while DuplexKV updates the backbone, we then compare under matched training, with existing policies on a backbone fine-tuned with our recipe and the gate removed, and with the gate objectives of TrimKV and LKV under our joint training.
  • The reviewer's 0.10 is AudioKV at M=32M=32, the closest baseline in Table R1.5, and 0.04 is R-KV at M=64M=64, with a paired difference of +0.040 that is not significant (n = 200). Table R1.5 takes, among the 15 baselines that we ran on the jointly trained backbone, the one with the smallest point difference.

Table R1.5: Closest baseline on the jointly trained backbone, ΔJ\Delta J = DuplexKV + JET minus the baseline.

MMClosest baselineΔJ\Delta J95% CIn
MiniCPM-o 4.5
32AudioKV+0.095[-0.030, +0.230]200
64DefensiveKV+0.020[-0.090, +0.130]200
Raon-SpeechChat
32DefensiveKV+0.389[+0.144, +0.639]180
64Ada-KV+0.166[+0.015, +0.317]199
  • On MiniCPM-o 4.5 the differences to the closest baselines are not significant at either budget. On Raon-SpeechChat DuplexKV + JET obtains a significantly higher judge score than all 15 baselines at M=32M=32 and than 14 of them at M=64M=64, where the one exception is DefensiveKV, at +0.190 without a significant difference (n = 184).
  • Table R1.6 adds budgets that hold about 1.02 s (M=16M=16) and 1.53 s (M=24M=24) of the stream on MiniCPM-o 4.5, at 15.7 tokens per second (Appendix I.1), on the jointly trained backbone. The methods are the four strongest baselines at M=32M=32 on MiniCPM-o 4.5 (AudioKV, DefensiveKV, CAKE, R-KV) and StreamingLLM, and Table R1.6 gives, at each budget, the closest of these baselines.

Table R1.6: Closest baseline at budgets below M=32M=32, ΔJ\Delta J = DuplexKV + JET minus the baseline.

MMDuplexKV + JET, Judge / TORClosest baselineΔJ\Delta J95% CIn
MiniCPM-o 4.5
163.250 / 1.000AudioKV+1.335[+1.065, +1.600]200
193.935 / 1.000AudioKV+0.830[+0.580, +1.075]200
244.240 / 1.000AudioKV+0.580[+0.365, +0.795]200
  • As the budget tightens, the margin of DuplexKV + JET over the closest baseline grows to +0.580 at M=24M=24 and +1.335 at M=16M=16. At M=24M=24 DuplexKV + JET obtains 4.240, close to the 4.330 of the Full KV cache on the jointly trained backbone in Table 7, against 3.660 for AudioKV. Every method takes the turn with a TOR of at least 0.995.
AuthorsOpenReview ↗

Response to Reviewer nEe8 (Cont.)

  • To separate the effect of adapting the backbone from that of the gate, we train a Full KV fine-tune with the recipe of DuplexKV and the gate removed (same data, steps, schedule, learning rates and seed).
  • Table R1.7 runs the policies on the Full KV fine-tune and compares them with DuplexKV + JET on the jointly trained backbone. For each backbone and budget the table lists the closest policy and Attn + recency (DuplexKV with the gate channel off).

Table R1.7: Closest policies on the Full KV fine-tune, ΔJ\Delta J = DuplexKV + JET minus the policy.

MMPolicyΔJ\Delta J95% CInTOR
MiniCPM-o 4.5
32AudioKV+0.260[+0.145, +0.380]2001.000
32Attn + recency+0.260[+0.125, +0.395]2001.000
64AudioKV+0.075[-0.045, +0.205]2001.000
64Attn + recency+0.095[-0.035, +0.220]2001.000
Raon-SpeechChat
32DefensiveKV+0.523[+0.196, +0.879]1070.540
32Attn + recency+1.951[+1.622, +2.294]1430.720
64SnapKV+0.288[+0.094, +0.482]1910.960
64Attn + recency+0.739[+0.497, +0.988]1610.810
  • Under matched training, DuplexKV + JET obtains a higher judge score than every policy on the Full KV fine-tune, significantly so against every policy at M=32M=32 and against three of the eight at M=64M=64 on MiniCPM-o 4.5, and against every policy at both budgets on Raon-SpeechChat. Tables R3.1 to R3.4 of our response to Reviewer QR8n and Figure 22 of the revised paper give the full comparison.
  • A second matched-training check keeps our joint training (same data, steps, schedule, learning rates and seed) and replaces our gate training method, the L0L_0-Gate objective with a top-MM straight-through mask, by that of TrimKV or of LKV (Zhou et al., 2026), each with its own gate structure and mask rule. Unlike the TrimKV baseline of Table 1, which trains its gate on the frozen backbone, these fine-tunes change only the gate training method. All three objectives use compression weight 0.01 on Raon-SpeechChat and 0.05 on MiniCPM-o 4.5, the value each backbone's DuplexKV uses, and the same inference settings.

Table R1.8: Gate objectives under our joint training, ΔJ\Delta J = DuplexKV minus the objective.

MMObjectiveJudge / TORΔJ\Delta J95% CIn
MiniCPM-o 4.5
32TrimKV objective4.080 / 1.000+0.290[+0.135, +0.450]200
32LKV objective4.290 / 1.000+0.080[-0.060, +0.225]200
32DuplexKV (ours)4.370 / 1.000-
64TrimKV objective4.320 / 1.000+0.075[-0.065, +0.220]200
64LKV objective4.280 / 1.000+0.115[-0.035, +0.275]200
64DuplexKV (ours)4.395 / 1.000-
Raon-SpeechChat
32TrimKV objective1.780 / 0.205+1.425[+0.650, +2.150]40
32LKV objective3.324 / 0.355+0.042[-0.409, +0.507]71
32DuplexKV (ours)3.531 / 0.970-
64TrimKV objective2.469 / 0.405+1.488[+1.075, +1.913]80
64LKV objective3.842 / 0.730+0.238[-0.000, +0.469]143
64DuplexKV (ours)4.071 / 0.980-
  • On MiniCPM-o 4.5 every fine-tune has TOR 1.000 and answers every interruption, and DuplexKV obtains a significantly higher judge score than the TrimKV objective at M=32M=32. On Raon-SpeechChat, with the compression weight matched, DuplexKV answers more interruptions than the TrimKV and LKV objectives at both budgets (TOR higher by 0.250 to 0.765), and also obtains a higher judge score than the TrimKV objective, which answers 41 and 81 of the 200 episodes against 194 and 196.
  • In the revised paper, Section 5.2 states that the margin on the jointly trained backbone is clearest on Raon-SpeechChat and reports the policies on the Full KV fine-tune (Appendix L, Figure 22).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful reading and for questions that helped us strengthen the paper. Here we summarize our answers to the shared points, including the real-time serving cost, and give the details in the individual responses.

Summary

  • Gate or backbone adaptation (QR8n W1/Q1, nEe8 W2). To separate the effect of adapting the backbone from that of the gate, we fine-tuned each backbone with our recipe but without the gate (the Full KV fine-tune) and applied existing eviction policies to this fine-tune. At each evaluated budget from M=16M=16 to M=64M=64, DuplexKV + JET obtains a higher judge score than every such policy (QR8n Tables R3.1 to R3.4, Figure 22 of the revised paper). On the jointly trained backbone, DuplexKV + JET also obtains a higher judge score than the 15 baselines run on this backbone, most clearly on Raon-SpeechChat (nEe8 Tables R1.5 and R1.6).
  • What the JET labels carry (nEe8 W1, q1YR W1). Training JET on labels that the judge gives with the generated reply yields a higher judge score than training JET on labels that the same judge gives without the reply, at both budgets on both backbones (+0.120 to +0.266). In a paired test on sets of four candidates at M=32M=32, following a keep verdict rather than reversing the verdict raises the judge score by 0.415 on Raon-SpeechChat and 0.331 on MiniCPM-o 4.5, whereas following or reversing a discard verdict changes the judge score by at most 0.043 (nEe8 Tables R1.1 to R1.4). JET preserves head diversity: the number of tokens held by at least one head increases and the overlap between heads decreases (q1YR Table R2.1).
  • Real-time serving cost (QR8n Q4, nEe8 Q2, q1YR W3). On an H200 GPU, the Full KV cache, run with the released inference code, does not sustain real-time inference for a single 20-minute session on either backbone. DuplexKV at M=64M=64 sustains one Raon-SpeechChat session and four MiniCPM-o 4.5 sessions per GPU within the frame and chunk deadlines, with a KV cache of 11.1 and 18.8 MiB per session (Tables G.1 and G.2 below, Table R1.10 in our response to Reviewer nEe8, Appendix J of the revised paper).

Real-time serving cost

  • On H200 GPUs we ran five 20-minute sessions per backbone, one per GPU, in which the model decides its own turns, with the Full KV cache on the released inference code of each model and H2O, StreamingLLM and DuplexKV at M=64M=64.
  • Table G.1 gives the end-to-end latency p50 / p95 in minutes 2, 10 and 20, per 80 ms frame on Raon-SpeechChat and per speaking chunk of 1 s on MiniCPM-o 4.5, median over the sessions with speaking chunks in that minute, and the growth of GPU memory from minute 1 to minute 20, median over the sessions.

Table G.1: Latency and GPU memory over five 20-minute sessions per backbone on H200 GPUs.

MethodMinute 2Minute 10Minute 20Memory growth by minute 20
Raon-SpeechChat
Full KV38 / 40 ms2.23 / 4.79 s241 / 261 s10.30 GB
StreamingLLM32 / 32 ms34 / 35 ms40 / 40 ms0 MB
H2O32 / 32 ms35 / 35 ms39 / 40 ms0 MB
DuplexKV33 / 33 ms35 / 36 ms40 / 40 ms0 MB
MiniCPM-o 4.5
Full KV724 / 806 ms730 / 796 ms817 ms / 1.26 s3.51 GB
StreamingLLM278 / 322 ms298 / 333 ms288 / 317 ms42 MB
H2O283 / 323 ms306 / 348 ms291 / 354 ms42 MB
DuplexKV292 / 332 ms294 / 342 ms298 / 324 ms21 MB
  • With the Full KV cache, GPU memory grows by 10.30 GB on Raon-SpeechChat and 3.51 GB on MiniCPM-o 4.5 by minute 20. The median latency of Raon-SpeechChat exceeds the 80 ms frame from minute 10, and on MiniCPM-o 4.5 the p95 latency of speaking chunks exceeds the 1 s chunk in every minute from minute 13 (Figures 18 and 19 of the revised paper). On MiniCPM-o 4.5 the latency of Full KV grows from 724 to 817 ms (p50) and from 806 ms to 1.26 s (p95) per speaking chunk between minutes 2 and 20, while that of DuplexKV changes by at most 8 ms. With StreamingLLM, H2O and DuplexKV at M=64M=64, GPU memory grows by at most 42 MB and the latency remains within the frame and the chunk over the 20 minutes. The KV cache of DuplexKV remains at 18.8 MiB on MiniCPM-o 4.5 and 11.1 MiB on Raon-SpeechChat.
AuthorsOpenReview ↗

General Response to All Reviewers (Cont.)

  • We also ran concurrent sessions on one H200 GPU, each in its own process. With the Full KV cache, a single session per GPU already fails to remain real-time on either backbone, whereas DuplexKV at M=64M=64 keeps one Raon-SpeechChat session and four MiniCPM-o 4.5 sessions per GPU within the frame and chunk deadlines throughout the session (Figure 20 of the revised paper).
  • Table G.2 reports the time of each part of the step at M=64M=64, measured with CUDA events over five 20-minute sessions per backbone, p50 / p95 (ms) per 1 s chunk on MiniCPM-o 4.5 and per 80 ms frame on Raon-SpeechChat. Ranking scores the candidate tokens of every KV head and selects the MM that stay. H2O runs in the same inference code as DuplexKV and differs only in the eviction rule, which ranks tokens by accumulated attention. StreamingLLM keeps the newest MM tokens and ranks nothing.

Table G.2: Step cost at M=64M=64 in 20-minute sessions on H200 GPUs, p50 / p95 (ms).

MethodForwardGateRankingCache movementFull step
Raon-SpeechChat
StreamingLLM8.59 / 8.70nonenone0.64 / 0.649.27 / 9.38
H2O8.91 / 9.08none0.23 / 0.230.63 / 0.639.75 / 9.97
DuplexKV8.92 / 9.010.27 / 0.290.37 / 0.380.63 / 0.6310.18 / 10.27
MiniCPM-o 4.5
StreamingLLM49.26 / 81.30nonenone3.87 / 6.4553.60 / 87.79
H2O52.03 / 86.13none1.90 / 3.183.79 / 6.3257.42 / 95.72
DuplexKV51.93 / 84.340.18 / 0.303.09 / 5.173.79 / 6.3158.35 / 97.42
  • The full step of DuplexKV is comparable to that of H2O: 97.42 and 95.72 ms per chunk and 10.27 and 9.97 ms per frame at p95. Against StreamingLLM, which ranks nothing, the full step of DuplexKV is longer by 9.6 ms per chunk and 0.9 ms per frame at p95, about 1% of the chunk and of the frame. Ranking takes 5.17 ms per chunk and 0.38 ms per frame for DuplexKV against 3.18 and 0.23 ms for H2O, the cache movement is the same in both within 0.01 ms, and the gate adds at most 0.30 ms. As in H2O, most of the step is the language model forward. The shape of the step stays fixed over the session because the cache width is fixed at the prefix plus MM.

Revisions to the paper

  • For Reviewer QR8n, we added the policies on the Full KV fine-tune (Figure 22 and Tables R3.1 to R3.4) as Appendix L, and the pause, turn-taking and backchannel tasks of Full-Duplex-Bench v1.0 (FDB v1.0) on MiniCPM-o 4.5, with backchannel measured by JSD and frequency, as Appendix M.
  • For Reviewer nEe8, we added the paired retention and deletion test (Tables R1.3 and R1.4) as Appendix K, and Table R1.9, the gate on the cache at inference against a Full KV replay, as Appendix N.
  • For Reviewers QR8n, nEe8 and q1YR, we added the latency and memory over 20-minute sessions, the sessions per GPU and the time of a step by component to Appendix J (Figures 18 to 21), with a summary in Section 5.4.

We hope these answers address the shared points, and we are glad to clarify anything that remains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.