- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how KV quantization affects the behavior of coding agents and specifically what kinds of efficiency failures KV quantization causes. The finding is interesting and intuitive: with aggressive KV quantization, agents suffer from tool misuse, repeated attempts, and insufficient debugging. These failures cause trajectory-level inefficiency. This paper then proposes failure-centered diagnosis and perturbations to find sensitive components. Evaluation shows improved task success with lower KV cache budget.
Critical strengths
- The challenges are siginificant and also well-recognized by some other works. Trajectory-level efficiency matters more than the token-level efficiency.
- The anatomy of the failure patterns is informative. The local diagnosis desing like token replay supports the sensitivity analysis.
- Behavioral and performance improvements are validated across benchmarks and models.
Critical weaknesses
- The models tested are a bit old. With recent models like DeepSeek V4, is KV cache quantization still a preferred or necessary approach? It has trained-in sparse KV cache and KV cache eviction mechanism.
- The baselines do not include KV cache quantization/eviction methods built for agentic workloads [1, 2].
- Behavioral annotation needs better validation. Table 1 looks like AI-generated. Specifically, is "Scope Creep" a coined phrase? What is the boundary and how do you define the behavior categories?
[1] Shen H, Wu H, Zhao Y, et al. TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks[J]. arXiv preprint arXiv:2605.17170, 2026. [2] Liu T T, Wong J T H, Xiao C, et al. AgentKV: Phase-Aware KV Eviction for Agentic LLMs[J]. arXiv preprint arXiv:2609.14872, 2026.
Questions for authors
Please see weaknesses.
4 comments
Response to Reviewer gXMB Part2
W3. Behavioral taxonomy and annotation criteria
We appreciate the reviewer's careful reading and the opportunity to clarify the origin and validation of our behavioral taxonomy.
-
Standardized taxonomy and terminology from prior work. The 13 behavioral categories in Table 1 in the manuscript are neither ad-hoc nor AI-generated labels; they are adopted directly from the validated agent-evaluation taxonomy of Wang et al. (2026) [7]. Specifically, Scope Creep is a standard software engineering term formalized in their rubric to denote unauthorized modifications or feature additions that exceed the specific issue scope requested by the user.
-
Rigorous annotation criteria and consistency checks. We strictly follow Wang et al.'s annotation templates and evidence-grounded protocol. As detailed in Appendix B.1 (Table 5 in the manuscript) and Appendix B.2, every assigned label must be anchored to concrete trajectory evidence (such as shell commands, diff edits, or execution feedback), with explicit boundary criteria separating adjacent behaviors (e.g., distinguishing Scope Creep from Insufficient Debugging). To validate reliability, Table 6 in the manuscript compares cross-model annotations between Claude Opus and Gemini.
References
[1] Qwen Team. Qwen3.6-35B-A3B release announcement, April 16, 2026.
[2] vLLM Team. DeepSeek V4 in vLLM: Efficient Long-context Attention. 2026.
[3] NVIDIA. Nemotron 3.5 Super and Lightning technical report. 2026.
[4] Google. Gemma 4 31B model configuration. 2026.
[5] Shen et al. TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks. 2026.
[6] Liu et al. AgentKV: Phase-Aware KV Eviction for Agentic LLMs. 2026.]
[7] Wang et al. A Rubric-Supervised Critic from Sparse Real-World Outcomes. 2026.
Thank you for the detailed responses. Most of my concerns have been addressed.
Thanks for your feedback! We are pleased to hear that our responses have addressed most of your concerns. We will incorporate your valuable suggestions to improve the quality of our paper.We sincerely appreciate your support!
Response to Reviewer gXMB Part1
Thank you for your thoughtful and critical assessment. Many of your comments will help us produce a more readable and self-contained version of the paper. Below, we address each of your specific concerns in turn.
W1. Relevance to recent model architectures
We thank the reviewer for highlighting recent advances in memory-efficient attention architectures. This is an important consideration for the applicability of our work. Architectural compression reduces the number or size of cached representations, while quantization controls their storage precision. The two can therefore be combined, motivating us to study how sensitivity-guided precision allocation can be adapted to these new cache designs.
Empirical evidence on a hybrid architecture (Qwen3.6-35B-A3B). We applied our failure-centered localization analysis to Qwen3.6-35B-A3B [1]. The model interleaves Gated DeltaNet with full attention. Among the ten full-attention layers that maintain token-level KV caches, Table 11 in the manuscript appendix identifies two sensitive layers.
Potential integration with DeepSeek-V4's compressed KV cache. DeepSeek-V4 compresses historical tokens into fewer KV entries through CSA and HCA, and further reduces storage costs by retaining BF16 for the RoPE dimensions while storing the remaining dimensions in FP8 [2]. TQ-Mix could build on this design by applying failure-centered sensitivity localization to the compressed KV representations, identifying layers that benefit from higher precision, and selectively restoring their precision. This would allow us to evaluate whether sensitivity-guided precision allocation improves agent task performance at a matched KV memory budget.
Potential extensions to other cache layouts. Nemotron 3.5 combines Mamba and Transformer layers [3]. TQ-Mix could target the attention layers that maintain KV caches, localizing sensitivity and selectively restoring their precision. Gemma 4 combines local and global attention [4]. For this architecture, we could identify sensitive layers across both types and account for their different cache sizes when allocating precision.
W2. Comparison with TriAxialKV and AgentKV
We sincerely thank the reviewer for pointing out these very recent, relevant agentic baselines. Including them substantially strengthens our empirical evaluation. We evaluated both agent-oriented baselines TriAxialKV [5] and AgentKV [6] under their recommended default configurations on SWE-bench Verified (Table R1).
Table R1. Comparison with agent-oriented KV compression baselines (TriAxialKV and AgentKV) on the SWE-bench Verified instances.
Under the evaluated configurations, TQ-Mix achieves a higher resolve rate (41.6%) than TriAxialKV (40.0%) and AgentKV (39.6%). It also exhibits lower rates of Loop Behavior (20% vs. 36% and 28%) and Improper Tool Use or Setup (43% vs. 51% and 53%), while matching BF16 on Insufficient Debugging (48%). One possible explanation is that token-level quantization or eviction may not fully preserve the information used by failure-sensitive attention heads. Restoring precision in failure-critical layers may help mitigate the resulting attention distortion and repetitive behavior.
These methods target different dimensions and compression budgets: TriAxialKV assigns INT2/INT4 precision across tokens by recency, modality, and semantic role, whereas AgentKV evicts historical KV entries via phase-specific queries. Conversely, TQ-Mix allocates precision across layers using head-wise failure sensitivity. These different design choices suggest potential for combining layer-wise precision allocation with token-wise quantization or retention.