acceptodds
Under review as a conference paper at ICLR 2027

Fewer Bits, More Tokens: Rethinking KV Cache Quantization in Agentic Coding

Abstract

Agentic coding has emerged as a prominent application of LLMs, but its multi-turn interaction trajectories continually expand the KV cache, imposing substantial memory demands during inference. KV cache quantization reduces per-token memory usage, but do these savings translate into more efficient agent trajectories? We find that low-bit quantization disproportionately amplifies three failures: repetitive action loops, improper tool use, and insufficient debugging, while increasing the actions and tokens required per task. To identify which attention heads are most sensitive to quantization around these failures, we measure changes in attention distributions and head outputs under controlled KV cache perturbations. Highly sensitive heads comprise only about 5% of all heads and cluster in a few layers. Guided by this localization, we introduce a simple and effective mixed-precision strategy that restores higher precision in selected layers while retaining low precision elsewhere. On SWE-bench Verified with Qwen3-Coder-30B-A3B-Instruct, increasing average precision from to bits substantially reduces the three behavioral failures and brings task resolution close to BF16. Despite higher per-token precision, observed KV cache usage per case decreases by 12% versus uniform 3-bit quantization. These results shift the focus of agent quantization from storing each token more cheaply to completing tasks more reliably with less memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Quantization, Agentic Coding, Behavioral Degradation Analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gXMBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how KV quantization affects the behavior of coding agents and specifically what kinds of efficiency failures KV quantization causes. The finding is interesting and intuitive: with aggressive KV quantization, agents suffer from tool misuse, repeated attempts, and insufficient debugging. These failures cause trajectory-level inefficiency. This paper then proposes failure-centered diagnosis and perturbations to find sensitive components. Evaluation shows improved task success with lower KV cache budget.

Critical strengths
  1. The challenges are siginificant and also well-recognized by some other works. Trajectory-level efficiency matters more than the token-level efficiency.
  2. The anatomy of the failure patterns is informative. The local diagnosis desing like token replay supports the sensitivity analysis.
  3. Behavioral and performance improvements are validated across benchmarks and models.
Critical weaknesses
  1. The models tested are a bit old. With recent models like DeepSeek V4, is KV cache quantization still a preferred or necessary approach? It has trained-in sparse KV cache and KV cache eviction mechanism.
  2. The baselines do not include KV cache quantization/eviction methods built for agentic workloads [1, 2].
  3. Behavioral annotation needs better validation. Table 1 looks like AI-generated. Specifically, is "Scope Creep" a coined phrase? What is the boundary and how do you define the behavior categories?

[1] Shen H, Wu H, Zhao Y, et al. TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks[J]. arXiv preprint arXiv:2605.17170, 2026. [2] Liu T T, Wong J T H, Xiao C, et al. AgentKV: Phase-Aware KV Eviction for Agentic LLMs[J]. arXiv preprint arXiv:2609.14872, 2026.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer gXMB Part1

Thank you for your thoughtful and critical assessment. Many of your comments will help us produce a more readable and self-contained version of the paper. Below, we address each of your specific concerns in turn.

W1. Relevance to recent model architectures

We thank the reviewer for highlighting recent advances in memory-efficient attention architectures. This is an important consideration for the applicability of our work. Architectural compression reduces the number or size of cached representations, while quantization controls their storage precision. The two can therefore be combined, motivating us to study how sensitivity-guided precision allocation can be adapted to these new cache designs.

  • Empirical evidence on a hybrid architecture (Qwen3.6-35B-A3B). We applied our failure-centered localization analysis to Qwen3.6-35B-A3B [1]. The model interleaves Gated DeltaNet with full attention. Among the ten full-attention layers that maintain token-level KV caches, Table 11 in the manuscript appendix identifies two sensitive layers.

  • Potential integration with DeepSeek-V4's compressed KV cache. DeepSeek-V4 compresses historical tokens into fewer KV entries through CSA and HCA, and further reduces storage costs by retaining BF16 for the RoPE dimensions while storing the remaining dimensions in FP8 [2]. TQ-Mix could build on this design by applying failure-centered sensitivity localization to the compressed KV representations, identifying layers that benefit from higher precision, and selectively restoring their precision. This would allow us to evaluate whether sensitivity-guided precision allocation improves agent task performance at a matched KV memory budget.

  • Potential extensions to other cache layouts. Nemotron 3.5 combines Mamba and Transformer layers [3]. TQ-Mix could target the attention layers that maintain KV caches, localizing sensitivity and selectively restoring their precision. Gemma 4 combines local and global attention [4]. For this architecture, we could identify sensitive layers across both types and account for their different cache sizes when allocating precision.

W2. Comparison with TriAxialKV and AgentKV

We sincerely thank the reviewer for pointing out these very recent, relevant agentic baselines. Including them substantially strengthens our empirical evaluation. We evaluated both agent-oriented baselines TriAxialKV [5] and AgentKV [6] under their recommended default configurations on SWE-bench Verified (Table R1).

Table R1. Comparison with agent-oriented KV compression baselines (TriAxialKV and AgentKV) on the SWE-bench Verified instances.

MetricBF16TriAxialKVAgentKVTQ-Mix
Improper Tool Use or Setup (↓)38%51%53%43%
Loop Behavior (↓)15%36%28%20%
Insufficient Debugging (↓)48%52%54%48%
Resolved Rate (↑)42.2%40.0%39.6%41.6%

Under the evaluated configurations, TQ-Mix achieves a higher resolve rate (41.6%) than TriAxialKV (40.0%) and AgentKV (39.6%). It also exhibits lower rates of Loop Behavior (20% vs. 36% and 28%) and Improper Tool Use or Setup (43% vs. 51% and 53%), while matching BF16 on Insufficient Debugging (48%). One possible explanation is that token-level quantization or eviction may not fully preserve the information used by failure-sensitive attention heads. Restoring precision in failure-critical layers may help mitigate the resulting attention distortion and repetitive behavior.

These methods target different dimensions and compression budgets: TriAxialKV assigns INT2/INT4 precision across tokens by recency, modality, and semantic role, whereas AgentKV evicts historical KV entries via phase-specific queries. Conversely, TQ-Mix allocates precision across layers using head-wise failure sensitivity. These different design choices suggest potential for combining layer-wise precision allocation with token-wise quantization or retention.

AuthorsOpenReview ↗

Response to Reviewer gXMB Part2

W3. Behavioral taxonomy and annotation criteria

We appreciate the reviewer's careful reading and the opportunity to clarify the origin and validation of our behavioral taxonomy.

  • Standardized taxonomy and terminology from prior work. The 13 behavioral categories in Table 1 in the manuscript are neither ad-hoc nor AI-generated labels; they are adopted directly from the validated agent-evaluation taxonomy of Wang et al. (2026) [7]. Specifically, Scope Creep is a standard software engineering term formalized in their rubric to denote unauthorized modifications or feature additions that exceed the specific issue scope requested by the user.

  • Rigorous annotation criteria and consistency checks. We strictly follow Wang et al.'s annotation templates and evidence-grounded protocol. As detailed in Appendix B.1 (Table 5 in the manuscript) and Appendix B.2, every assigned label must be anchored to concrete trajectory evidence (such as shell commands, diff edits, or execution feedback), with explicit boundary criteria separating adjacent behaviors (e.g., distinguishing Scope Creep from Insufficient Debugging). To validate reliability, Table 6 in the manuscript compares cross-model annotations between Claude Opus and Gemini.

References

[1] Qwen Team. Qwen3.6-35B-A3B release announcement, April 16, 2026.

[2] vLLM Team. DeepSeek V4 in vLLM: Efficient Long-context Attention. 2026.

[3] NVIDIA. Nemotron 3.5 Super and Lightning technical report. 2026.

[4] Google. Gemma 4 31B model configuration. 2026.

[5] Shen et al. TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks. 2026.

[6] Liu et al. AgentKV: Phase-Aware KV Eviction for Agentic LLMs. 2026.]

[7] Wang et al. A Rubric-Supervised Critic from Sparse Real-World Outcomes. 2026.

Thank you for the detailed responses. Most of my concerns have been addressed.

Thanks for your feedback! We are pleased to hear that our responses have addressed most of your concerns. We will incorporate your valuable suggestions to improve the quality of our paper.We sincerely appreciate your support!

Reviewer_BLXyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an approach to improve the performance of quantized kv-cache of the model on the agentic coding trajectories. The vanilla quantized kv-cache can lead to behaviors like repetitive action loops, improper tool use, and insufficient debugging. The paper first analyze the window-based generation difference between the bf16 and the quantized version on the KL divergence and the output projection to study the impact of kv-cache. Then the difference is aggregated across position and examples to reduce to layers and heads, which show the most sensitive layers and heads. The paper shows that by increase the quantize bits on these sensitive parts can recover the resolve rates and leads to fewer memory footprint than 3-bit quantization baseline because the later and have more bad behaviors that can consume more tokens.

Critical strengths
  • The paper presents an approach that has strong performance. By increasing the averaged quantization bits from 3.0 to 3.35, the resolve rate is recoverd and has less 12% kv cache usage because some trajectories are finished earilier.
  • Nice ablation studies and the paper show it can generalize across programming languages, model sizes, and architectures.
  • The paper is well-written.
Critical weaknesses
  • The paper does not show another study: conduct the sensitive analysis on a train-set of examples and then evaluate the performance on the test-set of examples to see whether the approach can generalize to unseen sets of tasks in one benchmark.
Questions for authors
  1. How to define the window span in Section 4.1? And what will happen if we use single-token diagnosis.
  2. In lines 250-251, it is helpful to write out the definition of oo to explain why the output cosine distance captures changes from both key and value quantization.
  3. In any benchmarks, did you observe the divergence between the kl divergence and the cosine distance, i.e., they shows opposite results?
  4. Lines 473-475, the paper talks about locl score and long-range score, which are not defined anywhere.
4 comments
AuthorsOpenReview ↗

Response to Reviewer BLXy Part1

Thank you for your thoughtful and critical assessment. Many of your comments will help us produce a more readable and self-contained version of the paper. Below, we address each of your specific concerns in turn.

W1. Generalization to unseen tasks within SWE-bench Verified

Thank you for this helpful suggestion. We conducted the sensitivity analysis on a 50-instance calibration set from SWE-bench Verified to determine the layer-wise precision allocation, and then evaluated all seven configurations on the held-out test set of unseen instances. The TQ-Mix allocation was fixed after calibration, with no further tuning on the test set (Table R1).

Table R1. Resolved Rate, behavioral failure rates, and KV cache on the held-out test-set of SWE-bench Verified.

MetricBF16TQ-3bitTQ-BdKVTunerCoopQAMQTQ-Mix
Improper Tool Use or Setup (↓)37%68%62%56%57%55%42%
Loop Behavior (↓)16%42%39%30%35%31%19%
Insufficient Debugging (↓)47%65%60%55%54%53%47%
Average Bits (↓)16bit3bit4.1bit3.35bit3.35bit3.35bit3.35bit
KV Cache per Case (↓)6.31 GB1.58 GB2.22 GB1.43 GB1.50 GB1.55 GB1.36 GB
Resolved Rate (↑)42.4%38.9%39.6%38.9%39.8%39.3%41.8%

The held-out test results closely match the full-benchmark results in Table 1 in the manuscript, with resolve rates differing by at most 0.20.2 pp across all seven configurations. Specifically, TQ-Mix achieves a 41.8% resolve rate2.9 pp above TQ-3bit and 2.0--2.9 pp above all same-budget baselines while substantially curbing behavioral failures and requiring 13.9% less KV-cache memory per case than TQ-3bit (1.36 vs. 1.58 GB). These results confirm that both the resolve-rate and memory gains generalize reliably to unseen tasks.

Q1. Window definition and single-token diagnosis

Thank you for this great question. Clarifying the window formulation and testing the w=1w=1 boundary case helps sharpen Section 4.1.

  • Exact window formulation: Given an audited failure segment [t⋆,tend)[t^{\star}, t_{\mathrm{end}}), diagnosis operates over two query-position spans, a failure-onset window [t⋆,t⋆+w)[t^{\star}, t^{\star}+w) and a post-failure transition window [tend,tend+w)[t_{\mathrm{end}}, t_{\mathrm{end}}+w) with a default span of w=24w=24 tokens. Crucially, windowing restricts only the evaluated query indices.

  • Single token diagnosis (w=1w=1): Setting w=1w=1 token selects layers {3, 7, 12, 18, 30}, compared with {3, 5, 16, 28, 40} at w=24w=24. While w=1w=1 still captures the dominant outlier (Layer 3), relying on a single boundary token may introduce high token-level variance that perturbs secondary layer rankings; using a short span (w=24w=24 tokens) smooths out single-token noise across the action onset.

Q2. Projected head output formulation and joint K/V sensitivity

Thank you for this helpful suggestion. We will explicitly write out the projected head output oℓ,h,qo_{\ell,h,q} in Section 4.1 as

pℓ,h,q=softmax⁡(qℓ,h,qKℓ,h⊤dh).p_{\ell,h,q} = \operatorname{softmax}\left(\frac{\mathbf{q}_{\ell,h,q}K_{\ell,h}^{\top}}{\sqrt{d_h}}\right).

oℓ,h,q=(pℓ,h,qVℓ,h)Wℓ,hO.o_{\ell,h,q} = \left(p_{\ell,h,q}V_{\ell,h}\right)W_{\ell,h}^{O}.

where qℓ,h,q\mathbf{q}_{\ell,h,q} is the query vector at position qq, Kℓ,hK_{\ell,h} and Vℓ,hV_{\ell,h} are the causally visible key and value caches, and Wℓ,hOW_{\ell,h}^{O} denotes the head-hh slice of the output projection matrix.

With the query qℓ,h,q\mathbf{q}_{\ell,h,q} and projection Wℓ,hOW_{\ell,h}^{O} held fixed, key quantization (K^ℓ,h\widehat{K}_{\ell,h}) perturbs the attention routing weights p^ℓ,h,q\widehat{p}_{\ell,h,q}, whereas value quantization (V^ℓ,h\widehat{V}_{\ell,h}) directly distorts the aggregated value vectors. Because the quantized output o^ℓ,h,q=(p^ℓ,h,qV^ℓ,h)Wℓ,hO\widehat{o}_{\ell,h,q} = (\widehat{p}_{\ell,h,q}\widehat{V}_{\ell,h})W_{\ell,h}^{O} depends on the product of both terms, the cosine distance between oℓ,h,qo_{\ell,h,q} and o^ℓ,h,q\widehat{o}_{\ell,h,q} captures directional shifts in the residual stream induced jointly by key and value quantization, complementing the routing-only attention-KL metric.

AuthorsOpenReview ↗

Response to Reviewer BLXy Part2

Q3. Consistency between attention KL and output cosine distance

Across our evaluated benchmarks, the two metrics show consistent layer-sensitivity trends rather than opposite conclusions. In the analysis shown in Fig. 5(a), both diagnostics identify Layer 3 as the strongest outlier, while only some less sensitive layers undergo minor rank changes. As discussed in Q2, this is because attention KL measures changes in routing alone, whereas output cosine distance additionally reflects perturbations from value quantization.

Q4. Formulation of local and long-range attention scores

Thank you for pointing this out. For query position qq and causally visible key position j≤qj \le q at relative distance Δ=q−j\Delta = q - j (in tokens), local and long-range scores quantify the attention mass allocated to near versus distant context:

slocal=∑j:,0≤Δ<64pj,slong=∑j:,Δ>256pj,s_{\mathrm{local}} = \sum_{j:\\,0 \le \Delta < 64} p_j, \qquad s_{\mathrm{long}} = \sum_{j:\\,\Delta > 256} p_j,

where pjp_j is the attention weight assigned to key position jj. Thus, slocals_{\mathrm{local}} and slongs_{\mathrm{long}} sum attention mass over distances below 6464 tokens and beyond 256256 tokens, respectively.

Reviewer_BLXyOpenReview ↗

Response

I thank authors for their response and additional experiment results. I will retain my score.

Thanks you for your reply and for maintaining your positive recommendation. We greatly appreciate your time, careful review, and constructive feedback. We will continue to improve the paper accordingly.

Reviewer_cgYsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how low‑bit KV cache quantization affects multi‑turn coding agents rather than static benchmarks. Using Qwen3‑Coder‑30B‑A3B with OpenHands on SWE‑bench Verified and TurboQuant 3‑bit, it reports (i) a trajectory‑level analysis showing that quantization disproportionately increases three “execution‑control” failures (improper tool use +29pp, loops +26pp, insufficient debugging +18pp per an LLM auditor) and lengthens trajectories (+29.3% turns, +3.1k toks/case) while lowering resolve rate (42.2%→38.8%); (ii) a “failure‑centered” diagnosis that replays recorded tokens around auditor‑annotated failure windows, quantizes one head’s KV at a time, and measures attention KL / output cosine distance, finding ~5% of heads sensitive and concentrated in a few layers (Layer 3 dominant); (iii) TQ‑Mix, a three‑tier allocation (Layer 3 BF16, layers 5/16/28/40 at 4‑bit, rest 3‑bit; 3.35 bits avg) that recovers resolve rate to 41.6%, cuts the three failure rates by 18–24pp, and reduces observed KV per case from 1.56 to 1.38 GB. Comparisons against adapted KVTuner/CoopQ/AMQ layer‑selection criteria at the same 3.35‑bit budget, plus results on SWE‑bench Multilingual, Devstral 24B/123B, and static benchmarks on LLaMA‑3‑8B, are given as generalization evidence.

Critical strengths
  1. The framing and the headline empirical finding are great: Evaluating KV quantization by trajectory‑level cost (turns, tokens, observed cache per case) rather than per‑token bits is a genuinely different lens, and Table 1’s 1.38 GB (3.35 bit) < 1.56 GB (3 bit) is a clean, interesting demonstration that per‑token compression and per‑task memory can diverge. The behavior‑class decomposition (Fig. 2, Table 1) is more informative than a resolve‑rate delta alone.

  2. The experimental program is very thorough: Same‑budget baselines with alternative layer‑selection criteria (Table 1), precision‑allocation ablation (Table 4a), cross‑quantizer and window‑length robustness (Table 4b, Table 7), a second auditor (Table 6), two extra models and a second benchmark (Tables 2, 10), and static/long‑context sanity checks (Tables 3, 13). The failure‑centered replay with per‑head perturbation is a sensible diagnostic design that avoids confounding from divergent generations

Critical weaknesses
  1. No variance on outcome metrics; key deltas are small. On 500 instances, 38.8→41.6→42.2% (Table 1) is 14 and 3 instances; TQ‑Mix vs CoopQ is 8 instances; Devstral‑24B TQ‑Mix vs BF16 (Table 2) is 2. No seeds, repeated runs, or CIs are reported, and temperature/max‑iteration settings are unstated. The ~20pp behavioral deltas likely hold, but “approaches BF16” and “beats same‑budget baselines” are not statistically supported

  2. The “failure‑centered” advantage is not isolated. TQ‑Mix differs from the KVTuner/CoopQ/AMQ baselines in both the score (attention KL) and the calibration windows (failure segments). There is no ablation using attention‑KL on random windows. Evidence points to a structural, not failure‑specific, effect: Layer 3 is an outlier under every quantizer including FP8 (Table 4b), and restoring Layer 3 alone yields 40.8 of the 41.6 (Table 4a). If generic calibration finds the same layers, the paper’s claimed novelty over prior mixed‑precision criteria collapses.

  3. The memory claim is under‑specified and missing a comparator. “KV cache per case” is undefined (peak vs. mean vs. cumulative; peak is what matters for serving). Turns/tokens per case are given only for BF16 vs TQ‑3bit (Fig. 1), so the “shorter trajectories offset higher precision” mechanism cannot be checked for TQ‑Mix. TQ‑4bit uniform is never run on SWE‑bench, though it is the obvious “one more bit” baseline.

Questions for authors
  1. How many runs per configuration, at what temperature, and can you report CIs or at least a second seed for Table 1 and Table 2? What are OpenHands max iterations and is context condensation ever triggered (which would confound the cache/turn numbers)?

2/ Please add attention‑KL localization on random windows (or on all tokens) from the same calibration trajectories. Does it select {3,5,16,28,40}? If so, what does “failure‑centered” buy?

3/ Please report turns/tokens per case for all seven configurations (currently only BF16 vs TQ‑3bit in Fig. 1), and define “KV cache per case” (peak? mean over steps? sum?). Peak is what matters for serving; a longer trajectory with cheaper tokens may still have a lower peak.

  1. Were any auditor labels validated against human annotation on this data? The Gemini cross‑check shows notable disagreement on some categories (e.g., Insufficient Analysis recovery −15 vs −4, Table 6), and the auditor is used both to select calibration windows and to score outcomes.

  2. For LLaMA‑3‑8B (Table 3), what precision does each of layers 9/10/11/13 receive, and what is the average bit‑width vs TQ‑Bd (which there restores 4 of 32 layers)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cgYs Part1

Thank you for your thoughtful and critical assessment. Many of your comments will help us produce a more readable and self-contained version of the paper. Below, we address each of your specific concerns in turn.

W1, Q1. Statistical robustness and evaluation protocol

To rigorously assess statistical stability, we expanded all evaluations from a single run to three independent runs per configuration on SWE-bench Verified (Table R1) and across both cross-dataset/cross-model benchmarks (Table R2).

  • Consistent improvements over quantized baselines: On SWE-bench Verified (Table R1), TQ-Mix's 95% CI ([40.86,43.00][40.86, 43.00]) is completely non-overlapping with those of TQ-3bit, TQ-Bd, KVTuner, and AMQ. Against the strongest baseline CoopQ ([39.48,41.19][39.48, 41.19]), marginal CIs slightly overlap; under a paired comparison, TQ-Mix outperforms CoopQ in every run by +1.4, +1.6, and +1.8 pp (mean gain $+1.60 $ pp). Across both generalization settings (Table R2), TQ-Mix likewise maintains non-overlapping 95% CIs over both TQ-3bit (+3.67+3.67 and +3.60+3.60 pp) and TQ-Bd (+2.56+2.56 and +2.60+2.60 pp).

  • Proximity to full-precision BF16: Across runs, TQ-Mix recovers 85.7% of the uniform 3-bit degradation on SWE-bench Verified (within 0.600.60 pp of BF16) and 75.1% and 84.3% on the two generalization benchmarks (within 1.221.22 and 0.670.67 pp of BF16, respectively), with 95% CIs overlapping with BF16 across settings.

All runs use greedy decoding (temperature = 0), with OpenHands [1] max_iterations set to 300. We inspected the trajectories and confirmed that context condensation was not triggered in any run.

Table R1. Resolve rates across three seeds on SWE-bench Verified benchmark. Approximate 95% confidence intervals use the normal approximation μ±1.96 σ/3{\mu} \pm 1.96\,\sigma/\sqrt{3}, where σ\sigma is the sample standard deviation across the three runs.

ConfigurationSeed 2025Seed 2026Seed 2027MeanApprox. 95% CI
BF1641.842.243.642.53[41.46, 43.60]
TQ-3bit36.638.839.638.33[36.58, 40.09]
TQ-Bd37.439.439.838.87[37.41, 40.32]
KVTuner38.239.040.039.07[38.05, 40.09]
CoopQ39.840.041.240.33[39.48, 41.19]
AMQ38.439.240.439.33[38.19, 40.47]
TQ-Mix41.241.643.041.93[40.86, 43.00]

Table R2. Generalization resolve rates (%) and approximate 95% confidence intervals (μ±1.96 σ/3\mu \pm 1.96\,\sigma/\sqrt{3}) across datasets and model families. Gains are relative to TQ-3bit.

SettingConfigurationMean resolve rate (%)Approx. 95% CIGain over TQ-3bit (pp)
SWE-Multilingual (Qwen3-30B)BF1638.11[37.32, 38.90]—
SWE-Multilingual (Qwen3-30B)TQ-3bit33.22[32.64, 33.80]—
SWE-Multilingual (Qwen3-30B)TQ-Bd34.33[33.95, 34.71]+1.11
SWE-Multilingual (Qwen3-30B)TQ-Mix36.89[36.46, 37.32]+3.67
SWE-Verified (Devstral-24B)BF1666.47[65.53, 67.41]—
SWE-Verified (Devstral-24B)TQ-3bit62.20[61.00, 63.40]—
SWE-Verified (Devstral-24B)TQ-Bd63.20[62.16, 64.24]+1.00
SWE-Verified (Devstral-24B)TQ-Mix65.80[64.89, 66.71]+3.60

W2, Q2. Contribution of failure-centered windows

Thank you for this insightful suggestion. To directly verify the effectiveness of our localization strategy, we ran additional controlled ablations holding the attention-KL diagnostic and calibration trajectories fixed.

  • Controlled random-window ablation: Random windows sampled from pre-failure, broad in-failure, and post-failure segments all miss Layer 3 and instead rank Layers 18, 25, and 30 highest, respectively (yielding distinct layer sets from {3,5,16,28,40}\{3, 5, 16, 28, 40\}). Under the same 3.35-bit budget, the resulting allocations achieve resolve rates of 39.2%, 40.4%, and 39.6%, compared with 41.6% for TQ-Mix and 38.8% for uniform TQ-3bit. Even within broad in-failure regions, random sampling dilutes the localized failure-onset signal, confirming that failure-centered windowing is key to pinpointing the most effective layers.

  • Clarifying Tables 4(a) and 4(b) in the manuscript: Table 4(b) does not evaluate per-layer outliers such as Layer 3; instead, it shows across quantizers (including FP8) that sensitive heads are consistently sparse (2.5%–5.3%2.5\%\text{--}5.3\%) and predominantly concentrate on short-range local attention (0.62–0.770.62\text{--}0.77 vs. 0.09–0.220.09\text{--}0.22 long-range). Meanwhile, Table 4(a) ablates precision allocation after our diagnostic identifies the sensitive layers: compared with restoring Layer 3 alone (40.8%40.8\% resolve rate, 41.7%41.7\% behavioral failure), TQ-Mix's adaptive multi-layer allocation reaches 41.6% while cutting behavioral failures down to 37.0% (−4.7-4.7 pp).

AuthorsOpenReview ↗

Response to Reviewer cgYs Part2

  • Comparison with prior mixed-precision criteria: Layer 3 is not trivially identified by generic calibration. In Section 5.1, under the exact same calibration trajectories, quantizer, and bit budget, none of the adapted KVTuner, CoopQ, or AMQ criteria selects Layer 3.

W3, Q3. Peak KV memory, trajectory length, and uniform 4-bit baseline

  • Clarifying "KV cache per case": This metric reports the average peak KV-cache memory across trajectories, rather than a step-wise mean or cumulative sum, directly reflecting peak memory demand for serving capacity planning.

  • Trajectory length and average peak memory: Table R3 reports trajectory statistics across all seven main configurations and uniform TQ-4bit. Compared with uniform TQ-3bit, TQ-Mix reduces turns per case by 19.7% (63.88→51.2863.88 \to 51.28) and generated tokens by 16.6% (15.83k→13.20k15.83\text{k} \to 13.20\text{k}), while also yielding the shortest trajectories among all 3.35-bit baselines. Consequently, despite higher average precision (3.353.35 vs. 3.003.00 bits), TQ-Mix lowers average peak KV-cache memory by 11.5% (1.56→1.381.56 \to 1.38 GB), confirming that shorter trajectories more than offset the higher per-token bitwidth.

  • Comparison against uniform TQ-4bit: Uniform TQ-4bit averages 53.2253.22 turns and 14.35k14.35\text{k} generated tokens per case, reaching a 40.8%40.8\% resolve rate. Compared with uniform TQ-4bit, TQ-Mix achieves a higher resolve rate (41.6% vs. 40.8%) with shorter trajectories (51.2851.28 vs. 53.2253.22 turns) and 31.7% lower average peak KV-cache memory (1.38 vs. 2.02 GB).

Table R3. Average precision, interaction turns, generated tokens (in thousands), and resolve rates across the seven main configurations and the additional uniform TQ-4bit baseline on SWE-bench Verified.

ConfigurationAvg. BitsTurns per caseGenerated tokens per case (k)Resolve Rate (%)
BF1616.0049.3812.7542.2
TQ-3bit3.0063.8815.8338.8
TQ-Bd4.1057.6615.2139.4
KVTuner3.3558.2413.9239.0
CoopQ3.3554.5814.0840.0
AMQ3.3560.2014.8539.2
TQ-Mix3.3551.2813.2041.6
TQ-4bit4.0053.2214.3540.8

Q4. Manual verification of auditor disagreements

  • Manual inspection and cross-auditor disagreements: Using our behavioral rubric, we manually inspected 20 randomly sampled trajectories. Clear execution-control failures aligned closely with human judgment, whereas cross-auditor disagreements (such as Insufficient Analysis in Table 6 in the manuscript) mainly arose on borderline cases where Opus applied stricter judgments than Gemini; nevertheless, both auditors consistently confirm directional behavioral recovery under TQ-Mix.

  • Independent outcome scoring: Auditor labels are used strictly for diagnostic behavioral analysis rather than task success evaluation. Crucially, our primary outcome metric resolve rate is scored solely by executing the official SWE-bench repository test suites, completely independent of any auditor labels.

Q5. Layer precision for LLaMA-3-8B

Below we provide the exact per-layer precision assignments and average bit-widths for LLaMA-3-8B (32 layers). Specifically, TQ-Mix assigns BF16 to Layer 11, 4-bit to Layers 9, 10, and 13, and 3-bit to the remaining 28 layers, giving an average bit-width of (16+3×4+28×3)/32=(16 + 3\times4 + 28\times3)/32 = 3.50 bits. By contrast, TQ-Bd retains four boundary layers in BF16 and 3-bit elsewhere, averaging (4×16+28×3)/32=(4\times16 + 28\times3)/32 = 4.625 bits. Thus, TQ-Mix outperforms TQ-Bd across all five benchmarks in Table 3 in the manuscript while using 1.125 fewer bits on average (3.50 vs. 4.625 bits).

References

[1] Wang et al. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. ICLR, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.