- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a quantization technique that applies different bit precisions token-blockwise and layerwise in LLM-based multi-agent systems. Within a fixed bitwidth budget, the tokens generated online are grouped into blocks, the layer-wise quantization sensitivity is estimated for each block, and a bit width is allocated to each layer. The experiments were conducted in a multi-agent system, and within the experimental results, it showed superiority over other baselines.
Critical strengths
- The process of deriving a calibration-free upper bound on the quantization error for each layer and deriving the layerwise sensitivity from it is reasonable. This derivation method can inspire other studies that aim to update the hyperparameters of quantization, pruning, compression, etc. online.
Critical weaknesses
-
The multi-agent graph in S3.1, the method in S4, and the experimental setting in S5 are not aligned with each other. The method presents a quantization approach that assigns layer-wise bit widths online. However, the estimation method used in the method can be used regardless of the multi-agent environment, and since a fixed budget is used for each token block within each agent, there is also no budget exchange between agents at all. Unlike the multi-agent graph described in S3.1, the experimental setting in S5 has no case in which two or more subtasks are executed in parallel, and it deals with a situation in which only one agent is always operating sequentially.
-
The experimental results are insufficient to support the claims. In Table 1, the proposed method outperforms the other baselines at 3 bits, but in Table 2, when the weighted mean is computed considering the number of questions in each benchmark, it is difficult to regard it as outperforming KIVI and PM-KVQ. In addition, in both tables, there are frequent cases in which the quantization methods exceed the accuracy of Vanilla, and considering the magnitude of the gap from Vanilla and the gap between the different methods, it is difficult to regard these as reliable experimental results.
Questions for authors
-
To me, the introduced methodology appears to be, rather than belonging to the field of multi-agent systems, the existing KIVI method with layer-wise online bit width assignment added. While each token block uses a fixed budget, could you explain how different agents or the state of the multi-agent graph can affect the precision decision?
-
In the ablation Table 8, is Uniform allocation the same method as the KIVI baseline? If that is the case, the accuracy difference between Online sensitivity updates and Uniform allocation on AIME25 (which is 3.33 %p) corresponds to only a single question. Since this paper deals with experiments that include the use of external tools, the accuracy variation due to changes in tool outputs can be large. (Please let me know if external tools were not used in Table 8.) Therefore, in order to claim superiority in accuracy, it would be good to report the main results, including this table, over multiple runs together with the variance.
4 comments
Response to Reviewer Upe7 (2/2)
W2(a) The experimental results are insufficient to support the claims. In Table 1, the proposed method outperforms the other baselines at 3 bits, but in Table 2, when the weighted mean is computed considering the number of questions in each benchmark, it is difficult to regard it as outperforming KIVI and PM-KVQ.
Table 2 of the original submission compares performance across four types of reasoning tasks, so Bamboogle, GAIA, AIME25, and GPQA each receive 25% of the weight. Weighting by question count gives GPQA’s 198 questions 43.42% of the overall score and AIME25’s 30 questions 6.58%, a 6.6-fold difference. Applying these weights to our cross-benchmark comparison would make the influence of each task type depend on benchmark size. For our evaluation objective, the equal-weight macro-average in the original submission provides a fair comparison across all four task types.
W2(b) In addition, in both tables, there are frequent cases in which the quantization methods exceed the accuracy of Vanilla, and considering the magnitude of the gap from Vanilla and the gap between the different methods, it is difficult to regard these as reliable experimental results.
A model with a quantized KV cache can score higher than its full-precision counterpart on individual tasks. One possible explanation is that quantization maps KV activations in the same bin to a single value, which can suppress small variations that are irrelevant to the task. The resulting changes in attention can alter subsequent generations and lead to more accurate answers for some inputs.
Prior work on KV-cache quantization reports similar results on specific tasks.
- KIVI, Table 3. On Llama-2-13B, 4-bit KIVI scores 23.65% on GSM8K versus 22.67% for the 16-bit reference, a gain of 0.98 percentage points.
- PM-KVQ, Table 2. On DeepSeek-R1-Distill-Llama-8B, the mixed 4/8-bit configuration with batch size 6 achieves a mean AIME 2024 pass@1 of 47.71% versus 44.17% for the 16-bit reference, a gain of 3.54 percentage points.
- KVQuant, Table 3. On LLaMA-2-7B-32K, the 3-bit configuration retaining 1% of values as outliers scores 13.67 F1 on Qasper in LongBench, compared with 10.51 for the FP16 reference, a gain of 3.16 points.
Q2(a) In the ablation Table 8, is Uniform allocation the same method as the KIVI baseline? If that is the case, the accuracy difference between Online sensitivity updates and Uniform allocation on AIME25 (which is 3.33 %p) corresponds to only a single question.
Uniform allocation is an ablation within CoMixKV. It uses the CoMixKV quantization implementation and fixes all quantized layers at 4 bits. Online allocation uses the same target mean bit width and assigns widths by layer sensitivity. KIVI is evaluated as a separate baseline in Section 5.5. The Uniform ablation therefore measures the effect of online allocation versus fixed allocation within the same quantization implementation.
Your interpretation of the difference is correct. AIME25 contains 30 problems, so 3.33 percentage points corresponds to one additional correct answer. In Section 5.5 of the revised manuscript, online allocation solves 11/30 problems and Uniform solves 10/30, so the observed difference in this ablation is one problem.
Q2(b) Since this paper deals with experiments that include the use of external tools, the accuracy variation due to changes in tool outputs can be large. (Please let me know if external tools were not used in Table 8.) Therefore, in order to claim superiority in accuracy, it would be good to report the main results, including this table, over multiple runs together with the variance.
-
Tool use in Table 8. AIME25 uses a closed-book setting with web search and page visits disabled. Local Python is available for computation. In the revised ablation, online allocation and Uniform use the same tool permissions and random seed. The execution logs show Python calls in both conditions (Section 5.5).
-
Repeated runs and variability. In the main 4-bit AIME25 comparison, we evaluated CoMixKV and KIVI with seeds 0 and 1. CoMixKV leads in both runs, with a mean difference of +5.00 percentage points and a question-paired 95% confidence interval of [−8.33, 16.67] (Section 5.5). The online-versus-Uniform ablation uses seed 0. Its observed difference is +3.33 percentage points, with a question-paired 95% confidence interval of [−13.33, 20.00] (Section 5.5). These intervals quantify uncertainty across questions.
Thank you for the additional experiments and the effort put into the revision. I note that the rebuttal was submitted less than a day before the end of the discussion period. I have only partially gone through it so far, and I have not been able to review it thoroughly enough to engage in further discussion before the deadline. I will carefully review the rebuttal and the revised manuscript during the remaining review period after the discussion closes.
Thank you for your thoughtful review and for taking the time to follow up. We recognize that our response came late in the discussion period, leaving little time for further exchange. We appreciate your willingness to read the rebuttal and revised manuscript after the discussion closes. Section 5.5 addresses your question about how one agent can affect another’s precision allocation. In the parallel coding workflow, the traces show that one worker could use KV-cache capacity left unused by the other while staying within the team’s total budget. On HumanEval+, sharing this capacity improved accuracy by 9.15 percentage points over independent allocation at the same total budget. Thank you again for raising this question and for taking the time to consider the revision.
Response to Reviewer Upe7 (1/2)
We greatly appreciate your time, detailed comments, and valuable suggestions.
Thank you for pointing out that the connection between these sections needed clarification. The original reasoning experiments in Section 5 apply the allocation method in Section 4 while following the execution dependencies described in Section 3.1.
The graph in Section 3.1 describes task dependencies and message passing among agents. The original reasoning experiments in Section 5 use a sequential schedule. For each problem, the manager delegates subtasks to specialized workers, combines their reports, and passes a candidate answer to the verifier. One invocation generates tokens at a time, with different agents carrying out different stages of the same task. This execution order follows the dependencies in the graph in Section 3.1.
To evaluate a graph with parallel branches, the revision adds a parallel coding experiment. After the manager finishes planning, two workers concurrently generate candidate code and test assertions. The verifier then revises the code using execution feedback (Section 5.5). We traced the same 16 prespecified tasks under both budgeting policies and observed concurrent KV cache residency for the two workers in all 32 traces. Under the shared budget, we recorded 632 allocations during cache overlap. In 525 events, a worker’s reservation exceeded its independent limit, with zero recorded team-budget violations (Section 5.5).