- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Since KV-caches cannot be reused across architectures, each model has to prefill or store caches for identical contexts. This work proposes MoT, a mixture-of-translators architecture, which dynamically assembles multiple translation modules at the token level. A context-correction loss is further introduced to enable KV-cache reuse among heterogeneous large language models while preserving downstream task performance as much as possible.
Critical strengths
-
MoT addresses KV‑cache reuse across heterogeneous large language models. Combined with the Free cache‑eviction policy in multi‑agent settings, MoT reduces normalized KV‑cache overhead while retaining decent downstream performance. Furthermore, theoretical analysis of competing errors together with ablation studies on routing selection and context‑correction loss interpret the rationality of design choices and the contribution of individual components.
-
The paper assesses MoT from multiple perspectives via architectural ablations, context‑correction loss ablations, baseline comparisons on two QA datasets, multi‑agent reasoning experiments and long‑context CAG experiments. Ablation studies analyze contributions of different components and their combinations; further experiments examine performance under various task settings. Results support the effectiveness of MoT and reveal the inherent trade‑off between task quality and inference latency.
Critical weaknesses
-
Although the paper aims to reduce redundant computation through KV-cache reuse, the proposed method still requires target-side context replay. The computational overhead of KV translation and context replay is not sufficiently analyzed, making the actual end-to-end efficiency gains unclear.
-
MoT introduces additional translator capacity and supervision compared with existing methods. However, the experiments lack sufficiently controlled comparisons under matched parameter counts, training objectives, and computational budgets, making it difficult to isolate the contributions of the proposed architecture.
-
In the heterogeneous multi-agent experiments, MoT uses the Free cache-management strategy, while competing methods use Retain. Consequently, the reported memory savings may partly result from different cache-management policies rather than the proposed KV-cache translation method itself.
-
Although the paper identifies propagation error and correction-deficit error, the theoretical analysis does not directly justify the effectiveness of the MoT architecture. Moreover, the experimental findings suggest that performance improvements are not primarily attributable to reducing the correction-deficit coefficient, weakening the connection between the theoretical motivation and the proposed approach.
Questions for authors
- When multiple distinct LLMbased agents coexist within a multiagent system, does the MoT framework require training a separate translator for every sourcetarget pair?
- For the observed Ushaped validationloss and competingerror phenomenon: does it stably occur across all evaluated QA datasets or only within specific subsets? Are there any discussions or experiments in the manuscript supporting its generality across different model pairs and diverse tasks?