- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Hello, authors! First of all, thank you very much for your work, found the materials very interesting. In this section, i will try to clearly provide a summary of the paper based on my own understanding.
After reading the paper, I supposed that your work scope is more eng impl combined with ml methods, so for a deeper understanding, I decided to rely on a careful reading of the source code; fortunately, there are not many implementation files there.
Essentially, your idea is to examine speculative decoding from the perspective of edge devices. This brings in classic constraints, like: bottleneck with target model does not fit into the available RAM (DRAM), requiring some of its layers to be loaded from flash memory during each pass. To address this, you propose CATS, a variant of self-speculative decoding. The model is divided into three stages based on memory-related heuristics: (1) a pre-generation (called draft) stage (initial layers + an adapter); (2) an intermediate verification stage (layers residing in DRAM + a second adapter), i genuinely think the key algo trick here, is that the proposed tokens are verified in a single parallel pass, with corrected alternatives provided in the event of a mismatch; and (3) a target stage, where the remaining layers are loaded from flash memory (once per cycle) to verify the tree of candidate sequences. Additionally, the adapters are trained using a loss function based on distillation with top-K truncation (referred to as "Reduced KL").
In summary, the contribution is as follows: (i) adapting the concept of speculative decoding for scenarios with significant resource constraints. (ii) The use of intermediate layers as a "lightweight" verifier, which proposes alternative token options to construct a tree of hypotheses, is, in my view as already mentioned, the most interesting aspect.
To evaluate your results, you conducted experiments using 5 open-source models 3 families (Vicuna-7B/13B, LLaMA2-7B/13B, LLaMA3-8B) across 5 benchmarks, running on Jetson AGX Orin (with emulated memory constraints) and Orin Nano hardware.
From the code, the implementation builds on Kangaroo (https://github.com/Equationliu/Kangaroo), uses a simplified variant of the EAGLE-2 tree, and follows Medusa's typical acceptance (https://github.com/FasterDecoding/Medusa) for the relaxed (temperature) setting. I put my concerns about it in weaknesses and questions below.
Critical strengths
- the proposed tokens are verified by shallow verifier in a single parallel pass, with corrected alternatives provided in the event of a mismatch. these alternatives go to the tree so the expensive target pass (layers streamed from flash) can accept more tokens per cycle. i think this is the main new part compared to Kangaroo
- setting is practical: when model does not fit in DRAM, flash->DRAM transfer is the bottleneck, so tokens accepted per streamed pass is the right thing to optimize. choosing SV depth by memory budget follows from this naturally
Critical weaknesses
Critical:
- eval not so strong and hard to reproduce: in provided code there is no flash streaming / memory budget harness (model is loaded fully on GPU with device_map="auto"), no baselines (Kangaroo, EAGLE, Medusa, AR) and no EAGLE-3 integration, so only τ can be reproduced, not speedups from Tables 1-5. also in chain mode with README settings (--tree-topk 1 --total-tokens -1) SV corrections never enter the tree (sib_budget=0 in CATS_dynamic.py), so not clear how chain results were obtained
- some numbers look inconsistent: CATS speedup is close or above τ while baselines get ~0.85τ, Kangaroo τ is 2.27 in Tables 4/5 but 2.49 in Tables 1/15, and with 1.47 s per streamed layer and 13 streamed layers at 8 GB the AR speed should be around 0.05 tok/s, lower than what Table 4 reports
- "outperforming all compared self-speculative decoding baselines including KANGAROO, MEDUSA, EAGLE and EAGLE-3" - when you compare with prior works is better to mention that you are basically extending their impl and idea. adapter, training and eval code are mostly from Kangaroo repo, tree is simplified EAGLE-2 tree (one trunk + depth-1 siblings), relaxed mode is Medusa typical acceptance. would be good to say it directly and show what SV adds on top
other:
- "CATS is model-agnostic: the cascaded framework applies to any transformer-based LLM without architectural modifications" - this is listed in contribution but is basically a side take, any self-speculative method with adapters is model-agnostic in this sense, and you still need to train adapters per model
- is Reduced KL Loss giving enough gain? Table 16 shows +0.17 pp hit rate and τ 3.227 -> 3.237 on single seed, this looks within noise, but it is listed as contribution
- 3 PRELIMINARY AND MOTIVATION maybe here you should directly mention other possible setups that gives opportunity to avoid this particular bottleneck (INT8 that fits 8 GB budget for 7B, optimized INT4 runtimes like llama.cpp / TensorRT-LLM, NVMe instead of eMMC, smaller model - Table 12 shows Llama-3.2-3B is close on MT-bench) so them we can better understand why need this exact machinery on this level. Appendix G compares only with NF4 from bitsandbytes
- closest prior work for speculative decoding under offloading (SpecExec, Sequoia offloading experiments) is not compared
- feels like paper writing is somehow circling around same takes and a bit vague, motivation is repeated several times and some appendices just restate main tables
Questions for authors
clarification:
- how chain mode results were obtained if SV corrections ar not added to the tree with README settings?
- why CATS speedup is equal or higher than τ while baselines are ~0.85τ? from Eq. 4 i would expect both ≤ τ. what Kangaroo exit threshold was used and why its τ differs between Tables 1/15 and 4/5?
- how EAGLE-3 is integrated? there is no code for it. can you release edge harness and baseline configs?
- which tables use relaxed mode? it is Medusa typical acceptance so not lossless, and Table 11 GSM8K differs from base model
- what tree was used for EAGLE self-speculative trwe baseline?
- what the exact strong difference with prior work (Kangaroo, EAGLE-2), except SV corrections?
suggestions:
- ablation with same draft and same tree budget with and without SV, to see the real gain from cascade
- the simplified tree (a single draft trunk with depth-1 alternatives) loses deeper branching after an alternative token and EAGLE-2's re-ranking of a multi-level tree by cumulative confidence. also since flash is ~99% of latency, verifying much bigger tree is almost free, why limit to 40 nodes?
- comparison with INT8 or optimized INT4 runtime, alone and combined with CATS, in absolute tok/s
- KnapSpec is cited but not discussed / compared, maybe its worth to compare directly
3 comments
Rebuttal for reviewer Tz3o(2)
S1: Ablation for SV
We appreciate the reviewer's suggestion. However, the shallow verifier (SV) is an integral component of our cascade design rather than an optional enhancement. Its ability to provide corrections using resident layers is a key advantage of the proposed architecture. Removing the SV would fundamentally alter the cascade mechanism rather than isolate the contribution of an auxiliary component. We believe comparisons under identical drafting configurations and forwarding passes, as provided in our experiments, more directly evaluate the benefit of our proposed method.
S2: Tree Structure and Node Budget
We thank the reviewer for raising these concerns. We would like to clarify that CATS is an architectural design rather than a specific tree-drafting strategy. Our proposed cascade is broadly applicable to existing speculative and self-speculative decoding methods, including EAGLE-2 and its multi-level dynamic tree.
Regarding the 40-node budget, although weight movement is the primary bottleneck, edge devices still have limited computational capacity, making larger verification trees increasingly costly. We selected 40 nodes after evaluating multiple configurations, as it provides a practical balance between acceptance gains and computational overhead.
S3: INT8 / INT4 Runtime Comparison
We appreciate the reviewer's thought about quantization. We would like to clarify that quantization is not a cost-free alternative: it may introduce generation quality degradation, and our method is quality-lossless. We provide the evaluated generation quality of using quantization below:
Table R1. Generation quality and memory footprint of CATS and quantized inference on Llama-3-8B.
| Method | Memory (GiB) | MT-bench | GSM8K |
|---|---|---|---|
| Llama-3-8B FP16 | 14.96 | 7.281 | 78.33% |
| Llama-3-8B NF4 (4-bit) | 5.21 | 6.856 | 76.00% |
| CATS (Llama-3-8B) | 14.96 | 7.338 | 79.33% |
As shown in Table R1, 4-bit NF4 quantization reduces Llama-3-8B's MT-bench score from 7.281 to 6.856 and GSM8K accuracy from 78.33% to 76.00%, whereas CATS maintains comparable or better generation quality than the FP16 baseline.
In addition, our method is orthogonal to quantization, and the two methods can be combined. In particular, a quantized model that still exceeds the resident budget must still stream non-resident layers; our method can reduce how often those layers are reread.
We additionally evaluate CATS with INT4 quantization under 2GB memory budget. As shown in Table R2, INT4 CATS achieves 4.54 tok/s on Vicuna-7B and 1.73 tok/s on Vicuna-13B, corresponding to 2.98× and 3.17× end-to-end speedups over INT4 autoregressive decoding. Compared with INT4 Kangaroo, CATS further improves throughput by 1.40× and 1.48×, respectively. These results demonstrate that CATS remains effective when combined with quantization.
Table R2. Absolute throughput and end-to-end speedup of INT4 methods under 2GB memory budget. Speedups are measured against INT4 autoregressive decoding.
| Method | Metric | Spec | MT | GSM8K | Alpaca | HumanEval | Mean |
|---|---|---|---|---|---|---|---|
| Vicuna-7B (INT4) | |||||||
| INT4 AR | tok/s | 1.52 | 1.52 | 1.52 | 1.52 | 1.52 | 1.52 |
| Speedup | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | |
| INT4 Kangaroo | tok/s | 2.97 | 3.22 | 3.24 | 2.99 | 3.83 | 3.25 |
| Speedup | 1.96× | 2.11× | 2.13× | 1.97× | 2.52× | 2.14× | |
| INT4 CATS | tok/s | 4.27 | 4.45 | 4.68 | 4.14 | 5.13 | 4.54 |
| Speedup | 2.81× | 2.92× | 3.08× | 2.72× | 3.38× | 2.98× | |
| Vicuna-13B (INT4) | |||||||
| INT4 AR | tok/s | 0.546 | 0.546 | 0.546 | 0.546 | 0.546 | 0.546 |
| Speedup | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | |
| INT4 Kangaroo | tok/s | 1.08 | 1.17 | 1.08 | 1.10 | 1.43 | 1.17 |
| Speedup | 1.97× | 2.14× | 1.98× | 2.00× | 2.61× | 2.14× | |
| INT4 CATS | tok/s | 1.63 | 1.72 | 1.68 | 1.60 | 2.04 | 1.73 |
| Speedup | 2.98× | 3.15× | 3.07× | 2.92× | 3.72× | 3.17× |
Rebuttal for reviewer Tz3o(3)
S4: KnapSpec
Thanks for providing this. KnapSpec did not release the official implementation codes, so we didn't do related experiments. We choose aonther open-source work, SpecExec, and conduct additional experiments using their implementation. We use Vicuna-68M (0.14 GB in fp16) as the draft model, which fits within the memory budget of edge devices, and evaluate on 40 nodes and 128 nodes separately.
Table R3. Comparison with SpecExec on Vicuna-7B (MT-bench, greedy decoding).
| Method | Tree Nodes | Speedup |
|---|---|---|
| SpecExec | 40 | 4.28× |
| CATS w/ SpecExec | 40 | 4.85× |
| SpecExec | 128 | 4.95× |
| CATS w/ SpecExec | 128 | 5.21× |
As shown in Table R3, our approach improves end-to-end speedup from 4.28× to 4.85× and from 4.95× to 5.21× under the 40-node and 128-node tree budgets, respectively. These results demonstrate the effectiveness of our cascade and correction mechanism in improving practical decoding efficiency.
Additional Weaknesses
W2: Autoregressive Throughput at the 8 GB Budget
Thanks again for pointing this out, the detailed response can be checked in C2. For the AR's speed you calculate, that is based on a more bandwidth-constrained condition as we presented at Section D.1 in the appendix of our revised paper.
W3: Attribution to Prior Implementations
Thanks for raising these concerns, detailed responses can be reviewed above.
O1: "Model-Agnostic" Claim
We thank the reviewer for pointing this out. We acknowledge that CATS requires training model-specific adapters for different base models, and will clarify this in the revised manuscript. But we want to clarify that we just need to fine-tune a lightweight adapter, rather than the large model. And it can be done at scale by tuning multiple during the development phase before deployment. We appreciate the reviewer's thought, and have included this in the Conclusion and Limitations section of the revised draft.
O2: Gain from the Reduced KL Loss
We thank the reviewer for this observation. Reduced KL Loss is a training optimization we identified while developing the adapters and is our secondary contribution.
O3: Alternative Setups in the Motivation
We thank the reviewer for raising this concern. We want to clarify that quantization is not a cost-free alternative: it may introduce generation quality degradation. And CATS is orthogonal to quantization and can be applied directly to quantized models. We also provide related experiment results for your reference in S3.
We also want to clarify that the major contribution of CATS is that it provides edge devices with the capacity to enable larger-model inference under limited DRAM budgets, which enables them to conduct complex reasoning and solve tasks that a small model cannot when necessary.
O4: SpecExec and Sequoia
We thank for reviewer's suggestions. We provide experiment results tested on SpecExec which can be checked in response for S4.
Rebuttal for reviewer Tz3o
C1: Chain-Mode Results
We thank the reviewer for pointing this out. Upon carefully reviewing our experimental settings, we found that the README had not been updated. Our actual chain-mode experiments used
--total-tokens 40, with SV corrections included as sibling nodes along the draft trunk (Section 4.1, Figure 4). We will correct the README accordingly.C2: Speedup vs. τ and Kangaroo Settings
We clarify that the accepted length τ and end-to-end speedup S are not equivalent. In our flash-offloading setting, speedup depends on both the accepted length and the per-cycle latency, including flash-to-DRAM weight transfers and computation overhead.
The reviewer's characterization of baseline speedups as approximately 0.85τ does not hold consistently: for example, REST achieves 1.75× speedup with τ=1.71, while Medusa achieves 2.49× with τ=2.48. Our measurements are conducted on a real Jetson AGX Orin device, where concurrent system processes may also introduce runtime variability. The ratio between speedup and τ is not expected to be constant across methods.
For τ in Table4/5, we thank the reviewer for pointing out this discrepancy. After checking the original experimental records, we confirmed that the reported value of 2.27 corresponds to Spec-bench and have corrected it in our revised paper. We also reran the other experiments and verified that the other results are consistent with the reported values.
C3: EAGLE-3 Integration and Code Release
We thank the reviewer for pointing this out. We have submitted related files in our revised supplementary.
C4: Relaxed Mode
For relaxed decoding, we adopt the acceptance rule from Medusa. Relaxed decoding (T=0.7) is reported in the lower blocks of Tables 1 and 2, the R columns of Table 3, the T=0.7 columns of Table 10 (now Table 14 in the revised paper; derived from Tables 1–3), and Figure 5. An exception is EAGLE-3 in Table 3, whose R results use its native distribution-preserving speculative sampling. All remaining results use greedy decoding.
C5: Tree of the EAGLE Self-Speculative Baseline
Our EAGLE-self implementation is based on the official EAGLE-1 implementation and adopts its original draft tree design.
C6: Difference from Kangaroo and EAGLE-2
Kangaroo is a simple self-speculative decoding method without cascade design. EAGLE-2 is a speculative decoding method with stronger independent draft model and dynamic tree design.
The key novelty of CATS lies in its memory-aware cascade and shallow-verification correction mechanism. By exploiting the intermediate layers that fit within the DRAM budget, our cascade can provide additional correction branches, enabling more candidate nodes for target verification without additional flash weight transfers and forward passes. This allows CATS to better utilize limited resident memory.
Importantly, the cascade is independent of the drafting strategy and can be integrated with many other speculative and self-speculative decoding methods.