acceptodds
Under review as a conference paper at ICLR 2027

CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration

Abstract

Auto-regressive decoding in Large Language Models (LLMs) is inherently memory-bound: every generation step requires loading the model weights and intermediate results from memory (e.g., High-Bandwidth Memory (HBM) for GPU servers), making throughput bottlenecked by memory bandwidth rather than compute. Speculative decoding addresses this by enabling parallel verification of multiple draft tokens, effectively amortizing the cost of each target-model call. However, existing speculative decoding methods are designed under the assumption that HBM is sufficiently large to hold both the target model and an auxiliary draft model simultaneously – an assumption that breaks down on memory-constrained devices such as edge platforms with limited DRAM. We analyze the inference bottleneck in this memory-limited regime and propose CATS, a self-speculative decoding framework that conducts cascaded verification and correction based on the memory budget and parameter offloading patterns on memory-limited devices. This design maximizes token acceptance rate and end-to-end speedup while keeping the peak memory footprint on the device equal to that of the target model alone. We evaluate CATS on different models across five benchmarks on real edge devices. CATS can achieve a wall-clock speedup of up to 5.08x with no degradation in generation quality, outperforming the SOTA method by up to 1.60x under edge memory constraints.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
self-speculative decoding, memory-limited LLM inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_Tz3oOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Hello, authors! First of all, thank you very much for your work, found the materials very interesting. In this section, i will try to clearly provide a summary of the paper based on my own understanding.

After reading the paper, I supposed that your work scope is more eng impl combined with ml methods, so for a deeper understanding, I decided to rely on a careful reading of the source code; fortunately, there are not many implementation files there.

Essentially, your idea is to examine speculative decoding from the perspective of edge devices. This brings in classic constraints, like: bottleneck with target model does not fit into the available RAM (DRAM), requiring some of its layers to be loaded from flash memory during each pass. To address this, you propose CATS, a variant of self-speculative decoding. The model is divided into three stages based on memory-related heuristics: (1) a pre-generation (called draft) stage (initial layers + an adapter); (2) an intermediate verification stage (layers residing in DRAM + a second adapter), i genuinely think the key algo trick here, is that the proposed tokens are verified in a single parallel pass, with corrected alternatives provided in the event of a mismatch; and (3) a target stage, where the remaining layers are loaded from flash memory (once per cycle) to verify the tree of candidate sequences. Additionally, the adapters are trained using a loss function based on distillation with top-K truncation (referred to as "Reduced KL").

In summary, the contribution is as follows: (i) adapting the concept of speculative decoding for scenarios with significant resource constraints. (ii) The use of intermediate layers as a "lightweight" verifier, which proposes alternative token options to construct a tree of hypotheses, is, in my view as already mentioned, the most interesting aspect.

To evaluate your results, you conducted experiments using 5 open-source models 3 families (Vicuna-7B/13B, LLaMA2-7B/13B, LLaMA3-8B) across 5 benchmarks, running on Jetson AGX Orin (with emulated memory constraints) and Orin Nano hardware.

From the code, the implementation builds on Kangaroo (https://github.com/Equationliu/Kangaroo), uses a simplified variant of the EAGLE-2 tree, and follows Medusa's typical acceptance (https://github.com/FasterDecoding/Medusa) for the relaxed (temperature) setting. I put my concerns about it in weaknesses and questions below.

Critical strengths
  • the proposed tokens are verified by shallow verifier in a single parallel pass, with corrected alternatives provided in the event of a mismatch. these alternatives go to the tree so the expensive target pass (layers streamed from flash) can accept more tokens per cycle. i think this is the main new part compared to Kangaroo
  • setting is practical: when model does not fit in DRAM, flash->DRAM transfer is the bottleneck, so tokens accepted per streamed pass is the right thing to optimize. choosing SV depth by memory budget follows from this naturally
Critical weaknesses

Critical:

  • eval not so strong and hard to reproduce: in provided code there is no flash streaming / memory budget harness (model is loaded fully on GPU with device_map="auto"), no baselines (Kangaroo, EAGLE, Medusa, AR) and no EAGLE-3 integration, so only τ can be reproduced, not speedups from Tables 1-5. also in chain mode with README settings (--tree-topk 1 --total-tokens -1) SV corrections never enter the tree (sib_budget=0 in CATS_dynamic.py), so not clear how chain results were obtained
  • some numbers look inconsistent: CATS speedup is close or above τ while baselines get ~0.85τ, Kangaroo τ is 2.27 in Tables 4/5 but 2.49 in Tables 1/15, and with 1.47 s per streamed layer and 13 streamed layers at 8 GB the AR speed should be around 0.05 tok/s, lower than what Table 4 reports
  • "outperforming all compared self-speculative decoding baselines including KANGAROO, MEDUSA, EAGLE and EAGLE-3" - when you compare with prior works is better to mention that you are basically extending their impl and idea. adapter, training and eval code are mostly from Kangaroo repo, tree is simplified EAGLE-2 tree (one trunk + depth-1 siblings), relaxed mode is Medusa typical acceptance. would be good to say it directly and show what SV adds on top

other:

  • "CATS is model-agnostic: the cascaded framework applies to any transformer-based LLM without architectural modifications" - this is listed in contribution but is basically a side take, any self-speculative method with adapters is model-agnostic in this sense, and you still need to train adapters per model
  • is Reduced KL Loss giving enough gain? Table 16 shows +0.17 pp hit rate and τ 3.227 -> 3.237 on single seed, this looks within noise, but it is listed as contribution
  • 3 PRELIMINARY AND MOTIVATION maybe here you should directly mention other possible setups that gives opportunity to avoid this particular bottleneck (INT8 that fits 8 GB budget for 7B, optimized INT4 runtimes like llama.cpp / TensorRT-LLM, NVMe instead of eMMC, smaller model - Table 12 shows Llama-3.2-3B is close on MT-bench) so them we can better understand why need this exact machinery on this level. Appendix G compares only with NF4 from bitsandbytes
  • closest prior work for speculative decoding under offloading (SpecExec, Sequoia offloading experiments) is not compared
  • feels like paper writing is somehow circling around same takes and a bit vague, motivation is repeated several times and some appendices just restate main tables
Questions for authors

clarification:

  • how chain mode results were obtained if SV corrections ar not added to the tree with README settings?
  • why CATS speedup is equal or higher than τ while baselines are ~0.85τ? from Eq. 4 i would expect both ≤ τ. what Kangaroo exit threshold was used and why its τ differs between Tables 1/15 and 4/5?
  • how EAGLE-3 is integrated? there is no code for it. can you release edge harness and baseline configs?
  • which tables use relaxed mode? it is Medusa typical acceptance so not lossless, and Table 11 GSM8K differs from base model
  • what tree was used for EAGLE self-speculative trwe baseline?
  • what the exact strong difference with prior work (Kangaroo, EAGLE-2), except SV corrections?

suggestions:

  • ablation with same draft and same tree budget with and without SV, to see the real gain from cascade
  • the simplified tree (a single draft trunk with depth-1 alternatives) loses deeper branching after an alternative token and EAGLE-2's re-ranking of a multi-level tree by cumulative confidence. also since flash is ~99% of latency, verifying much bigger tree is almost free, why limit to 40 nodes?
  • comparison with INT8 or optimized INT4 runtime, alone and combined with CATS, in absolute tok/s
  • KnapSpec is cited but not discussed / compared, maybe its worth to compare directly
3 comments
AuthorsOpenReview ↗

Rebuttal for reviewer Tz3o

C1: Chain-Mode Results

We thank the reviewer for pointing this out. Upon carefully reviewing our experimental settings, we found that the README had not been updated. Our actual chain-mode experiments used --total-tokens 40, with SV corrections included as sibling nodes along the draft trunk (Section 4.1, Figure 4). We will correct the README accordingly.


C2: Speedup vs. τ\tau and Kangaroo Settings

We clarify that the accepted length τ\tau and end-to-end speedup SS are not equivalent. In our flash-offloading setting, speedup depends on both the accepted length and the per-cycle latency, including flash-to-DRAM weight transfers and computation overhead.

The reviewer's characterization of baseline speedups as approximately 0.85τ0.85\tau does not hold consistently: for example, REST achieves 1.75× speedup with τ=1.71\tau=1.71, while Medusa achieves 2.49× with τ=2.48\tau=2.48. Our measurements are conducted on a real Jetson AGX Orin device, where concurrent system processes may also introduce runtime variability. The ratio between speedup and τ\tau is not expected to be constant across methods.

For τ\tau in Table4/5, we thank the reviewer for pointing out this discrepancy. After checking the original experimental records, we confirmed that the reported value of 2.27 corresponds to Spec-bench and have corrected it in our revised paper. We also reran the other experiments and verified that the other results are consistent with the reported values.


C3: EAGLE-3 Integration and Code Release

We thank the reviewer for pointing this out. We have submitted related files in our revised supplementary.


C4: Relaxed Mode

For relaxed decoding, we adopt the acceptance rule from Medusa. Relaxed decoding (T=0.7T=0.7) is reported in the lower blocks of Tables 1 and 2, the R columns of Table 3, the T=0.7T=0.7 columns of Table 10 (now Table 14 in the revised paper; derived from Tables 1–3), and Figure 5. An exception is EAGLE-3 in Table 3, whose R results use its native distribution-preserving speculative sampling. All remaining results use greedy decoding.


C5: Tree of the EAGLE Self-Speculative Baseline

Our EAGLE-self implementation is based on the official EAGLE-1 implementation and adopts its original draft tree design.


C6: Difference from Kangaroo and EAGLE-2

Kangaroo is a simple self-speculative decoding method without cascade design. EAGLE-2 is a speculative decoding method with stronger independent draft model and dynamic tree design.

The key novelty of CATS lies in its memory-aware cascade and shallow-verification correction mechanism. By exploiting the intermediate layers that fit within the DRAM budget, our cascade can provide additional correction branches, enabling more candidate nodes for target verification without additional flash weight transfers and forward passes. This allows CATS to better utilize limited resident memory.

Importantly, the cascade is independent of the drafting strategy and can be integrated with many other speculative and self-speculative decoding methods.


AuthorsOpenReview ↗

Rebuttal for reviewer Tz3o(2)

S1: Ablation for SV

We appreciate the reviewer's suggestion. However, the shallow verifier (SV) is an integral component of our cascade design rather than an optional enhancement. Its ability to provide corrections using resident layers is a key advantage of the proposed architecture. Removing the SV would fundamentally alter the cascade mechanism rather than isolate the contribution of an auxiliary component. We believe comparisons under identical drafting configurations and forwarding passes, as provided in our experiments, more directly evaluate the benefit of our proposed method.


S2: Tree Structure and Node Budget

We thank the reviewer for raising these concerns. We would like to clarify that CATS is an architectural design rather than a specific tree-drafting strategy. Our proposed cascade is broadly applicable to existing speculative and self-speculative decoding methods, including EAGLE-2 and its multi-level dynamic tree.

Regarding the 40-node budget, although weight movement is the primary bottleneck, edge devices still have limited computational capacity, making larger verification trees increasingly costly. We selected 40 nodes after evaluating multiple configurations, as it provides a practical balance between acceptance gains and computational overhead.


S3: INT8 / INT4 Runtime Comparison

We appreciate the reviewer's thought about quantization. We would like to clarify that quantization is not a cost-free alternative: it may introduce generation quality degradation, and our method is quality-lossless. We provide the evaluated generation quality of using quantization below:

Table R1. Generation quality and memory footprint of CATS and quantized inference on Llama-3-8B.

MethodMemory (GiB)MT-benchGSM8K
Llama-3-8B FP1614.967.28178.33%
Llama-3-8B NF4 (4-bit)5.216.85676.00%
CATS (Llama-3-8B)14.967.33879.33%

As shown in Table R1, 4-bit NF4 quantization reduces Llama-3-8B's MT-bench score from 7.281 to 6.856 and GSM8K accuracy from 78.33% to 76.00%, whereas CATS maintains comparable or better generation quality than the FP16 baseline.

In addition, our method is orthogonal to quantization, and the two methods can be combined. In particular, a quantized model that still exceeds the resident budget must still stream non-resident layers; our method can reduce how often those layers are reread.

We additionally evaluate CATS with INT4 quantization under 2GB memory budget. As shown in Table R2, INT4 CATS achieves 4.54 tok/s on Vicuna-7B and 1.73 tok/s on Vicuna-13B, corresponding to 2.98× and 3.17× end-to-end speedups over INT4 autoregressive decoding. Compared with INT4 Kangaroo, CATS further improves throughput by 1.40× and 1.48×, respectively. These results demonstrate that CATS remains effective when combined with quantization.

Table R2. Absolute throughput and end-to-end speedup of INT4 methods under 2GB memory budget. Speedups are measured against INT4 autoregressive decoding.

MethodMetricSpecMTGSM8KAlpacaHumanEvalMean
Vicuna-7B (INT4)
INT4 ARtok/s1.521.521.521.521.521.52
Speedup1.00×1.00×1.00×1.00×1.00×1.00×
INT4 Kangarootok/s2.973.223.242.993.833.25
Speedup1.96×2.11×2.13×1.97×2.52×2.14×
INT4 CATStok/s4.274.454.684.145.134.54
Speedup2.81×2.92×3.08×2.72×3.38×2.98×
Vicuna-13B (INT4)
INT4 ARtok/s0.5460.5460.5460.5460.5460.546
Speedup1.00×1.00×1.00×1.00×1.00×1.00×
INT4 Kangarootok/s1.081.171.081.101.431.17
Speedup1.97×2.14×1.98×2.00×2.61×2.14×
INT4 CATStok/s1.631.721.681.602.041.73
Speedup2.98×3.15×3.07×2.92×3.72×3.17×

AuthorsOpenReview ↗

Rebuttal for reviewer Tz3o(3)

S4: KnapSpec

Thanks for providing this. KnapSpec did not release the official implementation codes, so we didn't do related experiments. We choose aonther open-source work, SpecExec, and conduct additional experiments using their implementation. We use Vicuna-68M (0.14 GB in fp16) as the draft model, which fits within the memory budget of edge devices, and evaluate on 40 nodes and 128 nodes separately.

Table R3. Comparison with SpecExec on Vicuna-7B (MT-bench, greedy decoding).

MethodTree NodesSpeedup
SpecExec404.28×
CATS w/ SpecExec404.85×
SpecExec1284.95×
CATS w/ SpecExec1285.21×

As shown in Table R3, our approach improves end-to-end speedup from 4.28× to 4.85× and from 4.95× to 5.21× under the 40-node and 128-node tree budgets, respectively. These results demonstrate the effectiveness of our cascade and correction mechanism in improving practical decoding efficiency.


Additional Weaknesses

W2: Autoregressive Throughput at the 8 GB Budget

Thanks again for pointing this out, the detailed response can be checked in C2. For the AR's speed you calculate, that is based on a more bandwidth-constrained condition as we presented at Section D.1 in the appendix of our revised paper.


W3: Attribution to Prior Implementations

Thanks for raising these concerns, detailed responses can be reviewed above.


O1: "Model-Agnostic" Claim

We thank the reviewer for pointing this out. We acknowledge that CATS requires training model-specific adapters for different base models, and will clarify this in the revised manuscript. But we want to clarify that we just need to fine-tune a lightweight adapter, rather than the large model. And it can be done at scale by tuning multiple LSVL_{SV} during the development phase before deployment. We appreciate the reviewer's thought, and have included this in the Conclusion and Limitations section of the revised draft.


O2: Gain from the Reduced KL Loss

We thank the reviewer for this observation. Reduced KL Loss is a training optimization we identified while developing the adapters and is our secondary contribution.


O3: Alternative Setups in the Motivation

We thank the reviewer for raising this concern. We want to clarify that quantization is not a cost-free alternative: it may introduce generation quality degradation. And CATS is orthogonal to quantization and can be applied directly to quantized models. We also provide related experiment results for your reference in S3.

We also want to clarify that the major contribution of CATS is that it provides edge devices with the capacity to enable larger-model inference under limited DRAM budgets, which enables them to conduct complex reasoning and solve tasks that a small model cannot when necessary.


O4: SpecExec and Sequoia

We thank for reviewer's suggestions. We provide experiment results tested on SpecExec which can be checked in response for S4.


Reviewer_j5h8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CATS, a self-speculative decoding framework for memory-limited LLM inference with flash-offloaded model weights. CATS combines a shallow autoregressive drafter, an intermediate-layer verifier that proposes correction tokens, and final target-model verification over a tree containing the draft sequence and correction branches. The intermediate verifier reuses layers that fit within the memory budget, aiming to improve accepted length without repeatedly executing a deeper drafter. The paper also introduces top-K distillation for training the adapters.

The evaluation covers several 7B–13B models, five generation benchmarks, emulated resident-memory budgets on a Jetson AGX Orin, and an additional experiment on an Orin Nano. The reported results show improvements over the evaluated self-speculative baselines and approximately 10% additional acceleration when combined with native EAGLE-3.

Critical strengths
  1. A well-motivated allocation of computation under an offloading bottleneck. CATS keeps sequential drafting shallow while using intermediate target layers to propose corrections in parallel. This directly addresses the trade-off between draft quality and repeated drafting cost. Reusing the already-loaded layers for correction processing is a sensible design when flash transfers dominate latency.

  2. Consistent reported improvements across multiple configurations. The evaluation includes several model families, five benchmarks, different resident-weight budgets, and a natively memory-constrained device. The native EAGLE-3 comparison is particularly useful: the reported improvement from 5.55× to 6.13× mean greedy speedup suggests that the cascade can help beyond the authors’ self-speculative baseline implementations. The verifier-depth and correction-position analyses also provide evidence that intermediate verification produces useful candidates.

Critical weaknesses
  1. The reported absolute throughput is difficult to reconcile with the stated flash bandwidth and streamed weight volumes. Appendix D reports approximately 0.275 GB/s direct flash bandwidth and states that each non-resident layer is read for every target verification. At the 2 GB budget, the stated streamed volume is 11.33 GB. For autoregressive decoding, this implies approximately 41.2 seconds per token from flash reads alone, whereas Table 4 reports 0.117 tokens/s, or approximately 8.55 seconds per token. Similarly, at the 6 GB budget, 7.29 GB divided by 0.275 GB/s gives approximately 26.5 seconds, versus approximately 7.94 seconds implied by Table 4. Under the stated measurement model, these quantities do not appear mutually consistent.

The memory-accounting claim also needs stronger evidence. A resident-weight limit enforced in code on a 64 GB device does not establish that total peak memory equals that of target-only inference. Adapters, speculative KV states, activations, and transfer buffers also consume memory. Although the Nano experiment is helpful, it does not provide a complete peak-memory breakdown or comparative baseline evaluation. These issues are consequential because the contribution depends on acceleration under an actual shared-memory constraint. A consistent account of physical bytes read, timing boundaries, and measured peak memory is needed to substantiate the central systems claims.

  1. The novelty positioning and experimental isolation are incomplete. The statement that all existing speculative decoding methods assume memory-resident weights is too broad. SpecExec explicitly addresses speculative decoding with parameter offloading, while Cascade Speculative Drafting studies multilevel speculative execution. These works do not establish that CATS lacks a distinct contribution, but they make it necessary to identify precisely what is new about its shared-layer correction mechanism and memory-aware scheduling.

The comparisons also do not fully isolate the value of correction selection from the benefit of evaluating additional candidates. CATS adds correction branches even in the configuration labeled “chain,” and Appendix F’s chain/tree grouping does not establish equal total candidate-node budgets. The paper should specify actual node counts and compare alternative candidate-selection strategies under matched memory and verification budgets. Combined with the reliance on a modified EAGLE implementation for many comparisons, this leaves uncertainty about how much improvement is attributable specifically to the proposed cascade.

Questions for authors
  1. Measurement clarification: How can the flash bandwidth and streamed volumes in Appendix D be reconciled with the absolute throughput in Table 4? Please provide, for the same runs, measured physical bytes read per decoding cycle, cycle latency, committed tokens per cycle, and the precise timing boundary. Does every reported throughput measurement include the stated O_DIRECT reads?

  2. Memory and comparison clarification: What are the measured total peak CPU/GPU shared-memory footprints of target-only decoding, CATS, and the strongest baseline under identical context and generation settings? Please include adapters, speculative caches, activations, and staging buffers. For the matched-structure comparisons, are total verification-node counts also matched? If not, would the gains persist under an equal-node budget?

  3. Decoding clarification: Please specify the complete acceptance rule used for the relaxed results and identify the decoding settings behind Table 11. Is target-distribution preservation claimed only for greedy decoding, or also for relaxed acceptance? The MT-Bench and GSM8K results provide useful empirical quality evidence, but they do not establish distributional equivalence. Clarifying the guarantee and the quality-evaluation protocol would affect how I interpret the relaxed-mode speedups.

7 comments
AuthorsOpenReview ↗

Rebuttal for reviewer j5h8

Q1: Measurement Clarification

Thank you for raising this question. We clarify the storage configurations and provide additional measurements to reconcile the reported bandwidth and throughput.

Storage Configuration. Table 7 (now Table 11 in the revised paper) reports the measured bandwidth of the AGX Orin's internal HDD, illustrating a more bandwidth-constrained scenario where CATS can potentially offer greater benefits. However, due to limited onboard storage capacity, our actual experiments use an external SSD to store model weights and adapters, which is a more practical setting.

We emphasize that the throughput results in Table 4 were directly measured under the actual experimental setup, rather than analytically estimated.

Additional Measurements. To further address reviewer's concern, we provide detailed profiling results from the same runs, including physical bytes read, cycle latency, committed tokens, and throughput. To further optimize performance, we optimized the implementation at both the algorithmic and system levels, including removing redundant parameter streaming and adopting direct flash-to-GPU streaming with pinned memory, and report the related results in the table below.

Table R1. Measured decoding performance on AGX Orin. Physical reads and cycle latency are measured from the same runs. Throughput includes all O_DIRECT reads and decoding operations, excluding prefill.

MethodRead (GB)Cycle (s)τ\tautok/s
8 GB (15 resident layers)
AR6.881.5861.000.631
Kangaroo6.881.6892.491.472
CATS6.881.7853.051.709
6 GB (10 resident layers)
AR8.902.0151.000.496
Kangaroo8.902.1112.491.178
CATS8.902.1762.941.351
2 GB (5 resident layers)
AR10.932.4421.000.410
Kangaroo10.932.5422.490.978
CATS10.932.5762.801.087

Timing and I/O Accounting. Each decoding cycle starts after the previous cycle ends and ends after token commitment and KV-cache updates. All O_DIRECT reads, host-to-GPU transfers, and decoding operations are included in the measured latency. Throughput is computed over the complete decoding interval, with prefill and initialization excluded. The measured SSD read times are 1.24/1.60/1.97 s per cycle at 8/6/2 GB model weights keeping resident in DRAM, corresponding to approximately 5.55 GB/s for the bandwith. Physical SSD reads were independently verified using /proc/diskstats, confirming that all non-resident model layers were read from storage during each decoding cycle.


AuthorsOpenReview ↗

Rebuttal for reviewer j5h8(2)

Q2: Memory and Comparison Clarification

Thank you for raising these concerns. We clarify the peak-memory accounting and the fairness of our verification comparison.

Memory Budget and Measurement. The 8/6/2 GB settings refer to decoder-layer residency budgets rather than total system memory limits. For Vicuna-7B, these configurations retain the first R=15/10/5R=15/10/5 decoder layers in DRAM, respectively. All methods use the same resident-layer configurations. Since Jetson AGX Orin uses unified CPU–GPU memory, we measure peak system memory relative to the pre-launch baseline. All measurements use identical prompts, greedy decoding, and tested on MT-bench.

Memory Breakdown. Beyond the resident decoder layers, all methods retain the embedding and LM head (0.524 GB), staging buffers (0.945 GB), and runtime allocations (2.233 GB). Kangaroo introduces a 0.134 GB adapter, while CATS uses two adapters totaling 0.268 GB. The remaining memory consists of KV caches, temporary activations, and allocator buffers. The staging buffers are reused across layers, avoiding additional persistent copies of non-resident weights. The component breakdown accounts for the measured peaks within 0.05 GB.

Table R2. Peak shared CPU–GPU memory (GB) on AGX Orin.

BudgetTarget-onlyKangarooCATS
8 GB (R=15R=15)10.3410.3710.71
6 GB (R=10R=10)8.278.338.62
2 GB (R=5R=5)6.246.276.62

As shown in Table R2, CATS introduces only 0.35–0.38 GB of additional peak memory compared with target-only decoding, including its 0.268 GB adapters. Kangaroo adds 0.03–0.06 GB. These results demonstrate that the proposed cascade incurs modest memory overhead across different DRAM budgets.

Comparison with EAGLE-3. For the EAGLE-3 experiments based on Llama-3.1-8B, we allocate approximately 12 GB of DRAM for resident model weights, including an independent 1.901 GB drafter, with R=16R=16 target decoder layers kept resident. We apply the same memory accounting to EAGLE-3 and CATS w/ EAGLE-3.

Table R3. Peak shared CPU–GPU memory (GB) on AGX Orin for Llama-3.1-8B.

Memory ComponentEAGLE-3CATS w/ EAGLE-3
Model and buffers (excl. drafter)10.32310.411
Independent drafter1.9011.901
Runtime/allocator (residual)3.1523.069
Measured peak15.37615.381

As shown in Table R3, EAGLE-3 and CATS w/ EAGLE-3 exhibit nearly identical peak memory usage (15.376 vs. 15.381 GB), with a measured difference of only 0.005 GB.

Verification Fairness. Our comparison controls the number of target-model verification forward passes rather than requiring identical verification-node counts. The additional verification nodes arise from the proposed cascade design and constitute part of our algorithmic advantage.


AuthorsOpenReview ↗

Rebuttal for reviewer j5h8(3)

Q3: Decoding Clarification

Thank you for raising these questions. For relaxed decoding, we follow the typical acceptance rule introduced by Medusa, a widely adopted approach in speculative decoding, while our exact output equivalence guarantee applies only to greedy decoding. Following Medusa, a draft token xix_i is accepted if pT(xi∣x<i)>min⁡(ϵ,δ e−H(pT))p_T(x_i\mid x_{<i})>\min(\epsilon,\delta\,e^{-H(p_T)}), where pTp_T is the target distribution at T=0.7T=0.7, HH is its entropy, ϵ=0.3\epsilon=0.3, and δ=0.09\delta=0.09. We commit the longest accepted prefix across all verification-tree paths, followed by the target's top-1 token.

For Table 11 (now Table 15 in the revised paper) Settings, both rows use greedy decoding in FP16 on Llama-3-8B-Instruct. For the scores, for MT-bench, we employee GPT-4o for evaluation using FastChat's single-answer mode following MT-bench's official scripts. GSM8K uses the first 300 test examples with exact-match evaluation.

Empirical Effect of Relaxed Decoding. To quantify the impact of relaxed acceptance, we evaluate the generated tokens against the target distribution on Vicuna-7B using MT-bench.

Table R4. Effect of relaxed acceptance on Vicuna-7B. Metrics are computed by teacher-forcing generated outputs through the target model.

DecoderNon-Greedy Tokens (%)Outside Top-2 (%)Mean Log-Prob. Gap (nats)Target PPL
Target (greedy)0.030.000.0001.155
CATS (greedy)0.020.000.0001.156
CATS (relaxed)6.922.150.0671.222

In the table, non-Greedy and Outside Top-2 measure token deviations from the target's top-1 and top-2 predictions, respectively. Mean Gap is the average log-probability gap to the top-1 token, and PPL is the perplexity under the target model. As shown in Table R4, 93.08% of committed tokens under relaxed decoding remain the target's top choice under their respective prefixes. The mean log-probability gap is only 0.067 nats, with target perplexity increasing slightly from 1.156 to 1.222. These results suggest that relaxed decoding introduces only limited deviations from the target's greedy decisions.


Additional Weaknesses

W2a: Novelty Positioning

Thank you for highlighting these related works. After reading the papers, we notice that SpecExec focuses on efficient speculative tree construction for parameter-offloaded inference, with its main experiments using RAM offloading. Cascade Speculative Drafting primarily optimizes multi-level drafting efficiency. In contrast, CATS specifically addresses flash-to-DRAM weight streaming under strict edge-memory constraints, where minimizing expensive weight transfers is critical.

The key novelty of CATS lies in its memory-aware cascade and shallow-verification correction mechanism. By exploiting the intermediate layers that fit within the DRAM budget, our cascade can provide additional correction branches, enabling more candidate nodes for target verification without additional flash weight transfers and forward passes. This allows CATS to better utilize limited resident memory.

To further evaluate our approach, we conduct additional experiments using the official SpecExec implementation. We use Vicuna-68M (0.14 GB in fp16) as the draft model, which fits within the memory budget of edge devices, and evaluate on 40 nodes and 128 nodes separately.

Table R5. Comparison with SpecExec on Vicuna-7B (MT-bench, greedy decoding).

MethodTree NodesSpeedup
SpecExec404.28×
CATS w/ SpecExec404.85×
SpecExec1284.95×
CATS w/ SpecExec1285.21×

As shown in Table R5, our approach improves end-to-end speedup from 4.28× to 4.85× and from 4.95× to 5.21× under the 40-node and 128-node tree budgets, respectively. These results demonstrate the effectiveness of our cascade and correction mechanism in improving practical decoding efficiency.


W2b: Isolating Correction Selection from Extra Candidates

Thank you for raising this concern. We would like to clarify that achieving effective acceleration requires more than the basic idea of cascading. The generation and selection of correction branches are integral parts of our proposed design.

In our design, re-forwarding correction tokens through the resident layers does not require extra flash-to-DRAM weight transfers in our memory-bound setting. Our comparison controls the number of target-model verification forward passes rather than requiring identical verification-node counts. The additional verification nodes arise from the proposed cascade design and constitute part of our algorithmic advantage.

We also want to clarify that CATS is independent of EAGLE. We integrate our cascade into EAGLE's established implementation solely to enable fair comparisons under tree-based speculative decoding.

Reviewer_95Q1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CATS, a cascaded self-speculative decoding framework for memory-limited LLM inference. It combines lightweight drafting, shallow verification, and tree-based target verification to reduce costly Flash-to-DRAM weight transfers. Experiments across multiple models and benchmarks demonstrate substantial relative inference speedups.

Critical strengths
  1. Practical Motivation: The paper identifies the Flash-to-DRAM bottleneck in memory-constrained LLM inference and supports its motivation with hardware profiling.

  2. Effective Framework Design: The three-stage cascaded verification mechanism improves token acceptance while avoiding repeated execution of intermediate layers during drafting.

  3. Consistent Speedup Improvements: CATS achieves promising relative speedups across multiple models, benchmarks, and decoding configurations, with additional validation on edge devices.

Critical weaknesses
  1. Extremely Low Absolute Throughput: Despite the reported speedups, CATS achieves only 0.329–0.461 tokens/s under different memory budgets (Table 4), and 0.41 tokens/s on Jetson Orin Nano (Table 9). Such low throughput severely limits its practical usefulness for interactive LLM applications.

  2. Limited Practical Comparisons: The evaluation mainly focuses on FP16 models with frequent weight offloading. More comprehensive throughput comparisons against optimized quantized inference are needed to justify the practical advantages of CATS.

Questions for authors
  1. Given that CATS achieves less than one token per second on the evaluated edge devices, what practical deployment scenarios would benefit from such low absolute throughput?

  2. How does CATS compare with optimized quantized inference in terms of absolute tokens/s under the same memory constraints?

  3. Many recent LLMs adopt Mixture-of-Experts (MoE) architectures, where sparse expert activation leads to different memory-access patterns compared to dense models. Can CATS achieve similar inference speedups on MoE models, particularly under memory-constrained settings?

4 comments
AuthorsOpenReview ↗

Rebuttal for reviewer 95Q1

Q1: Practical Scenarios for Sub-1 Token/s Throughput

Thank you for raising this question. We acknowledge that the current absolute throughput reported in the paper is limited by the low flash-to-DRAM bandwidth of edge devices. But, as hardware advances and storage bandwidth improves, absolute decoding throughput is expected to increase. And as long as the hierarchical memory architecture exists on edge devices, CATS can continue to provide acceleration by reducing the amortized weight-loading cost per accepted token.

The hierarchical memory architecture is also the motivation of related works in industry, Apple's LLM in a Flash (Alizadeh et al., ACL 2024) investigates inference for models exceeding available DRAM, while Qualcomm AI Research (Federici et al., MLSys 2025) explicitly considers flash streaming for a 14B model.

The major contribution of CATS is that it provides edge devices with the capacity to enable larger-model inference under limited DRAM budgets, which enables them to conduct complex reasoning and solve tasks that a small model cannot when necessary.

We appreciate your valuable question, and we went back to check and optimize the implementation code, including removing redundant parameter streaming and adopting direct flash-to-GPU streaming with pinned memory. Here we provide with new results after optimization for your reference.

Table R1. Measured decoding performance on AGX Orin. Physical reads and cycle latency are measured from the same runs. Throughput includes all O_DIRECT reads and decoding operations, excluding prefill.

MethodRead (GB)Cycle (s)τ\tautok/s
8 GB (15 resident layers)
AR6.881.5861.000.631
Kangaroo6.881.6892.491.472
CATS6.881.7853.051.709
6 GB (10 resident layers)
AR8.902.0151.000.496
Kangaroo8.902.1112.491.178
CATS8.902.1762.941.351
2 GB (5 resident layers)
AR10.932.4421.000.410
Kangaroo10.932.5422.490.978
CATS10.932.5762.801.087

Q2: Comparison with Optimized Quantized Inference

We appreciate the reviewer's thought about quantization. We would like to clarify that quantization is not a cost-free alternative: it may introduce generation quality degradation, and our method is quality-lossless. We provide the evaluated generation quality of using quantization below:

Table R2. Generation quality and memory footprint of CATS and quantized inference on Llama-3-8B.

MethodMemory (GiB)MT-benchGSM8K
Llama-3-8B FP1614.967.28178.33%
Llama-3-8B NF4 (4-bit)5.216.85676.00%
CATS (Llama-3-8B)14.967.33879.33%

As shown in Table R2, 4-bit NF4 quantization reduces Llama-3-8B's MT-bench score from 7.281 to 6.856 and GSM8K accuracy from 78.33% to 76.00%, whereas CATS maintains comparable or better generation quality than the FP16 baseline.

In addition, our method is orthogonal to quantization, and the two methods can be combined. In particular, a quantized model that still exceeds the resident budget must still stream non-resident layers; our method can reduce how often those layers are reread.

We additionally evaluate CATS with INT4 quantization under 2GB memory budget. As shown in Table R3, INT4 CATS achieves 4.54 tok/s on Vicuna-7B and 1.73 tok/s on Vicuna-13B, corresponding to 2.98× and 3.17× end-to-end speedups over INT4 autoregressive decoding. Compared with INT4 Kangaroo, CATS further improves throughput by 1.40× and 1.48×, respectively. These results demonstrate that CATS remains effective when combined with quantization.

Table R3. Absolute throughput and end-to-end speedup of INT4 methods under 2GB memory budget. Speedups are measured against INT4 autoregressive decoding.

MethodMetricSpecMTGSM8KAlpacaHumanEvalMean
Vicuna-7B (INT4)
INT4 ARtok/s1.521.521.521.521.521.52
Speedup1.00×1.00×1.00×1.00×1.00×1.00×
INT4 Kangarootok/s2.973.223.242.993.833.25
Speedup1.96×2.11×2.13×1.97×2.52×2.14×
INT4 CATStok/s4.274.454.684.145.134.54
Speedup2.81×2.92×3.08×2.72×3.38×2.98×
Vicuna-13B (INT4)
INT4 ARtok/s0.5460.5460.5460.5460.5460.546
Speedup1.00×1.00×1.00×1.00×1.00×1.00×
INT4 Kangarootok/s1.081.171.081.101.431.17
Speedup1.97×2.14×1.98×2.00×2.61×2.14×
INT4 CATStok/s1.631.721.681.602.041.73
Speedup2.98×3.15×3.07×2.92×3.72×3.17×

AuthorsOpenReview ↗

Rebuttal for reviewer 95Q1(2)

Q3: Mixture-of-Experts Models

Thanks for raising this concern. Our cascade design can be naturally extended to Mixture-of-Experts (MoE) architectures. We additionally evaluate CATS on OLMoE-1B-7B-0125-Instruct to examine its effectiveness on Mixture-of-Experts architectures.

Table R4. End-to-end speedup over autoregressive decoding on OLMoE-1B-7B-0125-Instruct. CATS uses an SV at layer 8.

MethodMT-benchSpec-bench
Greedy Decoding
Kangaroo2.01×1.93×
CATS2.30×2.25×
Relaxed Decoding (T=0.7T=0.7)
Kangaroo2.47×2.32×
CATS2.90×2.77×

As shown in Table R4, CATS consistently outperforms Kangaroo, improving end-to-end speedup by 14.4% and 16.6% on MT-bench and Spec-bench under greedy decoding, and by 17.4% and 19.4% under relaxed decoding, respectively. These results demonstrate that CATS also provides effective acceleration on MoE models.

I appreciate the authors' detailed responses and additional experiments, which have addressed our main concerns, therefore I raise my rating from 2 to 3.

We sincerely thank the reviewer for their constructive feedback and for reconsidering the rating. We appreciate the time and effort devoted to reviewing our work.

Reviewer_LAbEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses inference acceleration for large models on memory-constrained edge devices, proposing a three-stage cascaded self-speculative decoding paradigm.

When the device DRAM cannot hold a 7B model's weights, every generation step must stream all weights from slower storage; the paper measures this read at 98.9% of the layer-loading-plus-compute chain, so per-token latency is set by streamed bytes over bandwidth, amortized by the mean accepted length. The method is a three-stage cascaded self-speculative scheme: the first few layers plus a lightweight adapter draft sequentially (DRAM-resident, adding no reads from slower storage); the middle layers serve as a shallow verifier that checks the draft in parallel and, at mismatched positions, emits correction candidates attached as siblings at the same trunk positions, forming a tree with the trunk; the remaining layers are streamed from slower storage once per cycle, and a single batched tree forward performs the final verification, lossless under greedy matching. The shallow verifier only decides which candidates are worth entering the tree; every emitted token is still verified by the target model itself. A per-token byte analysis shows that the cascade outperforms deepening the draft exit.

Critical strengths

It decouples "improving draft quality" from "adding streamed bytes": the shallow verifier runs on resident layers and the correction candidates are verified together within the target's single batched tree forward, so every mechanism that raises the accepted length adds no reads from slower hardware; correctness is structurally decoupled from the verifier's accuracy (every emitted token is still verified by the target model itself, strictly lossless under greedy decoding); there is no separate drafter and no extra heads, the additional resident overhead is minimal, and adapted EAGLE-family drafting structures plug into the framework for further gains — the first-generation EAGLE re-shaped into a self-speculative structure, EAGLE-3 attached as an external drafter, both yielding gains after adaptation.

Critical weaknesses
  1. The method's applicability is confined to the speculative-decoding paradigm (its main body is self-speculative) and constrained by the hardware configuration (it presumes the measured storage-read ratio).

The method has a limited scope of applicability: its gains are tied to the specific combination of self-speculation and very slow flash storage.

Paradigm: the main body is self-speculative; an independent drafter can only be an add-on. The three stages are stopping points of one forward pass at three depths, sharing a single hidden-state chain: layers 1-3 feed layers 4-15 (the SV stage), and layer 15 feeds the streamed layers 16-32 (the target stage). This chain exists only by reusing the target model itself; an independent drafter's hidden states live in a different representation space and cannot enter it, so the drafter cannot replace the cascade proper and can only replace the first stage as an external component — which is exactly what the paper does, as CATS w/ EAGLE-3 still gains over native EAGLE-3 under the same budget. But the add-on drafter's weights either crowd the resident budget or stream from flash themselves, and the paper does not decompose the CATS w/ EAGLE-3 gain into "saved drafter streaming cost" versus "the cascade itself"; the first-generation EAGLE baseline never appears in native form in the results tables — it was re-implemented as EAGLE-self (a self-speculative structure); the only native independent-drafter comparison is the EAGLE-3 group on Llama-3.1. Training-free drafters (REST/Lookahead) carry no weight overhead but yield low accepted lengths.

Hardware: the gains depend on the measured regime of extremely low flash bandwidth. The premise that added computation costs no time holds only while flash reads dominate weight movement (measured at 98.9%); Eq. 4 itself carries the "Because flash dominates" qualifier — with faster storage or slower compute, the added computation (the SV forward, wider trees, re-forwards after mismatches) enters the latency. The paper gives no end-to-end numbers for CATS on weight-resident hardware; the B200 comparison only plots baseline methods dropping between the two regimes. The two devices cover only about a tenfold bandwidth range from eMMC to NVMe, and whether the advantage survives on faster storage is untested. Absolute throughput remains set by storage bandwidth: CATS amplifies the constrained rate by the accepted length without removing the constraint; once the model is quantized or swapped for one small enough to fit entirely, the streaming problem disappears and the method loses its scenario (Appendix G, same-budget comparison). L_SV is hand-picked per budget; changing devices or models requires retuning.

  1. Moreover, all experiments are on LLaMA-family decoder-only models.

Whether the two premises — that a layer-3 early exit can be trained into a competent draft, and that layer-15 states can predict the final-layer output — hold for other models is only briefly discussed. If a supplement is possible, could the authors add generalization experiments on transformer models with other attention mechanisms?

  1. Baseline configurations are under-disclosed.
  • Medusa is configured with only 2 heads, far below its default 4-5; on LLaMA2-7B its accepted length is only 1.67-1.97 (1.78x), clearly below the method's usual performance; the rationale for the head count, whether the heads were retrained per model, and the training configuration are all undisclosed.
  • EAGLE is re-implemented as "EAGLE-self" (a self-speculative structure, drafting steps=5) rather than the native feature-level independent drafter. In the EAGLE-3 comparison, how much of native EAGLE-3's 5.55x is drafter streaming cost, and how the CATS w/ EAGLE-3 gain splits between "saved drafter streaming" and "the cascade itself", are both unreported.
Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Rebuttal for reviewer LAbE

W1a: Applicability: Self-Speculative Paradigm

Effectiveness across drafting strategies. Our method is not limited to the self-speculation setting. We focus on the self-speculative decoding setting in our paper because we want to minimize the total memory cost for edge devices. The improvement comes from utilizing the intermediate layers for pre-verification in our cascade design, rather than eliminating the independent drafter. On Llama-3-8B, CATS improves the five-benchmark mean greedy speedup from 2.27× to 2.98× over Kangaroo and from 2.90× to 3.33× over EAGLE-self (Table 3 in paper). For CATS w/ EAGLE-3, our implementation retains the original EAGLE-3 drafter. With the independent EAGLE-3 drafter on Llama-3.1-8B, it further improves speedup from 5.55× to 6.13×. These results demonstrate that the benefits of the cascade extend across different drafting strategies. We appreciate the reviewer's thoughts, and conduct experiments on native first-generation EAGLE using LLaMA-3.1-8B to further prove the improvement on independent drafter settings.

Table R1. End-to-end speedup over greedy decoding of native EAGLE-1 and CATS w/ EAGLE-1 across five benchmarks.

MethodSpecMTGSM8KAlpacaHumanEvalMean
Native EAGLE-12.90×2.98×3.13×2.82×3.46×3.06×
CATS w/ EAGLE-13.16×3.31×3.40×3.02×4.20×3.42×

As shown in Table R1, CATS w/ EAGLE-1 outperforms native EAGLE-1 across all five benchmarks, improving the end-to-end speedup from 3.06× to 3.42× (+11.8%) under greedy decoding. These results, together with the CATS w/ EAGLE-3 results, demonstrate that CATS also improves speculative decoding with an independent drafter.

Training-free drafters. We included training-free drafters as baselines to compare with the methods that are efficient in terms of weight overhead. We agree that there will be a trade-off between the weight overhead and drafter performance. We have included this discussion in the revised paper.


W1b: Applicability: Hardware Regime

Thank you for raising this concern. We would like to clarify that CATS specifically targets memory-constrained edge devices where model weights must be streamed from flash to DRAM. This is a practical deployment scenario actively studied in both academia and industry. Apple's LLM in a Flash (Alizadeh et al., ACL 2024) investigates inference for models exceeding available DRAM, while Qualcomm AI Research (Federici et al., MLSys 2025) explicitly considers flash streaming for a 14B model. These studies demonstrate the practical relevance of the memory-constrained setting addressed by CATS. It is a common characteristic of edge devices that flash-to-DRAM weight transfers dominate latency, which is the motivation for both works above and ours. We also believe that future devices will have faster storage, but as long as the memory hierarchy remains, which is most likely to be true, these methods, including ours, will remain applicable.

We appreciate the reviewer's thought about quantization. We would like to clarify that quantization is not a cost-free alternative: it may introduce generation quality degradation, and our method is quality-lossless. We provide the evaluated generation quality of using quantization below:

Table R2. Generation quality and memory footprint of CATS and quantized inference on Llama-3-8B.

MethodMemory (GiB)MT-benchGSM8K
Llama-3-8B FP1614.967.28178.33%
Llama-3-8B NF4 (4-bit)5.216.85676.00%
CATS (Llama-3-8B)14.967.33879.33%

As shown in Table R2, 4-bit NF4 quantization reduces Llama-3-8B's MT-bench score from 7.281 to 6.856 and GSM8K accuracy from 78.33% to 76.00%, whereas CATS maintains comparable or better generation quality than the FP16 baseline.

In addition, our method is orthogonal to quantization, and the two methods can be combined. In particular, a quantized model that still exceeds the resident budget must still stream non-resident layers; our method can reduce how often those layers are reread.

It is true that LSVL_{SV} is derived and needs retuning for different devices. But we want to clarify that we just need to fine-tune a lightweight adapter, rather than the large model. And it can be done at scale by tuning multiple LSVL_{SV} during the development phase before deployment. We appreciate the reviewer's thought, and have included this in the Conclusion and Limitations section of the revised draft.


AuthorsOpenReview ↗

Rebuttal for reviewer LAbE(2)

W2: Generalization Beyond LLaMA-Family Models

We further evaluate CATS on Gemma-2-9B-it under the same experimental settings as Table 1. As shown in Table R3, CATS consistently outperforms Kangaroo across all five benchmarks, improving the mean end-to-end speedup from 1.83× to 2.14× under greedy decoding and from 2.00× to 2.36× under relaxed decoding. These results further demonstrate the effectiveness of CATS across different model families.

Table R3. End-to-end speedup over autoregressive decoding on Gemma-2-9B-it. Experimental settings are identical to those in Table 1 of the main paper.

MethodSpec-benchMT-benchGSM8KAlpacaHumanEvalMean
Greedy Decoding
Kangaroo1.71×1.77×1.69×1.70×2.26×1.83×
CATS2.00×2.04×2.10×1.93×2.64×2.14×
Relaxed Decoding (T=0.7T=0.7)
Kangaroo1.90×2.00×1.80×1.92×2.37×2.00×
CATS2.25×2.32×2.23×2.22×2.78×2.36×

W3a: Baseline Configuration: Medusa

We thank the reviewer for pointing this out. After carefully checking our experimental settings, we confirmed that all experiments used 5 heads, and the reference to 2 heads in the table caption was outdated. We have corrected the caption and rerun the experiments on both Vicuna-7B and Vicuna-13B to further verify the reliability of our results, as shown in Table R4.

Table R4. Mean accepted length (τ\tau) of Medusa with the released Medusa heads for Vicuna-7B/13B v1.3 (5 heads) and the official sparse trees. Relaxed decoding uses Medusa's typical acceptance.

ModelSpec-benchMT-benchGSM8KAlpacaHumanEvalMean
Greedy
Vicuna-7b2.31292.51332.49402.39262.67762.4781
Vicuna-13b2.39532.58832.58332.39682.80712.5542
Relaxed (temperature = 0.7)
Vicuna-7b2.48372.67952.68752.62382.87062.6690
Vicuna-13b2.54322.73072.75072.63892.96122.7249

Compared with the Medusa results reported in Tables 1 and 2 of our paper, the mean accepted lengths differ by at most 0.21% under greedy decoding and 2.8% under relaxed decoding, confirming that our original results are trustworthy.

For reviewer's question on LLaMA2-7B, the official Medusa implementation does not report results on LLaMA2-7B, nor does it release the corresponding model weights. We trained our own heads following the official training code and procedure and got the results.


W3b: Baseline Configuration: EAGLE and EAGLE-3

Thank you for the question. The main response to this question is included in the response to Weakness 1-a.

More questions:

  • In your memory accounting (response to Reviewer j5h8), the independent drafter is resident in both native EAGLE-3 and CATS w/ EAGLE-3 (1.901 GB) — do the two peak-memory figures differ by only 0.005 GB? If so, this means the gain of CATS in this comparison cannot come from "saving the drafter's streaming reads"; it can only come from the cascade's own candidate expansion.
  • Is it the case that the main experiments actually ran on an external SSD (measured at about 5.55 GB/s), while the 0.275 GB/s figure corresponds to a different internal storage device? Are there more details of the experimental setup to be clarified?
  • After stacking NF4 or INT4, the target model itself has degraded accuracy. Does the basic logic of "quantization is lossy, CATS is lossless" therefore need a more precise formulation?

Thank you for your follow-up questions. We are happy to further clarify these points.

1. Independent drafter and cascade gains.

Your understanding is correct. The experiment provided in our response to Reviewer j5h8 is intended to demonstrate exactly this point: the additional acceleration comes from our cascade design rather than eliminating the drafter's weight transfers.

We also want to emphasize that our main contribution is the cascaded verification architecture, which leverages resident intermediate layers to improve acceptance performance without additional target-model weight streaming, regardless of the underlying drafting strategy. We primarily study self-speculative decoding to further minimize weight overhead and movement on memory-constrained devices.

2. Storage bandwidth and experimental setup.

Yes, the main experiments were conducted using an external SSD. The 0.275 GB/s measurement was included to illustrate a more extreme storage-bandwidth-constrained scenario. We have explicitly clarified this distinction in Appendix D.1, and we have updated the description of our experimental setup in the revised manuscript.

3. Losslessness and quantization.

We agree that a more precise formulation would be helpful. Quantization modifies the target model's weight values and may therefore degrade its generation quality. However, CATS accelerates inference without modifying the target model itself.

Following the principle of speculative decoding [1, 2], CATS preserves token-level output equivalence with autoregressive decoding under greedy decoding. This guarantee holds with respect to the underlying target model, whether quantized or not.

When combined with quantization, any quality degradation attributable to quantization originates from the quantized target model rather than CATS.

[1] Leviathan Y, Kalman M, Matias Y. Fast inference from transformers via speculative decoding[C]//International conference on machine learning. PMLR, 2023: 19274-19286.

[2] Chen C, Borgeaud S, Irving G, et al. Accelerating large language model decoding with speculative sampling[J]. arXiv preprint arXiv:2302.01318, 2023.

Reviewer_hNHEOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CATS (Cascaded Adaptive Tree Speculation), a self-speculative decoding framework designed to resolve the memory bandwidth bottleneck of Large Language Model (LLM) inference on memory-constrained edge platforms. Existing methods assume high-bandwidth memory (HBM) can simultaneously hold both target and draft models, but this paper identifies that flash-to-DRAM data movement is the actual bottleneck on edge devices. CATS addresses this by dividing inference into three stages based on the available DRAM budget: 1) iterative drafting via a shallow sub-network; 2) shallow verification using intermediate layers streamed from flash to generate correction candidates in parallel; and 3) target verification by offloading the remaining layers chunk by chunk. Experiments show CATS achieves up to 5.08x end-to-end wall-clock speedup on real edge hardware without generation quality loss, significantly outperforming existing self-speculative decoding baselines.

Critical strengths
  1. The paper accurately identifies that the core bottleneck of speculative decoding on edge devices is the flash↔DRAM data movement, rather than DRAM-to-chip transfer, effectively breaking the ideal memory-resident assumption of previous methods.

  2. The proposed three-stage architecture is highly ingenious. By dynamically setting the drafting boundary and shallow verification boundary, it effectively utilizes the available DRAM budget to increase the parallel verification candidate set, maximizing throughput without increasing the device's peak memory footprint.

Critical weaknesses

The current experiments primarily focus on models with 7B to 13B parameters. For larger models, the effectiveness of the Shallow Verifier has yet to be fully verified.

Questions for authors
  1. If CATS is applied to models with larger parameter scales (e.g., 30B or 70B), the proportion of intermediate layers covered by the shallow verifier will significantly drop under the same minimal edge DRAM budget. Will this lead to a significant decay in the acceptance rate?

  2. The Reduced KL Loss proposed in the paper focuses distillation supervision on the top-K high-probability tokens of the target distribution. Is the choice of the K value highly sensitive for different tasks or models of different scales?

  3. How do speed and peak memory change at longer contexts, where the KV cache is no longer negligible?

2 comments
AuthorsOpenReview ↗

Rebuttal for reviewer hNHE

Q1: Larger Models (30B / 70B)

Thank you for raising this important scalability question. We agree that, under a fixed DRAM budget, the shallow verifier (SV) covers a smaller fraction of the network as model size increases. However, this does not necessarily imply a proportional degradation in acceptance performance. We investigate this through the following experiments.

Sensitivity to SV depth. We first vary the SV depth on Vicuna-7B while keeping the drafter, tree budget, and training configuration fixed.

Table R1. SV depth ablation on Vicuna-7B (32 layers), MT-bench, greedy decoding. The drafter is fixed at layer 3 with a 40-node tree budget.

SV LayerDepth (%)PrecisionRecallτ\tau
LSV=15L_{SV}=1546.90.9110.9393.05
LSV=10L_{SV}=1031.30.8650.9082.94
LSV=5L_{SV}=515.60.8090.8832.80

As shown in Table R1, reducing the SV coverage from 46.9% to 15.6% decreases the average accepted length τ\tau by only 8.2% (from 3.05 to 2.80). Precision and recall also decline gradually, suggesting that CATS maintains relatively stable acceptance performance even with substantially reduced SV depth.

To discuss scaling to larger models, we have been running experiments on Vicuna-33B v1.3 with 60 layers. We will later provide with related results.


Q2: Sensitivity to KK in the Reduced KL Loss

Thank you for raising this question. We investigate the sensitivity of the top-KK parameter in Reduced KL Loss by comparing Vicuna-7B adapters trained with K=10K=10 and K=20K=20, keeping all other training configurations fixed. We evaluate both adapters across five benchmarks under identical inference settings.

Table R3. Effect of KK in the Reduced KL Loss on Vicuna-7B, greedy decoding, mean accepted length τ\tau.

Drafter KKSpec-benchMT-benchGSM8KAlpacaHumanEvalMean
K=10K=103.5293.7103.7313.4524.2053.725
K=20K=203.5313.7323.7113.4474.2113.726

As shown in Table R3, the average accepted length τ\tau differs by at most 0.6% across all five benchmarks, with nearly identical overall averages (3.725 vs. 3.726). These results suggest that Reduced KL Loss is robust to the choice of KK across different tasks, without requiring task-specific tuning.


Q3: Speed and Peak Memory at Longer Contexts

Thank you for raising this question. In our experiments, we use a 2K-token context window for evaluated models, which is sufficient for our evaluated generation tasks. To examine longer contexts, we additionally analyze the KV-cache memory overhead and decoding latency under our edge-device bandwidths.

Table R4. FP16 KV-cache memory at different context lengths.

ModelContextTarget-only KV (GB)CATS Extra KV (GB)
Vicuna-7B2K1.070.07
Llama-3-8B2K0.270.02
Llama-3-8B8K1.070.07
Llama-3.1-8B2K0.270.02
Llama-3.1-8B32K4.290.27

In Table R4, Target-only KV denotes the KV-cache memory required by the target model alone, while CATS Extra KV denotes the additional KV-cache memory introduced by CATS. As shown in Table R4, CATS introduces only 1/161/16 of the target-only KV-cache memory. Even at 32K tokens on Llama-3.1-8B, this amounts to just 0.27 GB compared to 4.29 GB for target-only decoding. The 40-node verification tree adds at most 20 MB. Our DRAM allocation already accommodates KV-cache growth, and the analytical estimates agree with on-device measurements.

We now provide the experimental results on Vicuna-33B v1.3 (60 layers).

Table R2. Accepted length on Vicuna-33B, MT-bench, greedy decoding. Both methods use the same layer-3 drafter.

MethodSV LayerSV Coverage (%)τ\tau
Kangaroo----2.26
CATS58.32.63
CATS1525.02.86

As shown in Table R2, even with SV coverage of only 8.3% and 25.0%, CATS improves τ\tau over Kangaroo by 16.4% and 26.5%, respectively. These results demonstrate that CATS retains substantial acceptance gains when scaling to larger models, even under severely limited SV coverage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.