- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CatchUp, a prefix-cache recovery method for hybrid LLMs. It restores a recurrent checkpoint, replays a fraction of the checkpoint gap only through the first recurrent group, and reuses cached full-attention KV states while retaining deeper recurrent states. Experiments on three hybrid models show substantial reductions in time-to-first-token compared with full-model recovery. The authors also evaluate storage overhead and task quality on LongBench, BFCL, and RULER.
Critical strengths
-
CatchUp aims to address a practical recovery bottleneck in prefix caching for hybrid large language models. Compared to full-model recovery, this approach significantly reduces latency and eliminates the need to store additional per-token context inputs for deeper recurrent groups.
-
The evaluation covers three hybrid models across various downstream tasks. Furthermore, the paper analyzes storage overhead and sensitivity to the "replay budget," and includes "no-replay" ablation experiments, providing valuable insights into the trade-off between latency and quality.
Critical weaknesses
-
The benefit of replay itself is not convincingly established. In Table 6, at an 8K gap and 10% replay, CatchUp improves over Stale on Qwen3.5-4B but performs worse on OLMo-Hybrid-7B and Qwen3.6-27B. Table 7 shows positive confidence intervals excluding zero in only 3 of the 60 configurations, while all 27 RULER comparisons in Table 16 have intervals including zero. Since replay adds computation, its advantage over simply reusing the checkpoint remains unclear.
-
The comparison with Tail-Replay does not fully isolate the source of the improvement. CatchUp replays 10% of the checkpoint gap, whereas Tail-Replay uses 10% of the matched prefix and processes more recurrent groups. The implementation also excludes Tail-Replay's original transfer-overlap optimization. The reported speedups are therefore specific to these different recovery and execution settings. A comparison under matched computational or latency budgets would better establish the relative benefit of CatchUp.
Questions for authors
-
Given that "first-group replay" offers only limited and inconsistent improvements over the "stale state" approach, under what conditions is the use of "first-group replay" actually beneficial? Could the authors specify the configurations in which the quality gains are sufficient to offset the additional latency incurred?
-
How does CatchUp perform compared to "tail-replay" when recovery costs or latency budgets are comparable? If the "transfer-overlap" optimization—a native feature of "tail-replay"—is enabled, do the claimed advantages of CatchUp still hold?
-
Proposition 1 assumes that gates, keys, and values are shared, while the deeper recurrent states remain unchanged within CatchUp. How does this analysis, which relies on specific assumptions, account for the actual performance observed during the full recovery process?
Response to Reviewer 4tTz
Thank you for the questions on replay, baseline fairness, and Proposition 1. CatchUp selectively recovers recurrent states while reusing matched FA KV, without an additional deeper-group input cache.
The original A100/vLLM evaluation reports 4.32–7.94× paired TTFT speedups over full-model recovery (Exact) across three models, at an 8K gap and 10% replay on 28 timing inputs. Quality was evaluated separately on 139 inputs, with differences of -1.83 to +0.14 versus Exact. The supplement tests replay's incremental effect and the FFN-skipping Tail comparison.
Q1. When do quality gains offset replay latency?
The state diagnostic asks whether replay repairs the first group. Median matrix-state relative L2 error against the exact producer state falls from 0.4021 for Stale to 0.1645 and 0.0649 for CatchUp 10% and 50%, improving on all 28 inputs. CatchUp uses token-derived first-group inputs and preserves deeper checkpoint states and matched FA KV. Stale also avoids the extra input cache.
Complete-answer evaluation tests whether this repair improves task scores:
Paired differences versus Stale are -0.69 (95% interval [-1.47, -0.07]) and -0.38 ([-1.11, 0.27]). On this cohort, replay's additional cost is not offset by a higher task score. These tests establish state repair and distinguish it from task-quality gains.
For Qwen3.5-4B, gap 8192, and 10% replay, the original 139-input result was +0.81 versus Stale (paper interval [-0.32, 2.44]); the 28-input timing subset gave -0.11. The new backend gives -0.69 on those 28 inputs. We report these evaluation scopes separately.
The supplement uses Qwen3.5-4B, gap 8192, BF16, RTX 4090, and HF/FLA with original greedy generation and scoring. Supplemental intervals use 10,000 paired context-cluster bootstrap resamples within tasks, equal task weights, and no multiple-comparison correction (28 inputs, 26 contexts). Timings average per-input medians within tasks and then across tasks, with three warmups and five shuffled rounds, including restoration, required transfers, replay, suffix processing, and returning the first token.
Q2. Comparable recovery budgets and transfer overlap
Latency calibration tests budget comparability: CatchUp budgets against the checkpoint gap, whereas Tail budgets against the matched prefix and replays all groups. With Tail's terminal FFN skipped, we tested windows of 16, 32, 64, 128, 256, 512, 1024, 1536, and 2048 tokens on four fixed, context-disjoint calibration inputs, one per task.
Timing-only selection chose 16 tokens as closest to CatchUp 50%, but it remained 15–37% slower. Tail windows 16–1024 differed in latency by less than 1% on each calibration input. After calibration, before inspecting evaluation quality, we retained the primary 16-token selection and added 1024 as the longest window in Tail's near-constant-cost range.
The paired comparison tests this baseline's quality and cost. Tail-16 scored 41.93. In the secondary run, CatchUp 50% scored 48.22 at 107.49 ms versus Tail-1024's 42.71 at 132.09 ms: +5.50 points, with a wide 95% interval [0.22, 11.96]. CatchUp's outputs matched the main run token-for-token on all inputs. This is a favorable measured operating point; no equal-latency point was found in the tested grid, and Tail-1024 was within 10% of CatchUp 50%'s latency on 0/28 inputs.
A separate FFN-omission test isolates this cost on the supplemental HF/FLA backend. At Tail's original 10%-of-prefix budget, states and first-generation logits remained bitwise identical on all 28 inputs; latency fell by 5.58% (task-balanced mean of paired per-input percentage reductions). CatchUp retains its FFN.
FA KV is GPU-resident, and required state/input transfers are timed. Host-resident KV with optimized transfer/replay overlap remains unmeasured. These results concern the tested backend and residency setting.
Q3. How does Proposition 1 relate to full recovery?
Proposition 1 explains single-layer initial-state error propagation under shared gates, keys, and values. In multilayer recovery, earlier-layer perturbations can alter later updates; deeper groups retain checkpoint states missing gap updates. Token-derived inputs do not ensure shared internal transitions.
The exact-first-group intervention asks whether perfect first-group recovery suffices to improve scores when deeper states remain stale. We installed exact producer recurrent/convolution states only in that group, keeping FA KV unchanged. It scored 47.97, or -0.63 versus Stale (95% interval [-1.60, 0.13]). This separates first-group fidelity from end-to-end behavior. The theory describes the conditional repair mechanism; the experiments assess actual multilayer recovery. The intervention is a diagnostic, not deployable recovery or a quality upper bound.