- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Raincheck reduces initial multimodal prefill by processing selected tokens first and deferring the remaining tokens to decoding and post-answer intervals for later-turn reuse. It combines attention-visibility control, approximate recurrent-state correction, and fused prefill–decode execution to support these context updates. Experiments on full-attention and hybrid MLLMs report improved quality over matched hard pruning while retaining lower first-token latency than full prefill.
Critical strengths
-
The paper implements the method for both full-attention and hybrid models, including the cache and state updates needed to add input tokens after generation starts. Supporting both architectures makes the method more broadly useful.
-
The comparison with recovery only after the answer is a useful control. Recovery during generation gives better quality even though both methods restore all input before the next turn, showing that when information becomes available matters.
Critical weaknesses
-
The experiments establish that deferred recovery improves quality over matched hard pruning, but leave its value relative to existing information-preserving methods unclear. Prior work already explores deferred token computation and visual-token recovery during generation. Raincheck differs from these methods, particularly in its recovery schedule and support for hybrid models, but the experiments do not show what these differences contribute relative to an applicable alternative. Testing four token selectors does not address this gap. Appendix A.3 also states that VisionZip-Select omits merging, which is an important part of how the original method preserves information from unselected tokens. The Full and post-answer recovery controls are useful, but answer different questions. Without a representative comparison that reports quality, latency, and cache requirements together, it is difficult to judge whether the additional recovery mechanism offers a useful tradeoff beyond improving over permanent information removal.
-
The small average decoding overhead is demonstrated under a workload that can dilute the cost of recovery. The MultiVerse timing experiment uses fixed 512-token outputs, while restoration usually finishes early in generation, leaving many later steps over which to average its overhead. The video experiments show a larger decoding penalty, and answer-completion latency excludes any remaining post-answer recovery. The paper reports these costs separately, but quality and timing are evaluated in different backends, and timing uses prescribed output traces. These are reasonable measurements for isolating execution overhead, but they do not directly establish the quality–latency tradeoff for naturally terminated answers in the same implementation. Since Raincheck eventually processes the deferred tokens and does not retain hard pruning’s persistent KV-capacity savings, the practical benefit depends on how much recovery can be accommodated without delaying the answer or the next turn. The current evaluation does not establish this clearly enough.
Questions for authors
-
Could you provide a comparison with one applicable information-preserving or recovery method on the full-attention model, reporting quality, latency, and cache requirements? If not, what specific implementation obstacle prevents this comparison?
-
On a small matched subset, could you report quality and latency using the same backend with naturally terminated answers, distinguishing answer-completion time from prefix-ready time?
5 comments
Thanks & Initial Response to Reviewer VzkY (W2, Q2)
[W2/Q2 - On a small matched subset, could you report quality and latency using the same backend with naturally terminated answers, distinguishing answer-completion time from prefix-ready time?]
Thank you for recommending this distinction. We report (1) quality and latency from matched natural-generation runs, (2) prefix readiness after original-prompt answers, and (3) unfinished recovery and immediate follow-ups with brief answers. This separates the time users receive an answer from the time its context is ready for reuse.
1. Matched natural-generation runs show lower first-turn TTFT than Full for both Raincheck configurations.
Original prompts use the dataset’s questions for the four configurations whose quality results are reported in our response to Q1. Ours denotes Raincheck with the pruning method in the same row; Original denotes complete VisionZip with merging and no recovery. Brief prompts add an instruction to answer briefly on eight MultiVerse dialogues; all-turn quality is 16.51 for Full and 33.29 for MMTok + Raincheck.
| Pruning method | Policy | TTFT (ms) | TPOT (ms) | E2E (s) | Output tokens |
|---|---|---|---|---|---|
| Original prompt · 48 dialogues × 2 | |||||
| N/A | Full | 157.15 | 6.8058 | 8.9458 | 1,257.5 |
| MMTok | Ours | 115.75 | 6.7131 | 5.2511 | 765.4 |
| VisionZip | Original | 118.91 | 6.7413 | 7.9819 | 1,143.8 |
| VisionZip | Ours | 119.83 | 6.7365 | 7.0722 | 1,015.4 |
| Brief prompt · 8 dialogues | |||||
| N/A | Full | 221.37 | 6.8268 | 0.3850 | 25.1 |
| MMTok | Ours | 108.11 | 7.5292 | 0.2906 | 24.8 |
Entries are first-turn means. TPOT averages each request’s mean token interval; client E2E runs from submission to receipt of the final output token.
Each configuration uses its own preceding answers as history. We allow up to 16K tokens per turn and stop generation at EOS. All 1,520 original-prompt answers across all turns and four configurations were scored, including nine that reached the limit; brief prompts have none.
2. Both Raincheck configurations finish restoration by first-answer return on original prompts.
| Pruning method | Policy | TPOT during restoration (ms) | Unfinished (n/N) | Post-response mean (ms) | P95 (ms) |
|---|---|---|---|---|---|
| Original prompt | |||||
| MMTok | Ours | 8.068 | 0/96 (0.00%) | 0.788 | 1.300 |
| VisionZip | Ours | 7.695 | 0/96 (0.00%) | 0.852 | 1.335 |
| Brief prompt | |||||
| MMTok | Ours | 9.837 | 5/8 (62.50%) | 13.175 | 24.469 |
Restoration TPOT includes only intervals executing deferred prefill. Unfinished counts first answers with deferred tokens remaining at answer return. Post-response time runs from server-side answer completion to prefix readiness across all first turns, separately from client E2E. Full’s separate readiness time was not recorded.
Under original prompts, neither Raincheck configuration leaves deferred tokens unfinished at first-answer return. The remaining mean post-response intervals are below 1 ms.
3. Brief answers can leave recovery unfinished and increase immediate-follow-up TTFT.
With brief prompts, 5/8 first answers return before recovery completes. For the 23 follow-ups, clients submit the next question immediately after receiving the complete answer, without a client-side inter-turn gap or waiting for prefix readiness. P95 TTFT is 23.17 ms for Full and 98.69 ms for MMTok + Raincheck, including server-side waiting and processing. Recovery waiting was not measured separately.
Decode and supported fused deferred-prefill/decode use CUDA Graph replay with FA3; other prefill and completion may run eagerly. We will add these results to the efficiency and recovery tables, and clarify that the original <0.5% mean TPOT overhead applies to single-request timing with fixed 512-token outputs.
Follow-up: Two Days Remaining in Discussion
Dear Reviewer VzkY,
Thank you for your positive initial recommendation and for recognizing Raincheck’s support for both full-attention and hybrid architectures, as well as the value of recovery during generation.
With only two days remaining in the discussion period, we are following up on our November 13 rebuttal. Following your suggestions, we added a comparison with the complete VisionZip pipeline and joint quality–latency measurements under natural generation, explicitly separating answer completion from prefix readiness.
Adding Raincheck to VisionZip improves first-answer quality by 7.10 points, with a 95% confidence interval of [1.58, 12.86], at similar first-token latency, while accounting for additional recovery compute and cache requirements. This demonstrates a useful combination: compact initial context enables a fast response, and recovery during generation supplies information that improves the ongoing answer.
We would greatly appreciate your feedback on whether these additions address your concerns. We hope further discussion will allow this evidence and Raincheck’s distinctive contributions and practical significance to be fully reflected in the final evaluation. We stand ready to clarify any outstanding points.
Wish you a pleasant day ahead.
Authors of Raincheck
Thanks & Initial Response to Reviewer VzkY (W1, Q1)
[W1/Q1 - Could you provide a comparison with one applicable information-preserving or recovery method on the full-attention model, reporting quality, latency, and cache requirements? If not, what specific implementation obstacle prevents this comparison?]
Thank you for suggesting this comparison. We evaluate (1) quality with the complete VisionZip pipeline, (2) latency and KV-cache costs, and (3) the scope of the current integration. The results offer an encouraging example of compression and recovery working together.
1. Adding Raincheck to VisionZip improves first-turn quality by 7.10 points.
We evaluated Qwen3-VL-8B on 48 MultiVerse image dialogues, each run twice, using SGLang, FA3, and CUDA Graphs on H20 at batch size 1. Full prefills all visual tokens; the other configurations use an initial visual-token budget of 30%. Following the paper, Ours denotes Raincheck with the pruning method in the same row. Original is the complete VisionZip pipeline with merging; Ours adds recovery to the same merging plan.
Scores follow the paper’s 0–100 formula. T1 denotes the first turn, Follow-up includes later turns, and Avg includes all scored turns. Bold marks the higher score within the VisionZip pair. The first-turn gain over VisionZip is 7.10 points [1.58, 12.86], while the all-turn gain is 2.16 [−0.77, 4.85] (dialogue-grouped 95% CIs). The clearest evidence therefore concerns the first answer.
2. The combination retains similar TTFT while using more restoration compute and KV cache.
TTFT is 118.91 ms for VisionZip and 119.83 ms with Raincheck. Timing includes online vision processing, merging, and LLM execution; selection and grouping plans are prepared offline.
At output positions where VisionZip + Raincheck performs deferred prefill, TPOT is 7.695 ms with recovery versus 6.714 ms for VisionZip (+14.6%). Natural-generation latency results are reported in our response to Q2.
With a common 512-token output length, estimated effective KV cache at prefix readiness is 101.91 MiB for VisionZip and 158.12 MiB for Full and both Raincheck configurations (+55.2% over VisionZip). These are effective-cache estimates, rather than measured peak GPU memory.
For applications that can accommodate this cache cost and value stronger first answers at similar startup latency, the combination is a promising option to consider.
3. The current adapter supports recovery alongside merging, with deeper integration left to explore.
The paper studies recovery of tokens omitted by selection before LLM prefill. Its VisionZip-Select baseline omits merging. Its Hard and Post-answer controls examine the value of recovery during generation for the current answer.
Our VisionZip adapter retains merged tokens and adds back their original source tokens; it does not exactly reverse merging. Layerwise pruning, blockwise merging, and explicit merge expansion require further integration. We will add the comparison to the evaluation and clarify the adapter’s scope in the related work.