acceptodds
Under review as a conference paper at ICLR 2027

Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference

Abstract

Multi-turn conversations with multimodal large language models (MLLMs) reuse input context, but long multimodal inputs increase time to first token (TTFT). Query-aware hard pruning reduces prefill tokens but permanently excludes information that later questions may require. We propose Raincheck, a framework for deferred multimodal prefill that decouples initial token selection from later information availability. An existing pruning method selects tokens for initial prefill; Raincheck preserves unselected tokens and progressively prefills them in source order using spare compute capacity during decode. Unfinished prefill continues after answer return during available scheduling gaps. Attention visibility control and affine state mapping enable context updates after generation begins in full-attention and hybrid recurrent models. Prefill-decode fusion reduces repeated weight reads. We evaluate Qwen3-VL-8B and Qwen3.5-9B with 30% initial multimodal-token retention. Across both models and four pruning methods, Raincheck recovers on average 73.7% and 59.3% of the quality lost to matched hard pruning on MultiVerse and ConvBench, respectively. On MultiVerse, the SGLang implementation reduces mean first-turn TTFT by up to 24.1% relative to full prefill, with less than 0.5% increase in mean time per output token (TPOT).

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, Token Compression, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VzkYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Raincheck reduces initial multimodal prefill by processing selected tokens first and deferring the remaining tokens to decoding and post-answer intervals for later-turn reuse. It combines attention-visibility control, approximate recurrent-state correction, and fused prefill–decode execution to support these context updates. Experiments on full-attention and hybrid MLLMs report improved quality over matched hard pruning while retaining lower first-token latency than full prefill.

Critical strengths
  1. The paper implements the method for both full-attention and hybrid models, including the cache and state updates needed to add input tokens after generation starts. Supporting both architectures makes the method more broadly useful.

  2. The comparison with recovery only after the answer is a useful control. Recovery during generation gives better quality even though both methods restore all input before the next turn, showing that when information becomes available matters.

Critical weaknesses
  1. The experiments establish that deferred recovery improves quality over matched hard pruning, but leave its value relative to existing information-preserving methods unclear. Prior work already explores deferred token computation and visual-token recovery during generation. Raincheck differs from these methods, particularly in its recovery schedule and support for hybrid models, but the experiments do not show what these differences contribute relative to an applicable alternative. Testing four token selectors does not address this gap. Appendix A.3 also states that VisionZip-Select omits merging, which is an important part of how the original method preserves information from unselected tokens. The Full and post-answer recovery controls are useful, but answer different questions. Without a representative comparison that reports quality, latency, and cache requirements together, it is difficult to judge whether the additional recovery mechanism offers a useful tradeoff beyond improving over permanent information removal.

  2. The small average decoding overhead is demonstrated under a workload that can dilute the cost of recovery. The MultiVerse timing experiment uses fixed 512-token outputs, while restoration usually finishes early in generation, leaving many later steps over which to average its overhead. The video experiments show a larger decoding penalty, and answer-completion latency excludes any remaining post-answer recovery. The paper reports these costs separately, but quality and timing are evaluated in different backends, and timing uses prescribed output traces. These are reasonable measurements for isolating execution overhead, but they do not directly establish the quality–latency tradeoff for naturally terminated answers in the same implementation. Since Raincheck eventually processes the deferred tokens and does not retain hard pruning’s persistent KV-capacity savings, the practical benefit depends on how much recovery can be accommodated without delaying the answer or the next turn. The current evaluation does not establish this clearly enough.

Questions for authors
  1. Could you provide a comparison with one applicable information-preserving or recovery method on the full-attention model, reporting quality, latency, and cache requirements? If not, what specific implementation obstacle prevents this comparison?

  2. On a small matched subset, could you report quality and latency using the same backend with naturally terminated answers, distinguishing answer-completion time from prefix-ready time?

5 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer VzkY (W1, Q1)

[W1/Q1 - Could you provide a comparison with one applicable information-preserving or recovery method on the full-attention model, reporting quality, latency, and cache requirements? If not, what specific implementation obstacle prevents this comparison?]

Thank you for suggesting this comparison. We evaluate (1) quality with the complete VisionZip pipeline, (2) latency and KV-cache costs, and (3) the scope of the current integration. The results offer an encouraging example of compression and recovery working together.

1. Adding Raincheck to VisionZip improves first-turn quality by 7.10 points.

We evaluated Qwen3-VL-8B on 48 MultiVerse image dialogues, each run twice, using SGLang, FA3, and CUDA Graphs on H20 at batch size 1. Full prefills all visual tokens; the other configurations use an initial visual-token budget of 30%. Following the paper, Ours denotes Raincheck with the pruning method in the same row. Original is the complete VisionZip pipeline with merging; Ours adds recovery to the same merging plan.

Pruning methodPolicyT1Follow-upAvg
N/AFull56.7173.2669.08
MMTokOurs54.2273.6568.74
VisionZipOriginal48.4673.5667.22
VisionZipOurs55.5674.0569.38

Scores follow the paper’s 0–100 formula. T1 denotes the first turn, Follow-up includes later turns, and Avg includes all scored turns. Bold marks the higher score within the VisionZip pair. The first-turn gain over VisionZip is 7.10 points [1.58, 12.86], while the all-turn gain is 2.16 [−0.77, 4.85] (dialogue-grouped 95% CIs). The clearest evidence therefore concerns the first answer.

2. The combination retains similar TTFT while using more restoration compute and KV cache.

TTFT is 118.91 ms for VisionZip and 119.83 ms with Raincheck. Timing includes online vision processing, merging, and LLM execution; selection and grouping plans are prepared offline.

At output positions where VisionZip + Raincheck performs deferred prefill, TPOT is 7.695 ms with recovery versus 6.714 ms for VisionZip (+14.6%). Natural-generation latency results are reported in our response to Q2.

With a common 512-token output length, estimated effective KV cache at prefix readiness is 101.91 MiB for VisionZip and 158.12 MiB for Full and both Raincheck configurations (+55.2% over VisionZip). These are effective-cache estimates, rather than measured peak GPU memory.

For applications that can accommodate this cache cost and value stronger first answers at similar startup latency, the combination is a promising option to consider.

3. The current adapter supports recovery alongside merging, with deeper integration left to explore.

The paper studies recovery of tokens omitted by selection before LLM prefill. Its VisionZip-Select baseline omits merging. Its Hard and Post-answer controls examine the value of recovery during generation for the current answer.

Our VisionZip adapter retains merged tokens and adds back their original source tokens; it does not exactly reverse merging. Layerwise pruning, blockwise merging, and explicit merge expansion require further integration. We will add the comparison to the evaluation and clarify the adapter’s scope in the related work.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer VzkY (W2, Q2)

[W2/Q2 - On a small matched subset, could you report quality and latency using the same backend with naturally terminated answers, distinguishing answer-completion time from prefix-ready time?]

Thank you for recommending this distinction. We report (1) quality and latency from matched natural-generation runs, (2) prefix readiness after original-prompt answers, and (3) unfinished recovery and immediate follow-ups with brief answers. This separates the time users receive an answer from the time its context is ready for reuse.

1. Matched natural-generation runs show lower first-turn TTFT than Full for both Raincheck configurations.

Original prompts use the dataset’s questions for the four configurations whose quality results are reported in our response to Q1. Ours denotes Raincheck with the pruning method in the same row; Original denotes complete VisionZip with merging and no recovery. Brief prompts add an instruction to answer briefly on eight MultiVerse dialogues; all-turn quality is 16.51 for Full and 33.29 for MMTok + Raincheck.

Pruning methodPolicyTTFT (ms)TPOT (ms)E2E (s)Output tokens
Original prompt · 48 dialogues × 2
N/AFull157.156.80588.94581,257.5
MMTokOurs115.756.71315.2511765.4
VisionZipOriginal118.916.74137.98191,143.8
VisionZipOurs119.836.73657.07221,015.4
Brief prompt · 8 dialogues
N/AFull221.376.82680.385025.1
MMTokOurs108.117.52920.290624.8

Entries are first-turn means. TPOT averages each request’s mean token interval; client E2E runs from submission to receipt of the final output token.

Each configuration uses its own preceding answers as history. We allow up to 16K tokens per turn and stop generation at EOS. All 1,520 original-prompt answers across all turns and four configurations were scored, including nine that reached the limit; brief prompts have none.

2. Both Raincheck configurations finish restoration by first-answer return on original prompts.

Pruning methodPolicyTPOT during restoration (ms)Unfinished (n/N)Post-response mean (ms)P95 (ms)
Original prompt
MMTokOurs8.0680/96 (0.00%)0.7881.300
VisionZipOurs7.6950/96 (0.00%)0.8521.335
Brief prompt
MMTokOurs9.8375/8 (62.50%)13.17524.469

Restoration TPOT includes only intervals executing deferred prefill. Unfinished counts first answers with deferred tokens remaining at answer return. Post-response time runs from server-side answer completion to prefix readiness across all first turns, separately from client E2E. Full’s separate readiness time was not recorded.

Under original prompts, neither Raincheck configuration leaves deferred tokens unfinished at first-answer return. The remaining mean post-response intervals are below 1 ms.

3. Brief answers can leave recovery unfinished and increase immediate-follow-up TTFT.

With brief prompts, 5/8 first answers return before recovery completes. For the 23 follow-ups, clients submit the next question immediately after receiving the complete answer, without a client-side inter-turn gap or waiting for prefix readiness. P95 TTFT is 23.17 ms for Full and 98.69 ms for MMTok + Raincheck, including server-side waiting and processing. Recovery waiting was not measured separately.

Decode and supported fused deferred-prefill/decode use CUDA Graph replay with FA3; other prefill and completion may run eagerly. We will add these results to the efficiency and recovery tables, and clarify that the original <0.5% mean TPOT overhead applies to single-request timing with fixed 512-token outputs.

AuthorsOpenReview ↗

Follow-up: Two Days Remaining in Discussion

Dear Reviewer VzkY,

Thank you for your positive initial recommendation and for recognizing Raincheck’s support for both full-attention and hybrid architectures, as well as the value of recovery during generation.

With only two days remaining in the discussion period, we are following up on our November 13 rebuttal. Following your suggestions, we added a comparison with the complete VisionZip pipeline and joint quality–latency measurements under natural generation, explicitly separating answer completion from prefix readiness.

Adding Raincheck to VisionZip improves first-answer quality by 7.10 points, with a 95% confidence interval of [1.58, 12.86], at similar first-token latency, while accounting for additional recovery compute and cache requirements. This demonstrates a useful combination: compact initial context enables a fast response, and recovery during generation supplies information that improves the ongoing answer.

We would greatly appreciate your feedback on whether these additions address your concerns. We hope further discussion will allow this evidence and Raincheck’s distinctive contributions and practical significance to be fully reflected in the final evaluation. We stand ready to clarify any outstanding points.

Wish you a pleasant day ahead.

Authors of Raincheck

Reviewer_2EREOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses the problem that one-shot pruning in multi-turn multimodal dialogue permanently discards information needed in later turns. The paper proposes Raincheck, a deferred-prefill framework. The system first processes selected visual tokens, then processes the remaining tokens during generation and in scheduling gaps after responses. Through attention visibility control and affine state updates in Gated DeltaNet, it integrates the recovered context into ongoing generation. Across two Qwen architectures and four pruning selectors, RAINCHECK recovers, on average, 77.7% and 60.4% of the performance lost to hard pruning under equivalent conditions on the MultiVerse and ConvBench datasets, respectively. On MultiVerse, the SGLang-based implementation reduces average first-turn time to first token by up to 24.1% compared with full prefill, while increasing average time per output token (TPOT) by less than 0.5%.

Critical strengths
  • The timing of recovery:All eight model–selector pairs improve in aggregate MultiVerse quality in Table 2. More importantly, Appendix D.6, Table 24 compares recovery during generation against recovery only after the first answer: the former improves all-turn scores by 4.93 and 5.70 points, with paired confidence intervals above zero. This establishes value beyond merely restoring the context eventually.

  • The implementation and cost analysis are substantive. The method supports both full-attention and hybrid recurrent architectures, and Appendix B distinguishes exact updates for recorded factors from approximation to a full model rerun. The fused-versus-serial control in Table 31 supports the execution optimization, while Table 28 measures immediate-follow-up delays. The explicit accounting that full recovery does not preserve Hard’s persistent KV-capacity savings further clarifies the trade-off.

Critical weaknesses
  • The headline near-zero average TPOT overhead depends on a specific output-length regime. MultiVerse timing replays 512-token outputs, whereas image restoration requires about 29.88 output positions on average and at most 63 (Appendix E.1), diluting the more expensive restoration intervals. The paper does include naturally terminated Full traces for video, a short-output stress test, and immediate-follow-up measurements; nevertheless, video TPOT increases by 19.93%/14.92% in Table 4. Joint quality and latency measurements with natural generation in the same SGLang path, especially for short answers and early follow-ups, would better establish the practical operating region.

  • Numerical validation of the state map does not quantify approximation error relative to a full model rerun. Table 8 uses identical recorded factors, whereas Appendix B.5 correctly explains that expanded context can change higher-layer activations and transition factors. Task scores show that the approximation can work, but do not isolate losses caused by retaining cached history, particularly with long histories or successive media additions. This limits assessment of the operating range; it is not an objection to Equation (2) under its stated fixed-factor assumptions.

Questions for authors
  • Q1 : On the same image-dialogue cohort, can you jointly report quality, first-turn TTFT, restoration-interval TPOT, and immediate-follow-up waiting time using natural SGLang generation rather than fixed 512-token replay? Please include output lengths and the fraction with unfinished recovery.

  • Q2 : Holding suffix tokens fixed, how different are the corrected states or next-token distributions from a full rerun with expanded context, and how does this vary with history length and recovery fraction? A small controlled diagnostic would directly address this concern.

3 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer 2ERE (W1, Q1)

[W1/Q1 - On the same image-dialogue cohort, can you jointly report quality, first-turn TTFT, restoration-interval TPOT, and immediate-follow-up waiting time using natural SGLang generation rather than fixed 512-token replay? Please include output lengths and the fraction with unfinished recovery.]

Thank you for this suggestion. We examine (1) quality under natural generation, (2) first-turn latency and restoration costs, and (3) recovery completion and immediate follow-ups. Together, these measurements give a clearer picture of Raincheck’s benefits and the interaction patterns it best supports.

1. Natural generation yields similar all-turn quality on 8B original prompts, while 9B shows a first-turn quality gap.

We evaluated Qwen3-VL-8B and Qwen3.5-9B on H20 with SGLang, FA3, and CUDA Graphs at batch size 1. Ours uses MMTok with Raincheck, initially retaining 30% of visual tokens. We used 48 MultiVerse image dialogues run twice for 8B original prompts and eight dialogues for each other condition. Brief prompts add an instruction to answer briefly.

PolicyT1Follow-upAvg
8B · Original prompt
Full56.7173.2669.08
Ours54.2273.6568.74
9B · Original prompt
Full66.0372.8471.08
Ours42.9071.1863.88
8B · Brief prompt
Full13.6717.5016.51
Ours37.9631.6633.29
9B · Brief prompt
Full6.5622.2618.21
Ours1.8122.3217.03

Scores follow the paper’s 0–100 formula. T1 denotes the first turn, Follow-up includes later turns, and Avg includes all scored turns. Each method uses its own preceding answers as history. We allow up to 16K tokens per turn and stop generation at EOS. Truncated answers remain included: five across all turns of Full and Ours for 8B original prompts and none for the other conditions.

On original prompts, the all-turn difference from Full is −0.34 points for 8B and −7.20 for 9B. The 9B gap mainly occurs in the first turn, although all eight first answers complete recovery before EOS. The brief-prompt results are exploratory eight-dialogue tests.

2. Raincheck reduces first-turn TTFT in all four conditions, with higher token latency during restoration.

PolicyTTFT (ms)TPOT (ms)E2E (s)Output tokens
8B · Original prompt
Full157.156.80588.94581,257.5
Ours115.756.71315.2511765.4
9B · Original prompt
Full261.377.23157.69931,029.8
Ours191.467.25097.3997996.6
8B · Brief prompt
Full221.376.82680.385025.1
Ours108.117.52920.290624.8
9B · Brief prompt
Full300.877.27290.515930.6
Ours196.387.72600.425030.0

Entries are first-turn means. TPOT averages each request’s mean token interval; client E2E runs from submission to receipt of the final output token. Timing includes online vision processing and LLM execution; token-selection plans are prepared offline.

For 8B original prompts, TPOT at the output positions where Raincheck performs deferred prefill is 6.754 ms for Full and 8.068 ms for Raincheck. Decode and supported fused deferred-prefill/decode use CUDA Graph replay; other prefill and post-answer completion may run eagerly.

3. Recovery finishes within original-prompt answers, while brief answers can increase immediate-follow-up TTFT.

ConditionTPOT during restoration (ms)Unfinished (n/N)Post-response mean (ms)P95 (ms)
8B · Original prompt8.0680/96 (0.00%)0.7881.300
9B · Original prompt8.5200/8 (0.00%)0.5350.560
8B · Brief prompt9.8375/8 (62.50%)13.17524.469
9B · Brief prompt8.6185/8 (62.50%)31.71854.984

Restoration TPOT includes only intervals executing deferred prefill. Unfinished counts first answers with deferred tokens remaining at answer return. Post-response time runs from server-side answer completion to prefix readiness across all first turns.

Both models leave recovery unfinished in 5/8 brief first answers. For the 23 follow-ups in the 8B brief-answer test, clients submit the next question immediately after receiving the complete answer, without a client-side inter-turn gap or waiting for prefix readiness. P95 TTFT is 23.17 ms for Full and 98.69 ms for Ours. This includes server-side waiting and processing; recovery waiting was not measured separately.

We will add these results beside the quality, efficiency, and recovery tables, and clarify that the original <0.5% mean TPOT overhead applies to single-request timing with fixed 512-token outputs.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer 2ERE (W2, Q2)

[W2/Q2 - Holding suffix tokens fixed, how different are the corrected states or next-token distributions from a full rerun with expanded context, and how does this vary with history length and recovery fraction? A small controlled diagnostic would directly address this concern.]

Thank you for proposing this diagnostic. We examine (1) the effect of fixed suffix length, (2) the effect of recovery fraction, and (3) the behavior with successive media. The results help characterize where the correction closely approximates a rerun and where further refinement would be useful.

1. Longer fixed suffixes yield smaller mean distribution differences in the tested inputs.

We evaluated Qwen3.5-9B in Transformers on eight MultiVerse image inputs, initially retaining 30% of visual tokens. Here, γ is the fraction of initially deferred visual tokens restored; the processed fraction of original visual tokens is approximately 0.30 + 0.70γ.

Full first generates a suffix, ignoring EOS to reach the tested lengths. The corrected run and reference rerun then use identical suffix tokens. At each γ, the reference recomputes the currently available input tokens and suffix from scratch, matching the corrected run’s token order and positions.

Fixed suffix tokensγ = 0γ = 0.25γ = 0.5γ = 1
322.67e-041.42e-021.45e-022.26e-02
1289.90e-055.85e-038.51e-038.30e-03
5123.29e-055.97e-048.93e-042.63e-03
2,0486.81e-072.20e-049.62e-059.02e-05

Each cell reports mean Jensen–Shannon divergence between next-token distributions over eight inputs, using natural logarithms. At full recovery, mean JS decreases from 2.26e-02 with 32 suffix tokens to 9.02e-05 with 2,048 tokens.

2. Distribution differences do not increase monotonically with the recovery fraction.

For example, with 2,048 suffix tokens, mean JS decreases from 2.20e-04 at γ = 0.25 to 9.02e-05 at γ = 1. The relationship therefore depends on the tested context and recovery state.

3. Successive-media inputs show larger differences and motivate further refinement.

On four two-image sequences, mean JS is 0.1189 with a 512-token suffix and γ = 1. This identifies successive-media contexts as a useful direction for extending the correction mechanism.

Equation (2) is exact for its recorded factors; a full rerun can change those factors and earlier activations. We will clarify this distinction and add both diagnostics to Appendix B.

AuthorsOpenReview ↗

Follow-up: Two Days Remaining in Discussion

Dear Reviewer 2ERE,

Thank you for your positive initial recommendation and for recognizing the value of recovery during generation, our cross-architecture implementation, and the substantive cost analysis.

With only two days remaining in the discussion period, we are following up on our November 13 rebuttal. We provided the matched natural-generation quality–latency evaluation and controlled full-rerun diagnostic you requested. The revised manuscript explicitly distinguishes the exact recorded-factor update from its approximation to a full rerun and documents the operating limits.

Raincheck makes initially omitted information useful during the current answer while restoring context for later turns. The recovery-timing control isolates this benefit, with all-turn gains of 4.93 and 5.70 points over post-answer-only recovery. The additional natural-generation experiments jointly report answer quality, startup latency, restoration costs, and immediate-follow-up behavior, making the practical trade-offs directly assessable.

We would greatly appreciate your feedback on whether these additions address your concerns. We hope further discussion will allow the supplementary evidence and Raincheck’s distinctive technical contributions and practical value to be fully reflected in the final evaluation. We stand ready to provide further clarification.

Wish you a pleasant day ahead.

Authors of Raincheck

Reviewer_PR9eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Raincheck, a deferred multimodal prefill framework that reduces initial latency while preserving information discarded by conventional token pruning. The method restores deferred tokens during decoding and after response completion using attention visibility control and recurrent state mapping to support multi-turn context reuse. Experiments across two model architectures and multiple pruning methods demonstrate improved multi turn response quality over hard pruning while reducing first turn latency relative to full prefill.

Critical strengths

The proposed approach effectively recovers information lost through token pruning while reducing first turn latency. Experiments across multiple pruning methods and model architectures demonstrate consistent improvements in multi turn response quality supporting the effectiveness and generality of the proposed deferred prefill mechanism.

Critical weaknesses

W1.

The proposed approach relies on unused compute capacity during decoding which may decrease under concurrent serving. Without evaluating performance under varying serving loads, it remains unclear whether the reported latency benefits can be maintained in practical deployment settings.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer PR9e (W1)

[W1 - The proposed approach relies on unused compute capacity during decoding which may decrease under concurrent serving. Without evaluating performance under varying serving loads, it remains unclear whether the reported latency benefits can be maintained in practical deployment settings.]

Thank you for suggesting this evaluation. We address this question through (1) first-turn latency under concurrent arrivals, (2) the effect of unified scheduling on token-delivery latency, and (3) the resulting deployment scope. The results retain Raincheck’s first-turn TTFT benefit and help clarify the scheduling trade-offs.

1. Raincheck reduces first-turn P95 TTFT relative to Full across all five tested concurrent workloads.

We evaluated Qwen3-VL-8B on one H20 with SGLang, FA3, and CUDA Graphs, with actual decode batches of 1–4. Each workload contains 24 MultiVerse image-dialogue sessions and 95 requests. Ours combines MMTok with Raincheck, initially retaining 30% of visual tokens and targeting full recovery.

Session workloadPolicyT1 TTFT P95 (ms)Max-ITL P95 (ms)SLO (n/N)
Low rateFull401.6934.7694/95
Low rateOurs205.9453.9692/95
Low rateOurs (unified)225.3341.8991/95
Medium rateFull407.3481.3392/95
Medium rateOurs203.39107.4589/95
Medium rateOurs (unified)223.3190.3891/95
Higher rateFull399.0288.0990/95
Higher rateOurs196.79110.9689/95
Higher rateOurs (unified)231.5589.3892/95
Bursty arrivalsFull806.7148.8594/95
Bursty arrivalsOurs307.88183.3485/95
Bursty arrivalsOurs (unified)364.5474.4391/95
Immediate follow-upFull396.2063.9293/95
Immediate follow-upOurs206.8156.6991/95
Immediate follow-upOurs (unified)226.0069.9993/95

The three arrival rates use Poisson session starts at 0.0075, 0.015, and 0.0225 sessions/s. Bursty arrivals start four sessions together at fixed intervals, matching the medium mean rate. Clients submit follow-ups after receiving the complete answer. Immediate follow-up uses the medium arrival plan without a client-side inter-turn gap; other workloads use pre-set gaps. Clients never wait for prefix readiness, so follow-up TTFT includes any server-side recovery waiting.

2. Unified scheduling improves Max-ITL P95 versus original Raincheck in four of five workloads, while recovery still incurs costs.

The unified scheduler interleaves incoming prefill, deferred prefill, and decode. Full and Ours (unified) use chunked prefill; original Ours does not. Both Raincheck configurations target full recovery. Under bursty arrivals, unified scheduling reduces Max-ITL P95 from 183.34 to 74.43 ms while retaining lower first-turn P95 TTFT than Full (364.54 versus 806.71 ms). Its Max-ITL P95 remains above Full in all five workloads. Its SLO attainment is close to Full (within 3.2 percentage points).

T1 P95 covers 24 first turns; Max-ITL P95 covers the per-request maximum intervals between consecutive output tokens across 95 requests. SLO requires normal EOS, TTFT ≤ 1 s, and maximum ITL ≤ 100 ms. Each method uses its own answers as history, generating up to 16K tokens per turn and stopping at EOS. All requests are included, with eight token-limit hits counted as failures. Each setting was run once. Decode and fused deferred-prefill/decode use CUDA Graph replay; ordinary prefill, including prefill mixed with decode, runs eagerly.

3. Raincheck is a promising option for premium, low-traffic services that prioritize ultra-low response latency.

A concrete use case is a VIP multimodal service provisioned with high-end GPUs and low per-GPU traffic, where serving a few users with very fast responses is the priority. Our small-batch results support considering Raincheck for this setting. We would be happy to explore recovery scheduling tailored to such latency-focused services. We will clarify this deployment scope and add the concurrent results to the efficiency evaluation, distinguishing them from the paper’s single-request mean measurements.

My concerns are fully addressed.

I will maintain my score.

AuthorsOpenReview ↗

Follow-up: Two Days Remaining in Discussion

Dear Reviewer PR9e,

Thank you for your positive initial recommendation and for confirming that all your concerns are now fully addressed. We are encouraged by your recognition of Raincheck’s effectiveness and generality.

With only two days remaining in the discussion period, we would like to emphasize its practical value: users can receive a faster initial response while the system recovers visual information needed by the ongoing answer and later questions.

Our additional experiments directly support this use case. Across all five tested small-batch concurrent workloads, Raincheck retains lower first-turn P95 TTFT than full prefill. Under bursty arrivals, unified Raincheck reduces it from 806.71 to 364.54 ms, with SLO attainment within 3.2 percentage points of full prefill. Adding Raincheck to complete VisionZip also improves first-answer quality by 7.10 points at similar first-token latency.

Raincheck turns initial pruning into a scheduling decision, enabling fast responses and subsequent information recovery across full-attention and hybrid architectures. We hope the additional evidence, together with the resolution of your concerns, allows this technical contribution and its practical significance to be fully reflected in the final evaluation.

Thank you for your time, consideration, and constructive discussion.

Wish you a pleasant day ahead.

Authors of Raincheck

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions.

We appreciate R1’s recognition of improved multi-turn quality across pruning methods and model architectures, together with lower first-turn latency, and the suggestion to evaluate concurrent serving. We thank R2 for recognizing the value of recovery during generation and the implementation and cost analysis, and for proposing joint quality–latency measurements and GDN diagnostics. We also thank R3 for recognizing support for full-attention and hybrid models and the importance of recovery timing, and for recommending a comparison with a complete information-preserving method. These suggestions helped us strengthen the evaluation and clarify Raincheck’s practical benefits and costs.

We have responded to each reviewer’s questions in our individual replies. Below, we summarize our core contributions and updates to experiments and discussions.

Core Contributions of Our Work

  1. Recovery during generation for current answers. Raincheck makes initially omitted visual information available during generation. In the paper’s Post-answer control, this improves all-turn scores by 4.93 points on 8B and 5.70 on 9B over recovery only after the answer, although both methods complete recovery before later context reuse.
  2. Deferred prefill across model architectures. Attention visibility control and approximate GDN state correction allow deferred tokens to be added after generation starts. Prefill–decode fusion reduces repeated weight reads. We implement this design on Qwen3-VL-8B and Qwen3.5-9B with four initial token selectors.
  3. Quality recovery with low first-token latency. Raincheck recovers on average 77.7% and 60.4% of the quality lost to matched hard pruning on MultiVerse and ConvBench. In the original single-request timings with fixed 512-token outputs, mean first-turn TTFT is reduced by up to 24.1% relative to Full.

Updates of experimental results during Rebuttal

  • Section 3.2: Added a matched comparison with complete VisionZip on 8B. Adding Raincheck improves first-answer quality by 7.10 points at similar TTFT, with more recovery compute and a larger estimated effective KV cache. Both VisionZip configurations include merging.
  • Section 3.3: In the tested small-batch concurrent workloads, MMTok + Raincheck retains lower first-turn P95 TTFT than Full. Unified scheduling further reduces tail token latency relative to original Raincheck in most workloads.
  • Appendix E.4: Added joint quality–latency measurements using natural SGLang generation, covering original prompts, brief answers, and immediate follow-ups.
  • Section 3.4: Added a controlled GDN comparison against a full rerun, holding suffix tokens fixed while varying history length and recovery fraction, including successive media additions.

Updates of in-depth discussions during Rebuttal

  • Section 3.3: Discussed how recovery shares compute with incoming prefill and decode (R1). Coordinating these tasks helps retain first-token gains while limiting token-delivery delays in small-batch interactive services.
  • Appendix E.4: Discussed how recovery timing affects the current answer and later context reuse (R2/R3). Recovered information can guide later generation. For short answers, remaining recovery can use an available client-side inter-turn gap or delay an immediate follow-up, depending on server capacity.
  • Section 3.2: Explained why initial merging and recovery during generation can complement each other (R3). Merging supplies a compact initial context, while recovery makes original source tokens available during the answer.
  • Appendix B.7: Explained how the state map for recorded factors differs from a full model rerun (R2). The approximation depends on both existing state error and changes in factors after context expansion.

We thank the reviewers again for their constructive feedback and look forward to further discussion. New results and supporting experimental details are marked in blue in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.