acceptodds
Under review as a conference paper at ICLR 2027

Bitwise-Identical Latent Reuse for Causal Video Encoders

Abstract

Causal video encoders recompute shared work across overlapping clips, but pixel overlap alone does not justify feature reuse: independently initialized clips differ in temporal support and sampling phase. We derive compatible reuse regions at multiple encoder depths. Shallow features become reusable earlier; deeper features become reusable later but avoid more computation. ClipStore combines both within a request, preserves request-local suffix state, and retains only planned reads using bounded rolling storage. On a frozen Wan2.1 encoder, all timed output comparisons across four videos match independent encoding exactly. On the reference plan, complete construction takes less time and retains fewer feature bytes than the faster fixed-cut baseline, with a speedup over independent encoding. Request-count, length, overlap, and grouping sweeps identify where source construction is amortized. The reported gain in a 50-step text-to-video pipeline is ; encoder reuse adds no output error when combined with DiT-side caching. Decoder continuation illustrates the complementary use of retained causal state to preserve an existing stream’s outputs.

Primary area:
generative models
Keywords:
video diffusion, VAE inference efficiency, latent caching, training-free acceleration, exact reuse, causal convolutions, receptive field, streaming video generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_e3QEOpenReview ↗
Rating
2
Confidence
4(+2)
Informativeness
1(−1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ClipStore which enables exact feature reuse when encoding overlapping video clips for latent-cache construction. It identifies reusable regions from temporal support and phase, combines shallow and deep cached features, and preserves request-local encoder state. A request plan determines which features are retained. Experiments on Wan2.1 check output equality, complete construction cost, and the effects of different request structures.

Critical strengths

The paper addresses a useful problem that reusing computation across overlapping clips while preserving the outputs of independent encoding. The timing includes feature construction and final writes, and the authors report cases where reuse does not help. This makes the comparison more credible within the tested setting.

Critical weaknesses
  1. The design's contribution and generality remain limited. Propositions 1–2 establish correctness and read-set sufficiency, but do not explain how to choose reuse depths for another encoder. The experiments mainly compare two selected cuts with those same cuts used separately in one Wan encoder, leaving other feasible cuts and simpler budget-based choices untested. On the reference plan, the added time saving over the faster fixed cut is 9.93%; the 21.9% payload reduction corresponds to only 3.21% lower total GPU peak. These are valid local gains, but without cut-selection and transfer evidence, their broader significance is not convincing.

  2. The presentation is poor and overly dense, making the reuse procedure hard to follow. The full executor is deferred to Appendix B. A clearer worked example should connect the boundary calculation, A/B switching, and retained features for one complete request.

Questions for authors
  1. Can you give a worked request that connects raw-frame offsets, feature indices, phase matching, and the A/B switching points?
  2. Can you replay requests from the intended workflow after sampling, crop, and augmentation constraints, and report compatible-group sizes and complete construction costs?
  3. How were cuts A and B selected? How do other feasible cuts or a simple budget-based choice compare, and what must be changed and revalidated for another encoder or precision?
Reviewer_qgA9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In dataset preprocessing and video latent cache construction, video latents with overlapping temporal frames are typically recomputed independently from scratch because each clip initializes a distinct causal boundary state, incurring computational redundancy. This paper introduces a new method, ClipStore, to address this redundancy by introducing Depth Stratification: rather than relying on the encoder's global temporal receptive field, exact feature reuse is derived at intermediate encoder depths. The key observation is that exact reuse depends on the cumulative support and stride at each intermediate cut rather than on the global field. ClipStore runs a two-depth schedule per non-origin latent computation request: blocks 0–5 compute A and B from the request boundary, blocks 6–9 read source A features while continuing the request's B state, the remaining blocks read source B features, and C always keeps continuous request-local state. Driven by the host-side plan, ClipStore pre-allocates packed read unions and employs a rolling source builder to enforce a constant peak storage bound. Experiments on 4 benchmarks demonstrates that ClipStore achieves exactness in latent feature extraction against independent encoding. On a reference plan of 16 requests, ClipStore achieves a 2.51x speedup over independent encoding, reducing execution time by 9.93% and memory consumption by 21.9% compared to single-cut fixed baselines.

Critical strengths
  • The paper provides a formal dependency-graph derivation that effectively disentangles initialization boundary effects from temporal sampling phases, enabling bitwise-identical latents compared to the original independent encoding. This makes ClipStore a drop-in replacement for independent encoding.
  • The host-side plan analyzer and rolling source builder ensure that intermediate feature overhead is bounded, keeping total GPU memory allocation stable and predictable regardless of the source video length.
Critical weaknesses
  • Negligible latency gain in full Text-to-Video diffusion pipelines: For standard 50-step text-to-video generation, the resulting overall pipeline speedup is only 1.006x. On this side, the practical system benefit is somewhat limited.
  • All empirical results are restricted to a single model family (Wan2.1 VAE). While the receptive field stratification theory is general and applies to any causal convolution/attention network, the specific layer cuts and input block schedules are manually derived for Wan2.1. Demonstrating automated cut selection on alternative video foundation models (e.g., CogVideoX or HunyuanVideo VAEs) would strengthen the system's generality.
  • The comparison with the independent encoding is against a speed-unoptimized setting. The 2.51x speedup is measured against unbatched (i.e. batch size 1) independent encoding. 16 equal-length 129-frame clips can be batched, which typically improves GPU utilization substantially for a 480×832 3D-conv VAE.
Questions for authors
  • What is the speedup over batched inference version of independent encoding (e.g., all 16 reference-plan clips in one or a few batches?
  • Could the depth stratification cuts (Cuts A and B) be selected automatically for an arbitrary causal architecture via automated graph analysis of receptive fields and tensor byte sizes?
  • How possible is it to adapt the proposed method for other causal 3DVAEs? What performance gains can be expected?
Reviewer_vzKtOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how a causal video encoder can reuse intermediate features when processing multiple overlapping clips with different starting points, while preserving the outputs of independent encoding. The paper points out that pixel overlap alone is insufficient to justify feature reuse, because independent initialization of different clips makes early features depend on each request’s own boundary, and temporal downsampling further introduces phase differences. Accordingly, the authors derive, from dependency graphs, cumulative receptive fields, and sampling phase, the boundaries at different network depths at which features become compatible. The authors propose ClipStore, which partitions the Wan2.1 video encoder into three stages, A, B, and C. Within the same request, it first computes boundary regions, then reuses shallow A features and deeper B features in sequence, while suffix C always maintains the request’s own continuous state. The system uses a pre-known request plan to compute the positions that will actually be read, stores only these positions, and uses a rolling builder to bound additional memory. The theoretical part provides sufficient conditions for compatible reuse and state continuity, while the implementation retains the original temporal chunks, BF16 features, and a fixed execution path. Experiments are conducted on a frozen Wan2.1 encoder, an A100 GPU, and four public videos. All 4,944 timed output comparisons are element-wise identical to independent encoding. On the reference plan, the complete construction time of ClipStore is 28.497 s, lower than 31.637 s for Fixed A, 34.059 s for Fixed B, and 71.626 s for independent encoding; this reduces time by 9.93% relative to the faster fixed-cut baseline and retains 21.9% fewer feature bytes than Fixed A. Sweeps over request count, length, spacing, and grouping further clarify when the additional construction cost of deeper features can be amortized. The paper also explicitly notes that the end-to-end speedup in a 50-step text-to-video pipeline is only 1.006×, and that the main benefits are more applicable to latent cache construction and multi-clip preprocessing.

Critical strengths
  1. The correctness argument is well integrated with implementation validation. The paper separately analyzes “different boundary initialization” and “different temporal sampling phase,” and derives reuse boundaries using the cumulative receptive field at each cut point; the state-continuity argument explains why replacing only numerically equal feature blocks without resetting the suffix state preserves the complete output. The experiments not only report that all 4,944 zero-tolerance comparisons are equal, but also show that shortening the repair interval or choosing the wrong phase produces differences. These counterexamples make the bit-level consistency conclusion more credible and directly support the paper’s core technical claim.
  2. The system design centers on full cost rather than single-operator speedup. ClipStore merges the actual read intervals according to the request plan, combines shallow and deep reuse within a single request, and uses a rolling builder to control additional feature storage. The evaluation accounts for source feature construction, request execution, final writes, and fsync, and separately reports retained features, construction peak, and overall GPU peak. Experiments on request count, clip length, overlap density, and temporal grouping also clearly show the method’s applicable regime, rather than reporting only a single most favorable configuration.
Critical weaknesses
  1. External validity remains limited. All main results come from the same Wan2.1 encoder, mixed-precision and execution configuration, A100 GPU, batch size 1, and four public videos. Bit-level consistency depends on fixed kernels, batch shape, precision, and autocast boundaries, and the paper also acknowledges that these conditions need to be revalidated when changed. Therefore, the current evidence sufficiently demonstrates feasibility for this implementation, but is not yet sufficient to show that multi-depth exact reuse can reliably transfer to other causal video encoders, hardware, or production configurations.
  2. The method relies on knowing the complete request plan in advance, and no optimizer has yet been provided for automatically selecting reuse depth, grouping strategy, and storage location. Experiments show that ClipStore is not always superior and requires a trade-off between time and storage. Fixed A is faster when requests are fewer or shorter; in the mixed plan, ClipStore is 1.4% slower than Fixed A, and its retained memory on the reference plan is also higher than that of Fixed B. Most sweep points have only one observation per video, and only the full range across four videos is reported; the lack of more repetitions, confidence intervals, real workload traces, and broader system baselines leaves the robustness and deployment value of the sweep gains uncertain. The end-to-end improvement of only 1.006× in the conventional 50-step generation pipeline further limits immediate impact.
Questions for authors
  1. Can you validate the automatically derived reuse boundaries and bit-level consistency on at least one structurally different causal video encoder, or on another GPU and precision configuration? If bit-level consistency cannot be maintained, which operators or execution differences are the first to break this property? This result would directly affect my judgment of the method’s generality.
  2. Given a request plan, memory budget, and transfer cost, can you automatically choose among Fixed A, Fixed B, ClipStore, or more cut points, while simultaneously determining temporal grouping? Please report the gap between such a selector and the post hoc optimal choice for each plan. If manual selection is required, the paper’s claims about practical deployment value should be narrowed accordingly.
  3. Can you add independent repetitions to the sweeps over request count, length, and spacing, report confidence intervals for paired differences, and include request traces from real latent cache construction pipelines? This would help determine whether the approximately 10% reference-plan gain is stable and under what plan distributions the gain disappears.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.