acceptodds
Under review as a conference paper at ICLR 2027

Residual Motion Splatting for Continuous Space-Time Video Super-Resolution

Abstract

Continuous space-time video super-resolution produces a high-resolution, high-frame-rate video from a low-resolution, low-frame-rate one at arbitrary spatial scales and temporal instants. Classical spatial multi-frame super-resolution has shown that detail beyond a single frame’s sampling limit can be recovered by aligning complementary observations with sub-pixel precision and accumulating them on a finer grid. Existing learning-based methods largely depart from this principle: some align and fuse frames only at low resolution before regressing a continuous high-resolution representation, while others interpolate each frame to the output resolution before splatting them. In both cases, the per-frame samples are resampled before fusion, so their sub-pixel positions are lost before frames are combined. Restoring this principle in a learned continuous model raises two challenges: estimating sub-pixel motion at unobserved times, and handling holes and occlusions when splatting a coarse grid onto a finer one. We address the first by predicting per-pixel continuous trajectories that refine linear optical-flow paths with smooth residual motion, using features integrated over the full video sequence. We address the second by directly splatting consecutive low-resolution features in a learned latent space onto a high-resolution grid at their predicted sub-pixel positions with predicted visibility weights, before a decoder resolves remaining ambiguities. This makes temporal interpolation and multi-frame super-resolution a single operation. Our method improves both reconstruction fidelity and perceptual quality on multiple standard benchmarks over the state of the art, while running up to an order of magnitude faster.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video super-resolution, space-time video super-resolution, continuous space-time video super-resolution, forward splatting, motion trajectory modeling, video frame interpolation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KND3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper brings classical sub-pixel accumulation into learned latent features for continuous space-time video super-resolution, using flow-based residual trajectories and direct LR-to-HR splatting.

Critical strengths
  1. The motivation is clear. By predicting residual motion and directly splatting low-resolution features at their estimated sub-pixel positions onto a high-resolution grid, the method preserves sub-pixel positional information and follows the classical principle of precise alignment and accumulation in multi-frame super-resolution.

  2. The method achieves leading results on GoPro and Adobe240, with convincing quantitative and qualitative improvements over strong baselines. The discussion of failure cases also helps clarify its limitations.

Critical weaknesses
  1. Two key ablations are missing: upsample-then-splat versus direct LR-to-HR splatting, and full-trajectory prediction versus flow-residual prediction. Without these comparisons under matched training settings, it is difficult to isolate the contribution of the proposed design choices.

  2. Sub-pixel accumulation and B-spline motion modeling have both been explored in prior work. The novelty is relatively modest, making stronger ablations important for establishing the value of the combined design.

  3. The architectural presentation is difficult to follow. A detailed pipeline diagram and a side-by-side schematic comparison with prior methods would improve readability.

  4. I could not find a code attachment. Do the authors plan to release the code and pretrained models?

  5. Metric reporting is inconsistent across experiments. Please provide PSNR, SSIM, and LPIPS consistently across the main comparison tables to support a more complete assessment. In table 1, only PSNR and LPIPS are reported. However, in table 2, PSNR and SSIM are reported.

Questions for authors

None.

5 comments
AuthorsOpenReview ↗

Response to Reviewer KND3

We thank the reviewer for the positive assessment and the precise requests.

Two missing ablations, with the encoder, decoder, data and training schedule fixed (new Appendix D.1, Table 6):

  • Upsample-then-splat versus direct LR-to-HR splatting. Upsampling each frame (learned conv + PixelShuffle) before the same splat gives the same PSNR-Y (LPIPS within 0.004) but costs 21% more latency, 18% more FLOPs and 10% more training memory. Direct LR-to-HR splatting is what makes precise sub-pixel fusion cheap.
  • Full-trajectory versus flow-residual prediction. Same network and parameter count, 150k iterations, same recipe; it is running and we will post it by Mon 16 Nov (as announced in the general response).

Novelty. We agree that sub-pixel splatting and B-spline motion exist individually. What is new is how the trajectories are obtained and where the samples land. First, and most important, our B-spline trajectories are estimated from features propagated over the whole sequence (Sec. 3.1). MoTIF and BF-STVSR also splat along learned trajectories at intermediate times, but estimate them without multi-frame context (BF-STVSR from two frames, where acceleration is not observable); to our knowledge, ours is the first C-STVSR method to estimate explicit trajectories from multi-frame context. Appendix A shows that the trained estimator relies on this context, and that given the same four frames it outperforms QVI's closed-form quadratic model on every video. Figures 4, 5 and 9 illustrate its effect under complex motion. Second, we splat each low-resolution feature directly at its precise sub-pixel position, whereas MoTIF and BF-STVSR lift each frame with a nonlinear implicit representation first. The controls requested by 2eK7 test this placement: our splat outperforms fusing at LR then PixelShuffle, and rounding to the HR grid, in PSNR-Y on every dataset and frame type (Table 6 of the revised paper, reproduced below).

PSNR-Y ↑ / LPIPS ↓, ×4 space / ×8 time on GoPro and Adobe240, key frames (τ = 0) and interpolated frames, ordered by how much of each sample's sub-pixel position survives until fusion; * = mean of two seeds; best of the controlled variants in bold, as in Table 6 (the last row is the efficiency control).

ControlSub-pixel position at fusionKey GoProKey Adobe240Interp. GoProInterp. Adobe240
Ours: splat at the predicted sub-pixel position*full precision35.58 / 0.118534.68 / 0.096431.41 / 0.142531.86 / 0.1138
(1) Fuse at LR, conv + PixelShuffle*encoded in channels35.47 / 0.120834.53 / 0.098431.34 / 0.148031.60 / 0.1230
(1) Fuse at LR, RVRT's upsamplerencoded in channels35.45 / 0.121234.49 / 0.099631.27 / 0.149331.54 / 0.1248
(3) Splat rounded to the HR grid*quantised (¼ LR pixel)35.50 / 0.120134.60 / 0.096631.23 / 0.145631.61 / 0.1159
(3) Splat rounded to the LR pixelnone34.81 / 0.131633.75 / 0.113129.61 / 0.171029.68 / 0.1478
(2) Upsample (linear), then the same splat (efficiency control)full precision35.59 / 0.116934.69 / 0.094231.40 / 0.145631.87 / 0.1151

Architecture presentation. New Figure 2 is a detailed pipeline diagram: for one interval, it shows the model's intermediate outputs (optical flow, spline residual and visibility, splatted features, output) and which steps run once per interval and once per output time. New Appendix F (Table 13) gives the side-by-side comparison, in table form, with prior methods (SPMC, MoTIF, BF-STVSR, GS-STVSR, V³): from which frames each estimates motion, which trajectory model it predicts and on which grid, and where it fuses the frames at an output time.

Code. Anonymized inference code and pretrained weights are in the supplementary material.

Metric consistency. Tables 1 and 2 now report PSNR-Y, SSIM-Y and LPIPS for every method with public weights (published values otherwise). Ours is best in PSNR-Y and SSIM-Y in every setting of Tables 1 and 2, and in LPIPS on GoPro and Adobe240 in Table 1. At ×12 space, BF-STVSR has the best LPIPS; in the crops of these outputs (new Figure 8), ours is higher in PSNR-Y and shows more detail (e.g. the window frames of the facade in the first crop), so we read the LPIPS ranking at this scale with caution (Appendix E.4).

We thank the reviewer for the constructive feedback, which improved the paper.

AuthorsOpenReview ↗

Follow-up to Reviewer KND3: residual vs full-displacement comparison

The residual-on-flow vs full-displacement comparison you asked for is now in the general thread: the residual is better on average by a small margin and markedly better on large motion (+0.18 / +0.37 dB PSNR-Y on the 10% of frames with the largest motion). The same post explains that the Table 6 reproduced in our reply was trained with a wrong version of the trajectory implementation; we will share the updated table on Wednesday 18th.

The updated version of the Table 6 reproduced in our reply to you is now in the general thread, re-trained with the released implementation. The conclusions are unchanged: our splat is better than fusing at low resolution then PixelShuffle (+0.24 / +0.36 dB PSNR-Y on interpolated frames, GoPro / Adobe240) and than rounding to the high-resolution pixel (+0.15 / +0.24 dB), and linearly lifting first matches it at higher cost.

Reviewer_2eK7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a continuous spatiotemporal video super-resolution (C-STVSR) method: At any intermediate moment, the low-resolution features are directly splatted onto the fixed ×4 high-resolution grid at the predicted sub-pixel positions. The features are obtained by propagating the RVRT encoder throughout the sequence. The trajectory of each pixel is modeled as a linear motion of the optical flow plus a centered cubic B-spline residual. Each splatted feature is weighted by a visibility term derived from the spline basis. Then, it is decoded by U-Net. Other spatial scales are obtained by double cubic resampling of the ×4 output.

Critical strengths
  1. The motivation is clear. The author points out that the existing C-STVSR method resamples each frame sample before fusion, thereby losing the sub-pixel position information. This paper propose the idea that residual motion splatting for continuous space-time video super-resolution.

  2. The evaluation has a wide coverage, including standard and extreme magnifications, as well as perception and temporal consistency indicators. The improvements are relatively consistent across all aspects, and the training computing power is much less than that of V3.

Critical weaknesses
  1. The splatting refers to the deep features, not the original observations. The reason why the classic multi-frame super-resolution can restore frequencies above Nyquist is that each sample is the original, aliased pixel observation, and there are different sub-pixel phases between them. In this paper, splatting has been guided by the optical flow of RVRT for deformable attention and has integrated the information of multiple previous frames. That is to say, the sub-pixel information across frames is likely to have been implicitly utilized once within the encoder. Performing an additional explicit accumulation would not be clear about how much additional information is added.

  2. The motivation is clear but limited. The main components of this paper mostly come from existing works: sub-pixel splats from low resolution to high resolution are seen in SPMC, B-spline trajectories and visibility derived from spline basis are from BF-STVSR, and the encoder is RVRT. The truly new part is the flow-based centered residual spline and the estimation of trajectories in the low-resolution domain. The former is a reasonable but incremental design, and the latter mainly brings efficiency improvement. Works that are mainly based on combination may still have sufficient contributions, provided that it can be proved that a certain design choice is the key to performance improvement; however, since the paper has not independently verified the effect of the splat method under fixed other components, it is currently impossible to confirm this, and the contribution of this paper is thus closer to an engineeringly well-performing system combination. We suggest that the authors, while keeping the encoder, trajectory network, decoder, and training settings unchanged, supplement the following control experiments: (1) Replace the splat with low-resolution fusion followed by pixel-shuffle; (2) Upsample the features to high resolution first, and then splat along the same trajectory; (3) Quantize the splat points to integer high-resolution grids to test the contribution of sub-pixel accuracy itself. If the complete model is significantly better than these three variants, it can demonstrate that the splat design is the key to performance improvement, and the novelty of this paper will also be substantiated.

Questions for authors

See Critical Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2eK7 (1/2)

We thank the reviewer for a precise review and for stating a clear criterion: the splat is the key design choice if the full model is better than the three controls with everything else fixed. We ran all three (150k iterations) and report them in the new Appendix D.1 and Table 6 of the revised paper, reproduced below: (1) and (3) test the precision of the placement, and (2) its cost.

Measured against this criterion. Our splat outperforms (1) and the HR rounding of (3) in every column of Table 6, in PSNR-Y and LPIPS, by 0.07–0.32 dB. These gaps can look small in PSNR, but they are systematic: each exceeds the seed spread (at most 0.045 dB). PSNR also understates them: it averages the error over all pixels, and most pixels lie in smooth regions that every variant reconstructs well. Removing the sub-pixel position altogether costs 0.8–2.2 dB. The 0.07–0.32 dB margins come from changing a single operation with everything else fixed, and are of the same order as the gap between ours and the previous state of the art, V³, on GoPro and Adobe240 in Table 1 (0.36–0.67 dB PSNR-Y). Control (2) lifts each frame with a linear upsampler before the same splat. The positions do not change, so no quality difference is expected, and PSNR-Y is indeed the same, but lifting costs 21% more latency. What matters is the precision of the placement, and splatting at LR keeps it at the lowest cost.

PSNR-Y ↑ / LPIPS ↓, ×4 space / ×8 time on GoPro and Adobe240, key frames (τ = 0) and interpolated frames, ordered by how much of each sample's sub-pixel position survives until fusion; * = mean of two seeds; best of the controlled variants in bold, as in Table 6 (the last row is the efficiency control).

ControlSub-pixel position at fusionKey GoProKey Adobe240Interp. GoProInterp. Adobe240
Ours: splat at the predicted sub-pixel position*full precision35.58 / 0.118534.68 / 0.096431.41 / 0.142531.86 / 0.1138
(1) Fuse at LR, conv + PixelShuffle*encoded in channels35.47 / 0.120834.53 / 0.098431.34 / 0.148031.60 / 0.1230
(1) Fuse at LR, RVRT's upsamplerencoded in channels35.45 / 0.121234.49 / 0.099631.27 / 0.149331.54 / 0.1248
(3) Splat rounded to the HR grid*quantised (¼ LR pixel)35.50 / 0.120134.60 / 0.096631.23 / 0.145631.61 / 0.1159
(3) Splat rounded to the LR pixelnone34.81 / 0.131633.75 / 0.113129.61 / 0.171029.68 / 0.1478
(2) Upsample (linear), then the same splat (efficiency control)full precision35.59 / 0.116934.69 / 0.094231.40 / 0.145631.87 / 0.1151

What each control shows.

  • (1) Fusing at LR, then PixelShuffle. Before the upsampler, each feature's sub-pixel offset survives only implicitly, in how its weight is split over neighbouring LR cells; PixelShuffle then assigns each channel group to a fixed sub-position, so the position must be decoded from the channels, and features of both frames that fall in the same LR cell are summed first. Our splat instead writes each feature directly at its position on the HR grid. Ours is also better with the full 500k recipe (one run: +0.06 / +0.15 dB PSNR-Y, −0.0017 / −0.0034 LPIPS on GoPro / Adobe240).
  • (2) Upsampling each frame first, then the same splat. A learned linear upsampler (3×3 conv + PixelShuffle) lifts the features before splatting, so positions keep full precision and only the resolution of the splatted features changes. It gives the same PSNR-Y as ours, at 21% more latency, 18% more FLOPs and 10% more training memory.
  • (3) Rounding the landing positions. Rounding them to the HR grid costs 0.08–0.25 dB; rounding them to the LR pixel, which removes the sub-pixel position entirely, costs 0.8–2.2 dB, key frames included.

In Table 6, the two variants that keep each feature's position at full precision (ours and (2)) are the best in every column, and the one that removes it entirely (rounding to the LR pixel) is the worst.

(continued in 2/2)

AuthorsOpenReview ↗

Response to Reviewer 2eK7 (2/2)

W1: accumulation on encoder features. We separate two kinds of output frames. Key frames are at the times of the input frames (τ = 0). Interpolated frames are at intermediate times t+τ that no input frame shows; they are what C-STVSR adds over VSR.

  • Key frames. The reviewer is right, and our framing suggested more than our experiments support at key frames. RVRT already passes information across frames: removing the second input frame's contribution to the splat at key frames changes PSNR-Y by only 0.01 dB on GoPro and Vid4 and 0.02 dB on Adobe240. We removed the two sentences that suggested otherwise (abstract: "detail beyond a single frame's sampling limit can be recovered"; conclusion: the splat "gathers detail from neighbouring frames") and report this test in Appendix D.1. Still, with the same architecture and training recipe, our splat outperforms fusing at LR then PixelShuffle (1) at key frames too (Table 6). There, each frame's own features land at their own positions, so this gain comes from how the features reach the HR grid (written at their HR positions rather than through a learned PixelShuffle), not from motion.
  • Interpolated frames. RVRT aligns each input frame with its neighbours, but every feature stays at the position it has at its own input time; no part of the encoder moves content to t+τ. The splat does: it places each feature of both input frames at its predicted sub-pixel position at t+τ. Rounding that position to the HR grid costs up to 0.25 dB on interpolated frames, and decoding each interpolated frame from the nearer input frame alone costs 0.5 to 1.2 dB on average (Vid4 to GoPro) and 1.1 to 2.2 dB at the midpoint τ = 1/2, so both input frames contribute there.

W2: novelty. We agree the components have precedents, and would like to add one element to the summary that we see as central: trajectories are estimated from features propagated over the whole sequence (Sec. 3.1). MoTIF and BF-STVSR also splat along learned trajectories at intermediate times, but estimate them without multi-frame context (BF-STVSR from two frames, where acceleration is not observable), and V³ has no explicit motion model. To our knowledge, ours is the first C-STVSR method to learn to predict explicit trajectories from context beyond two frames (QVI, for frame interpolation, fits a fixed quadratic model to optical flow over four frames). Appendix A shows that the trained estimator relies on this context, and that given the same four frames it outperforms QVI's closed-form quadratic model on every video. Figures 4, 5 and 9 illustrate the consequence under complex motion, where prior methods duplicate or blur the moving object.

The two parts the reviewer identifies build on this.

  • Residual on flow: the spline corrects a measured motion instead of learning it outright. The controlled comparison with full displacement is still running; we will post it by Mon 16 Nov.
  • Splatting at LR: each feature lands directly at its precise sub-pixel position, whereas MoTIF and BF-STVSR lift each frame with a nonlinear implicit representation before splatting (SPMC splats pixels, at the times of its input frames only). The controls above show that the most precise placement drives quality, and control (2), the lift-then-splat variant the reviewer asked for, that lifting first performs on par at a higher cost. Splatting at LR also brings efficiency with higher quality: 79 ms per frame against 184 ms for V³ and 356–391 ms for MoTIF and BF-STVSR (Table 3).

We hope the new controls address the reviewer's concerns, and would be grateful if the reviewer would consider updating the score in light of them.

Revisions. New Appendix D.1 with all controls. The two contributions now read: (1) splatting low-resolution features at their precise sub-pixel positions, which outperforms fusing at low resolution, while lifting each frame first matches it at higher cost; (2) to our knowledge, the first learned trajectory model in C-STVSR to predict from context beyond two frames. The abstract and conclusion are rephrased accordingly, without the sentence on the sampling limit.

AuthorsOpenReview ↗

Follow-up to Reviewer 2eK7: residual comparison and Table 6 correction

We posted the residual-on-flow vs full-displacement comparison in the general thread: the residual is better on average by a small margin and markedly better on large motion (+0.18 / +0.37 dB PSNR-Y on the 10% of frames with the largest motion). The same post explains that the current Table 6 was trained with a wrong version of the trajectory implementation; we will share the updated table on Wednesday 18th.

Reviewer_5927OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a method for continuous space-time video super-resolution that combines long-range temporal context with B-spline residual motion estimation. Low-resolution features from adjacent frames are directly splatted onto a high-resolution grid along predicted trajectories, with visibility weights handling occlusions, before being decoded into RGB frames. The reported results show improved reconstruction fidelity and perceptual quality on GoPro and Adobe240, together with substantial inference speedups over the evaluated baselines.

Critical strengths
  1. The method builds on classical multi-frame super-resolution principles and integrates sub-pixel feature accumulation with continuous temporal interpolation.
  2. The controlled ablation provides evidence that additional temporal context improves trajectory estimation while keeping the splatted content features unchanged.
  3. The reported experiments demonstrate competitive reconstruction and perceptual quality, alongside substantial speed improvements under full-frame evaluation.
Critical weaknesses
  1. The paper does not provide anonymized inference code or model weights, making it difficult to independently verify the reported results and runtime. I would consider raising my score if the authors provide complete, runnable code and weights.
  2. Although parameter counts and inference speed are reported, a comparison of peak GPU memory usage under the same hardware, input resolution, and space-time scaling factors is missing.
  3. The existing ablations mainly examine temporal context and visibility weighting, without isolating the contributions of directly splatting low-resolution features onto the high-resolution grid and predicting residual motion. Comparisons against upsampling before splatting and predicting the full displacement, with matched encoders, decoders, and parameter budgets, would help clarify the source of the improvements.
  4. STNO-c (IEEE TIP 2025) also supports continuous space-time video super-resolution. A comparison under the same evaluation protocol is needed to more convincingly establish the proposed method’s performance advantage.
Questions for authors

Could the authors provide anonymized access to the inference code together with pretrained model checkpoints, to enable reproduction of the reported results?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 5927

We thank the reviewer for the constructive review.

Code and weights. The supplementary material contains anonymized, runnable inference code and the trained checkpoint that produced every "Ours" number in Tables 1–3. Its README gives the download links for GoPro, Adobe240 and Vid4, the train/test splits, the degradation and the frame selection for any temporal and spatial scale, then one command per benchmark: following it reproduces the evaluation of Tables 1 and 2, and a benchmark command measures latency and peak GPU memory as in Table 3.

Peak GPU memory on the same hardware, resolution and scales. New Table 3 measures every method on one RTX 6000 Ada GPU (fp32, ×4 space, ×8 time), on the full GoPro test set as evaluated in Table 1 (28-frame windows, 24 for V³, whose tiling requires it; 720p output) and on the 14 × 80×80 crops on which V³ reports its runtime. Best per column in bold (lower is better everywhere).

MethodFull set: ms / frameFull set: peak memory / window (GiB)Full set: GFLOPs / frameCrops: latency (s)Crops: peak memory (GiB)Crops: GFLOPs / frame
MoTIF355.99.7041156.211.00436
STNO-c72.97.9512791.070.74136
BF-STVSR390.89.2737966.140.95422
V³ (JAX)184.16.02n/a*3.856.02n/a*
Ours78.818.767670.891.3187

We report it as measured: on 28-frame 720p windows our peak memory is the highest (18.76 GiB); it grows with the window, since the encoder processes the whole window jointly and keeps every frame's features until all intervals are decoded. It can be reduced with shorter windows, or bounded as V³ does by keeping idle features in CPU memory (V³ also processes fixed-size tiles, at 2.3× our latency); we did not implement this offloading. We take 8% longer per frame than STNO-c (78.8 vs 72.9 ms), which is 3.3–3.6 dB lower in PSNR-Y over all GoPro / Adobe240 frames; on the V³ crops, ours is 17% faster than STNO-c. On the full test set, MoTIF and BF-STVSR are 4.5–5.0× slower than ours. Among the PyTorch methods, ours needs the fewest FLOPs. For V³ (JAX), memory is read from the programs XLA compiles for the same inputs; *we do not report its FLOPs, since XLA and PyTorch count operations differently.

Ablations with matched encoders, decoders and budgets (Appendix D.1, Table 6):

  • Upsampling before splatting. Same encoder, trajectory network and decoder; the learned upsampler (conv + PixelShuffle) adds 0.6 M parameters. It gives the same PSNR-Y as ours (LPIPS within 0.004) at 21% more latency, 18% more FLOPs and 10% more training memory.
  • Full displacement instead of the residual on flow. Same network and parameter count, 150k iterations, same recipe; running, we will post it by Mon 16 Nov (as announced in the general response).
  • Fusing at LR, then PixelShuffle (also 0.6 M more parameters than ours): our splat outperforms it in every column of Table 6 (key and interpolated frames, GoPro and Adobe240), in PSNR-Y and LPIPS.

STNO-c. We evaluated STNO-c with its official code and weights under our protocol (Tables 1–2); running the same pipeline on STNO-fix, the fixed-scale model of the same paper, reproduces its published results within 0.11 dB, so our protocol does not disadvantage it. Ours is better in PSNR-Y, SSIM-Y and LPIPS in every cell of Tables 1 and 2: by 3.3 / 3.6 dB on all GoPro / Adobe240 frames, and by 1.8 / 2.3 dB on the Center subset used in the STNO paper.

We hope the code and the new measurements resolve the reviewer's concerns, and that the reviewer may consider updating the score accordingly.

AuthorsOpenReview ↗

Follow-up to Reviewer 5927: residual vs full-displacement comparison

The residual-on-flow vs full-displacement comparison you asked for is now in the general thread: the residual is better on average by a small margin and markedly better on large motion (+0.18 / +0.37 dB PSNR-Y on the 10% of frames with the largest motion). The same post explains that the Table 6 runs cited in our reply were trained with a wrong version of the trajectory implementation; we will share the updated table on Wednesday 18th.

The updated version of the Table 6 runs cited in our reply is now in the general thread, re-trained with the released implementation. The conclusions are unchanged: our splat is better than fusing at low resolution then PixelShuffle (+0.24 / +0.36 dB PSNR-Y on interpolated frames, GoPro / Adobe240) and than rounding to the high-resolution pixel (+0.15 / +0.24 dB), and linearly lifting first matches it at higher cost.

Comments on the paper

3 comments

We thank the reviewers for their careful reading and concrete suggestions. We ran the requested experiments (one result is still running, see below) and revised the paper (changes in blue). In short:

Controls of the splat, with the encoder, trajectory network, decoder, data and training schedule fixed (2eK7, KND3, 5927; new Appendix D.1, Table 6):

  1. Fusing at LR, then PixelShuffle (2eK7-1), including RVRT's own upsampler: our splat outperforms both variants in every column of Table 6 (key and interpolated frames, GoPro and Adobe240; PSNR-Y and LPIPS), and also with the full 500k schedule (one run).
  2. Upsampling each frame (learned conv + PixelShuffle) before the same splat (2eK7-2) gives the same PSNR-Y but costs 21% more latency, 18% more FLOPs, 10% more training memory and 0.6 M more parameters.
  3. Rounding the splat to the integer HR grid (2eK7-3) costs up to 0.25 dB on interpolated frames; rounding it to whole LR pixels costs 0.8–2.2 dB.

What the splat adds beyond the encoder (2eK7). We call key frames the output frames at the times of the input frames (τ = 0), and interpolated frames those at intermediate times t+τ that no input frame shows. At key frames, the reviewer is right and our framing suggested more than our experiments support: removing the second input frame's contribution to the splat changes PSNR-Y by only 0.01 dB on GoPro and Vid4 and 0.02 dB on Adobe240, because the encoder already passes information across frames. We removed the two sentences that suggested otherwise (abstract and conclusion) and report this test in Appendix D.1. At interpolated frames, the encoder features stay at the positions they have at their own input times; the splat is the step that places each feature of both input frames at its sub-pixel position at t+τ, and items 1 and 3 show that the precision of this position matters: losing part of it lowers quality in every column of Table 6, and removing it entirely costs 0.8–2.2 dB.

Other revisions.

  • Tables 1 and 2 report PSNR-Y, SSIM-Y and LPIPS for every method with public weights (KND3).
  • STNO-c, evaluated with its official code and weights under our protocol, added to Tables 1–2 (5927).
  • New Table 3: peak GPU memory and speed of all methods on the same GPU, resolution and scales (5927).
  • New Figure 2: detailed pipeline diagram, and new Appendix F: side-by-side comparison with prior methods (KND3).
  • Anonymized inference code and weights in the supplementary material (KND3, 5927). Every "Ours" number in Tables 1–3 and in the video-quality metrics of Table 11 comes from this checkpoint and evaluation code, and the README reproduces Tables 1–3, data preparation included.

Correction. We re-measured V³ under the same protocol as all other methods (Table 3). Our earlier timing of V³ was too slow because we had not configured its JAX runtime correctly, so our speed-up over V³ is 2.3×, not an order of magnitude as stated in the submission; the abstract, contributions and Section 4 are corrected.

Still running. The comparison of the residual on flow with full displacement (same network and parameter count, 150k iterations, same recipe as Table 6), asked for by KND3 and 5927, will be posted here on Mon 16 Nov, whatever the outcome, and added to the paper.

This post delivers the residual-on-flow vs full-displacement comparison announced in our first response, and corrects Table 6 (see the note at the end).

Residual on flow vs full displacement (KND3, 5927, 2eK7 W2). Same network and parameter count, 150k iterations, same recipe, released implementation, one seed each. ×4 space / ×8 time.

1. Average over all output frames. The residual is better on average, but the gap is limited: +0.06 / +0.07 dB PSNR-Y on GoPro / Adobe240 (above the 0.045 dB seed spread), and better LPIPS on both.

Trajectory (PSNR-Y ↑ / LPIPS ↓)GoProAdobe240
Residual on optical flow (ours)32.28 / 0.132632.38 / 0.1083
Full displacement32.22 / 0.133532.31 / 0.1095

2. By motion magnitude. Averages are dominated by small motion, which both handle well. To see what averaging hides, we reuse the per-frame scores of the full test-set evaluation above (same inference, nothing re-run) and group the interpolated frames by the motion of their keyframe interval. Each interval's motion is the mean optical-flow magnitude between its two input keyframes (pretrained RAFT, LR pixels), and its 7 interpolated frames go to the same bin. Bins are split at the 50th and 90th percentiles over all intervals of each dataset: small (50% of frames), medium (40%) and large (10%) motion. No frame or clip is selected.

PSNR-Y ↑ / LPIPS ↓GoPro smallGoPro mediumGoPro largeAdobe240 smallAdobe240 mediumAdobe240 large
Residual (ours)31.59 / 0.133732.16 / 0.135431.45 / 0.135932.63 / 0.096731.50 / 0.119731.35 / 0.1376
Full displacement31.53 / 0.134932.11 / 0.136531.27 / 0.136132.61 / 0.097331.43 / 0.121630.97 / 0.1409

The residual is ahead in every bin, in PSNR-Y and LPIPS, on both datasets, and the gap grows with motion: on the 10% of frames with the largest motion it reaches +0.18 / +0.37 dB PSNR-Y (GoPro / Adobe240), about 3.5 and 14 times the gain on small motion, and the residual is better on 61% / 77% of these frames. For example, on a fixed 105-frame segment around the fast-moving truck of Figure 4 (chosen before these experiments), the residual gains +1.39 dB PSNR-Y. Large motion is the hardest case for C-STVSR and where prior methods duplicate or blur moving objects (Figures 4, 5 and 9), so this is the regime where the residual matters most. Anchoring the trajectory on the measured flow is therefore a sound default: it matches or slightly improves on full displacement for typical motion, and handles large displacements markedly better. We have added this comparison to the paper (new Appendix G, with a pointer in Section 4 and in the Discussion).

Note on Table 6. While preparing this comparison, we noticed that the Table 6 runs were trained with a wrong version of the trajectory implementation: not the one described in the paper and released with the code. This version is common to all Table 6 rows, so the comparisons within the table remain controlled; the re-trained values will show whether the margins carry over to the released implementation. The two runs above use the correct, released implementation. We are re-training all Table 6 controls with it, will share the updated Table 6 here on Wednesday 18th, and will update the paper accordingly. We apologise for the confusion.

This post delivers the updated Table 6 announced on Monday.

Updated Table 6 (KND3, 5927, 2eK7 W2). All rows are re-trained with the released implementation. Only the fusion step changes: encoder, trajectory network, decoder, data and schedule (150k iterations) are identical. One seed each, ×4 space / ×8 time. We focused the re-training on the requested experiments, all reported here (PSNR-Y ↑ / LPIPS ↓; bold: best among ours, (1) and (3)):

Fusion stepKeyframes GoProKeyframes Adobe240Interp. GoProInterp. Adobe240
Ours: splat at the predicted sub-pixel position35.61 / 0.118934.70 / 0.096431.80 / 0.134632.05 / 0.1100
(1) Fuse at LR, then conv + PixelShuffle35.50 / 0.120934.55 / 0.098931.56 / 0.145031.69 / 0.1224
(3) Splat rounded to the HR pixel35.56 / 0.119834.64 / 0.097231.65 / 0.136731.81 / 0.1124
Same splat, features lifted to high resolution first (efficiency comparison)
(2) Lift each frame (conv + PixelShuffle), then same splat35.59 / 0.117334.68 / 0.094831.78 / 0.138032.05 / 0.1118
  • Fusing at low resolution before upsampling (1) is worse in every column, in PSNR-Y and LPIPS: −0.24 / −0.36 dB on interpolated frames (GoPro / Adobe240).
  • Rounding the splat to the high-resolution pixel (3) is worse in every column: −0.15 / −0.24 dB on interpolated frames. The sub-pixel precision of the splat itself matters.
  • Lifting each frame to high resolution before the same splat (2) is on par in quality (PSNR-Y within 0.02 dB; LPIPS slightly better on key frames, slightly worse on interpolated frames), at +13% latency, +18% FLOPs and +10% training memory. Splatting the low-resolution features directly gives the same quality at lower cost.

The conclusions of the paper are unchanged, and the gap to fusing at low resolution is larger than in the previous table. We have updated Table 6, its discussion in Appendix D.1, and the cost figures in Section 4 in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.