- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Align-FG, an efficient game frame generation framework that combines Target-Aligned Motion Generation (TAMG) for training-time motion prior construction with a Progressive Motion Fusion and Refinement (PMFR) network for coarse-to-fine motion estimation and frame synthesis. At inference, the method constructs motion priors from endpoint motion vectors under a linear motion assumption. Experiments on Unreal Engine demo scenes and real-game data demonstrate improved perceptual quality and efficient inference compared with existing baselines.
Critical strengths
-
The paper tackles an important real-world task of game frame interpolation.
-
The method is light-weight, simple, and achieve good performance against baselines.
-
The supplement video clearly shows the advantage of the proposed method.
Critical weaknesses
-
The motivation and comparison with existing VFI methods are insufficiently justified. The paper attributes the limitations of image-based video frame interpolation (VFI) to large motion, rapid camera rotation, disocclusion, and complex rendered effects. However, these are general challenges in frame interpolation rather than limitations specific to VFI. The lack of access to rendering-pipeline geometry is a more distinctive limitation of RGB-only methods, but the paper does not sufficiently establish why existing VFI approaches are inadequate for the targeted scenarios. For example, TimeTracker (CVPR 2025) explores continuous point tracking for nonlinear motion using event data, while Realtime Video Frame Interpolation Using One-Step Diffusion Sampling (ICLR 2026) models high-order continuous pixel trajectories to handle complex motion. Although these methods are not necessarily directly comparable to engine-assisted game frame generation, they illustrate progress beyond the VFI baselines considered in this work. The authors should provide a more nuanced discussion of recent VFI methods and better justify the advantages of their approach, ideally with targeted experiments demonstrating where engine-provided information offers substantial benefits over modern alternatives.
-
The linear motion assumption remains at inference, potentially limiting the method's effectiveness under nonlinear motion. Although the paper identifies the linear motion assumption as a key limitation of existing approaches, Align-FG still constructs its inference-time motion prior from endpoint motion vectors under this assumption. Furthermore, training uses target-aligned motion priors constructed with intermediate-frame motion information, whereas inference relies on approximate linear priors, introducing a potential train–test mismatch. Table 3 shows a substantial gap between the TAMG/Linear and TAMG/TAMG configurations, suggesting that interpolation quality remains sensitive to the accuracy of the motion prior. The authors should investigate how effectively the refinement network compensates for inaccurate priors under different degrees of motion nonlinearity. For example, they could quantify the discrepancy between the linear and target-aligned motion priors, stratify test sequences by this discrepancy, and compare the performance of different training and inference configurations within each group.
-
The experimental evaluation could be more comprehensive and better controlled. The runtime comparisons involve different inference backends, including PyTorch with CUDA Graphs, Slang, and GLSL, which complicates attributing the observed latency differences to the methods themselves. The authors should provide more controlled runtime comparisons, or more clearly qualify the conclusions drawn from measurements obtained with different backends.
-
The technical presentation could be clearer. For example, the relationship between the two directional motion priors generated by TAMG and the stage-specific \hat{M}_{eng}^{(\ell)} used in PMFR is insufficiently explained. The training loss description is also incomplete, as the loss weights and perceptual loss implementation are not specified. These ambiguities hinder understanding and reproducibility.
Questions for authors
Please see weaknesses.
3 comments
Response to Reviewer vwn2-part2
Q2: Linear inference priors
TAMG reduces geometric misalignment with the ground-truth target during training but does not provide the true nonlinear trajectory at inference. PMFR instead learns appearance-dependent residual corrections to refine unreliable motion estimates in regions affected by occlusion, disocclusion, and complex rendering effects, without explicitly recovering the underlying nonlinear trajectories.
Fig. 7 illustrates that warped endpoints may be mutually aligned yet remain displaced from the ground-truth target under the Linear prior. TAMG reduces this additional training-time discrepancy, providing a more accurate geometric reference for learning motion refinement and blending. Fig. 8 shows that TAMG reduces the early-stage loss from 0.4825 to 0.4125 (14.5%) and improves final validation LPIPS from 0.0457 to 0.0384 (16.0%). Table 4 further confirms improvements across all four quality metrics under the same architecture, optimization schedule, and Linear evaluation prior.
Rather than explicitly recovering nonlinear trajectories, PMFR learns appearance-dependent motion corrections that may remain useful under approximate Linear priors. The advantage of Ours* highlights the importance of accurate target-position information, but does not imply that PMFR can recover such privileged information from endpoint observations alone. While training and inference priors are not identically distributed, the controlled results demonstrate that TAMG training improves interpolation performance under the same Linear evaluation prior. We therefore claim improved geometric training references and interpolation quality, rather than explicit nonlinear trajectory recovery or gains specifically attributable to nonlinear-motion regions. The revised manuscript clarifies this distinction.
Q3: Runtime control
The original RIFE/IFRNet benchmarks followed their respective default full-resolution inference settings (1280 × 720). To provide a more comparable evaluation, we additionally benchmarked variants with half-resolution motion prediction (640 × 360) and full-resolution synthesis (1280 × 720). On the NVIDIA L20 using PyTorch with CUDA Graphs (CG), RIFE, IFRNet, and Align-FG take 5.009 ms, 3.023 ms, and 1.604 ms, respectively. Align-FG thus achieves 3.12× and 1.88× speedups over RIFE and IFRNet. These experiments better control for prediction resolution and backend differences, although model capacity and computational complexity remain unmatched.
The originally reported ~1.3 ms NFRU latency excluded optical-flow computation. We apologize for this omission and provide the corrected timings below:
| NFRU backend | Excluding flow computation (ms) | Including flow computation (ms) |
|---|---|---|
| PyTorch + CG | 8.261 | 10.002 |
| Slang + CG | 1.340 | 2.982 |
All NFRU configurations use 320 × 180 fusion-network inputs, 640 × 360 optical-flow estimation, and 1280 × 720 output. Thus, NFRU's fusion network operates at 320 × 180, whereas Align-FG predicts motion at 640 × 360; both produce full-resolution outputs. Flow-excluded timings omit only the computation cost of optical flow, not its use as input.
We retain NFRU's official Slang implementation because it achieves lower latency than its PyTorch counterpart, which incurs substantial preprocessing and postprocessing overhead. Including optical-flow computation, the complete Slang-based NFRU pipeline takes 2.982 ms, compared with 1.604 ms for Align-FG, yielding a 1.86× speedup. We explicitly acknowledge that this is a cross-backend comparison. The reported FLOPs account only for network computation at the respective input resolutions, rather than complete pipeline costs.
The revised manuscript clarifies the resolutions used for motion prediction, synthesis, and metric evaluation, distinguishing the half-resolution quality evaluation in Table 1 from the 720p synthesis benchmark in Table 3. Quality and latency results obtained under different configurations should not be combined into a single matched quality–latency operating point. Our efficiency conclusions are limited to the tested FP32 implementations on the NVIDIA L20 and do not establish mobile-device latency or energy efficiency. We also acknowledge that Mob-FG remains faster at 0.568 ms.
Response to Reviewer vwn2-part3
Q4: Motion notation and training details
During training, TAMG constructs two target-to-endpoint motion fields, each containing horizontal and vertical displacement components. Their concatenation forms a four-channel geometric prior. At each scale, the prior is obtained by resampling the same underlying motion field and adjusting displacement magnitudes to the corresponding pixel units, rather than independently estimating a new field. The four scales (, , , and ) are defined relative to the network input resolution, which is half the original image resolution. At inference, the Linear prior follows the same four-channel representation and multiscale processing, but does not require target-frame correspondences.
The geometric prior is resampled using nearest-neighbor interpolation, whereas propagated motion estimates are bilinearly upsampled with appropriate displacement scaling. At each finer scale, a learned motion-fusion gate adaptively combines the propagated motion estimates with the geometric prior, followed by residual motion refinement. This gate is distinct from the image-blending mask, which controls the fusion of warped RGB frames. The DMS coverage mask represents projected pixel support rather than an explicit occlusion label.
For final synthesis, the half-resolution motion fields are bilinearly upsampled, with their displacement magnitudes multiplied by two before warping the original full-resolution frames. The image-blending mask is similarly upsampled, but without displacement scaling.
Our training objective combines L1 reconstruction and VGG-based perceptual losses, with equal weights of 1 for both terms. To ensure reproducibility, we will specify the VGG architecture and pretrained weights, selected feature layers and their weights, input normalization, feature-distance computation, loss reduction, and parameter-freezing configuration. We will also clarify any auxiliary losses retained in the matched-loss baselines. These details will facilitate reproduction of the motion representation and training configuration.
Response to Reviewer vwn2-part1
We sincerely thank the reviewer for the careful evaluation and constructive feedback. We appreciate your insightful suggestions regarding our motivation, recent VFI baselines, and experimental fairness, and address these concerns in detail below.
Q1: Motivation and SPEED
We agree that large motion, disocclusion, and complex rendering effects pose general challenges to VFI. Our key advantage lies in leveraging engine-provided motion vectors (MVs) and depth, which offer geometric information without relying solely on RGB-based estimation. We will clarify that our contribution focuses on efficiently exploiting these geometric signals and refining unreliable motion estimates in regions affected by occlusion, disocclusion, and complex rendering effects, rather than suggesting that existing VFI methods cannot model nonlinear motion.
Regarding the suggested Realtime Video Frame Interpolation Using One-Step Diffusion Sampling [1], a public training and evaluation implementation was unavailable when we conducted our experiments. We therefore additionally evaluated SPEED (ACM MM 2026) [2], an open-source one-step pixel-space diffusion method, as a contemporary generative baseline.
To better control for differences in training objectives, we trained SPEED for 2,000 epochs using the same L1+VGG loss as Align-FG and the updated RIFE/IFRNet baselines, with equal weights of 1 for both terms. Apart from adaptations to the dataset, loss function, and training duration, we retained SPEED's released implementation settings. We will explicitly identify this as a matched-loss adaptation rather than its original training configuration.
As shown in Table 1, Align-FG achieves average LPIPS/FloLPIPS of 4.46/7.24, compared with 7.68/10.39 for SPEED, 7.74/11.29 for RIFE, and 9.23/13.32 for IFRNet (all scaled by 100). This represents 41.9%/30.3% reductions over SPEED. On Real Game Hard, SPEED substantially outperforms RIFE (13.60/16.80 vs. 21.64/26.31), while Align-FG achieves 7.38/11.38. SPEED also slightly outperforms NFRU in average FloLPIPS and achieves higher SSIM than Align-FG across the three Real Game groups. Thus, we do not claim superiority across all metrics or scenarios.
For 4× multi-frame interpolation on UE Demo 4x (Table 2), Align-FG achieves LPIPS/FloLPIPS of 4.69/6.50, compared with SPEED's 8.68/12.08, corresponding to 46.0%/46.2% reductions. Without additional training, our midpoint-trained model directly synthesizes frames at normalized target timestamps t∈{0.25,0.5,0.75} from the same endpoint frames. The corresponding Linear motion priors are constructed from the directly recorded full-interval endpoint MV following Eq. (3), without recursive interpolation or target-frame MV inputs. Although training uses uniformly spaced triplets, nonlinear motion and time-varying velocity mean that the target position need not coincide with a linear interpolation of endpoint motion. By leveraging recorded target-frame correspondences, TAMG provides diverse target-aligned geometric references for learning motion refinement, which may facilitate transfer across interpolation timestamps. However, these complete-model results do not isolate TAMG's specific contribution to this generalization.
Unlike flow-based methods, SPEED directly generates RGB frames. Reducing its generation resolution would require subsequent image upsampling, unlike low-resolution motion prediction followed by warping of the original high-resolution endpoints. At native 720p resolution, SPEED takes 63.227 ms versus 1.604 ms for Align-FG on the NVIDIA L20, corresponding to a 39.4× latency ratio. Their network FLOPs are 1046.9 G and 5.63 G, respectively. These measurements reflect the evaluated implementations and configurations rather than strictly capacity-matched architectural comparisons.
We will further clarify the input assumptions of all baselines, including the event information used by TimeTracker [3]. The SPEED evaluation provides additional evidence of Align-FG's quality–efficiency trade-off against a contemporary generative method, without implying that all generative approaches exhibit comparable computational costs or reconstruction quality.
References
[1] Y. Ma, S. Zhao, M. Yao, et al., "Realtime Video Frame Interpolation Using One-Step Diffusion Sampling," in The Fourteenth International Conference on Learning Representations (ICLR), 2026.
[2] Z. Zhang, H. Zhao, S. Yang, et al., "SPEED: One-Step Pixel Diffusion for High-quality Video Frame Interpolation," arXiv preprint arXiv:2607.15585, 2026.
[3] H. Liu, J. Xu, Y. Chang, et al., "TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 17649–17659.