acceptodds
Under review as a conference paper at ICLR 2027

Align-FG: Efficient Game Frame Generation with Target-Aligned Motion and Progressive Refinement

Abstract

Game frame generation improves temporal smoothness by synthesizing intermediate frames with limited additional rendering cost. Existing two-stage methods generate complementary warped candidates using image-based optical flow and engine-provided motion vectors (MVs), and subsequently fuse them for frame synthesis. However, intermediate motion derived from engine MVs under a linear motion assumption can misalign the estimated initial positions with their targets, while standalone optical-flow estimation introduces substantial computational overhead. We present Align-FG, an efficient framework that addresses both limitations with two complementary designs. First, Target-Aligned Motion Generation (TAMG) uses engine MVs and depth during training to construct a target-aligned bidirectional geometric prior, reducing initialization errors and allowing the network to focus on appearance changes that are poorly described by engine MVs, such as reflections, shadows, and semi-transparent effects. At inference, the same motion representation is constructed from endpoint MVs under a linear motion assumption, enabling deployment without access to the target frame. Second, we develop a lightweight one-stage Progressive Motion Fusion and Refinement (PMFR) network that adaptively fuses propagated flow with the geometric motion prior and predicts residual corrections across scales, incorporating appearance cues without a standalone optical-flow estimator. Experiments on our Unreal Engine-based and real-game datasets show that Align-FG improves perceptual quality over the evaluated baselines, while generating a frame in ms on an NVIDIA L20 GPU, approximately – faster than RIFE and IFRNet.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Align-FG: Efficient Game Frame Generation with Target-Aligned Motion and Progressive Refinement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vwn2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Align-FG, an efficient game frame generation framework that combines Target-Aligned Motion Generation (TAMG) for training-time motion prior construction with a Progressive Motion Fusion and Refinement (PMFR) network for coarse-to-fine motion estimation and frame synthesis. At inference, the method constructs motion priors from endpoint motion vectors under a linear motion assumption. Experiments on Unreal Engine demo scenes and real-game data demonstrate improved perceptual quality and efficient inference compared with existing baselines.

Critical strengths
  1. The paper tackles an important real-world task of game frame interpolation.

  2. The method is light-weight, simple, and achieve good performance against baselines.

  3. The supplement video clearly shows the advantage of the proposed method.

Critical weaknesses
  1. The motivation and comparison with existing VFI methods are insufficiently justified. The paper attributes the limitations of image-based video frame interpolation (VFI) to large motion, rapid camera rotation, disocclusion, and complex rendered effects. However, these are general challenges in frame interpolation rather than limitations specific to VFI. The lack of access to rendering-pipeline geometry is a more distinctive limitation of RGB-only methods, but the paper does not sufficiently establish why existing VFI approaches are inadequate for the targeted scenarios. For example, TimeTracker (CVPR 2025) explores continuous point tracking for nonlinear motion using event data, while Realtime Video Frame Interpolation Using One-Step Diffusion Sampling (ICLR 2026) models high-order continuous pixel trajectories to handle complex motion. Although these methods are not necessarily directly comparable to engine-assisted game frame generation, they illustrate progress beyond the VFI baselines considered in this work. The authors should provide a more nuanced discussion of recent VFI methods and better justify the advantages of their approach, ideally with targeted experiments demonstrating where engine-provided information offers substantial benefits over modern alternatives.

  2. The linear motion assumption remains at inference, potentially limiting the method's effectiveness under nonlinear motion. Although the paper identifies the linear motion assumption as a key limitation of existing approaches, Align-FG still constructs its inference-time motion prior from endpoint motion vectors under this assumption. Furthermore, training uses target-aligned motion priors constructed with intermediate-frame motion information, whereas inference relies on approximate linear priors, introducing a potential train–test mismatch. Table 3 shows a substantial gap between the TAMG/Linear and TAMG/TAMG configurations, suggesting that interpolation quality remains sensitive to the accuracy of the motion prior. The authors should investigate how effectively the refinement network compensates for inaccurate priors under different degrees of motion nonlinearity. For example, they could quantify the discrepancy between the linear and target-aligned motion priors, stratify test sequences by this discrepancy, and compare the performance of different training and inference configurations within each group.

  3. The experimental evaluation could be more comprehensive and better controlled. The runtime comparisons involve different inference backends, including PyTorch with CUDA Graphs, Slang, and GLSL, which complicates attributing the observed latency differences to the methods themselves. The authors should provide more controlled runtime comparisons, or more clearly qualify the conclusions drawn from measurements obtained with different backends.

  4. The technical presentation could be clearer. For example, the relationship between the two directional motion priors generated by TAMG and the stage-specific \hat{M}_{eng}^{(\ell)} used in PMFR is insufficiently explained. The training loss description is also incomplete, as the loss weights and perceptual loss implementation are not specified. These ambiguities hinder understanding and reproducibility.

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vwn2-part1

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We appreciate your insightful suggestions regarding our motivation, recent VFI baselines, and experimental fairness, and address these concerns in detail below.

Q1: Motivation and SPEED

We agree that large motion, disocclusion, and complex rendering effects pose general challenges to VFI. Our key advantage lies in leveraging engine-provided motion vectors (MVs) and depth, which offer geometric information without relying solely on RGB-based estimation. We will clarify that our contribution focuses on efficiently exploiting these geometric signals and refining unreliable motion estimates in regions affected by occlusion, disocclusion, and complex rendering effects, rather than suggesting that existing VFI methods cannot model nonlinear motion.

Regarding the suggested Realtime Video Frame Interpolation Using One-Step Diffusion Sampling [1], a public training and evaluation implementation was unavailable when we conducted our experiments. We therefore additionally evaluated SPEED (ACM MM 2026) [2], an open-source one-step pixel-space diffusion method, as a contemporary generative baseline.

To better control for differences in training objectives, we trained SPEED for 2,000 epochs using the same L1+VGG loss as Align-FG and the updated RIFE/IFRNet baselines, with equal weights of 1 for both terms. Apart from adaptations to the dataset, loss function, and training duration, we retained SPEED's released implementation settings. We will explicitly identify this as a matched-loss adaptation rather than its original training configuration.

As shown in Table 1, Align-FG achieves average LPIPS/FloLPIPS of 4.46/7.24, compared with 7.68/10.39 for SPEED, 7.74/11.29 for RIFE, and 9.23/13.32 for IFRNet (all scaled by 100). This represents 41.9%/30.3% reductions over SPEED. On Real Game Hard, SPEED substantially outperforms RIFE (13.60/16.80 vs. 21.64/26.31), while Align-FG achieves 7.38/11.38. SPEED also slightly outperforms NFRU in average FloLPIPS and achieves higher SSIM than Align-FG across the three Real Game groups. Thus, we do not claim superiority across all metrics or scenarios.

For 4× multi-frame interpolation on UE Demo 4x (Table 2), Align-FG achieves LPIPS/FloLPIPS of 4.69/6.50, compared with SPEED's 8.68/12.08, corresponding to 46.0%/46.2% reductions. Without additional training, our midpoint-trained model directly synthesizes frames at normalized target timestamps t∈{0.25,0.5,0.75}t \in \{0.25, 0.5, 0.75\} from the same endpoint frames. The corresponding Linear motion priors are constructed from the directly recorded full-interval endpoint MV following Eq. (3), without recursive interpolation or target-frame MV inputs. Although training uses uniformly spaced triplets, nonlinear motion and time-varying velocity mean that the target position need not coincide with a linear interpolation of endpoint motion. By leveraging recorded target-frame correspondences, TAMG provides diverse target-aligned geometric references for learning motion refinement, which may facilitate transfer across interpolation timestamps. However, these complete-model results do not isolate TAMG's specific contribution to this generalization.

Unlike flow-based methods, SPEED directly generates RGB frames. Reducing its generation resolution would require subsequent image upsampling, unlike low-resolution motion prediction followed by warping of the original high-resolution endpoints. At native 720p resolution, SPEED takes 63.227 ms versus 1.604 ms for Align-FG on the NVIDIA L20, corresponding to a 39.4× latency ratio. Their network FLOPs are 1046.9 G and 5.63 G, respectively. These measurements reflect the evaluated implementations and configurations rather than strictly capacity-matched architectural comparisons.

We will further clarify the input assumptions of all baselines, including the event information used by TimeTracker [3]. The SPEED evaluation provides additional evidence of Align-FG's quality–efficiency trade-off against a contemporary generative method, without implying that all generative approaches exhibit comparable computational costs or reconstruction quality.

References

[1] Y. Ma, S. Zhao, M. Yao, et al., "Realtime Video Frame Interpolation Using One-Step Diffusion Sampling," in The Fourteenth International Conference on Learning Representations (ICLR), 2026.

[2] Z. Zhang, H. Zhao, S. Yang, et al., "SPEED: One-Step Pixel Diffusion for High-quality Video Frame Interpolation," arXiv preprint arXiv:2607.15585, 2026.

[3] H. Liu, J. Xu, Y. Chang, et al., "TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 17649–17659.

AuthorsOpenReview ↗

Response to Reviewer vwn2-part2

Q2: Linear inference priors

TAMG reduces geometric misalignment with the ground-truth target during training but does not provide the true nonlinear trajectory at inference. PMFR instead learns appearance-dependent residual corrections to refine unreliable motion estimates in regions affected by occlusion, disocclusion, and complex rendering effects, without explicitly recovering the underlying nonlinear trajectories.

Fig. 7 illustrates that warped endpoints may be mutually aligned yet remain displaced from the ground-truth target under the Linear prior. TAMG reduces this additional training-time discrepancy, providing a more accurate geometric reference for learning motion refinement and blending. Fig. 8 shows that TAMG reduces the early-stage loss from 0.4825 to 0.4125 (14.5%) and improves final validation LPIPS from 0.0457 to 0.0384 (16.0%). Table 4 further confirms improvements across all four quality metrics under the same architecture, optimization schedule, and Linear evaluation prior.

Rather than explicitly recovering nonlinear trajectories, PMFR learns appearance-dependent motion corrections that may remain useful under approximate Linear priors. The advantage of Ours* highlights the importance of accurate target-position information, but does not imply that PMFR can recover such privileged information from endpoint observations alone. While training and inference priors are not identically distributed, the controlled results demonstrate that TAMG training improves interpolation performance under the same Linear evaluation prior. We therefore claim improved geometric training references and interpolation quality, rather than explicit nonlinear trajectory recovery or gains specifically attributable to nonlinear-motion regions. The revised manuscript clarifies this distinction.

Q3: Runtime control

The original RIFE/IFRNet benchmarks followed their respective default full-resolution inference settings (1280 × 720). To provide a more comparable evaluation, we additionally benchmarked variants with half-resolution motion prediction (640 × 360) and full-resolution synthesis (1280 × 720). On the NVIDIA L20 using PyTorch with CUDA Graphs (CG), RIFE, IFRNet, and Align-FG take 5.009 ms, 3.023 ms, and 1.604 ms, respectively. Align-FG thus achieves 3.12× and 1.88× speedups over RIFE and IFRNet. These experiments better control for prediction resolution and backend differences, although model capacity and computational complexity remain unmatched.

The originally reported ~1.3 ms NFRU latency excluded optical-flow computation. We apologize for this omission and provide the corrected timings below:

NFRU backendExcluding flow computation (ms)Including flow computation (ms)
PyTorch + CG8.26110.002
Slang + CG1.3402.982

All NFRU configurations use 320 × 180 fusion-network inputs, 640 × 360 optical-flow estimation, and 1280 × 720 output. Thus, NFRU's fusion network operates at 320 × 180, whereas Align-FG predicts motion at 640 × 360; both produce full-resolution outputs. Flow-excluded timings omit only the computation cost of optical flow, not its use as input.

We retain NFRU's official Slang implementation because it achieves lower latency than its PyTorch counterpart, which incurs substantial preprocessing and postprocessing overhead. Including optical-flow computation, the complete Slang-based NFRU pipeline takes 2.982 ms, compared with 1.604 ms for Align-FG, yielding a 1.86× speedup. We explicitly acknowledge that this is a cross-backend comparison. The reported FLOPs account only for network computation at the respective input resolutions, rather than complete pipeline costs.

The revised manuscript clarifies the resolutions used for motion prediction, synthesis, and metric evaluation, distinguishing the half-resolution quality evaluation in Table 1 from the 720p synthesis benchmark in Table 3. Quality and latency results obtained under different configurations should not be combined into a single matched quality–latency operating point. Our efficiency conclusions are limited to the tested FP32 implementations on the NVIDIA L20 and do not establish mobile-device latency or energy efficiency. We also acknowledge that Mob-FG remains faster at 0.568 ms.

AuthorsOpenReview ↗

Response to Reviewer vwn2-part3

Q4: Motion notation and training details

During training, TAMG constructs two target-to-endpoint motion fields, each containing horizontal and vertical displacement components. Their concatenation forms a four-channel geometric prior. At each scale, the prior is obtained by resampling the same underlying motion field and adjusting displacement magnitudes to the corresponding pixel units, rather than independently estimating a new field. The four scales (1/81/8, 1/41/4, 1/21/2, and 11) are defined relative to the network input resolution, which is half the original image resolution. At inference, the Linear prior follows the same four-channel representation and multiscale processing, but does not require target-frame correspondences.

The geometric prior is resampled using nearest-neighbor interpolation, whereas propagated motion estimates are bilinearly upsampled with appropriate displacement scaling. At each finer scale, a learned motion-fusion gate adaptively combines the propagated motion estimates with the geometric prior, followed by residual motion refinement. This gate is distinct from the image-blending mask, which controls the fusion of warped RGB frames. The DMS coverage mask represents projected pixel support rather than an explicit occlusion label.

For final synthesis, the half-resolution motion fields are bilinearly upsampled, with their displacement magnitudes multiplied by two before warping the original full-resolution frames. The image-blending mask is similarly upsampled, but without displacement scaling.

Our training objective combines L1 reconstruction and VGG-based perceptual losses, with equal weights of 1 for both terms. To ensure reproducibility, we will specify the VGG architecture and pretrained weights, selected feature layers and their weights, input normalization, feature-distance computation, loss reduction, and parameter-freezing configuration. We will also clarify any auxiliary losses retained in the matched-loss baselines. These details will facilitate reproduction of the motion representation and training configuration.

Reviewer_ZtMhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Align-FG is a lightweight framework for game frame interpolation using rendered endpoint images, engine motion vectors, and depth. During training, Target-Aligned Motion Generation constructs geometric motion priors using recorded intermediate-frame correspondences. At inference, these are replaced by deployable priors derived from endpoint motion vectors under a linear-motion assumption. A single-stage coarse-to-fine network progressively combines the geometric prior with propagated motion estimates through learned gates and residual corrections. Experiments on Unreal Engine scenes and real-game recordings show improved perceptual metrics, while generating a 720p frame in 1.60 ms on an NVIDIA L20. Ablations separately examine target-aligned training, adaptive fusion, and residual refinement.

Critical strengths
  1. The method achieves a useful quality–computation trade-off. The deployable configuration reduces average LPIPS and FloLPIPS relative to NFRU by 34.8% and 30.7%, respectively, with modest additional measured latency. Integrating motion refinement and fusion avoids a separate optical-flow estimator while retaining appearance-based corrections for effects poorly captured by engine motion vectors.
  2. The ablations distinguish training benefits from privileged evaluation information. Table 3 compares Linear/Linear with TAMG/Linear under the same inference prior, demonstrating that target-aligned training improves deployable performance. The target-assisted TAMG/TAMG configuration is explicitly identified as non-deployable, and the fusion/refinement ablations further support the architectural choices.
Critical weaknesses

TAMG trains the network with priors derived from recorded intermediate-frame correspondences, whereas deployment uses endpoint motion vectors under a linear-motion assumption. Table 3 demonstrates that target-aligned training improves performance with the deployable prior, which is a meaningful result. However, this ablation is restricted to the four UE Demo scenes and reports aggregate performance. It remains unclear whether the benefit persists when nonlinear motion causes substantial errors in the inference prior—the situation motivating TAMG. A targeted analysis by motion nonlinearity or prior error, together with the corresponding ablation on Real Game, would clarify the robustness and scope of this contribution.

Questions for authors

How are training and validation sequences separated temporally, and are any reported results obtained on completely unseen scenes or games? Does the benefit of TAMG training persist on the Real Game dataset and on strongly nonlinear-motion subsets, using the same deployable linear inference prior?

5 comments

As the authors have not addressed the reviewers' questions throughout the review process, many concerns remain unresolved. Therefore, I have reconsidered my rating.

AuthorsOpenReview ↗

Response to Reviewer ZtMh

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We appreciate your recognition of the value of target-aligned training and the quality–computation trade-off, and address your concerns in detail below.

Q1: Training scope and multi-frame interpolation

TAMG reduces geometric misalignment with the target during training for both linear and nonlinear motion. PMFR learns appearance-dependent motion refinement to correct unreliable motion estimates in regions affected by occlusion, disocclusion, and complex rendering effects, without explicitly reconstructing the underlying nonlinear trajectories.

Fig. 7 distinguishes mutual alignment of warped endpoints from alignment with the ground-truth target. Under the Linear prior, warped endpoints may be mutually aligned yet remain displaced from the target. TAMG reduces this additional training-time discrepancy, providing a more accurate geometric reference for learning motion refinement and blending.

Fig. 8 shows that TAMG reduces the early-stage loss from 0.4825 to 0.4125 (14.5%). Under the same Linear evaluation prior, Table 4 presents this controlled comparison, showing improvements in LPIPS from 4.57 to 3.84 and FloLPIPS from 6.15 to 5.68, alongside higher PSNR and SSIM. These results demonstrate the benefit of target-aligned training in the evaluated UE setting.

We additionally evaluate 4× multi-frame interpolation on UE Demo 4x (Table 2) using the same midpoint-trained model without additional training. This held-out dataset comprises two UE captures, each containing 1,200 frames, recorded with a modified rendering pipeline that directly provides backward MVs spanning one to four frame intervals. Given endpoints at T−4T{-}4 and TT, Align-FG constructs time-dependent Linear motion priors for normalized target timestamps t∈{0.25,0.5,0.75}t \in \{0.25, 0.5, 0.75\} from the directly recorded T→(T−4)T{\to}(T{-}4) MV, following Eq. (3). Each intermediate frame is synthesized independently from the same endpoint frames, without recursive interpolation or target-frame MV inputs. Align-FG achieves LPIPS/FloLPIPS of 4.69/6.50, outperforming the next-best results of 7.49 (RIFE) and 11.10 (Mob-FG) by 37.4%/41.4%, respectively. It also achieves the highest SSIM (66.00%) and a PSNR of 29.14 dB, only 0.10 dB below IFRNet. LPIPS and FloLPIPS are scaled by 100.

These results suggest that the learned refinement can generalize beyond midpoint interpolation to different target timestamps. Although training uses uniformly spaced triplets, nonlinear motion and time-varying velocity mean that the target position need not coincide with the spatial midpoint of endpoint motion. By leveraging recorded target-frame correspondences, TAMG provides diverse target-aligned geometric references that help PMFR learn to refine unreliable motion estimates, rather than explicitly recover nonlinear trajectories. This offers a plausible explanation for its effectiveness with different time-dependent Linear priors, although it does not guarantee generalization to arbitrary timestamps.

The 4× results support the effectiveness of the complete framework for direct interpolation at multiple target timestamps. However, they do not isolate the contribution of TAMG to this generalization, establish improvements specifically in strongly nonlinear regions, or replace a Real Game training-prior ablation. The controlled TAMG ablation remains limited to UE data, and we will clarify these evaluation scopes accordingly.

Q2: Training and validation separation

We partition the data at the captured-sequence-group level, rather than randomly splitting individual frames or sampling a fixed proportion from each sequence. We will clarify the grouping criteria, sequence membership, and temporal relationships between recordings to make the data separation explicit.

Importantly, held-out captures do not necessarily imply unseen scenes. The four UE maps are represented in both training and validation recordings; thus, these results evaluate performance on held-out recordings from known maps rather than unseen-map generalization. Similarly, the Easy/Medium/Hard categories in Real Game do not establish cross-game generalization. We will explicitly specify the level of data separation for each experiment, including 4× multi-frame interpolation on UE Demo 4x, and qualify our generalization claims accordingly.

Regarding input provenance, the engine-assisted configurations in the original Table 1 use full-interval MVs composed from consecutive frame-to-frame motion fields. We further evaluated directly recorded endpoint MVs on UE Demo 4x using the same UE-trained model without additional training. Compared with composed MVs, directly recorded endpoint MVs improve all four quality metrics, including PSNR from 32.1855 to 32.5835 dB. These results provide additional evidence of the model's effectiveness with directly recorded endpoint motion, but do not replace the requested Real Game TAMG ablation.

AuthorsOpenReview ↗

Apology for the Delayed Rebuttal Response

Dear Reviewer ZtMh,

I sincerely apologize for the delay in submitting our rebuttal. I had originally planned to finish addressing all three reviewers' comments and submit the responses together early on November 18. However, when preparing the submission, I realized there was a 5,000-character limit. Since I had prepared detailed responses to all three reviewers, carefully shortening them while preserving the key explanations took longer than expected.

I am truly sorry for the delay and sincerely hope for your understanding.

I have now submitted our response. If possible, I would be very grateful if you could review it and let us know whether it adequately addresses your concerns.

Thank you very much for your time, patience, and constructive feedback.

Best regards, The Authors

Reviewer_LTAsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of engine-assisted frame interpolation in games: given two rendered frames, along with depth and backward motion vectors provided by the game engine, the goal is to synthesize intermediate frames to increase the frame rate with minimal additional overhead. The authors identify two primary limitations in existing methods. First, intermediate positions derived from MVs—based on a linear motion assumption—often deviate from the true positions when motion is non-linear; this error propagates into subsequent warping and blending stages. Second, relying on separate optical flow estimation to supplement MVs makes it difficult to balance output quality with computational efficiency.

To address the first issue, the authors propose Target-Aligned Motion Generation. During the training phase, it utilizes recorded intermediate-frame MVs and depth-aware motion splatting to construct a bidirectional geometric motion prior aligned with the target positions. During inference, a prior in the same format is constructed using endpoint MVs under a linear motion assumption. To address the second issue, the authors introduce PMFR, a single-stage, coarse-to-fine lightweight network. Across four scales—ranging from 1/8 resolution to full resolution—the network employs pixel-wise gated fusion to integrate the optical flow and geometric priors passed from the previous scale, followed by residual prediction for refinement. Finally, it predicts optical flow and a blending mask at half-resolution, which are then upsampled to warp and blend the original frames—eliminating the need for a separate optical flow estimator.

Experiments were conducted on both a custom-collected dataset of Unreal Engine scenes and a dataset of actual game footage. Compared to RIFE, IFRNet, Mob-FG, and NFRU, the proposed method achieves the best LPIPS scores across all seven test sets and the best FloLPIPS scores on six of them, though its average PSNR and SSIM values ​​are lower than those of RIFE and IFRNet. On an NVIDIA L20 GPU, the method takes 1.60 ms to generate a single 720p frame—6 to 7 times faster than RIFE or IFRNet. Ablation studies demonstrate that, when using a linear inference prior, training with TAMG outperforms training with a standard linear prior; furthermore, both the gated fusion mechanism and the residual refinement contribute positively to the overall performance.

Critical strengths
  1. TAMG is a training technique characterized by a clean experimental design and zero deployment cost. Table 3 compares two types of training priors—keeping the network architecture and the inference-time linear prior constant—and demonstrates that TAMG-based training yields superior results across all four metrics. This approach is straightforward, incurs no additional inference overhead, and can be seamlessly integrated into existing engine-assisted frame interpolation pipelines, offering significant practical value.

  2. Among engine-assisted, learning-based methods, this approach delivers higher quality for the same computational budget. While PMFR and NFRU share nearly identical FLOP counts, PMFR achieves a reduction of approximately 35% in average LPIPS and 31% in average FloLPIPS. Its advantage is most pronounced on the challenging "Real Game Hard" subset, with an inference latency only 0.3 ms higher than that of NFRU. This confirms that employing a gated, progressive refinement of engine-based priors—rather than relying on a standalone optical flow estimator—is an effective strategy for enhancing quality.

Critical weaknesses
  1. Train-inference distribution shift invalidates the core claim of TAMG. TAMG is only used in training while inference still relies on linear motion prior. The source of performance improvement is not proven to come from nonlinear motion mitigation, lacking required ablations and analysis.
  2. Fundamental identifiability problem for nonlinear motion. With only two endpoint frames, PMFR has no inherent information to recover true intermediate trajectories. The large gap between Ours and Ours* suggests results are dominated by prior quality, conflicting with its claimed motion correction capability.
  3. Unfair and inconsistent efficiency benchmarks. Reported speedup mostly comes from half-resolution and narrow channels rather than architectural gains. Mismatched resolution, backends and missing edge-device measurements cannot support the efficiency claim.
Questions for authors

Q1. TAMG introduces a distribution shift between training and inference, which contradicts the paper’s core motivation.The paper’s premise is that the linear motion assumption leads to misalignments of intermediate positions when motion is non-linear (Sec. 1, Fig. 1(a)). However, TAMG is employed only during training; at inference, the prior is still constructed via linear extrapolation (Eq. (3)—identical to the baseline), meaning the initial misalignment at deployment remains unmitigated. Furthermore, the distributions of these two priors differ significantly. During training, M^t0\hat{M}_{t0} is derived directly from the recorded motion vectors (MV) defined on the target grid (Eq. (2))—approximating the ground truth and reflecting the motion of visible surfaces at time tt. In contrast, the inference-time prior is obtained by linearly scaling and projecting the MVs from the first frame, reflecting the motion of surfaces visible at t=0t=0. While the paper claims "consistency in representation between training and inference" (Sec. 1, Sec. 3.2), this refers only to the tensor format, not the statistical distribution. Since the network never encounters the specific misalignment patterns characteristic of the linear prior during training, it should theoretically lack the capacity to correct them. Yet, Table 3 indicates that TAMG/Linear outperforms Linear/Linear. The authors should provide a mechanistic explanation for this phenomenon, supported by direct evidence that the performance gains indeed stem from regions of non-linear motion—for instance, by analyzing errors grouped by the degree of trajectory non-linearity, comparing the distribution of gating weights between training and inference, or conducting an ablation study where both priors are randomly mixed during training. Without such evidence, the central claim—that "TAMG mitigates misalignments caused by the linear assumption"—remains unsubstantiated.

Q2. The problem of non-linear motion is inherently unidentifiable when only the two endpoints are observed.Given a set of inputs (I0,I1,D1,M1→0)(I_0, I_1, D_1, M_{1\to0}), there exist infinitely many non-linear trajectories that could connect the same pair of endpoint positions. During inference, the input to PMFR—excluding the endpoint frames—contains no information regarding the intermediate trajectory (such as the preceding frame I−1I_{-1}, the camera pose at time tt, or object transformations). From an information-theoretic perspective, the network cannot, in principle, recover the true intermediate positions under non-linear motion; it can only learn a conditional expectation based on the training data. Could the authors clarify: what information does PMFR rely on during inference to correct non-linear misalignments? If the answer is "it cannot correct them," then the ~4.4 dB PSNR gap between "Ours*" and "Ours" (Tables 1 and 3) suggests that the output quality is primarily determined by the quality of the prior. This implies that PMFR’s capacity to correct motion using appearance cues is limited, which contradicts the paper’s description of PMFR’s function.

Q3. The source of the full-interval motion vector (MV) M1→0M_{1→0} used during evaluation, and the potential for information leakage.The dataset is recorded at 60 FPS, providing only the backward MVs between adjacent frames (Sec. 4.1), where the triplet (I0,It,I1)(I_0, I_t, I_1) consists of consecutive frames. The training samples listed in Sec. 3.1 do not include M1→0M_{1→0}, nor does the paper explain how it is obtained during evaluation. If M1→0M_{1→0} is derived by concatenating M1→tM_{1→t} and Mt→0M_{t→0} via the intermediate frame, then the supposedly "deployable" linear prior is, in fact, indirectly utilizing information from the intermediate frame (including occlusions and visibility at time tt). This is not equivalent to the MVs produced by the engine under real-world 30 FPS deployment conditions, which would invalidate the "Ours" results presented in Table 1. Please clarify how M1→0M_{1→0} is generated and justify its consistency with the MVs that would be available under actual deployment conditions.

Q4. The observed advantages in perceptual metrics may stem from the loss function rather than the method itself. This method is trained using an L1 + VGG perceptual loss (Sec. 4.1), which primarily yields advantages in deep-feature-based metrics such as LPIPS and FloLPIPS. However, its average PSNR and SSIM scores are lower than those of RIFE and IFRNet—lagging behind in 5 out of the 7 test sets in terms of PSNR. The paper does not clarify whether the baselines were trained using the same loss functions and the same number of training epochs (2,000 epochs in this work). If the baselines utilized their original, distinct loss functions, then a significant portion of the disparity in perceptual metrics could simply stem from differences in the optimization objectives. Please provide a comparison based on identical loss functions and training budgets, and explain why reconstruction fidelity consistently drops in non-"hard" scenarios.

Q5. The claim of being a "single-stage" approach that "does not rely on a separate optical flow estimator" is conceptually flawed.Each stage of PMFR predicts a residual optical flow, which is then refined through a coarse-to-fine cascade (Eq. (4), (5)). In essence, this constitutes a multi-scale optical flow estimator—a structure highly similar to the cascaded IFBlocks in RIFE or the design of IFRNet. RIFE itself is also a single-stage method that directly estimates intermediate optical flow. Therefore, the assertion that the approach "replaces the two-stage process of optical flow estimation and frame fusion with a unified, progressive motion reconstruction" (Fig. 1(c-d)) does not represent a fundamental methodological distinction. The authors should precisely define the novelty of PMFR relative to the cascaded structures of RIFE and IFRNet, and clarify whether there are any substantive structural contributions beyond the "introduction of geometric priors and gating mechanisms."

Q6. The conclusions regarding efficiency lack a comparable experimental setup; the efficiency gains attributable to the PMFR architecture itself remain unproven.(a) The 6–7x speedup over RIFE and IFRNet is primarily a result of the network operating at half-resolution with a very narrow channel width (Appendix A), rather than being an inherent property of the PMFR architecture. (b) Among comparable engine-accelerated methods, the proposed approach has a FLOP count nearly identical to that of NFRU (5.63 G vs. 5.65 G) but exhibits higher latency (1.60 ms vs. 1.30 ms)—which is also higher than that of Mob-FG (0.57 ms). Furthermore, as the authors themselves acknowledge, a direct comparison is not feasible because the three methods utilize different inference backends (Table 2). (c) Quality is evaluated at half-resolution, while latency is reported for a 1280×720 resolution; additionally, the resolution used for timing RIFE and IFRNet is not specified. Consequently, quality and efficiency metrics are not measured under consistent settings. (d) Although the paper’s primary motivation centers on edge devices and power consumption, the provided measurements are limited to FP32 precision on a data-center-grade GPU. Moreover, the ablation studies (Table 4) lack any data regarding computational cost or latency. In summary, the current evidence is insufficient to support the claim that the proposed method outperforms existing approaches in terms of efficiency. Please provide a comparison of quality and latency using a unified backend and a consistent resolution, and include a comparison against a two-stage baseline—consisting of "lightweight optical flow + fusion"—under equivalent computational constraints.

Q7. Inconsistency in the tabular data. Averaging the results for the four "UE Demo" scenarios in Table 1 yields values ​​for "Ours" that align perfectly with those reported for "TAMG/Linear" in Table 3 (LPIPS 3.84, PSNR 29.15 dB, SSIM 86.94%, FloLPIPS 5.68%). However, the averaged values ​​for "Ours*"—LPIPS 1.97, PSNR 34.88 dB, SSIM 96.20%, and FloLPIPS 4.55%—do not match the figures reported for "TAMG/TAMG" in Table 3 (1.94, 35.01 dB, 96.29%, and 4.48%, respectively). Please clarify whether these discrepancies arise from the use of different checkpoints or distinct evaluation settings, and confirm the consistency of the remaining results.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LTAs-part1

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We appreciate your insightful comments and provide detailed clarifications on the roles and scope of TAMG and PMFR, MV provenance, controlled experimental comparisons, and numerical corrections below. All table and figure references correspond to the revised manuscript.

Q1: Target-aligned training and Linear-prior inference

TAMG reduces geometric misalignment with the target during training. Our training data encompass both linear and nonlinear motion, for which TAMG leverages recorded target-frame correspondences. PMFR learns to refine unreliable motion estimates in regions affected by occlusion, disocclusion, and complex rendering effects, including shadows, reflections, and semi-transparency, without explicitly reconstructing the underlying nonlinear trajectories.

Fig. 7 highlights the distinction between mutual alignment of warped endpoints and alignment with the ground-truth target. In commonly visible geometric regions, both priors can produce relatively consistent warped endpoints, as illustrated by the two rightmost pre-cur error maps in the bottom row. However, under the Linear prior, these observations may still exhibit substantial discrepancies with the ground-truth target, as shown by its pre/cur-to-GT error maps. In contrast, TAMG reduces these target discrepancies, providing a more accurate geometric reference for learning motion refinement and blending. Training with the Linear prior must additionally accommodate misalignment arising from inaccurate motion approximation.

Rather than explicitly distinguishing linear from nonlinear trajectories, PMFR learns appearance-dependent corrections to unreliable motion estimates. This provides a plausible basis for its effectiveness under different motion priors, without assuming identical prior distributions or guaranteeing target alignment at inference.

Fig. 8 shows that TAMG reduces the training loss from 0.4825 to 0.4125 (14.5%) at approximately epoch 20, while improving final validation LPIPS from 0.0457 to 0.0384 (16.0%). These results indicate that improved training-time alignment is accompanied by better final interpolation quality.

Table 4 further confirms these improvements under a controlled setting with identical architecture, optimization schedule, and Linear evaluation prior. TAMG improves FloLPIPS from 6.15 to 5.68, PSNR from 28.77 to 29.15 dB, and SSIM from 86.21% to 86.94%. LPIPS and FloLPIPS are scaled by 100 in Table 4. Since the evaluation prior remains unchanged, these results support the benefit of TAMG training rather than a more accurate inference prior. The full-interval MV construction is clarified in Q3.

We will therefore clarify our claim as follows: TAMG provides a more accurate geometric training reference and improves interpolation performance under the same Linear evaluation prior. These aggregate improvements support the effectiveness of TAMG without attributing the gains specifically to nonlinear-motion regions.

Q2: Identifiability and Ours versus Ours*

We agree that endpoint observations alone cannot uniquely determine an arbitrary nonlinear intermediate trajectory. Explicit trajectory recovery is not the objective of PMFR. Instead, PMFR learns appearance-dependent residual motion updates to refine unreliable motion estimates in regions affected by occlusion, disocclusion, and complex rendering effects, while adaptively fusing motion and source-frame information.

Ours* leverages recorded target-frame correspondences that are unavailable to the standard Ours configuration at inference. Its approximately 4.4 dB higher average PSNR across the seven groups in Table 1 highlights the benefit of accurate target-position information, rather than implying that PMFR can recover such privileged information from endpoint observations alone.

Importantly, the comparison between Ours and Ours* changes the motion prior and therefore does not isolate the contribution of PMFR's refinement. Table 5 provides a more direct ablation under the same Linear evaluation prior: removing residual motion updates increases LPIPS from 3.84 to 4.61, while replacing learned motion-fusion gating with fixed averaging increases it to 3.98. These results support the effectiveness of residual motion refinement and adaptive motion fusion, despite the inherent ambiguity of nonlinear trajectories. They do not imply explicit recovery of the underlying nonlinear motion.

AuthorsOpenReview ↗

Response to Reviewer LTAs-part2

Q3: Full-interval motion vectors

In the original 60 FPS UE and Real Game recordings, current-to-previous-frame motion vectors (MVs) were available without modifying the rendering pipeline. For midpoint interpolation, we constructed full-interval MVs by composing two consecutive motion fields: the current-to-intermediate MV and the intermediate-to-previous MV sampled at the corresponding warped location.

All deployable configurations in Table 1 that require engine motion use these composed full-interval MVs, whereas RGB-only baselines do not use engine MVs. Ours* is separately reported as a target-assisted reference. Although PMFR does not take the intermediate RGB frame as input, MV composition relies on intermediate-frame motion correspondences. We acknowledge this preprocessing dependency and will explicitly distinguish composed full-interval MVs from directly recorded endpoint MVs.

To further evaluate MV composition, we modified the UE rendering pipeline to directly record backward MVs from frame TT to T−1T{-}1, T−2T{-}2, T−3T{-}3, and T−4T{-}4, without chaining. The resulting UE Demo 4x dataset contains two scenes with 1,200 frames each (2,400 frames in total), with the remaining capture settings matching UE Demo. It supports both multi-frame interpolation and the direct-versus-composed MV analysis. The MV comparison below evaluates 1,176 samples per interval; this is the comparison sample count, distinct from the number of captured frames.

Interval kkOverall EPE (px)Nonoccluded EPEOccluded EPEPixels below 1 px
20.2240.1272.35398.2%
30.8420.4394.94696.1%
41.6600.9467.88094.1%

At k=2k = 2, 98.6% of pixels exhibit an endpoint error (EPE) below 2 px. Errors are more pronounced in occluded regions and increase with temporal interval length. Fig. 9 compares the direct and composed T→(T−2)T{\to}(T{-}2) motion fields and the corresponding warps of the T−2T{-}2 image onto the TT grid. This is an endpoint-warping diagnostic, separate from the midpoint synthesis experiment below. These results indicate close agreement for most pixels, rather than exact equivalence.

We further evaluate final synthesis quality on the UE Demo 4x captures using the same UE-trained model without additional training. For 2× midpoint interpolation, we compare two sources of full-interval MVs, obtaining the following results:

MetricDirect MVChained MVChained minus direct
PSNR (dB) ↑32.583532.1855−0.3980
LPIPS ↓0.027640.02901+0.00137
SSIM ↑0.749300.74214−0.00716
FloLPIPS ↓0.0351610.037668+0.002507

LPIPS, SSIM, and FloLPIPS are reported without scaling, using the same precision as Table 7. Directly recorded MVs improve all four metrics. Compared with direct MVs, chaining increases LPIPS and FloLPIPS by approximately 5.0% and 7.1%, respectively, while decreasing PSNR by 0.398 dB and SSIM by 0.716 percentage points.

These results show that MV chaining offers no synthesis-quality advantage in this paired comparison. Instead, the existing model achieves better performance with directly recorded endpoint MVs, without relying on intermediate-frame motion correspondences to construct the inference prior. This provides evidence of its effectiveness with the intended endpoint inputs.

Since Table 1 uses composed MVs, the new results suggest that directly recorded endpoint MVs may further improve performance, as observed on this dataset. However, this does not guarantee consistent improvements across all scenes or baselines, nor does it make the original evaluation protocol equivalent to direct MV capture. We will explicitly identify the MV source used in each experiment and report the new direct-MV validation separately.

AuthorsOpenReview ↗

Response to Reviewer LTAs-part3

Q4: Loss functions and reconstruction fidelity

We agree that differences in training objectives can affect perceptual comparisons. In the original experiments, RIFE and IFRNet used their released reconstruction-oriented losses, while NFRU employed L1+VGG. All methods, including Align-FG, were trained for 2,000 epochs, retaining their respective learning-rate settings.

To better control for the effect of training objectives, we additionally retrained RIFE and IFRNet using the same L1+VGG objective as Align-FG, with equal weights of 1 for both loss terms. We preserved their original learning-rate settings and the 2,000-epoch training schedule. Updated Table 1 presents these matched-loss results. Although the loss formulation, weights, and epoch counts are aligned, other optimization settings may still differ.

MethodLPIPS ×100 ↓FloLPIPS ×100 ↓PSNR (dB) ↑SSIM ×100 ↑
IFRNet9.2313.3228.3885.85
RIFE7.7411.2927.6584.11
Align-FG4.467.2428.4886.50

Under this setting, Align-FG reduces average LPIPS/FloLPIPS by 51.7%/45.6% compared with IFRNet and 42.4%/35.9% compared with RIFE. It also outperforms IFRNet in average PSNR and SSIM by 0.10 dB and 0.65 percentage points, respectively. Thus, Align-FG achieves higher average perceptual and reconstruction quality than both baselines under the matched-loss setting.

However, the improvements are not uniform across all scenes. IFRNet retains higher PSNR on A.Village, Egypt, and Real Game Easy and Medium, whereas Align-FG performs better on F.Village, Highrise, and Real Game Hard. Residual spatial misalignment may affect pixelwise fidelity despite improved perceptual similarity. Linear-prior approximation, reduced-resolution prediction, and model capacity may also contribute, although their individual effects have not been isolated.

Overall, Align-FG maintains substantial perceptual advantages even when RIFE and IFRNet are trained with the same L1+VGG objective and epoch count. These results support the effectiveness of the proposed engine-assisted framework and indicate that its perceptual gains cannot be explained solely by the use of VGG supervision.

Q5: Single stage and the PMFR contribution

We agree that coarse-to-fine flow estimation is well established, including in RIFE and IFRNet. Our use of "single-stage" refers to a unified pipeline that integrates motion estimation, geometric-prior fusion, and refinement, rather than a standalone optical-flow estimator followed by engine-assisted fusion. We do not claim novelty in coarse-to-fine estimation itself.

In the two-stage engine-assisted pipelines considered here, image-based flow is first estimated and subsequently fused with engine MVs, either directly in the motion domain or through images warped by the respective motion fields. In contrast, PMFR progressively integrates motion fusion and refinement within a unified multiscale network. At each finer scale, it takes propagated bidirectional motion estimates, geometric priors, and appearance features as inputs, and jointly predicts motion-fusion gates, residual motion updates, and image-blending logits. The motion candidates are adaptively fused and refined before the updated estimates are propagated to the next scale.

Crucially, the geometric prior remains an explicit motion candidate at successive scales, rather than serving only as an initialization. The motion-fusion gates adaptively combine the propagated motion estimates with the geometric prior, while the image-blending mask controls the fusion of the final warped endpoint frames. Our contribution lies in integrating repeated geometric-prior fusion and appearance-dependent motion refinement within a lightweight network, complemented by TAMG training, rather than introducing a fundamentally new multiscale backbone.

Table 5 further validates the contributions of residual motion refinement and learned motion fusion under otherwise identical experimental settings. Compared with removing residual refinement, the full model reduces LPIPS from 4.61 to 3.84 (16.7%); compared with fixed averaging, it reduces LPIPS from 3.98 to 3.84 (3.5%). These results support the effectiveness of the proposed refinement and fusion mechanisms within PMFR.

AuthorsOpenReview ↗

Response to Reviewer LTAs-part4

Q6: Controlled runtime comparisons

The original RIFE/IFRNet benchmarks followed their respective default inference settings, with full-resolution processing at 1280 × 720. The reported 6–7× speedups therefore reflected the combined effects of different inference configurations, including resolution, model capacity, and pipeline design. To provide a more comparable evaluation, we additionally benchmarked RIFE and IFRNet variants with half-resolution motion prediction (640 × 360) and full-resolution synthesis (1280 × 720), using PyTorch with CUDA Graphs (CG):

MethodPrediction domainTotal (ms)
RIFEHalf resolution5.009
IFRNetHalf resolution3.023
Align-FGHalf resolution1.604

Under these settings, Align-FG achieves 3.12× and 1.88× speedups over RIFE and IFRNet, respectively. Updated Table 3 includes these more comparable resolution-controlled results. However, differences in model capacity and computational complexity remain, so these comparisons are not strictly compute-matched.

The originally reported ~1.3 ms NFRU latency excluded optical-flow computation. We apologize for this omission and provide the corrected end-to-end timings below. All NFRU configurations use 320 × 180 fusion-network inputs, 640 × 360 optical-flow estimation, and 1280 × 720 output. Flow-excluded timings omit the computation cost of optical flow, not its use as input.

NFRU backendFlow computation includedPre (ms)Net (ms)Post (ms)Total (ms)
PyTorch + CGNo3.3341.1073.8208.261
PyTorch + CGYes5.0621.1003.84010.002
Slang + CGNo0.1711.1180.0511.340
Slang + CGYes1.8311.1000.0512.982

We retain NFRU's official Slang implementation, as its PyTorch counterpart incurs substantially higher preprocessing and postprocessing overhead. Including optical-flow computation, the complete Slang-based NFRU pipeline takes 2.982 ms, compared with 1.604 ms for Align-FG, yielding a 1.86× speedup. We explicitly acknowledge that this is a cross-backend comparison.

The reported FLOPs (5.65 G for NFRU and 5.63 G for Align-FG) account only for network computation at their respective input resolutions, rather than complete pipeline costs. Thus, NFRU serves as a practical flow-plus-fusion baseline, not a strictly compute-matched ablation. Table 5 supports the quality contributions of individual components but does not isolate their latency costs.

The revised manuscript distinguishes the half-resolution quality evaluation in Table 1 from the 720p synthesis benchmark in Table 3 and clarifies the corresponding input and output resolutions. Quality and latency measurements obtained under different configurations should not be combined into a strictly matched quality–latency comparison. Furthermore, our efficiency conclusions are limited to the tested FP32 implementations on the NVIDIA L20 and do not establish edge-device latency or energy efficiency. We also acknowledge that Mob-FG remains faster at 0.568 ms.

Q7: Corrected privileged evaluation averages

We identified a checkpoint-selection inconsistency: Ours* in Table 1 used the checkpoint selected by validation LPIPS, whereas TAMG/TAMG in Table 4 inadvertently used the checkpoint selected by validation PSNR. We apologize for this oversight and sincerely thank the reviewer for bringing this inconsistency to our attention.

We have corrected Table 4 to use the same checkpoint as Table 1. The updated UE averages are LPIPS 1.97, FloLPIPS 4.55, PSNR 34.88 dB, and SSIM 96.20%, with LPIPS and FloLPIPS scaled by 100.

Ours in Table 1 and TAMG/Linear in Table 4 were already consistent within rounding precision. Therefore, the TAMG/Linear versus Linear/Linear comparison and its conclusions remain unchanged. We also verified that the seven-group averages in Table 1 match the arithmetic means of the reported per-group values.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.