- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes photometric flow, a dense, motion-aligned RGB increment field intended to represent appearance changes along optical-flow trajectories. A Photometric Flow Network (PFN) predicts an additive base field followed by appearance-conditioned calibration and is trained using frame reconstruction with staged joint fine-tuning of RAFT. The estimated fields are completed in missing regions and used either to correct source appearance before propagation or to guide a separate spatio-temporal residual refinement network. The method is evaluated on video inpainting, outpainting, and frame interpolation.
Critical strengths
-
The paper addresses a practically relevant limitation of flow-guided synthesis. Geometrically aligned content can still exhibit appearance discrepancies under illumination or exposure changes. Explicitly retaining an appearance-change signal for downstream synthesis is a reasonable approach to this problem and gives the work potential practical value.
-
The reported results provide encouraging evidence of empirical usefulness. Improvements are reported across several synthesis tasks and backbones. In particular, Table 5 reports a 15.6% reduction in photometric alignment error under fixed ground-truth optical flow, providing evidence that the correction can improve appearance alignment independently of changes in estimated geometry. These results support the usefulness of the proposed system.
Critical weaknesses
- The presentation makes the work difficult to assess. The main text does not explain the central training procedure and refinement configuration clearly enough, while the figures offer limited support for understanding the proposed mechanisms and evaluating the claimed improvements. These issues hinder a clear assessment of the method and its experimental evidence. Overall, I think this manuscript needs substantial revision of both the writing and the figures for better understanding.
- The central contribution is not sufficiently substantiated. The distinction between photometric flow and existing RGB residual correction approaches is not convincingly established. Although the overall system shows improvements, the experiments do not sufficiently isolate the proposed representation’s contribution from the additional refinement network and training. This leaves both the novelty and the source of the gains unclear.
Questions for authors
- Dependence on the optical-flow estimator. Since PFN is jointly fine-tuned with RAFT, how sensitive are the results to the choice of optical-flow model? Is there evidence that the benefits persist with other optical flow model, such as WAFT[1], particularly when transferring the trained PFN without additional fine-tuning?
2 comments
I appreciate the authors’ efforts, which have partially addressed my concern about the method’s sensitivity to the choice of optical flow model. However, additional experiments alone are not sufficient to address my remaining concerns; the manuscript itself still requires substantial revision to be accepted. I will therefore maintain my recommendation of weak reject.
We thank you for the constructive feedback and for recognizing the practical value of our approach. We provide additional experiments to clarify the contribution of photometric flow, distinguish it from conventional residual correction, and evaluate its transferability across optical-flow estimators. We also clarify the training and refinement configurations.
1. Transfer to WAFT without PFN fine-tuning
Yes: the same frozen PFN improves WAFT-based photometric alignment on Sintel Final without additional fine-tuning. We replace RAFT with pretrained WAFT and retain the original PFN, including calibration. For adjacent-frame reconstruction, each row below uses identical flow and evaluation pixels for warp-only and +PFN.
The first row directly addresses the requested zero-fine-tuning transfer. The second is a separate adaptation experiment trained on YouTube-VOS and evaluated on Sintel; its warp-only result uses the adapted WAFT.
2. Isolating photometric guidance from additional refinement and training
On DAVIS inpainting, we train an STRM control with the same architecture, training data, losses, and 200k-step budget as the full STRM. Its six photometric channels are zeroed during both training and testing. All configurations use the same ProPainter coarse outputs and evaluation protocol.
C versus B isolates the contribution of photometric guidance under matched refinement capacity and training: PSNR increases by 0.60 dB, SSIM by 0.0384, while LPIPS and VFID decrease by 30.18% and 27.65%, respectively. C also outperforms the original ProPainter (A) across all four metrics.
D and E further evaluate cross-estimator transfer in downstream synthesis. In D, we replace RAFT with WAFT while keeping PFN, the completion modules, and STRM frozen. The direct substitution results in lower PSNR and SSIM than the original configuration (C), indicating sensitivity to the optical-flow estimator in the full pipeline. After jointly adapting WAFT and PFN (E), PSNR improves from 33.76 to 34.43 dB, while SSIM increases from 0.9593 to 0.9698, LPIPS decreases from 0.0256 to 0.0238, and VFID decreases from 0.1112 to 0.0912. The adapted configuration also improves all four metrics over the original ProPainter baseline (A).
These results demonstrate that photometric guidance provides benefits beyond additional refinement capacity, while cross-estimator adaptation further improves its effectiveness in downstream synthesis.
We will add the detailed experimental settings to the revised main text and appendix.
3. Distinction from RGB residual correction
ECFVI's ECN predicts a residual to correct already-propagated content. Our photometric flow instead models photometric changes between motion-corresponding input observations and retains them as an intermediate field for trajectory-wise accumulation, missing-region completion, and synthesis guidance. The distinction concerns what the field represents and how it is used, rather than merely whether the output is a dense RGB map.
4. Training and refinement configuration
Due to space constraints, these implementation details were provided in Appendix A.1–A.2. The training stages on YouTube-VOS are:
In the revised manuscript, we will move this summary into the main text and provide clearer figures showing the training stages, the two inference paths, their inputs, and trainable/frozen modules.