Photometric Flow For Temporal Photometric Consistency

Flow–guided video synthesis typically assumes photometric constancy: corresponding pixels preserve appearance as they move across frames. In practice, lighting changes, shadows, auto-exposure, and white-balance shifts violate this assumption, producing color drift and visible seams after warping. Prior remedies either enforce strict consistency—risking artifacts and hallucinations—or learn generic error compensation that conflate geometric and photometric errors without explicitly representing temporal photometric variation. We instead treat photometric variation as a cue complementary to geometry and explicitly model how pixel appearance evolves along motion trajectories. Specifically, we introduce photometric flow, a dense, motion-aligned representation of temporal color change, together with a lightweight Photometric Flow Network (PFN) for its estimation. PFN first predicts a stable additive base field and then refines it through appearance-conditioned residual calibration, relaxing the limitations of a purely additive correction while retaining a structured representation suitable for temporal composition. The resulting photometric flow provides explicit and temporally consistent appearance guidance that complements optical flow. Integrated into an iterative, flow-guided pipeline, our approach consistently improves color fidelity and suppresses seams, demonstrating broad applicability across diverse tasks including video inpainting, outpainting, and frame interpolation. To our knowledge, this is the first formulation and learning of a photometric flow tailored for downstream flow-guided video synthesis, offering a principled bridge between motion and appearance change.