acceptodds
Under review as a conference paper at ICLR 2027

Photometric Flow For Temporal Photometric Consistency

Abstract

Flow–guided video synthesis typically assumes photometric constancy: corresponding pixels preserve appearance as they move across frames. In practice, lighting changes, shadows, auto-exposure, and white-balance shifts violate this assumption, producing color drift and visible seams after warping. Prior remedies either enforce strict consistency—risking artifacts and hallucinations—or learn generic error compensation that conflate geometric and photometric errors without explicitly representing temporal photometric variation. We instead treat photometric variation as a cue complementary to geometry and explicitly model how pixel appearance evolves along motion trajectories. Specifically, we introduce photometric flow, a dense, motion-aligned representation of temporal color change, together with a lightweight Photometric Flow Network (PFN) for its estimation. PFN first predicts a stable additive base field and then refines it through appearance-conditioned residual calibration, relaxing the limitations of a purely additive correction while retaining a structured representation suitable for temporal composition. The resulting photometric flow provides explicit and temporally consistent appearance guidance that complements optical flow. Integrated into an iterative, flow-guided pipeline, our approach consistently improves color fidelity and suppresses seams, demonstrating broad applicability across diverse tasks including video inpainting, outpainting, and frame interpolation. To our knowledge, this is the first formulation and learning of a photometric flow tailored for downstream flow-guided video synthesis, offering a principled bridge between motion and appearance change.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video synthesis, optical flow, photometric flow

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rALvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes photometric flow, a dense, motion-aligned RGB increment field intended to represent appearance changes along optical-flow trajectories. A Photometric Flow Network (PFN) predicts an additive base field followed by appearance-conditioned calibration and is trained using frame reconstruction with staged joint fine-tuning of RAFT. The estimated fields are completed in missing regions and used either to correct source appearance before propagation or to guide a separate spatio-temporal residual refinement network. The method is evaluated on video inpainting, outpainting, and frame interpolation.

Critical strengths
  1. The paper addresses a practically relevant limitation of flow-guided synthesis. Geometrically aligned content can still exhibit appearance discrepancies under illumination or exposure changes. Explicitly retaining an appearance-change signal for downstream synthesis is a reasonable approach to this problem and gives the work potential practical value.

  2. The reported results provide encouraging evidence of empirical usefulness. Improvements are reported across several synthesis tasks and backbones. In particular, Table 5 reports a 15.6% reduction in photometric alignment error under fixed ground-truth optical flow, providing evidence that the correction can improve appearance alignment independently of changes in estimated geometry. These results support the usefulness of the proposed system.

Critical weaknesses
  1. The presentation makes the work difficult to assess. The main text does not explain the central training procedure and refinement configuration clearly enough, while the figures offer limited support for understanding the proposed mechanisms and evaluating the claimed improvements. These issues hinder a clear assessment of the method and its experimental evidence. Overall, I think this manuscript needs substantial revision of both the writing and the figures for better understanding.
  2. The central contribution is not sufficiently substantiated. The distinction between photometric flow and existing RGB residual correction approaches is not convincingly established. Although the overall system shows improvements, the experiments do not sufficiently isolate the proposed representation’s contribution from the additional refinement network and training. This leaves both the novelty and the source of the gains unclear.
Questions for authors
  1. Dependence on the optical-flow estimator. Since PFN is jointly fine-tuned with RAFT, how sensitive are the results to the choice of optical-flow model? Is there evidence that the benefits persist with other optical flow model, such as WAFT[1], particularly when transferring the trained PFN without additional fine-tuning?
2 comments

We thank you for the constructive feedback and for recognizing the practical value of our approach. We provide additional experiments to clarify the contribution of photometric flow, distinguish it from conventional residual correction, and evaluate its transferability across optical-flow estimators. We also clarify the training and refinement configurations.

1. Transfer to WAFT without PFN fine-tuning

Yes: the same frozen PFN improves WAFT-based photometric alignment on Sintel Final without additional fine-tuning. We replace RAFT with pretrained WAFT and retain the original PFN, including calibration. For adjacent-frame reconstruction, each row below uses identical flow and evaluation pixels for warp-only and +PFN.

SettingWarp-only L2+PFN L2Reduction
Pretrained WAFT + frozen original PFN5.85545.47686.47%
Jointly adapted WAFT + PFN5.83895.013414.14%

The first row directly addresses the requested zero-fine-tuning transfer. The second is a separate adaptation experiment trained on YouTube-VOS and evaluated on Sintel; its warp-only result uses the adapted WAFT.

2. Isolating photometric guidance from additional refinement and training

On DAVIS inpainting, we train an STRM control with the same architecture, training data, losses, and 200k-step budget as the full STRM. Its six photometric channels are zeroed during both training and testing. All configurations use the same ProPainter coarse outputs and evaluation protocol.

ConfigurationPSNR ↑SSIM ↑LPIPS ↓VFID ↓
A: ProPainter33.970.96640.02650.1010
B: +STRM, zero photometric input34.120.93760.03280.1331
C: +STRM, RAFT/PFN guidance34.720.97600.02290.0963
D: C with WAFT, all modules frozen33.760.95930.02560.1112
E: D after adapting WAFT + PFN34.430.96980.02380.0912

C versus B isolates the contribution of photometric guidance under matched refinement capacity and training: PSNR increases by 0.60 dB, SSIM by 0.0384, while LPIPS and VFID decrease by 30.18% and 27.65%, respectively. C also outperforms the original ProPainter (A) across all four metrics.

D and E further evaluate cross-estimator transfer in downstream synthesis. In D, we replace RAFT with WAFT while keeping PFN, the completion modules, and STRM frozen. The direct substitution results in lower PSNR and SSIM than the original configuration (C), indicating sensitivity to the optical-flow estimator in the full pipeline. After jointly adapting WAFT and PFN (E), PSNR improves from 33.76 to 34.43 dB, while SSIM increases from 0.9593 to 0.9698, LPIPS decreases from 0.0256 to 0.0238, and VFID decreases from 0.1112 to 0.0912. The adapted configuration also improves all four metrics over the original ProPainter baseline (A).

These results demonstrate that photometric guidance provides benefits beyond additional refinement capacity, while cross-estimator adaptation further improves its effectiveness in downstream synthesis.

We will add the detailed experimental settings to the revised main text and appendix.

3. Distinction from RGB residual correction

ECFVI's ECN predicts a residual to correct already-propagated content. Our photometric flow instead models photometric changes between motion-corresponding input observations and retains them as an intermediate field for trajectory-wise accumulation, missing-region completion, and synthesis guidance. The distinction concerns what the field represents and how it is used, rather than merely whether the output is a dense RGB map.

4. Training and refinement configuration

Due to space constraints, these implementation details were provided in Appendix A.1–A.2. The training stages on YouTube-VOS are:

  • PFN: 80k steps with RAFT frozen; 40k jointly with RAFT; 5k calibration-only steps; then 40k joint steps for the complete PFN and RAFT.
  • Photometric completion: 200k steps, using PFN outputs on unmasked videos as targets under simulated masks.
  • STRM: 200k steps with upstream estimation/completion modules frozen. A separate STRM is trained for each frozen backbone. Its input is coarse RGB (3 channels), bidirectional photometric flow (6), and a validity mask (1).

In the revised manuscript, we will move this summary into the main text and provide clearer figures showing the training stages, the two inference paths, their inputs, and trainable/frozen modules.

I appreciate the authors’ efforts, which have partially addressed my concern about the method’s sensitivity to the choice of optical flow model. However, additional experiments alone are not sufficient to address my remaining concerns; the manuscript itself still requires substantial revision to be accepted. I will therefore maintain my recommendation of weak reject.

Reviewer_6WWNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the violation of photometric constancy, an assumption implicitly made by flow-guided video synthesis, caused by illumination changes, shadows, exposure, and white-balance shifts. To this end, it proposes photometric flow, an additive three-channel field that represents RGB appearance changes along optical-flow trajectories.

Photometric flow is estimated by a Photometric Flow Network (PFN). PFN predicts a base field using an architecture built on the BrightFlow corrector, and then applies appearance-conditioned calibration through a per-pixel 1×1 MLP. Training is self-supervised and performed jointly with RAFT under a forward-sharing, backward-decoupled strategy.

The estimated photometric flow is used in two ways: (1) a Correct-then-Propagate scheme that corrects source appearance before propagation, and (2) guidance for a plug-in refinement module (STRM) applied to the outputs of existing synthesis models. The authors report consistent improvements on video inpainting, outpainting, and frame interpolation.

Critical strengths
  1. The problem is well-motivated and practically relevant. Color seams caused by appearance changes are frequently observed in flow-guided propagation. The motivation to treat geometry and appearance as separate cues is intuitive.

  2. The approach is broadly applicable. The paper shows improvements across three tasks and two usage modes, and it is a positive point that refinement is applied to multiple inpainting backbones (E2FGVI, ProPainter, MiniMax Remover).

  3. The paper attempts diagnostic analysis. The Sintel alignment experiment with ground-truth optical flow (Tab. 5), which fixes the geometry, is a reasonable attempt to isolate the photometric effect.

  4. The paper is clearly organized and the figures aid understanding. The limitations are also discussed fairly candidly.

Critical weaknesses

W1. The technical novelty is limited, and the relationship to prior work is insufficiently discussed. Both the network architecture (BrightFlow corrector) and the training strategy (forward-sharing, backward-decoupled) directly follow BrightFlow. BrightFlow's brightness correction map is essentially the same quantity as the photometric flow in this paper. The substantive differences appear to be limited to (a) using this field for synthesis at inference time and (b) a small 1×1 MLP calibration. Given this, the claim of being the first to formulate and learn photometric flow is overstated.

W2. The learned photometric flow may be a trivial warping residual.

  • PFN takes the aligned target I_warped as input and is trained so that I_t + photometric flow ≈ I_warped. A trivial solution, photometric flow = I_warped − I_t, therefore exists. The per-pixel calibration MLP sees both I_t and I_warped and can easily learn it, and no regularization is mentioned.

  • The flow may then act as a warping residual that also absorbs geometric errors. Its large magnitudes at motion boundaries (Figs. 10–11) support this concern, which is the same conflation the paper criticizes in ECN.

  • This also makes the Sintel gain (Tab. 5) largely expected. Comparisons with simple baselines, such as global affine color correction or low-pass(I_warped − I_t), are needed.

W3 Missing controls to isolate the source of the gains.

  • STRM is a sizable network (U-Net with a Transformer bottleneck) trained for 200k extra steps. Without an STRM trained identically but without photometric flow input, the gains cannot be attributed to photometric flow.

  • It is unclear whether STRM is trained per backbone, which affects the plug-and-play claim.

  • In Tab. 4, calibration adds 45k training steps without a step-matched control, and the small total gain (+0.34 dB) is reported without variance across seeds.

W4. The target scenario is not directly evaluated.

  • DAVIS and YouTube-VOS with synthetic masks contain few videos with strong illumination or exposure changes.

  • Evaluation on data with large inter-frame photometric variation is needed, e.g., synthetic exposure/white-balance perturbations or challenging real data such as the alternating-exposure videos of Chen et al., "HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset," ICCV 2021.

  • Error versus propagation distance should be reported to support the stability of additive composition, and temporal metrics beyond VFID (e.g., warping error) would help.

W5. Unconvincing propagation-integration results.

  • Correct-then-Propagate is validated only on the authors' reproduced ECFVI* (31.81 dB), far below the original ECFVI (33.52 dB). Even with PFN (32.15 dB) it does not reach the original, and confounds such as the SD1.5 synthesis module make the comparison unclear.

  • This setting is evaluated only on DAVIS. Numbers for existing methods (e.g., ProPainter) also seem to differ from published ones, so the protocol (resolution, masks, evaluated regions) should be specified.

W6. Limited outpainting and interpolation experiments.

  • Each task uses only DAVIS and one comparison model. Interpolation omits standard benchmarks such as Vimeo90K (Xue et al., "Video Enhancement with Task-Oriented Flow," IJCV 2019), X4K1000FPS (Sim et al., "XVFI: eXtreme Video Frame Interpolation," ICCV 2021), and SNU-FILM (Choi et al., "Channel Attention Is All You Need for Video Frame Interpolation," AAAI 2020), and using the inpainting method ECFVI* as the propagation baseline is questionable.
Questions for authors
  1. How does an STRM trained without photometric flow input perform?

  2. What prevents the trivial solution photometric flow = I_warped − I_t? How does low-pass(I_warped − I_t) compare as a baseline?

  3. How does the photometric flow at motion boundaries change with ground-truth versus RAFT flow?

  4. How does the method compare with FGVC-style gradient-domain/Poisson fusion and global affine color correction?

  5. How does it perform on data with large photometric variation (e.g., Chen et al., ICCV 2021), and how does error grow with propagation distance?

  6. Is STRM trained per backbone, and under what protocol were baseline numbers reproduced?

4 comments

Thank you for the detailed review. We conducted additional analyses to isolate the contribution of photometric flow and clarify the evaluation protocol. Due to the length limit, we will address your concerns in two separate comments.

W1. The relationship to prior work.

BrightFlow predicts a pairwise brightness correction for the optical-flow training objective. Building on its architecture, we formulate photometric flow as a representation of appearance evolution along motion trajectories. Adjacent photometric increments are explicitly composed across time, and used to correct propagated content or guide refinement. This establishes a reusable temporal representation whose role extends beyond correcting an individual frame pair. Our contribution therefore centers on this temporal field formulation and its integration into downstream synthesis. The temporal-composition and conditioning ablations directly support these modeling choices.

W2. Residual solution and simple baselines.

We clarify that PFN is not directly trained to regress the residual Iwarped−ItI_{\mathrm{warped}}-I_t. As described in Eqs. (5)–(6), PFN predicts photometric flow to correct the source image, which is then warped back using backward optical flow to reconstruct the target frame:Lrec=∥It+1−W(It+Φt→t+1,Ft+1→t)∥1.\mathcal{L}_{\mathrm{rec}} = \left\lVert I_{t+1} - W\left(I_t + \Phi_{t\rightarrow t+1}, F_{t+1\rightarrow t}\right) \right\rVert_1.

Moreover, stop-gradient serves as a safeguard against geometric-photometric coupling during optimization: gradients through the warped observation and occlusion guidance are blocked from flowing back to the optical-flow estimator, while the direct reconstruction path remains trainable.

To further evaluate whether PFN provides benefits beyond simple residual correction, we compare it with several baselines on Sintel Final. All methods use the same predicted RAFT flow and are evaluated against GT-correspondence RGB increments on common valid pixels.

MethodMean L2 ↓MAE ↓
Raw warping residual2.336271.29029
Low-pass residual2.818441.54059
Global affine correction6.264933.47773
PFN (ours)2.112711.17141

PFN reduces mean L2 and MAE by 9.57% and 9.21% compared with the raw residual, and by 25.04% and 23.96% compared with the low-pass baseline.

These results demonstrate that PFN provides more accurate photometric estimation than simple residual-based alternatives under the same imperfect geometric correspondences, supporting its effectiveness as an explicit appearance representation for downstream video synthesis.

W3. Controls to isolate the source of gains.

1. STRM without photometric input. We train a ProPainter-specific STRM with the same architecture, data, losses, and 200k-step budget. Its six photometric input channels are zeroed during both training and testing.

SettingPSNR ↑SSIM ↑LPIPS ↓VFID ↓
ProPainter33.970.96640.02650.1010
+ STRM without photometric input34.120.93760.03280.1331
+ STRM with photometric guidance34.720.97600.02290.0963

The full model improves PSNR by 0.60 dB and SSIM by 0.0384, while reducing LPIPS and VFID by 30.18% and 27.65%, respectively, over the matched control.

2. STRM training strategy. STRM is trained separately for each backbone, while the original backbone remains frozen. The trained STRM can be plugged into the corresponding backbone without modifying or retraining it.

3. Calibration with matched training budget. We additionally train PFN without calibration for 45k extra steps to control for the additional training budget.

SettingAdditional stepsPSNR ↑SSIM ↑LPIPS ↓VFID ↓
PFN without calibration031.830.94990.09820.1228
PFN without calibration45k31.850.95110.09940.1237
PFN with calibration45k32.150.95580.09690.1209

This control helps isolate the contribution of calibration from the additional training budget.

We will add the detailed experimental settings to the revised main text and appendix.

W4. Photometric variation and distance.

We agree that evaluating challenging photometric variations is important. Our current evaluation focuses on DAVIS and YouTube-VOS, and the limitations of PFN under strong specularities, saturation, overexposure, and nonlinear camera responses are discussed in Appendix C. In future work, we will conduct detailed evaluations on datasets containing scenes with substantial brightness changes.

To further evaluate temporal composition, we conduct a long-range experiment on Sintel Final using GT optical-flow trajectories to isolate photometric accumulation from geometric errors. We compare no correction, accumulated adjacent-frame PFN predictions, and a single endpoint PFN prediction.

GapNo correctionAccumulated PFNEndpoint PFN
12.741442.26512.2651
23.791822.87693.9835
45.524293.69027.6812
88.615425.237214.6732

At an 8-frame gap, accumulated PFN reduces the error by 39.21% compared with no correction and 64.31% compared with a single endpoint prediction. These results demonstrate the benefit of accumulating local photometric corrections along motion trajectories over the tested temporal ranges.

We will add the detailed experimental settings to the revised main text and appendix.

W5. Propagation-integration results and evaluation protocol.

We clarify that ECFVI* is our reproduced and adapted propagation baseline, with its complete pipeline detailed in Appendix A.3 (Fig. 8). It follows a Propagate-then-Compensate strategy: optical-flow estimation and completion, direct content propagation, ECN-style residual compensation, and Stable Diffusion v1.5 synthesis for regions that remain unfilled.

The reported 33.52 dB is the published result of the original ECFVI, whereas 31.81 dB is obtained from our local ECFVI* implementation under our evaluation protocol. These numbers are not directly comparable. Our controlled comparison is between ECFVI* and ECFVI* + PFN under the same evaluation setting:

Stable Diffusion v1.5 is used in both configurations to synthesize regions that cannot be filled by propagation, with the same synthesis setting. Thus, it is not an additional variable introduced only in the PFN setting. The key difference is the correction strategy: our method explicitly corrects source appearance before propagation using photometric flow, rather than applying generic residual compensation after propagation.

Regarding other baselines such as ProPainter, reported numbers are obtained through our own local evaluations. We follow the standard evaluation protocol across methods, including the same DAVIS 50-video subset, resolution, masks, and evaluation region.

W6. Outpainting and interpolation.

Our DAVIS experiments evaluate the cross-task applicability of two integration modes: PFN-based Correct-then-Propagate and photometric-guided refinement. ECFVI* serves as an adapted propagation baseline for testing the benefit of photometric correction across tasks, rather than as a state-of-the-art interpolation method. For refinement, we additionally use task-specific backbones: M3DDM+ for outpainting and BiM-VFI for interpolation.

The improvements across these settings support the applicability of photometric flow in both propagation and refinement pipelines. We will clarify this experimental scope and extend the evaluation to additional task-specific benchmarks and synthesis models in future work.

Thank you again for your careful review of our work and your valuable suggestions. With approximately 29 hours remaining in the author–reviewer discussion period, we wanted to ask whether there is anything in our previous responses or additional experiments that you would like us to clarify or explain further. If you have a chance to review them, we would be very grateful for any further feedback and would be happy to provide additional explanations in the remaining time. Thank you for your time and effort amid your busy schedule.

I thank the authors for the detailed response and the additional experiments, which have partially addressed my concerns about the trivial-residual solution and the contribution of photometric guidance to STRM. However, the core claim is still not validated on videos with strong illumination or exposure changes, and the new STRM control does not fully isolate the source of the gains. I will therefore maintain my recommendation of weak reject.

Reviewer_sqBCOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces photometric flow, a dense RGB correction field aligned with optical-flow correspondences, to model appearance changes during video synthesis. A Photometric Flow Network predicts an additive base field with appearance-conditioned calibration and supports temporal composition and completion in missing regions, and guides either content propagation or spatio-temporal refinement.

Critical strengths
  1. The method try to address appearance variation that geometric warping alone cannot resolve. Improvements across three synthesis tasks and several backbones support its practical applicability.
  2. With ground-truth optical flow fixed, PFN reduces photometric alignment error on Sintel by a large marigin. This supports a benefit beyond changes in flow estimation.
Critical weaknesses

The benefit of photometric flow as a temporally composable representation is not sufficiently established. Table 5 validates pairwise appearance correction, while Table 7 compares additive and multiplicative parameterizations. Neither directly tests whether composing local corrections improves long-range consistency over direct source-to-target correction. The downstream quality gains therefore do not isolate the contribution of temporal compositionality.

Questions for authors

When optical flow is inaccurate, what prevents the calibration branch from absorbing geometric misalignment into the photometric field?

1 comment

Thank you for highlighting the need to isolate temporal compositionality. We report a new controlled long-range comparison and clarify how our training handles imperfect geometric correspondences.

1. Temporal composition versus a single endpoint correction

We agree that Tables 5 and 7 do not directly answer this question: Table 5 evaluates pairwise appearance correction under fixed geometry, while Table 7 compares additive and multiplicative parameterizations.

We therefore conducted a controlled experiment on fully observed Sintel Final sequences. Adjacent ground-truth flows define the motion trajectories. We compare zero correction, accumulated adjacent-frame PFN increments sampled along these trajectories (Eq. 3), and a single PFN prediction from the two endpoint observations. The methods share the same endpoint correspondences and the same valid pixels at each temporal gap. We measure mean per-pixel RGB-vector L2 error against the RGB increment between the GT-corresponding endpoints.

Gap (frames)No correctionAccumulated PFNSingle endpoint PFN
12.741442.26512.2651
23.791822.87693.9835
45.524293.69027.6812
88.615425.237214.6732

At an 8-frame gap, accumulation reduces error by 39.21% relative to no correction and 64.31% relative to a single endpoint prediction. It improves over no correction at every tested gap and over the endpoint prediction at gaps 2, 4, and 8; the two PFN strategies coincide at gap 1.

This comparison therefore tests how to use the current local estimator at inference time and supports composition for long-range estimation under fixed geometry.

2. Geometric misalignment and appearance-conditioned calibration.

The calibration branch refines the base photometric flow conditioned on the base prediction, source image, and warped observation (Eq. 4). It is trained through the reconstruction objective in Eq. (6), where the corrected source image is warped back using backward optical flow.

To reduce geometric-photometric coupling, we employ stop-gradient in Eq. (5): gradients through the warped observation and occlusion guidance are blocked from flowing back to the optical-flow estimator, while the direct reconstruction path remains trainable. This provides a safeguard during optimization, although it does not strictly prevent the calibration branch from responding to inaccurate correspondences.

We further evaluate the complete PFN under imperfect RAFT correspondences, using GT-correspondence RGB increments as the reference. Compared with the raw warping residual, PFN reduces mean L2 from 2.33627 to 2.11271 (9.57%) and MAE from 1.29029 to 1.17141 (9.21%).

These results indicate that PFN learns a more accurate photometric correction than directly using the warping residual, even with imperfect geometric correspondences.

We will add the detailed experimental settings to the revised main text and appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.