acceptodds
Under review as a conference paper at ICLR 2027

TrackLUT: Semantic Color Tracking for Adaptive Video Color Transfer

Abstract

Reference-based video color transfer remains challenging in dynamic scenes, where color transformations must adapt to evolving content and illumination while avoiding temporal flicker. Existing keyframe-based approaches provide limited adaptability, whereas independent frame-wise predictions often sacrifice temporal consistency. To address these limitations, we propose TrackLUT, a frame-adaptive framework that predicts a 3D lookup table (LUT) for each frame and exploits semantic correspondence for both spatial color transfer and temporal propagation. Specifically, semantic features establish reliable correspondences between target content and reference regions, while decoupled color features encode transferable appearance cues. This design enables selective color retrieval from semantically relevant regions, reducing erroneous mappings caused by spatial and appearance discrepancies. To maintain temporal coherence, we further align the historical color state with each incoming frame through semantic correspondence, allowing relevant appearance information to propagate as the scene evolves. The retrieved reference features and aligned temporal state are jointly decoded into a frame-specific LUT. Experiments demonstrate that TrackLUT adapts effectively to substantial content and illumination changes while preserving reference-style fidelity, structural consistency, and temporal stability, enabling efficient real-time video color transfer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video color transfer, Semantic awareness, Temporal consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_D2hdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses reference-based video color transfer in dynamic scenes. Existing keyframe-based methods struggle with changing illumination and scene content, and independent per-frame predictions tend to cause temporal flicker. The authors propose TrackLUT, which predicts a 3D LUT for each frame. Using semantic correspondence for color mapping and temporal state propagation, the model separates semantic and color features to sample colors from semantically consistent reference regions and avoid wrong mappings. Reference features and aligned historical states are decoded into frame-adaptive LUTs. The method is claimed to cope with content and lighting changes while preserving style, structure, temporal stability and real-time performance.

Critical strengths
  1. The technical idea of frame-adaptive 3D LUT prediction guided by semantic correspondence is well aligned with the task requirement. It attempts to combine the computational efficiency of LUT-based color mapping with adaptive capacity for dynamic lighting, which is a reasonable direction for reference-based video color transfer.
  2. The design decouples semantic matching and color appearance representation, which is an intuitive strategy to mitigate misalignment-induced wrong color transfer.
Critical weaknesses
  1. The evaluation is inadequate to support the main claims. While quantitative results are reported, visual evidence is insufficient to confirm that semantic propagation mitigates the inherent flicker and color drift of per-frame LUTs. There is no targeted ablation or comparison to justify the advantages of the frame-adaptive LUT over prior LUT-based approaches. Qualitative results are only shown for some baselines, and tests are limited to simple videos without complex sequences or supplementary demos. This makes it hard to reliably judge the practical effectiveness of TrackLUT.
  2. The presentation and motivation of the paper are suboptimal. The citation format contains errors, and the writing over-emphasizes technical details at the cost of intuitive motivation. The high-level rationale is not easy to follow, which hinders readers from quickly grasping the core insight and contributions.
Questions for authors
  1. From the visual results you have included, could you elaborate on how your temporal semantic propagation alleviates flicker and color drift of per-frame LUTs under complex dynamic scenes? What key observations support this claim?
  2. In your current experiments, what evidence or analysis can demonstrate the advantages of your frame-adaptive 3D LUT design compared with other LUT-based color transfer methods?
1 comment

We sincerely thank the reviewer for the insightful and constructive comments, which have helped us improve the clarity, motivation, and experimental completeness of our work. We have revised the manuscript and conducted additional analyses to address the concerns regarding temporal stability, qualitative comparisons, and frame-wise adaptability. Our point-by-point responses are provided below.

W1:

Thank you for this valuable comment.

(1) Temporal stability. Table 3 provides targeted ablations of HAF and Temporal LUT Gate, showing reductions of 12.0% and 7.5% in wΔEw\Delta E-1 and wΔEw\Delta E-5, respectively. We further provide a qualitative comparison with EMA in Supplementary Fig. A.1, illustrating how semantic-guided history propagation handles newly appearing foreground content.

(2) Qualitative comparisons. All baselines are quantitatively evaluated in Table 1. Due to space constraints, we select representative methods with competitive performance for visualization, including NLUT (best Content Sim.), CAP-VSTNet (best Style Sim.), and ColorFM (competitive overall performance), covering different methodological paradigms. Figure 3 further compares their results under different illumination conditions.

(3) Real-world evaluation. Our test set contains 55 real-world video-reference pairs. We additionally provide temporal lightness response analysis on four representative videos in Supplementary Fig. B.1, demonstrating TrackLUT's responsiveness to changing video appearance.

W2:

We thank the reviewer for this constructive comment. We have revised the manuscript to improve the clarity of its motivation and overall presentation. First, we have corrected several citation formatting errors. Second, we have refined the Introduction to provide a more intuitive motivation by emphasizing the fundamental trade-off between frame-wise adaptability and temporal consistency.

Q1:

Thank you for this insightful comment.

TrackLUT mitigates temporal flickering and color drift through semantic-guided historical feature propagation. Specifically, HAF establishes patch-level semantic correspondences between consecutive frames using DINOv2 features. For each current-frame patch, it retrieves the corresponding historical color state and adaptively determines its contribution according to semantic similarity. Regions with reliable correspondences retain more historical color information, reducing inconsistent frame-wise color retrieval, while newly appearing or semantically mismatched regions rely more on current-frame features to avoid propagating irrelevant historical information.

Several visual observations support this design. First, in Supplementary Fig. A.1, when a foreground subject enters the scene, the HAF weight maps exhibit spatially selective responses, illustrating content-aware historical information propagation. Compared with EMA, TrackLUT better accommodates the newly appearing content while maintaining a coherent appearance. Second, Fig. 3 shows that under substantial indoor-to-outdoor illumination changes, fixed keyframe LUT methods exhibit under- or over-stylization, whereas TrackLUT adapts its color mapping while maintaining style coherence. Finally, Supplementary Fig. B.1 shows that the transferred videos preserve the major temporal lightness trends of the input sequences.

Q2:

Thank you for this insightful comment.

First, Table 1 shows that TrackLUT achieves the lowest distance to the ideal point (0.282), outperforming ColorFM (0.295).

Furthermore, Figs. 3 and 5 demonstrate that our method achieves stronger semantic matching capabilities compared with other methods.

Finally, Fig. 3 provides direct visual evidence under indoor-to-outdoor illumination changes. When the appearance of the current frame deviates from that of the keyframe, fixed keyframe LUTs tend to result in under- or over-stylization. In contrast, TrackLUT dynamically adjusts the predicted frame-specific LUTs to accommodate evolving content and illumination while maintaining overall style coherence. Supplementary Fig. A.1 further illustrates the advantages of adaptive LUT prediction over the EMA baseline.

Reviewer_5BDgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TrackLUT, a reference-based video color transfer method that predicts a separate 3D LUT for each frame. TrackLUT uses DINOv2 features to establish semantic correspondences between the input frame and reference image, along with a separate color encoder to retrieve appearance information. For temporal consistency, it aligns historical color features using semantic similarity and fuses them with the current features. A gated fusion module is also introduced to stabilize the LUT predictions over time. The results are competitive in terms of style similarity, content preservation, temporal consistency, and inference speed, although the improvements over existing methods are relatively modest.

Critical strengths
  • The paper addresses a practical problem. The trade-off between frame-wise color adaptation and temporal consistency is well motivated.

  • The overall framework is reasonably designed and easy to follow. Using semantic correspondence for both reference color retrieval and temporal alignment makes sense.

  • The method achieves competitive results with low inference latency. The qualitative examples also look promising, especially under changing lighting conditions.

Critical weaknesses
  • Missing simple baselines. A frame-wise LUT predictor with simple EMA or adaptive smoothing might achieve similar results, but such comparisons are missing.
  • The temporal training relies on synthetic sequences generated from still images, with fixed target LUTs within each segment. It's unclear how well this setup captures continuous lighting and appearance changes in real videos.
  • In Table 3, wΔE-35 only improves from 5.257 to 5.239 with both temporal modules. This is a very small difference and doesn't really support the claims about long-term temporal stability.
  • The temporal consistency metric seems to mainly measure color differences between flow-aligned frames. However, lower color differences do not necessarily mean better temporal consistency, especially when the lighting changes significantly. How can this metric distinguish unwanted color flickering from legitimate color changes caused by scene dynamics?
  • Incomplete temporal consistency evaluation. The paper only uses flow-aligned CIEDE2000 (wΔE) to evaluate temporal consistency, which I think is not enough. There are several other commonly used metrics, such as Color Distribution Consistency (CDC) from TCVC [1], temporal warping error [2], and temporal perceptual metrics like tLP from TecoGAN [3]. It would be useful to include some of these metrics for a more comprehensive evaluation. Also, lower frame-to-frame color differences do not necessarily mean better results, especially when lighting or scene content changes. I would like to see separate evaluations on stable and dynamic scenes to better understand the trade-off between temporal consistency and frame-wise adaptation.

[1] Liu et al., Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning. Computational Visual Media, 2024.

[2] Lai et al., Learning Blind Video Temporal Consistency. ECCV, 2018.

[3] Chu et al., Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation. ACM TOG (SIGGRAPH), 2020.

Questions for authors

Please see the above weaknesses.

2 comments

We sincerely thank the reviewer for the insightful and constructive comments, which have helped us improve the clarity, experimental completeness, and technical rigor of our work. We have conducted additional experiments and analyses to address the concerns regarding temporal consistency and frame-wise adaptability, as detailed in our point-by-point responses below.

W1:

Thank you for this valuable suggestion. We have added a frame-wise LUT prediction baseline with EMA smoothing in Supplementary Appendix A, evaluating different smoothing coefficients (β∈{0.1,0.3,0.5,0.7,0.9}\beta \in \{0.1, 0.3, 0.5, 0.7, 0.9\}).

As shown in Table A.1, EMA with β=0.1\beta=0.1 achieves slightly lower temporal errors than TrackLUT. However, fixed-coefficient EMA uniformly smooths the LUT without considering changes in semantic content, whereas TrackLUT adaptively updates historical information through semantic alignment and gated fusion.

More importantly, Fig. A.1 provides a qualitative comparison with EMA (β=0.1\beta=0.1) as a foreground subject gradually enters the scene. The heatmaps visualize the history fusion weights of HAF, illustrating how TrackLUT adjusts historical information according to the current semantic content. This highlights the distinction between fixed temporal smoothing and content-aware temporal adaptation.

These results are included in Supplementary Appendix A (Table A.1 and Fig. A.1).

W2:

Thank you for raising this important concern. Our design aims to provide controlled supervision for two complementary behaviors: temporal stability under gradual frame variations and adaptation to changes in semantic content.

Specifically, we apply smooth geometric and photometric perturbations within each segment, while keeping the target LUT fixed. At a randomly sampled transition point, the content switches to another image and the target LUT changes accordingly. This provides explicit supervision for maintaining consistent color mappings under gradual variations and updating historical information when the scene content changes.

To evaluate the applicability of our method to dynamic real-world videos, we have included a temporal lightness response analysis in Supplementary Appendix B (Fig. B.1). By comparing the frame-wise mean CIE Lab lightness and its relative temporal changes, we demonstrate that TrackLUT preserves the major temporal lightness trends of the input videos.

W3:

Thank you for this observation. We agree that the improvement in wΔEw\Delta E-35 in Table 3 is marginal. This is partly related to the design of our temporal modules. HAF establishes semantic correspondence with the immediately preceding frame to selectively propagate relevant historical color information, while the temporal LUT gate regulates the transition between consecutive LUT representations. Therefore, their primary effect is to stabilize local temporal updates rather than explicitly model long-range dependencies.

However, we would like to clarify that the long-term temporal consistency of TrackLUT is also reflected in the overall comparison in Table 2, rather than solely in the incremental gains of these two modules. TrackLUT achieves a wΔEw\Delta E-35 of 5.239, outperforming frame-wise ColorFM (6.345) and its global variant (5.464).

W4:

Thank you for this insightful comment. We would like to clarify that our evaluation considers not only flow-aligned wΔEw\Delta E, but also the method's ability to respond to dynamic scene changes. The flow-aligned metric evaluates temporal color consistency in motion-corresponding regions, accounting for object motion and occlusions.

As illustrated in Fig. 3, we include challenging scenarios with substantial illumination changes, particularly an indoor-to-outdoor transition. Compared with methods relying on frame-independent predictions or fixed keyframe transformations, TrackLUT adapts its color transformations to evolving illumination conditions while maintaining temporal coherence.

To further examine this behavior, we have added a temporal lightness response analysis in Supplementary Appendix B (Fig. B.1), comparing the frame-wise mean CIE Lab lightness and its relative temporal changes between the input and transferred videos across four real-world sequences. The results show that TrackLUT preserves the major brightness trends of the original videos.

W5:

Thank you for this constructive suggestion. We agree that temporal consistency should be evaluated from multiple perspectives and that smaller inter-frame differences do not necessarily indicate better video color transfer.

Additional temporal metrics. We have incorporated tLP and temporal warping error (EwarpE_{\mathrm{warp}}) into Table 2 of the revised manuscript. TrackLUT achieves the best tLP (0.0040) and the second-best warping error (0.000536), demonstrating competitive temporal consistency across different metrics.

Evaluation under different scene dynamics. Following your suggestion, we divide the 55 test videos into 27 static and 28 dynamic scenes and report separate evaluations in Supplementary Appendix C. TrackLUT achieves the lowest tLP in static scenes and the lowest warping error with the joint-lowest tLP in dynamic scenes.

Trade-off between temporal consistency and adaptability. We further investigate this trade-off through comparisons with EMA under different smoothing strengths (Appendix A) and temporal lightness response analysis (Appendix B). The results show that stronger temporal smoothing can reduce temporal errors but may hinder adaptation to newly appearing content, highlighting the importance of balancing temporal stability and frame-wise adaptability.

Table R1. Quantitative comparison of temporal color consistency. Lower values indicate better temporal consistency. Methods marked with (∗*) perform frame-wise inference, while the remaining methods apply a globally inferred transformation to the entire video. Best and second-best results are shown in bold and italics, respectively.

TypeMethodwΔEw\Delta E-1 ↓wΔEw\Delta E-5 ↓wΔEw\Delta E-35 ↓tLP ↓EwarpE_{\mathrm{warp}} ↓CDC ↓
Color-onlyWCT$^2$ (∗*)1.5002.7455.7900.00880.0005930.0049
Color-onlyNLUT1.2242.2914.9400.00730.0006490.0020
Color-onlyNeural Preset (∗*)1.2872.3185.2440.00900.0006600.0078
Color-onlyVideoColorGrading1.4852.6685.9000.00490.0006290.0019
Structure-semantic-awareCAP-VSTNet (∗*)1.4372.6955.8090.00750.0005080.0030
Structure-semantic-awareSA-LUT (∗*)2.8894.8528.6860.00900.0013660.0047
Structure-semantic-awareColorFM (∗*)1.9923.3596.3450.00700.0008230.0236
Structure-semantic-awareColorFM1.3262.3805.4640.00550.0006740.0019
Structure-semantic-awareOurs (∗*)1.2642.3175.2390.00400.0005360.0105

Discussion on CDC. We also evaluated CDC, on which TrackLUT achieves a score of 0.0105, indicating that its global color distribution consistency remains an area for improvement. For professional video color grading, maintaining consistent color transformations while adapting to changes in scene content and illumination remains a challenging problem. Although TrackLUT takes an initial step toward addressing this challenge through semantic-guided reference retrieval and historical color propagation, its semantic-aware temporal modeling can be further improved. We will continue to improve global color distribution consistency and explore semantic-aware evaluation metrics that distinguish undesirable temporal fluctuations from necessary appearance adaptation.

Comments on the paper

1 comment

We sincerely thank the reviewer for the insightful and constructive comments, which have helped us improve the experimental completeness, methodological clarity, and technical rigor of our work. We have conducted additional experiments and analyses and revised the manuscript accordingly to address the concerns regarding temporal consistency, frame-wise adaptability, computational efficiency, and training data construction. Our point-by-point responses are provided below.

W1:

Thank you for this detailed comment. We agree that temporal consistency should be evaluated jointly with transfer quality and adaptability, rather than relying solely on frame-to-frame color differences. We have therefore supplemented our evaluation from several complementary perspectives.

(1) Transfer quality and temporal consistency. As reported in Table 3, we evaluate Content Similarity and Style Similarity alongside temporal consistency for all ablated configurations. Both metrics remain comparable across configurations (Content Sim.: 0.760–0.763; Style Sim.: 0.851–0.855), indicating that temporal stabilization does not substantially compromise transfer quality. Meanwhile, the joint application of HAF and LUT Gate reduces wΔEw\Delta E-1 and wΔEw\Delta E-5 by 12.0% and 7.5%, respectively. We acknowledge that the improvement at the 35-frame interval is marginal (0.34%) and have moderated our claims accordingly.

(2) Comparison with generic temporal smoothing. We have added an EMA baseline in Supplementary Appendix A, applying first-order IIR smoothing directly to the same frame-wise predicted LUTs with different update coefficients (β∈{0.1,0.3,0.5,0.7,0.9}\beta \in \{0.1, 0.3, 0.5, 0.7, 0.9\}). The results reveal a clear smoothing-adaptation trade-off: stronger smoothing reduces measured temporal errors, and EMA with β=0.1\beta=0.1 even achieves slightly lower errors than TrackLUT. However, unlike EMA, TrackLUT does not uniformly blend consecutive LUT predictions. Instead, HAF selectively propagates historical color information according to semantic correspondence, while the learned LUT gate adaptively regulates temporal updates. As illustrated in Supplementary Fig. A.1, this distinction becomes particularly relevant when new foreground content enters the scene.

(3) Adaptability to dynamic appearance changes. Figure 3 provides qualitative comparisons under both stable illumination and substantial indoor-to-outdoor lighting changes. Furthermore, we have added a temporal lightness response analysis in Supplementary Appendix B (Fig. B.1), examining four real-world video sequences. These results show that TrackLUT preserves the major brightness trends of the input videos, providing complementary evidence of its responsiveness to dynamic appearance changes.

W2 & Q1:

Thank you for pointing out the ambiguity in our timing protocol. We have revised Table 1 and clarified the timing details in the manuscript.

(1) TrackLUT's initialization and steady-state inference costs. The reported 13 ms represents the complete per-frame inference latency, including DINOv2 semantic feature extraction, Color Encoder, History-Aligned Fusion (HAF), Temporal LUT Gate, LUT decoding, and final LUT interpolation and application. The computational difference between the initial and steady states arises only from HAF and Temporal LUT Gate, whose combined overhead is less than 1 ms. Additionally, applying the predicted LUT takes less than 1 ms per frame. Therefore, TrackLUT maintains an end-to-end inference latency of approximately 13 ms per frame (about 77 FPS) at 512×512512\times512 resolution, supporting real-time frame-adaptive video color transfer.

(2) Initialization and application costs of global LUT methods. For NLUT and VideoColorGrading, the reported 6.256 s and 5.648 s correspond to the one-time generation of video-level LUTs, rather than per-frame inference. Once generated, these LUTs are reused across subsequent frames, and their application at 512×512512\times512 resolution takes less than 1 ms per frame. We have explicitly marked these two methods with asterisks in Table 1 and clarified this distinction in the caption.

Q2:

Thank you for this question. The 1,000 training triplets were constructed through a curated selection and professional color grading process. Specifically, the target images were manually color-graded by professional in-house colorists to match the desired reference styles, providing high-quality supervision for our model.

Regarding data separation, the training triplets were constructed from still images, while the test set consists of independently collected real-world videos. The training and test data are completely disjoint, with no overlap between the training images and test video frames.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.