acceptodds
Under review as a conference paper at ICLR 2027

Leveraging Multispectral Images for Cross-Camera Color Restoration

Abstract

Color restoration, comprising white balance and color correction, is a core stage of the image signal processor (ISP) that maps the raw sensor response into a standard, device-independent color space. Yet with only three broadband RGB channels, disentangling illumination from reflectance is ill-posed, and these operations are tied to a camera-specific raw space that must be recalibrated for every new sensor. Multispectral (MS) sensors sample the spectrum more densely and provide a physical constraint that breaks this underdetermination. Existing methods still underuse this signal, as most apply it only to illuminant estimation, while the few that fuse multispectral image (MSI) with RGB across the whole pipeline train a separate model per camera and cannot transfer to unseen sensors. We present a physics-guided framework that uses a single fixed multispectral camera, treating the MSI as a *camera-agnostic spectral bridge* and exploiting it across the entire color-restoration pipeline. The three-stage framework predicts CSS and white-balance gains, followed by per-image color correction. Crucially, we optimize CSS in the sensor-response domain and use CSS projection to separate sensor dependence from illumination learning. We train the framework once, and it transfers to a new target RGB camera using only the DNG ColorMatrices of the target camera and a small set of MSI–RGB pairs, without retraining. Experimental results show that our framework outperforms existing cross-camera color-constancy and illuminant-estimation baselines and, in the cross-camera setting, approaches the color-restoration accuracy of single-camera MSI methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multispectral image processing, cross-camera generalization, color restoration, computational color constancy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JrApOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper proposes to solve the color restoration task for new cameras, including both white balancing and color correction. To avoid training separate model for each camera with a specific sensor response, they use a fixed 16-band multispectral camera as a common "spectral bridge" between different cameras, since the spectral radiance and reflectance are censor independent.
  2. The proposed method has three stages:
    1. Camera spectral sensitivity recovery. Starting from the cemera's two DNG ColorMatrics from the meta data, they estimate the CSS using paired MSI-RAW images using ridge regression. Linear regression here is to avoid the overfitting. The ground-truth CSS of the camera is unnecessary but the CSS of the MSI is needed.
    2. Estimate the illumination together with the white balance. A shallow network takes in both MSI and RAW images predict the illumination and white balancing curve.
    3. A scene-specific color correction matrix, They create a virtual color chart using the estimated illumination and CSS from the first two stages.
Critical strengths
  1. What I like the most is they ideally combine the physics and learning. Instead of learning RAW+MSI to XYZ end-to-end, they explicitly decompose the problem into CSS, illumination, white balancing and CCM estimation, which makes the pipeline more interpretable.
  2. Ablation study in Table 3(c) shows that the CSS projection is convincing. Other ablation studies also makes the contribution of each part clear.
  3. The proposed method work well on unseen cameras compared with other methods, as shown in Table 1.
  4. Experiments in Table 12 show the importance of the quality of SPD estimation, which is a good analysis.
Critical weaknesses
  1. What I confused is the paper mentioned that the propose method works on new cameras. But given a new camera, the stage-1 still need paired MSI-RAW captures for CSS optimization on 60 pairs. This is not pure zero-shot transferring. This makes it hard to deploy. Besides, calibration between MSI and RGB cameras is also non-trivial.
  2. Synthetic training, which may not work well on real data.
  3. Table 2 shows the performance is better than RGB-Only methods. This is reasonable, but it is only comparable with other RGB+MSI methods.
  4. Performance on the real data in Table 14 still shows a noticeable domain gap.
Questions for authors
  1. Why do you think the MSI and RAW together are enough to recover the illumination? It is still highly ill-posed.
  2. How much target-camera data is really necessary in practice?
  3. How sensitive is the method to MSI-RAW misalignment?
  4. In practice, it is impossible to get the perfect MSI CSS, how sensitive is the method to MSI CSS error.
5 comments
AuthorsOpenReview ↗

Response to Reviewer JrAp (Part 1/3): Calibration Requirements and Error Sensitivity

Dear Reviewer JrAp,

We sincerely thank you for your thoughtful review and constructive feedback. We appreciate your recognition of the integration of physics and learning in our framework. Your comments highlight the need for clearer calibration requirements, stronger experimental validation, and further analysis of illuminant estimation and robustness.

We have uploaded a revised manuscript. Major additions and revisions are highlighted in blue, including the updated and expanded real-world experiments in Appendix G and the new sensitivity analyses of calibration errors, measurement errors, and MSI–RAW spatial misalignment in Appendix H. We address your concerns individually below.

W1 & Q2 & Q3: Target-camera calibration requirements and sensitivity to MSI-RAW misalignment

Thank you for highlighting the calibration requirements for adapting to a new camera. We agree that our method requires one-time CSS calibration for a new camera and therefore does not constitute entirely calibration-free, zero-shot transfer. The cross-camera generalization we emphasize is that a model can be transferred to an unseen camera after one-time calibration using a small set of paired MSI-RAW observations without tuning the model.

To further evaluate the amount of calibration data required, we compare different numbers of MSI-RAW calibration pairs in Appendix C, Table 7. With all network weights and the test set fixed, using only 20 pairs achieves a mean white-point angular error of 1.362° ± 0.005° and a mean ΔE00\Delta E_{00} of 2.385 ± 0.008, close to 1.349° and 2.376 with the default 60 pairs. These results show that, under the evaluated setting, 20 calibration pairs are sufficient to achieve accuracy comparable to the default configuration. Moreover, these calibration pairs can be captured from everyday scenes, making the required calibration data easy to collect.

Regarding your concern about calibration between MSI and RGB cameras, we conduct additional spatial-misalignment sensitivity experiments, with detailed settings and results provided in Appendix H.4. During calibration, as the typical input displacement before registration increases from 0 to 41.34 pixels, the final mean ΔE00\Delta E_{00} remains within 2.3758-2.3772 after applying the existing registration and spatial-averaging procedure. During inference, increasing the typical input displacement from 22.53 to 40.64 pixels raises the mean ΔE00\Delta E_{00} only from 2.3764 to 2.4491, an increase of approximately 3.06%. These results demonstrate that our framework is robust to spatial misalignment between MSI and RGB within the evaluated range.

Q4: Sensitivity to MSI spectral-response calibration errors

Thank you for highlighting the inevitable errors in practical MSI spectral-response calibration. We have added sensitivity experiments to separately evaluate wavelength shifts and relative per-band gain errors. We perturb only the MSI response curves used in Stage 1 and re-estimate the target RGB-camera CSS, while keeping all network weights fixed. Detailed experimental settings and results are provided in Appendix H.1.

For wavelength calibration errors, the method maintains good color-restoration accuracy when the response curves are shifted by +1 or +3 nm, with mean ΔE00\Delta E_{00} values of 2.416 and 2.765, respectively. At a larger shift of +5 nm, performance degrades to 3.356, but remains better than the DNG interpolation baseline of 3.86.

For per-band gain calibration errors, maximum relative gain deviations of 0.1%, 0.3%, and 0.5% yield mean ΔE00\Delta E_{00} values of 2.392, 2.476, and 2.623, respectively. These values remain close to the unperturbed baseline of 2.376, indicating good tolerance to gain calibration errors over the tested range. Overall, these results show that our framework maintains good color-restoration performance under the tested MSI spectral-response calibration errors and does not require perfectly accurate MSI spectral responses.

AuthorsOpenReview ↗

Response to Reviewer JrAp (Part 2/3): Real-Data Validation and the Domain Gap

W2 & W4: Real-data validation and the synthetic-to-real domain gap

Thank you for highlighting the issues of real-data validation and the domain gap. Due to differences between synthetic image formation and real observations, as well as the use of different MS cameras in our synthetic setup and the real dataset, models trained on synthetic data are difficult to transfer directly to real datasets. Meanwhile, Beyond-RGB contains only two smartphone cameras, making it difficult to support cross-camera training and independent evaluation with sufficient camera diversity. We therefore train and evaluate the complete pipeline on the Samsung single-camera subset to assess the effectiveness of our framework under real imaging conditions. Detailed experiments are provided in Appendix G.

(1) Mismatch between real observations and the ideal imaging model

Real MSI observations exhibit a noticeable mismatch with the ideal linear imaging model used for synthetic training. We synthesize channel responses using the authors' measured chart reflectances, measured illuminant spectra, and published multispectral response curves, and fit one global exposure scalar per scene. On the test scenes, the mean scene-wise relative Frobenius residual is 22.5%, and the mean 16-channel response angular error is 4.99°. After excluding saturated observations, the relative residual remains 15.4%. Introducing a shared empirical power-response model fitted only on the training set further reduces the residual to 11.42%, indicating a systematic intensity-dependent mismatch.

To address this mismatch, we adopt a CSS calibration variant based on scene-shared illuminant estimation. Without using the measured illuminant spectra of the calibration scenes, the mean imaging angular error decreases from 1.596° at initialization to 0.967°, approaching the reference result of 0.915° obtained using ideal radiance inputs. This demonstrates that the calibration variant alleviates the effect of the mismatch in measured observations.

(2) Illuminant estimation on real data

We evaluate Stage 2 on the Samsung subset of Beyond-RGB, comparing SPD prediction with Spectrum-Aware and white-point estimation with MSI+RAW-extended FC4 and ConvMean under the same data splits. Detailed experimental settings and results are provided in Appendix G.2.

MethodParameters (M) ↓Mean SPD angular error (°) ↓Median (°) ↓
Spectrum-Aware16.3013.5742.704
Ours0.6782.9971.524
MethodMean white-point angular error (°) ↓Median (°) ↓
FC4 (extended)1.2600.871
ConvMean (extended)1.8981.474
Ours1.1470.670

Our method achieves lower mean and median SPD errors while reducing the parameter count by approximately 95.8% relative to Spectrum-Aware. It also achieves lower white-point errors than both extended baselines under the same MSI+RAW input setting, supporting the effectiveness of our Stage 2 design on real data.

(3) Complete color restoration and residual refinement on real data

We have additionally trained the residual refinement module on real data, as detailed in Appendix G.3. Since Beyond-RGB does not provide dense per-pixel XYZ ground truth, we train the residual module using sparse color supervision from 24 ColorChecker patches in the training images. We also include per-scene chart-fitted matrices as a reference. The results are as follows:

MethodMean ΔE00\Delta E_{00} ↓
DNG interpolation5.286
Dynamic physical CCM4.354
Dynamic physical CCM + residual refinement3.989
Per-scene chart-fitted 3×3 matrix3.584

Residual refinement further reduces the color error of the physical CCM from 4.354 to 3.989. To understand the remaining errors on real data, we also directly fit an independent 3×3 matrix using the 24 chart patches in each test scene and evaluate it on those same patches. Even in this setting, the mean ΔE00\Delta E_{00} remains 3.584, indicating that errors on real data also involve the limited expressiveness of linear color mappings, as well as acquisition, chart-patch extraction, and reference-color consistency. Our complete pipeline approaches this per-scene fitting reference, demonstrating the effectiveness of our model on the real dataset.

Although our method still exhibits some performance degradation on real data, the experiments above show that physical color correction and residual refinement remain effective in improving color restoration. Compared with MSCC, which is trained using dense XYZ ground truth, our method can train the residual module using sparse color supervision from a chart, providing a practical route to training on real data without dense XYZ annotations.

AuthorsOpenReview ↗

Response to Reviewer JrAp (Part 3/3): Same-Input Comparisons and Ill-Posed Illuminant Recovery

W3: Comparison with methods using the same RGB+MSI input

We agree that comparison with methods using the same RGB+MSI input is important for evaluating our framework. Table 2 already includes two MSCC models that use RGB+MSI input. We further address this concern through cross-camera comparisons and additional simple baselines using the same inputs.

(1) Cross-camera color restoration with multispectral information

In Table 2, MSCC is trained specifically for the target camera and achieves a lower mean color error than our method (2.31 versus 2.42). Our method achieves comparable accuracy after one-time CSS calibration of the target camera. To further evaluate cross-camera performance, we additionally apply the models to unseen cameras with their network weights fixed. Results for three target cameras are shown below.

CameraOurs: mean ΔE00\Delta E_{00} ↓MSCC cmKAN-light: mean ΔE00\Delta E_{00} ↓MSCC LPIENet: mean ΔE00\Delta E_{00} ↓
Sony ILCE-64002.1233.0363.059
Panasonic DC-GX92.3292.7752.635
Hasselblad L2D-20c2.3972.7252.815
Mean2.2832.8452.837

Our method achieves accuracy comparable to camera-specific MSCC models without retraining on the target camera. When these models are also transferred directly to unseen cameras, our method achieves substantially lower color errors on all three cameras above. These results demonstrate that our framework supports cross-camera transfer while retaining competitive color-restoration accuracy.

(2) Additional simple baselines with the same MSI+RGB input

We extend FC4 and ConvMean from their original inputs to jointly accept a 16-band MSI and a three-channel linear RAW image. We area-downsample RAW to the MSI resolution, concatenate the two inputs along the channel dimension, and adjust the input convolution accordingly, leaving the remaining architecture unchanged. Both baselines directly predict the camera-space white point, followed by conventional DNG interpolation for color correction.

Both baselines are trained specifically on Sony ILCE-7RM2 data and evaluated on the same 770 test images used in Table 2.

MethodInputTraining settingMean white-point angular error (°) ↓Mean ΔE00\Delta E_{00} ↓
FC4RGBTarget-camera training1.4633.69
FC4 (extended)MSI+RGBTarget-camera training1.6574.08
ConvMeanRGBTarget-camera training1.6634.01
ConvMean (extended)MSI+RGBTarget-camera training1.4693.82
Ours (full framework)MSI+RGBSource-camera training1.4612.42

Even with the same MSI+RGB input and target-camera training, these extended baselines exhibit a substantial gap in final color-restoration accuracy: their mean ΔE00\Delta E_{00} values are 4.08 and 3.82, compared with 2.42 for our full framework. This indicates that the improvement in color-restoration accuracy comes not only from the introduction of MSI information but also from the design of our three-stage framework.

Q1: Ill-posedness of illuminant spectrum estimation

Thank you for pointing out the ill-posedness of illuminant recovery. We agree that, even with both MSI and RAW observations, unknown surface reflectance and illumination remain coupled, and these observations alone cannot guarantee unique recovery of the illuminant spectrum. Our method estimates illumination under the constraints of multispectral observations and learned priors, rather than assuming that MSI and RAW make the problem fully well-posed.

Specifically, estimating a high-dimensional illuminant spectrum from three-channel RAW alone is constrained by very limited observations. Introducing a 16-band MSI provides additional spectral constraints, thereby alleviating the ill-posedness of illuminant estimation; the network further uses priors learned during training to estimate the illuminant spectrum. Experimentally, Table 3(b) shows that the spectral-preserving backbone using only MSI achieves a mean SPD angular error of 5.08°, which decreases to 4.91° with the addition of the RAW branch, indicating that the two observations provide complementary information. The full model achieves a mean SPD angular error of 4.71° on unseen target cameras. These results show that, although illuminant recovery remains inherently ambiguous, multispectral observations combined with learned priors support effective illuminant spectrum estimation.

Thank you again for your careful assessment and valuable suggestions. We hope our responses and the added experiments address your concerns. We welcome any further questions or discussion.

Reviewer_fAstOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a cross-camera color-restoration framework that uses a fixed multispectral camera as a camera-agnostic "spectral bridge." Rather than learning an end-to-end RGB-to-XYZ mapping, the method follows the ISP pipeline and decomposes restoration into three stages: (1) recovery of the target camera's spectral sensitivity (CSS) from DNG ColorMatrices plus a small set of paired MSI-RAW observations, refined in the response domain; (2) joint estimation of the illuminant spectrum and white-balance gains, where a shared spectral representation is projected through the target CSS to obtain the camera-space white point; (3) construction of a per-image physical CCM via virtual-chart calibration, followed by a learned residual correction. The framework is trained once on 36 source cameras and applied to 14 held-out target cameras (Sony, Panasonic, Hasselblad) without retraining, requiring only a one-time CSS calibration per target camera.

Critical strengths
  • The motivation is clear and physically grounded: multispectral observations can provide spectral constraints unavailable from RGB measurements alone.

  • The proposed CSS recovery, illuminant estimation, and dynamic CCM stages form a coherent pipeline, and each component is mathematically formulated.

  • The experimental protocol covers multiple relevant settings and the paper has potential practical value for camera interoperability and color-consistent imaging.

  • The work is potentially useful beyond the specific benchmark setting, as it highlights the value of spectral information for physically grounded color restoration.

Critical weaknesses

(1) Method

  • The proposed method receives multispectral information, while the RGB baselines appear to be evaluated using RGB-based white balance and CCM estimation through DNG interpolation. Therefore, the current comparison may primarily demonstrate the value of additional multispectral measurements rather than the effectiveness of the proposed three-stage design. To isolate the methodological contribution, the paper should include simple MSI-aware baselines that use the same multispectral input.
  • Because the method relies on paired RGB and multispectral images, what is the impact of the registration quality to CSS recovery and the final color restoration?

(2) Experiment

  • The main empirical evidence relies heavily on synthetic data. More extensive real-camera validation is needed to support the claimed practical applicability. And the paper should explain why the performance on real data is relatively weak.
  • Compared with MSCC, which also exploits multispectral information, the gains are not consistently substantial across all reported metrics. Since MSCC is a more directly comparable method than RGB-only baselines, the paper should clarify where the improvement comes from and provide per-stage evidence that CSS recovery, illuminant estimation, and dynamic CCM each contribute meaningfully.
  • In Table 3(a), Why the CSS angular error increases substantially, from 6.94 to 32.10, after adding only the CM anchor.
Questions for authors
  • Can the authors compare against simple MSI-aware baselines that use the same multispectral input but replace CSS recovery, illuminant estimation, and dynamic CCM with standard or fixed alternatives? Such comparisons are necessary to distinguish the contribution of the proposed three-stage framework from the benefit of having access to MSI measurements.

  • What are the main causes of the weaker real-data performance, and can the authors provide a failure analysis?

6 comments
AuthorsOpenReview ↗

Response to Reviewer fAst (Part 1/3): MSI–RAW Misalignment and Smoothness Regularization

Dear Reviewer fAst,

We appreciate the care you have taken in evaluating our work and the constructive suggestions you have offered. Thank you for recognizing the design and potential practical value of our framework. Your comments help us identify where the methodological evidence and practical limitations need clearer discussion.

We have uploaded a revised manuscript. Major additions and revisions are highlighted in blue, including the updated and expanded real-world experiments in Appendix G and the new sensitivity analyses of calibration errors, measurement errors, and MSI–RAW spatial misalignment in Appendix H. We address your concerns individually below.

W2: Impact of MSI-RAW spatial misalignment

We have added systematic sensitivity experiments to separately evaluate the effects of MSI-RAW spatial misalignment during CSS calibration and inference. All network weights remain fixed throughout these experiments. Detailed experimental settings and result tables are provided in Appendix H.4.

(1) Calibration stage: Impact on CSS recovery

We retain the original 60 calibration scenes and MSI inputs for each target camera, vary the geometric transformations applied to RAW images, and recover CSS using the existing registration and spatial-averaging procedure. As the typical input displacement increases from 0 to 41.34 pixels, the imaging angular error of the recovered CSS remains close to 0.157°, while the final mean ΔE00\Delta E_{00} stays within 2.3758-2.3772. These results indicate that the calibration procedure maintains stable CSS recovery and downstream color-restoration accuracy under the evaluated geometric variations.

(2) Inference stage: Impact on final color restoration

We fix the recovered CSS and MSI inputs and vary the geometry of the test RAW images. The corresponding XYZ ground truth undergoes the same transformation to maintain consistency with the RAW input. Increasing the typical input displacement from 22.53 pixels in the original setting to 40.64 pixels raises the mean ΔE00\Delta E_{00} only from 2.3764 to 2.4491, an increase of approximately 3.06%. Thus, larger spatial misalignment causes only modest degradation in final color-restoration accuracy over the tested range.

Together, these experiments demonstrate that our framework maintains stable CSS calibration and color-restoration performance across the evaluated range of MSI-RAW spatial misalignment.

W5: CSS estimation instability without spectral smoothness

Thank you for drawing attention to this observation. The distinction between the two configurations in Table 3(a) may not have been sufficiently clear: 6.94° corresponds to the CM-anchor-only initialization, whereas 32.10° corresponds to adding paired MSI-RAW response fitting to this initialization without spectral smoothness regularization. The increase in error therefore reflects the instability of response-domain fitting in the absence of regularization. Without smoothness regularization, the estimated sensitivities may depart from plausible spectral curves, increasing both CSS and imaging errors. With second-order spectral smoothness regularization, the CSS angular error decreases to 4.34°, and the imaging angular error decreases from 2.22° at initialization to 0.157°. This ablation demonstrates that smoothness regularization is an important component for stably incorporating paired response information and effectively refining the CSS initialization. Appendix B provides further stability experiments with different smoothness weights.

AuthorsOpenReview ↗

Response to Reviewer fAst (Part 2/3): Real-Data Validation and Failure Analysis

W3 & Q2: Real-data validation and stage-wise analysis

We have added a real-data evaluation of the complete pipeline and analyze its effectiveness and remaining errors across the three stages. The revised experiments are detailed in Appendix G.

(1) Dataset limitations

Real-world cross-camera validation requires paired MSI-RAW images, measured illuminant spectra, and color references. Currently, Beyond-RGB is the only publicly available real dataset we can access that meets these requirements. However, it contains only two smartphone cameras, making it difficult to support cross-camera training and independent evaluation with sufficient camera diversity. We therefore evaluate the complete pipeline in a single-camera setting using the Samsung subset. In future work, we will seek to expand the coverage of real cameras and scenes to further evaluate cross-camera generalization.

(2) Stage-wise effectiveness and error analysis on real data

Stage 1: Mismatch between measured MSI and the ideal imaging model

We synthesize channel responses using the authors' measured chart reflectances, measured illuminant spectra, and published multispectral response curves, and fit one global exposure scalar per scene. On the test scenes, the mean scene-wise relative Frobenius residual is 22.5%, and the mean 16-channel response angular error is 4.99°. After excluding saturated observations, the relative residual remains 15.4%. Introducing a shared empirical power-response model fitted only on the training set further reduces the residual to 11.42%, indicating a systematic intensity-dependent mismatch.

To address this mismatch, we adopt a CSS calibration variant based on scene-shared illuminant estimation. The mean imaging angular error decreases from 1.596° at initialization to 0.967°, approaching the reference result of 0.915° obtained using ideal radiance inputs. This demonstrates that the calibration variant alleviates the effect of the mismatch in measured observations.

Stage 2: Illuminant prediction on real data

We evaluate Stage 2 on the Samsung subset of Beyond-RGB, comparing SPD prediction with Spectrum-Aware and white-point estimation with MSI+RAW-extended FC4 and ConvMean under the same data splits. Detailed experimental settings and results are provided in Appendix G.2.

MethodParameters (M) ↓Mean SPD angular error (°) ↓Median (°) ↓
Spectrum-Aware16.3013.5742.704
Ours0.6782.9971.524
MethodMean white-point angular error (°) ↓Median (°) ↓
FC4 (extended)1.2600.871
ConvMean (extended)1.8981.474
Ours1.1470.670

Our method achieves lower SPD errors while reducing the parameter count by approximately 95.8% relative to Spectrum-Aware. It also achieves lower white-point errors than both extended baselines under the same MSI+RAW input setting, supporting the effectiveness of our Stage 2 design on real data.

Stage 3: Complete color restoration and remaining errors of per-scene matrices

We trained the residual refinement module on real data, as detailed in Appendix G.3. We also include per-scene chart-fitted matrices as a reference. Test results:

MethodMean ΔE00\Delta E_{00} ↓
DNG interpolation5.286
Dynamic physical CCM4.354
Dynamic physical CCM + residual refinement3.989
Per-scene chart-fitted 3×3 matrix3.584

The final result approaches the reference obtained by fitting a matrix directly to each test scene's chart, demonstrating the effectiveness of the complete pipeline on real data.

Further least-squares fitting diagnostics show that errors are not distributed uniformly across colors. Across 155 valid scenes, the mean ΔE00\Delta E_{00} of the darkest patch is 8.246, substantially higher than the mean of 3.663 over all patches. When this dark patch is excluded from matrix fitting, its prediction error remains 8.254. These results show that per-scene 3×3 matrices struggle to accurately restore dark colors, making dark patches a prominent source of error in the current real-data evaluation.

In Appendix H, we provide sensitivity experiments on MSI measurement noise, black-level offsets, spectral calibration errors, and MSI-RAW spatial misalignment, showing their effects on final color restoration. Such errors in real acquisition and calibration can likewise degrade performance and make color restoration more difficult.

Although our method still exhibits some performance degradation on real data, the experiments above show that physical color correction and residual refinement remain effective in improving color restoration. Compared with MSCC, which is trained using dense XYZ ground truth, our method can train the residual module using sparse color supervision from a chart, providing a practical route to training on real data without dense XYZ annotations.

AuthorsOpenReview ↗

Response to Reviewer fAst (Part 3/3): Same-Input Comparisons and Stage-Wise Contributions

W1 & W4 & Q1: Distinguishing the benefit of multispectral input from the contribution of the three-stage design

We agree that baselines using the same inputs and controlled ablations are needed to distinguish the benefit of multispectral measurements from that of the proposed design. We address this concern from the following three perspectives.

(1) Cross-camera color restoration with multispectral information

Existing methods, including MSCC, have already demonstrated the value of multispectral images. Our work further addresses how to maintain color-restoration accuracy when the RGB sensor response changes. We explicitly recover the target camera's CSS, separate illumination prediction from sensor responses, and use these quantities to construct the color transform. This allows a network trained on source cameras to transfer to an unseen target camera after one-time CSS calibration, without retraining its network weights.

In Table 2, MSCC is trained specifically for the target camera and achieves a lower mean color error than our method (2.31 versus 2.42). To further evaluate cross-camera performance, we additionally apply the models to unseen cameras with their network weights fixed. Results for three target cameras are shown below.

CameraOurs: mean ΔE00\Delta E_{00} ↓MSCC cmKAN-light: mean ΔE00\Delta E_{00} ↓MSCC LPIENet: mean ΔE00\Delta E_{00} ↓
Sony ILCE-64002.1233.0363.059
Panasonic DC-GX92.3292.7752.635
Hasselblad L2D-20c2.3972.7252.815
Mean2.2832.8452.837

Our method achieves accuracy comparable to camera-specific MSCC models without retraining on the target camera. When these models are also transferred directly to unseen cameras, our method achieves substantially lower color errors on all three cameras above. These results demonstrate that our framework supports cross-camera transfer while retaining competitive color-restoration accuracy.

(2) Additional simple baselines with the same MSI+RGB input

We extend FC4 and ConvMean from their original inputs to jointly accept a 16-band MSI and a three-channel linear RAW image. We area-downsample RAW to the MSI resolution, concatenate the two inputs along the channel dimension, and adjust the input convolution accordingly, leaving the remaining architecture unchanged. Both baselines are trained specifically on Sony ILCE-7RM2 data.

MethodInputTraining settingMean white-point angular error (°) ↓Mean ΔE00\Delta E_{00} ↓
FC4RGBTarget-camera training1.4633.69
FC4 (extended)MSI+RGBTarget-camera training1.6574.08
ConvMeanRGBTarget-camera training1.6634.01
ConvMean (extended)MSI+RGBTarget-camera training1.4693.82
Ours (full framework)MSI+RGBSource-camera training1.4612.42

Even with the same MSI+RGB input and target-camera training, these extended baselines exhibit a substantial gap in final color-restoration accuracy: their mean ΔE00\Delta E_{00} values are 4.08 and 3.82, compared with 2.42 for our full framework. This indicates that simply adding multispectral input is insufficient to achieve the color-restoration accuracy of our framework; explicitly modeling sensor responses and illumination is also important.

(3) Controlled ablations of the individual stages

  • CSS recovery: In Table 3(a), compared with the ColorMatrix-only initialization, response-domain refinement with spectral smoothness reduces the CSS angular error from 6.94° to 4.34° and the imaging angular error from 2.22° to 0.157°. This demonstrates more accurate modeling of the target sensor's image-formation behavior.

  • CSS-guided illumination estimation: Table 3(c) fixes the backbone and MSI+RAW inputs. Replacing direct white-point regression with CSS projection reduces the white-point angular error from 2.23° to 1.35° and the SPD angular error from 5.52° to 4.71°. Our projection helps the network disentangle scene illumination from surface reflectance without requiring it to directly learn a camera-specific RGB white-point mapping. This design improves both SPD estimation and white balance.

  • Physical color correction: Table 3(d) fixes the upstream predictions. Replacing DNG interpolation with the physical CCM alone reduces mean ΔE00\Delta E_{00} from 3.86 to 2.59. Adding residual refinement trained with error injection further reduces the error to 2.38. This directly demonstrates the contributions of the physical color transform and its residual refinement.

We are grateful for the opportunity to discuss these aspects of our work in greater depth. We hope the explanations and evidence presented above help clarify the framework’s methodological contributions, robustness, and practical limitations. We would be happy to elaborate on any points that remain unclear.

Reviewer_5SvqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a multispectral-image-guided framework for cross-camera color restoration. It estimates the target camera's spectral sensitivity, predicts scene illumination, and constructs a scene-adaptive color correction matrix (CCM), followed by learned residual refinement. The goal is to transfer the model to unseen cameras without camera-specific retraining. Experiments on 14 simulated target cameras show encouraging results, while additional experiments on real Samsung smartphone data evaluate parts of the proposed pipeline.

Critical strengths
  1. Meaningful technical contributions. The integration of CSS-aware spectral projection and illuminant-dependent dynamic CCM provides a physically interpretable solution to cross-camera color restoration.
  2. Extensive quantitative results. The experiments across 14 target camera models and the ablation studies demonstrate the effectiveness of the proposed framework, particularly the physics-based dynamic CCM and residual refinement. The proposed dynamic CCM reduces the average ΔE00 from 3.86 to 2.59, further improving to 2.38 with error-aware residual correction.
Critical weaknesses
  1. Methodological novelty remains somewhat limited. Spectral sensitivity estimation, physics-based CCM construction, and residual correction are established techniques. The paper would benefit from a clearer explanation of what is fundamentally new compared with MSCC and related methods.
  2. Real-world cross-camera generalization is insufficiently demonstrated. The main 14-camera benchmark is synthetic. Appendix G evaluates real data from one smartphone, but does not validate the complete pipeline, including learned residual refinement, across multiple real cameras.
  3. Practical requirements deserve more attention. The approach relies on multispectral measurements, camera calibration information, and paired MSI–RAW samples. The real-data experiments also reveal discrepancies between measured responses and the assumed imaging model. Their effects on calibration cost, spatial alignment, and deployment across different cameras are not sufficiently investigated.
  4. The comparison with camera-specific methods needs further clarification. In Table 2, MSCC achieves a slightly lower mean ΔE00 (2.31 versus 2.42). Although MSCC benefits from target-camera-specific training, a clearer comparison of calibration effort, adaptation cost, and reconstruction accuracy would help establish the practical advantage of the proposed method.
Questions for authors
  1. What is the main methodological contribution beyond combining existing spectral sensitivity estimation and physics-based color correction techniques? How does it differ from MSCC?
  2. Can the authors demonstrate the complete pipeline on multiple real cameras, including the learned residual refinement?
  3. How sensitive is the method to errors in multispectral measurements, spectral calibration, and MSI–RAW spatial registration?
  4. What calibration resources are required for each new camera, and how do these requirements compare with camera-specific learning approaches?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5Svq (Part 1/3): Sensitivity Analyses

Dear Reviewer 5Svq,

We sincerely thank you for your time and constructive feedback. We appreciate your recognition of our physically interpretable design and our comprehensive quantitative evaluation.

We have uploaded a revised manuscript. Major additions and revisions are highlighted in blue, including the updated and expanded real-world experiments in Appendix G and the new sensitivity analyses of calibration errors, measurement errors, and MSI–RAW spatial misalignment in Appendix H. Below, we respond to each of your concerns.

Our response is organized into three comments: Part 1 addresses W3/Q3; Part 2 addresses W1/Q1; and Part 3 addresses W2/Q2 and W4/Q4.

W3 & Q3: Sensitivity to multispectral measurement, spectral calibration, and spatial registration errors

We have added systematic sensitivity experiments to evaluate how these errors affect individual stages and final color restoration. Detailed experimental settings and result tables are provided in Appendix H.

(1) Multispectral measurement errors

Under mild random noise, the full-pipeline mean ΔE00\Delta E_{00} increases only from 2.376 to 2.392; under stronger random noise, it reaches 2.663. For residual black-level correction errors, small biases of 1-4 equivalent DN yield errors of 2.39-2.41. As the residual increases to 8, 10, 12, and 14 DN, the corresponding errors are 2.525, 2.611, 2.707, and 2.816. These results indicate that our method remains stable under small measurement errors and degrades gradually as the perturbation increases, while retaining good color-restoration accuracy.

(2) Spectral calibration errors

We separately evaluate MSI spectral-response calibration errors and target RGB-camera CSS calibration errors.

For MSI spectral-response calibration errors, the method maintains good color-restoration accuracy when the response curves are shifted by +1 or +3 nm, with mean ΔE00\Delta E_{00} values of 2.416 and 2.765. At a larger shift of +5 nm, performance degrades to 3.356, but remains better than the DNG interpolation baseline of 3.86. For per-band gain calibration errors, maximum relative gain deviations of 0.1%, 0.3%, and 0.5% yield mean ΔE00\Delta E_{00} values of 2.392, 2.476, and 2.623, respectively. These values remain close to the unperturbed baseline of 2.376, indicating good tolerance to gain calibration errors over the tested range.

For RGB-camera CSS calibration errors, increasing the mean CSS spectral angular error from 4.339° to 10° along the tested perturbation direction raises the final color error only from 2.376 to 2.625, demonstrating that downstream color restoration tolerates the tested CSS calibration errors well. Residual refinement further improves upon the physical CCM at every error level.

(3) MSI-RAW spatial misalignment

We separately evaluate geometric variations during calibration and inference. During calibration, as the typical input geometric displacement increases from 0 to 41.34 pixels, the final mean ΔE00\Delta E_{00} remains nearly unchanged at 2.3758-2.3772 after applying the existing registration and spatial-averaging procedure. During inference, increasing the typical input displacement from 22.53 pixels in the original setting to 40.64 pixels raises the mean ΔE00\Delta E_{00} only from 2.3764 to 2.4491, an increase of approximately 3.06%. These results demonstrate stable color-restoration performance across the evaluated range of spatial misalignment.

AuthorsOpenReview ↗

Response to Reviewer 5Svq (Part 2/3): Methodological Contributions

W1 & Q1: Methodological contributions and distinction from prior work

Our core contribution is to enable multispectral-based color restoration to generalize to unseen cameras. Existing RGB-based cross-camera methods primarily focus on white balance, while MSCC relies on camera-specific training for color restoration. We recover the target camera's spectral sensitivity (CSS) through response-domain optimization and explicitly model sensor differences, enabling cross-camera color restoration without retraining the network. Below, we explain the three key methodological contributions that support this capability.

(1) Response-domain constraints for CSS recovery

CSS recovery methods based on DNG matrices or spectral priors, such as Solomatov & Akkaynak (2023), do not sufficiently constrain the target camera's scene responses. Consequently, even recovered curves with low spectral errors may not accurately predict imaging responses. We introduce paired MSI-RAW observations to provide direct response-domain constraints for CSS recovery. Additional experiments under the same evaluation setting show that Solomatov's method achieves CSS spectral angular and imaging angular errors of 8.72° and 5.670°, whereas our method achieves 4.34° and 0.157°. Our method not only reduces spectral curve errors but, more importantly, substantially reduces imaging response errors, providing a more accurate sensor representation for downstream white balance and color correction.

(2) Extending cross-camera adaptation from white balance to color correction

Cross-camera white-balance methods such as C5 and CCMNet still rely on networks to account for sensor differences when estimating white points and optimize only white balance. We explicitly compute the camera-space white point through the recovered target CSS, reducing the network's reliance on camera-specific white-point mappings. We also predict the illuminant spectral power distribution (SPD), providing the spectral information needed for subsequent color correction and extending cross-camera adaptation from white balance to complete color restoration. Experiments show that our method achieves a lower mean white-point angular error. Meanwhile, the complete pipeline achieves a mean ΔE00\Delta E_{00} of 2.38, substantially outperforming the best result of 3.70 among the compared cross-camera methods.

(3) Cross-camera color correction without retraining or dense XYZ supervision

MSCC learns a supervised RGB-MSI→XYZ mapping with two practical limitations. First, it relies on camera-specific training, making direct transfer to unseen cameras challenging. Second, training requires dense XYZ ground truth, which is difficult to acquire for real scenes and prevents direct training on real datasets. Our framework addresses these two limitations as follows.

Cross-camera transfer without network retraining. We use the recovered target CSS and predicted illuminant SPD to construct a per-image CCM via virtual-chart calibration, explicitly incorporating sensor and illumination differences into the color transform. Adapting to an unseen camera therefore requires only one-time CSS calibration, without network retraining. As shown in the table below, MSCC exhibits degraded performance when transferred to unseen cameras, whereas our method maintains good color-restoration accuracy without network retraining.

CameraOurs: ΔE00\Delta E_{00} ↓MSCC cmKAN-light: ΔE00\Delta E_{00} ↓MSCC LPIENet: ΔE00\Delta E_{00} ↓
Sony ILCE-64002.1233.0363.059
Panasonic DC-GX92.3292.7752.635
Hasselblad L2D-20c2.3972.7252.815
Mean2.2832.8452.837

Training on real data with sparse XYZ supervision. Our physical CCM requires no real RAW-XYZ training pairs; only residual refinement needs sparse XYZ supervision. In Appendix G.3, we demonstrate this on the real Beyond-RGB dataset using supervision from 24 ColorChecker patches per training sample. On 157 test scenes, residual refinement reduces the mean ΔE00\Delta E_{00} from 4.354 with the physical CCM alone to 3.9893.989. For further details, please refer to W2 & Q2.

CSS estimation, physics-based CCM construction, and residual correction are established techniques. Our contribution is to propose a physics-based three-stage framework with specific improvements at each stage to address the challenges of cross-camera color restoration. The residual module is trained to handle upstream estimation errors, reducing error propagation through the pipeline. Meanwhile, our physical decomposition reduces reliance on dense XYZ ground truth and enables color-correction training on real data with sparse supervision. This design supports both transfer without target-camera network retraining and training on real datasets that lack dense color ground truth.

AuthorsOpenReview ↗

Response to Reviewer 5Svq (Part 3/3): Real-Data Validation and Adaptation Cost

W2 & Q2: Real-data validation and residual refinement

We have added experiments on training and evaluating the residual refinement module on real data in Appendix G.3. Below, we explain the limitations of available real datasets and present the real-data evaluation of the complete pipeline.

(1) Limitations of available real datasets

Real-world cross-camera validation requires paired MSI-RAW images, measured illuminant spectra, and color references. Currently, Beyond-RGB is the only publicly available real dataset we can access that meets these requirements. However, it contains only two smartphone cameras, making it difficult to support cross-camera training and independent evaluation with sufficient camera diversity.

We therefore evaluated our framework on the single-camera subset (Appendix G), and the experimental results support the effectiveness of our method under real imaging conditions. In future work, we will expand the coverage of real cameras and evaluate the generalization of the complete pipeline across multiple real cameras.

(2) Training residual refinement with sparse XYZ supervision

Beyond-RGB does not provide dense per-pixel XYZ ground truth. We therefore train the residual module using sparse supervision from 24 patches in paired ColorChecker images, as detailed in Appendix G.3.

Under the same test setting, DNG interpolation, the physical CCM, and the physical CCM with residual refinement achieve mean ΔE00\Delta E_{00} values of 5.286, 4.354, and 3.989, respectively. As a reference, per-scene matrices fitted directly to each test scene's chart achieve an error of 3.584. Our complete pipeline approaches this reference without fitting a matrix to each test scene's chart, demonstrating that residual refinement trained with sparse real-chart supervision effectively improves color restoration.

This also distinguishes our method from MSCC: MSCC requires dense XYZ ground truth to train its supervised mapping, whereas our physical CCM requires no real RAW-XYZ training pairs, and our residual module can be trained on real data using sparse chart supervision. Our design thus provides a practical route to training on real datasets without dense XYZ annotations.

W4 & Q4: Calibration resources, adaptation cost, and comparison with MSCC

With a fixed MSI module whose spectral response is known, adapting our method to a new RGB camera requires two DNG ColorMatrices and 60 MSI-RAW pairs. The ColorMatrices can be obtained from existing Adobe camera profiles. After calibration, the network weights remain fixed, with no retraining or fine-tuning for the new camera.

The comparison with MSCC is summarized below.

Comparison dimensionMSCCOur method
Resources for a new cameraRGB-MSI training data for the target camera with paired XYZ labelsTwo DNG ColorMatrices and paired MSI-RAW observations
Data per new cameraApproximately 75,000 synthetic training samples with dense XYZ labels60 calibration pairs by default; 20 pairs achieve accuracy close to the default configuration
Network adaptation costA separate network is trained for each RGB sensorNetwork weights remain fixed after one-time CSS calibration
Mean ΔE00\Delta E_{00} ↓2.312.42
Median ΔE00\Delta E_{00} ↓1.931.90
Trimean ΔE00\Delta E_{00} ↓2.022.02
Best-quartile ΔE00\Delta E_{00} ↓1.050.84

Table 7 shows that using 20 calibration pairs achieves a mean ΔE00\Delta E_{00} of 2.385, close to 2.376 with the default 60 pairs. This indicates that, under this experimental setting, fewer calibration observations can retain accuracy close to the default configuration. Moreover, these calibration pairs can be captured from everyday scenes, making the required calibration data easy to collect.

To further evaluate cross-camera performance, we additionally apply the models to unseen cameras with their network weights fixed. Results for three target cameras are shown below.

CameraOurs: mean ΔE00\Delta E_{00} ↓MSCC cmKAN-light: mean ΔE00\Delta E_{00} ↓MSCC LPIENet: mean ΔE00\Delta E_{00} ↓
Sony ILCE-64002.1233.0363.059
Panasonic DC-GX92.3292.7752.635
Hasselblad L2D-20c2.3972.7252.815
Mean2.2832.8452.837

These results demonstrate that our framework supports cross-camera transfer while retaining competitive color-restoration accuracy. The advantage of our method is that it adapts to a new camera using a small set of paired observations and achieves overall accuracy comparable to camera-specific models without retraining the network on the target camera.

Thank you again for your thoughtful review and valuable suggestions. We hope our responses clarify the contributions and practical scope of our framework. We welcome any further questions or discussion.

AuthorsOpenReview ↗

Follow-up on W2/Q2: Additional Real-World Cross-Camera Evaluation

Dear Reviewer 5Svq,

Thank you for your careful review and constructive feedback. To further address your questions in W2/Q2 regarding complete-pipeline validation on real data and cross-camera generalization, we conducted an additional experiment to provide further evidence for your assessment.

The Samsung single-camera experiments included in our earlier response demonstrated the feasibility of the complete method, including residual refinement, on real data. Specifically, the residual refinement module trained with sparse chart-based XYZ supervision reduced the mean ΔE00\Delta E_{00} from 4.354 to 3.989. This provides a practical path to training our framework on real-world datasets without dense XYZ ground truth. Building on these results, we further investigated cross-camera applicability on real data.

The Beyond-RGB dataset contains limited RGB-camera diversity, whereas learning to generalize across different camera responses with the RAW branch, AWB head, and residual refinement module requires more diverse multi-camera training data. We therefore performed an initial evaluation using a simplified variant: we removed the RAW branch, AWB head, and residual refinement module, retaining only the MSI branch and SPD prediction head, followed by color restoration using calibrated camera CSS.

After training on real MSI observations, we kept the network weights fixed and combined the illumination prediction for each scene with the separately calibrated CSS of Samsung and OPPO. Under matched evaluation conditions, the mean white-point angular errors were 1.341° and 1.649°, and the mean ΔE00\Delta E_{00} values were 4.476 and 4.626, respectively. For reference, the previously reported Samsung single-camera experiment achieved a mean white-point angular error of 1.147° and a mean ΔE00\Delta E_{00} of 4.354 for physical color restoration. The simplified model achieved performance close to these previously reported single-camera results. This experiment provides preliminary evidence of cross-camera generalization for a subset of our framework’s components on real-world data. In future work, we plan to expand the camera coverage of real-world dataset and use more diverse multi-camera training data to further evaluate the generalization of the complete method across real devices.

As the discussion deadline approaches, we would appreciate any further feedback you may have on these additional results and our earlier responses. If any concerns remain or further clarification would be helpful, we would be happy to address them. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.