- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This paper proposes to solve the color restoration task for new cameras, including both white balancing and color correction. To avoid training separate model for each camera with a specific sensor response, they use a fixed 16-band multispectral camera as a common "spectral bridge" between different cameras, since the spectral radiance and reflectance are censor independent.
- The proposed method has three stages:
- Camera spectral sensitivity recovery. Starting from the cemera's two DNG ColorMatrics from the meta data, they estimate the CSS using paired MSI-RAW images using ridge regression. Linear regression here is to avoid the overfitting. The ground-truth CSS of the camera is unnecessary but the CSS of the MSI is needed.
- Estimate the illumination together with the white balance. A shallow network takes in both MSI and RAW images predict the illumination and white balancing curve.
- A scene-specific color correction matrix, They create a virtual color chart using the estimated illumination and CSS from the first two stages.
Critical strengths
- What I like the most is they ideally combine the physics and learning. Instead of learning RAW+MSI to XYZ end-to-end, they explicitly decompose the problem into CSS, illumination, white balancing and CCM estimation, which makes the pipeline more interpretable.
- Ablation study in Table 3(c) shows that the CSS projection is convincing. Other ablation studies also makes the contribution of each part clear.
- The proposed method work well on unseen cameras compared with other methods, as shown in Table 1.
- Experiments in Table 12 show the importance of the quality of SPD estimation, which is a good analysis.
Critical weaknesses
- What I confused is the paper mentioned that the propose method works on new cameras. But given a new camera, the stage-1 still need paired MSI-RAW captures for CSS optimization on 60 pairs. This is not pure zero-shot transferring. This makes it hard to deploy. Besides, calibration between MSI and RGB cameras is also non-trivial.
- Synthetic training, which may not work well on real data.
- Table 2 shows the performance is better than RGB-Only methods. This is reasonable, but it is only comparable with other RGB+MSI methods.
- Performance on the real data in Table 14 still shows a noticeable domain gap.
Questions for authors
- Why do you think the MSI and RAW together are enough to recover the illumination? It is still highly ill-posed.
- How much target-camera data is really necessary in practice?
- How sensitive is the method to MSI-RAW misalignment?
- In practice, it is impossible to get the perfect MSI CSS, how sensitive is the method to MSI CSS error.
5 comments
Response to Reviewer JrAp (Part 2/3): Real-Data Validation and the Domain Gap
W2 & W4: Real-data validation and the synthetic-to-real domain gap
Thank you for highlighting the issues of real-data validation and the domain gap. Due to differences between synthetic image formation and real observations, as well as the use of different MS cameras in our synthetic setup and the real dataset, models trained on synthetic data are difficult to transfer directly to real datasets. Meanwhile, Beyond-RGB contains only two smartphone cameras, making it difficult to support cross-camera training and independent evaluation with sufficient camera diversity. We therefore train and evaluate the complete pipeline on the Samsung single-camera subset to assess the effectiveness of our framework under real imaging conditions. Detailed experiments are provided in Appendix G.
(1) Mismatch between real observations and the ideal imaging model
Real MSI observations exhibit a noticeable mismatch with the ideal linear imaging model used for synthetic training. We synthesize channel responses using the authors' measured chart reflectances, measured illuminant spectra, and published multispectral response curves, and fit one global exposure scalar per scene. On the test scenes, the mean scene-wise relative Frobenius residual is 22.5%, and the mean 16-channel response angular error is 4.99°. After excluding saturated observations, the relative residual remains 15.4%. Introducing a shared empirical power-response model fitted only on the training set further reduces the residual to 11.42%, indicating a systematic intensity-dependent mismatch.
To address this mismatch, we adopt a CSS calibration variant based on scene-shared illuminant estimation. Without using the measured illuminant spectra of the calibration scenes, the mean imaging angular error decreases from 1.596° at initialization to 0.967°, approaching the reference result of 0.915° obtained using ideal radiance inputs. This demonstrates that the calibration variant alleviates the effect of the mismatch in measured observations.
(2) Illuminant estimation on real data
We evaluate Stage 2 on the Samsung subset of Beyond-RGB, comparing SPD prediction with Spectrum-Aware and white-point estimation with MSI+RAW-extended FC4 and ConvMean under the same data splits. Detailed experimental settings and results are provided in Appendix G.2.
| Method | Parameters (M) ↓ | Mean SPD angular error (°) ↓ | Median (°) ↓ |
|---|---|---|---|
| Spectrum-Aware | 16.301 | 3.574 | 2.704 |
| Ours | 0.678 | 2.997 | 1.524 |
| Method | Mean white-point angular error (°) ↓ | Median (°) ↓ |
|---|---|---|
| FC4 (extended) | 1.260 | 0.871 |
| ConvMean (extended) | 1.898 | 1.474 |
| Ours | 1.147 | 0.670 |
Our method achieves lower mean and median SPD errors while reducing the parameter count by approximately 95.8% relative to Spectrum-Aware. It also achieves lower white-point errors than both extended baselines under the same MSI+RAW input setting, supporting the effectiveness of our Stage 2 design on real data.
(3) Complete color restoration and residual refinement on real data
We have additionally trained the residual refinement module on real data, as detailed in Appendix G.3. Since Beyond-RGB does not provide dense per-pixel XYZ ground truth, we train the residual module using sparse color supervision from 24 ColorChecker patches in the training images. We also include per-scene chart-fitted matrices as a reference. The results are as follows:
| Method | Mean ↓ |
|---|---|
| DNG interpolation | 5.286 |
| Dynamic physical CCM | 4.354 |
| Dynamic physical CCM + residual refinement | 3.989 |
| Per-scene chart-fitted 3×3 matrix | 3.584 |
Residual refinement further reduces the color error of the physical CCM from 4.354 to 3.989. To understand the remaining errors on real data, we also directly fit an independent 3×3 matrix using the 24 chart patches in each test scene and evaluate it on those same patches. Even in this setting, the mean remains 3.584, indicating that errors on real data also involve the limited expressiveness of linear color mappings, as well as acquisition, chart-patch extraction, and reference-color consistency. Our complete pipeline approaches this per-scene fitting reference, demonstrating the effectiveness of our model on the real dataset.
Although our method still exhibits some performance degradation on real data, the experiments above show that physical color correction and residual refinement remain effective in improving color restoration. Compared with MSCC, which is trained using dense XYZ ground truth, our method can train the residual module using sparse color supervision from a chart, providing a practical route to training on real data without dense XYZ annotations.
Response to Reviewer JrAp (Part 3/3): Same-Input Comparisons and Ill-Posed Illuminant Recovery
W3: Comparison with methods using the same RGB+MSI input
We agree that comparison with methods using the same RGB+MSI input is important for evaluating our framework. Table 2 already includes two MSCC models that use RGB+MSI input. We further address this concern through cross-camera comparisons and additional simple baselines using the same inputs.
(1) Cross-camera color restoration with multispectral information
In Table 2, MSCC is trained specifically for the target camera and achieves a lower mean color error than our method (2.31 versus 2.42). Our method achieves comparable accuracy after one-time CSS calibration of the target camera. To further evaluate cross-camera performance, we additionally apply the models to unseen cameras with their network weights fixed. Results for three target cameras are shown below.
| Camera | Ours: mean ↓ | MSCC cmKAN-light: mean ↓ | MSCC LPIENet: mean ↓ |
|---|---|---|---|
| Sony ILCE-6400 | 2.123 | 3.036 | 3.059 |
| Panasonic DC-GX9 | 2.329 | 2.775 | 2.635 |
| Hasselblad L2D-20c | 2.397 | 2.725 | 2.815 |
| Mean | 2.283 | 2.845 | 2.837 |
Our method achieves accuracy comparable to camera-specific MSCC models without retraining on the target camera. When these models are also transferred directly to unseen cameras, our method achieves substantially lower color errors on all three cameras above. These results demonstrate that our framework supports cross-camera transfer while retaining competitive color-restoration accuracy.
(2) Additional simple baselines with the same MSI+RGB input
We extend FC4 and ConvMean from their original inputs to jointly accept a 16-band MSI and a three-channel linear RAW image. We area-downsample RAW to the MSI resolution, concatenate the two inputs along the channel dimension, and adjust the input convolution accordingly, leaving the remaining architecture unchanged. Both baselines directly predict the camera-space white point, followed by conventional DNG interpolation for color correction.
Both baselines are trained specifically on Sony ILCE-7RM2 data and evaluated on the same 770 test images used in Table 2.
| Method | Input | Training setting | Mean white-point angular error (°) ↓ | Mean ↓ |
|---|---|---|---|---|
| FC4 | RGB | Target-camera training | 1.463 | 3.69 |
| FC4 (extended) | MSI+RGB | Target-camera training | 1.657 | 4.08 |
| ConvMean | RGB | Target-camera training | 1.663 | 4.01 |
| ConvMean (extended) | MSI+RGB | Target-camera training | 1.469 | 3.82 |
| Ours (full framework) | MSI+RGB | Source-camera training | 1.461 | 2.42 |
Even with the same MSI+RGB input and target-camera training, these extended baselines exhibit a substantial gap in final color-restoration accuracy: their mean values are 4.08 and 3.82, compared with 2.42 for our full framework. This indicates that the improvement in color-restoration accuracy comes not only from the introduction of MSI information but also from the design of our three-stage framework.
Q1: Ill-posedness of illuminant spectrum estimation
Thank you for pointing out the ill-posedness of illuminant recovery. We agree that, even with both MSI and RAW observations, unknown surface reflectance and illumination remain coupled, and these observations alone cannot guarantee unique recovery of the illuminant spectrum. Our method estimates illumination under the constraints of multispectral observations and learned priors, rather than assuming that MSI and RAW make the problem fully well-posed.
Specifically, estimating a high-dimensional illuminant spectrum from three-channel RAW alone is constrained by very limited observations. Introducing a 16-band MSI provides additional spectral constraints, thereby alleviating the ill-posedness of illuminant estimation; the network further uses priors learned during training to estimate the illuminant spectrum. Experimentally, Table 3(b) shows that the spectral-preserving backbone using only MSI achieves a mean SPD angular error of 5.08°, which decreases to 4.91° with the addition of the RAW branch, indicating that the two observations provide complementary information. The full model achieves a mean SPD angular error of 4.71° on unseen target cameras. These results show that, although illuminant recovery remains inherently ambiguous, multispectral observations combined with learned priors support effective illuminant spectrum estimation.
Thank you again for your careful assessment and valuable suggestions. We hope our responses and the added experiments address your concerns. We welcome any further questions or discussion.
Response to Reviewer JrAp (Part 1/3): Calibration Requirements and Error Sensitivity
Dear Reviewer JrAp,
We sincerely thank you for your thoughtful review and constructive feedback. We appreciate your recognition of the integration of physics and learning in our framework. Your comments highlight the need for clearer calibration requirements, stronger experimental validation, and further analysis of illuminant estimation and robustness.
We have uploaded a revised manuscript. Major additions and revisions are highlighted in blue, including the updated and expanded real-world experiments in Appendix G and the new sensitivity analyses of calibration errors, measurement errors, and MSI–RAW spatial misalignment in Appendix H. We address your concerns individually below.
W1 & Q2 & Q3: Target-camera calibration requirements and sensitivity to MSI-RAW misalignment
Thank you for highlighting the calibration requirements for adapting to a new camera. We agree that our method requires one-time CSS calibration for a new camera and therefore does not constitute entirely calibration-free, zero-shot transfer. The cross-camera generalization we emphasize is that a model can be transferred to an unseen camera after one-time calibration using a small set of paired MSI-RAW observations without tuning the model.
To further evaluate the amount of calibration data required, we compare different numbers of MSI-RAW calibration pairs in Appendix C, Table 7. With all network weights and the test set fixed, using only 20 pairs achieves a mean white-point angular error of 1.362° ± 0.005° and a mean ΔE00 of 2.385 ± 0.008, close to 1.349° and 2.376 with the default 60 pairs. These results show that, under the evaluated setting, 20 calibration pairs are sufficient to achieve accuracy comparable to the default configuration. Moreover, these calibration pairs can be captured from everyday scenes, making the required calibration data easy to collect.
Regarding your concern about calibration between MSI and RGB cameras, we conduct additional spatial-misalignment sensitivity experiments, with detailed settings and results provided in Appendix H.4. During calibration, as the typical input displacement before registration increases from 0 to 41.34 pixels, the final mean ΔE00 remains within 2.3758-2.3772 after applying the existing registration and spatial-averaging procedure. During inference, increasing the typical input displacement from 22.53 to 40.64 pixels raises the mean ΔE00 only from 2.3764 to 2.4491, an increase of approximately 3.06%. These results demonstrate that our framework is robust to spatial misalignment between MSI and RGB within the evaluated range.
Q4: Sensitivity to MSI spectral-response calibration errors
Thank you for highlighting the inevitable errors in practical MSI spectral-response calibration. We have added sensitivity experiments to separately evaluate wavelength shifts and relative per-band gain errors. We perturb only the MSI response curves used in Stage 1 and re-estimate the target RGB-camera CSS, while keeping all network weights fixed. Detailed experimental settings and results are provided in Appendix H.1.
For wavelength calibration errors, the method maintains good color-restoration accuracy when the response curves are shifted by +1 or +3 nm, with mean ΔE00 values of 2.416 and 2.765, respectively. At a larger shift of +5 nm, performance degrades to 3.356, but remains better than the DNG interpolation baseline of 3.86.
For per-band gain calibration errors, maximum relative gain deviations of 0.1%, 0.3%, and 0.5% yield mean ΔE00 values of 2.392, 2.476, and 2.623, respectively. These values remain close to the unperturbed baseline of 2.376, indicating good tolerance to gain calibration errors over the tested range. Overall, these results show that our framework maintains good color-restoration performance under the tested MSI spectral-response calibration errors and does not require perfectly accurate MSI spectral responses.