- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper propose a new metric and benchmark: albedo drift, which measures the variation of predicted albedo of intrinsic image decomposition methods under varying illuminations. The paper found modern intrinsic image decomposition is often not as robust as even classical Retinex for albedo drift. The author propose a trained U-Net base adapter refinement on albedo prediction, which is trained on multi-illumination data, to improve albedo drift. The authors found albedo drift is improved at not cost to fidelity as defined by whdr.
Critical strengths
there can be interesting points if true (although I believe the current evidence presented in the paper is too weak). For example, albedo drift is significantly worse for out of domain examples.
Critical weaknesses
-
I have a philosophical concern regarding the paper: like the author said, there has been plenty of work which study albedo invariance within an image. The author propose another invariance of albedo which is measured across illumination. I am worried the community as a whole can keep proposing many more invariants, but nothing really specifies what correct albedo should be exactly. It is also not clear if albedo drift an extraordinary important invariance in downstream tasks. While illustrative of the albedo drift, I find the example where the author relight the albedo recovered from multiple illumination under the same new illumination to be a bit artificial, as it is not clear if this is representative for how IID will be used in downstream tasks.
-
As a benchmark paper, the benchmark is not very exhaustive. recent methods such as "RGB↔X" or "IntrinsicAnything" are not benchmarked. Additional regression baselines such as NIID-net, or "Inverse Rendering for Complex Indoor Scenes" is also not benchmarked. I think it is difficult to draw reliable conclusions for each class of method based 1 or 2 techniques. "Findings" section of the paper. It is also not clear why Marigold-IID fails so badly on albedo drift. providing qualitative results would be really helpful.
-
As a dataset paper, I feel the qualitative results is way too few and readers might have difficulty intuitively understanding how the albedo drift of various different methods manifest on these examples. For the only result of albedo in the main paper, figure 2, a large percentage of the visible pixels are glasses, and it is really not clear what is the correct albedo in those regions. There is also no qualitative results for albedo spectral drift at all.
-
Albedo drift isn't really an entirely new concept. Even back in 2018, "Learning Intrinsic Image Decomposition from Watching the World" proposed to minimize albedo variations across different illuminations. The proposed albedo drift and adapter is very close to "Learning Intrinsic Image Decomposition from Watching the World".
-
Why many fields of table 1 is left empty? e.g. drift (diffuse), drift (spectral), WHDR, relight? Completeness of the results is very important to assess the authors' conclusion of the paper
-
minor. I do not believe eq.4 is gauge transform. please clarify.
Questions for authors
Why is albedo drift worthwhile single out as an invariance for the intrinsic image decomposition problem? How is albedo drift different from constant albedo assumption that is used by technical such as "Learning Intrinsic Image Decomposition from Watching the World" .
4 comments
Response to Reviewer zHrQ (2/2): Marigold failure explained, qualitative results, Table 1 filled, RGB→X and NIID-Net added
Response to Reviewer zHrQ (2/2): the evidence
Why does Marigold-IID "fail so badly"? Not as badly as we reported: the submitted number was a shading measurement. Our Marigold scripts selected the wrong output: list(pipe.target_properties).index("albedo") indexes the keys of the property dictionary and returns the shading target. Every submitted Marigold number was a shading measurement. With the albedo target and matched encoding, Marigold's median drift is 0.441 [0.370, 0.620] against 0.200 for C&A, and its full-IIW WHDR is 0.215 (C&A: 0.217). The two models therefore have nearly the same full-IIW WHDR and a two-fold difference in invariance, which is the corrected form of Finding 4. Sampling noise contributes to Marigold's drift but does not create it: five seeds on four scenes put same-image variation at 0.31 of the cross-illumination variation, and a 10-sample ensemble lowers the median from 0.441 to 0.408 (−7.5%, paired p < 1e-4). Marigold also returns near-black albedo for dark objects and, in one scene, for almost the whole scene under one light (Fig. 4); such outputs are detected by thresholding the prediction's median and counted separately (§3.2).
Qualitative results, and the glass pixels of Fig. 2. Regenerated under the matched encoding: Fig. 2 (drift structure maps on everett_kitchen2, the scene with the largest C&A drift), a new Fig. 4 (six scenes × {input, C&A, Marigold} under the two lights whose normalized inputs differ most; per-scene ratios in Appendix C) and Fig. 5 (relit crops). You are right that much of Fig. 2 is glass and specular surfaces, where the correct albedo is ambiguous; that is why the protocol has the diffuse-only control: Eq. (7) removes saturated pixels and moving highlights, and the ratio restricted to the remaining pixels is 0.206 for C&A (0.200 on all pixels), 0.423 for Marigold (0.441), 0.228 for Kocsis et al. and 0.289 for RGB→X (Table 1). Drift persists after the diffuse-only control; it is not an artifact of the non-Lambertian regions alone. The strong colour casts in the submitted panels were themselves partly an artifact of the mixed encoding. There are no qualitative spectral results in the revision either: the simulated probe is withdrawn as a measurement (§4.1), and the spectral axis is listed as unmeasured (Limitation 2).
Incomplete Table 1. Table 1 is now complete. Kocsis et al. is evaluated on multilum, MIT Intrinsics and BigTime: 0.200 [0.179, 0.210] (C&A 0.200), 0.187, and 0.380 (C&A 0.595); spectral probe 3.67 (ill-conditioned, not interpreted), WHDR 0.297 (300-image subset), diffuse-only 0.228, relighting flicker 0.20 (all with the authors' x^2.2 input linearization, general response item 4). It matches C&A's invariance at a worse WHDR, a second instance of the fidelity–invariance decoupling. The last empty cell, the WHDR of C&A 2023, is filled (0.254 on the 300-image subset).
Additional public methods. We added the RGB→X model of RGB↔X (Zeng et al., 2024), run with the authors' demo settings (prompt "Albedo (diffuse basecolor)", 50 DDIM steps) and with the pipeline's 1/2.2 output gamma inverted so that its albedo enters the metric in linear RGB like the other learned models. Results (matched encoding, medians; multilum, MIT Intrinsics and BigTime): 0.264 [0.224, 0.279] (diffuse-only 0.289), 0.277, 0.674; WHDR 0.257 on the 300-image IIW subset. It sits between Kocsis et al. (0.200) and Marigold (0.441) in drift, and it shows the same off-domain growth (0.264 → 0.674 on BigTime). We also added NIID-Net (Luo et al., 2020), the regression baseline you named, run with the authors' inference script (it consumes the encoded image, runs at a fixed 320×240 and upsamples its linear reflectance to the frame; stated in §4): 0.193 [0.181, 0.206] on multilum (diffuse-only 0.229), 0.148 on MIT Intrinsics, 0.315 on BigTime, WHDR 0.235 on the 300-image subset. It matches C&A's in-domain invariance at a worse WHDR and grows off-domain by half rather than threefold; its low internal resolution accounts for little of this (run at the frame's resolution: 0.208 on multilum, 0.324 on BigTime; Appendix A). Both models are in Tables 1–2, Fig. 1, Findings 3–4 and the appendix mapping. IntrinsicAnything and the inverse-rendering network of Li et al. (2020) remain unevaluated and are named as such in §2 and Limitation (5).
Eq. 4 and the term "gauge". You are right. The scalar ambiguity of I = R·S (and its per-pixel generalization R·f(x) · S/f(x)) is now described as a scale ambiguity; "gauge" is removed. §3.2 states what the per-image median normalization fixes and what it leaves (a global per-channel cast is additionally forgiven by the per-channel variant, reported as a control: C&A 0.205).
If this resolves your concerns, we would be grateful if you would reflect it in your assessment; if something remains, we would like to know what it is before the discussion closes.
I thank the authors for an thorough and honest revision. The authors reported 3 defects of the orignal version (linear/srgb mismatch, marigold measured shading instead of albedo, ill-conditioned simulated psectral probe). The authors therefore updated the results, and added baselines, completed table 1, added second base model all made me more confident in the authors reported numbers. My point 2, 5, 6 are resolved.
My core remaining concern is fact my philosophical concern raised in point 1. The community can keep adding more invariance dimensions, but invariance is merely a property and does not specify what is correct. The results could still be wrong even all invariants are met. The newly reported results furthers my concern. On MIT intrinsics for example, the proposed adapter reduces drift by 20% (0.197->0.157), and lowers WHDR, but the ground-truth albedo error rises by 33% (si-MSE 0.00147 -> 0.0195), and by 111%(0.0217->0.0459).
As the authors optimizing for this invariance, predicted albedo becomes much more distant from groundtruth albedo. The results in fact strongly supports that optimizing for one extra invariance could be counterproductive and new technique that evaluate albedo comprehensively is needed.
Additionally, the author fail to support why such invariance is particularly important in downstream tasks, which should be the core motivation of the paper. No discussions nor experiments or user studies are performed in this regard. The adapter proposed is also not very useful, as it optimizes an invariance that is not necessarily important in downstream use cases, while harming the error when compared to groundtruth.
Therefore, I believe the proposed metric is very limited, and do not believe the paper meets the high bar of ICLR. The paper could even misled the community toward optimizing for wrong objective if published.
Other concerns:
-
I still do not believe the idea proposed in this paper is meaningfully different from Li & Snavely (2018), as both papers use cross illumination variance as objective for training and evaluation.
-
I still think the qualitative results are quite limited for a computational photography paper, where the qualitative results often communications more about the properties of the systems to the reader. fig.2 is still mostly glasses. fig. 4 only has two examples. Most techniques that are presented in table one are missing. Moreover, there is no qualitative results shown for the proposed adapter technique to allow reader understand what the adapter did. There is also no MIT intrinsics qualitative results or IIW qualitative results. Finally adding various quantitative metrics (drift, WHDR, si-MSE), could help reader intuitively understand what each level of quantitative results looks like qualitatively.
-
Given major errors was discovered in the original paper, I am not very confident about the quality of the paper can be completely addressed within one conference review cycle, which is more fast paced rather then comprehensive. There could be still undiscovered major errors in the paper, and I think the paper could benefit from going through another review cycle.
Response to Reviewer zHrQ (round 2)
Thank you for the second reading and for confirming that points 2, 5 and 6 are resolved. New in the PDF uploaded with this comment: a contrast-matched drift ratio (Eq. 6, requested by Reviewer uUXy), MIT ground-truth error for all operating points but one (Table 4), five qualitative figures (Figs. 5–9) and the Li & Snavely distinction (§2).
1. "Invariance is a property, not correctness; optimizing it can be counterproductive." We agree, and the new measurements sharpen your point. Under the contrast-matched form (each frame standardized to zero mean and unit spatial std; identity = 1), the signed change of medians for the C&A adapter is −28.5% under Eq. 5 but −6.9% [−15.6%, +0.4%] under Eq. 6; for the NIID-Net adapter −38.5% vs. −5.9%; for the λ=3 fine-tune −39.9% vs. +11.7% (worse than the base) at 0.54 of the base's spatial contrast. On MIT every one of the 12 refinements of C&A has a higher mean ground-truth error (adapter +32.8%, λ=3 fine-tune +80.0%, NIID-Net adapter +111.3%; your 0.00147 should read 0.0147), and the change in drift ratio and the change in error are rank-correlated at ρ = −0.79 (p = 0.002). At their headline operating points the consistency objectives lower the drift ratio without worsening full-IIW WHDR mainly by flattening the albedo; after contrast matching the adapters show no resolved improvement, the fine-tune is worse, and the albedo is further from the truth.
We have therefore withdrawn "the axis is improvable at no fidelity cost" (Note on this revision, §1, §2, §5, Limitation 4, conclusion, Fig. 1 caption). The adapter and the fine-tune remain as probes showing that Eq. 5 can be moved while WHDR stays put, and how — evidence that WHDR alone is not a sufficient fidelity axis either. Abstract and conclusion now state your recommendation: report Eq. 5 with its contrast-matched form and ground-truth error where available; do not optimize invariance alone.
Where we still differ: that a necessary condition can be gamed when optimized in isolation is not an argument against measuring it. WHDR has the same property — it is blind to a flattening that nearly halves the albedo's contrast (Table 4, Fig. 9) — and our ground-truth check covers 20 lab objects. Two-light photographs give a controlled test of a necessary property under the stated image-formation assumptions (measured drift is an upper bound, §3.4), which the learned models fail at 0.193–0.441 before any optimization (0.391–0.711 contrast-matched).
2. Downstream importance. Not established: no downstream task end-to-end, no user study (Limitation 4). The relighting flicker of §6 is a proxy that inherits the flattening. Applications that reuse albedo across captures (relighting, material extraction from multi-view or time-lapse captures, video editing) are exposed to the change we measure — motivation, not a demonstrated outcome.
3. Li & Snavely (2018). We re-read the paper. Their cross-frame reflectance consistency is a training loss (their §5.2); their evaluation is WHDR on IIW, AP on SAW and MSE/LMSE/DSSIM on MIT (their §6); no cross-illumination consistency of any output is reported. They train one model with the signal; we measure it, under a stated encoding, normalization, identity control and contrast-matched form, on current models none of which was trained with it, and show that within-image scores do not predict it (§2). Our adapter's objective is of their family (§2, §5); their evaluation did not measure cross-illumination instability or the contrast-matched diagnostic.
4. Qualitative results. Appendix D now has Figs. 5–6 (all Table 1 methods and the three refined models, six scenes, two lights, light-b panels annotated with both drift ratios), Fig. 7 (base vs. refined albedo, |A_a − A_b| maps, signed residuals, metrics per row), Fig. 8 (five MIT objects by a fixed rule, ground truth and every method with a MIT value, drift / contrast-matched drift / si-MSE) and Fig. 9 (four IIW images by a fixed rule, every method, per-image WHDR). The flattening is visible, and so is its invisibility to WHDR. Fig. 2 keeps everett_kitchen2, the scene with the largest C&A drift (0.50); its glassware is why the diffuse-only value (C&A 0.206) is reported beside it; the new figures cover five other scenes. The previous Fig. 4 had six scenes but only two models and moved to the appendix.
5. Another review cycle. You and Reviewer uUXy identified the spectral-probe problem; checking it, we found the encoding and target-selection defects, and an interface audit found the Kocsis input transform. Every number comes from one macro pipeline over the result files, with identity controls and recomputation against the archived values (Appendix B), to be released with the code. The contrast-matched analysis found no new pipeline defect; it found a limit of the claim, which the paper now states. These checks do not exclude undiscovered errors; the revision documents the corrections, the checks and the remaining limitations.
Response to Reviewer zHrQ (1/2): the question of the review, Retinex premise, class-level finding withdrawn
Response to Reviewer zHrQ (1/2): the question of the review
Thank you for the detailed review. This comment answers your conceptual question; the next, the evidence.
Why single out this invariance, and what "correct albedo" is. We do not propose a new invariant; we test the one that defines the quantity. Albedo is, by definition, the property of the surface that does not change when the light does; WHDR scores relative reflectance within one photograph under one light and cannot test it. Drift is a necessary condition, not a sufficient one: a correct albedo has zero drift (up to the scale ambiguity of §3.2 and the interreflection floor of Limitation 3), but zero drift does not make an albedo correct, which is why every result is read jointly with fidelity (WHDR, and ground-truth error where it exists) and why a constant output, perfect on drift, is rejected on fidelity. We do not claim to specify the correct albedo. We show that every learned model evaluated keeps a fifth to two fifths of the normalized change (0.193–0.441 on multilum; §3.4 treats this as an upper bound on invariance failure), the best regression model on its own training distribution, and that WHDR cannot see this. Whether that matters downstream is not established here: relighting flicker is a proxy and, as you say, an artificial one; the applications that need consistency across captures (relighting, material extraction from several photographs, editing across frames) are not tested, and Limitation 4 says no downstream task is evaluated end-to-end. The difference from Li & Snavely (2018) is one of role, not of physics: they use the same assumption as a training loss on time-lapse video; we measure how much of it current models violate after training without it, in a normalized ratio comparable across methods (its absolute scale is dataset-dependent, §3.3), read jointly with fidelity. The adapter's objective is in the same family as theirs, and §5 says so; our contribution is the measurement and its consequences, not the loss.
What the measurement adds beyond the definition. (a) Fidelity does not predict invariance: C&A v2 and Marigold-IID have WHDR 0.217 and 0.215 but drift 0.200 and 0.441; Kocsis et al. scores 0.200 at WHDR 0.297 and NIID-Net 0.193 at 0.235, both WHDR values on the 300-image subset (Finding 4, corrected). (b) The off-domain growth you singled out as interesting if true holds with the corrected numbers: C&A 0.200 → 0.595 on BigTime, Kocsis 0.200 → 0.380, RGB→X 0.264 → 0.674, Marigold 0.441 → 0.916, NIID-Net 0.193 → 0.315 (Finding 3). (c) The encoding discipline of §3.4 and its identity-in-linear control apply to any evaluation comparing a model's linear-RGB output with sRGB photographs: a no-op scores 3.86 on multilum under the submitted convention.
Two submitted claims that the correction reverses. Retinex is not "more robust than modern decomposition": under matched encoding it is less invariant than C&A (0.327 vs 0.200) and less faithful (WHDR 0.450 vs 0.217). And the class-level conclusion of the submission (Finding 4, "distribution match, not architecture, governs drift") rested on Marigold's shading value and is withdrawn; the corrected paper states its findings per model and claims only that, among the methods evaluated, no architecture family is invariant and the within-image score does not reveal which model keeps less of the light. We agree that one or two models per class cannot support class-level conclusions; the revision draws none.
What the corrected paper claims. Less than the submitted one: the median directional drift of the five learned pipelines on multilum is 0.193–0.441, the leak is structural and grows off-domain, WHDR does not predict it, and on two base models a consistency objective reduces it without WHDR degradation, while ground-truth albedo error on MIT worsens on both. Every number traces to a result file to be released with the code.
If this resolves your concerns, we would be grateful if you would reflect it in your assessment; if something remains, we would like to know what it is before the discussion closes.