Albedo Drift: What Single-Image Intrinsic Decomposition Evaluation Does Not Measure

Albedo is illumination-invariant by definition, yet the standard evaluation of single-image intrinsic decomposition never tests invariance: WHDR on IIW scores relative reflectance within one image, so a model that bakes illumination into its albedo is never penalized. We name this failure albedo drift and introduce a protocol that measures it directly: the ratio of cross-illumination albedo variation to input variation on scenes photographed under many lighting conditions. Across three datasets and six methods we find that (i) a state-of-the-art model absorbs half of the illumination change into its albedo even within its training domain (drift ratio 0.55), and remains at 0.50 on photometrically diffuse pixels, ruling out non-Lambertian ambiguity as the cause; (ii) outside the training domain, decomposition amplifies illumination variation (drift ratio 1.5, worse than returning the input); (iii) under simulated illuminant color changes, learned models amplify chromatic variation up to $10\times$; and (iv) what determines drift is not the architecture family but the match between training and test distributions — a domain-matched diffusion model is as invariant as regression models, while a mismatched one is $6\times$ worse. Invariance alone is trivial (a constant albedo is perfectly invariant), so we evaluate it jointly with fidelity and observe that no existing method occupies the region that is both faithful and invariant. This corner is reachable: a 1M-parameter consistency adapter trained on held-out scenes halves drift ($-49\%$, sign test $p<10^{-9}$) while slightly improving WHDR, generalizes across capture domains, and cannot be replicated by smoothing. Albedo drift is not a fundamental trade-off; it is an axis the field has never optimized because it has never been measured.