acceptodds
Under review as a conference paper at ICLR 2027

Capture White-Point Sensitivity in Frozen Vision Encoders: A Calibrated Audit on Human Faces

Abstract

Frozen vision encoders are widely used as feature extractors for color-sensitive tasks such as dermatology, skin-tone auditing, and cosmetics, yet their color constancy under real capture changes has not been evaluated against physical measurements of the imaged surface. We audit twelve frozen encoders on 2,257 subjects photographed under a fully crossed design of capture white point and illuminance, each subject independently measured by a contact spectrophotometer. Changing white point at fixed illuminance moves embeddings by a large fraction of the distance between different people, along a direction shared across subjects; removing chroma eliminates this structure. Comparing encoder inputs and outputs, the perturbation enters as an almost shared translation and leaves as a subject-dependent residual more distributed than the difference between two people; a single-image gain ablation reproduces this with no pose or metering difference, while the white-point–illuminance interaction is already present at the encoder input at the level a separable multiplicative model predicts. The consequence is measurable in physical units: a skin-tone probe trained at one white point shifts the same person's predicted typology angle by several degrees under another and frequently changes the assigned skin-tone bin. We then compare corrections on one utility–invariance trade-off. Blind white balancing and a diagonal correction fitted to measured skin color do not reliably remove the sensitivity; projecting out a capture subspace estimated from unlabeled calibration captures recovers much of the cross-condition loss while preserving same-condition accuracy, and condition-mean differences estimate that subspace as well as paired displacements; pooling capture conditions when fitting the probe recovers the most.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Color Constancy, Frozen Vision Encoders, Representation Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3BzfOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies pre-trained vision feature capture actual skin tone compare with camera setting. Authors show that changing white balance alone can change the features and result in different skin tone predictions for the same person. Multiple settings works best, and other fixes offer limited benefits.

Critical strengths

Real photos and measured skin tone provide reliable GT. Experiments are good, with different models, layers, and strategies. I can understand the impact on skin tone classification in this research.

Critical weaknesses

The main experiments use only one phone and one studio setup, I am not sure about the generalization to other cameras. The main findings are intuitive. There is no comparison with standard color augmentation.

Questions for authors

Does multi-condition training still help with the same number of images or compared with color augmentation? Would learned white balance correction change the results?

Reviewer_1i5COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates the sensitivity of 12 frozen vision encoders to variations in capture conditions, specifically camera white point and illuminance. Utilizing a dataset of 2,257 subjects photographed under four fully crossed capture conditions, alongside physical skin color measurements (Individual Typology Angle, ITA) from an independent contact spectrophotometer, the study demonstrates that changing the white point causes representations to shift significantly. This shift is large enough to translate into substantial prediction errors in physical skin-tone estimation, frequently causing the exact same person to be re-binned into a different skin-tone category. The authors found that standard blind white balancing or simple diagonal gain corrections often fail to resolve—and sometimes exacerbate—the issue for language-supervised models, whereas projecting out a capture subspace or pooling capture conditions during probe training successfully mitigates the cross-condition loss.

Critical strengths

Instrumental Ground Truth: The use of an actual physical measurement (contact spectrophotometer) as a ground-truth reference, rather than relying on subjective human labels or synthetic corruptions, provides a highly rigorous and objective evaluation of color constancy.

Comprehensive Backbone Evaluation: The audit covers a wide array of 12 vision backbones with different architectures and pretraining objectives (CLIP, DINOv2, MAE, ResNet), establishing that this vulnerability is widespread.

Rigorous Ablation of the Encoder: Through a carefully designed synthetic gain ablation study, the paper successfully isolates the exact contribution of the neural network from the physical changes occurring in the camera pipeline.

Critical weaknesses

Dataset Diversity Constraints: The evaluation is strictly dependent on a single camera model (Galaxy S24 Ultra) and a specific studio lighting protocol from the AI Hub dataset, meaning its generalizability to different camera pipelines or unconstrained outdoor environments is unverified.

Domain Scope: The experiments are strictly limited to human faces and skin tones, leaving it unclear if the observed representation shifts behave similarly in other color-critical applications like agricultural sorting or textiles.

Input-side Correction Baseline: The evaluated input-side correction baseline relies primarily on a global diagonal gain, which may not adequately represent the robustness achievable by a fully color-checker-calibrated raw-to-sRGB processing pipeline.

Questions for authors

How might the white-point sensitivity and the geometric structure of the errors change if the encoders were fine-tuned end-to-end for the specific task, rather than being kept strictly frozen with linear probes?

Do you expect the proposed capture-subspace projection method to generalize and provide meaningful color constancy corrections on entirely different visual domains (e.g., product images, natural landscapes) without task-specific recalibration?

Here are the limitations of the proposed methods when considered as general-purpose AI solutions, written in an academic tone:

While the proposed solutions—such as pooled-condition probes and capture-subspace projection—are highly effective in controlled environments, they possess distinct limitations that hinder their seamless deployment as general-purpose AI solutions:

Strict Requirement for Multi-Condition Data: Both the projection and pooling methods fundamentally require multi-condition captures of the exact same subjects (e.g., the same person photographed under different white points and illuminance levels). Acquiring such perfectly paired, geometrically controlled data is highly impractical for "in-the-wild" real-world applications or web-crawled datasets.

Mandatory Probe Refitting (Lack of Plug-and-Play Capability): The subspace projection cannot be reliably inserted as a simple pre-processing step in front of an already deployed, frozen probe. The study demonstrates that a "fixed-probe deployment" actually degrades performance for certain models (e.g., worsening the error for SigLIP-B/16) because the projection may inadvertently remove useful directions that the existing probe already relies on. Consequently, practitioners must entirely refit the task probe on the corrected features, which adds operational overhead.

Narrow Domain and Hardware Validation: The efficacy of these mitigations was validated strictly on human faces and skin tones, utilizing a single camera pipeline and a specific studio lighting protocol. It remains completely unverified whether these compensation methods would successfully generalize to other color-critical visual domains (such as agriculture, textiles, or cosmetics) or function robustly under complex, unconstrained outdoor lighting conditions.

Reviewer_LoiFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper audits frozen encoders for skin-tone estimation. It finds they are not color-constant. They shoot 2257 people under a 2x2 white-point x illuminance design. Ground truth is a spectrophotometer ITA. Just changing the white point shifts the same person's predicted ITA by a few degrees. Many people even get re-binned. Input white-balancing does not reliably fix it. Projecting out a capture subspace helps. Pooling conditions when fitting the probe helps the most.

Critical strengths

Real recaptures plus a physical ground truth. That lets them blame skin color, not a label. The work is honest and well-controlled.

Critical weaknesses
  • No new method. All the fixes are off-the-shelf. Worse, their own results show the simplest baselines win. So it is an audit, not real progress. It does not fit ICLR well.
  • One camera, one studio. Here "white point" is a device setting, not real lighting. So the size, direction, even the sign of the effect could just be artifacts. No second camera. No in-the-wild test.
Questions for authors
  • Any second-camera or in-the-wild check? Does the effect and its fix transfer to other settings?
  • The writing needs improvement. The paper is difficult to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.