- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VLM-Gauge, which replaces human-provided attribute labels in weakly supervised causal representation learning with labels generated by vision-language models. The central question is how much representation quality is lost when VLM-generated supervision substitutes for human annotation. The authors calibrate each VLM annotator on 500 human-labeled examples per attribute and use the Youden index J to characterize label quality. They theoretically argue that, under class-conditional label noise and a weak-signal approximation, n VLM-generated labels provide approximately the effective signal of nJ^2 human labels. Experiments primarily use DEAR on CelebA with four to twelve attributes, with additional experiments using CausalVAE, SCM-VAE, and Pendulum. The paper also studies label-noise locality, several correction strategies, real versus simulated label errors, and inconsistencies in MIC/TIC evaluation conventions in prior causal representation learning work.
Critical strengths
-
Using VLMs as scalable semantic annotators is a natural direction, and explicitly quantifying the quality–cost trade-off rather than treating pseudo-labels as ground truth is valuable. The study also evaluates two annotators and several representation-learning architectures rather than relying on a single model.
-
The authors analyze six different MIC scoring conventions and show that unreported evaluation choices can substantially change scores on the same trained models. This is a meaningful contribution to reproducibility in causal representation learning, independent of the VLM-supervision proposal itself.
Critical weaknesses
-
Proposition 2 assumes class-conditional label noise, a weak-signal regime, and matched label prevalence, while the more general derivation introduces an additional variance factor . The paper nevertheless uses J^2 as the main pricing rule because it correlates better empirically. More importantly, the manuscript explicitly shows that the class-conditional assumption is violated for some attributes, with error rates varying substantially across subgroups. This weakens the interpretation of J^2 as a generally reliable pre-training estimate of supervision value.
-
The paper treats CelebA attribute labels as ground truth when estimating false-positive/false-negative rates and computing the Youden index, but it also acknowledges that CelebA annotations contain errors and that some attributes are subjective. As a result, the measured J reflects agreement with CelebA annotations rather than necessarily with the true semantic attribute. This affects claims about VLM “errors,” annotator quality, and the effective value of a generated label. An independently re-annotated subset or a human-consensus reference would make the conclusions more convincing.
-
The framework still requires 500 human-labeled calibration images per attribute, so the human supervision cost grows linearly with the number of attributes. In addition, almost all real-image experiments are conducted on CelebA, and scaling from four to twelve attributes is evaluated mainly with DEAR. The authors themselves acknowledge that larger attribute sets, other real-image datasets, other VLM families, and different causal orderings remain untested. Therefore, statements suggesting that VLMs can broadly replace human supervision for “any attribute that can be named” are stronger than the current evidence supports.
-
The appendix reports that changing the wording of attribute questions can alter agreement with human labels by up to 13.4 percentage points for the stronger annotator, indicating that annotator quality is strongly prompt dependent. However, the main analysis largely treats J as an attribute-specific property. In addition, the theoretical results are derived for fixed features or population minimizers of the supervised cross-entropy term, while the experiments jointly optimize learned latent representations together with a generative/causal objective. Thus, conclusions about representation learning and the ineffectiveness of correction methods extend beyond what the theory directly guarantees.
-
The claims that “label noise stays local,” correction methods do not help, and the human–VLM gap narrows with increasing attribute count are interesting but are based on a relatively restricted set of attributes, architectures, and datasets. For example, the narrowing gap may partly arise because DEAR’s human-label performance decreases as more attributes are added, potentially reflecting capacity or multi-task interference rather than improved relative effectiveness of VLM supervision. Similarly, the locality conclusion is established only for selected corrupted attributes and specific models. Stronger evidence would require more attributes, additional real-image datasets, prompt variations, capacity controls, and broader corruption settings.
Questions for authors
See the weaknesses
4 comments
Reply to Reviewer vBuq (2/2)
W4a. "Annotator quality is strongly prompt dependent"
We agree that depends on the wording: it belongs to the model with its wording and threshold, as Sec. 3.2 now states. The 13.4 points you cite are Qwen3.5-9B's (the original submission's "stronger annotator"), the largest standard deviation across wordings (App. C). The revision makes the wording a choice the price guides: calibrated labels pick it per attribute, and the change in predicts which attributes gain (change test 0.65 and 0.75 for Qwen3.5-9B and Gemma-4-31B, less clearly 0.56 for Qwen2.5-VL-7B; Sec. 6.2).
W4b. "The theoretical results are derived for fixed features or population minimizers"
This is the intended scope: the theory prices a label at fixed features, and the experiments, all with full joint training, test whether the price carries over. Sec. 4.1 now adds that Proposition 1 covers maps of the annotator's answer, which human labels on some images are not, and the experiments match: rules reading only Qwen3.5-9B's answers move normalized MIC by to (none significant after Holm correction), while targeted human labels, weighted within each attribute, gain where it is near chance (Sec. 6.4). The results are consistent with the theory, not guaranteed by it.
W5a. "The narrowing gap may partly arise because DEAR's human-label performance decreases"
The capacity check finds no detectable capacity loss: with human labels, the first four attributes score 48.1 when four are supervised and 47.8 when twelve are (), and no annotator's score on them changes detectably (Sec. 6.1). The gap narrows only because the added attributes have smaller gaps, and not significantly, so we no longer claim it.
W5b. "The locality conclusion is established only for selected corrupted attributes"
We now relabel eight attributes one at a time in DEAR with Qwen3.5-9B's labels, all seven where its plus SMILING (three in the original submission). Only one untouched attribute moves significantly after Holm correction, YOUNG, and it improves (Fig. 3a); each newly relabeled attribute's own score follows its price (Pearson 0.98; Sec. 6.3). Locality also holds in SCM-VAE, while in CausalVAE near-chance labels can make a whole run fail (App. E.3). On Pendulum, corrupting the shadow position still moves the pendulum angle (, Holm-corrected ), so corruption now "mostly" does not travel along the graph.
W5c. "Correction methods do not help"
We narrowed this claim rather than dropping it: rules that read only Qwen3.5-9B's answers still do not help, while targeted human labels, weighted within each attribute, help where it is near chance (W4b; details in our reply to Reviewer YWHk). Of the further evidence you list, we added more relabeled attributes and the capacity check for these claims, and a second real-image dataset and other wordings for the price; attribute sets beyond twelve and other Pendulum corruption levels remain untested.
Does this address your concerns? We are happy to clarify further.
Maintain the score
Thank you for your response. These have addressed some of my concerns. However, I still believe that my initial score accurately reflects my assessment of this paper, so I have decided to maintain that score.
Thank you for reading our responses, and for a detailed review that led to several of the new experiments: the re-checked CelebA labels, the capacity check and the additional relabeled attributes. We respect your assessment. The limits you raised remain stated in the paper (Sec. 8): CelebA's label errors, attribute sets beyond twelve, and that only DEAR scales beyond four attributes.
Reply to Reviewer vBuq (1/2)
We thank the reviewer for the detailed review. J is the Youden index; gaps are in points of normalized MIC; the change test asks whether the change in J2 predicts the change in score when only the label source changes.
W1. "This weakens the interpretation of J2 as a generally reliable pre-training estimate"
As you note, the paper reports this violation (the largest gap in error rate between groups of images with the same label is 0.41, 0.32 and 0.52 for Qwen2.5-VL-7B, Qwen3.5-9B and Gemma-4-31B; App. C), and Proposition 3 covers it: errors on images resembling the other class cost less than J says, as Qwen3.5-9B's do (Sec. 6.5). We use J2 as an ordering and an order of magnitude, tested out of sample with Gemma-4-31B (change test 0.98 and 0.96), on three architectures, on MPI3D-real and with DEAR's ordering reversed (Sec. 6.2, 6.6); CausalVAE's order test and MPI3D-real's change test are not significant. We keep J2 rather than J2κ because it orders the results better for every annotator and κ needs the prevalence (App. A).
W2. "An independently re-annotated subset ... would make the conclusions more convincing"
We agree that J measures agreement with CelebA's labels, and tested what that changes. On MPI3D-real, labeled by the recording equipment, J2 still orders the attributes (0.91; Sec. 6.6). Against the labels of Lu & Zhong (2024), who re-checked 1,000 random CelebA images wherever a vision-language model disagreed with CelebA, J rises on 11, 12 and 11 of the twelve attributes for the three annotators in the order above, and on the 150 of them we hold out, scoring against these labels shrinks the gap to human labels by 2.4, 2.3 and 3.1 points (Holm-corrected p=0.088, 0.088, 0.014; App. C). The sample is small, only disputed labels were re-checked, and the re-check is a majority vote that includes the vision-language model, so it shows a direction, not a size: where CelebA's labels are wrong, scoring against them favors the runs trained on them, which works against the annotators.
W3. "Statements suggesting that VLMs can broadly replace human supervision for 'any attribute that can be named'"
We withdraw this phrase and the claim that human annotation is what limits prior work to four to six attributes (Sec. 1, 5.2). The paper now claims what it measured, on two model families, two real-image datasets (MPI3D-real with DEAR) and three architectures, and names where the annotators fail: Pendulum's continuous factors and MPI3D-real's LARGE OBJECT, whose J of 0.00 to 0.02 flags the failure before training (Sec. 8, Table 16). Human labels go mainly to calibration, which grows with attributes, not images, and needs few: J from 500 images is within 0.04 of its value on all 20,000 for all but one attribute, and 50 images order the attributes nearly as well (Sec. 3.2). The annotator labels the rest at one forward pass per image and attribute. Only DEAR scales beyond four attributes, as CausalVAE and SCM-VAE are written for four.