acceptodds
Under review as a conference paper at ICLR 2027

VLM-Gauge: Pricing Vision-Language Supervision for Causal Representation Learning

Abstract

Weakly supervised causal representation learning ties each latent variable to a labeled attribute and assumes those labels exist; because they come from human annotators, published experiments on real images supervise four to six attributes at most. We replace the human annotator with a vision-language model, which labels any attribute that can be named at negligible cost and so removes that limit, and ask what the substitution costs the representation. Calibrating the model on 500 human-labeled images per attribute yields its Youden index , and we show, in theory and in experiments with DEAR on four to twelve CelebA attributes, that this index prices a free label before any training: a label of index carries roughly the signal of human labels, and annotator labels on 17,000 images outperform human labels on 4,000. The gap to human labels narrows from four attributes to twelve, as the human-label score falls and both annotators' scores hold, and persists for two annotators, for CausalVAE and SCM-VAE, and for a second attribute set. Three further findings follow. Label noise stays local: replacing one attribute's labels with the model's costs that attribute up to 31 points without significantly harming any other, and removing the weakest attributes' labels does not help the rest. Seven correction rules, among them forward correction, posterior relabeling, learned loss weights and human labels spent where the annotator is weakest, do not improve on the uncorrected labels. Errors from a real annotator cost less than simulated errors at matched rates. Finally, the scores the existing literature reports are underdetermined: for CausalVAE and SCM-VAE on Pendulum, Flow and CelebA, six scoring conventions that neither paper states move MIC on the same trained models by up to 71 points. We release the calibration protocol, the convention audit and the scores of every run.

Primary area:
causal reasoning
Keywords:
causal representation learning, disentanglement, vision-language models, annotation cost, weak supervision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vBuqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VLM-Gauge, which replaces human-provided attribute labels in weakly supervised causal representation learning with labels generated by vision-language models. The central question is how much representation quality is lost when VLM-generated supervision substitutes for human annotation. The authors calibrate each VLM annotator on 500 human-labeled examples per attribute and use the Youden index J to characterize label quality. They theoretically argue that, under class-conditional label noise and a weak-signal approximation, n VLM-generated labels provide approximately the effective signal of nJ^2 human labels. Experiments primarily use DEAR on CelebA with four to twelve attributes, with additional experiments using CausalVAE, SCM-VAE, and Pendulum. The paper also studies label-noise locality, several correction strategies, real versus simulated label errors, and inconsistencies in MIC/TIC evaluation conventions in prior causal representation learning work.

Critical strengths
  1. Using VLMs as scalable semantic annotators is a natural direction, and explicitly quantifying the quality–cost trade-off rather than treating pseudo-labels as ground truth is valuable. The study also evaluates two annotators and several representation-learning architectures rather than relying on a single model.

  2. The authors analyze six different MIC scoring conventions and show that unreported evaluation choices can substantially change scores on the same trained models. This is a meaningful contribution to reproducibility in causal representation learning, independent of the VLM-supervision proposal itself.

Critical weaknesses
  1. Proposition 2 assumes class-conditional label noise, a weak-signal regime, and matched label prevalence, while the more general derivation introduces an additional variance factor κ\kappa. The paper nevertheless uses J^2 as the main pricing rule because it correlates better empirically. More importantly, the manuscript explicitly shows that the class-conditional assumption is violated for some attributes, with error rates varying substantially across subgroups. This weakens the interpretation of J^2 as a generally reliable pre-training estimate of supervision value.

  2. The paper treats CelebA attribute labels as ground truth when estimating false-positive/false-negative rates and computing the Youden index, but it also acknowledges that CelebA annotations contain errors and that some attributes are subjective. As a result, the measured J reflects agreement with CelebA annotations rather than necessarily with the true semantic attribute. This affects claims about VLM “errors,” annotator quality, and the effective value of a generated label. An independently re-annotated subset or a human-consensus reference would make the conclusions more convincing.

  3. The framework still requires 500 human-labeled calibration images per attribute, so the human supervision cost grows linearly with the number of attributes. In addition, almost all real-image experiments are conducted on CelebA, and scaling from four to twelve attributes is evaluated mainly with DEAR. The authors themselves acknowledge that larger attribute sets, other real-image datasets, other VLM families, and different causal orderings remain untested. Therefore, statements suggesting that VLMs can broadly replace human supervision for “any attribute that can be named” are stronger than the current evidence supports.

  4. The appendix reports that changing the wording of attribute questions can alter agreement with human labels by up to 13.4 percentage points for the stronger annotator, indicating that annotator quality is strongly prompt dependent. However, the main analysis largely treats J as an attribute-specific property. In addition, the theoretical results are derived for fixed features or population minimizers of the supervised cross-entropy term, while the experiments jointly optimize learned latent representations together with a generative/causal objective. Thus, conclusions about representation learning and the ineffectiveness of correction methods extend beyond what the theory directly guarantees.

  5. The claims that “label noise stays local,” correction methods do not help, and the human–VLM gap narrows with increasing attribute count are interesting but are based on a relatively restricted set of attributes, architectures, and datasets. For example, the narrowing gap may partly arise because DEAR’s human-label performance decreases as more attributes are added, potentially reflecting capacity or multi-task interference rather than improved relative effectiveness of VLM supervision. Similarly, the locality conclusion is established only for selected corrupted attributes and specific models. Stronger evidence would require more attributes, additional real-image datasets, prompt variations, capacity controls, and broader corruption settings.

Questions for authors

See the weaknesses

4 comments
AuthorsOpenReview ↗

Reply to Reviewer vBuq (1/2)

We thank the reviewer for the detailed review. JJ is the Youden index; gaps are in points of normalized MIC; the change test asks whether the change in J2J^2 predicts the change in score when only the label source changes.

W1. "This weakens the interpretation of J2J^2 as a generally reliable pre-training estimate"

As you note, the paper reports this violation (the largest gap in error rate between groups of images with the same label is 0.41, 0.32 and 0.52 for Qwen2.5-VL-7B, Qwen3.5-9B and Gemma-4-31B; App. C), and Proposition 3 covers it: errors on images resembling the other class cost less than JJ says, as Qwen3.5-9B's do (Sec. 6.5). We use J2J^2 as an ordering and an order of magnitude, tested out of sample with Gemma-4-31B (change test 0.98 and 0.96), on three architectures, on MPI3D-real and with DEAR's ordering reversed (Sec. 6.2, 6.6); CausalVAE's order test and MPI3D-real's change test are not significant. We keep J2J^2 rather than J2κJ^2\kappa because it orders the results better for every annotator and κ\kappa needs the prevalence (App. A).

W2. "An independently re-annotated subset ... would make the conclusions more convincing"

We agree that JJ measures agreement with CelebA's labels, and tested what that changes. On MPI3D-real, labeled by the recording equipment, J2J^2 still orders the attributes (0.91; Sec. 6.6). Against the labels of Lu & Zhong (2024), who re-checked 1,000 random CelebA images wherever a vision-language model disagreed with CelebA, JJ rises on 11, 12 and 11 of the twelve attributes for the three annotators in the order above, and on the 150 of them we hold out, scoring against these labels shrinks the gap to human labels by 2.4, 2.3 and 3.1 points (Holm-corrected p=0.088p = 0.088, 0.0880.088, 0.0140.014; App. C). The sample is small, only disputed labels were re-checked, and the re-check is a majority vote that includes the vision-language model, so it shows a direction, not a size: where CelebA's labels are wrong, scoring against them favors the runs trained on them, which works against the annotators.

W3. "Statements suggesting that VLMs can broadly replace human supervision for 'any attribute that can be named'"

We withdraw this phrase and the claim that human annotation is what limits prior work to four to six attributes (Sec. 1, 5.2). The paper now claims what it measured, on two model families, two real-image datasets (MPI3D-real with DEAR) and three architectures, and names where the annotators fail: Pendulum's continuous factors and MPI3D-real's LARGE OBJECT, whose JJ of 0.00 to 0.02 flags the failure before training (Sec. 8, Table 16). Human labels go mainly to calibration, which grows with attributes, not images, and needs few: JJ from 500 images is within 0.04 of its value on all 20,000 for all but one attribute, and 50 images order the attributes nearly as well (Sec. 3.2). The annotator labels the rest at one forward pass per image and attribute. Only DEAR scales beyond four attributes, as CausalVAE and SCM-VAE are written for four.

AuthorsOpenReview ↗

Reply to Reviewer vBuq (2/2)

W4a. "Annotator quality is strongly prompt dependent"

We agree that JJ depends on the wording: it belongs to the model with its wording and threshold, as Sec. 3.2 now states. The 13.4 points you cite are Qwen3.5-9B's (the original submission's "stronger annotator"), the largest standard deviation across wordings (App. C). The revision makes the wording a choice the price guides: calibrated labels pick it per attribute, and the change in J2J^2 predicts which attributes gain (change test 0.65 and 0.75 for Qwen3.5-9B and Gemma-4-31B, less clearly 0.56 for Qwen2.5-VL-7B; Sec. 6.2).

W4b. "The theoretical results are derived for fixed features or population minimizers"

This is the intended scope: the theory prices a label at fixed features, and the experiments, all with full joint training, test whether the price carries over. Sec. 4.1 now adds that Proposition 1 covers maps of the annotator's answer, which human labels on some images are not, and the experiments match: rules reading only Qwen3.5-9B's answers move normalized MIC by −3.0-3.0 to +1.1+1.1 (none significant after Holm correction), while targeted human labels, weighted within each attribute, gain 5.0±1.25.0 \pm 1.2 where it is near chance (Sec. 6.4). The results are consistent with the theory, not guaranteed by it.

W5a. "The narrowing gap may partly arise because DEAR's human-label performance decreases"

The capacity check finds no detectable capacity loss: with human labels, the first four attributes score 48.1 when four are supervised and 47.8 when twelve are (−0.3±13.7-0.3 \pm 13.7), and no annotator's score on them changes detectably (Sec. 6.1). The gap narrows only because the added attributes have smaller gaps, and not significantly, so we no longer claim it.

W5b. "The locality conclusion is established only for selected corrupted attributes"

We now relabel eight attributes one at a time in DEAR with Qwen3.5-9B's labels, all seven where its J<0.5J < 0.5 plus SMILING (three in the original submission). Only one untouched attribute moves significantly after Holm correction, YOUNG, and it improves (Fig. 3a); each newly relabeled attribute's own score follows its price (Pearson 0.98; Sec. 6.3). Locality also holds in SCM-VAE, while in CausalVAE near-chance labels can make a whole run fail (App. E.3). On Pendulum, corrupting the shadow position still moves the pendulum angle (−5.4-5.4, Holm-corrected p=0.041p = 0.041), so corruption now "mostly" does not travel along the graph.

W5c. "Correction methods do not help"

We narrowed this claim rather than dropping it: rules that read only Qwen3.5-9B's answers still do not help, while targeted human labels, weighted within each attribute, help where it is near chance (W4b; details in our reply to Reviewer YWHk). Of the further evidence you list, we added more relabeled attributes and the capacity check for these claims, and a second real-image dataset and other wordings for the price; attribute sets beyond twelve and other Pendulum corruption levels remain untested.

Does this address your concerns? We are happy to clarify further.

Reviewer_vBuqOpenReview ↗

Maintain the score

Thank you for your response. These have addressed some of my concerns. However, I still believe that my initial score accurately reflects my assessment of this paper, so I have decided to maintain that score.

Thank you for reading our responses, and for a detailed review that led to several of the new experiments: the re-checked CelebA labels, the capacity check and the additional relabeled attributes. We respect your assessment. The limits you raised remain stated in the paper (Sec. 8): CelebA's label errors, attribute sets beyond twelve, and that only DEAR scales beyond four attributes.

Reviewer_6RTqOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the use of VLM-generated labels for weakly supervised causal representation learning, aiming to reduce reliance on human annotations. The authors propose a label-pricing framework based on the Youden index, suggesting that the effective supervision value of a VLM-generated label can be approximated by J2J^2 human labels under certain assumptions. Through experiments primarily using DEAR, with additional validation on CausalVAE and SCM-VAE, the paper demonstrates that VLM-generated labels can provide useful supervision for learning attribute-aligned representations, and that label quality is correlated with the resulting representation quality. The authors further analyze the effects of label noise, investigate several noise correction strategies, and compare real VLM annotation errors with simulated noise. Finally, the paper examines existing CRL evaluation protocols and highlights the substantial influence of scoring conventions on reported results.

Critical strengths
    1. Well-motivated and practically relevant research question
      The paper addresses an important limitation of weakly supervised causal representation learning: its reliance on human-annotated attributes. By exploring VLM-generated labels as an alternative, the work provides a practical approach to scaling supervision. Furthermore, the proposed label-pricing analysis offers an interpretable perspective on how annotation quality affects representation learning, making the study more informative than a straightforward application of pseudo-labeling.
    1. Systematic empirical analysis and valuable evaluation insights
      The paper conducts extensive experiments to investigate how VLM-generated supervision affects learned representations, considering different annotators, attribute counts, CRL architectures, and noise correction strategies. These experiments provide meaningful evidence supporting the main findings. Additionally, the analysis of existing CRL evaluation protocols reveals substantial sensitivity to scoring conventions, highlighting an important reproducibility issue for the research community.
Critical weaknesses
    1. Limited generalizability of the experimental findings
      Although the paper provides a meaningful analysis of VLM-generated supervision, the main experiments are conducted on CelebA using two Qwen-based annotators, with most label-pricing experiments relying on DEAR. It remains unclear whether the observed relationship between annotation quality and representation performance would generalize to other datasets, attribute types, or VLM families. This limits the broader applicability of the proposed label-pricing framework.
    1. Limited analysis of human annotation requirements for label pricing
      Although the proposed NJ2NJ^2 formulation aims to quantify the value of VLM-generated supervision, estimating JJ still requires human-labeled calibration data. The paper uses a fixed budget of 500 human annotations per attribute but does not systematically investigate how the calibration size affects the reliability of the estimated label values. Since annotation requirements may vary across different VLMs, datasets, and attribute distributions, the actual human annotation cost remains unclear. This limits the assessment of the proposed framework's practical scalability and cost-effectiveness.
Questions for authors
    1. Applicability of Label Pricing
      The proposed J2J^2 formulation is derived under specific assumptions, while the paper also observes instance-dependent annotation errors. Could the authors further clarify how reliably J2J^2 can estimate the effective supervision value when these assumptions are not fully satisfied? In particular, how should its predicted label value be interpreted beyond the evaluated settings?
    1. Human Annotation Requirements for Calibration
      The proposed framework requires 500 human-labeled samples per attribute to estimate the Youden index. Could the authors clarify whether fewer annotations would provide similarly reliable estimates? In particular, how sensitive is the estimated label value to calibration set size, and would the required number of annotations vary across different VLM annotators, datasets, or attribute distributions?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer 6RTq

We thank the reviewer for the positive assessment and the questions. JJ is the Youden index; the order test asks whether attributes with a higher J2J^2 keep more of the human-label score, and the change test whether the change in J2J^2 predicts the change in score when only the label source changes.

W1. Generalizability to "other datasets, attribute types, or VLM families"

We extended the tests along each line (Sec. 6.2, 6.6): a second VLM family, Gemma-4-31B, whose JJ was measured before any of its runs; MPI3D-real, photographs of a robot arm whose attributes describe the object, the camera and the background rather than faces, labeled by the recording equipment; CausalVAE and SCM-VAE with all three annotators, as well as DEAR; and DEAR with the causal ordering reversed.

SettingOrder testChange test
Gemma-4-31B (JJ measured before any run)0.66 (p=0.037p = 0.037)0.98 and 0.96 (with Qwen2.5-VL-7B and Qwen3.5-9B)
Three annotators, within attributesn/a0.95 (p=2.5×10−5p = 2.5 \times 10^{-5})
MPI3D-real (DEAR)0.91 (p=1.2×10−4p = 1.2 \times 10^{-4})0.59 (p=0.051p = 0.051)
CausalVAE, three annotators0.43 (p=0.10p = 0.10)0.74 (p=0.020p = 0.020)
SCM-VAE, three annotators0.67 (p=0.033p = 0.033)0.95 (p=0.022p = 0.022)
DEAR, ordering reversed0.68 to 0.780.90 (p=0.002p = 0.002)

Two tests are not significant, and the paper says so. Still untested: more than twelve attributes, non-binary attributes and other orderings (Sec. 8).

W2 and Q2. "Whether fewer annotations would provide similarly reliable estimates"

The 500-image estimate is accurate: on all 20,000 CelebA images we train and score on, each annotator's JJ is within 0.04 of it on every attribute but one (Sec. 3.2). For ranking, 50 images per attribute (600 human labels for twelve attributes) come close: the median order test stays within 0.05 of its resampled value at 500 for every annotator, and on MPI3D-real it is 0.90 against 0.91 (App. C, Table 13). An exact price needs more: at 50 images, the median largest error in JJ is about 0.2. The bound on JJ's standard error, 1/(2m)\sqrt{1/(2m)} with mm images per class, does not depend on the annotator or dataset; what varies is the search for rare attributes, about 1,050 images to find 25 of each class for BALD (2.4% of images). Beyond calibration, training labels come from the annotator at one forward pass per image and attribute, with human labels added only where it is near chance.

Q1. "How reliably J2J^2 can estimate the effective supervision value when these assumptions are not fully satisfied"

J2J^2 is a first-order guide, not a bound: it ranks attributes and gives a label's order of magnitude (Sec. 6.2, App. E.2). Qwen3.5-9B's MOUTH OPEN, priced at 1,257 human labels, beats human labels on 1,000 images and loses to 4,000, while Qwen2.5-VL-7B's HEAVY MAKEUP, at the same price, loses to 250 (Table 17). Where Assumption 1 fails, Proposition 3 predicts that errors on images resembling the other class cost less than JJ says, and Qwen3.5-9B's errors do cost less than random flips at its measured rates (Sec. 6.5). Single attributes can miss their price in either direction (App. E.2), and the variance factor κ\kappa does not rank results better for any annotator (App. A). Beyond our settings, we recommend J2J^2 for the two decisions the paper tests: which attributes to trust, and where human labels are worth spending.

Does this answer your questions? We are happy to clarify further.

Reviewer_YWHkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The goal of this paper is to investigate whether vision-language models can replace human annotations in weakly supervised causal representation learning. Authors evaluate two Qwen-based annotators, calibrated using 500 human-labeled CelebA images per attribute. They assess annotation quality using Youden's index (J) and theoretically establish that n annotator-generated labels provide approximately the same statistical signal as nJ² human labels. This suggests that J² can be used to estimate the value of annotations before model training. The experiments use DEAR with 4 to 12 CelebA attributes. Results show that human labels consistently outperform VLM-generated labels, although the performance gap decreases as the number of attributes increases. Interestingly, 17,000 labels from the stronger annotator outperform 4,000 human labels. Authors also find that noise in one attribute does not affect the others, none of the seven correction techniques improves performance over uncorrected labels, and actual annotator errors are less damaging than simulated errors at comparable rates. Finally, authors examine the reported MIC scores for CausalVAE and SCM-VAE and find that six previously unspecified scoring choices can change MIC by as much as 71 points for the same trained models.

Critical strengths

The theoretical treatment presented in this paper is sound and well connected to the experimental observations. Proposition 1 explains why label correction techniques cannot improve rank-based criteria, which is consistent with the experimental findings in Section 6.4. The experiment in which authors change only the annotator while keeping the images, attributes, and model architecture fixed is particularly useful. The performance changes closely follow the changes in J², with a Pearson correlation of 0.91. This provides strong empirical support for the theoretical argument. The experimental methodology is also appropriate, with five random seeds, paired confidence intervals, and Holm correction. I appreciate that authors openly report their negative findings.

Another strength of this paper is the evaluation audit in Section 7. Using the bound MIC ≤ H(Y), authors show that some previously published MIC values for CelebA-Beard cannot represent the mean of raw MIC scores. This is an important observation for causal representation learning research. How can MIC scores from different studies be compared if their scoring conventions are not clearly specified? Authors also provide the scores from all experimental runs and the corresponding scripts, which is useful for reproducibility.

Critical weaknesses

First, I am not sure how the results from targeted human labeling are consistent with the price of a label discussed in Section 4.2. According to the mixture formula, when annotator labels are weighted by J, their contribution should be very small for attributes where J is close to zero. Therefore, I expected the mixture to perform similarly to using only human labels for these attributes. Figure 2b shows that even 250 human labels outperform the annotator on three of the four near-chance attributes. Why, then, does using 3,500 human labels on weak attributes not improve the results significantly? Authors report an improvement of only 0.4 ± 1.9 points against a performance gap of 14.1 points. They mention that weighting by J does not work, but do not provide a clear explanation for this observation. How are these 3,500 labels distributed across the seven attributes? What happens if we consider only the four near-chance attributes? I think this is important because the price of a label is the central argument of this paper, and the experimental results do not appear to support it.

Second, I have some questions about how the annotator labels are generated. Authors use a threshold of 0.5 on the probability of "Yes" to obtain binary labels. However, Youden's index can be used to select a threshold that maximizes J. Authors already have 500 human-labeled images for each attribute. Why not use these images to determine an appropriate threshold? For example, the poor results for HIGH CHEEKBONES and POINTY NOSE could be due to the threshold selection rather than a lack of useful information from the annotator. Also, the VLM probabilities contain information that is lost when converted into binary labels. Would using soft labels improve the results? This possibility is not addressed by the argument in Section 4.3. Authors recommend improving the annotator instead of modifying the loss, but these relatively simple alternatives are not tested. Another concern is the motivation of this work. Authors argue that human annotation restricts existing research to four to six attributes. However, CelebA already provides human annotations for 40 attributes, and authors themselves use 12 attributes with human labels for DEAR. I am not sure whether annotation is really the limiting factor. The model design and causal graph may impose more important limitations. I think the practical significance of this approach would be better established using a dataset where attribute labels are unavailable, which is not considered in this paper.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer YWHk

We thank the reviewer for the detailed review. JJ is the Youden index; gains are in points of normalized MIC; the change test asks whether the change in J2J^2 predicts the change in score when only the label source changes.

W1. "Why, then, does using 3,500 human labels on weak attributes not improve the results significantly?"

Your reading of the mixture formula is right, and the corrected experiment agrees with it. The 3,500 labels are 500 training images labeled on the seven attributes where Qwen3.5-9B has J<0.5J < 0.5, four of them near chance. The original rule normalized the JJ weights over all attributes in a batch, which at twelve attributes cut a near-chance attribute's share of the loss from 8.3% to as little as 0.5%. The price (Sec. 4.2) assumes weights within each attribute's own term, so we added that rule (Sec. 6.4, Table 9). On the four near-chance attributes, against Qwen3.5-9B's labels:

LabelsGain
Human labels on 500 images (the price's benchmark)+2.4±1.7+2.4 \pm 1.7
Targeted, weighted within each attribute+5.0±1.2+5.0 \pm 1.2
Weighted within, minus 500 human labels+2.6±2.2+2.6 \pm 2.2 (p=0.029p = 0.029, uncorrected)
Weighted within, minus the original rule+3.7±2.0+3.7 \pm 2.0 (p=0.007p = 0.007, uncorrected)
Human labels on all 17,000 images+12.3±1.2+12.3 \pm 1.2

The figures you cite, 0.4±1.90.4 \pm 1.9 and 14.1, are the original rule and the gap to all 17,000 human labels on the seven targeted attributes, pooled over runs at six and twelve attributes; both stay in Table 9. Averaged over all twelve attributes, the corrected rule's gain is not significant (+2.5±2.7+2.5 \pm 2.7). Sec. 6.4 now states that rules reading only Qwen3.5-9B's answers do not help, while human labels spent where it is near chance, weighted within each attribute, do. The annotator swap you found useful (0.91) also extends to Gemma-4-31B, whose JJ was measured before any of its runs (0.98 and 0.96), and to CausalVAE and SCM-VAE (0.74 and 0.95; Sec. 6.2, 6.6).

W2a. "Why not use these images to determine an appropriate threshold?" and soft labels

We now do. Calibrated labels take the wording and threshold with the highest JJ on the calibration images (Sec. 3.2), and the price predicts which attributes gain: change test 0.65 and 0.75 for Qwen3.5-9B and Gemma-4-31B (Holm-corrected p=0.042p = 0.042 and 0.0160.016), and less clearly for Qwen2.5-VL-7B (0.56, p=0.056p = 0.056). The mean gain, 2.5 to 4.7 points, is not significant after Holm correction. On HIGH CHEEKBONES and POINTY NOSE, no threshold can help Qwen3.5-9B: on its four near-chance attributes, the AUROC of its probability is 0.50 to 0.55. Another wording helps a little there (held-out JJ of 0.14 and 0.19, App. C; gains of +3.6±2.0+3.6 \pm 2.0 and +1.5±1.2+1.5 \pm 1.2, Table 15). Soft labels give no reliable gain for any annotator (Table 15), and Sec. 4.3 now restricts its argument to maps of the yes/no answer. These results are consistent with the advice in Sec. 8 to improve the annotator rather than the loss.

W2b. "CelebA already provides human annotations for 40 attributes"

CelebA is our testbed because its labels let us measure what annotator labels cost; its human labels are the reference, not the use case. The use case is an attribute or a dataset without labels. There, both kinds of label grow with images times attributes, but an annotator label is one forward pass of the model, and Lu & Zhong (2024) put an 8B vision-language model's labels at under 1% of the cost of CelebA's human annotation (cited in Sec. 1); human labels are then needed mainly to calibrate, 50 to 500 per attribute, and where the annotator is near chance (Sec. 8). We agree that architectures also limit the attribute count (CausalVAE and SCM-VAE are written for four; Sec. 5.2), and the paper no longer names annotation as the cause (Sec. 1). As for the graph, DEAR needs only an ordering of the twelve attributes, and reversing it leaves the price tests in place (Sec. 3.1, 6.6).

W2c. "A dataset where attribute labels are unavailable"

That is the setting the method is for: with no labels to score against, JJ from a few hundred calibration labels tells, before training, which attributes an annotator can supply. Checking this needs true labels, so we added MPI3D-real, whose five attributes come from the recording equipment (Sec. 6.6). There J2J^2 orders the attributes (Spearman 0.91), the annotator with the highest JJ comes closest to the true labels (7.5 points for Gemma-4-31B, against 21.7 and 23.6 for Qwen3.5-9B and Qwen2.5-VL-7B), and calibration flags LARGE OBJECT, which all three annotators answer at chance, before training. The change test is not significant here (p=0.051p = 0.051).

Does this address your concerns? We are happy to clarify further.

Thank you for the detailed response and the additional experiments. I appreciate that the authors found the reason behind the targeted labels result. Normalizing the J weights over all attributes in a batch explains why the near-chance attributes did not gain, and with weighting within each attribute, the gain of 5.0 points now agrees with the price in Section 4.2. The threshold and soft-label experiments also address my second concern, and it is now clear that the near-chance attributes have no signal (AUROC of 0.50 to 0.55), not a poorly chosen threshold. The new results with Gemma-4-31B, whose J was measured before any training, make the price more convincing.

I am curious whether the same batch normalization also affected the other weighted rules in Figure 4, for example, the learned per-attribute loss weights. If it did, their results may need to be revisited as well.

The MPI3D-real experiment is a reasonable proxy for the unlabeled setting, and I appreciate that the abstract claims only the ordering there. Still, with only 5 attributes, the evidence is limited, and it would be good to mention this in the discussion of that experiment. I would also appreciate a clarification on the DCI scale in Section 7 and on the rule for excluding the two failed runs.

AuthorsOpenReview ↗

Reply to Reviewer YWHk (follow-up)

Thank you for the careful follow-up. We answer the three points in turn. Revised paper version 2, now uploaded, adds each clarification to the paper and changes nothing else.

1. "Whether the same batch normalization also affected the other weighted rules in Figure 4"

No. Only the original targeted rule divided by the total weight over all attributes in a batch. The learned per-attribute weights (Kendall et al., 2018) first average each attribute's loss over its own labeled images and then scale it by that attribute's learned factor, so no attribute's share depends on the others' weights. Forward correction, posterior relabeling and the unweighted targeted rule keep every attribute at its plain 1/12 share. One related effect: with the seven weak attributes unlabeled, the supervised term averages over the five that keep labels, so each holds 1/5 of it instead of 1/12. That gives them more weight, not less, yet none of them improves, and λ\lambda from 1 to 10 leaves the gap in place (Table 10). App. D now states both.

2. "With only 5 attributes, the evidence is limited"

We agree. Each annotator contributes only five points on MPI3D-real; the order test pools the 15 annotator and attribute pairs within annotators (p=1.2×10−4p = 1.2 \times 10^{-4}), and the change test is not significant (p=0.051p = 0.051). Sec. 6.6 now adds that five attributes give limited evidence.

3. "A clarification on the DCI scale in Section 7 and on the rule for excluding the two failed runs"

DCI scores lie between 0 and 1, so "within 0.23" in Sec. 7 is a difference in DCI, not in MIC points; Sec. 7 now says so. Our scores against ICM-VAE's published ones (App. E.5):

ICM-VAE's DCIOursPublishedGap
Flow, disentanglement0.9800.9880.008
Flow, completeness0.6000.5980.002
Pendulum, disentanglement0.9400.9970.057
Pendulum, completeness0.6560.8820.226

No run is excluded. As App. D states, runs that under-trained or collapsed are kept: the comparison of Sec. 6.5 uses all five seeds (−8.4±7.5-8.4 \pm 7.5). Two of its simulated-label runs did not train, with DCI informativeness of −0.01-0.01 and 0.020.02 against at least 0.190.19 for every run on Qwen3.5-9B's labels, so we also report the other three seeds (−4.2±2.4-4.2 \pm 2.4); both are in Table 9. Sec. 6.5 and App. E.2 now say that all five are kept.

Thank you again for engaging with the details.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reviews, and for noting the sound theory (YWHk), the motivation (6RTq) and the audit of scoring conventions (all three). We have uploaded a revised paper and supplement with 150 new runs (428 in total), and answer each point in the reply to each reviewer. "Original submission" below is the version you reviewed; section, table and figure numbers are the revised paper's unless stated.

What the new experiments show. The price of a label, set by the Youden index JJ from a few hundred human labels before training, now holds beyond the original setting (order and change tests as in Sec. 5.3):

  1. Gemma-4-31B, a second model family [6RTq, vBuq]: with its JJ measured before any of its runs, it follows the price (change test 0.98 and 0.96) and reaches 87% of the human-label MIC at twelve attributes (Sec. 6.1, 6.2).
  2. MPI3D-real, labeled by the recording equipment [YWHk, 6RTq, vBuq]: with DEAR, J2J^2 orders its attributes (Spearman 0.91; Sec. 6.6).
  3. CausalVAE, SCM-VAE and a reversed ordering [6RTq, vBuq]: the change test holds on all three architectures and with DEAR's ordering reversed; CausalVAE's order test and MPI3D-real's change test are not significant (Sec. 6.6).
  4. Targeted human labels [YWHk, vBuq]: weighted within each attribute, they gain 5.0±1.25.0 \pm 1.2 points where Qwen3.5-9B is near chance, at least what the price predicts (500 human labels alone gain 2.4±1.72.4 \pm 1.7; Sec. 6.4).
  5. Calibration [YWHk, 6RTq, vBuq]: chosen wording and threshold raise mean normalized MIC by 2.5 to 4.7 points (not significant after Holm correction); 50 images per attribute rank nearly as well as 500 (Sec. 3.2, 6.2).
  6. Label errors, locality, capacity [vBuq]: re-checked CelebA labels shrink the gap to human labels (a direction, significant for Gemma-4-31B); noise stays local across eight relabeled attributes in DEAR; more attributes do not detectably change earlier ones (Sec. 6.1, 6.3, App. C).

Claims we narrowed. We withdraw "any attribute that can be named" and the TL;DR's "any attribute a causal representation needs" (the revision form has no TL;DR field), human annotation as the reason prior work uses four to six attributes, and the narrowing gap, which is not significant. Section 6.4, "Corrections do not help" in the original submission, is now "What corrections can and cannot do"; annotator labels on 17,000 images beat human labels on 4,000 for Qwen3.5-9B and Gemma-4-31B only; and on Pendulum, with Holm correction now applied, corruption "mostly" does not travel along the graph.

Names and numbers. The original submission's "weaker" and "stronger" annotators are Qwen2.5-VL-7B and Qwen3.5-9B; its Table 6 is now Tables 6 to 11, its Table 7 is Table 12, and Tables 13 to 17 are new.

We hope these changes address your concerns, and thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.