VLM-Gauge: Pricing Vision-Language Supervision for Causal Representation Learning

Weakly supervised causal representation learning ties each latent variable to a labeled attribute and assumes those labels exist; because they come from human annotators, published experiments on real images supervise four to six attributes at most. We replace the human annotator with a vision-language model, which labels any attribute that can be named at negligible cost and so removes that limit, and ask what the substitution costs the representation. Calibrating the model on 500 human-labeled images per attribute yields its Youden index $J$, and we show, in theory and in experiments with DEAR on four to twelve CelebA attributes, that this index prices a free label before any training: a label of index $J$ carries roughly the signal of $J^2$ human labels, and annotator labels on 17,000 images outperform human labels on 4,000. The gap to human labels narrows from four attributes to twelve, as the human-label score falls and both annotators' scores hold, and persists for two annotators, for CausalVAE and SCM-VAE, and for a second attribute set. Three further findings follow. Label noise stays local: replacing one attribute's labels with the model's costs that attribute up to 31 points without significantly harming any other, and removing the weakest attributes' labels does not help the rest. Seven correction rules, among them forward correction, posterior relabeling, learned loss weights and human labels spent where the annotator is weakest, do not improve on the uncorrected labels. Errors from a real annotator cost less than simulated errors at matched rates. Finally, the scores the existing literature reports are underdetermined: for CausalVAE and SCM-VAE on Pendulum, Flow and CelebA, six scoring conventions that neither paper states move MIC on the same trained models by up to 71 points. We release the calibration protocol, the convention audit and the scores of every run.