- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a sampling-based method to recover the implicit visual priors of multimodal large language models. By combining controllable image generation with pairwise comparisons, it estimates joint distributions over visual attributes and analyzes how these preferences vary across concepts and models.
Critical strengths
-
The paper introduces a cognitively inspired framework for quantitatively estimating the implicit visual priors of multimodal large language models (MLLMs).
-
By combining interpretable slider spaces with FLUX-based image generation and adaptive sampling, the proposed method moves beyond fixed stimulus sets and extends prior work by Zhu et al. on implicit prior elicitation from LLMs to the multimodal setting.
-
The method represents implicit priors as interpretable joint distributions over visual attributes, enabling systematic comparisons across models and concepts while capturing dependencies between attributes.
Critical weaknesses
-
Convergence and reliability of recovered priors. Appendix E reports substantial convergence issues. For example, in Aesthetics, 50% of parameters for Qwen3-VL and 58% for Gemini 3 Flash have split-R-hat > 1.5 with median acceptance rates below 10%. Although the authors acknowledge these limitations, it remains unclear how they affect the reliability of the recovered distributions. Further convergence analysis and stability checks would strengthen the results.
-
Insufficient comparison with explicit baselines. The explicit baselines in Appendix C rely on one-dimensional slider sweeps and do not adequately model inter-slider dependencies. Thus, it remains unclear whether IMH's advantage comes from implicit sampling or simply from capturing interactions neglected by the baselines. Moreover, sharper distributions are not necessarily more accurate. An interaction-aware explicit baseline with a comparable query budget and additional independent validation would help establish the proposed method's advantages.
-
Potential biases in slider selection and labeling. The authors manually select 10 out of 64 candidate axes, potentially introducing subjective biases into the recovered priors. I would encourage a human-subject evaluation involving multiple independent participants to establish group-level consensus and assess selection reliability. Additionally, the automatic semantic labeling pipeline would benefit from quantitative validation against human judgments.
-
Limited surprise of the recovered priors. Although the authors claim to uncover "surprising novel priors," several reported associations involving facial appearance, personality, and demographic attributes resemble biases documented in previous studies [1, 2, 3]. The authors could better distinguish genuinely novel findings from previously documented biases.
-
Limited cross-modal validation. Although the framework is described as modality-agnostic, all experiments focus on images. Evaluating another modality would provide stronger empirical support for its generality.
-
Limited quantitative evidence regarding direct slider manipulation. The authors discuss practical difficulties with direct slider manipulation, including oscillation and inefficient exploration. While these observations provide useful motivation, a quantitative comparison would help clarify the practical advantages of the proposed sampling approach.
[1] Howard, P., Fraser, K. C., Bhiwandiwalla, A., & Kiritchenko, S. (2025). Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2025), 5946–5991. https://doi.org/10.18653/v1/2025.naacl-long.305
[2] Gulati, A., D’Incà, M., Sebe, N., Lepri, B., & Oliver, N. (2025). Beauty and the Bias: Exploring the Impact of Attractiveness on Multimodal Large Language Models. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), 8(2), 1154–1168. https://doi.org/10.1609/aies.v8i2.36619
[3] Lehr, S. A., Lothe, Y., & Banaji, M. R. (2026). Like Humans, Language Models Demonstrate Face-to-Character Biases. PNAS Nexus, 5(8), pgag247. https://doi.org/10.1093/pnasnexus/pgag247
Questions for authors
-
How do the convergence issues reported in Appendix E affect the reliability of the recovered priors? Are the main findings stable across independent chains or longer sampling runs?
-
Have the authors considered an explicit baseline that directly evaluates multidimensional slider combinations? How can we distinguish the benefits of implicit sampling from those of modeling inter-slider interactions?
-
How were the 10 axes selected from the 64 candidates? Was the selection based on judgments from multiple independent participants? How sensitive are the recovered priors to different axis selections?
-
Has the automatic semantic labeling pipeline been validated against human judgments? To what extent do the assigned labels reflect human interpretations of the visual attributes?
-
Which of the recovered priors do the authors consider genuinely novel compared with previously documented MLLM biases? What additional insights does the proposed method reveal beyond existing findings?
9 comments
(W1 and Q1) Convergence and reliability of recovered priors
Our predictive checks show that these chains do meaningfully capture models' preferences. Within chains, we predict held-out accept and reject decisions with median accuracies of $\approx$ 0.70--0.83 across all models and domains (Figure 15). This includes conditions with the highest $\hat{R}$. Then, across chains, we fit densities on all chains but one and predict decisions from the held-out chain. We again predict above chance for every model and domain, with medians of $\approx$ 0.55--0.70 (Figure 16). 26/32 of tests against random chance are significant at $p < .05$
Our $\hat{R}$ values are similar to those of GSP [1]. In GSP study 4a, median $\hat{R}$ across sliders (per our reanalysis from their data) ranges from 1.07--1.23 across targets, and the max $\hat{R}$ is 2.41. Between 4 and 8 of their 10 sliders exceed 1.10 per target. Across their other face studies, $\hat{R}$ goes up to 5.78. On our face domain, median $\hat{R}$ ranges from 1.11--1.33 across models, and worst $\hat{R}$ ranges from 1.42--2.79 (as per Table 2)
Our HSL experiment shows how chains can recover correct distributions even under suboptimal $\hat{R}$. In this experiment, we know the correct answer for every target and we clearly recover the correct color for all 8 targets (Figure 17). Yet on saturation and lightness, which are bounded and non-circular like our sliders, $\hat{R}$ goes up to 1.70 for lavender, 1.80 for eggshell, etc. These values cover most of the range we report in our main experiments, where median $\hat{R}$ goes from 1.02--1.60 (as per Table 2)
Importantly, convergence issues appear in only some conditions. We acknowledge it lowers precision there, and include these values precisely for this reason (i.e. transparency). GPT-5.4 and Claude Sonnet 4.6 have median split $\hat{R}$ of 1.02--1.20 across all 4 domains, and at most 1% of their parameters exceed 1.5. We therefore draw our main conclusions from effects that replicate across independent chains and models. We would thus argue that it is misleading to interpret the least converged conditions as being representative of our full evaluation
(W2 and Q2) Insufficient comparison with explicit baselines.
The inability of independent, one-dimensional elicitation procedures to capture joint dependencies is precisely one of the motivations for our method, as detailed in Appendix C.
Appendix C compares our method against two widely used explicit elicitation strategies: (1) scalar ratings and (2) fixed pairwise comparisons. These test whether simpler, nonadaptive evaluations of individual sliders recover similar structure as adaptively sampling joint preferences. We show that this is not true, which motivates a method like ours.
Consider alternate ways of approximating a posterior over $d=10$ sliders (this is the number of generations/queries). #1 and #2 would be interaction-aware in the sense that you suggest:
| Method | Complexity | Evaluations per model–target |
|---|---|---|
| 10-level factorial | $O(10^d)$ | $10^{10}$ = 10 Billion |
| 5-level factorial | $O(5^d)$ | $5^{10}$ = 9.8 Million |
| Uniform random sampling | $O(N)$ | $N$ |
| Independent slider sweeps | $O(dm)$ | $10 \times 10 = 100$ |
| Ours: MH-within-Gibbs | $O(T)$ | $10 \times 2,000 = 20,000$ |
Factorial approaches preserve interactions but scale exponentially, and independent sweeps are cheap but discard interactions. Random sampling is flexible depending on budget $N$, and thus is equivalent to (joint) IMH-without-Gibbs and less efficient unless the posteriors are near-uniform. Our method preserves interactions while making the cost linear in the number of sampling iterations.
Furthermore, Appendix C compares much more than distribution sharpness. It measures magnitude and ranking disagreements between methods, investigates relationships with inter-slider dependencies, and reports association between interaction strength and baseline disagreement
RE: sharper distributions are not necessarily more accurate; We explicitly make this point ourselves in Appendix C, where we caution that posterior concentration can be misleading when an elicitation procedure is biased or misspecified, and we do not claim ground-truth accuracy.
(W3 and Q3 and Q4) Potential biases in slider selection and labeling.
Neither of these points establishes a failure of our method.
Slider selection. In Section 3.2, we mention that these 10 sliders are selected from 64 candidates. These are the interpreters' degrees of freedom. Selecting a tractable, interpretable subset is also necessary for the experimental protocol, since sampling all 64 dimensions for every model and target is computationally unfeasible.
Crucially, this selection occurs before the downstream target concepts are evaluated and is independent of their labels and resulting preferences. We claim nowhere to have exhaustive coverage of visual features or invariance to arbitrary slider subsets. The fact that alternative subsets might reveal additional associations or alter inferred distributions is in fact a desirable feature of our method, since it supports diverse analysis goals without methodological alterations.
RE: automatic labeling: A slider being mislabeled would require correcting the semantic interpretation of its results, but would not retrospectively change anything in the sampling procedure (e.g. models’ preferences, etc.). Therefore, validating (or not) the auto-interpreted labels does not affect the underlying correctness of our method. Validating against human judgements could make the labels more reliable, but it’s not strictly necessary.
Importantly, the semantic labels are post-hoc interpretations of the learned coordinates and not inputs to the MLLM acceptance decisions.
### (W4 and Q5) Limited surprise of the recovered priors
While we agree that some of these facial associations have been reported previously, a new method recovering familiar associations is in fact strong validation (rather than a weakness). If our method could not recover such established effects, this would conceivably raise questions about construct validity. Even so, the methodological contribution does not require that all priors be unprecedented.
We will update the manuscript to cite the papers you listed as evidence that our method recovers known priors. Even in cases where we recover known priors, our method also provides joint distributions over multiple visual features, which conventional bias evaluations do not.
However, we do show new and surprising associations which invite further investigation. For example, in Criminal, “Receding Hairline” is robustly the leading slider. In humans, criminal stereotypes traditionally involve long, dark, shaggy hair [2]. Human studies of male pattern baldness that we can find appear to disagree, e.g. balding men rated as less aggressive [3]. “Receding Hairline” also is in the bottom three for Trustworthy, Hardworking, and Fun. Humans do traditionally penalize balding men on attractiveness [per 3] though, and our results reproduce this effect.
### (W5) Limited cross-modal validation
Our proposed method requires two components:
- A controllable generator with interpretable sliders
- A judge capable of providing binary comparisons between generated stimuli.
Nothing depends on the stimuli being images, and multimodal judges are widely available. So, using this method with a different modality just requires a controllable generator with interpretable sliders. For example, we could replace SliderSpace with an equivalent controllable method (e.g., AudioGen as an audio model, with SAEs as controls).
We do not claim to have empirically demonstrated this method across audio and other modalities. We invite the community to explore applications of our method in other modalities and problems leveraging their domain expertise.
### (W6) Limited quantitative evidence regarding direct slider manipulation
The direct slider manipulation (if it worked) would be equivalent to our method, and thus, the quantitative comparison you suggest would have no effect on the method proposed in this paper. If direct slider manipulation had worked, it would still be an MCMC sampling method using a generative model, MLLM judge, and interpretable controls. The only difference would be that we could avoid sampling over the slider via MH, since the judge would be doing that part as well. We had to work around that issue to make a method like this one possible.
We respectfully disagree with this review, and we discuss below in detail why. To summarize the main points:
[1](we recomputed from their faces data). We discuss below how our HSL experiments also add internal validity. We report $\hat{R}$s for transparency