- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper uses Bayesian linear classifiers on pre-generation hidden states to forecast distributions over visualization choices. It decomposes predictive uncertainty into aleatoric and epistemic terms. Across three LLMs, forecasts are stronger for broad design choices than field bindings, and epistemic uncertainty correlates with distributional error on several attributes.
Critical strengths
1. A useful focus on output distributions. Predicting distributions is appropriate when several chart designs are plausible. Distinguishing output variability from uncertainty in the forecaster is also a useful goal.
2. Substantial evaluation with useful controls. The reading/design/binding split exposes clear differences in difficulty. Source datasets are disjoint across splits. The supplement adds MLP and feature baselines, training-budget comparisons, and an exploratory comparison with human chart-effectiveness data.
Critical weaknesses
1. The added value of Bayesian uncertainty needs a clearer test. The MLP comparison changes model capacity, so it does not isolate the contribution of Bayesian inference. A matched linear baseline using maximum a posteriori (MAP) baseline and a direct comparison of epistemic uncertainty with predictive entropy would test whether the decomposition helps identify unreliable forecasts.
2. Some uncertainty interpretations need qualification. The paper acknowledges readout limitations, but the entropy gap alone does not separate missing representation information from limitations of the learned forecaster. The paper should present hidden-state information loss as a possible explanation for this gap throughout, as it does in the conclusion. Fig. 3(a) attributes entropy reduction from a uniform reference to hidden-state information, although class imbalance alone can reduce entropy. Class-frequency baselines appear elsewhere, but this interpretation still needs correction.
3. Inference details and computational costs are underreported. Appendix C omits posterior sample counts, warm-up, chain counts, and convergence diagnostics. This makes the uncertainty estimates harder to reproduce and assess. Reporting offline fitting costs separately from online prediction latency would also make the efficiency claim more concrete.
4. Reference comparisons would benefit from robustness checks. The supplement discloses filtering, uses entropy correction, and reports similar readout gaps on matched evaluation prompts. However, F1 scores and ceilings use different populations, with only 10-19 reference draws per eligible prompt. A larger-sample check using matched prompts and comparable decision rules would strengthen the evidence. Filtering also narrows the population covered by these results. This is a scope limitation, not a demonstrated source of error.
Questions for authors
1. What additional capability does the decomposition provide? Given prior work on behavior prediction, distribution recovery, and uncertainty probes [1-4], what added capability does this decomposition provide? Which results in the visualization setting establish that capability beyond what matched point-estimate probes and predictive entropy can provide?
2. How stable are the uncertainty estimates and reference comparisons? Please provide the posterior-sampling settings and convergence diagnostics. How sensitive are the readout gaps and F1 ceiling comparisons to the number of reference generations and to using matched evaluation prompts?
References
[1] Dong et al. Emergent Response Planning in LLMs. ICML, 2025.
[2] Ashok et al. Language Models Can Predict Their Own Behavior. NeurIPS, 2025.
[3] Piskorz et al. Eliciting Numerical Predictive Distributions of LLMs Without Auto-Regression. ICLR, 2026.
[4] Kossen et al. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs. ICML FM-Wild Workshop, 2024.
Response to the Authors
Thank the authors for the clarifications. The response makes the intended role of epistemic uncertainty clearer. However, the following key concerns remain unresolved:
The main contents and appendix I have currently downloaded is identical to the initial versions, so I could not verify the proposed additions. Therefore, I tend to maintain my score.