- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper addresses the instance-level reliability of VLMs in quantitative physical reasoning, where ground-truth values are unavailable at deployment and standard self-consistency fails due to "stable-but-wrong" predictions and over-reliance on textual numerical priors. To overcome these limitations, the authors propose answer-level trust selection (ATS), a training-free, verifier-free, black-box selective prediction framework that operates without access to internal model probabilities. Using controlled probing around an initial prediction, ATS computes eight behavioral diagnostic signals and aggregates them against a validation-calibrated threshold to selectively accept or reject outputs. Evaluated across 20 VLMs on QuantiPhy, ATS demonstrates its effectiveness in filtering out unreliable numerical predictions and mitigating both stable-but-wrong and prior-tracking failure modes.
Critical strengths
-
The paper formulates a well-motivated and useful problem by shifting the evaluation paradigm from aggregate benchmark performance to instance-level selective prediction in quantitative physical reasoning. Because ground-truth measurements are unavailable at deployment, determining when to trust or abstain from an individual numerical output is essential for safety-critical downstream applications. Furthermore, the core premise of actively probing models via controlled input interventions offers a compelling approach to uncover behavioral vulnerabilities that passive repeated sampling inherently misses.
-
The proposed ATS framework features a portable, model-agnostic black-box architecture. By relying purely on inference-time input perturbations and behavioral consistency checks, ATS can be seamlessly applied across diverse VLM architectures.
-
The empirical evaluation is extensive, spanning 20 VLMs across both proprietary LLMs and open-weight families of varying scales. In addition to benchmarking against standard self-consistency, generic uncertainty quantification baselines, and oracle rankings, the authors provide diagnostic ablations, weight-sensitivity analyses, and paired cross-model comparisons that validate the contributions of the different ATS variants.
Critical weaknesses
-
On standard selective prediction metrics, ATS fails to demonstrate a consistent advantage over simpler baselines. On the primary Qwen2.5-VL-7B backbone, both Prior-weighted and Learned ATS underperform either SC-only or Generic UQ rank fusion across MRA@30%, MRA@50%, and AURC. Across all 20 evaluated VLMs, the average gains are marginal. Ablating the signature counterfactual diagnostic actually improves cross-model performance, raising average MRA@50% from 0.4525 to 0.4646 and reducing AURC from 0.5488 to 0.5433. The authors need to substantiate that behavioral probing yields genuine overall risk reduction and clarify when ATS is preferable as a general-purpose selective predictor.
-
The paper's headline results on targeted error subsets ("stable-but-wrong" and "prior-tracking") suffer from severe definitional coupling between the evaluation slices and the selection criteria. Specifically, "stable-but-wrong" instances are defined by combining low ground-truth accuracy with high repeated-sampling stability, making SC-only's dismal 1.3% rejection rate on this subset a tautological artifact of slice construction rather than a fair reflection of its general error-detection capability. Similar problem can be observed in terms of "prior-tracking" subset. While these analyses confirm that ATS responds as engineered to its constituent features, they offer limited independent validation of selective prediction performance.
-
The ostensibly impressive error rejection rates achieved by ATS on targeted subsets come at the cost of severe false rejections of correct predictions, calling into question the framework's practical utility. For instance, while Prior-weighted ATS rejects 95.8% of erroneous prior-tracking predictions, it retains only 11.1% of correct prior-tracking predictions. In contrast, Learned Generic UQ achieves a much more balanced operating point, rejecting 63.8% of errors while preserving 52.3% of correct predictions. Without evaluating selection utility under matched operating points, it remains unclear whether ATS's aggressive filtering provides a net benefit over indiscriminate abstention in practical deployments.
-
The statistical reliability of the empirical conclusions is constrained by the small sample size of the evaluation benchmark. Each of the 20 VLMs is evaluated on 159 questions from the QuantiPhy validation split; while the authors employ 15 random video-level splits to assess partitioning sensitivity, resampling a small fixed pool cannot substitute for an independent, large-scale test set. Compounded by unweighted macro-averaging across models regardless of subset size, the reported rates carry substantial unquantified variance. The authors might validate their claims on a larger benchmark.
Questions for authors
See weaknesses