- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Fisher–Rao Decision Geometry (FRDG), a training-free, white-box monitor that assigns risk scores to completed VLM responses. It combines relative-state local predictive sensitivity, summarized by the leading eigenvalue of a state Fisher operator, with late-layer trajectory excess, defined as accumulated predictive distance minus the endpoint distance. Components are aggregated over answer positions and standardized using nominal reference responses. The paper derives a local perturbation interpretation and trajectory properties, presents a marginal nominal conformal alarm guarantee, and evaluates failure ranking on four VLMs and nine benchmark subsets. Reported macro AUROC/AP improve over the compared monitors, and ablations support combining the two measurements.
Critical strengths
- The local score has a precise interpretation under a specified relative state-perturbation budget. This is a coherent measurement distinct from evidential conflict or ignorance. Combining it with trajectory excess gives an interpretable monitor of completed answers without fitting a detector or generating additional answers.
- The reported gains merit attention: macro AUROC improves by 4.67–9.30 percentage points over the strongest baseline in the main macro comparison (Appendix D.1.2) across four VLMs, with trajectory excess adding 0.97–1.39 points over local sensitivity alone. The paper supplies extensive component and substitution controls, distinguishes nominal alarm guarantees from failure detection, and discloses variants outperforming the proposed construction. For the layer/position ablations, Appendix D.4.1 states that the main configuration was retained rather than replaced based on the test comparisons.
Critical weaknesses
-
The methodological increment beyond existing internal-uncertainty monitoring remains limited. The local perturbation interpretation differs from EUQ, but the experiments support a useful family of sensitivity-plus-trajectory scores more strongly than the particular Fisher–Rao choices. In Table A40, replacing only the trajectory distance with L2 or JS matches or slightly improves macro AUROC on all four models; these controls retain the local Fisher component. The leading eigenvalue has no consistent AUROC advantage over the trace, although it achieves higher AP, so these alternatives are not equivalent across metrics. Several simpler aggregation variants also outperform the proposed combination. These observations do not invalidate the method, but weaken the case for its specific design. The paper does not yet clearly establish which failures its features distinguish beyond EUQ-CF, EUQ-IG, and ordinary confidence; the exploratory stratifications in Appendix C.5 do not provide a joint or tightly controlled comparison.
-
The evaluation history and configuration behind parts of the central evidence remain unresolved. Appendix B.2 gives incompatible InternVL3 records: a 28-block decoder with expected indices 17/21/25/28 versus exported indices 19/24/29/32. The actual configuration producing that model’s scores is therefore unclear. Appendix D.2.2 also states that the reported intervals do not account for outcome-based selection recorded in the companion artifact and must not be interpreted as independent validation of a selected configuration. This needs reconciliation with the statements in Section 3.4/Appendix B.6 that test responses and behavior labels do not determine score parameters. Retaining the main configuration despite better reported variants is reassuring, but does not identify the scope or data used for the acknowledged selection. These are questions about experimental traceability and evaluation independence, not evidence of fabrication or proof that all results are invalid. Until resolved, they limit how strongly I can rely on the promising empirical improvements.
Questions for authors
- Which configurations were selected using outcomes, on which data partitions, and at what stage? Please reconcile Appendix D.2.2 with Section 3.4/Appendix B.6 and identify which main results and ablations were evaluated on data untouched by that selection.
- What exact InternVL3 checkpoint, decoder depth, and block indices generated the reported scores, and can the affected results be confirmed after reconciling the records? Separately, please clarify whether Table 3’s spectral checks use the actual full-dimensional, full-vocabulary model operators and how these relate to the metadata for local checks in Appendix D.3.2.
- The manuscript includes a custom CF+IG control . Beyond this aggregate comparison, can the recorded scores establish additional failure information conditional on EUQ-CF, EUQ-IG, and output confidence? A joint or matched analysis with an evaluation protocol fixed independently would help assess whether the contribution extends beyond improved aggregate ranking. This is a request for evidence supporting the current contribution, rather than a suggestion for a new application.