- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the confidence score of a multimodal model can both reflect the accuracy of answers and be associated with explicit visual evidence. TraceConf first employs structured SFT to enable the model to output reasoning, evidence regions, answers, and confidence scores. Subsequently, it utilizes a frozen validator to construct evidence gating mechanisms under four experimental conditions: full image, blurred evidence regions, only blurred evidence regions, and fully blurred image. This gating mechanism is incorporated into both the GRPO reward for answering and the local confidence loss function; however, the confidence selection tokens do not receive shared sequence-level advantage updates. Results across six benchmarks demonstrate that TraceConf achieves superior macro-average ECE and AUROC scores compared to the compared methods, and the study also presents visual intervention experiments along with additional results on an 8B model.
Critical strengths
- The research question is well-defined, and the methodology exhibits a substantive alignment with this question. Relying solely on answer correctness calibration would indeed fail to distinguish between responses that benefit from sufficient visual support and those that primarily depend on linguistic prior knowledge. In this work, we require the model to declare evidence, followed by interventions to remove or retain such evidence, thereby establishing a testable interface for this objective. By excluding confidence tokens from the shared GRPO objective and employing an independent proper loss function, we also specifically address the challenge of conflating answer rewards with confidence supervision.
- The enhanced ability to differentiate confidence levels demonstrates strong cross-task consistency. In Table 1, TraceConf achieves the highest AUROC across all six benchmarks; its macro-average AUROC increases from the strongest baseline's 0.7489 to 0.7982, while its macro-average ECE decreases from 0.0951 to 0.0840. Appendix C additionally reports complete training results on another backbone network. These results substantiate the practical effectiveness of this approach, although the 8B model experiments cannot yet replace comparisons against the same backbone baselines.
Critical weaknesses
- The visual traceability experiment still suffers from confounding factors. In Appendix A.2, the declaration region and its complement were not area-matched; consequently, the differences shown in Figure 4 may be influenced by the amount of retained information and the degree of ambiguity in the ranges. Furthermore, the experiment fixed the reasoning and answer prefix while measuring only changes in confidence, which is insufficient to establish evidence of fidelity across the entire answering process; Table 2 also restricted the interpretability of cross-method comparisons by using the respective "correct" / "correct" subsets for each method.
- Ablation studies failed to adequately isolate the contributions of key components. The un gated variant in Table 3 simultaneously altered both the sequence reward and the local confidence loss, and did not separately evaluate the role of confidence token masking. The AUROC and Support Gap of this un gated variant approached those of the full method. Although the full method significantly improved ECE, the current results are still insufficient to determine which specific component contributes to this improvement, or whether each design element was necessary. 3.The reliability of the gating mechanism and its performance on weak-evidence samples remain unverified. The gating relies on a single frozen verifier, yet the paper does not provide manual consistency checks or robustness experiments involving alternative verifiers. Since gating simultaneously affects both the sequence reward and confidence supervision, verifier bias could potentially impact both training objectives. Particularly when W=0, local confidence supervision is entirely absent; however, the paper does not report the proportion of such samples or their calibration performance, making it unclear whether the method can effectively mitigate overconfidence in weak-evidence scenarios.
Questions for authors
- Under random box or position-shuffled controls that match the number and area of regions, does the region still exhibit a larger Necessity Drop and a smaller Sufficiency Gap? Table 2 reports the number of valid samples for each method, and whether the conclusions hold after applying a shared sample set or stratification based on the benchmark.
- Can we separately report: (1) using evidence gating exclusively in sequence rewards; (2) using gating exclusively for local confidence loss; and (3) using matching controls with confidence tokens disabled? These results will determine whether the independent methodological contributions of this paper are supported.
- What backbone networks, training datasets, and inference budgets were specifically used for each training-based baseline in Table 1? The Appendix provides detailed information on TraceConf. However, this alone is insufficient to establish that the training and computational conditions across the comparisons are comparable.
2 comments
Thank you for your detailed response and for providing additional experiments. I will raise my score from 2 to 3.
Q1: Thank you for this suggestion. To test region specificity, we added random-region controls matched in number and total area, and position-shifted controls preserving box shape and area. Following the Figure 4 protocol, we fixed the original reasoning, declared evidence, and answer, changing only the image condition and rereading confidence. Across 265 originally correct outputs, blurring the declared regions produces a larger confidence drop (higher NecDrop) than either control, while keeping only the declared regions clear better preserves full-image confidence (lower SufGap). All four paired advantages have 95% confidence intervals excluding zero, supporting location-specific confidence dependence rather than an effect explained solely by region area.
All methods use the same pool of 378 input pairs and independently retain pairs answered correctly under both the original and blurred image conditions. CoT, TopK, ConfTuner, RewardingDoubt, VL-Calibration, and TraceConf retain 109, 129, 116, 104, 109, and 115 pairs, respectively. The shared subset contains 49 pairs spanning all six benchmarks. On this subset, TraceConf achieves the highest Pairwise Preference and lowest HCM, with a Support Gap close to that of VL-Calibration. Thus, its more consistent confidence response to reduced visual support persists on identical samples, as shown below.
Q2: Thank you for this suggestion. We added three ablations while holding the SFT checkpoint, RL data, rollout count, training steps, and evaluation protocol fixed: evidence gating only in sequence rewards (reward-only), evidence gating only in the local confidence loss (conf-loss-only), and removing the shared-advantage mask for the confidence token (no-token-mask). The third ablation retains the confidence output and local loss, changing only whether the confidence token participates in the shared sequence-advantage update to test the masking design.
The results show that evidence weighting of the local loss improves calibration and confidence responses to changes in visual support; the sequence-level evidence reward strengthens confidence dependence on the declared regions; and confidence-token masking further improves calibration and confidence drops when the declared regions are blurred (NecDrop). These components provide complementary benefits. Full TraceConf achieves the best overall performance, supporting the combined design. Detailed results are reported below.
Q3: Thank you for this suggestion. All training-based methods in Table 1 use Qwen3.5-4B. ConfTuner uses the same 5,368 source examples as TraceConf’s SFT stage, drawn from the VL-Calibration-12K training split of ViRL-39K. RewardingDoubt and VL-Calibration use the same 3,000 training prompts as TraceConf’s RL stage, drawn from MMFineReason-Visual-Verifiable-v2-10k. At inference, the maximum output and context lengths are 16,384 and 32,768 tokens, respectively, with a thinking-token budget capped at 12,288. We retain each method’s native prompts and confidence readout.