acceptodds
Under review as a conference paper at ICLR 2027

TraceConf: Visually Traceable and Calibrated Confidence in Multimodal Large Language Models

Abstract

As multimodal large language models are increasingly used, confidence estimates are becoming central to assessing the reliability of their predictions. Existing calibration objectives, however, primarily ask whether an answer is likely to be correct, without revealing which visual evidence warrants that confidence. This limitation arises because binary correctness assigns the same calibration target to predictions with different evidential bases, allowing confidence to be statistically calibrated while remaining visually untraceable. We introduce TraceConf, a post-training approach that jointly generates declared visual evidence, an answer, and a confidence choice. TraceConf counterfactually assesses the necessity and sufficiency of the declared regions by removing and isolating them. The resulting evidence gate rewards evidence-grounded response generation and weights a local proper confidence loss, while the confidence-choice token is excluded from the shared sequence-level GRPO advantage to prevent response-level rewards from misassigning credit to confidence. Because the gate is determined before the confidence report, evidence weighting preserves the proper conditional target wherever the gate is positive. Across six multimodal benchmarks, TraceConf improves confidence calibration on average and error discrimination across all benchmarks while maintaining competitive answer accuracy. Further experiments establish a direct link between model confidence and its declared visual evidence: removing the declared regions causes larger confidence drops, while retaining only these regions better preserves confidence than retaining their complementary regions. Together, these results demonstrate that visually traceable confidence can enhance multimodal reliability without compromising answer quality.

Primary area:
interpretability and explainable AI
Keywords:
MLLM,confidence calibration,Visual Evidence, Uncertainty Estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ieUMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the confidence score of a multimodal model can both reflect the accuracy of answers and be associated with explicit visual evidence. TraceConf first employs structured SFT to enable the model to output reasoning, evidence regions, answers, and confidence scores. Subsequently, it utilizes a frozen validator to construct evidence gating mechanisms under four experimental conditions: full image, blurred evidence regions, only blurred evidence regions, and fully blurred image. This gating mechanism is incorporated into both the GRPO reward for answering and the local confidence loss function; however, the confidence selection tokens do not receive shared sequence-level advantage updates. Results across six benchmarks demonstrate that TraceConf achieves superior macro-average ECE and AUROC scores compared to the compared methods, and the study also presents visual intervention experiments along with additional results on an 8B model.

Critical strengths
  1. The research question is well-defined, and the methodology exhibits a substantive alignment with this question. Relying solely on answer correctness calibration would indeed fail to distinguish between responses that benefit from sufficient visual support and those that primarily depend on linguistic prior knowledge. In this work, we require the model to declare evidence, followed by interventions to remove or retain such evidence, thereby establishing a testable interface for this objective. By excluding confidence tokens from the shared GRPO objective and employing an independent proper loss function, we also specifically address the challenge of conflating answer rewards with confidence supervision.
  2. The enhanced ability to differentiate confidence levels demonstrates strong cross-task consistency. In Table 1, TraceConf achieves the highest AUROC across all six benchmarks; its macro-average AUROC increases from the strongest baseline's 0.7489 to 0.7982, while its macro-average ECE decreases from 0.0951 to 0.0840. Appendix C additionally reports complete training results on another backbone network. These results substantiate the practical effectiveness of this approach, although the 8B model experiments cannot yet replace comparisons against the same backbone baselines.
Critical weaknesses
  1. The visual traceability experiment still suffers from confounding factors. In Appendix A.2, the declaration region and its complement were not area-matched; consequently, the differences shown in Figure 4 may be influenced by the amount of retained information and the degree of ambiguity in the ranges. Furthermore, the experiment fixed the reasoning and answer prefix while measuring only changes in confidence, which is insufficient to establish evidence of fidelity across the entire answering process; Table 2 also restricted the interpretability of cross-method comparisons by using the respective "correct" / "correct" subsets for each method.
  2. Ablation studies failed to adequately isolate the contributions of key components. The un gated variant in Table 3 simultaneously altered both the sequence reward and the local confidence loss, and did not separately evaluate the role of confidence token masking. The AUROC and Support Gap of this un gated variant approached those of the full method. Although the full method significantly improved ECE, the current results are still insufficient to determine which specific component contributes to this improvement, or whether each design element was necessary. 3.The reliability of the gating mechanism and its performance on weak-evidence samples remain unverified. The gating relies on a single frozen verifier, yet the paper does not provide manual consistency checks or robustness experiments involving alternative verifiers. Since gating simultaneously affects both the sequence reward and confidence supervision, verifier bias could potentially impact both training objectives. Particularly when W=0, local confidence supervision is entirely absent; however, the paper does not report the proportion of such samples or their calibration performance, making it unclear whether the method can effectively mitigate overconfidence in weak-evidence scenarios.
Questions for authors
  1. Under random box or position-shuffled controls that match the number and area of regions, does the region still exhibit a larger Necessity Drop and a smaller Sufficiency Gap? Table 2 reports the number of valid samples for each method, and whether the conclusions hold after applying a shared sample set or stratification based on the benchmark.
  2. Can we separately report: (1) using evidence gating exclusively in sequence rewards; (2) using gating exclusively for local confidence loss; and (3) using matching controls with confidence tokens disabled? These results will determine whether the independent methodological contributions of this paper are supported.
  3. What backbone networks, training datasets, and inference budgets were specifically used for each training-based baseline in Table 1? The Appendix provides detailed information on TraceConf. However, this alone is insufficient to establish that the training and computational conditions across the comparisons are comparable.
2 comments

Q1: Thank you for this suggestion. To test region specificity, we added random-region controls matched in number and total area, and position-shifted controls preserving box shape and area. Following the Figure 4 protocol, we fixed the original reasoning, declared evidence, and answer, changing only the image condition and rereading confidence. Across 265 originally correct outputs, blurring the declared regions produces a larger confidence drop (higher NecDrop) than either control, while keeping only the declared regions clear better preserves full-image confidence (lower SufGap). All four paired advantages have 95% confidence intervals excluding zero, supporting location-specific confidence dependence rather than an effect explained solely by region area.

ControlMetricDeclared regionsControl regionsPaired advantage [95% CI]
Shifted regionsNecDrop ↑11.946.885.06 [4.40, 6.34]
Shifted regionsSufGap ↓3.288.975.69 [4.47, 6.11]
Random regionsNecDrop ↑11.946.605.34 [3.64, 5.71]
Random regionsSufGap ↓3.289.416.13 [5.37, 7.09]

All methods use the same pool of 378 input pairs and independently retain pairs answered correctly under both the original and blurred image conditions. CoT, TopK, ConfTuner, RewardingDoubt, VL-Calibration, and TraceConf retain 109, 129, 116, 104, 109, and 115 pairs, respectively. The shared subset contains 49 pairs spanning all six benchmarks. On this subset, TraceConf achieves the highest Pairwise Preference and lowest HCM, with a Support Gap close to that of VL-Calibration. Thus, its more consistent confidence response to reduced visual support persists on identical samples, as shown below.

MethodSupport Gap ↑ [95% CI] (pp)Pairwise Preference ↑HCM ↓
CoT1.53 [0.51, 2.65]0.28571.0000
TopK2.61 [−0.35, 5.92]0.22450.9388
ConfTuner5.90 [2.95, 9.07]0.36730.8163
RewardingDoubt8.98 [4.08, 14.90]0.36730.8367
VL-Calibration13.11 [7.65, 19.27]0.63270.7959
TraceConf13.54 [8.27, 19.42]0.83670.5918

Q2: Thank you for this suggestion. We added three ablations while holding the SFT checkpoint, RL data, rollout count, training steps, and evaluation protocol fixed: evidence gating only in sequence rewards (reward-only), evidence gating only in the local confidence loss (conf-loss-only), and removing the shared-advantage mask for the confidence token (no-token-mask). The third ablation retains the confidence output and local loss, changing only whether the confidence token participates in the shared sequence-advantage update to test the masking design.

The results show that evidence weighting of the local loss improves calibration and confidence responses to changes in visual support; the sequence-level evidence reward strengthens confidence dependence on the declared regions; and confidence-token masking further improves calibration and confidence drops when the declared regions are blurred (NecDrop). These components provide complementary benefits. Full TraceConf achieves the best overall performance, supporting the combined design. Detailed results are reported below.

Training variantACC ↑ECE10 ↓AUROC ↑Support Gap ↑ (pp)NecDrop ↑ (pp)SufGap ↓ (pp)
SFT only0.70180.14860.765317.9210.354.64
W=10.71260.12430.796020.598.964.65
Reward-only0.70210.11300.795516.8611.223.32
Conf-loss-only0.69640.09190.789220.0410.374.95
No-token-mask0.70490.09640.793419.7310.133.29
Full TraceConf0.73380.08400.798220.6711.943.28

Q3: Thank you for this suggestion. All training-based methods in Table 1 use Qwen3.5-4B. ConfTuner uses the same 5,368 source examples as TraceConf’s SFT stage, drawn from the VL-Calibration-12K training split of ViRL-39K. RewardingDoubt and VL-Calibration use the same 3,000 training prompts as TraceConf’s RL stage, drawn from MMFineReason-Visual-Verifiable-v2-10k. At inference, the maximum output and context lengths are 16,384 and 32,768 tokens, respectively, with a thinking-token budget capped at 12,288. We retain each method’s native prompts and confidence readout.

Thank you for your detailed response and for providing additional experiments. I will raise my score from 2 to 3.

Reviewer_FAC9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TraceConf, a post-training framework that connects answer confidence to explicitly declared visual evidence. A structured SFT stage teaches the model to produce reasoning, evidence boxes, an answer, and a binary confidence choice, with continuous confidence obtained from the final yes/no logits. During reinforcement learning, a frozen verifier assesses the declared regions through removal and retention interventions. The resulting evidence gate contributes to the response reward and weights a local logarithmic confidence loss. The confidence-choice token is excluded from the sequence-level GRPO advantage. The paper establishes that positive, context-dependent gating preserves the conditional optimum of the local proper loss, and evaluates calibration, error discrimination, and visual-evidence dependence on six benchmarks, with an additional 8B experiment.

Critical strengths
  1. The paper makes a useful distinction between calibrated correctness and inspectable visual support, and implements it coherently. Requiring an answer to identify evidence that can be removed or retained gives the confidence claim a concrete, testable visual basis. The separation between sequence-level response optimization and local confidence supervision is also well motivated: it prevents the confidence token from directly inheriting rewards intended for answer quality or formatting. Importantly, Equation (11) and Appendix D state the properness result at the appropriate conditional level and explicitly acknowledge both zero-gate contexts and the limits of the result for joint neural optimization. This careful formulation strengthens the conceptual contribution.

  2. The empirical results provide encouraging evidence for both confidence quality and evidence dependence. In Table 1, TraceConf has the highest AUROC on all six benchmarks, with macro-average AUROC increasing from the strongest baseline's 0.7489 to 0.7982, and the best macro-average ECE of 0.0840. Tables 2–3 and Figure 4 go beyond answer accuracy by examining confidence responses to visual interventions and the contribution of evidence gating. Appendix A provides substantial implementation detail, including separate construction and online-verification procedures, and Appendix C reports a second backbone. Although the controls below would strengthen the conclusions, the combination of consistent discrimination gains and a testable evidence interface supports a weak acceptance.

Critical weaknesses

W1. The visual-intervention experiments support evidence dependence, but do not yet isolate its specificity sufficiently.

  • Table 2 evaluates each method on its own subset of examples that remain correct under both image conditions. This controls correctness within a method, but the compared methods can retain different questions and subset sizes. Consequently, their Support Gap, Pairwise Preference, and High-confidence Masked Rate are not necessarily measured on comparable populations. Reporting the retained counts and a common-cohort analysis would make the cross-method comparison more persuasive.
  • Figure 4 compares declared regions with their complements. Appendix A.2 explicitly states that these regions are not area matched and that confidence is evaluated with the original reasoning, evidence, and answer held fixed. This is a useful test of confidence dependence conditional on that prefix, but spatial extent and the interaction between the retained textual prefix and the altered image remain potential confounders. Area-matched random or displaced regions would better establish that the effect is specific to the declared evidence locations.
  • The principal declared-region plot uses 265 originally correct outputs. The behavior on incorrect answers is especially relevant to the reliability motivation, yet is not quantified in the same analysis. These limitations narrow the interpretation of the traceability evidence; they do not invalidate the reported paired effects.

W2. The ablations do not fully identify which parts of the training design produce the gains, and the comparison protocol needs clarification.

  • In Table 3, setting the gate to one simultaneously removes its discriminative role in the sequence reward and its weighting of the local confidence loss. The experiment therefore does not isolate evidence-based response learning from evidence-weighted confidence learning. It also does not separately test excluding the confidence token from the shared advantage, a central design choice. The full gate improves ECE from 0.1243 to 0.0840, Necessity Drop from 8.96 to 11.94 percentage points, and Sufficiency Gap from 4.65 to 3.28 percentage points. AUROC and Support Gap change much less. These are meaningful benefits, but their attribution to the gate's two distinct roles remains unresolved.
  • As Appendix D correctly notes, a zero gate supplies no local confidence supervision. The prevalence of zero or very small gates, particularly among incorrect outputs, is not reported. The conditional properness result is valid, but does not resolve whether precisely the unsupported errors of interest receive enough confidence training in practice.
  • Section 5.1 specifies native confidence readouts and a common judging protocol, while Appendix A mainly details TraceConf. The exact baseline checkpoints, training-data access, and generation budgets should be documented together. Table 1 also shows nonuniform trade-offs: for example, DynaMath-2V ECE is 0.1209 versus 0.0638 for VL-Calibration, and macro-average accuracy is 0.7338 versus 0.7448 for ConfTuner. Uncertainty estimates and valid-confidence counts would help assess these differences. The 8B results are useful, but lack corresponding 8B baseline comparisons.
Questions for authors
  1. Comparable intervention cohorts: How many correct/correct pairs remain for each method in Table 2, and do the conclusions hold on their shared subset and when results are stratified by correctness transitions?

  2. Evidence specificity: Does the Figure 4 effect remain against area-matched random or displaced masks? What happens on the originally incorrect outputs excluded from the main plot?

  3. Component attribution: Can you separate the evidence reward, evidence weighting of the local loss, and exclusion of the confidence token from the sequence advantage while holding the training protocol fixed?

  4. Low-support errors: What fraction of correct and incorrect rollouts receive zero or near-zero evidence weight, and how well calibrated are those groups after training?

  5. Comparison protocol: Please specify the checkpoints, training data, inference budgets, and valid-confidence coverage for all Table 1 methods, together with uncertainty estimates for the main differences. Are matched baseline results available for the 8B experiment?

2 comments

Q1: Thank you for highlighting this comparability issue. Table 2 uses the same pool of 378 input pairs. Each method independently generates answers and confidence estimates and retains pairs answered correctly under both image conditions. CoT, TopK, ConfTuner, RewardingDoubt, VL-Calibration, and TraceConf retain 109, 129, 116, 104, 109, and 115 pairs, respectively. The intersection across all six methods contains 49 pairs, covering all six benchmarks. On these 49 shared pairs, TraceConf achieves the highest Pairwise Preference and the lowest High-confidence Masked Rate, with a Support Gap close to that of VL-Calibration. These results support its more consistent confidence reduction in response to weakened visual support on the same questions. We further stratify by correctness transitions: CC denotes correct answers under both conditions; CW denotes correct on the original image but incorrect after blurring; and WW denotes incorrect under both conditions. Within each group, all six methods are evaluated on the same shared samples. TraceConf’s mean confidence drop is positive in all three groups, with 95% confidence intervals excluding zero. Thus, this response occurs not only in the CC group, where correctness remains unchanged, but also in the CW and WW groups. The shared WC group (incorrect on the original image but correct after blurring) is empty and is therefore not reported. Detailed results are provided in the tables below.

MethodSupport Gap ↑ [95% CI] (pp)Pairwise Preference ↑HCM ↓
CoT1.53 [0.51, 2.65]0.28571.0000
TopK2.61 [−0.35, 5.92]0.22450.9388
ConfTuner5.90 [2.95, 9.07]0.36730.8163
RewardingDoubt8.98 [4.08, 14.90]0.36730.8367
VL-Calibration13.11 [7.65, 19.27]0.63270.7959
TraceConf13.54 [8.27, 19.42]0.83670.5918
MethodCC(n=49)CW(n=37)WW(n=11)
CoT1.53 [0.51, 2.65]11.22 [6.22, 17.84]8.64 [1.82, 16.82]
TopK2.61 [−0.35, 5.92]23.92 [16.35, 31.89]15.91 [7.27, 25.00]
ConfTuner5.90 [2.95, 9.07]39.64 [28.53, 51.35]29.29 [11.11, 50.51]
Rewarding Doubt8.98 [4.08, 14.90]43.51 [30.27, 57.03]37.27 [17.27, 58.18]
VL-Calib.13.11 [7.65, 19.27]26.70 [18.04, 35.74]17.47 [5.15, 31.21]
TraceConf13.54 [8.27, 19.42]33.96 [24.15, 43.74]27.02 [15.02, 40.52]

Q2: Thank you for this suggestion. To distinguish region-area effects from location specificity, we added random regions matched in count and total area, and displaced regions that preserve box shape and area while changing their locations within the image. All interventions hold the original reasoning, declared evidence, and answer fixed and re-read confidence under the altered image, avoiding confounding from regenerated content.

On the 265 originally correct outputs, blurring the declared regions causes a larger confidence drop (higher NecDrop), while keeping only those regions clear better preserves full-image confidence (lower SufGap), relative to both controls. All paired advantages are positive, with 95% confidence intervals excluding zero, supporting confidence dependence on the declared locations beyond region-area effects.

Control typeMetricDeclared regionsControl regionsPaired advantage [95% CI]
Displaced regionsNecDrop ↑11.946.885.06 [4.40, 6.34]
Displaced regionsSufGap ↓3.288.975.69 [4.47, 6.11]
Random regionsNecDrop ↑11.946.605.34 [3.64, 5.71]
Random regionsSufGap ↓3.289.416.13 [5.37, 7.09]

We applied the same analysis to the 119 originally incorrect outputs excluded from the main plot. The declared regions again yield higher NecDrop and lower SufGap, with all paired-advantage confidence intervals excluding zero. Thus, location-specific confidence dependence is also present in incorrect outputs. Detailed results are reported below.

Control typeMetricDeclared regionsControl regionsPaired advantage [95% CI]
Displaced regionsNecDrop ↑6.735.501.22 [0.48, 2.04]
Displaced regionsSufGap ↓3.996.072.08 [1.21, 3.04]
Random regionsNecDrop ↑6.754.991.75 [0.81, 2.80]
Random regionsSufGap ↓3.986.642.66 [1.59, 3.82]

Q3: Thank you for this suggestion. We added reward-only gating, conf-loss-only gating, and no-token-mask ablations to isolate the three design choices, holding the SFT checkpoint, RL data, rollout count, training steps, and evaluation protocol fixed.

The results show that evidence weighting of the local loss improves calibration and confidence sensitivity to visual support; the sequence-level evidence reward strengthens confidence dependence on the declared regions; and excluding the confidence token from the shared sequence advantage further improves calibration and confidence drops when the declared regions are blurred (NecDrop). These components provide complementary benefits, with full TraceConf achieving the best overall performance, supporting the combined design. Detailed results are reported below.

Training variantACC ↑ECE10 ↓AUROC ↑Support Gap ↑ (pp)NecDrop ↑ (pp)SufGap ↓ (pp)
SFT only0.70180.14860.765317.9210.354.64
W=10.71260.12430.796020.598.964.65
Reward-only0.70210.11300.795516.8611.223.32
Conf-loss-only0.69640.09190.789220.0410.374.95
No-token-mask0.70490.09640.793419.7310.133.29
Full TraceConf0.73380.08400.798220.6711.943.28

Q4: Thank you for this suggestion. We report the evidence-weight distribution of training rollouts. Among rollouts with computable evidence weights, 28.32% of correct rollouts and 32.71% of incorrect rollouts receive zero or near-zero weights (W ≤ 0.05). The proportions of zero and near-zero weights are reported separately in the table below.

CorrectnessW=00<W≤0.05W>0.05
Correct23.34%4.98%71.68%
Incorrect26.56%6.16%67.29%

We further evaluate the final model on random subsets of the six test benchmarks, grouping outputs by their evidence weights. The zero-weight and near-zero-weight groups have ECE values of 0.0575 and 0.0900, respectively, with mean confidence close to within-group accuracy. Among incorrect outputs in these groups, 11.50% and 6.90%, respectively, have confidence above 0.8. These results provide empirical evidence of the final model’s calibration performance on outputs with zero or near-zero evidence weights.

Evidence-weight groupACCMean confidenceECE10Mean confidence on incorrect outputsHigh-confidence rate among incorrect outputs (p > 0.8)
All0.70150.64060.06090.400110.36%
W=00.64350.62130.05750.430111.50%
0<W≤0.050.57970.55660.09000.35806.90%
W>0.050.73460.65540.07920.390010.23%

Q5: Thank you for this suggestion. All methods in Table 1 use Qwen3.5-4B. CoT and TopK are training-free. ConfTuner uses LoRA-based SFT on the same 5,368 source examples as TraceConf’s SFT stage. RewardingDoubt and VL-Calibration use 40 PPO steps and 20 GRPO steps, respectively, on the same 3,000 training prompts as TraceConf’s RL stage. TraceConf uses 504 SFT steps followed by 125 RL steps. The maximum output and context lengths are 16,384 and 32,768 tokens, respectively, with a thinking-mode budget capped at 12,288 tokens. Each method retains its native prompts and confidence readout.

Valid-confidence coverage is 87.31%, 96.03%, 94.14%, 96.83%, 81.92%, and 100% for CoT, TopK, ConfTuner, RewardingDoubt, VL-Calibration, and TraceConf, respectively. The 8B experiment in Appendix C has no matched 8B baselines and is therefore reported only as a supplementary replication of the full training pipeline on another backbone, not as evidence of a same-backbone comparative advantage.

Reviewer_jL8QOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes TraceConf - a training-based approach for confidence calibration. TraceConf trains multimodal language models to jointly predict responses, evidence for the responses as regions in images, and corresponding confidence scores. For counterfactual analysis, it assesses the regions through removal and retention interventions, then uses the resulting evidence score both as a response-level reward and as a weight on confidence supervision. The approach is evaluated on six benchmarks, including MATH-Vision, MMStar, MMMU-Pro-v2, where TraceConf achieves the best AUROC.

Critical strengths
  1. Interesting setup for calibration: I like the idea of generating confidence that is grounded in evidence. Jointly training allows the models to learn evidence-based confidence calibration.

  2. The proposed counterfactual analysis helps avoid wrongly generated evidence and makes the evidence-based calibration more reliable.

  3. Impressive results across six benchmarks: TraceConf achieves the highest AUROC across all six benchmarks while improving average calibration and maintaining accuracy.

  4. The paper is well organized and easy to follow.

Critical weaknesses
  1. My main concern for training-based calibration approaches is generalization. Eventually, the models are supposed to deal with novel queries and assuming access to training data is not practical. It is perhaps okay to calibrate the confidence on a large calibration set. How does the approach perform on novel benchmarks or questions?

  2. The approach assumes the evidence is always available in the images. However, this may not be the case for various types of reasoning questions (e.g., abductive reasoning), where the complete evidence is not available. The MLMs are good at this type of reasoning due to the inherent knowledge of the world beyond the input image/video. How does the proposed approch work in these situations or in situations where the evidence is distributed in multiple regions in the image/video. "The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning", ECCV 2022.

  3. The approch considers a Qwen model, how about the other more state-of-the-art models?

  4. The evidence gate affects both response rewards and confidence supervision, but Table 3 removes both together, leaving their individual effects unclear. Although the gate improves calibration and evidence dependence, its gains in AUROC (from 0.7960 to 0.7982) and Support Gap (from 20.59 to 20.67) are not significant.

Questions for authors

Please respond to the comments in the weaknesses section. Overall, I like the approch and I think some additional experimental support could better justify the claims.

3 comments

Weakness 1: Thank you for raising the question of generalization. Table 1 directly evaluates this setting. We evaluate the same TraceConf model, trained with SFT and RL on our training sets, on six benchmarks to assess generalization, without benchmark-specific training or additional confidence calibration. Appendix A.1 describes the distinct sources of our training data and evaluation benchmarks. These benchmarks cover visual mathematical reasoning, general visual understanding, and multidisciplinary reasoning. TraceConf achieves the highest AUROC among the evaluated methods on all six benchmarks, with a macro-average AUROC of 0.7982 and ECE of 0.0840. These results support the transfer of its learned confidence estimation capability to questions not used in post-training. At inference time, TraceConf requires only the image and question, with no target-task labels, reference answers, or online verifier. We will clarify this cross-benchmark evaluation setting in the revised manuscript.

Weakness 2: Thank you for raising this question. TraceConf assesses the visual support that declared regions provide for an answer; it does not require the entire reasoning chain to be explicit in the image and allows world knowledge to complement visual clues in abductive reasoning. We evaluated the existing checkpoint on an inference-plausibility judgment task comprising 1,000 candidate inferences from the Sherlock v1.1 test split, using human Likert scores of 3 and 1 as high- and low-plausibility labels, respectively. Without training or additional calibration on Sherlock, TraceConf achieves 89.0% ACC, 0.0296 ECE, and 0.7987 AUROC, outperforming the compared baselines on all three metrics and supporting transfer to this abductive reasoning setting. For spatially distributed evidence, TraceConf supports multiple declared boxes and intervenes on their union. Training annotations retain multiple regions rather than merging them into a single enclosing box. Multi-box outputs account for 34.18%–70.31% of predictions across the six benchmarks. The tables below report the Sherlock results and multi-box output rates; Example 1 in Appendix B illustrates a multi-region declaration.

MethodACC ↑ECE10 ↓AUROC ↑
TraceConf89.0%0.02960.7987
CoT86.8%0.07350.7379
TopK83.8%0.13820.6362
ConfTuner87.6%0.04200.7097
RewardingDoubt85.9%0.04190.6947
VL-Calibration83.7%0.03980.6907
DatasetMulti-box output rate
MathVision45.40%
MathVerse-V70.31%
DynaMath-2V54.96%
MMStar54.20%
MathVista62.30%
MMMU-Pro34.18%

Weakness 3: Thank you for this suggestion. We replicated the complete SFT-to-RL pipeline on InternVL3.5-8B using the same training data, evidence gate, frozen verifier, and evaluation protocol as in the main experiments, and compared it with two training-free baselines, CoT and Top-k, on the same backbone. Across the six benchmarks, TraceConf achieves higher macro-averaged ACC and AUROC and lower macro-averaged ECE than both baselines, supporting its effectiveness on another backbone. Detailed results are reported in the table below.

DatasetMethodACC ↑ECE10 ↓AUROC ↑
MathVisionCoT46.0%0.250.62
Top-k47.5%0.270.70
TraceConf50.50%0.10190.8386
MathVerse-VCoT53.0%0.290.57
Top-k54.3%0.280.71
TraceConf57.84%0.07280.7857
DynaMath-2VCoT66.0%0.220.65
Top-k67.0%0.200.66
TraceConf70.04%0.19300.7522
MMStarCoT67.6%0.230.70
Top-k67.1%0.200.68
TraceConf70.07%0.05300.6926
MathVistaCoT72.4%0.170.72
Top-k73.4%0.160.76
TraceConf75.90%0.16070.8428
MMMU-Pro-v2CoT61.9%0.210.72
Top-k62.9%0.210.73
TraceConf65.87%0.09530.7570
Macro-averageCoT61.2%0.230.66
Top-k62.0%0.220.71
TraceConf65.04%0.11280.7781

Weakness 4: Thank you for this suggestion. We added reward-only and confidence-loss-only gating ablations under the same training and evaluation protocol to isolate the gate’s two roles. The results show that the sequence-level evidence reward strengthens confidence dependence on the declared regions, while evidence weighting of the local loss improves calibration and confidence responses to reduced visual support. These roles provide complementary benefits, with the full method achieving the best overall performance, as shown in the table below. Relative to W=1, the gains in AUROC and Support Gap are small, whereas the improvements in ECE, NecDrop, and SufGap are more pronounced. AUROC measures correct–incorrect ranking; it does not directly measure probability calibration or dependence on declared evidence. The gate’s primary benefits therefore lie in improving confidence calibration and strengthening confidence dependence on declared visual evidence, rather than merely improving ranking.

Training variantACC ↑ECE10 ↓AUROC ↑Support Gap ↑ (pp)NecDrop ↑ (pp)SufGap ↓ (pp)
SFT only0.70180.14860.765317.9210.354.64
W=10.71260.12430.796020.598.964.65
Reward-only0.70210.11300.795516.8611.223.32
Conf-loss-only0.69640.09190.789220.0410.374.95
Full TraceConf0.73380.08400.798220.6711.943.28

Thank you again for your time and thoughtful feedback. If any concerns remain after our responses, we would be happy to address them and provide further clarification. We appreciate the opportunity to discuss and improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.