- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes "trajectory agreement" — the fraction of training checkpoints whose prediction agrees with the final model's prediction — as a logit-free confidence proxy. The same signal underlies the published selective-prediction method SPTD (Rabanser et al., 2025, TMLR), which up-weights late disagreement by . The authors argue that as training converges, checkpoints agree more and more, so agreement saturates at 1 and stops carrying error information ("convergence-induced saturation"). Two bounds make this precise: (i) the calibration error of agreement is pinned to within of the model's error rate , i.e. ; (ii) , and when the score is constant on the window-agreement event , . A corollary gives the late limit: implies and . Experiments use 3D visual grounding (ScanRefer, 3 seeds; Nr3D, 1 run; a 3.8M-parameter GeoClassifier over frozen Uni3D + DistilBERT features with ground-truth proposals, ) and CIFAR-10 / ResNet-18 (3 seeds, ). Key results: moving a fixed-size window later raises CE and lowers AUROC in 13 of 14 run-by-window-size cells; in the last-checkpoint window 89-94% of errors receive agreement 1; temperature scaling and a 3-seed ensemble are calibrated on the same predictions while agreement is badly overconfident (ECE 0.228 vs 0.031 on ScanRefer); SPTD's late weighting helps on CIFAR-10 (+0.016 to +0.020) but hurts on 3D (-0.002 to -0.016). A secondary finding is that agreement fabricates a subgroup gap.
Critical strengths
- Honest, reproducible execution: three seeds agree closely; scene-grouped and paired bootstraps are used; the single non-significant cell is flagged; weak "four-group" evidence is reported as descriptive only; the supplement ships scripts with unit tests and JSON number sources.
Critical weaknesses
-
The central empirical comparison is confounded by checkpoint density. SPTD's original experiments used , but this paper uses (3D) and (CIFAR-10). SPTD's weights depend on trajectory resolution. The authors' own thinning study (Table 7) shows the effect size moves with density — the CIFAR-10 gain shrinks from +0.016..+0.020 to +0.007..+0.008 when thinned to . By symmetry, the 3D degradation observed at (-0.002 to -0.016) may also change (vanish or reverse) at realistic density, and the authors state denser 3D trajectories were not stored. So the headline claim "SPTD hurts on 3D grounding" is not established at the density the method was designed for.
-
Mismatch between the object analyzed (fixed-final agreement) and the marketed target (SPTD). The genuine failure is the limit, which nobody uses; at (the original setting) SPTD improves AUROC on CIFAR-10, and the 3D degradation is small (). The authors themselves write that their results "qualify rather than contradict" the method, yet the abstract and introduction are framed adversarially. Moreover, Proposition 2 only covers the case where the score is constant on ; it does not directly bound SPTD's actual score , so the theory aligns with the method only at and .
Questions for authors
- (Key) Can you store denser 3D trajectories (e.g. ), or run SPTD on CIFAR-10 near the original , to test whether the 3D AUROC degradation persists at realistic density?
- If SPTD's weighted-disagreement score is recalibrated (Platt/isotonic), does it rival softmax? If so, the finding is better stated as "the raw proxy needs recalibration".
- Please add the within-trajectory probability-averaging baseline (average the softmax over the same checkpoints, report ECE/AUROC). Does it also saturate, or is saturation specific to agreement?
- Can Proposition 2 be extended from "constant on " to the general weighted , and can you give a matching lower bound to establish tightness?
- How does the picture change with non-ground-truth proposals (a real detector front end) and at lower error rates / larger models — does saturation still scale with ?
- Can you add a small LLM experiment (e.g. self-consistency agreement across samples vs training-trajectory agreement)?
Thank you for the thorough response and the revisions.
My questions are addressed or scoped by authors' rebuttal. However, my remaining reservation is that the contribution is an empirical diagnosis of one published method’s failure regime.
Thus, I will keep my original rating. I won't argue for a rejection.