- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies multi-agent trajectory prediction when semantic agent-type labels may be noisy or missing. The proposed POSTERIOR method treats the underlying type as latent and combines two sources of evidence: a type distribution inferred from observed motion, and a learned noisy-label channel with an agent-specific reliability estimate. Bayes' rule yields a posterior over types, whose expected embedding conditions the trajectory predictor. The posterior probability assigned to the observed type label is additionally reported as a per-agent label-trust score. The empirical study compares type-agnostic, hard-label, motion-only soft, and posterior conditioning on a shared GRU-attention backbone. Experiments span AV2, SDD, COSMOS, and a synthetic benchmark, under uniform label flips, structured class-confusion flips, and missing labels.
The findings indicate that corruption augmentation accounts for most of the pooled robustness: once HARD is trained with flipped and missing labels, POSTERIOR is statistically indistinguishable from HARD in pooled minADE across all real-dataset conditions. The clearest forecasting benefit appears for AV2 pedestrians, where POSTERIOR improves over augmented HARD by roughly 2–7%, with the largest gain under missing labels. The trust score detects injected corruption well on AV2 and SDD and recovers interpretable label–motion inconsistencies. The paper further shows that this score should not be read as confidence in the trajectory prediction itself.
Critical strengths
1. The comparison among conditioning strategies is carefully controlled. NONE, HARD, SOFT, and POSTERIOR share the same prediction backbone and differ mainly in how type information enters. The evaluation covers multiple corruption mechanisms, shifted splits, repeated runs, class-specific breakdowns, and augmentation ablations. This level of control supports the central empirical claim that corruption augmentation matters more than the conditioning rule for pooled robustness. The negative result for POSTERIOR is nonetheless informative for practitioners choosing between these designs.
2. The trust-score analysis is unusually thorough and appropriately qualified. The paper examines corruption detection, calibration, kinematic correlates, a real-label audit, and selective prediction. Notably, it draws a clean distinction between confidence in the supplied label and confidence in the predicted trajectory, and the coverage and partial-correlation analyses back this up. The transparent reporting of the failed COSMOS run and the absence of annotation errors in the SDD audit add credibility rather than detracting from it.
Critical weaknesses
1. The latent-type formulation offers limited methodological novelty relative to its benefit. POSTERIOR combines a motion-derived type distribution with a learned class-conditional observation channel and a per-agent reliability estimate. While the context-dependent reliability term is a useful addition, the overall construction follows naturally from established noisy-label and annotator-channel models (Dawid–Skene-type estimation in particular), and the paper does not articulate what the Bayesian combination buys that a well-tuned simpler scheme cannot. Against this, the empirical returns are narrow: the conditioning rule makes no significant difference to pooled error once augmentation is in place, and the one consistent predictive gain (AV2 pedestrians, 2–7%) is confined to a single class on a single dataset. The incremental formulation paired with a localized gain makes the contribution feel more like a careful empirical study than a methodological advance.
2. The experiments do not directly test the deployment scenario the method is motivated by. Trust evaluation relies almost entirely on injected label flips and synthetically constructed behavioral anomalies. The SDD audit turns up no actual annotation errors, and COSMOS's perception-derived labels — while unanimous within each track — have no independent human verification, so track-level unanimity cannot be taken as correctness. The trust score is therefore validated as a detector of motion–label disagreement, not as a detector of errors produced by a real perception pipeline, which is the setting the abstract and introduction emphasize. The label-permuted solution observed in one of five COSMOS runs compounds this concern: it occurs precisely where motion carries little type information, suggesting the latent variable may not be identifiable from data alone in the weak-evidence regime that matters most.
3. The headline claims rest on modest statistical footing, and "indistinguishability" is overstated. Conclusions are drawn from five training runs (three for several baselines) on hardware that is explicitly nondeterministic, and the key positive result — AV2 pedestrians — is supported by a paired test at p = 0.047 on clean labels across several datasets, classes, and corruption conditions, with no correction for multiplicity. Meanwhile, the central negative claim that POSTERIOR and augmented HARD are "statistically indistinguishable" is an equivalence assertion that failure to reject on five runs does not establish; a formal equivalence test would be needed. The pedestrian effect is also the sole consistent gain, with only a weaker trend on SDD and nothing on COSMOS, which raises the question of whether it reflects the posterior mechanism itself or compensates for the rarity of the class in training.
Questions for authors
-
How stable is latent-type identification across seeds? One of five COSMOS runs converged to a label-permuted solution. Did permutation or collapse failures occur in any other prespecified or exploratory runs, and is there a detection criterion that does not require access to clean type labels?
-
How should the statistical evidence for the AV2 pedestrian result be read under multiple comparisons? The clean-label comparison uses five paired runs and reports p = 0.047, while the paper examines multiple datasets, classes, and corruption conditions. Does the result survive an appropriate correction, and was the pedestrian-specific analysis specified before the pooled results were inspected?
-
Can the trust score be evaluated against genuine perception errors with independently verified type labels? Detector or tracker outputs checked against human-verified agent types would separate true semantic errors from correctly labeled agents exhibiting atypical motion. Such evidence would substantiate the deployment claim considerably.
-
Does the AV2 pedestrian improvement survive stronger minority-class controls? Please compare against HARD with class-balanced sampling, a class-weighted prediction loss, or a pedestrian-specific prediction head. If these controls absorb the 2–7% gain, the result would suggest POSTERIOR mainly compensates for insufficient minority-class supervision rather than exploiting the label channel.
-
Do the conclusions transfer to a stronger trajectory-prediction backbone? The compact reference architecture is the right choice for isolating the conditioning mechanism, but a single result on a modern higher-capacity predictor would clarify whether semantic type uncertainty matters beyond this experimental setup.
Thank you for the reponse
Thanks for running all these additional experiments. They address most of my questions.
In particular, the HARD-FALLBACK baseline and the 3way ablation help separate the effects of posterior conditioning and the probe objective. The results point to posterior conditioning as the source of the gain. The real-error experiment on SDD and the lane-context results are helpful too. However, the SDD study is small (although it is understandable given the short rebuttal period), and the pooled differences in the lane-context study aren’t statistically significant. Appendix F also shows that a constant-ρ channel matches the full model with a learned noise model.
I’d like to give the paper’s contributions some more thought during the discussion period.