acceptodds
Under review as a conference paper at ICLR 2027

Latent-Type Posterior Conditioning for Heterogeneous Multi-Agent Trajectory Prediction under Noisy and Missing Type Labels

Abstract

Heterogeneous trajectory predictors condition each agent’s representation on a semantic type label (pedestrian, cyclist, vehicle) and treat it as ground truth. In deployment the label is a perception output that may be wrong, missing, or in-consistent with the agent’s motion, which raises the question of when a predictor should rely on it. We propose posterior type conditioning: a dynamics-based type classifier and a learned, context-dependent noisy-label channel are combined by Bayes’ rule into a per-agent posterior over types, which conditions the predictor. Both are trained jointly from the prediction loss and the label likelihood, without supervision of label reliability. On a controlled benchmark and three real-world datasets spanning metric- and pixel-space trajectories and hand-annotated and perception-derived labels (COSMOS, Stanford Drone, Argoverse 2), we find that the value of the label is strongly class-dependent: relative to a type-agnostic model it improves prediction for a behaviorally distinct minority class by 8–15% (pedestrians among vehicles) and by 1–3% where motion already identifies the type. Naive conditioning degrades by 11–34% at 50% label corruption; label-noise augmentation removes most of this loss but forfeits part of the minority-class gain. Posterior conditioning retains most of that gain, degrades more gracefully, and is statistically indistinguishable from the best model on held-out scenes, cities, and recording dates. Its per-agent trust score detects corrupted labels with AUROC 0.91–0.97 on two of three datasets and recovers interpretable label–motion relationships, such as crediting a cyclist label only at cycling speeds.

Primary area:
applications to robotics, autonomy, planning
Keywords:
trajectory prediction, heterogeneous multi-agent systems, label noise, Bayesian conditioning, label uncertainty, robustness, representation learning, autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JzQnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies multi-agent trajectory prediction when semantic agent-type labels may be noisy or missing. The proposed POSTERIOR method treats the underlying type as latent and combines two sources of evidence: a type distribution inferred from observed motion, and a learned noisy-label channel with an agent-specific reliability estimate. Bayes' rule yields a posterior over types, whose expected embedding conditions the trajectory predictor. The posterior probability assigned to the observed type label is additionally reported as a per-agent label-trust score. The empirical study compares type-agnostic, hard-label, motion-only soft, and posterior conditioning on a shared GRU-attention backbone. Experiments span AV2, SDD, COSMOS, and a synthetic benchmark, under uniform label flips, structured class-confusion flips, and missing labels.

The findings indicate that corruption augmentation accounts for most of the pooled robustness: once HARD is trained with flipped and missing labels, POSTERIOR is statistically indistinguishable from HARD in pooled minADE across all real-dataset conditions. The clearest forecasting benefit appears for AV2 pedestrians, where POSTERIOR improves over augmented HARD by roughly 2–7%, with the largest gain under missing labels. The trust score detects injected corruption well on AV2 and SDD and recovers interpretable label–motion inconsistencies. The paper further shows that this score should not be read as confidence in the trajectory prediction itself.

Critical strengths

1. The comparison among conditioning strategies is carefully controlled. NONE, HARD, SOFT, and POSTERIOR share the same prediction backbone and differ mainly in how type information enters. The evaluation covers multiple corruption mechanisms, shifted splits, repeated runs, class-specific breakdowns, and augmentation ablations. This level of control supports the central empirical claim that corruption augmentation matters more than the conditioning rule for pooled robustness. The negative result for POSTERIOR is nonetheless informative for practitioners choosing between these designs.

2. The trust-score analysis is unusually thorough and appropriately qualified. The paper examines corruption detection, calibration, kinematic correlates, a real-label audit, and selective prediction. Notably, it draws a clean distinction between confidence in the supplied label and confidence in the predicted trajectory, and the coverage and partial-correlation analyses back this up. The transparent reporting of the failed COSMOS run and the absence of annotation errors in the SDD audit add credibility rather than detracting from it.

Critical weaknesses

1. The latent-type formulation offers limited methodological novelty relative to its benefit. POSTERIOR combines a motion-derived type distribution with a learned class-conditional observation channel and a per-agent reliability estimate. While the context-dependent reliability term is a useful addition, the overall construction follows naturally from established noisy-label and annotator-channel models (Dawid–Skene-type estimation in particular), and the paper does not articulate what the Bayesian combination buys that a well-tuned simpler scheme cannot. Against this, the empirical returns are narrow: the conditioning rule makes no significant difference to pooled error once augmentation is in place, and the one consistent predictive gain (AV2 pedestrians, 2–7%) is confined to a single class on a single dataset. The incremental formulation paired with a localized gain makes the contribution feel more like a careful empirical study than a methodological advance.

2. The experiments do not directly test the deployment scenario the method is motivated by. Trust evaluation relies almost entirely on injected label flips and synthetically constructed behavioral anomalies. The SDD audit turns up no actual annotation errors, and COSMOS's perception-derived labels — while unanimous within each track — have no independent human verification, so track-level unanimity cannot be taken as correctness. The trust score is therefore validated as a detector of motion–label disagreement, not as a detector of errors produced by a real perception pipeline, which is the setting the abstract and introduction emphasize. The label-permuted solution observed in one of five COSMOS runs compounds this concern: it occurs precisely where motion carries little type information, suggesting the latent variable may not be identifiable from data alone in the weak-evidence regime that matters most.

3. The headline claims rest on modest statistical footing, and "indistinguishability" is overstated. Conclusions are drawn from five training runs (three for several baselines) on hardware that is explicitly nondeterministic, and the key positive result — AV2 pedestrians — is supported by a paired test at p = 0.047 on clean labels across several datasets, classes, and corruption conditions, with no correction for multiplicity. Meanwhile, the central negative claim that POSTERIOR and augmented HARD are "statistically indistinguishable" is an equivalence assertion that failure to reject on five runs does not establish; a formal equivalence test would be needed. The pedestrian effect is also the sole consistent gain, with only a weaker trend on SDD and nothing on COSMOS, which raises the question of whether it reflects the posterior mechanism itself or compensates for the rarity of the class in training.

Questions for authors
  1. How stable is latent-type identification across seeds? One of five COSMOS runs converged to a label-permuted solution. Did permutation or collapse failures occur in any other prespecified or exploratory runs, and is there a detection criterion that does not require access to clean type labels?

  2. How should the statistical evidence for the AV2 pedestrian result be read under multiple comparisons? The clean-label comparison uses five paired runs and reports p = 0.047, while the paper examines multiple datasets, classes, and corruption conditions. Does the result survive an appropriate correction, and was the pedestrian-specific analysis specified before the pooled results were inspected?

  3. Can the trust score be evaluated against genuine perception errors with independently verified type labels? Detector or tracker outputs checked against human-verified agent types would separate true semantic errors from correctly labeled agents exhibiting atypical motion. Such evidence would substantiate the deployment claim considerably.

  4. Does the AV2 pedestrian improvement survive stronger minority-class controls? Please compare against HARD with class-balanced sampling, a class-weighted prediction loss, or a pedestrian-specific prediction head. If these controls absorb the 2–7% gain, the result would suggest POSTERIOR mainly compensates for insufficient minority-class supervision rather than exploiting the label channel.

  5. Do the conclusions transfer to a stronger trajectory-prediction backbone? The compact reference architecture is the right choice for isolating the conditioning mechanism, but a single result on a modern higher-capacity predictor would clarify whether semantic type uncertainty matters beyond this experimental setup.

Reviewer_A969OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies heterogeneous multi-agent trajectory prediction when semantic agent-type labels may be incorrect or missing. The proposed POSTERIOR approach combines a motion-based type estimate with a learned noisy-label model to infer a posterior over agent types, which is used for prediction and provides a per-agent label trust score. Experiments on AV2, SDD, and COSMOS show that simple corruption augmentation provides most of the pooled robustness, while POSTERIOR offers additional benefits mainly for AV2 pedestrians and under missing labels.

Critical strengths
  • Considering upstream perception errors is a meaningful direction for trajectory prediction.

Rather than assuming that semantic type labels provided by the perception/tracking pipeline are always correct, the paper explicitly studies how incorrect or missing upstream information affects downstream prediction. This is a useful step toward more realistic trajectory prediction.

  • The experimental study is carefully controlled.

The comparison among different conditioning strategies uses a common backbone, and the evaluation covers multiple datasets, corruption settings, distribution shifts, and independent runs, making the empirical comparisons relatively convincing.

  • The analysis provides useful insights beyond demonstrating the proposed method.

In particular, the paper shows that simple corruption augmentation accounts for most of the pooled robustness, identifies when semantic type information is actually beneficial, and clarifies that the learned trust score measures label–motion consistency rather than prediction confidence.

Critical weaknesses
  1. The correspondence between the simulated semantic-label corruption and errors from real perception systems is insufficiently established.

The central motivation concerns incorrect or unavailable semantic labels produced by an upstream perception system, but the primary evaluation simulates these failures through uniform random label flips and random dropout at predefined rates. The class-confusion experiment is somewhat more structured, but its confusion probabilities are manually specified rather than estimated from an actual perception system. These protocols provide useful controlled stress tests, but it is unclear whether they reproduce the distribution and structure of semantic errors encountered in deployment.

This issue is particularly important because the proposed posterior explicitly learns when motion evidence is inconsistent with a supplied label. The reported robustness and trust-detection performance could therefore depend substantially on the assumed corruption model. An empirical characterization of semantic errors from an actual detector/tracker, followed by evaluation using either the real perception outputs or a corruption model estimated from them, would provide considerably stronger support for the paper's practical motivation. COSMOS seems particularly relevant here because its labels already originate from a perception pipeline, although the lack of human ground truth currently prevents measuring its actual error distribution.

  1. The generalizability of the main findings across different class–motion distributions remains unclear.

The method-specific advantage is relatively narrow. Once HARD conditioning is trained with corruption augmentation, POSTERIOR provides no measurable pooled advantage across the three datasets. Its clearest benefit is concentrated on AV2 pedestrians, where the minority class is particularly distinctive in its motion characteristics; the corresponding evidence on SDD is weaker, and COSMOS does not show the same benefit.

More fundamentally, the posterior and trust mechanism rely on learned relationships between semantic class and observed motion, and the analysis suggests that these relationships can be highly dataset-specific. For example, the trust behavior for SDD vehicles is strongly affected by the fact that the training distribution contains predominantly stationary vehicles, causing valid moving vehicles to appear atypical. Although the held-out scene/city/date evaluations provide useful distribution shifts, they do not establish whether the learned label–motion relationships remain reliable when class-specific motion distributions change substantially. This makes it difficult to know how broadly conclusions such as when semantic type information is valuable, or when a supplied type label should be trusted, can be generalized.

Questions for authors
  1. Can the authors provide evidence that the synthetic corruption distributions approximate errors produced by a real perception system?

For example, could a detector with semantic ground truth be used to estimate an empirical confusion matrix and missing-label distribution, or could prediction be evaluated directly using its naturally occurring semantic errors? This would help distinguish robustness to generic synthetic corruption from robustness to realistic perception errors.

  1. Why is semantic object type the appropriate latent variable for trajectory prediction, rather than a motion- or dynamics-oriented latent type?

Semantically different objects may have very similar short-term motion characteristics (e.g., certain bicycles and motorcycles), while members of the same semantic class can exhibit substantially different behaviors. It would be helpful to discuss whether semantic labels should instead be viewed as one source of evidence for a latent motion/dynamics category that is more directly relevant to forecasting.

  1. How robust is POSTERIOR when the observed trajectory itself is imperfect?

The current experiments isolate semantic-label corruption while retaining reliable trajectory histories. In a real perception/tracking pipeline, semantic errors can coexist with localization noise, missed observations, track fragmentation, or identity switches. This seems particularly relevant because POSTERIOR uses observed motion as evidence for deciding whether the semantic label should be trusted. If that motion evidence is itself corrupted, the model might attribute a tracking error to an incorrect semantic label.

  1. The related work could better cover prediction under imperfect upstream perception/tracking.

Although this paper specifically addresses noisy and missing semantic type labels, prior work has studied the effects of other upstream perception/tracking errors on trajectory forecasting, e.g., Liang et al. (CVPR 2020) and Weng et al., “Whose Track Is It Anyway?” (CVPR 2022). Discussing this literature would better contextualize the proposed treatment of semantic-type uncertainty within the broader problem of prediction with imperfect upstream information.

  1. Could the authors further test generalization of the learned label–motion relationship?

A particularly informative experiment would train the posterior/trust mechanism in one environment and evaluate it where the motion distribution of a semantic class changes substantially, rather than only using within-dataset scene/city/date shifts. This would help establish whether the trust score captures a transferable relationship between semantics and dynamics or primarily dataset-specific motion statistics.

Reviewer_RmhqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses multi-agent trajectory prediction when agent type labels are noisy or missing. The key idea is to replace hard type labels with a probability distribution over possible types. Specifically, a motion-based classifier predicts a type distribution from the observed trajectory. This distribution is then combined with the provided type label through a learned noise model to obtain a posterior distribution. The expected type embedding is used to condition the trajectory predictor through FiLM.

Critical strengths
  • The overall presentation is clear.
  • The paper reports results across multiple runs and clearly discusses the conditions under which the proposed method is effective.
Critical weaknesses

1. The contribution of the learned label noise model is unclear

In Table 3, the proposed method achieves its largest improvement when type labels are missing. However, when a label is missing, the posterior reduces to the motion-based type distribution, and the learned label noise model does not directly affect this posterior at inference time. It is therefore unclear how much of the improvement is attributable to the learned noise model, rather than simply using motion-based type estimates when labels are unavailable. A targeted ablation would help isolate the contribution of the proposed mechanism.

2. The effectiveness on real type label errors is insufficiently validated

The label error detection results in Table 4 rely on artificially corrupted labels. In the real-data audit, low-trust examples primarily correspond to atypical motion patterns, such as moving vehicles or stopped cyclists, rather than confirmed labeling errors. These findings demonstrate sensitivity to inconsistencies between labels and motion, but they do not sufficiently establish the method’s ability to detect actual labeling errors.

The authors acknowledge this limitation. Nevertheless, noisy type labels are a central motivation for the method, so experiments using errors from an actual perception system or independently verified labeling errors would provide more direct evidence of its practical effectiveness.

Questions for authors
  1. Does the proposed method outperform a simple baseline that uses the provided type label when available and a motion-based type estimate only when the label is missing?
  2. Could you evaluate the method on real type-label errors from a perception system, using human-annotated ground-truth labels?
  3. Could you evaluate whether the gains from posterior type conditioning remain when map or lane information is added to the AV2 backbone?

Comments on the paper

1 comment
Reviewer_RmhqOpenReview ↗

Thank you for the reponse

Thanks for running all these additional experiments. They address most of my questions.

In particular, the HARD-FALLBACK baseline and the 3way ablation help separate the effects of posterior conditioning and the probe objective. The results point to posterior conditioning as the source of the gain. The real-error experiment on SDD and the lane-context results are helpful too. However, the SDD study is small (although it is understandable given the short rebuttal period), and the pooled differences in the lane-context study aren’t statistically significant. Appendix F also shows that a constant-ρ channel matches the full model with a learned noise model.

I’d like to give the paper’s contributions some more thought during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.