- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper combines synthetic data generation with a constrained adaptation method for motion forecasting. A conditional diffusion model generates rare driving futures from specified longitudinal, lateral and interaction attributes. Samples are refined and checked for behavioral and physical validity, producing 3,000 synthetic trajectories. The forecaster then undergoes 1,000 adaptation updates in a small subspace built from behavior gradients and their effects on existing predictions. It seeks updates that improve both naturalistic and tail objectives; when these conflict, it limits naturalistic degradation while improving the tail objective. On nuScenes, ParetoTail reports minADE5 of 1.17 m and minFDE5 of 2.34 m. Against the same architecture trained on naturalistic data, Common error falls from 1.16 to 1.11 m and Tail error from 2.80 to 2.27 m. Evaluation includes three seeds for the training controls, component ablations and results on NGSIM and HighD. The joint improvement regime activates in 40% of updates, with both losses actually decreasing in 90% of those cases.
Critical strengths
-
The paper evaluates both sides of its central trade-off using a fixed Common and Tail partition and the same forecasting architecture. The budget ablation is particularly informative: removing the constraint improves Tail error from 2.27 to 2.15 m but worsens Common error from 1.11 to 1.15 m. This provides concrete evidence that the preservation budget affects the intended balance. Adaptation also adds no modules or computation at inference time.
-
The implementation is carefully worked out. Appendix C avoids dense parameter matrices through a low rank covariance representation, Jacobian products and conjugate gradient solves, making the proposed adaptation procedure concrete. The generator also uses physically meaningful controls, a differentiable decoder and explicit validity checks. Its reported validity and condition hit rates provide useful checks on whether the synthetic supervision meets the requested constraints.
Critical weaknesses
W1. ParetoTail receives 1,000 additional updates, while Naive and Grouped are evaluated at the end of their earlier training. The missing controls are plain fine-tuning from the same checkpoint on naturalistic data, naturalistic plus generated data, and a weighted combination of the two objectives. Without matched update budgets and checkpoint selection, the reported improvement cannot be cleanly attributed to Pareto adaptation.
W2. The paper cites established long-tail learners and GALTraj, the closest generation-based approach, but compares only Natural-only, Grouped and Naive under the shared architecture. It also omits standard alternatives for balancing conflicting objectives, such as MGDA, PCGrad and CAGrad. These comparisons, together with a scalarization-weight sweep, are needed to establish what the proposed machinery adds over existing approaches.
W3. Requested behaviors determine both the synthetic data and the coverage weights used during adaptation, yet their selection procedure, numerical targets and sampling proportions are not sufficiently documented. The main Common/Tail evaluation also uses the same behavioral descriptor that guides adaptation. Sensitivity to different target distributions and an independent difficulty-based partition would help establish broader relevance. Several adaptation hyperparameters are selected on the validation split used for reporting, without a clearly separate final evaluation.
W4. The constraints operate on local surrogates, with curvature estimated from probe updates. A per-step degradation budget does not bound cumulative degradation over 1,000 updates, and a positive mean-minus-standard-deviation tail margin does not ensure improvement in every behavior region. The reported success and violation rates omit the magnitude of failed updates and cumulative drift. Aggregate Common/Tail errors also leave possible regressions within individual behaviors hidden.
W5. Important details remain unclear, including the forecaster architecture and loss, anchor count, kernel bandwidth and subspace refresh schedule. The component ablations also do not fully specify the replacement rule when a component is removed. Each adaptation update requires behavior-conditioned gradients, geometry solves and numerical optimization, but no adaptation time or memory usage is reported. The unchanged inference time does not establish whether these training costs are worthwhile.
W6. The correction field subtracts the same naturalistic gradient from every behavior gradient. This leaves their centered covariance unchanged, while adding the naturalistic gradient back recovers the ordinary deficit-weighted mean gradient. The remaining construction combines a prediction-space metric, a generalized eigenspace and constrained multi-objective optimization. This is a concrete design, but the paper does not clearly identify the new insight that makes this particular combination necessary or preferable.
Questions for authors
Q1. Starting from the same Natural-only checkpoint, please compare 1,000 updates of plain fine-tuning, fine-tuning with the same synthetic pool, a weighted objective with λ swept, and MGDA or CAGrad. Please report Overall, Common and Tail errors over three seeds with matched checkpoint selection. Does ParetoTail retain an advantage once additional training and objective balancing are controlled?
Q2. How were the external baseline numbers in Table 3 obtained: reruns, released predictions or published results? Please provide a paired comparison on a fixed difficulty subset defined by Natural-only or a Kalman predictor, and include an established long-tail learner and GALTraj under matched conditions. This would test whether the gains extend beyond the paper’s own controls and behavioral partition.
Q3. How were the requested behavior categories, numerical targets and sampling proportions chosen? Were they fixed using training data alone? Were adaptation hyperparameters selected on the same validation instances used for reporting? Please state the full selection protocol, whether baselines received comparable tuning, and whether an independent final evaluation is available.
Q4. What replaces the update rule when the correction field or two-regime switch is removed in Table A.2? Were the 10% of Regime I updates that failed to improve both losses applied or rejected, and how large were the failures? Please also report cumulative degradation relative to the initial predictor and whether aggregate Tail improvement hides regressions in individual behavior regions.
Q5. Please specify the forecaster architecture and loss, anchor count, kernel bandwidth and subspace refresh schedule. What are the measured wall-clock time and peak memory for adaptation, including all 1,000 updates? How do these compare with base training and ordinary fine-tuning, and what accuracy benefit justifies the additional cost?
4 comments
Response to Reviewer 3m7M (Part 2/3)
4. Local Adaptation Reliability (W4, Q4).
(1) Ablation Settings (W4, Q4). Removing the Correction Field retains the mean tail direction but excludes heterogeneous corrections (, ); removing the Two-Regime Switch applies Regime II throughout adaptation (revised Table 5). These ablations isolate heterogeneous corrections and regime selection.
(2) Update Reliability (W4, Q4). Across three runs, Table R3 shows that 90.0% of Regime I decisions reduce both naturalistic and tail losses, while only 1.0% of Regime II decisions violate the naturalistic budget. This supports the reliability of the two-regime optimization in identifying compatible updates and controlling naturalistic degradation.
Table R3. Reliability of local Pareto adaptation across three independent runs.
| Run | Regime I Decisions | Joint Improvement | Regime II Decisions | Budget Violations |
|---|---|---|---|---|
| Run 1 | 400 | 360 (90.0%) | 600 | 6 (1.0%) |
| Run 2 | 400 | 359 (89.8%) | 600 | 6 (1.0%) |
| Run 3 | 400 | 361 (90.3%) | 600 | 6 (1.0%) |
| Overall | 1,200 | 1,080 (90.0%) | 1,800 | 18 (1.0%) |
(3) Cumulative Performance (W4, Q4). To further assess whether local adaptation leads to cumulative performance degradation, we track naturalistic and tail losses relative to the initial predictor throughout the 1,000 updates. As shown in Table R4, the cumulative Common and Tail losses decrease by 0.70% and 8.20%, respectively. Despite minor fluctuations during adaptation, both objectives improve at the final checkpoint, indicating that the proposed method maintains a favorable Common–Tail balance over the complete adaptation process.
Table R4. Cumulative relative loss changes during ParetoTail adaptation. Negative values indicate loss reduction.
| Adaptation Step | Common | Tail |
|---|---|---|
| 0 | 0.00% | 0.00% |
| 200 | -0.18% | -2.10% |
| 400 | -0.32% | -3.80% |
| 600 | -0.25% | -5.10% |
| 800 | -0.45% | -6.70% |
| 1,000 | -0.70% | -8.20% |
(4) Behavioral Generalization. To further examine the distribution of improvements across different rare behaviors, we evaluate performance across five behavioral groups. As shown in Table R5, ParetoTail achieves substantial Tail improvements in four of the five groups, with reductions of up to 8.50% for mild lateral behaviors and 7.20% for longitudinal behaviors. Interaction-heavy and aggressive lateral behaviors also benefit consistently. Composite behaviors, which involve multiple simultaneous rare attributes, remain more challenging, with a relatively small Tail loss increase of 0.80%. Overall, these results highlight the effectiveness of ParetoTail across diverse long-tail behaviors while identifying composite scenarios as a promising direction for further refinement.
Table R5. Relative loss changes across behavioral groups after adaptation. Negative values indicate improvement.
| Behavioral Group | Common | Tail |
|---|---|---|
| Longitudinal | -0.40% | -7.20% |
| Mild lateral | -0.65% | -8.50% |
| Aggressive lateral | +0.20% | -5.30% |
| Interaction-heavy | +0.35% | -3.10% |
| Composite behavior | +0.55% | +0.80% |
5. Implementation and Efficiency (W5, Q5).
(1) Implementation Details (W5, Q5). Our forecaster consists of multiple standard encoder–decoder modules, with a hidden dimension of 64, dropout rate of 0.1, and 10 output modes. The training objective combines minADE$_5$ and Laplace losses with equal weights. The predictor is pretrained for 120 epochs using AdamW. For Pareto adaptation, we use a Gaussian kernel with bandwidth , retaining 142 of 180 candidate behavioral anchors. We employ 32 quadrature nodes and retain generalized eigen-directions. Both the adaptation subspace and curvature estimates are refreshed at each update, with conjugate gradients and SLSQP used for geometry solving and constrained optimization, respectively (revised Appendix D.1).
(2) Computational Efficiency (W5, Q5). ParetoTail introduces additional computation only during offline adaptation, without modifying the forecasting architecture. As shown in Table R6, ParetoTail completes 1,000 updates in 9.89 minutes, only 1.9% longer than FT-Naive, while reducing Tail minADE$_5$ by 14.3%. It also achieves the lowest peak GPU memory of 6.23 GiB. Furthermore, its inference time remains approximately 14 ms per batch on an RTX 3090, identical to Natural-only. These results demonstrate a favorable trade-off between adaptation cost and forecasting performance.
Table R6. Computational cost of 1,000 adaptation updates.
| Method | Wall Time | Peak Memory |
|---|---|---|
| FT-Natural | 7.75 min | 6.41 GiB |
| FT-Naive | min | GiB |
| MGDA | 16.78 min | 6.51 GiB |
| ParetoTail | 9.89 min | 6.23 GiB |
Response to Reviewer 3m7M (Part 3/3)
6. Long-Tail Pareto Insight (W6).
The key motivation for defining the correction field is to characterize how underrepresented behaviors seek to modify the predictor relative to ordinary naturalistic learning. Rather than treating behavior-conditioned gradients as isolated learning signals, we use as a common reference to distinguish the baseline learning direction from tail-induced corrections. The deficit-weighted mean correction captures the overall adaptation demand of underrepresented behaviors, while the correction covariance characterizes their heterogeneous learning directions. Subtracting the shared reference does not alter the centered covariance; its purpose is to explicitly separate naturalistic learning, average tail correction, and behavioral heterogeneity.
The central contribution lies in translating this distributional structure into effective adaptation directions. ParetoTail combines the naturalistic gradient , the deficit-weighted tail gradient , and heterogeneous correction directions selected according to the naturalistic prediction geometry . This identifies directions that retain substantial tail-learning benefits while limiting disturbances to naturalistic predictions. The subsequent two-regime optimization exploits compatible Common–Tail improvements and explicitly controls naturalistic degradation when the objectives conflict.
The component ablations further validate this design. Removing the correction field or prediction geometry increases Tail minADE$_5$ from 2.27 m to 2.47 m and 2.50 m, respectively, while disabling the two-regime switch increases it to 2.39 m. Removing the naturalistic preservation budget reduces Tail error to 2.15 m but increases Common error from 1.11 m to 1.15 m. These results demonstrate the importance of jointly modeling heterogeneous tail corrections, naturalistic prediction geometry, and Pareto-aware adaptation.
Thank you for the additional experiments and clarifications. Matching the fine-tuning budget addresses some of my concern. My main concern remains the lack of a direct comparison with long-tail forecasting methods such as GALTraj. The motivation for the correction field is clearer, and the authors acknowledge that subtracting the same naturalistic gradient from each behavior gradient leaves their covariance unchanged. The ablation removes the directions derived from this covariance and the variance penalty together, so it remains unclear whether a simpler formulation could achieve similar gains. The authors mention a separate validation set for model selection but do not specify how it was constructed. No code link is provided for ParetoTail. So I am maintaining my original rating.
Response to Reviewer 3m7M (Part 1/3)
Thank you for recognizing the motivation and technical contributions of our work, and for your detailed feedback and constructive suggestions.
1. Fair Comparison (W1, Q1).
To isolate the contribution of Pareto adaptation from additional training, we conduct comprehensive comparisons starting from the same pretrained Natural-only checkpoint, with 1,000 subsequent parameter updates and a consistent checkpoint-selection rule. The baselines include naturalistic fine-tuning (FT-Natural), density-grouped fine-tuning (FT-Grouped), direct synthetic-data augmentation (FT-Naive), weighted-objective fine-tuning (Weighted FT), and MGDA. FT-Naive, Weighted FT, and ParetoTail use the same naturalistic data and frozen pool of 3,000 generated trajectories. We further sweep λ∈{0.1,0.3,1,3,10} for Weighted FT.
As shown in Table R1 (revised Table 4), ParetoTail achieves the best mean Overall/Common/Tail minADE$_5$ of 1.17/1.11/2.27 m. Compared with the strongest Tail-performing weighted baseline (λ=3), ParetoTail reduces Common and Tail errors by 9.0% and 9.2%, respectively. These results demonstrate the effectiveness of the proposed adaptation strategy beyond continued training and conventional multi-objective optimization.
Table R1. Comparison of adaptation strategies on nuScenes after 1,000 updates. Results are minADE$_5$ (m), reported as mean ± SD over three runs.
2. Existing Methods and Generalization (W2, Q2).
(1) Existing Methods (W2, Q2). The external baselines in Table 3 were reproduced using their publicly released implementations. These comparisons establish the competitiveness of ParetoTail against existing motion forecasting methods.
To further examine generalization across forecasting architectures, we apply ParetoTail to the MTR backbone. As shown in Table R2, ParetoTail reduces Tail minADE$_5$ by 9.0% compared with weighted fine-tuning, while simultaneously improving Common and Overall performance. These results demonstrate that the proposed adaptation strategy can effectively transfer to another forecasting backbone.
Table R2. Comparison of adaptation strategies on the MTR backbone using nuScenes. Results are minADE$_5$ (m).
(2) Paired Evaluation (W2, Q2). We further rank test samples by the prediction errors of Natural-only and evaluate all methods on identical difficulty-defined subsets. As shown in Figure 4 (revised manuscript), ParetoTail achieves pronounced improvements on the most challenging cases across the hardest 1%–100% evaluation curve. This complements Table 3 by directly comparing forecasting performance on the same challenging samples.
3. Request Design and Model Selection (W3, Q3).
The generation requests are constructed exclusively from the naturalistic training data, covering ten rare-event categories across longitudinal, lateral, and interaction behaviors. Numerical targets are selected from the 75th, 90th, and 95th percentiles of the corresponding training attributes, with a normalized tolerance of 0.15. The sampling proportions are set to 30%, 35%, and 35% for longitudinal, lateral, and interaction behaviors, respectively. All generation conditions are fixed using training data alone, independently of the evaluation set.
For model selection, the adaptation hyperparameters r, κ, ϵN, ωN, and ωT are tuned on the validation set, while final performance is evaluated on the separate test set. For Weighted FT, we consider five coefficients λ∈{0.1,0.3,1,3,10} under the same continuation and checkpoint-selection protocol. This separation ensures that the reported test results are not used for hyperparameter selection.