acceptodds
Under review as a conference paper at ICLR 2027

Beyond Joint Refinement: Bounded Residual Flows for Multi-Modal Trajectory Forecasting

Abstract

Multi-future trajectory prediction is a safety-critical bottleneck between perception and planning: an over-confident wrong-mode prediction can steer a planner into an unsafe manoeuvre. The dominant paradigm appends a refinement module to a strong backbone and trains the two jointly from random initialisation — leaving the refined output without an a-priori bound and letting the refiner lean on a learned gate that masks a poorly-behaved refiner on clean data and fails wherever its protection lapses: out-of-distribution input, closed-loop recursion, or low-bit quantisation. We therefore argue that a refiner should be judged by whether its raw correction is valid without the gate, and we construct one that is. BReF (Bounded Residual Flow) is a plug-and-play module that attaches to any frozen multi-modal predictor and refines its trajectories by integrating a learned drift field along a latent “refinement” time, kept separate from the physical trajectory time. A smoothness penalty in this latent time turns the refiner into a contraction, so its raw correction stays small and physically plausible on its own and the gate becomes near-redundant. Because the refiner is identity-initialised on the frozen backbone, the backbone acts as a verifiable performance floor that jointly-trained refiners cannot give. On nuScenes, BReF improves a strong backbone among twenty recent baselines on every multi-mode metric and on single-mode minADE while preserving the full mode set; the decisive gate-forced diagnostic — removing the gate at inference — separates refiners that are indistinguishable on displacement error by nearly three orders of magnitude, tracing the advantage to the latent-time contraction.

Primary area:
generative models
Keywords:
Multi-Modal Trajectory Forecasting, Bounded Residual Flows, Frozen-Backbone Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uUZxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes BReF, a residual correction module attached to a frozen multimodal trajectory predictor. The module uses a Neural ODE to update candidate trajectories along an artificially defined correction time, and controls the correction behavior through drift-magnitude regularization, gating, and zero initialization. The paper also proposes a “force-open gating” diagnostic that analyzes the dependence of different correction modules on gating protection by comparing prediction errors before and after removing gating. The authors report prediction accuracy, cross-backbone adaptation, variations in solvers and integration steps, and several distribution-shift experiments on nuScenes.

Critical strengths
  1. It highlights internal behavioral differences that conventional prediction metrics may obscure. The paper shows that correction modules with similar normal prediction errors can exhibit markedly different performance degradation after removing gating. This provides an instructive perspective for analyzing the behavior of correction modules.

  2. The experimental analysis covers multiple dimensions. The authors examine regularization weights, integration steps, solvers, input noise, geographic and behavioral splits, and report three training seeds for BReF. These experiments help understand the stability of the reported phenomena.

  3. It has a degree of engineering practicality. Freezing an existing predictor and training only a lightweight correction module allows existing models to be reused. Adaptation results on multiple frozen predictors also provide some empirical support for the method’s portability.

Critical weaknesses

Weaknesses

  1. The methodological contribution is relatively incremental. BReF mainly combines existing ideas such as residual correction, frozen backbones, gating, zero initialization, and Neural ODEs. The paper has not yet clearly established a substantive methodological advance of this combination over existing trajectory correction methods. Although the diagnostic of removing gating is somewhat illuminating, whether it can constitute a sufficiently independent and important contribution still requires more thorough argumentation.

  2. There is a lack of cross-dataset validation, and the generality of the conclusions remains unclear. The experiments are concentrated on nuScenes. Adapting to multiple backbone models can support cross-architecture portability, but it cannot replace cross-dataset validation. Therefore, it is currently difficult to determine whether the observed gains and robustness depend on a specific data distribution, scene characteristics, or evaluation protocol, and the evidence is insufficient to support broader generalization claims.

  3. The experiments have not sufficiently demonstrated the necessity of the overall design. Existing ablations show performance changes under different configurations, but they do not sufficiently rule out the possibility that simpler residual correction and regularization methods could achieve similar results. Thus, the current evidence shows more that the approach is effective under the tested settings, but it is not yet enough to prove that its complex design yields irreplaceable benefits or reveals a new mechanism of general significance.

  4. There remains a gap between the scope supported by the experiments and the paper’s claims. The paper emphasizes robustness, stability, and deployment value, but the evidence mainly comes from open-loop prediction errors, gating-removal diagnostics, and artificial perturbation tests. These results cannot yet sufficiently demonstrate advantages under real distribution shifts, recursive use, or downstream planning. The relevant conclusions require more direct experimental support or a clearer limitation of their scope of applicability.

Questions for authors

What is the core novelty? Compared with existing residual learning and trajectory correction methods, what substantive methodological advance does BReF offer? Beyond combining existing components, what unique conclusion does the paper aim to establish?

How is the necessity of the current design validated? Can the current experiments demonstrate that the reported benefits depend on the overall design of BReF, rather than on simpler residual correction and regularization methods? What evidence could distinguish these explanations?

Do the conclusions hold across datasets? The current evidence is concentrated on nuScenes and is insufficient to determine whether these findings are general. Can the authors validate the core conclusions on other motion prediction datasets (e.g., Navsim, nuplan, bench2drive, Argoverse 2, Waymo Open Motion Dataset) with different scene and interaction characteristics? Or, could you provide strong enough justification that the experiments on nuScenes is sufficient?

Do the experiments sufficiently support the paper’s claims? To what extent can the current open-loop prediction results and gating diagnostics support the paper’s claims about robustness, stability, and deployment value? Can the authors provide validation corresponding to these claims, or explicitly narrow the scope of the conclusions?

4 comments

Response to Reviewer uUZx (1/3): what the contribution is (W1, Q1)

We thank the reviewer for a careful review. We agree with most of it, and we have narrowed the paper's claims accordingly. The other two reviewers raised overlapping points, so we ran new experiments; we summarise here what they establish.

About the new numbers. The original training machine no longer exists, and our archive kept only aggregate tables. We therefore rebuilt the full chain (LAformer stage-1 → frozen backbone → refiners) and ran every comparison below on it, with N = 9041 nuScenes val samples. 95% CIs come from a bootstrap clustered on keyframes, and we also report Wilcoxon signed-rank tests. The questions here are relative comparisons on identical samples, which the reproduction can answer; it does not certify the paper's absolute numbers.

What we are no longer claiming. Re-examination confirmed several overclaims, and we withdraw them:

  • That the smoothness penalty makes the refiner a contraction. It does not. The measured full-map Jacobian has σ(JF)>1\sigma(J_F)>1 for every refiner.
  • That the backbone is a guaranteed performance floor. That only holds at initialisation.
  • That frozen refinement beats joint training. Joint training is more accurate, by 9.6 mm minADE₅.
  • That Δ5≈0\Delta_5\approx0 shows a good correction. A zero residual achieves the same.

What remains, and what we now claim. We agree the components (residual correction, frozen backbone, gate, zero-init, Neural ODE) are individually known. We do not claim a new component. We claim two findings, both now supported by paired, per-sample evidence:

  1. A methodological finding about evaluating gated refiners. Refiners that are indistinguishable on displacement error can differ by more than an order of magnitude in how much their accuracy depends on the gate (4–7 mm versus 179–439 mm). At a similar applied-correction size (4.7–9.4 cm), forcing the gate open costs BReF 4–7 mm, but costs an unpenalised ODE 179 mm and an MLP refiner 308–439 mm. Those refiners hide raw residuals of 0.65 m and 1.38 m RMS behind a nearly closed gate. Standard metrics cannot see this, and you identify this as the paper's strength. We now present it as a gate-reliance probe, always reported next to the correction size and a zero-residual control, not as a quality score.
  2. An empirical finding about what makes a residual both useful and gate-safe. A magnitude penalty alone can make any refiner gate-safe. But at four matched penalty weights, an MLP loses its accuracy gain when penalised (no clustered CI excludes 0), while a penalised ODE keeps a significant gain at every weight (details in comment 2/3).

Joint training. Matched joint training turned out to be backbone fine-tuning. In all three joint runs, the refiner's gate ended at exactly zero. Fine-tuning the backbone alone, with no refiner, matched the joint result to within 0.8 mm minADE₅. BReF's role is therefore narrower and, we think, clearer: a cheap correction for backbones that cannot be retrained. It trains 7.7% of the backbone's parameter count, never modifies a backbone weight, and gives a gain of about 1 mm minADE₅ that is significant in each of three independent runs (−0.92 / −1.04 / −1.14 mm, each CI excluding 0).

We will rewrite the abstract and introduction around these two findings and this scope.

Response to Reviewer uUZx (2/3): is the design necessary, or would simpler residual correction do? (W3, Q2)

This is the right question. The paper's ODE-versus-MLP comparison was confounded: only the ODE had a magnitude penalty, because the smoothness term acts on the ODE's drift log, which an MLP does not have. We now test the simpler alternative directly. We add an explicit residual-energy penalty ∥δ∥2\|\delta\|^2 to an MLP refiner, and sweep it and the ODE's smoothness penalty over the same four weights. Each refiner is compared with its own zero-residual control (Δ minADE₅ in mm, 95% CI). "gate-forced" is the error added when the gate is forced open.

wODE + smoothness: gainMLP + energy: gainODE − MLP, minADE₅ODE − MLP, minFDE₅gate-forced, ODE / MLP
0.003 †−0.92 [−1.77, −0.09]−0.25 [−1.18, +0.68]−0.70 [−1.54, +0.16]−2.09 [−3.91, −0.30]+8.4 / +10.1
0.01 (BReF)−0.92 [−1.65, −0.20]−0.48 [−1.44, +0.47]−0.44 [−1.35, +0.44]−2.04 [−3.75, −0.36]+5.6 / +2.3
0.03 †−0.84 [−1.43, −0.25]−0.33 [−0.89, +0.23]−0.53 [−1.13, +0.06]−0.14 [−1.39, +1.11]+1.1 / +0.7
0.1 †−1.12 [−1.62, −0.67]−0.27 [−0.60, +0.04]−0.87 [−1.32, −0.45]−0.27 [−0.96, +0.39]+0.6 / −0.1

† Trained on a second cluster with a different GPU model. Its backbone outputs differ from the first cluster's by only +0.02 mm [−0.00, +0.05].

What this rules out and what it does not:

  • "Simpler regularisation achieves the same result" is not supported here. An energy-penalised MLP becomes gate-safe, but its gain has a clustered CI including 0 at all four weights. Every penalised ODE has a significant gain: four weights, and six runs including two more seeds at w = 0.01 (−1.04, −1.14).
  • Head to head, the ODE is ahead at every weight. Its paired difference has a CI excluding 0 on at least one of minADE₅ / minFDE₅ at three of four weights, and it is never significantly worse on either.
  • Against an unpenalised MLP, BReF is not more accurate (paired CIs include 0). Its advantage there is gate reliance alone: 4–7 mm versus 308–439 mm.

We will state the limits with the result. Each added weight is a single run, while the seed spread at w = 0.01 is 0.11 mm. The two penalties are different functionals, so equal weights are not equal regularisation; matching on realised correction size gives the same picture, as shown in our reply to th2h. The margin is small, 0.4–0.9 mm minADE₅. So the evidence supports the ODE parameterisation helps, at this size, on this backbone. It does not show that BReF yields irreplaceable benefits, and we will not claim that.

Response to Reviewer uUZx (3/3): cross-dataset evidence and the scope of our claims (W2, W4, Q3, Q4)

Cross-dataset (W2, Q3): conceded. All of our experiments use nuScenes. Testing many backbones shows portability across architectures; it is not cross-dataset evidence, and we should not have let the two blur together. We could not port the full chain (backbone pre-training, frozen backbone, refiners, and the paired per-sample evaluation) to Argoverse 2 or WOMD within the rebuttal window, and we will not present a partial run as validation. We therefore make two changes:

  • every conclusion in the paper is now explicitly scoped to nuScenes;
  • cross-dataset validation is listed as required future work, not implied coverage.

We do not claim that nuScenes alone suffices to establish generality.

Scope of the robustness, stability and deployment claims (W4, Q4): we narrow all three to what was measured.

  • Robustness / OOD. Our "OOD" test added Gaussian noise to the backbone's output. That is a perturbation of the prediction, not a distribution shift of the input scene, and we relabel it as such. The geographic and manoeuvre splits are genuine distribution shifts. We will make only the narrow claim those splits support.
  • Stability under recursive use. We replaced the extrapolated decay with a measurement of repeated application on the full val split.
    • Applied repeatedly to the backbone's prediction (re-anchored), BReF converges to a fixed point within 2–3 steps. The fixed point lies 8 mm from the backbone, and minADE₅ stays within 0.05 mm of the deployed value.
    • Applied to its own output (feedback), every refiner drifts: the step ratio exceeds 1 at every iteration, and accuracy degrades after the first one or two steps. BReF drifts 6.9 cm after 8 steps, against 22.0 cm for the MLP, but it does drift.
    • We will claim the first behaviour and disclaim the second. This is open-loop self-iteration, not closed-loop control, and we will not call it that.
  • Closed-loop and quantisation. We ran no planner-in-the-loop experiment and no low-bit quantisation experiment. Both motivations are removed from the abstract and introduction.
  • Deployment value. We claim only what we measured. BReF trains 575,747 parameters (7.7% of the backbone's count) in 0.43× the wall time of joint training, and leaves every backbone weight untouched. It gives a small (~1 mm minADE₅) gain that is significant in each of three independent runs, and it transfers to a second backbone (LAformer: minFDE₅ −2.6 mm, p = 0.0014). We make no claim about downstream planning.

The diagnostic you singled out as the paper's strength is what we keep and sharpen. It shows that refiners with near-identical displacement error can differ by more than an order of magnitude in how much of their accuracy depends on the gate. We will present it as a behavioural probe, reported beside the correction size and a zero-residual control, so that it cannot be satisfied by a residual that is simply zero.

Area_Chair_74JKOpenReview ↗

Please help to join the discussion

Dear Reviewer uUZx,

The discussion deadline is approaching. Please help to join the discussion and offer the reviewer's opportunity to further respond to your concerns.

Thanks!

AC

Reviewer_th2hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BReF adds an identity-initialized residual ODE to a frozen trajectory predictor. A gate and a drift-magnitude penalty control the correction. The paper evaluates forecasting and transfer across backbones, and argues that the refiner preserves the backbone's performance and is stable under repeated application.

Critical strengths

The refiner can be trained without changing the backbone, and identity initialization recovers the original predictions. The paper includes extensive ablations on architecture, regularization, gate forcing, and transfer across 19 backbones.

Critical weaknesses
  1. Eq. (3) penalizes drift magnitude, which does not bound its spatial Jacobian. Appendix F.5.2 measures the Jacobian of the residual δ, whereas repeated refinement applies F(x) = x + δ(x). A small residual Jacobian does not imply contraction of this full map: δ(x) = 0.018x gives F'(x) = 1.018. The measured value therefore does not justify the recursive decay extrapolated in Table 13.

  2. Identity initialization guarantees that the initial prediction matches the frozen backbone, but does not guarantee that subsequent training preserves its performance. Appendix C relies on a smooth full-gradient descent argument, while the implemented loss includes a nonsmooth Laplace term and training uses stochastic updates. Small learning rates alone do not guarantee monotone expected loss descent, and lower training NLL does not guarantee lower test minADE. The claimed performance floor is therefore not established after training.

  3. Under normal inference, BReF and the matched MLP have nearly identical forecasting accuracy. The large reported advantage appears when the gate is forced open, but this diagnostic also favors a refiner that simply makes negligible corrections: an identically zero residual would incur no gate-forced degradation. The experiments need to distinguish useful, robust refinement from merely staying close to the backbone. A matched MLP with a comparable residual-magnitude penalty would help establish whether the benefit requires the ODE formulation.

Questions for authors

See weaknesses

5 comments

Response to Reviewer th2h (1/4): the contraction claim (W1)

We thank the reviewer for a precise and correct review. All three weaknesses identify real errors, and we act on each with new measurements.

About the new numbers. The machine that trained the submitted checkpoints no longer exists, and our archive kept aggregate tables only. We therefore rebuilt the full chain (LAformer stage-1 → frozen backbone → refiners) and ran every comparison below on it: N = 9041 nuScenes val samples, 95% CIs from a bootstrap clustered on keyframes (3076 clusters), and Wilcoxon signed-rank tests. Absolute values differ from the paper (backbone minADE₅ 1.2327 here vs 1.1864). Every question you raise is a relative comparison on identical samples, which the reproduction answers.

W1 is correct; we withdraw the contraction claim. Our probe differentiated the raw residual δ\delta alone, without the identity term or the gate. The deployed map is F(x)=x+g(x)⊙δ(x)F(x)=x+g(x)\odot\delta(x), whose Jacobian is JF=I+diag(g)Jδ+δ(∇g)⊤J_F=I+\mathrm{diag}(g)J_\delta+\delta(\nabla g)^\top, because the gate is itself a function of xx. A small ∥Jδ∥\|J_\delta\| bounds σ(JF)\sigma(J_F) only by 1+∥g∥∞σ(Jδ)+∥δ∥∥∇g∥>11+\|g\|_\infty\sigma(J_\delta)+\|\delta\|\|\nabla g\|>1, which cannot certify contraction. We first reproduced your counterexample as a self-test of the new probe: δ(x)=0.018x\delta(x)=0.018x gives σ(Jδ)=0.0180\sigma(J_\delta)=0.0180 and σ(JF)=1.0180\sigma(J_F)=1.0180.

What the latent-time penalty does bound is the energy of the correction. With fixed-step Euler, δ=1K∑kfk\delta=\frac1K\sum_k f_k, so ∥δ∥2≤1K∑k∥fk∥2\|\delta\|^2\le\frac1K\sum_k\|f_k\|^2, and the penalty controls E∥δ∥2\mathbb E\|\delta\|^2. This is an expected-magnitude bound with no constraint on ∂δ/∂x\partial\delta/\partial x, and we will state it as exactly that.

We then measured all four maps on the trained models (exact JVP power iteration, gate differentiated through; mean over 12 val batches):

refinerσ(Jδ)\sigma(J_\delta)σ(JF)\sigma(J_F)σ(JF)\sigma(J_F), gate = 1raw rms ∥δ∥\|\delta\|
BReF (ODE + smooth), run 1 / 2 / 30.73 / 0.94 / 1.581.38 / 1.28 / 1.241.54 / 1.62 / 2.290.078 / 0.090 / 0.084 m
ODE, no penalty2.691.203.150.647 m
MLP, no penalty2.011.352.441.381 m

No refiner is contractive (σ(JF)>1\sigma(J_F)>1 in every row), so your objection holds on the trained models, not only on the counterexample. What the penalty does do is visible in the gate-free columns. In all three BReF runs, σ(Jδ)\sigma(J_\delta) is lower than in both unpenalised models (0.73–1.58 vs 2.01–2.69). So is the gate-forced σ(JF)\sigma(J_F) (1.54–2.29 vs 2.44–3.15). The raw residual is also 7–18× smaller. In short, the penalty makes the raw residual map smaller and tamer, but it does not make the deployed map a contraction. We will describe it only in those terms.

Response to Reviewer th2h (2/4): measured recursion replaces Table 13 (W1, continued)

Rather than extrapolate, we iterated the trained refiner on the full val split under both readings of "repeated refinement":

  • feedback: xn+1=xn+g(xn)δ(xn)x_{n+1}=x_n+g(x_n)\delta(x_n);
  • re-anchor: xn+1=μ+g(xn)δ(xn)x_{n+1}=\mu+g(x_n)\delta(x_n), with μ\mu the backbone's prediction.

Feedback, BReF:

nnstep (m)drift from backbone (m)minADE₅step ratio
0 (backbone)—01.23263—
1 (deployed)0.00790.00791.23174—
20.00810.01601.231701.025
30.00830.02431.232401.024
50.00870.04141.235871.022
80.00940.06871.246211.026

After 8 iterations:

refinerfeedback: drift after 8 / step ratiofeedback: minADE₅ n=1→8n=1\to8re-anchor: fixed point reached / its drift
BReF0.069 m / 1.022–1.0261.2317 → 1.2462n = 2–3 / 0.0082 m
ODE, no penalty0.040 m / 1.041–1.0591.2320 → 1.2451n = 2–3 / 0.0045 m
MLP0.220 m / 1.016–1.0171.2314 → 1.2662n = 2–3 / 0.0264 m

The measurement settles the question in your favour. Under feedback, the step never shrinks: the ratio exceeds 1 at every iteration for every refiner. The iterate drifts away from the backbone, and after the first one or two steps accuracy degrades monotonically. Table 13's geometric decay does not happen, and we withdraw it.

Two results are, however, positive for BReF, and we will report both.

  1. Re-anchored application is stable. Every refiner reaches a fixed point within 2–3 steps. BReF's fixed point is 8 mm from the backbone, and its minADE₅ stays within 0.05 mm of the deployed value (1.23174 → 1.23179). Repeated application to the backbone's prediction is therefore safe. Repeated application to the refiner's own output is not, and we will disclaim it.
  2. The penalty slows the feedback drift. After 8 feedback steps BReF has drifted 6.9 cm and lost 14.5 mm minADE₅, against 22.0 cm and 34.8 mm for the MLP. The unpenalised ODE takes smaller steps but expands fastest (ratio up to 1.059).

(An earlier version of this probe used only 384 samples. On that subset, one refinement step happened to be worse than the backbone. The full-split numbers above replace it.)

Edits. We will:

  • delete "turns the refiner into a contraction" from the abstract and §3;
  • restate App. F.5.2 as a raw-residual Jacobian, explicitly not a bound on the update map, and add the full-map Jacobians above;
  • replace Table 13 with the measured recursion;
  • replace the contraction proposition with the energy bound and its limits.

All Jacobians are local, empirical, in-distribution quantities. We claim no global Lipschitz property.

Response to Reviewer th2h (3/4): the "performance floor" (W2)

W2 is correct; we reduce the claim to an initialisation property.

What survives is exact: the drift head is zero-initialised, so at step 0 δ≡0\delta\equiv0 and F(x)=xF(x)=x. The refiner starts at the frozen backbone, which is therefore recoverable by construction.

What does not survive is the Appendix C argument, for the reasons you give and one more we found in our own code:

  • the loss contains a Laplace NLL, which is non-smooth, so the smooth descent lemma does not apply;
  • training is stochastic (Adam, minibatch 16, cosine LR), so small steps do not give monotone expected-loss descent;
  • lower training NLL would not imply lower test minADE in any case;
  • best_mode is selected from the refined trajectories, so the training target moves with the refiner, and identity initialisation does not pin the loss surface the way the appendix assumes.

Edits. We remove "verifiable performance floor" and "cannot do worse" from the abstract, §1 and §5. We state instead: the refiner starts exactly at the backbone, so the backbone is recoverable by construction; this is an initialisation property, not a post-training guarantee.

In place of the guarantee, we report the measured post-training outcome. Each trained refiner is compared with its own zero-residual control: the same checkpoint, through the identical evaluation path, with δ forced to 0, which outputs the frozen backbone's prediction (minADE₅ 1.2327).

BReF (Δ vs own backbone)minADE₅ (mm) [95% CI]Wilcoxon pminFDE₅ (mm)MR₅
run 1−0.92 [−1.65, −0.20]4.3e-4−0.84 (n.s.)+0.01 pp (n.s.)
run 2−1.04 [−1.78, −0.31]0.041−0.30 (n.s.)n.s.
run 3−1.14 [−1.78, −0.56]4.5e-4−0.83 (n.s.)n.s.

Empirically, then, the trained refiner does not fall significantly below its backbone on any metric we track. minADE₅ improves significantly in all three independent runs, and minFDE₅ and MR₅ are unchanged within noise. We will report this as a measured outcome on nuScenes val, with its CI, and without "floor" or "guarantee" language.

The gain is small, about 1 mm. Its value is that it is reliable and comes without touching the backbone. Of the two refiners we trained three times each, only BReF clears the clustered CI in every run (unpenalised MLP: 2 of 3), and its run-to-run spread is half the MLP's (0.11 vs 0.23 mm).

Reviewer_1uHAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work address that a refiner should be judged by whether its raw correction is valid without the gate, and propose BReF, which is a plug-and-play module that can attach to any frozen multi-modal predictor and refines its trajectories by integrating a learned drift field. The proposed refiner is identity-initialised on the frozen backbone. The method is evaluated on nuScenes against twenty baselines, using both multi-mode and single-mode evaluation metrics.

Critical strengths

This work makes an interesting observation about gated refiners, showing that a learned gate can compress the raw corrections of refiners. And it proposes several principles for designing a refiner, including the raw correction should be valid without the gate, the backbone should act as a performance floow, and the refiner should be plug-and-play and preserves the backbone's modes. For the evaluation, the work evaluates about 20 predictors on nuScenes, which is broad and thorough.

Critical weaknesses
  1. The improvement from BReF itself is very small, and its significance is not verified. For the results of the proposed method, BReF, in Table 1, minADE5 is 1.185, minFDE5 is 2.324, which should refers to the improved LAformer backbone + BReF. In table 9a, it shows the frozen backbone minADE5 is 1.186, minFDE5 is 2.325. So from the frozen backbone, the improvement brought by BReF is minADE5: 1.186->1.185, minFDE5: 2.325->2.324. Most of the improvment to the original LAformer (1.213/2.367) comes from the improved backbone, rather than the proposed BReF refiner.
  • With this level of improvment, it is necessary to show the statistic significant, like the results of std over multiple random seeds, paired-sample tests against baselines. In Table 10, there are three seeds, showing that minADE5 values are 1.1858, 1.1855, 1.1865. Compared with the frozen backbone result of 1.1864, the improvment of BReF is not clear.

  • And for comparison, only the original LAformer results are 1.213/2.367 shown in the table, the frozen backbone results (where the role of BReF use) should also be included in Table 1 on all metrics for comparison, to show the effectiveness of the proposed refiner.

  • Evaluation results of BReF is reported inconsistently. It is 1.185 in Table 1, 1.1858 in Table 5, and 1.184 in Table 9.

  1. Table 2 compares each predictor with and without BReF, which mainly shows the effectiveness of the proposed refiner. But among the evaluated method, the main baseline, LAformer is not included,and neither is MBRM.

  2. A central claim is that freezing the backbone and identity-initialising the refiner is preferable to joint training. It would strengthen the claim, to show the results when the same backbone and refiner trained jointly, and compare with the frozen setting.

  3. About the gate discussion. This work claims that without a gate, the raw correction from refiner should also be valid. And the proposed method aims this. But from the results, in Table 5, default (with gate) minADE5 is 1.1858, gate-forced value is 1.1865, and frozen backbone is 1.1864. So with the raw correction (gate forced), the results are not better. The improvement is mainly from the gate. The results are inconsistent with the claim.

  • Also, the evaluation is largely focus on Δ5\Delta_5, which is (gate-forced - default). Δ5\Delta_5 is close to 0, does not essentially imply a valid correction, but could also be due to a nearly-zero raw correction value ∣∣δ∣∣||\delta||. A refiner that outputs ∣∣δ∣∣=0||\delta|| = 0 would also obtain Δ5\Delta_5 = 0.
Questions for authors

Same as in weaknesses

6 comments

Response to Reviewer 1uHA (1/4): attribution, significance, and inconsistent numbers (W1)

We thank the reviewer. All four weaknesses are correct. We answer each with new, paired, per-sample measurements, and we have changed the paper's claims accordingly.

About the new numbers. The machine that trained the submitted checkpoints no longer exists, and our archive kept only aggregate tables, no per-sample predictions. We therefore rebuilt the full chain (LAformer stage-1 → frozen backbone → refiners) and ran every comparison below on it, with N = 9041 nuScenes val samples. 95% CIs come from a bootstrap clustered on keyframes (3076 clusters, because agents in one scene are not independent), alongside a Wilcoxon signed-rank test. Absolute values differ from the paper; for example, the frozen backbone's minADE₅ is 1.2327 here versus 1.1864 there. Your questions are relative comparisons on identical samples, and those the reproduction answers.

Attribution: you are right. Our own development ladder [reported] reads: LAformer 1.2136 → frozen backbone 1.1864 → + refiner 1.1842. That is roughly 92% from the backbone and 8% from the refiner. Comparing Table 1 against the original LAformer credited BReF with the backbone's gain. We will add the frozen backbone as its own row on every metric in Table 1 and state BReF's contribution in mm. We will also state explicitly that this "frozen backbone" already contains its own internal refiner and a goal-anchor blend, and that BReF is a second, outer refiner on top of it.

Significance: also right. The paper's three seeds (1.1858 / 1.1855 / 1.1865) straddle the backbone's 1.1864, so they did not establish an improvement. On the reproduction, we compare each trained refiner with its own zero-residual control: the same checkpoint, through the identical evaluation path, with δ forced to 0. The control's output is exactly the frozen backbone's prediction.

refiner (3 independent runs)Δ minADE₅ vs own backbone (mm) [95% CI]Wilcoxon pruns with CI excluding 0seed std
BReF−0.92 [−1.65, −0.20] / −1.04 [−1.78, −0.31] / −1.14 [−1.78, −0.56]4.3e-4 / 0.041 / 4.5e-43 / 30.11 mm
MLP refiner−1.18 [−2.26, −0.11] / −0.73 [−1.80, +0.32] / −0.93 [−1.85, −0.06]0.067 / 0.005 / 0.0732 / 30.23 mm

BReF also improves top-1 minADE₁ by 1.4–2.1 mm, with the CI excluding 0 in all three runs. Its changes to minFDE₅ and MR₅ are not significant. So the honest size of BReF's contribution is about 1 mm minADE₅. It is small, but it is reliable: significant in every run, with half the MLP's run-to-run spread, and obtained without modifying a single backbone weight. That is how we will report it. The three runs are described as independent runs rather than pinned seeds, because we found that the original trainer did not pass the seed to its distributed workers (now fixed).

Inconsistent numbers: right. The values 1.185, 1.1858 and 1.184 are three different configurations quoted under one name. Our sweep gives w=0.01→1.1858w=0.01\to1.1858 and w=0.03→1.1843w=0.03\to1.1843, and a 15-epoch run gives 1.1854 [reported]. Every table will be regenerated from one named checkpoint, with its config and SHA256 recorded. The same audit showed that our sentence claiming every w≥0.001w\ge0.001 gives Δ5∈[0,1.3]\Delta_5\in[0,1.3] mm is false (w=0.001→13.0w=0.001\to13.0 mm), and we will correct it.

Response to Reviewer 1uHA (2/4): LAformer and MBRM in the plug-and-play table (W2)

LAformer: we ran it, but did not report it. LAformer was trained and evaluated in the same batch as the other plug-and-play backbones. It went through a different code path that wrote no result file, so the script that builds the table skipped it. This was a reporting error, not a missing experiment. The archived refined value is minADE₅ 1.2157 [reported]. Its backbone-only partner run was never made, and those weights are gone, so that number cannot be paired with anything.

We therefore trained a fresh matched pair on our rebuilt LAformer. The configuration is the archived one (540,803 trainable parameters, matching the archived count exactly). We evaluated it twice through the same script, checkpoint and sample order: once as trained, and once with δ ≡ 0, which outputs the frozen LAformer's own prediction.

metricfrozen LAformer+ BReFpaired Δ (mm) [95% CI]Wilcoxon pbetter / tie / worse
minADE₅1.24091.2403−0.58 [−1.12, −0.06]0.17195 / 8682 / 164
minFDE₅2.45282.4502−2.61 [−4.13, −1.14]0.0014208 / 8681 / 152
MR₅0.42870.4288+0.01 pp (n.s.)1.06 / 9028 / 7
minADE₁3.06913.0690−0.07 [−0.62, +0.48]0.86203 / 8646 / 192

BReF therefore transfers to LAformer. minFDE₅ improves by 2.6 mm, and both tests agree. minADE₅ improves by 0.6 mm: the CI excludes 0, but the signed-rank test does not. The effect is small. The refiner changes the prediction on only about 4% of samples and is neutral on the rest. Importantly, it does not significantly hurt any metric. We will add this row to Table 2 with its paired Δ and CI, labelled as a reproduction on a rebuilt backbone and not as the submitted 1.2157.

MBRM cannot host the adapter, and we will say why. MBRM is a post-hoc ensemble of seven separately trained predictors, not a single backbone. The plug-and-play adapter needs one backbone that exposes K-mode trajectories, per-mode scales and a scene encoding for the refiner to cross-attend to. An ensemble has no single scene encoding. Attaching BReF to it would mean defining a new method rather than applying ours. Table 2 will state this.

Response to Reviewer 1uHA (3/4): frozen vs joint training (W3)

Accepted, and we ran it. One disclosure first: in our code this comparison was not merely missing, it was impossible to run. Six separate mechanisms cut the gradient to the backbone, five of them silently: requires_grad=False, a no_grad() around the backbone, three .detach() calls, a forced eval(), and DDP wrapping only the refiner. A flag alone would have produced a run whose loss fell normally while its result stayed bit-identical to the frozen one. We fixed all six behind a switch that defaults to off. We verified the fix by counting gradients rather than trusting the flag: 315/347 backbone tensors receive a gradient with the switch on, and 0/347 with it off. The EMA was extended to cover the backbone, so both arms are evaluated as EMA snapshots.

Matched comparison. Same initialisation, refiner, data, loss, 30 epochs, global batch, seed and EMA. The only change is that the backbone trains at lr 1e-5.

frozen (BReF)joint, run 1paired Δ (frozen − joint) [95% CI]
minADE₅1.23181.2222+9.6 mm [+2.8, +16.4], p = 7e-10
MR₅0.44170.4153+2.6 pp [+1.7, +3.6]
minADE₁3.08413.0202+63.9 mm [+53.6, +74.1]
trainable params / wall time0.58 M / 96 min8.01 M / 223 min

Joint training is more accurate, and we concede it. We drop any reading of our title as "frozen beats joint". We then asked why it wins, using three more controls:

armminADE₅refiner gate
backbone fine-tune only (joint recipe, refiner frozen at identity)1.2223δ ≡ 0
joint, runs 1 / 2 † / 3 †1.2222 / 1.2215 / 1.2223exactly 0 in all three
BReF trained on the fine-tuned backbone1.22230 (identical output on 9041/9041 samples)
BReF trained on the joint backbone1.2226mean 0.031

† Trained on a second cluster with a different GPU model; its backbone outputs differ from the first cluster's by only +0.02 mm [−0.00, +0.05] minADE₅.

  1. The joint advantage is backbone fine-tuning. Fine-tuning the backbone alone matches every joint run within 0.8 mm minADE₅ (CIs ≈ ±2.3 mm). In all three joint runs the refiner's gate ends at exactly zero. "Joint refinement" did not produce a better refiner; it produced a fine-tuned backbone with the refiner as a passenger. The +9.6 mm is stable across the three joint runs (9.5–10.2 mm, each significant).
  2. BReF does not stack on a fine-tuned backbone. On the plain fine-tuned backbone it has no effect. On the joint backbone it is 0.40 mm worse [+0.09, +0.77]. We will not claim complementarity.
  3. We do not read the zero gate as the refiner deciding there is nothing to correct. The gate starts only 0.5 logit above its hard-sigmoid saturation, and a saturated gate receives no gradient from the trajectory loss. The mechanism remains open.

Positioning. BReF is not a substitute for fine-tuning when fine-tuning is available. It is for the common case where the backbone cannot or should not be retrained: shared across products, supplied as a black box, or already validated. There, BReF gives a reliable gain of about 1 mm minADE₅ while training 7.7% of the backbone's parameter count, in 0.43× the joint wall time, and it never touches a backbone weight. We will add these tables and state this scope.

Comments on the paper

1 comment
Area_Chair_74JKOpenReview ↗

Reviewers, please help to join the discussion.

Dear reviewers,

The authors have uploaded the rebuttal. Please help to join the discussion and the further process.

Thanks!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.