- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether trajectory planners act for the intended event or a correlated visual cue, even when their predicted trajectories appear reasonable. TREAT combines three stages: LOCATE compares braking aligned to cue onset and hazard-event onset; CURE fine-tunes on logged records covering all four cue/event configurations; and CERTIFY measures audit sensitivity using injected cue-locked output perturbations. Temporal identification is formulated under explicit latency and disturbance-isolation assumptions, and certificates are restricted to particular trained weights and audit conditions.
Critical strengths
- The paper addresses a meaningful limitation of trajectory accuracy: similar actions can arise from different dependencies. Its black-box timing audit provides evidence that complements representation probing and image interventions, and the trained positive control helps test sensitivity to a known cue-driven behavior.
- The evaluation includes useful controls and transparent limitations. It reports replica fleets, a disjoint-split Argoverse 2 evaluation, cue/window sensitivity, human references, and continued-training controls. Audited scenes are excluded from repair data, and failed repairs, train–audit overlap, and ADE degradation are explicitly disclosed.
- The certification guarantees are insufficiently validated. Calibration relies on bootstrapping already-cleared responses and transferring results between planners and cues. These procedures provide useful sensitivity estimates, but do not yet establish the claimed false-positive and missed-defect bounds.
Critical weaknesses
- The timing audit may confuse reasonable anticipation with incorrect cue reliance. A planner can react to an agent before it crosses the 20 m threshold. The planner is also measured using its next-three-second planned speed, while the human reference uses current logged speed. These differences weaken the causal interpretation, despite the reported geometry and window checks.
- Independent confirmation and repair evidence are limited. The sole confirmed natural case uses 11 scenes, eight overlapping original training scenes, without a new held-out confirmation set. The CURE–control difference has a paired p=0.16, and repair substantially worsens ADE. These results leave the repair benefit and its accuracy trade-off uncertain.
Questions for authors
- Could you measure the human reference using the same three-second future-speed quantity as the planner? How does LOCATE distinguish cue reliance from appropriate anticipation of an agent visible before crossing the 20 m threshold?
- What justifies treating bootstrap flag rates from cleared responses as false-positive bounds for causal dependence? Please clarify the assumptions behind transferring calibration between planners and cues, and validate them with known aligned negative controls.
- Can the confirmed case and repair effect be reproduced on independent unseen scenes? Please report uncertainty for the paired CURE–control difference and explain how the ADE increase should be weighed against the improved locking statistic.
- What fixes the scale of the response kernel in Proposition 2? With an unknown, unnormalized kernel, the observable mean appears to identify the contribution times the kernel's average height, rather than the coefficient alone. Please clarify the identification claim and its connection to LOCATE.
4 comments
Q2.The false-positive rate. A bootstrap of cleared responses does not justify a bound on false positives for causal dependence, and the revision no longer reads it as one. Such a bootstrap resamples the reading of one planner, and whether that planner is free of cue dependence is what the audit was meant to establish. The reviewer describes these numbers as sensitivity estimates. That is the right description, and we adopt it.
The false-positive rate is now obtained from planners whose alignment is known, as the reviewer asks. Each control planner follows a written rule that never reacts to the crosswalk. There are four rules. The planner ignores both events, brakes when the agent enters the danger zone, slows progressively while the agent approaches, or finishes braking before the agent reaches 20 m. Its noise is resampled from the pre-onset residuals of the audited planners. For each rule, each of three braking magnitudes and each of two noise models, we audit 10,000 planners on the same cue-first scenes with the unchanged flag rule. A second check needs no model of the planner. We flip the signs of each audited planner's own scene-level contrasts, which imposes the null, and count the flags over all sign assignments.
| Planner under test | nuScenes (11 scenes) | Argoverse 2 (17 scenes) |
|---|---|---|
| Nominal level of the flag rule | 2.5% | 2.5% |
| Audited planners, null imposed on their own contrasts (range) | 2.1–2.9% | 2.5–2.6% |
| Flagged LFM2-VL-1.6B instance, null imposed on its own contrasts | 2.7% | – |
| Rule-based, no response to either event | 2.3–2.6% | 2.3–2.5% |
| Rule-based, brakes when the agent enters the danger zone | 0.0–0.5% | 0.0–0.3% |
| Rule-based, slows progressively while the agent approaches | 0.0–0.5% | 0.0–0.3% |
| Rule-based, completes its braking before the agent reaches 20 m (lead varied) | 2.0–6.2% | 3.1–8.2% |
The rule stays at or near its nominal level except for the planner that finishes braking early, which is the case treated under Q1 in our first comment. We also removed the transfers between planners and between cues. Every instance is calibrated on its own contrasts. This includes the flagged one, which no longer takes a value from another planner. Every cue is calibrated on its own frames, and the rescaling by scene count is gone. The certificate is limited to the audited weights, cue set and scene population, and its detection limit refers only to the injected family of defects. This is in Section 4.3 and Appendix C.3 (Table 7) of the revision.
Q3 & W2.Unseen scenes, the repair arms. We tested the natural case on unseen scenes in two ways, and neither confirms it. The revision therefore reports the case as flagged on the pre-specified audit set and no longer as confirmed. First, we formed a second audit set with the same rule on the Boston part of the nuScenes test split, which is in no training manifest. Its agent onsets come from a camera detector, because the split has no public object annotations. On its twelve usable scenes the instance leans the same way but is not flagged (8/12, +0.78 m/s, p = 0.21). Second, the single-cause scenes introduced under Q1 in our first comment are unseen as well, and on the agent-only scenes the instance is not separated from the drivers (0.96 against 1.22 m/s). The repair comparison that follows is made on the eleven audit scenes, and we do not claim it beyond them.
On the eleven audit scenes, a single run per arm does not separate CURE from the control. The paired difference is +0.56 m/s, with a 95% interval of [−0.26, +1.38] (p = 0.16), and the revision says so. We therefore trained each arm four times and added a control that takes as many optimizer steps as CURE. CURE ends 0.88 m/s below this step-matched control (p = 0.003) and 0.52 m/s below the original control (p = 0.084). On the overlap, the eight scenes that are also in the training manifest were trained with the drivers' own futures as targets. A planner that memorized them would read like the drivers, who are not locked to the crosswalk on the audit scenes (+0.19 m/s, p = 0.58), so the overlap does not produce the flag. The three unseen scenes all lean toward the crosswalk (3/3, +1.98 m/s).
The ADE increase comes from the training recipe of the repair arms and not from the decorrelation records. Over four runs the ADE is 2.51 m for CURE, 2.83 m for the original control and 2.64 m for the step-matched control, against 1.62 m before repair. We do not claim that the repaired planner is the better planner overall. The claim is that the arm with decorrelation records lowers the cue-ward contrast on the audit scenes more than the arms without them. Whether 0.9 m of ADE is an acceptable price depends on the deployment, and an open-loop audit cannot settle it. The revision therefore reports both quantities for every arm (Table 2).
Q4.Proposition 2. The reviewer is right. With an unknown and unnormalized kernel, the window mean identifies the product , and the coefficient alone is not identified. The revision fixes the scale by the convention on the cue-first window. With this convention, is the mean cue-driven change of the response in that window, in m/s, and Proposition 2 is restated for this quantity. Two statements hold for any scale, provided the kernel has one sign on the window, which we now assume for a braking response. The window mean is zero only if is zero, and its sign is the sign of . The screening step needs only these two statements. LOCATE estimates the same window mean as the braking after the cue onset (Eq. 12) and compares it with the braking after the event onset, so it never used the scale of the kernel. The projection of Eq. 7 requires the shape of the kernel, and the revision now gives it as the estimator for a known kernel only. These changes are in Section 3.2 of the revision.
We hope these answers and the new controls resolve the concerns, and we thank the reviewer again for the questions, which made the claims of the paper more precise.
Thank you again for your careful review. As the discussion period is about to close, we would like to check whether our response and the revised paper have addressed your concerns.
If any point remains open, we would be glad to answer it in the time that is left. If your concerns are resolved, we would be grateful if you could consider updating your rating.
We thank the reviewer for the careful reading and the four questions. We answer the questions in order, in three comments, and each answer covers the weakness or the remark it belongs to.
Q1 & W1.The human reference and anticipation. We recomputed the human reference with the planner's quantity, the mean speed over the next three seconds of the logged future. In this quantity the drivers show no locking in either direction on both datasets.
Looking three seconds ahead moves the drivers' braking about 1.5 s earlier, so part of it falls into the cue-first window. The strong agent-ward contrast of the submission on nuScenes (−2.67 m/s) therefore came partly from the mismatch that the reviewer points out, and we no longer state that the drivers lock to the agent there. In neither quantity are the drivers locked to the crosswalk. One nuScenes scene is left out of the second row, because its log ends less than three seconds after the windows. Table 1 of the revision gives both references, so planners and drivers can be compared in the same quantity.
On anticipation, LOCATE separates it from cue reliance in one case and not in another. A planner that slows progressively while the agent approaches brakes more in the event window than in the cue-first window. Rule-based planners of this kind are flagged in at most 0.5% of 10,000 draws (table under Q2, in our second comment). A planner that has finished braking by the time the agent crosses 20 m gives a cue-ward contrast although its cause is the agent. Such planners are flagged in up to 6–8% when they finish early. Timing cannot decide this second case, so we added unseen scenes in which only one of the two causes is present. In 16 nuScenes scenes a crosswalk comes into view with no agent on the ego path. In 30 scenes an agent enters the danger zone with no crosswalk within 60 m. None of these scenes is in the audit set, the training manifest or the repair mixtures. The table reports the mean braking after each onset (Eq. 12), and the drivers are measured with the planner's quantity.
At a crosswalk without an agent, the flagged instance slows by 0.81 m/s and the drivers by 1.22 m/s. This slowing is ordinary caution and not a defect, and we changed Definition 2 to say so. Alignment now requires that the agent drives the braking with or without a cue. Misalignment means that the cue takes the agent's place. The agent-only scenes test this. There the drivers brake by 1.22 m/s and the cleared planners by 0.94–1.19 m/s. The flagged instance brakes by 0.96 m/s on the same scenes and in the same window, and the paired difference to the drivers has a 95% interval of [−0.69, +0.31]. The control therefore does not show that the crosswalk takes the agent's place. Section 5.3 and Appendix D.2 (Table 8) of the revision report both controls. A flag is confirmed only if the instance fails the agent-only one, so the revision reports this instance as flagged and no longer as confirmed.