acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Target-State Risk-Aligned Conditional Execution for Embodied Tracking

Abstract

Embodied visual tracking requires a robot to follow a language-specified person under occlusion, look-alike distractors, and ambiguous instructions. We show that improving tracking persistence can simultaneously increase safety violations. Using a compact B vision-language-action (VLA) tracker with a frozen backbone, we update only the planner head with proximal policy optimization (PPO). PPO improves success and following rates across all three EVT-Bench settings, but increases the collision rate on distracted tracking because episodes that previously ended in target loss can instead end in contact. We introduce , an execution-time target-state risk gate that preserves the learned policy and intervenes selectively. A lightweight risk head processes cues, including candidate boxes, identity margins, a learned range-and-bearing probe, and the policy's commands. When the calibrated risk exceeds a threshold, rescales the translational component of the next waypoint while leaving yaw unchanged. The executed action is identical to PPO's on more than of steps. In paired evaluations on identical episodes, reduces the distracted-tracking collision rate by percentage points, while further improving success and following rates. Matched slowdown controls without target-state timing do not reproduce this improvement. Concurrent evaluation reproduces a collision reduction of points, compared with points for identical-arm null pairs, indicating that the effect exceeds evaluation variability. These results show that target-state-aware execution can recover the safety cost of policy post-training with minimal intervention.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied visual tracking, vision-language-action models, compact VLA, PPO, runtime safety, risk gate, EVT-Bench, collision avoidance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Mh2COpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACE combines tracking-policy post-training with a learned risk head that selectively scales motion commands. It aims to reduce collisions while preserving embodied target-following performance.

Critical strengths

The paper provides comprehensive experimental results. The collision reduction under oracle perception is supported by the reported data.

Critical weaknesses

Practical validation is insufficient. All experiments are simulated without real world deployment.

Some test episodes were also used to train the risk predictor. The remaining episodes were used to choose the intervention threshold, so they are not an independent test set either. Performance on unseen episodes remains unclear.

Questions for authors

Does the method still reduce risks on new episodes that were heldout form training nor threshold selection?

What inputs in the 84 dim allow the risk predictor to detect walls or other obstacles?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Mh2C

Thank you for your thoughtful review and for noting our comprehensive experimental results. We ran new experiments directly addressing each question and added them to the revised PDF; our replies follow your numbering.


W2 & Q1 — Does the method still reduce collisions on new episodes held out from both training and threshold selection?

Yes. The revision adds a strictly disjoint test in which fitting, threshold selection, and measurement use three non-overlapping sets of scenes (Appendix U, p. 38). The 101 DT scenes are split at the scene level, balanced on collision rate (5.49%, 5.49%, 5.53%):

SplitScenesEpisodesRole
FIT32906Fitting the risk head (published recipe, unchanged)
CAL30223Selecting τ (matched to the deployed 2.006% trigger budget)
TEST39251Measurement only (unseen scenes, episodes, and targets)

On TEST (5 policy seeds, 1,255 paired episode rollouts across the 251 unseen episodes, concurrent-pair protocol):

ComparisonΔCR95% CIpPer-seed ΔCR
TRACE vs. no gate−3.67[−5.74, −1.67]0.0005−2.79, −5.18, −3.19, −3.59, −3.59

On unseen scenes, episodes, and targets, collisions fall by 3.67 points—in line with the full-set −3.96—with every seed negative and SR/FR intervals spanning zero (ΔSR +1.35 [−1.35, +4.06], ΔFR −0.17 [−1.41, +1.03]). §4, Table 2, and Appendix J now report this test.


Q2 — What inputs in the 84 dimensions allow the risk predictor to detect walls or other obstacles?

§4.1 now states EVT-Bench's exact collision rule: HumanCollision (which defines CR) triggers when the robot comes within 0.5 m of the followed person; walls and furniture do not enter CR (navmesh constraints govern scene geometry). All 712 collision terminations in our logs end with robot–target distance < 0.5 m.

The inputs that anticipate this event are on-board target-state cues (Appendix K): target box and scale, identity margin, candidate competition, probe distance/bearing bins, memory state, previous command, and predicted waypoints—computed from RGB and policy outputs with no simulator distance.

Appendix V (p. 38) evaluates predicting collision within H steps (733,203 steps, 6,900 episodes) against two privileged quantities the head never sees:

HRisk score (84 inputs)Wall/furniture clearance (privileged)True robot–target distance (privileged)
50.8510.5380.981
200.8020.5460.841
400.7720.5410.771
  • Anticipates collisions from on-board inputs: AUC 0.851 at H = 5, reaching the level of true robot–target distance at H = 40 (0.772 vs. 0.771).
  • Foresight beyond distance: Within every band of matched true robot–target distance, the score still separates steps followed by a collision within 20 steps (AUC 0.63–0.72) by reading commanded motion and planned waypoints.
  • Scene geometry preserved: Wall clearance carries near-chance signal (≈0.54), and since Eq. (2) only scales down translation, wall clearance is unchanged across percentiles (§4.4; median 0.159 m with gate vs. 0.155 m without on TEST).

W1 — Practical validation: oracle perception and real-world deployment.

We strengthened practical validation on both fronts:

  • Closed-loop safety without oracle perception (Appendix G, pp. 21–25, Table 12): Replacing benchmark boxes and association with learned RT-DETR + learned SigLIP ReID + an on-robot cue scorer (no benchmark input at run time; 10 replicate pairs across the 251 unseen TEST episodes, 2,510 paired rollouts, two per policy seed) cuts CR from 14.8% to 11.1% (ΔCR −3.75 [−5.38, −2.23], p < 10⁻⁵, negative in all 10 pairs), reaching the level of benchmark perception (−3.67; diff −0.08 [−2.19, +2.03]) while maintaining SR/FR (ΔSR +1.67 [−0.28, +3.71], ΔFR −0.50 [−1.47, +0.48]).
  • On-board robot deployment (Appendix Y, pp. 41–43, Figs. 10–11): RT-DETR detection, nearest-target selection, SigLIP probe, and the 84-input risk head run on a Unitree quadruped's on-board NVIDIA AGX Orin from live RealSense RGB with zero simulator inputs. RT-DETR runs at ≈15 fps (67 ms/frame) (above the 10 Hz control rate), and under teleoperation behind a walking person, the stack produces target detections (target #1) and online risk readouts (Figs. 10–11). Resolving a 3.67-point binary collision shift requires ≈780 paired trials at 80% power, which we measure across the 2,510 paired rollouts on unseen scenes (Appendix G).

We hope these new experiments and clarifications address your concerns, and we respectfully invite you to reconsider your assessment in light of the updated results.

Thank you for the following-up experiments and detailed responses, which have addressed most of my concern. I am raising the score to weak acceptance.

AuthorsOpenReview ↗

Thank You for Reconsidering Your Assessment

Thank you very much for reading our rebuttal and revised manuscript, and for raising your score! Your constructive questions directly motivated the scene-disjoint evaluation, the collision-foresight analysis, and the learned-perception and on-board validations, which substantially strengthened the paper.

Reviewer_MhP1OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the safety–performance trade-off in reinforcement learning for embodied visual tracking. Planner-head PPO improves tracking persistence but increases collisions. The proposed TRACE mechanism predicts collision risk and selectively reduces translational motion without changing the underlying policy. Experiments on EVT-Bench show improved tracking and safety under benchmark-provided perception, supported by matched slowdown controls, ablations, and repeated evaluations.

Critical strengths
  1. Useful finding about the safety cost of policy post-training. PPO improves tracking persistence but can replace target-loss failures with collision terminations. This finding highlights an important tension between task performance and safety.
  2. Careful validation of the main benchmark result. Multiple runs, matched slowdown controls, episode-level analyses, and identical-arm null comparisons strengthen the evidence. The transparent reporting of negative and inconclusive results is also valuable.
Critical weaknesses
  1. TRACE follows a risk-conditioned intervention framework explored in prior work [1, 2]. In particular, Recovery RL uses a learned state-action safety critic to trigger a recovery policy when estimated risk exceeds a threshold [1]. TRACE adds target-specific features and fixed action attenuation, but the paper does not establish why these changes constitute a substantial methodological advance. Matched slowdown and random-intervention controls demonstrate that intervention timing matters, but do not establish an advantage over existing safety critics or risk-aware controllers. Feature ablations primarily assess offline prediction rather than closed-loop safety. The strongest contribution therefore appears empirical rather than methodological. [1] Thananjeyan, B., et al. (2021). Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones. IEEE Robotics and Automation Letters, 6(3), 4915–4922. [2] Srinivasan, K., Eysenbach, B., Ha, S., Tan, J., & Finn, C. (2020). Learning to be Safe: Deep RL with a Safety Critic. arXiv preprint arXiv:2010.14603.
  2. The main results use simulator-provided target association, including privileged target identity information described in Appendix K. This matters because identity-related features contribute substantially to risk prediction. Under learned detection and association, Appendix G reports a collision-rate difference of −0.22 percentage points (95% CI: [−0.96, +0.52]), compared with −3.77 percentage points under benchmark perception. This does not establish ineffectiveness, but it leaves the practical safety benefit unproven. Overlapping training and evaluation episode identities further limit generalization; the held-out analysis does not fully establish transfer to unseen scenes and targets.
Questions for authors
  • What distinguishes TRACE methodologically from existing safety filters, particularly Recovery RL? Could a conventional safety critic achieve comparable results with the same features and intervention budget?
  • Can you demonstrate reproducible safety improvements under fully learned detection and target association?
  • Does TRACE generalize to unseen scenes and target identities under strictly disjoint evaluation? Among remaining collisions, how many result from missed interventions versus insufficient action attenuation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer MhP1 (1/2)

Thank you for a careful and constructive review. Every question led to a new closed-loop experiment in the revised PDF; our replies follow your numbering. This post addresses W1 & Q1; the second post addresses W2 & Q2 and Q3.


W1 & Q1 — What distinguishes TRACE methodologically from Recovery RL and safety critics, and could a conventional safety critic achieve comparable results with the same features and budget?

§2 now positions TRACE explicitly relative to Recovery RL [1] and safety critics [2], which learn Q_risk(s, a) (including from visual observations in [1]) via discounted TD backups and, in [1], hand control to a separately trained recovery policy above a threshold. TRACE adapts risk-conditioned gating to frozen-VLA embodied tracking along three task-specific axes:

  1. Explicit relational target-state conditioning vs. ego-motion/action features or raw frames: Rather than gating on (s_ego, a) alone or training an image-level critic, the head reads compact tracker-level relational cues (candidate boxes, association confidence, top-2 identity margin, target memory) where distracted-tracking collisions arise.
  2. In-place yaw-preserving modulation vs. training and switching to a separate recovery policy [1]: TRACE keeps the frozen VLA continuously in authority and scales only forward/lateral translation (Eq. 2; leaving retreat unattenuated, proven load-bearing in Appendix W, and keeping yaw at ×1.0 to maintain visual orientation toward the moving target), requiring zero auxiliary recovery-policy training. (Among training-based or residual baselines on our benchmark, PPO-Lagrangian was unstable in Table 24 and a supervised recovery residual lowered SR with higher seed variance in §4.4.)
  3. Post hoc episode-outcome calibration on frozen-policy logs vs. discounted TD backups [1, 2]: Fitted offline with no policy modification, avoiding early-pursuit over-triggering from backward risk discounting.

We tested both readings of Q1 ("ego-motion/action inputs" and "same 84 TRACE features") in closed loop on the 251 unseen TEST episodes (Appendix X, pp. 39–41, Table 34). Every gate shares FIT scenes, τ rate-matched on CAL to the 2.006% budget, and identical scales (5 seeds = 1,255 paired rollouts per row; 10 replicate pairs = 2,510 for the 39-input row):

Gate (closed loop on unseen TEST scenes)InputsOffline AUCClosed-loop ΔCR [95% CI]p
TRACE (deployed, episode-outcome fit)84 (target state)0.782−3.67 [−5.74, −1.67]0.0005
TRACE without identity channels770.759−2.31 [−4.22, −0.56]0.016
Ego motion + selected target box390.777−2.23 [−3.78, −0.80]0.004
Ego-only critic (ego motion + action, no target cues)270.708+0.48 [−0.80, +1.75]0.55
Same 84 TRACE inputs, TD-backup critic [1, 2] (γ = 0.95)84 (target state)0.792−5.26 [−7.57, −3.19] (ΔFR −1.44)< 10⁻⁵
  • Without explicit target-state cues (27 inputs): Ego-motion/action gating yields no measurable collision reduction (+0.48, p = 0.55). Paired on the same TEST episodes, TRACE improves ΔCR by 4.14 points [1.51, 6.77], p < 0.001, and adding only the selected target's box accounts for 2.71 [0.88, 4.62] (p = 0.003), whereas a nearest-bin proximity rule (Appendix H, −0.68, p = 0.10) does not reproduce the effect.
  • Same 84 features and budget (TD-backup critic [1, 2]): Given TRACE's 84 target-state inputs, a TD critic (γ = 0.95) also strongly cuts collisions (−5.26 [−7.57, −3.19]; paired diff vs. TRACE −1.59 [−3.90, +0.64]), confirming that our target-state representation transfers across critic objectives. At the same time, discounted TD backups significantly lower following rate (ΔFR −1.44 [−2.64, −0.24]), whereas TRACE's episode-outcome fit maintains it (ΔFR −0.17 [−1.41, +1.03]).
  • Closed-loop ablations replace offline AUC: Offline AUC (0.708, 3.44× lift) makes the ego-only head appear informative despite +0.48 in closed loop because fixed-log ranking ignores state-distribution shift; every feature claim now rests on paired closed-loop runs (§3.3, Appendix X).

Our contribution thus combines (i) uncovering how planner post-training converts target loss into collisions (§4.3), (ii) proving in closed loop that relational target-state cues—not ego-motion/action features—unlock safety gating across critic objectives while episode-outcome calibration preserves following rate, and (iii) realizing a lightweight post hoc forward-only, yaw-preserving gate on a frozen VLA with zero policy retraining.

AuthorsOpenReview ↗

Response to Reviewer MhP1 (2/2)

W2 & Q2 — Privileged identity information; reproducible safety improvements under fully learned detection and target association.

Yes. Appendix G (pp. 21–25, Table 12) demonstrates reproducible closed-loop safety gains under fully learned detection and target association on the 251 unseen TEST episodes (39 unseen scenes). All 84 inputs are deployable in form—none reads simulator pose or distance, and all 84 run live on-board in Appendix Y. Keeping the scene-disjoint risk head frozen (not refitted) and matching τ on CAL to the 2.006% budget, we replaced benchmark boxes and association step by step (5-pair rows = 1,255 paired rollouts, one per policy seed; 10-pair rows = 2,510 paired rollouts, two replicate pairs per seed; full ΔSR/ΔFR CIs in Table 12):

Candidate boxesTarget associationClosed-loop ΔCR [95% CI]p
BenchmarkBenchmark (App. U, 5 pairs)−3.67 [−5.74, −1.67]0.0005
Learned (RT-DETR)Benchmark (10 pairs)−2.75 [−4.34, −1.24]0.0007
Learned (RT-DETR)Grounding probe (5 pairs)−1.04 [−2.79, +0.72]0.29
Learned (RT-DETR)Learned ReID, designated once (10 pairs)−2.23 [−3.90, −0.68]0.008
Learned (RT-DETR)Learned ReID, no benchmark input (10 pairs)−3.75 [−5.38, −2.23]< 10⁻⁵
  • Fully learned stack (row 5): RT-DETR supplies every box; association uses contrastive 128-d SigLIP crop embeddings plus a logistic scorer over on-robot cues (language-probe probability and box geometry) fitted on FIT scenes only (τ on CAL; TEST untouched), with nothing read from the benchmark at run time. CR falls from 14.8% to 11.1%, ΔCR is negative in all 10 replicate pairs, SR and FR intervals span zero (ΔSR +1.67 [−0.28, +3.71], ΔFR −0.50 [−1.47, +0.48]), and the collision reduction reaches the level of benchmark perception on the same episodes (paired diff −0.08 [−2.19, +2.03]).
  • Why the submitted learned variant (−0.22) differed (row 3 vs. row 5): Under the same frozen head, CAL rule, and TEST episodes, raw grounding-probe association yields −1.04 (p = 0.29), matching the submitted −0.22 and multi-person H = 5 AUC dropping 0.837 → 0.686 while < 2-person AUC holds (0.886 → 0.855). Upgrading only association to learned ReID improves ΔCR by −2.71 [−4.70, −0.76] and ΔSR by +4.38 [+1.31, +7.45] on the same episodes, restoring the full closed-loop safety effect.

Q3 — Strictly disjoint evaluation; missed interventions vs. insufficient attenuation.

Disjoint evaluation (Appendix U, p. 38). Splitting the 101 DT scenes into FIT (32 scenes, 906 episodes; head fitting), CAL (30, 223; τ selection), and TEST (39, 251; measurement only) ensures strictly unseen scenes, episodes, and targets. On TEST (5 seeds, 1,255 paired rollouts): ΔCR −3.67 [−5.74, −1.67], p = 0.0005 (all 5 seeds negative, in line with full-set −3.96; ΔSR +1.35, ΔFR −0.17).

Remaining collisions (Appendix W, p. 39). Across the 712 collisions remaining under the gate:

Remaining collisions (712 total)CountShare
Gate fired within the last 20 steps45263.5%
Gate never fired (missed intervention)26036.5%

Thus 36.5% are missed triggers, while 63.5% follow a recent firing (with the target actively held, median memory age 0). Splitting by final-step commanded motion clarifies what slowdown covers: on TEST, deployed s_f = 0.25 removes 44% of forward-ending collisions (105 → 59), whereas 48.9% of all remaining collisions (348/712; 67/126 on TEST) end while the policy is already commanding reverse motion (67 → 67).

Testing reverse attenuation (Appendix W). Applying ×0.25 to reverse commands too (5 seeds on TEST) still cuts forward-ending collisions (100 → 56) but nearly triples reverse-ending ones (67 → 190), raising overall CR by +8.61 points [+5.66, +11.79], p < 10⁻⁵ paired directly against TRACE—confirming Eq. (2)'s forward-only design and showing why close-range retreating encounters lie outside a velocity-damping gate's scope (§3.3).

We hope these new closed-loop experiments and analyses resolve your concerns, and we respectfully invite you to reconsider your assessment in light of the updated results.

Thanks for the detailed rebuttal and additional experiments. Most of my concerns are addressed now.

My main remaining issue is that the safety improvements are still largely limited to the distracted-tracking setting, and the real-robot results don’t yet show closed-loop safety benefits. I think this is still a limitation of the current evaluation.

AuthorsOpenReview ↗

Thank You + Context on Task Selectivity (DT vs. STT/AT) and Hardware Scope

Thank you for reading our rebuttal and additional experiments closely, and for confirming that most of your concerns are now addressed! On the two points you noted, we share brief context from our results:

  1. Behavior across tracking settings (DT vs. STT and AT): Across EVT-Bench, TRACE intervenes selectively where post-training actually introduces collision risk (Table 1b, §4.5):

    • In Single-Target Tracking (STT) (unambiguous single person), planner-head PPO causes no collision increase over the SFT base (4.13% → 4.09%), and the DT risk gate stays silent (0% firing rate)—avoiding unnecessary interventions when multi-person crowding risk is absent (87.51% SR).
    • In Ambiguity Tracking (AT), where PPO's collision increase is milder (7.39% → 8.74%), transferring the DT gate unchanged still lowers CR across all five runs (8.74% → 8.01%, §4.5 & Appendix I), while AT-specific calibration prioritizes preserving SR and FR (64.41% / 72.57%).
    • In Distracted Tracking (DT)—crowded multi-person scenes where look-alike pedestrians create identity competition and PPO drives collisions up sharply (11.01% → 13.78%)—TRACE removes more than the entire post-training collision surge (9.83%, below the SFT base, and ΔCR −3.75, p < 10⁻⁵ on unseen scenes) while improving both SR and FR.
  2. Closed-loop learned-perception safety & on-board hardware execution: While statistical closed-loop collision rates require hundreds of paired multi-person encounters (≈780 pairs at 80% power for a 3.67-point shift; Appendix Y notes the hardware scope explicitly), our new experiments establish both end-to-end closed-loop safety gains with zero runtime simulator inputs under learned RT-DETR detection + learned SigLIP ReID across 2,510 paired rollouts on unseen scenes (ΔCR −3.75 [−5.38, −2.23], p < 10⁻⁵, negative in all 10 replicate pairs; Appendix G, Table 12), and real-time on-board execution of the perception and 84-input risk-scoring pipeline on the Unitree quadruped's AGX Orin (≈15 fps > 10 Hz control rate from live RealSense RGB, zero simulator inputs; Appendix Y).

Thank you again for engaging so constructively with our work—your feedback substantially strengthened the paper, and we hope this context is helpful as you finalize your assessment.

Reviewer_34gAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper raises the concern that increasing tracking persistence through reinforcement learning (PPO) in embodied visual tracking can lead to safety violations and therefore proposes TRACE, which adaptively intervenes by rescaling actions when collision risks are detected to improve tracking safety. Experiments on distracted tracking show that TRACE decreases the collision rate while improving the tracking success rate.

Critical strengths
  1. This paper raises a practical concern about the safety risks associated with reinforcement-learning-based improvements in tracking persistence and provides experimental evidence for this concern. This finding is interesting and motivates evaluating tracking policies beyond success rate alone.

  2. TRACE is lightweight, leaves the underlying policy unchanged, and modifies only a small fraction of executed actions. Various experiments distinguish its effects from those of slowdown controls, highlighting the importance of adaptive intervention.

Critical weaknesses
  1. The authors do not evaluate TRACE on a real robot, which limits the evidence for its reliability in real-world deployment. In particular, inputs including candidate bounding boxes, target associations, identity matching scores, top-2 identity margins, and visible-person counts are derived from simulator-provided information, whereas their estimated counterparts may be inaccurate in real-world settings. The collision reduction of the learned-perception variant is not statistically significant under concurrent re-evaluation. Further assessment under realistic perception errors and end-to-end real-robot evaluation are therefore needed.

  2. Since TRACE intervenes only by rescaling translational commands, its benefits appear most directly relevant to risks caused by overly aggressive motion. The gate itself does not generate alternative routes or recovery actions, leaving it unclear how well it addresses situations requiring active replanning or correction of an incorrectly tracked target. The authors should clarify the scope of tracking safety that TRACE is intended to address and the limitations of its intervention mechanism.

Questions for authors
  1. As mentioned in Weakness 1, how do errors in person detection and target association affect TRACE’s intervention decisions and safety benefits, and how robust is the complete system in real-robot deployment?

  2. As mentioned in Weakness 2, what types of tracking-related safety risks can TRACE address, and which remain outside the capabilities of its action-rescaling mechanism?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 34gA

Thank you for your clear and constructive review. Both questions now have measured closed-loop and on-robot answers in the revised PDF; our replies follow your numbering.


W1 & Q1 — How do person-detection and target-association errors affect TRACE's safety benefits, and how robust is the system in real-robot deployment?

We isolated detection from association errors by replacing benchmark perception one module at a time under the concurrent-pair protocol on strictly unseen TEST scenes (Appendix G, pp. 21–25, Table 12; 32 FIT / 30 CAL / 39 TEST scenes, 251 unseen episodes). The scene-disjoint risk head is frozen and τ is matched on CAL to the 2.006% budget (5-pair rows = 1,255 paired rollouts, one per policy seed; 10-pair rows = 2,510 paired rollouts, two replicate pairs per seed; full ΔSR/ΔFR CIs in Table 12):

Candidate boxesTarget associationClosed-loop ΔCR [95% CI]p
BenchmarkBenchmark (App. U, 5 pairs)−3.67 [−5.74, −1.67]0.0005
Learned (RT-DETR)Benchmark (10 pairs)−2.75 [−4.34, −1.24]0.0007
Learned (RT-DETR)Grounding probe (5 pairs)−1.04 [−2.79, +0.72]0.29
Learned (RT-DETR)Learned ReID, designated once (10 pairs)−2.23 [−3.90, −0.68]0.008
Learned (RT-DETR)Learned ReID, no benchmark input (10 pairs)−3.75 [−5.38, −2.23]< 10⁻⁵
  • Significant safety gain under fully learned perception in concurrent evaluation (row 5): With RT-DETR boxes plus contrastive SigLIP ReID and an on-robot cue scorer (fitted on FIT scenes only; no benchmark input at run time), CR falls from 14.8% to 11.1%, ΔCR is negative in all 10 replicate pairs, SR/FR intervals span zero (ΔSR +1.67 [−0.28, +3.71], ΔFR −0.50 [−1.47, +0.48]), and the collision reduction reaches the level of benchmark perception (−3.67; paired diff −0.08 [−2.19, +2.03]).
  • Detection errors have minor impact (row 2): RT-DETR boxes with benchmark association cut CR by −2.75 points (p = 0.0007, ΔSR +1.91 [+0.12, +3.75]), and < 2-person risk ranking stays intact (H = 5 AUC: 0.886 → 0.855).
  • Multi-person association governs the benefit (row 3 vs. row 5): RT-DETR + grounding probe gives ΔCR −1.04 (p = 0.29, consistent with the submitted −0.22 and ≥ 2-person AUC dropping 0.837 → 0.686). Upgrading only association to learned ReID improves ΔCR by −2.71 [−4.70, −0.76] and ΔSR by +4.38 [+1.31, +7.45] on the same TEST episodes.

On-board physical robot deployment (Appendix Y, pp. 41–43, Figs. 10–11). We ported and ran the perception and risk-scoring pipeline—RT-DETR detection, nearest-target selection, SigLIP grounding probe, and the 84-input risk head—on a Unitree quadruped's on-board NVIDIA AGX Orin:

  • Real-time on-board sensing: RT-DETR runs at ≈15 fps (67 ms/frame) from live RealSense RGB (above the 10 Hz control rate), and all 84 features compute live on-board with zero simulator inputs.
  • Live test & evaluation protocol: while teleoperating the quadruped behind a walking person, the stack produces target detections (target #1) and online risk readouts (Figs. 10–11). Because resolving a 3.67-point binary collision shift requires ≈780 paired trials at 80% power, closed-loop collision effects are measured across the 2,510 paired rollouts on unseen scenes (Appendix G).

W2 & Q2 — What tracking-related safety risks can TRACE address, and which lie outside its action-rescaling mechanism?

§3.3 now states this boundary explicitly. Eq. (2) rescales forward/lateral translation while preserving yaw (×1.0), keeping the person in view without switching policies:

  • In scope: collisions driven by the policy's own closing motion while following a target—the failure mode planner-head PPO creates when trading target loss for collisions (§4.3).
  • Outside by design: obstacles requiring routing around them; recovering a mis-associated target; other agents walking into the robot; and close-range encounters whose remedy is faster retreat rather than slowdown.

Appendix W (p. 39) quantifies this boundary. Deployed s_f = 0.25 removes 44% of forward-ending collisions (105 → 59 on TEST). Of the 712 remaining collisions, 36.5% (260/712) are missed triggers and 63.5% (452/712) follow a firing, with 48.9% (348/712) ending while the policy already commands reverse motion; attenuating reverse commands too (×0.25) nearly triples reverse-ending collisions (67 → 190; +8.61 CR paired vs. TRACE, p < 10⁻⁵), confirming Eq. (2)'s forward-only rule. Obstacle replanning and target re-acquisition (such as ReID above) act as complementary modules alongside TRACE.

We hope these new experiments and clarifications address your concerns, and we respectfully invite you to reconsider your assessment in light of the updated results.

I thank the authors for their response and additional experiments. I appreciate the authors' efforts during the rebuttal period.

However, I would like to see experiments on TRACE's end-to-end closed-loop robustness and safety on a real robot under real-world perception errors. The additional real-robot experiments reported by the authors are limited to perception and risk scoring under teleoperation. The rebuttal therefore does not sufficiently address my concerns. I decide to maintain my score.

AuthorsOpenReview ↗

Thank You for Engaging + Summary of Closed-Loop Learned-Perception and On-Board Results

Thank you for reading our response and revised manuscript, and for acknowledging our additional experiments during the rebuttal period.

To briefly summarize what the new experiments establish on the points raised in your review:

  • Closed-loop safety under realistic perception errors (W1, Q1): Under the concurrent-pair protocol on strictly unseen scenes (2,510 paired rollouts), replacing all simulator boxes and association with learned RT-DETR detection and learned SigLIP ReID (zero simulator inputs at run time) yields a statistically significant collision reduction of ΔCR −3.75 [−5.38, −2.23] (p < 10⁻⁵, negative in all 10 replicate pairs; CR 14.8% → 11.1%) while maintaining SR and FR—isolating multi-person association as the key factor (ΔCR −2.71 improvement over probe association on the same episodes) and resolving the learned-perception gap from the initial submission (Appendix G, Table 12).
  • On-board physical robot execution & statistical safety protocol (W1, Q1, W2, Q2): On the physical Unitree quadruped, running the perception and 84-input risk-scoring pipeline live on the on-board AGX Orin (RT-DETR at ≈15 fps > 10 Hz control rate from live RealSense RGB, zero simulator inputs; Appendix Y) verifies real-time embedded execution on real-world camera streams, while controlled multi-person rollouts provide the statistical power (≈780 paired encounters at 80% power for a 3.67-point shift) needed to measure binary human-collision reductions safely and reproducibly alongside our explicit intervention-scope validation (§3.3, Appendix W).

We sincerely thank you again for your constructive feedback, which directly motivated both the five-stage closed-loop learned-perception study and the on-board robot deployment in the revised paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response to All Reviewers

We thank all three reviewers for their constructive feedback. Four requests recur across the reviews: (1) a strictly disjoint evaluation on unseen scenes and targets (Mh2C, MhP1), (2) closed-loop safety under fully learned detection and target association (MhP1, 34gA, Mh2C), (3) closed-loop comparisons against ego-only gating and TD safety-critic fitting [1, 2] (MhP1), and (4) physical-robot validation plus explicit intervention scope (Mh2C, 34gA). We ran new experiments for every point and added them to the revised PDF (end of Appendix G, pp. 21–25, and new Appendices U–Y, pp. 38–43; existing appendix letters A–T are unchanged).

Substantive revisions to existing text and captions are highlighted in blue; new Appendices U–Y are labeled "Added in revision".

Summary of new evidence

QuestionNew experimentResultRevised PDF
Unseen scenes & targets (Mh2C W2/Q1, MhP1 W2/Q3)Head fitted (32 scenes), τ calibrated (30 scenes), measured on 39 strictly disjoint scenes (251 episodes; 5 pairs, 1,255 paired rollouts)ΔCR −3.67 [−5.74, −1.67], p = 0.0005; all 5 seeds negative (full set: −3.96); SR/FR maintained (ΔSR +1.35, ΔFR −0.17)App. U (p. 38), §4, Table 2
Fully learned perception (MhP1 W2/Q2, 34gA W1/Q1, Mh2C W1)Closed loop on unseen TEST scenes with RT-DETR + learned SigLIP ReID; no benchmark input at run time (10 replicate pairs, 2,510 paired rollouts)ΔCR −3.75 [−5.38, −2.23], p < 10⁻⁵, all 10 pairs negative, at the level of benchmark perception (diff −0.08 [−2.19, +2.03]); ΔSR +1.67, ΔFR −0.50; upgrading probe → ReID improves ΔCR by −2.71 [−4.70, −0.76]App. G (pp. 21–25), Table 12, §4
Target-state vs. ego-only gating (MhP1 W1/Q1)Closed-loop feature ladder on unseen scenes at the same 2.006% budget and calibration ruleEgo-motion/action gating (27 inputs) yields no measurable collision drop (+0.48, p = 0.55); paired on the same episodes, TRACE improves ΔCR by 4.14 points [1.51, 6.77], p < 0.001 (2.71 [0.88, 4.62] from adding the target's box)App. X (pp. 39–41), §2, §3.3
Same 84 inputs with TD-backup critic (MhP1 W1/Q1)Safety-critic TD fit [1, 2] (γ = 0.95) given TRACE's 84 target-state inputs and 2.006% budgetTarget-state inputs also unlock a TD critic (ΔCR −5.26 [−7.57, −3.19]; diff vs. TRACE −1.59 [−3.90, +0.64]), while TRACE's episode-outcome fit avoids TD's following-rate drop (ΔFR −0.17 vs. −1.44 [−2.64, −0.24])App. X (pp. 39–41), Table 34
On-board robot deployment (Mh2C W1, 34gA W1/Q1)RT-DETR + nearest-target + SigLIP probe + 84-input risk head on a Unitree quadruped's on-board AGX OrinRT-DETR runs at ≈15 fps (above the 10 Hz control rate) from live RealSense RGB with zero simulator inputs; target detections and online risk readouts are produced on board under teleoperationApp. Y (pp. 41–43), Figs. 10–11
What the risk score encodes (Mh2C Q2)EVT-Bench defines collision as robot–target distance < 0.5 m; predict collision within H steps (733,203 steps) vs. privileged distancesScore AUC 0.851 at H = 5, reaching true robot–target distance at H = 40 (0.772 vs. 0.771) and separating collisions at matched distance (AUC 0.63–0.72); scene clearance AUC is 0.54, while wall clearance is unchanged (§4.4)App. V (pp. 38–39), §4.1
Remaining collisions & scope (MhP1 Q3, 34gA W2/Q2)Attribute the 712 remaining collisions; ablate attenuating reverse motion tooDeployed s_f = 0.25 removes 44% of forward-ending collisions (105 → 59); 48.9% of residuals end while already retreating, where attenuating reverse commands raises CR (+8.61 paired vs. TRACE, p < 10⁻⁵)App. W (pp. 39–40), §3.3

Main-text and appendix revisions

  • §2: Task-specific positioning relative to Recovery RL [1] and safety critics [2].
  • §3.3: Explicit scope of the intervention, forward-only ablation reference (App. W), closed-loop value of target-state inputs (App. X), and episode-terminal label wording aligned with App. J.
  • §4.1: Explicit EVT-Bench collision definition (robot within 0.5 m of the followed person).
  • §4, Table 2, App. G, App. H & App. J: Scene-disjoint test (App. U) replaces the validation-split check as the independent test; §4 and App. G report the diagnostic and 5-row closed-loop perception ladder.
Area_Chair_3sLJOpenReview ↗

AC Reminder: Please Confirm Whether Your Concerns Are Addressed

Dear Reviewers,

The authors replied on 13 Nov with a common response and individual answers, backed by new closed-loop, scene-disjoint and on-robot experiments in the revised appendix. Please read the response addressed to your review and post a short reply confirming whether your concerns are addressed, or listing what remains unresolved and what would settle it. If you feel you do not have sufficient expertise to evaluate parts of the submission, please let me know so that I can arrange additional expert input. Either way, a response before the discussion period ends would be much appreciated.

Thank you for your time. Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.