acceptodds
Withdrawn from ICLR 2027

Physically Consistent Multimodal Representations for Persistent Target Tracking and Capture with Legged Manipulators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Persistent target tracking and capture in unstructured environments require physically grounded multimodal representations that preserve object identity and support interaction under uncertain observations. We present a unified framework that integrates physical consistency into multimodal representation, persistent target reasoning, and coordinated capture for legged manipulators. First, synchronized visual, thermal, and radar observations are organized into a pixel-aligned physical representation for constructing metric-semantic maps and estimating target states. Subsequently, an acquire–maintain–recover mechanism combines semantic compatibility with geometric, motion, and thermal consistency to preserve target identity under occlusion and visual degradation. The resulting representation and target confidence guide coordinated locomotion and manipulation during approach, alignment, and capture. Extensive real-robot experiments cover indoor and outdoor environments, day and night conditions, and four target categories. The framework achieves 91.3% capture success across 480 episodes per method and 84.8% post-occlusion recovery, compared with 76.7% and 61.9% for the strongest baseline. Component ablations further demonstrate complementary contributions from physical representation, semantic mapping, persistent reasoning, and coordinated execution.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multimodal Representation Learning, Embodied AI, Persistent Object Tracking, Legged Manipulation, Multimodal Perception

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_wgdfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors builds a system for a quadruped with an arm to follow and grab a target initialized by a text prompt, using RGB, thermal, and 4D radar. Radar and thermal are projected onto the RGB image so each pixel carries depth, Doppler velocity, and heat in addition to color. On top of this they run an open-vocabulary detector and a video segmentation tracker, and they check each tracking update against a constant-velocity prediction using position, Doppler, and thermal residuals. When the check fails, they re-detect the target and prefer candidates near the predicted position. A confidence threshold decides when the robot keeps approaching and when it attempts the grasp. All experiments are on the real robot with four toy targets, indoor/outdoor and day/night. The authors report 91.3% capture success against 76.7% for the best baseline, and better recovery after occlusion.

Critical strengths

This work addresses a practical problem: following and grabbing a specific object with a legged robot when the camera image is poor, e.g. at night, outdoors, or under heavy occlusion. The authors test the whole pipeline on a real robot, from sensing to grasping, and do not stop at perception benchmarks.

Critical weaknesses
  1. My major concern is the reliability of the results. The baseline numbers in Table 7 look implausible, and the main claims rest on them. Each cell has only 30 trials, yet across all 16 cells, every gap from KineMan-RL to BirdieRL and from BirdieRL to DQ-Net is exactly +1 or +2. That is 32 gaps with no ties and no reversals. In outdoor night, every gap is +2. Some columns are also repeated across conditions. For example, indoor-night Mouse and outdoor-night Rabbit are both 14/16/18/26. The supplement does not run the baselines, and its per-trial table cannot be matched to Table 7. It is also unclear how DQ-Net, which was evaluated only in simulation on a B1 + Z1, was deployed on a real Go2 + PiPER.
  2. The method has limited novelty. Radar and thermal projection onto the image, TSDF mapping, constant-velocity gating with Doppler, and re-detection near the predicted position are all standard, mostly from radar-based tracking in autonomous driving.
Questions for authors
  1. The results from table 7 look implausible. Please provide raw per-episode logs for all methods in Table 7, with episode ID, target, condition, layout, timestamp, and outcome. How do you explain why every gap between neighboring baselines is exactly +1 or +2, and why some columns repeat across conditions?
  2. How was each baseline adapted to the Go2 + PiPER? Please give the inputs used, whether it was retrained and how, and how sim-to-real was handled. DQ-Net was only evaluated in simulation on a different robot.
  3. The supplementary videos show a tabletop block-sorting arm, a red ball, and a robot walking in a corridor. It seems not relevant to robot tracking or chasing a target.

If authors can address point 1 and 2, I would consider raising my rating.

Reviewer_igdJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses visual, thermal and radar measurements to track and capture targets with a legged manipulator. It uses a metric–semantic map and acquire/maintain/recover reasoning to maintain target identity. Target confidence guides approach and capture. There are four inanimate target categories and four environmental conditions. Each method is evaluated on 480 capture episodes plus a separate 640-trial occlusion test. Reported capture success is 91.3% and post-occlusion recovery is 84.8%.

Critical strengths
  1. Keeping target identity is hard when visibility gets worse. The robot also needs to decide when to approach and capture. Using persistent tracking with confidence-dependent execution makes sense for this practical problem.

  2. The real-robot tests cover several targets and conditions. Appendices C–E give the failure denominators. They separate recovery frequency from recovery time conditional on success. They also state that confidence AUC and the capture coverage score are not calibrated probabilities. This helps with reading the results. The capture totals and ablation Fisher/Holm calculations match the supplied tables.

Critical weaknesses
  1. The full system works better but the contribution of the physical-consistency checks is not isolated. KineMan-RL, BirdieRL and DQ-Net have different observation channels (Appendix D.1). The 91.3% versus 76.7% capture result therefore changes what information each method gets. Sec. 4.5 swaps whole modules and changes sensing, persistence and control together.

The paper acknowledges this gap. A matched persistent tracker still needs tests with the checks on and off. Separate geometric/Doppler/thermal controls are needed to tell the cues apart. The mapping comparison doesn't resolve this for target identity and capture. Baseline adaptation and training budgets/checkpoints need more detail before the baselines can be judged fairly. So does tuning.

  1. The tested system is still hard to identify from Secs. 3–4 and Appendices A–D. There's too little detail on the deployed semantic model's architecture and training data/class protocol. It's not clear which tests use auxiliary online LiDAR. Recovery must occur within 5 s and stay stable for ten frames. What 3D-state error is allowed though?

sequence_count is unknown in the mapping CSV. The mean±SD is descriptive. It cannot establish precision or the number of independent sequences. Confidence AUC in Appendix D.3 is time-averaged confidence. Comparisons are hard to read without the score scales and time windows for each method. Appendix D.4 notes that the pooled Fisher/Wilson analyses ignore the dependence across the eight shared layouts/trajectories.

The supplementary reference implementation has a substitute linear semantic head and a kinematic execution surrogate. So it can't fill these deployment gaps. There is a difference in locking.py too. It compares consecutive radial-velocity summaries. Appendix B predicts radial velocity using the current line of sight and ego-motion compensation. The link needs explaining. This alone doesn't prove the deployed algorithm wrong.

The videos show rectangular-bar handling and front-gripper ball capture. The quadruped movement has no identifiable recovery sequence. It's not clear how they relate to the four target categories and the Go2–PiPER tests. A full deployment release is not required. Sufficient detail is needed to identify the actual system and the test protocols. These gaps prevent reliable assessment of the main result and support clear rejection.

Questions for authors
  1. What changes when only the physical-consistency checks are disabled? Keep the sensor streams, detector, persistent tracker, controller and tuning budget fixed. Can separate geometric/Doppler/thermal controls show what each cue contributes? How were the three task-adapted baselines trained, adapted and tuned? These tests address the paper's central mechanism.

  2. Which deployed semantic model, training/evaluation split and class set produced the results? Which experiments used online LiDAR? How many mapping sequences were evaluated? What metric-state/identity acceptance thresholds were used? For confidence AUC, give each method's score scales and time windows. Which reference-code settings match deployment, especially radial-velocity prediction and recovery acceptance?

  3. Please link the videos to the evaluated robot, targets and protocols. Can you link representative complete capture/recovery traces to the reported tests? Can matched starts or layout/trajectory grouping be used to assess the success differences? Please provide trial-level identifiers or appropriately grouped results. A new application or complete hardware-software release is not being requested.

Reviewer_FpN9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a pixel-aligned representation combining RGB, depth, radar Doppler, reflectivity, and thermal observations for language-specified target tracking and capture with a legged manipulator. Physical compatibility checks support acquisition, maintenance, and recovery, while confidence regulates coordinated execution. Real-robot experiments evaluate mapping, occlusion recovery, and capture.

Critical strengths

PMV preserves explicit physical measurements and channel validity on a common pixel lattice, connecting semantic mapping with metric target association while accommodating sparse observations. The acquire-maintain-recover mechanism validates propagated masks before updating identity and inhibits capture when association is unresolved. Table 2 reports 84.8% recovery and 2.8% identity switches, compared with 61.9% and 9.7% for the evaluated DQ-Net adaptation, supporting the system’s practical utility.

Critical weaknesses
  1. The experimental semantic network and training recipe remain insufficiently specified. Although Appendix D.1 states that mapping comparisons share a semantic backbone, the actual architecture, checkpoints, and training configurations are unclear; the released linear head is explicitly a substitute. This limits reproducibility and verification of the comparison conditions.
  2. The reference radial-motion check differs from Appendix B.2: it uses the last radial observation instead of projecting predicted velocity onto the current line of sight. The repository explicitly limits its correspondence to deployed code, so this does not establish an error in the robot experiments; it leaves a central consistency mechanism unverified.
Questions for authors
  1. Which architectures and checkpoints were used for the semantic and tracking frontends, and which components were trained or frozen? Please specify the actual f_theta architecture and its multimodal inputs.
  2. Could you provide the training data splits, losses, optimization settings, and deployed consistency thresholds, distinguishing them from the reference-only configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.