- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces the Sensor Generalist, a framework designed to automate robotic sensor suite selection (e.g., deciding which tactile taxels, joint encoders, or LiDAR rays to install or read). Rather than relying on human engineering intuition or training separate reinforcement learning (RL) policies for each candidate sensor configuration—which is computationally intractable due to combinatorial explosion—the authors propose to formulate the selection process as a joint optimization of policy learning and sensor configurations. The authors propose to first learn a generalist policy, which is able to take both sparse and dense sensory information as inputs and output control actions. Then with the learned generalist policy, they are able to select the optimal sensory configuration based on the task feedback.
Experiments across simulated and real-world benchmarks—including dexterous manipulation (tactile and proprioceptive sensing) and quadruped locomotion (LiDAR rays)— demonstrate that the framework matches or outperforms separately trained specialist policies while drastically cutting training costs and uncovering non-intuitive, compact sensor placements.
Critical strengths
(1) The problem proposed in the paper, automated sensory selection, is quite interesting. It is an practical and useful topic that is worth pursuing.
(2) In order to avoid extensive training and evaluation cost, the authors propose to first train a generalist policy and then optimize sensory selection based on the learned generalist policy.
(3) Experiments are extensive and across both simulation and real world.
Critical weaknesses
(1) The performance boost comes from policy architecture design, policy training and sensor configuration optimization with the generalist policy. In the ablation study part, the author demonstrates that distillation is essential for the training and the generalist policy acts as a good estimator for sensor selection. However, it seems like the performance improvement mainly comes from the generalist policy design and training, rather than sensor selection, which is the main topic for this paper.
[After rebuttal] The question is well addressed by rebuttal from the authors.
(2) As the sensor selection space is fixed when designing the generalist policy, the reviewer is wondering about whether it is possible to extend the method to unseen or out-of-distribution sensory space.
[After rebuttal] The authors mentioned that this will be included in the future work part.
Questions for authors
Please refer to the weakness part.
2 comments
(Nov 17) Rebuttal update
Dear reviewer Cjp6, as we near the end of the rebuttal period, we wanted to check in with you to see if there are any remaining concerns.
We addressed your concern about the performance boost of our method mainly coming from the generalist policy and not sensor selection, by pointing to our results that show that sensor selection does increase the performance of our generalist policy by a considerable margin in some tasks, and that the selected sensors from our generalist policy transfer to other specialist policies.
Next, we have updated the conclusion to include your question about how to address unseen sensors. We are grateful for your feedback, as the additional results generated from this discussion will further strengthen our manuscript. If satisfied, could you consider raising your score?
(Nov 12) Initial Response
Thank you for your review!
A central premise of our paper is that sensor selection should be viewed as a joint search over both sensor specifications and policy optimization, since a sensor configuration is only valuable in the context of a policy that is able to effectively exploit its information and achieve high return. Under this framework, we show that our method effectively selects sensors that maximize the returns of the generalist policy. In Figure 5, the difference between Generalist, Random and Generalist, Bsel isolates the gains from sensor selection while controlling for policy optimization. At low sensing levels, sensor selection vastly improves our method’s performance in Baoding, Bounce, and Forest Run.
Even when viewing sensor selection as agnostic to the policy optimization technique, we see evidence that our method discovers valuable sensor configurations which generalize to different policy synthesis techniques. This is visualized in the Additional Ablations section on our website. In the Specialist-PPO policy class, our discovered masks outperform random masks in Baoding, Bounce, and Rotate Real, showing positive transfer to expert policies. Similarly, in Specialist-Distill-Recon, discovered masks outperform random masks across all tasks.
Altogether, these results show that the performance boost of our method comes quite significantly from sensor selection, especially at low sensing budgets, rather than solely policy optimization; furthermore, the discovered sensors show evidence of transfer to specialist policies.
This is an interesting question! A straightforward approach would be to simply finetune the generalist on the new sensors, by adding in new modality encoders for the new sensors and finetuning the transformer on the combination of old and new sensor tokens. We leave this for future work.