Sensor Generalist Policies for Fast Optimization of Robotic Sensor Suites

While data-driven policy synthesis techniques have received much attention in robotics, sensor suites are still largely designed through human intuition and costly trial-and-error. This choice is especially consequential as robot learning scales: sensors determine what information is captured in the experience used for learning and what feedback is available for control. We formulate sensor selection as jointly searching for a policy and its associated sensor specification that maximize downstream task performance, across different sensing levels. Evaluating candidate configurations by independently training policies, however, is prohibitively expensive across a combinatorial search space. We address this bottleneck with the *Sensor Generalist*, a single policy trained in simulation across diverse sensor configurations that amortizes policy learning across sensing budgets. The generalist enables lightweight evaluation and search, yielding high-performing policies and their associated sensor specifications at different sensing levels. We analyze the performance of this approach for real-world and simulated dexterous manipulation with proprioception and tactile sensors, and quadruped locomotion with LiDAR-based obstacle avoidance in simulation. Our approach matches or outperforms the evaluated baselines while reducing training cost, and discovers non-intuitive, often compact sensor configurations that support effective learned behaviors, yielding interesting insights into the sensory requirements of these tasks. Website: https://sites.google.com/view/sensor-selection