acceptodds
Under review as a conference paper at ICLR 2027

Sensor Generalist Policies for Fast Optimization of Robotic Sensor Suites

Abstract

While data-driven policy synthesis techniques have received much attention in robotics, sensor suites are still largely designed through human intuition and costly trial-and-error. This choice is especially consequential as robot learning scales: sensors determine what information is captured in the experience used for learning and what feedback is available for control. We formulate sensor selection as jointly searching for a policy and its associated sensor specification that maximize downstream task performance, across different sensing levels. Evaluating candidate configurations by independently training policies, however, is prohibitively expensive across a combinatorial search space. We address this bottleneck with the *Sensor Generalist*, a single policy trained in simulation across diverse sensor configurations that amortizes policy learning across sensing budgets. The generalist enables lightweight evaluation and search, yielding high-performing policies and their associated sensor specifications at different sensing levels. We analyze the performance of this approach for real-world and simulated dexterous manipulation with proprioception and tactile sensors, and quadruped locomotion with LiDAR-based obstacle avoidance in simulation. Our approach matches or outperforms the evaluated baselines while reducing training cost, and discovers non-intuitive, often compact sensor configurations that support effective learned behaviors, yielding interesting insights into the sensory requirements of these tasks. Website: https://sites.google.com/view/sensor-selection

Primary area:
applications to robotics, autonomy, planning
Keywords:
sensor selection, generalist policy, reinforcement learning, robotics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Cjp6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces the Sensor Generalist, a framework designed to automate robotic sensor suite selection (e.g., deciding which tactile taxels, joint encoders, or LiDAR rays to install or read). Rather than relying on human engineering intuition or training separate reinforcement learning (RL) policies for each candidate sensor configuration—which is computationally intractable due to combinatorial explosion—the authors propose to formulate the selection process as a joint optimization of policy learning and sensor configurations. The authors propose to first learn a generalist policy, which is able to take both sparse and dense sensory information as inputs and output control actions. Then with the learned generalist policy, they are able to select the optimal sensory configuration based on the task feedback.

Experiments across simulated and real-world benchmarks—including dexterous manipulation (tactile and proprioceptive sensing) and quadruped locomotion (LiDAR rays)— demonstrate that the framework matches or outperforms separately trained specialist policies while drastically cutting training costs and uncovering non-intuitive, compact sensor placements.

Critical strengths

(1) The problem proposed in the paper, automated sensory selection, is quite interesting. It is an practical and useful topic that is worth pursuing.

(2) In order to avoid extensive training and evaluation cost, the authors propose to first train a generalist policy and then optimize sensory selection based on the learned generalist policy.

(3) Experiments are extensive and across both simulation and real world.

Critical weaknesses

(1) The performance boost comes from policy architecture design, policy training and sensor configuration optimization with the generalist policy. In the ablation study part, the author demonstrates that distillation is essential for the training and the generalist policy acts as a good estimator for sensor selection. However, it seems like the performance improvement mainly comes from the generalist policy design and training, rather than sensor selection, which is the main topic for this paper.

[After rebuttal] The question is well addressed by rebuttal from the authors.

(2) As the sensor selection space is fixed when designing the generalist policy, the reviewer is wondering about whether it is possible to extend the method to unseen or out-of-distribution sensory space.

[After rebuttal] The authors mentioned that this will be included in the future work part.

Questions for authors

Please refer to the weakness part.

2 comments
AuthorsOpenReview ↗

(Nov 12) Initial Response

Thank you for your review!

The performance boost comes from policy architecture design, policy training and sensor configuration optimization with the generalist policy. In the ablation study part, the author demonstrates that distillation is essential for the training and the generalist policy acts as a good estimator for sensor selection. However, it seems like the performance improvement mainly comes from the generalist policy design and training, rather than sensor selection, which is the main topic for this paper.

A central premise of our paper is that sensor selection should be viewed as a joint search over both sensor specifications and policy optimization, since a sensor configuration is only valuable in the context of a policy that is able to effectively exploit its information and achieve high return. Under this framework, we show that our method effectively selects sensors that maximize the returns of the generalist policy. In Figure 5, the difference between Generalist, Random and Generalist, Bsel isolates the gains from sensor selection while controlling for policy optimization. At low sensing levels, sensor selection vastly improves our method’s performance in Baoding, Bounce, and Forest Run.

Even when viewing sensor selection as agnostic to the policy optimization technique, we see evidence that our method discovers valuable sensor configurations which generalize to different policy synthesis techniques. This is visualized in the Additional Ablations section on our website. In the Specialist-PPO policy class, our discovered masks outperform random masks in Baoding, Bounce, and Rotate Real, showing positive transfer to expert policies. Similarly, in Specialist-Distill-Recon, discovered masks outperform random masks across all tasks.

Altogether, these results show that the performance boost of our method comes quite significantly from sensor selection, especially at low sensing budgets, rather than solely policy optimization; furthermore, the discovered sensors show evidence of transfer to specialist policies.

As the sensor selection space is fixed when designing the generalist policy, the reviewer is wondering about whether it is possible to extend the method to unseen or out-of-distribution sensory space.

This is an interesting question! A straightforward approach would be to simply finetune the generalist on the new sensors, by adding in new modality encoders for the new sensors and finetuning the transformer on the combination of old and new sensor tokens. We leave this for future work.

AuthorsOpenReview ↗

(Nov 17) Rebuttal update

Dear reviewer Cjp6, as we near the end of the rebuttal period, we wanted to check in with you to see if there are any remaining concerns.

We addressed your concern about the performance boost of our method mainly coming from the generalist policy and not sensor selection, by pointing to our results that show that sensor selection does increase the performance of our generalist policy by a considerable margin in some tasks, and that the selected sensors from our generalist policy transfer to other specialist policies.

Next, we have updated the conclusion to include your question about how to address unseen sensors. We are grateful for your feedback, as the additional results generated from this discussion will further strengthen our manuscript. If satisfied, could you consider raising your score?

Reviewer_A5LCOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers the setting of robot sensor selection, which aims to find an ideal sensor configuration subject to certain constraints. In this work, a configuration of sensors is evaluated by how well it enables a downstream policy performance. However, the paper discusses that it is costly to train a separate policy for each sensor configuration. This leads the paper to propose training a single generalist policy that can perform well on diverse sensor configurations, which can then be used for downstream sensor selection. Before training the generalist policy a teacher policy is trained with PPO using full sensing configurations by default. Training for the sensor generalist follows a DAgger-like setup where each episode a configuration is sampled, and during an episode actions are sampled from a mixture of the teacher and the current generalist policy. From this trajectory distribution, a) the generalist is trained with an on-policy distillation loss using the reverse KL between the generalist and teacher, b) a PPO update, c) a reconstruction loss, predicting privileged state information and un-masked observations. The paper evaluates this approach on dexterous hands with tactile sensing, in sim and real, as well quadruped locomotion with lidar sensors. On these benchmarks, the paper evaluates how their generalist policy compares to various baselines (specialists, removing sensors from teacher, etc.) over different numbers of sensors. To obtain reduced sensor configurations the paper proposes using backwards selection, iteratively removing sensors by evaluating performance of configurations the generalist. The paper also demonstrates that their generalist training suffers the most without OPD, and then reconstruction, where the RL loss plays a smaller role. Finally, the paper evaluates their approach in the real world with a LEAP hand, where first in sim, a generalist policy is used to search over sensor configuration ,and then, sensor-specific policies are trained on the discovered configurations. These policies are transferred to the real-world, and seem to have lower energy usage on average and attain increased object rotation rates.

Critical strengths
  • The paper proposes a strong training procedure for obtaining generalist sensor policies relying on multiple supervisory signals (RL, teacher distillation, reconstruction).
  • This approach allows for amortizing the computation used for evaluating different sensor configurations, reducing the compute compared to training sensor-specific experts.
  • The paper has thorough evaluation across baselines, ablation, and real-world results validating their approach.
Critical weaknesses
  • A weakness in the setup, is the need for several different steps. For example, my understanding of the real-world setup, is first training the teacher, then the generalist & search, and finally training an expert on each best configurations. Is there a reason that the generalist can not be directly deployed like in the sim experiments? This can allow for more broadly testing sensor configurations in the real world.

  • The ablations show the large benefit that reconstruction & distillation provide for the generalist, however it is not compared how these objectives could benefit the task-specific expert baselines. Would this help close the gap to the generalist? or could this increase performance when training on discovered configurations for the real-world experiments?

Questions for authors

How robust the generalist policy is to say changing / dropping sensors during the episode? While this is not the focus of the work, it could help further demonstrate the utility of the generalist.

From the experiments, it is shown that distillation and reconstruction play much larger roles than RL. Thus, for training the generalist policy, would it be possible to train it offline, on trajectories collected solely from the teacher (e.g. behavior cloning with sensor masking + reconstruction objective)? or is the use of online, on-policy training crucial?

4 comments
AuthorsOpenReview ↗

(Nov 12) Initial Response, Part 1

Thank you for your review!

A weakness in the setup, is the need for several different steps. For example, my understanding of the real-world setup, is first training the teacher, then the generalist & search, and finally training an expert on each best configurations. Is there a reason that the generalist can not be directly deployed like in the sim experiments? This can allow for more broadly testing sensor configurations in the real world.

The main reason is due to the tuning costs of our particular sim to real pipeline for tactile dexterous manipulation, which uses MuJoCo Playground. It features a well tuned, proven PPO implementation for doing the sim-to-real LEAP Hand rotation task, with preset hyperparameters and domain randomization settings for training specialist MLP policies. Integrating the generalist transformer policy would require changing the policy architecture and associated training and deployment settings (hyperparameters, domain randomization, PID settings, action smoothing, etc.) and would require a nontrivial amount of time to tune, so we opted to just use the existing PPO sim-to-real pipeline to train specialists.

Our four simulated tasks do demonstrate direct deployment of the generalist, and those simulated tasks are capable of sim-to-real transfer (with adequate tuning), since those tasks are built in mature robotic simulators like IsaacLab and MuJoCo and/or were tasks directly ported from sim-to-real papers (e.g. Multi-obj. Rotate is from [1]). We leave direct deployment of the generalist policy for future work.

[1] Yin, Zhao-Heng, et al. "Rotating without seeing: Towards in-hand dexterity through touch." RSS (2023).

The ablations show the large benefit that reconstruction & distillation provide for the generalist, however it is not compared how these objectives could benefit the task-specific expert baselines. Would this help close the gap to the generalist? or could this increase performance when training on discovered configurations for the real-world experiments?

Following the suggestion of reviewer A5LC and jmLS, we ran a specialist baseline with the same distillation and reconstruction objectives as our generalist. These supervision terms should improve the performance for baselines, but would still not fully close the gap, since the generalist trains a single policy over multiple sensor configurations. We share the results at the Distillation Specialist Baseline section on our website. The baseline is labeled Specialist-Distill-Recon, Random.

The results now more clearly show the benefits of training a generalist across multiple sensor configurations. The specialists do well in Multi-obj. Rotate and Forest Run, but our generalist is still competitive, showing that a single generalist policy is able to match our outperform hundreds of distinct specialist policies. Next, the distillation specialists do poorly in Baoding and Bounce, where with 2-16 active sensors our generalist achieves 2.3x and 4.4x higher return on average; the gap only closes when nearly all sensors are available (k ≥ 32). This shows that generalist training is still important for the final performance, though distillation does benefit the task-specific experts.

How robust is the generalist policy is to say changing / dropping sensors during the episode? While this is not the focus of the work, it could help further demonstrate the utility of the generalist.

This is an interesting question! We ran some robustness experiments to investigate changing and dropping out sensors throughout an episode. We have posted initial results on our website under the Generalization Experiments section.

First, we investigated the robustness of the generalist under sensor dropout/noise. In Multi-obj. Rotate, we 1) dropped out tactile sensors per-observation with probability p and 2) multiplied the force magnitude by (1 + σε) where ε ~ N(0, 1). For Forest Run, we added noise to the LiDAR sensors by perturbing each measured distance d as clip(d + 5σε, 0, 5), where ε ~ N(0, 1) and σ controls the noise magnitude. Across all forms of added noise, the generalist was able to maintain most of its original performance, showing its robustness to sensor noise/failures.

Next, we tried selecting k random sensors at every observation throughout an episode, shown in the “Per-step Random Sensors” plot. The generalist here performs similarly to if it were given a fixed random mask with k sensors throughout the episode, demonstrating that the generalist is not sensitive to changing sensors during an episode.

We will also update our website with robustness results on Baoding and Bounce during the rebuttal period, when experiments complete.

AuthorsOpenReview ↗

(Nov 12) Initial Response, Part 2

From the experiments, it is shown that distillation and reconstruction play much larger roles than RL. Thus, for training the generalist policy, would it be possible to train it offline, on trajectories collected solely from the teacher (e.g. behavior cloning with sensor masking + reconstruction objective)? or is the use of online, on-policy training crucial?

It is possible to train the generalist offline, but we found this to have mixed results previously. In earlier experiments, we tried sampling trajectories from only the teacher trajectory distribution for the entire training run, and this worked well for some tasks like Multi-obj. Rotate but not for the other tasks, which indicates that online exploration with the student policy is still generally important.

On the sim2real setup, this overall makes sense. Out of curiosity, while algorithm hyperparameters need to be tuned, how much would you need to tune “domain randomization, PID settings, action smoothing”, or do they work across methods?

Thank you for performing the experiments integration the distillation and reconstruction into the specialists, as well as the generalization experiments testing the effect of changing/dropping sensors.

Thank you for providing more context on “training offline”; I think showing some of these mixed results could be interesting to highlight the benefit of your method, however I do not think this is necessary to add, as this may require more experiments to flesh out.

I also hope to see the results on the website integrated into the paper. Overall, I believe my main concerns are resolved, and I have leaned towards increasing my assessment to a 4.

AuthorsOpenReview ↗

(Nov 17) Response and experiment updates

We’re happy to hear that your main concerns have been resolved, and thank you for updating your assessment!

For the sim-to-real setup, the existing pipeline’s parameters would serve as a good starting point. The PID parameters and action smoothing parameters would need to be adjusted since the generalist has a different action distribution than a specialist policy. For domain randomization, we could begin with the existing robot and object randomization settings (mass, friction, size), along with the sensor noise and thresholds. These settings would then need to be adjusted based on how the generalist interacts with the object and triggers sensors.

Since our last response, we also report back with updated generalization results and show the results in the Sensor Noise section. We evaluated two types of intra-episode sensor noise models.

Bernoulli Dropout: We independently drop sensors with probability p at each timestep across tasks. This experiment has now finished for Baoding and Bounce. The generalist retains most of its performance despite sensor dropout.

Per-Step Mask: At each timestep, we randomly select k sensors to retain. We compare this against per-episode dropout, where a single randomly sampled sensor mask is held fixed throughout the episode. In Multi-obj. Rotate and Forest Run, per-step and per-episode dropout yield similar performance. In Bounce, per-step dropout performs worse, suggesting that consistent sensor configurations are important. In Baoding, per-step dropout outperforms per-episode dropout at k=16 and k=32. One possible explanation is that there are only a handful of useful sensors, and per-step sampling allows the policy to more frequently access informative sensor configurations.


Thank you again for your thoughtful suggestions which have improved our manuscript. We have already begun updating the pdf, such as adding the generalization experiments and refining our claims on our real world sensor selection experiment (changes in blue). We are actively working on integrating the website results into the draft.

Reviewer_jmLSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sensor Generalist, a policy trained across sensor configurations using distillation, reinforcement learning, and reconstruction to enable efficient sensor selection. Experiments on manipulation and locomotion tasks demonstrate competitive performance with substantially lower training costs, while real-world results validate the selected sensor configurations.

Critical strengths
  1. I like the problem of sensor selection -- it is practically motivated, and amortizing policy training across sensor configs is a reasonably compelling approach to reducing the cost of sensor-policy co-design.

  2. The authors do a good job of evaluating across various tasks and embodiments in sim.

  3. It was interesting to see that sparse proprioception preserved substantial performance in boading and bounce tasks with the dexterous hand. The qualitative analysis connecting selected sensors with learned behaviors adds interpretability.

Critical weaknesses
  1. The efficiency comparison in Section 4.2 (Fig. 4) is potentially confounded by differences in training methodology. The specialist baselines are trained from scratch using PPO, whereas the Sensor Generalist policies are first trained with privileged-state, and then teacher-student policy distillation. So the reported 17–150× reduction in training steps may reflect not only amortization but also the additional supervision and improved optimization efficiency.

I'd like to see a fair comparison where the specalist baselines also use the same distillation and reconstruction objectives.

  1. The real-world experiment evaluates specialists retrained on selected configurations, rather than deploying the generalist itself. Therefore, it validates sensor selection but not direct zero-retraining deployment. Furthermore, only one real robot and manipulation task are considered.

  2. The paper does not evaluate how well the selected sensor configurations perform under sensor noise (e.g., noisy tactile readings, calibration errors, or sensor failures), environmental distribution shifts (e.g., changes in object friction, mass, or obstacle layouts), or varying task conditions (e.g., manipulating unseen objects or performing different manipulation tasks). This leaves it unclear whether the discovered configurations capture robust sensing requirements or are highly specialized to the training settings.

Questions for authors
  1. I'm curious so as to how the backward selection compare with ES (evolutionary search) or even random search algorithm under matched evaluation budgets, particularly at low sensing levels?

  2. How well do generalist-predicted sensor rankings correlate with returns achieved by fully trained specialists? Reporting rank correlations or top-k regret for existing evaluated configurations would directly validate the central selection assumption.

  3. Can the authors compare against specialist policies trained with the same distillation and reconstruction objectives under matched computational budgets, to isolate how much of the reported efficiency gain comes specifically from amortization across sensor configurations? (related to weaknesses #2)

5 comments
AuthorsOpenReview ↗

(Nov 12) Initial Response, Part 1

Thank you for your review!

The efficiency comparison in Section 4.2 (Fig. 4) is potentially confounded by differences in training methodology … So the reported 17–150× reduction in training steps may reflect not only amortization but also the additional supervision and improved optimization efficiency… I'd like to see a fair comparison where the specialist baselines also use the same distillation and reconstruction objectives.

In Section 4.2, we do not claim that the performance gains of our method come solely from amortization, but from the whole system which includes both our choice of objectives, and amortized training across multiple sensor configurations. Nevertheless, we agree with the reviewer to run a specialist baseline with the same training objectives of distillation and reconstruction as our generalist to give more insight into the gains from amortization, and share the results at the Distillation Specialist Baseline section on our project website. The baseline is labeled Specialist-Distill-Recon, Random.

The results now more clearly show the benefits of training a generalist across multiple sensor configurations. The distillation specialists do well in Multi-obj. Rotate and Forest run, but our generalist is still competitive, showing that a single generalist policy is able to match or outperform hundreds of distinct specialist policies. Next, the distillation specialists do poorly in Baoding and Bounce, where with 2-16 active sensors our generalist achieves 2.3x and 4.4x higher return on average; the gap only closes when nearly all sensors are available (k ≥ 32). This shows that training across multiple sensor configurations is still important for the final performance.

Specialist baselines are trained from scratch using PPO, whereas the Sensor Generalist policies are first trained with privileged-state, and then teacher-student policy distillation…

The reviewer appears to be concerned about potential differences in setup between our method and the PPO specialist. We address them one by one below.

Both methods access privileged information: There is no information asymmetry, PPO specialist has a privileged critic that conditions on simulator state, as mentioned in line 266. We will update the text to make it more clear.

Compute budgets currently favor specialists: We do not think it makes sense to match the training steps between the generalist and specialist because it will be extremely unfavorable for the specialist. For example, the generalist policy is trained for 200M steps in Multi-obj. Rotate, which has 65,636 possible configurations with just 16 tactile sensors. When first sampling the sensing level uniformly, then selecting a random specification, this results in a paltry average training budget of just ~1.5M steps per sensor configuration.

However, the PPO specialist requires ~200M steps to converge. Furthermore, the distillation specialist results show that even with a more sample efficient method like distillation, a distilled specialist that dedicates all 200M steps to a single sensor configuration still underperforms the generalist in 2 out of the 4 tasks.

Matching Training objectives: This is addressed by running the distillation+reconstruction specialist experiment you suggested, and we show that the generalist policy still matches or outperforms the specialists in the tasks. We would like to thank you again for suggesting this comparison, as it more clearly shows that the gains of the sensor generalist are not just due to its training objectives, but also due to its amortization.

AuthorsOpenReview ↗

(Nov 12) Initial Response, Part 2

The real-world experiment evaluates specialists retrained on selected configurations, rather than deploying the generalist itself. Therefore, it validates sensor selection but not direct zero-retraining deployment. Furthermore, only one real robot and manipulation task are considered.

We agree that the real-world experiment validates the transfer of selected sensor configurations rather than direct deployment of the generalist policy. Since MuJoCo Playground features a well tuned, proven PPO implementation for doing the sim-to-real LEAP Hand rotation task, with preset hyperparameters and domain randomization settings for training specialist MLP policies, we chose to preserve the established pipeline and isolate the effects of sensor selection. Integrating the generalist transformer policy would require changing the policy architecture and associated training and deployment settings (hyperparameters, domain randomization, PID settings, action smoothing, etc.) and would require a nontrivial amount of time to tune, so we opted to just use the existing PPO sim-to-real pipeline to train specialists.

However, our four simulated tasks do demonstrate direct deployment of the generalist, and those simulated tasks are capable of sim-to-real transfer (with adequate tuning), since those tasks are built in mature robotic simulators like IsaacLab and MuJoCo and/or were tasks directly ported from sim-to-real papers (e.g. Multi-obj. Rotate is from [1]). We leave direct deployment of the generalist policy for future work.

[1] Yin, Zhao-Heng, et al. "Rotating without seeing: Towards in-hand dexterity through touch." RSS (2023).

The paper does not evaluate how well the selected sensor configurations perform under sensor noise (e.g., noisy tactile readings, calibration errors, or sensor failures), environmental distribution shifts (e.g., changes in object friction, mass, or obstacle layouts), or varying task conditions (e.g., manipulating unseen objects or performing different manipulation tasks). This leaves it unclear whether the discovered configurations capture robust sensing requirements or are highly specialized to the training settings.

Our experiments already include several forms of sensor noise and environmental variation. In Multi-obj. Rotate, training includes 16 objects, randomized object mass, friction, size, actuator gains, external pushes, and noisy proprioception. In Forest Run, obstacle layouts are randomized across episodes, and Baoding and Bounce randomize initial joint and object positions. Furthermore, our real-world rotation experiment trains with sensor noise in simulation, and transferring to real inherently introduces differences in object dynamics and tactile sensing.

Further evaluation of robustness would be valuable and we will conduct additional generalization experiments during the rebuttal period. We have posted some initial results on our website under the “Generalization Experiments” section.

First, for Multi-obj. Rotate, we design a set of 8 new objects and evaluate our generalist on them. At varying sensing budgets (k=4, 8), our discovered sensor configurations from the original object set outperform random sensor configurations, showing the generalizability of our discovered sensors. Similarly, for Forest Run, we designed three new tree layouts: orchard, thick trunks, and groves (these are randomized per-episode). Across all new layouts, our discovered sensor configurations from the original obstacle layout outperform random ones, demonstrating positive transfer.

We also investigated the robustness of the generalist under sensor noise. In Multi-obj. Rotate, we 1) dropped out tactile sensors per-observation with probability p and 2) multiplied the force magnitude by (1 + σε) where ε ~ N(0, 1). For Forest Run, we added noise to the LiDAR sensors by perturbing each measured distance d as clip(d+ 5σε, 0, 5), where ε ~ N(0, 1) and σ controls the noise magnitude. Across all forms of added noise, the generalist was able to maintain most of its original performance, showing its robustness to sensor noise/failures.

I'm curious so as to how the backward selection compare with ES (evolutionary search) or even random search algorithm under matched evaluation budgets, particularly at low sensing levels?

Under low sensing levels, the search space shrinks (e.g. the number of possible 2-sensor candidates for 16 total sensors is 120). Therefore, we expect ES and random search to do well since the search space is smaller. We will look into running further ES / random search comparisons during the rebuttal period.

AuthorsOpenReview ↗

(Nov 12) Initial Response, Part 3

How well do generalist-predicted sensor rankings correlate with returns achieved by fully trained specialists? Reporting rank correlations or top-k regret for existing evaluated configurations would directly validate the central selection assumption.

To measure the utility of our selected sensors, we use the performance gain of policies trained on our sensors over policies trained on randomly selected ones rather than computing rank correlations. We do see the sensors chosen by the sensor generalist lead to performance gains. Following the Additional Ablations figure on our website, the performance gain is computed by taking the difference between the solid and dashed lines for various policy synthesis methods (Generalist, Specialist-PPO, Specialist-Distill-Recon). The degree of performance gain depends on the task and the type of policy.

We observe the following trends. Generalist and Specialist-distill-recon show performance gains in using backward selected masks over random sensors in all tasks, with large gains in Baoding, Bounce, and Forest Run, and smaller gains in Multi-obj Rotate. Specialist-PPO trained on our sensors show positive transfer in Baoding, Bounce, and our sim-to-real experiment, but tie in performance with random sensors in Multi-obj. Rotate, and underperforms in Forest Run.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Initial Response (Nov 12)

Thank you for the thoughtful and constructive reviews! We are happy to see that the initial manuscript was generally well received with positive reviews of 3,3,3. The reviewers found the problem of sensor selection well-motivated and compelling (jmLS,Cjp6) the generalist policy approach to efficiently find performant sensors and policies to be sensible (A5LC, Cjp6), and the experiments to be extensive (jmLS, A5LC, Cjp6). We have addressed each reviewer individually below, and also included some questions, results, and ongoing experiments for context.

(jmLS, a5LC) Baseline suggestion - Specialist policies trained with same training objectives as our method: The reviewers suggested training specialist policies using distillation and reconstruction terms rather than PPO, to isolate the performance gain of generalist training. We ran this, see results here. While this narrows the gap, the generalist policy still outperforms or is competitive with the specialist across the tasks, showing that training across multiple sensor configurations is still important for the final performance.

(jmLS,a5LC) Additional generalization experiments: The reviewers are curious about how well the generalist can generalize to changes during evaluation time, such as sensor noise, new objects, new obstacle layouts, and sensor failures within an episode. We have begun experiments, and the preliminary results on the website show that the generalist is relatively robust to these changes. We will update the results and notify the reviewers when they are completed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.