Squeeze to Plan: Learning Dense Visual Representations for Efficient Planning via Bisimulation Information Bottleneck

Visual foundation models offer robust perceptual priors for robotic manipulation, but it is computationally expensive to plan over their high-dimensional outputs. This inefficiency arises from the fact that for any given task, foundation models tend to capture irrelevant information. For example, a robot performing a pick-and-place need not reason about the visual texture of the table on which it operates. This leads us to ask – how can we develop planners which both benefit from these robust visual priors, and remain fast enough for real-time planning? Towards this goal, we introduce Squeeze-to-Plan (S2P), a framework for compressing the outputs of frozen foundation visual models into compact, task-specific state representations suitable for real-time planning. S2P achieves this using a novel Bisimulation Information Bottleneck objective, to encourage the compressed representations to discard task-irrelevant visual information while retaining task-relevant reward and dynamics information. S2P solves two real-time visual manipulation tasks where prior methods for planning in pretrained visual feature spaces generally fail, while achieving an orders-of-magnitude reduction in time needed to generate a plan. Additionally, we show that S2P's compression does not sacrifice the task-relevant perceptual priors despite its large compression factor, achieving similar robustness to prior methods against test-time perturbations to lighting and object-colour. Our results show that our method can enable efficient planning with pretrained visual features, while preserving their robustness to certain visual changes.