acceptodds
Under review as a conference paper at ICLR 2027

Training with Borrowed Eyes: Decentralized Multi-Robot Manipulation via Privileged Information Distillation

Abstract

Learning decentralized policies provides a scalable paradigm for multi-robot manipulation by enabling flexible collaboration without centralized observations or communication at deployment. However, agents operating solely on local observations often struggle to infer the intentions and behaviors of their partners, resulting in degraded coordination. In this work, we present a decentralized manipulation framework that transfers coordination knowledge from a privileged centralized policy to locally conditioned agents. During training, a privileged teacher with access to global observations captures coordinated behaviors, while privileged information distillation transfers its action distribution to decentralized students conditioned only on local observations. At deployment, each agent independently predicts its own actions without access to the states, observations, or actions of partners. This formulation retains the scalability and robustness of fully decentralized execution while exploiting privileged global information during training to foster implicit coordination. Experiments on real-world multi-arm manipulation tasks show that our approach consistently outperforms centralized baselines and prior decentralized methods, providing a practical approach for transforming pretrained centralized policies into scalable decentralized robotic systems.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic manipulation, Multi-robot manipulation, Vision-Language-Action models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fN9mOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper converts a pretrained centralized π0.5\pi_{0.5} policy into a decentralized multi-robot controller. A privileged teacher still sees the full team observation but predicts only one robot's action chunk. Its flow-matching velocity field is then distilled into a student that, at deployment, uses only its local observation, proprioception, and a robot-identity prompt, with no access to other robots' observations, states, or actions. Execution is asynchronous. On 50 RoboTwin 2.0 tasks the method reaches 71.48% success, above π0.5\pi_{0.5}-Dec (68.22%) and CHORUS (69.76%) but below centralized π0.5\pi_{0.5} (72.90%). On RoboFactory tasks with three or four arms it averages 76.8%, above CHORUS (73.0%) and centralized π0.5\pi_{0.5} (67.3%). On a real Franka plus two Piper arms, with 184 demonstrations over four tasks, reported average success and progress rise from 16.8% / 41.9% for centralized π0.5\pi_{0.5} to 50.2% / 72.3%, about 11.4 success points and 7.5 progress points above CHORUS.

Critical strengths
  1. The training interface is concrete and the ablations match the stated hypothesis. A privileged teacher that still sees the team observation is distilled into a locally conditioned student by aligning flow-matching velocity fields at the same noisy action state. Varying the field-loss weight and the teacher-quality gate shows a small, consistent gain, and the gain is larger on cooperative tasks than on non-cooperative tasks. The paper also reports a useful training fact: the teacher checkpoint with the highest success is not the checkpoint that produces the best student.
  2. The real-robot system makes the deployment claim tangible. One Franka and two Piper arms differ in embodiment, control frequency, and action space. The paper specifies decentralized collection, asynchronous execution, and four tasks that include deformable cloth and sustained interaction in table hockey. The appendix documents task splits, identity prompts, progress scoring, and training hyperparameters, which gives the experiments a reproducible starting point.
Critical weaknesses
  1. The technical step beyond existing decentralized VLA and privileged-distillation work is small relative to the claims. Centralized training with decentralized execution, privileged teacher-student transfer, and distilling multi-agent dynamics into a locally inferred representation are all established. CHORUS already finetunes one pretrained π0.5\pi_{0.5} for heterogeneous robots using local observations and an identity prompt, with asynchronous execution and no inference-time communication. In Table 3, copying the teacher and training with local flow matching alone reaches 69.93% overall success, while the full distillation method reaches 71.48%. Most of the decentralized performance is already present before the proposed field distillation.

  2. The evidence does not yet support the paper's main claim at the strength asserted in the abstract. In the RoboTwin conversion, each arm's local observation includes the shared agent view together with its own wrist view, so the partner arm remains visible and the information setting is weaker than strict local isolation. Simulation gains have no reported trial counts, seeds, or confidence intervals, and Table 4 contains large per-task regressions, which makes it hard to be confident in the robustness of the results. There is no real-world ablation that keeps decentralized finetuning and removes field distillation, so the 11.4-point gain over CHORUS cannot be attributed to privileged distillation.

Questions for authors
  1. The RoboTwin local observation includes a shared agent view that can see the other arm. Please report a variant that removes this view and retains only the wrist camera and proprioception. If the shared view remains part of the method, please revise the claim that local observations do not contain partner information. Please also state when the method fails if the partner is occluded or outside the local field of view, and qualify the abstract so that it agrees with Table 1, where centralized π0.5\pi_{0.5} remains ahead overall and on cooperative tasks.

  2. How many evaluation episodes and random seeds produce each success rate in Tables 1 and 4? Please add confidence intervals or a paired test. The margins over CHORUS are about 1.7 points overall and 2.8 points on cooperative tasks. Please also explain the large drops on Place Bread Skillet, Scan Object, and Move Playingcard Away, including whether they come from the identity prompt, interference from the shared view, or distillation of an inaccurate teacher field. In Figure 9(b), the partial correlation after controlling for π0.5\pi_{0.5}-Dec is r=0.476r=0.476 with p=0.053p=0.053. Please discuss what remains of the claimed teacher-to-student transfer once task difficulty is controlled for.

  3. Please list per-task success, progress, and intervals for all four real-world tasks and all three methods, and add the λfield=0\lambda_{\mathrm{field}}=0 control initialized from the teacher and trained only with local flow matching. Please separate the gain caused by decentralized execution from the gain caused by field distillation. The progress score is assigned by hand, and table hockey equates success with progress. Please state how these scores were adjudicated.

  4. The claim that ``weight sharing alone does not explicitly bridge the gap between globally informed coordination knowledge and locally conditioned decentralized decision-making'' in the introduction (Line 80-81) is unsupported. CHORUS reports that a shared pretrained VLA with local observations and an identity prompt already enables decentralized collaboration without inference-time communication. Please justify the claim against that result. Table 3 does not do so: λfield=0\lambda_{\mathrm{field}}=0 still initializes the student from the privileged teacher, and full distillation improves it by only 1.55 points (69.93% to 71.48%).

  5. Please explain why CLS-DP, LatentToM, and MIMIC-D are not compared in all the main results. A from-scratch diffusion policy and a pretrained VLA differ by much more than the proposed distillation term.

  6. The asynchronous study pauses one robot for 5, 10, or 15 steps at the start of four cooperative tasks. Please add mid-episode delays, packet loss, and measured communication latency, and state whether the centralized failure is an out-of-distribution input or a limitation of the joint action head.

Reviewer_rN75OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Privileged Information Distillation (PID) for decentralized multi-robot manipulation using a pretrained VLA. A teacher conditioned on team-wide observations predicts individual robot actions, while a locally conditioned student learns through demonstration supervision and flow-field distillation. Teacher-aware weights adjust sample importance. Evaluation on RoboTwin, RoboFactory, and four real-world tasks shows improvements over the tested decentralized baselines, including in a heterogeneous multi-arm system.

Critical strengths
  1. The F2S teacher and L2S student use compatible action parameterizations, making privileged flow-field supervision straightforward to integrate into a pretrained VLA. This is a practical contribution to decentralized policy learning.

  2. The simulation benchmarks and heterogeneous real-world system provide concrete evidence that the framework can support multi-arm manipulation across different embodiments and control representations.

Critical weaknesses
  1. The methodological novelty is not sufficiently established. LatentToM built a decentralized framework based on generative action policies. CHORUS establishes shared pretrained VLAs for heterogeneous, asynchronous decentralized execution, while CLS-DP studies privileged information transfer for local coordination. PID’s distinctive contribution therefore lies in its F2S teacher, flow-field transfer, and weighting scheme. The paper does not sufficiently explain or demonstrate why these choices offer a substantive advantage over other privileged-transfer mechanisms. This limits the methodological contribution beyond a useful combination of established ideas.

  2. The experimental evidence does not fully establish PID’s contribution or its claimed scalability advantage. Table 3 shows that adding distillation improves success from 69.93% for the teacher-initialized student to 71.48% for full PID, supporting its usefulness on RoboTwin. However, equivalent ablations are not reported for RoboFactory or the real-world tasks. It therefore remains unclear how much of the improvement in these settings comes specifically from distillation beyond teacher initialization. Furthermore, evaluating different tasks with two, three, and four robots demonstrates applicability across team sizes, but does not isolate the effect of increasing the number of robots. These comparisons alone cannot establish that PID scales better than centralized policies in small scale team size, since task characteristics and team size change together.

Questions for authors
  1. Are F2S teacher and initialization-only student results available for RoboFactory or the real-world tasks? If so, how much of the reported improvement is specifically attributable to distillation?

  2. Which results distinguish improved partner modeling from improved individual action prediction, and isolate PID’s recovery benefit from that of decentralization itself?

  3. Which camera feed(s) are shared at deployment?

Reviewer_JGN7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Privileged Information Distillation (PID) adapts π0.5 to multi-robot manipulation with local observations and no explicit communication at deployment. A privileged teacher uses joint observations to predict one robot's action chunk. The shared student starts from the teacher's weights and learns through both local flow matching and action-field distillation, with sample weights based on the teacher's relative prediction error. Evaluation covers 50 RoboTwin 2.0 tasks, four RoboFactory tasks, asynchronous execution, and four real-world tasks on a Franka/Piper setup. Average performance improves over the evaluated decentralized baselines, while centralized π0.5 remains stronger overall on RoboTwin 2.0.

Critical strengths
  1. The evaluation goes beyond bimanual simulation to three- and four-robot tasks and real robots with different action spaces and control rates. On RoboFactory, PID averages 76.8% success versus 73.0% for CHORUS. Figure 8 also reports improvements over CHORUS on all four real-world tasks in both SR and APR, supporting the approach's practical value.
  2. Field distillation fits the existing flow-matching formulation, while supervision from demonstrations gives the student a learning signal alongside the teacher. Table 3 controls for teacher initialization and reports an increase from 69.93% to 71.48% overall success when distillation is added. This is useful evidence of an additional benefit beyond transferring the teacher's weights.
Critical weaknesses
  1. The contribution of privileged observations is not sufficiently isolated. Table 3 controls for teacher initialization, but does not compare PID with a student trained using the same two-stage procedure with a teacher restricted to local observations. The teacher–student analysis offers supporting evidence, but does not provide this control. It remains unclear how much of the improvement specifically comes from access to partner observations.
  2. The reliability of the component gains is unclear. Table 3 reports overall gains of 1.55 percentage points over teacher weight copy and 0.93 percentage points for teacher-aware weighting over uniform weighting. The reproducibility statement mentions repeated experiments, but variation across independent training runs is not reported. This makes it difficult to judge how consistently these relatively small gains hold.
Questions for authors
  1. How were total training budgets and checkpoint selection handled for PID, π0.5-Dec., and CHORUS? Were the teacher checkpoint and distillation weight selected using separate validation data?
  2. Can you report end-to-end latency or task-completion time under matched hardware? These measurements would help substantiate the efficiency advantage discussed in Section 3.3.
  3. Figure 5 pauses one arm at task onset. How does PID compare with CHORUS when the interruption occurs during a handover or shared-object manipulation? This would test recovery during ongoing coordination.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.