acceptodds
Under review as a conference paper at ICLR 2027

Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination

Abstract

Robots operating in the physical world will increasingly need to coordinate with other robots, particularly in manipulation tasks where an object may be too large or heavy for a single robot to carry alone. Physical limitations caused by hardware degradation or actuator faults can restrict the actions a robot can reliably execute, yet these limitations may be unknown to its partner. We study whether a helper can infer a robot partner's physical constraints from observing it coordinate with another robot, then use the inferred capability to coordinate with the same partner on a new task. This is difficult because a demonstration shows what the constrained robot did, but not what it could have done. In physically coupled tasks, the other robot may also compensate for its limitations, making those limitations difficult to identify from the constrained robot's behavior alone. Our key insight is that these constraints shape the joint behavior of the team, making the actions of both robots informative about the constrained partner’s capability. We introduce Watch, Infer, Coordinate, a benchmark spanning three physically coupled manipulation settings, together with an inference approach that scores candidate constraints using observed joint behavior. Across all three settings, our method substantially improves constraint inference and zero-shot coordination, approaching an oracle with access to the true constraints.

Primary area:
applications to robotics, autonomy, planning
Keywords:
multi-robot coordination, capability inference, physical constraint inference, physically coupled manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F8yROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new method by which a robot can infer the physical limitations of a teammate and adapt its behavior for future collaboration. In the proposed Watch, Infer, Coordinate framework, a helper robot first observes a physically constrained robot cooperating with another demonstrator, infers persistent low-level constraints such as joint position limits, velocity limits, or restricted motion directions from their joint behavior, and then replaces the demonstrator to coordinate with the same constrained robot on a new task. The key idea is that the constrained robot’s limitations affect not only its own actions but also how its partner compensates, making the joint behavior of both robots informative for capability inference. The inferred constraint model is then incorporated into planning for zero-shot coordination, leading to substantially better performance than capability-blind or imitation-based baselines.

Critical strengths

A key strength of this paper is its novel and interesting problem setting: inferring a robot partner’s hidden physical constraints from prior collaboration and using them for zero-shot coordination on new tasks. The whole pipeline is reasonably well-designed and works well in the simulated cases.

Critical weaknesses

My main concern is whether the proposed setting captures a sufficiently important real-world problem.

The paper assumes that a robot’s low-level physical constraints are unavailable to its partner and must instead be inferred indirectly from prior collaborative behavior. However, in many practical robotic systems, constraints such as joint limits, actuator degradation, or velocity restrictions could potentially be obtained more directly through diagnostics, system identification, or communication with the robot. The paper does not sufficiently motivate when such direct access is unavailable while prior demonstrations and detailed state-action observations are available.

This makes the practical applicability of the proposed setting unclear, especially given that the current evaluation is entirely simulation-based and relies on known dynamics, goals, full observations, and a predefined finite constraint set.

Questions for authors
  1. Why is inference from observation the right tool for this problem, and how often does this information asymmetry actually arise in practice?
  2. About reliability: What happens when the observed demonstrations are not sufficiently informative to distinguish between different constraints?
3 comments
AuthorsOpenReview ↗

Response to Reviewer F8yR

Thank you for your confidence in our work, and for acknowledging our paper's novel and interesting problem setting, and the well designed pipeline. We address your questions below:

Q1: Inference from observation: Imagine watching two people carry a heavy object. You notice that one struggles to raise an arm and their partner adjusts how they lift the object to compensate. Even without being explicitly told about the limitation, you can infer restricted shoulder mobility and account for it when you later collaborate with that person. We want to enable robots to gain this same ability to learn about a prospective partner from observing their previous physical collaborations. Our work studies the corresponding problem in robot teams, where a robot may be watching other robots collaborate but do not have access to their controller configuration and diagnostic interface. The only information available in our setting are both the agent's states and actions and goals. We also plan on extending this setting to human robot teams in the future.

Q2: Uninformative demonstrations: If the observed demonstrations are not sufficiently informative, then it would not be possible to distinguish between constraints. For example, if the agents acted completely at random, it's not possible to tell if the agent didn't take an action because it couldn't or because it chose not to. There are two ways we make sure that the observed demonstrations are sufficiently informative. We mention in our paper that the demonstrators are acting boundedly rational, which is similar to how humans make decisions, and ensures that the agents take actions that are not necessarily optimal but rather "good enough". We also ensure that in the evaluation, the constraints actually led to a change of behavior in the demonstrations, which ensures the demonstrations are informative.

We hope these clarifications and additional experiments address your main concerns.

AuthorsOpenReview ↗

Follow-up to rebuttal

Thank you again for your encouraging assessment and thoughtful questions! We have clarified the practical motivation for inference from observation and discussed how our approach handles insufficiently informative demonstrations.

We would greatly appreciate your thoughts on whether these clarifications address your concerns.

Thanks for your reply. I would keep my scores, but since this paper does not exactly align with my expertise, I'm not going to champion this paper.

Reviewer_3gjAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how a helper can infer a robot partner’s physical constraints by observing it collaborate with another robot, then use that information on a new task. The key idea is that the demonstrator’s compensatory behavior can reveal limitations that are ambiguous from the constrained robot’s behavior alone. The inferred capability is incorporated into CEM-MPC and evaluated in three joint-carrying environments.

Critical strengths

I find the problem interesting and well motivated. Distinguishing what a partner cannot do from what it simply did not need to do is an important issue for capability inference. The partner-only ablation directly tests the central idea. The results suggest that observing the demonstrator provides useful information beyond observing the constrained agent alone. The paper connects inference to downstream performance rather than evaluating constraint recovery in isolation. Using the same planner across CEM-based methods makes this comparison informative.

Critical weaknesses

My main concern is the reliance on a matched behavioral model. The model used to generate demonstrations also underlies inference, with known goals, dynamics, and complete state/action observations. I understand the motivation for this controlled setup, but it remains unclear whether the method would recover the same physical constraints when the demonstrator follows a different strategy. This distinction matters because the paper aims to infer capability, which should ideally remain useful across changes in behavior. The final evaluation is also limited to three constraint cases per environment, despite much larger candidate spaces. The relationship between scorer training and these evaluation cases needs to be stated more clearly. The learned scoring models are another important source of uncertainty. For the UR5 settings, I could not find enough detail about the training targets, losses, or candidate construction to reproduce the scoring procedure. Since inference also removes physically inconsistent candidates, a filtering-only baseline is needed to establish the additional benefit of learning from joint behavior.

Questions for authors
  1. What are the exact training objectives and capability-level splits for the UR5 scorers? Were the final evaluation capabilities represented during training?
  2. How well does physical-consistency filtering alone perform when combined with the same prior and planner?
  3. Does the inference remain effective under a modest change in the demonstration policy, without changing the physical constraints?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3gjA

Thank you for acknowledging that our problem is interesting and well motivated, and that our insight that distinguishing what a partner cannot do from what it did not do is important for capability inference. We have performed additional experiments following your suggestions and added them to the paper. We address your questions below.

Q1: Scorer Training: For our training objective, we match reference evidence from both robots’ observed actions using

ℒ = KL_episode + 0.25 · SmoothL1(β = 0.5) + λₜ · KL_group

Here, KLepisode compares reference and predicted probabilities over candidate capabilities for each demonstration. The Smooth L1 term matches the corresponding evidence scores, while KLgroup compares capability probabilities after combining multiple demonstrations. Reference targets are computed from the behavioral model, and both KL terms run from the reference distribution to the predicted distribution. We set λₜ=0 for the first 100 updates and 0.5 afterward. To ensure fairness, the final evaluation capabilities are not represented during training. We have added these details to the appendix.

Q2: Physical-consistency filtering: Following your suggestion, we evaluate a baseline of physical-consistency filtering alone, while keeping the prior and planner fixed. This baseline filters out capability candidates that are inconsistent with the observed motion, and samples among the remaining candidates. On the obstacle task, we obtain the following success rates:

MethodRodFixed-base UR5Mobile UR5
Filtering only59.3%44.4%29.6%
Our method92.6%85.2%63.0%

These results support the contribution of behavioral evidence beyond excluding physically inconsistent hypotheses.

Q3: Robustness to changes in demonstration policy, matched behavioral model: We directly test this concern by replacing the soft-response demonstrator with an independent joint CEM controller, while keeping our method and the physical constraints completely frozen. The success rates of our method compared with CE-CM-Div, the prior work baseline, are shown below:

MethodRodFixed-base UR5Mobile UR5
CE-CM-Div51.9%44.4%37.0%
Our method92.6%85.2%59.3%

This shows that our method is robust to changes in the demonstration policy.

We hope these clarifications and additional experiments address your main concerns, and that you will consider raising the score.

AuthorsOpenReview ↗

Follow-up to rebuttal

Thank you again for your helpful suggestions! We have added the filtering-only baseline and independent-controller experiments you requested, along with further clarification of our scorer training procedure.

We would greatly appreciate your feedback on whether these additions address your main concerns, or if anything remains unclear.

Reviewer_7s6oOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

knowledge for coordination on a new task.

  1. The method scores candidate constraints using both agents’ behavior and incorporates the inferred capability into CEM-MPC planning.

  2. Experiments cover three simulated carrying settings, demonstrating improved constraint inference and coordination success approaching an oracle with known constraints.

Critical strengths
  1. An intuitive and useful modeling insight. A teammate’s compensatory behavior can reveal limitations that are ambiguous from the constrained robot’s actions alone.

  2. Direct ablation support. Partner-only comparisons support the benefit of observing both agents, particularly in the fixed- base and mobile UR5 settings (Table 2).

  3. Downstream benefits under controlled planning conditions. All CEM-based methods share the same planner and budget, helping attribute improved task success to better capability information (Table 3).

Critical weaknesses
  1. The practical need for indirect inference is not established. The paper does not identify concrete deployment conditions where partner limitations are unavailable through communication or onboard diagnostics, yet sufficiently accurate joint states and actions are accessible for inference.

  2. Simulation-only, model-matched evaluation limits external validity. Demonstrations follow the assumed behavioral model, with known goals and dynamics and fully observed states/actions. Without physical-robot or mismatch experiments, it remains unclear whether the method distinguishes genuine physical limitations from controller differences, sensing errors, or environmental disturbances.

  3. The evidence for broad generalization is limited. Each setting evaluates only three fixed constraint cases and 27 obstacle- task trials per method. Training/test capability separation is insufficiently documented, and the method is not evaluated on constraints outside its predefined candidate set.

Questions for authors
  1. What realistic deployment scenario requires behavioral inference rather than direct capability communication or fault diagnosis? Which information is available to the helper, and why are the physical limits themselves unavailable?

  2. Can you demonstrate the approach on physical robots with an undisclosed actuator limitation, or at least test independently designed controllers, observation noise, and dynamics mismatch?

  3. Can you clarify the capability-level training/test split and evaluate more held-out constraint combinations and severities? How does inference and planning behave when the true constraint is absent from the candidate set?

2 comments
AuthorsOpenReview ↗

Response to reviewer 7s6o

Thank you for acknowledging our paper's intuitive and useful modeling insights, as well as the downstream benefits and our paper's presentation, informativeness, and soundness. Your suggestions also motivated additional experiments that further strengthened our claims, and we have added the experiments to our paper. We address your questions below:

Q1: Realistic deployment scenario and available information: Imagine watching two people carry a heavy object, and you realize that one person struggles to lift their arm above their head, so you infer that they have limited shoulder mobility, and when you collaborate with them on a physical task you can plan around their limited mobility. We want to enable robots to gain this capability. Our work studies the corresponding problem in robot teams, where a robot may be watching other robots collaborate but do not have access to their controller configuration and diagnostic interface. The only information available in our setting are both the agent's states, actions and goals. We also plan on extending this setting to human robot teams in the future.

Q2: Independent controllers, observation noise, and dynamics mismatch: We have added evaluations addressing each of these suggestions while keeping our method frozen. We evaluated demonstrations generated by independent CEM controllers instead of the soft response demonstrator. The success rates of our method compared with CE-CM-Div, the prior work baseline, are shown below:

MethodRodFixed-base UR5Mobile UR5
CE-CM-Div51.9%44.4%37.0%
Our method92.6%85.2%59.3%

We also evaluate five temperature multipliers and an independent CEM controller across all three settings, using three capability cases and ten demonstration seeds per condition. Across all settings, our method achieves lower inference error than CE-CM-Div.

For dynamics mismatch, increasing the fixed-UR5 payload from 2 kg to 3 kg and 4 kg yields the following success rates, without changing the planner's dynamics model:

Payload massSuccess rate
3 kg92.6%
4 kg88.9%

Additionally, adding Gaussian noise to both robots’ joint-angle observations, with a standard deviation of 0.001 rad does not increase our inference error. These results further support our method's robustness.

Q3: Capacity Coverage: We intentionally evaluate a finite candidate library to provide a controlled and reproducible evaluation. Our method scores the likelihood of proposed capability hypotheses given the observed demonstrations, so it is not limited to this set. A natural extension is to generate capability hypotheses adaptively using CEM. Such a search would sample constraint combinations and continuous severity values, including joint-position and velocity limits, and progressively concentrate on hypotheses with higher scores. This would allow inference to explore capabilities beyond the candidate set.

We hope these clarifications and additional experiments address your concerns, and that you will consider raising the score.

AuthorsOpenReview ↗

Follow-up on rebuttal

Thank you again for your constructive feedback! We wanted to kindly follow up, as we have added new experiments addressing your concerns about independent demonstration controllers, observation noise, and dynamics mismatch.

We would greatly appreciate your thoughts on whether these results and our clarifications address your main concerns, or if any questions remain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

General Response:

We thank all the reviewers for their helpful comments, and appreciate reviewers 3gjA and F8yR for recognizing our novel and interesting problem setting, as well as reviewer 7s6o for recognizing the soundness of our experimentation and usefulness of our insight and method in downstream coordination. The reviewers suggested additional experiments that directly evaluate the robustness of our method to changes in demonstration behavior, and strengthen the support for our claims. We have added the following evaluations while keeping our method frozen.

Physical consistency filtering baseline: As suggested by reviewer 3gjA, this baseline filters out capability candidates that are inconsistent with the observed motion, and samples among the remaining candidates. On our obstacle evaluation task, the results are:

MethodRodFixed-base UR5Mobile UR5
Filtering only59.3%44.4%29.6%
Our method92.6%85.2%63.0%

This shows the value of using behavioral evidence in our method.

Independent Demonstration Controller: We address concerns brought up by reviewers 7s6o and 3gjA regarding model matched evaluation settings and change in demonstration policy affecting the model's inference by generating demonstrations with an independent CEM controller instead of the soft response demonstrator.

MethodRodFixed-base UR5Mobile UR5
CE-CM-Div (prior work baseline)51.9%44.4%37.0%
Our method92.6%85.2%59.3%

This shows that our method is robust to variations in demonstration behavior.

Robustness under different dynamics: As suggested by reviewer 7s6o, we also evaluate our method under different dynamics. We evaluated heavier payload masses on our fixed-UR5 task, 3kg and 4kg (baseline was 2kg), with the following results:

Payload massSuccess rate
3kg92.6%
4kg88.9%

Our method was still able to achieve these success rates without changing the planner's dynamics model.

We hope these substantial additional experiments in addition to our main results and novel problem setting addresses the reviewers’ key concerns and strengthens their confidence in our contributions and method. We have incorporated these changes in our main paper and appendix, with changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.