acceptodds
Under review as a conference paper at ICLR 2027

MatchingPolicy: Correspondence-Aware Policy Enables Cross-Object In-Context Learning

Abstract

In-context imitation learning enables few-shot policy generalization but struggles to maintain performance on unseen objects and novel scenarios. To address this, we introduce MatchingPolicy, a correspondence-driven framework that explicitly decouples demonstration-to-scene matching from policy learning. Central to our method is a correspondence-aware diffusion policy that conditions robotic actions directly on dense semantic correspondences. This architectural separation resolves the inherent conflict between correspondence identification and action adaptation, enabling robust out-of-distribution transfer. Our framework integrates vision foundation models with a novel two-stage matching algorithm to dynamically establish reliable correspondences. Extensive evaluations on RLBench and real-world manipulation tasks confirm that MatchingPolicy achieves superior few-shot performance, generalizing reliably across unseen object instances and semantic categories.

Primary area:
applications to robotics, autonomy, planning
Keywords:
In-Context Imitation Learning, Semantic Correspondence, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8v4pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MatchingPolicy, a correspondence-aware framework for in-context imitation learning that enables robot manipulation across different objects and scenes. It decouples semantic correspondence extraction from action prediction and proposes a graph-based diffusion policy conditioned on explicit correspondences. Extensive experiments on RLBench and real-world manipulation tasks demonstrate strong generalization to unseen tasks and objects.

Critical strengths
  1. The paper addresses an important and challenging problem in robotic manipulation, particularly few-shot generalization to unseen objects and scenarios.
  2. The idea of decoupling correspondence extraction from action prediction is intuitive and interesting.
  3. The method achieves strong performance on both simulation and real-world tasks, demonstrating its effectiveness across different manipulation scenarios.
Critical weaknesses
  1. While the paper claims cross-category generalization, Table 3 primarily evaluates layout and shape variations, whereas cross-category transfer is demonstrated mainly through a qualitative example in Figure 9. I would like to see more systematic quantitative evaluations across different object categories.
  2. Could the authors discuss whether MatchingPolicy could be extended to other settings (e.g., [R1]), where demonstrations of the target task are unavailable? What limitations or modifications would be required?

[R1] Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization. NeurIPS 2025

Questions for authors

See weakness

2 comments

W1: While the paper claims cross-category generalization... I would like to see more systematic quantitative evaluations across different object categories.

We have added quantitative evaluations of MatchingPolicy on four cross-category transfer tasks in RLBench. For each task, the object category used during evaluation differs from that in the demonstrations. Each transfer success rate is evaluated over 100 trials, while the success rates on the original object categories are taken from the MP results in Table. We use VLM to assist in extracting local semantic correspondences across object categories.


TaskOriginal ObjectSuccess RateCategory TransferSuccess Rate
Pick Up CupCup94%Cup → Bowl100%
Close LaptopLaptop100%Laptop → Lidded bin42%
Open WineWine bottle71%Wine bottle → Cream jar34%
Slide BlockBlock100%Block → Mouse94%

The videos of cross-category tasks are available on our anonymous website.

W2: Could the authors discuss whether MatchingPolicy could be extended to other settings [1], where demonstrations of the target task are unavailable? What limitations or modifications would be required?

W2: Extension to settings without target-task demonstrations.

MatchingPolicy could be extended to settings without target-task demonstrations. As shown in W1, MatchingPolicy can transfer interactions across object categories, suggesting the feasibility of reusing demonstrations from related tasks. Specifically, we could construct a general demonstration database covering diverse tasks and objects. Given target instruction and observation, a VLM could retrieve demonstrations with transferable interaction patterns and identify corresponding functional parts. MatchingPolicy could then leverage these correspondences to transfer the retrieved interactions to the target scene.

Limitation: This extension relies on the availability of relevant interaction patterns in the database, as MatchingPolicy cannot generate fundamentally novel interactions without suitable demonstrations.

We will discuss this extension and its limitations in the revised manuscript.

[1] Zhou et al. Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization. NeurIPS 2025.

Thank you for the authors' response. After carefully reading the authors' response and the other reviewers' comments, I think most of my concerns have been addressed. Therefore, I decide to maintain my rating of Weak Accept. Good luck!

Reviewer_oNZaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MatchingPolicy, an in-context imitation learning framework that separates visual correspondence matching from action prediction. It uses vision foundation models to match points between demonstrations and the current scene, then feeds these correspondences into a graph-based diffusion policy. The policy is trained on synthetic trajectories with ground-truth correspondences and deployed using visually estimated correspondences without real-world fine-tuning.

The experiments cover 36 RLBench tasks and four real-world tasks. MatchingPolicy outperforms InstantPolicy across object and layout variations, and additional comparisons with KAT and RDT-1B show promising performance with only two demonstrations.

Critical strengths
  1. Clear motivation and sensible design. Explicitly matching demonstrations to the current scene gives the policy useful information for adapting actions. The modular design also makes it possible to use pretrained visual models without learning correspondence extraction from robot data alone.

  2. Promising empirical results. The method performs well across simulated tasks and real-world object and layout changes. Transferring a policy trained on synthetic data to real-world observations without fine-tuning is particularly encouraging. The additional KAT and RDT-1B comparisons strengthen the evaluation.

  3. Improved presentation. The correspondence inputs and graph structure are explained more clearly in this version. The overview makes the overall pipeline easier to follow.

Critical weaknesses
  1. The source of the improvement is still unclear. The main motivation is that separating correspondence extraction from action prediction improves generalization. However, the ablations focus on the matching algorithm and action-chunk execution, rather than testing this claim directly. A comparison with a policy trained on the same data and architecture but without explicit correspondence conditioning would be more convincing. It would also be useful to compare against a simpler trajectory-retargeting method using the same correspondences, to understand what the learned policy adds.

  2. Robustness to correspondence errors is not well studied. Simulation evaluation uses ground-truth correspondences, while real-world deployment depends on estimated matches and tracking. The real-world results show that this can work, but do not explain when it fails or how sensitive the policy is to incorrect or missing matches. Evaluating estimated correspondences in simulation would help connect the larger simulation study with the complete deployed system.

  3. The dependence on the camera setup remains unresolved. The system uses three calibrated RGB-D cameras with approximately 360-degree coverage. Although the matching procedure is described more clearly, there is no experiment showing how performance changes with fewer cameras. This matters particularly for large pose changes, where view alignment appears important. The paper should also clarify how task-relevant object masks are selected and associated across scenes in multi-object tasks.

  4. The real-world success criteria do not always measure task completion. For example, Pour Water is considered successful when the spout is above the cup and the teapot is tilted by at least 15 degrees; actual water transfer is not measured. Similarly, Cut Egg is evaluated through knife motion rather than a completed cut. These are useful measures of motion transfer, but provide weaker evidence of functional manipulation than the task names suggest. The evaluation is also limited to four tasks with ten trials per setting.

Questions for authors
  1. Can the authors isolate the benefit of explicit correspondence conditioning under matched training data, architecture, and execution settings? How does the policy compare with direct trajectory retargeting using the same matches?

  2. How much does performance drop when estimated correspondences replace ground-truth correspondences in simulation? What are the main failure modes caused by matching or tracking errors?

  3. How does performance change with one, two, and three cameras, especially under Layout-Hard? What human input is needed to select and associate object masks?

  4. Can the authors report actual task completion alongside the geometric success criteria, as well as the end-to-end inference time?

2 comments

W1: ...A comparison with a policy trained on the same data and architecture but without explicit correspondence conditioning would be more convincing.

We added ablations on explicit correspondence conditioning to evaluate the gain from correspondence-based input construction and explicit correspondence connections independently. MatchingPolicy improves success under matched training and execution settings, supporting the benefits of these designs. Please see the Common Response for details.

Q1: Can the authors isolate the benefit of explicit correspondence ... How does the policy compare with direct trajectory retargeting ...?

We compare our method with a typical trajectory-retargeting method (DITTO [1]). It uses the same matched points obtained through the same correspondence sampling strategy as ours. We evaluate it on all our RLBench tasks with 100 trials, reporting the average success rate across tasks.

MethodSuccess Rate
Trajectory Retargeting41.3%
MatchingPolicy82.3%

We attribute this improvement to the gripper-object / object-object interaction prior learned through synthetic pretraining with sparse correspondences, as well as the policy's ability to adapt its execution to the observed scene configuration.

We provide a qualitative comparison on the Put Umbrella task on our anonymous website.

Q2: How much does performance drop when estimated correspondences replace GT in simulation? What are the main failure modes ...?

Following our real-world pipeline, we evaluate MatchingPolicy on RLBench using visually estimated correspondences. Specifically, we use MASt3R [2] for cross-scene matching and temporal correspondence estimation, replacing RADIOv2.5 which yielded unreliable matches on low-texture RLBench objects in our preliminary experiments. We report success rates on OOD-Hard tasks, with 100 trials per task.

TaskEstimatedOriginalΔ Success Rate (pp)
Close Drawer92%90%+2
Close Grill94%97%-3
Open Grill65%99%-34
Pick Up Cup99%94%+5
Open Door91%98%-7
Turn Tap77%100%-23
Frame Off45%33%+12
Sweep Dust47%41%+6
Open Wine72%71%+1
Put Money42%50%-8
Put Roll41%52%-11
Pull Plug15%46%-31
Average65.0%72.6%-7.6

We summarize two main failure modes (img)

Correspondence errors. Initial mismatches or temporal drift can associate points with incorrect physical locations, potentially misdirecting the gripper. In an Open Grill example, 3/4 selected lid points were already mismatched before execution.

Occlusion-induced correspondence loss. Occlusion can remove points from the candidate set. The evaluation does not restore lost correspondences after reappearance, reducing alignment cues, especially for small objects. In a failed Pull Plug episode, plug candidates dropped from 21 to zero, leaving no valid correspondence for the manipulated object.

Q3: How does performance change with cameras, especially under Layout-Hard? What human input is needed ...?

We apologize that we could not complete systematic real-world camera ablations within the rebuttal period and leave this for future work. Our preliminary tests suggest that MatchingPolicy can operate with different camera counts and layouts, provided that sufficiently reliable correspondences are available. More cameras can improve correspondence coverage to benefit performance.

We use a VLM to identify task-relevant objects, and to locate the matched objects in the current scene. Their segmentation masks are then obtained using SAM2. Both object selection and cross-scene association are performed automatically, without manual intervention. We will clarify these details in the revised manuscript.

Q4: Can the authors report actual task completion alongside the geometric success criteria, as well as the end2end inference time?

Our real-world evaluation employs strict geometric success criteria that capture key task requirements. We believe these criteria provide evidence of MatchingPolicy's ability to transfer demonstrated spatial relationships and motions to real-world scene layouts, particularly given that the policy is trained solely on synthetic trajectories and deployed without real-world fine-tuning. Due to safety and cost considerations, we primarily adopt geometric success criteria rather than evaluating full task completion, which we leave for future work.

In the real-world deployment, our inference time mainly consists of tracking and policy computation. On an RTX 4090, interaction-graph preparation and encoding take ~80 ms, while diffusion policy inference takes ~60 ms per planning. For tracking, CoTracker takes ~40 ms per update on average.

[1] Heppert, Nick, et al. "Ditto: Demonstration imitation by trajectory transformation." IROS 2024.

[2] Leroy, Vincent, et al. "Grounding image matching in 3d with mast3r." ECCV 2024.

Thank you for the additional experiments. The retargeting comparison and estimated-correspondence evaluation strengthen the paper. However, substantial correspondence-related failures, untested camera dependence, and geometric success criteria that do not establish actual pouring or cutting still limit the evidence for robust real-world manipulation. My assessment is borderline, leaning toward rejection, so I maintain my score under this year’s rating scale.

Reviewer_PKLHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MatchingPolicy, an in-context imitation learning framework that explicitly separates visual correspondence extraction from action prediction. It combines a correspondence-aware graph diffusion policy, synthetic correspondence-rich training data, and a VFM-based matching pipeline. Experiments on RLBench and real-world manipulation tasks show its effectiveness.

Critical strengths
  1. Explicit correspondence conditioning is a meaningful design choice for cross-object manipulation, as it also enhances interpretability.

  2. The reported results, especially on unseen RLBench tasks and real-world object variations, are promising.

Critical weaknesses
  1. The contribution is unclear. The graph diffusion policy, keypoint-based action representation, and synthetic demonstration training are closely related to Instant Policy. The principal contribution appears to be explicit correspondence conditioning, but its independent benefit is not sufficiently isolated.

  2. Insufficient experimental ablations. The improvements over Instant Policy may reflect differences in correspondence supervision, graph structure, and synthetic training data. These factors are not adequately disentangled. Detailed ablation studies should be provided to better distinguish the proposed method from the baseline method.

Questions for authors

Please refer to Weakness.

2 comments

[W1–W2] Contribution and isolation of benefits.

“The principal contribution appears to be explicit correspondence conditioning, but its independent benefit is not sufficiently isolated.”

“These factors are not adequately disentangled.”

Thank you for this helpful assessment. We agree that explicit correspondence conditioning is our primary contribution relative to Instant Policy. We distinguish two components—correspondence-based input construction and explicit correspondence connections—and isolate their benefits through controlled ablations in the Common Response, using matched training data, budgets, and execution settings. We will clarify these contributions and incorporate the ablations into the revised manuscript.

Thanks for the authors' response. I have no other questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.