acceptodds
Under review as a conference paper at ICLR 2027

GICAT: Goal-conditioned In-Context Action Translation for Generalizable Robot Manipulation

Abstract

General-purpose robots need to interpret diverse task goals, anticipate feasible changes in the environment, and realize those changes through embodiment-specific control. Video generation models provide a promising action-free planning interface, but their planning predictions cannot be directly executable by robots. Therefore, we present GICAT, a decoupled closed-loop framework that separates video planning from action generation. Given a current image and task instruction, a video generation model in GICAT predicts a short-horizon visual future. Unlike frame-by-frame action translation, an in-context action translation part of GICAT conditions a flow-matching model on the current image, goal image, robot state, and retrieved in-context demonstrations to generate executable action chunks. By training the action translation model as an in-context learner, we can apply it to unseen tasks during the inference stage without any weight updates. During execution, GICAT follows the predicted action chunk in a receding-horizon loop and replans from real environment observations. GICAT outperforms decoupled video-to-action baselines on Meta-World and ManiSkill and shows stronger cross-task generalization to unseen tasks during in-context action translation training. These results highlight the potential of decoupled video planning and in-context action learning as an effective route toward generalizable video-model-based robot control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
inverse dynamics model, in-context learning, world model, robot manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d8AfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GICAT pairs a pretrained Wan2.2-TI2V-5B planner, adapted with LoRA, with a demonstration-conditioned action transformer. Frozen LILA features from robot-masked current/goal images, robot state and four retrieved demonstration tokens condition eight-action chunks. Separate planners and action models are trained for Meta-World and ManiSkill. Held-out action-model tasks appear in planner training and receive demonstration support at evaluation; adaptation changes the memory without updating action-model weights.

Critical strengths
  1. The Meta-World result is substantial: 71.0% overall success versus DreamGen's 29.7% and the UniPi-style baseline's 17.7%, using the same planner and rollout loop (Fig. 4). This establishes practical value for the complete translator.
  2. Demonstration-conditioned action translation offers adaptation without retraining. Figure 5 shows that changing the context of a fixed model can improve execution on tasks excluded from its weight training.
Critical weaknesses
  1. The contribution beyond existing video-to-action and demonstration-conditioned policies is not clearly established. The particular combination may be useful, but its distinction from prior in-context/retrieval approaches needs clearer explanation and a suitable demonstration-conditioned baseline. The current comparisons also differ in state access and visual representation, leaving the retrieval/context increment unclear. Figure 5 helps, but Context0's training status is unspecified, and Context40/80 change memory size and task coverage together.
  2. The retrieval map needs clarification. Eq. (12) and Algorithm 1 use eta_q only for hard Top-N selection, without specifying an objective for any separate trainable parameters. Shared visual/state projections can still learn through the transformer; the question concerns the query map's implementation (Appendix A.1.2).
Questions for authors
  1. What distinguishes VG-IDM from prior in-context/retrieval policies? Can you provide a suitable demonstration-conditioned comparison with matched planner, inputs and support data?
  2. Was Context0 trained separately? What controls or existing analysis separate the retrieval/context benefit from changes in input features, memory size and target-task support?
  3. Is eta_q fixed or learned? If it has separate learned parameters, what objective updates them through the hard retrieval step?
Reviewer_G2A7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GICAT, a two-stage framework for video-based robot manipulation. A fine-tuned video model predicts a short future video from the current image and instruction, and its last frame is used as the goal image. Then a flow-matching inverse dynamics model (VG-IDM) predicts action chunks from the current and goal images, the robot state, and retrieved demonstrations as in-context examples. Experiments on Meta-World and ManiSkill demonstrate the effectiveness of GICAT.

Critical strengths
  • The topic is important, and using retrieved demonstrations as in-context examples for the IDM is a nice idea.
  • The comparison of action models is controlled, since all methods share the same video planner and closed-loop execution. On the Meta-World seen tasks GICAT is much better than the baselines (90.7% vs 44.7% and 26.0%).
Critical weaknesses
  • The biggest weakness and the main question for this paper is the unseen-task generalization claim. At test time GICAT retrieves eight expert demonstrations of every evaluation task, including the unseen ones, while the baselines get no demonstrations. Without context (Context0), GICAT gets 0% on all unseen tasks (Fig. 5). Also, three of the five unseen tasks (door-close, faucet-close, window-open) are the reverse of training tasks, and the video planner is fine-tuned on all evaluation tasks (App. B.1). So I don't think these tasks are really unseen for the system.
  • The baselines differ from GICAT in many aspects at once (robot state, retrieval, visual features, masking), and their training details are not reported. Some results look odd: DreamGen gets only 6.7% on faucet-open, an IDM training task, and the UniPi and DreamGen columns in Table 1 are identical. Also, if the 39-D Meta-World state is the standard observation, it contains object and goal positions, which DreamGen does not get.
  • The experiments are limited. ManiSkill results are very low (17.8% overall, 0% on PickCube and StackCube for all methods), and the unseen gain there is 5/60 vs 3/60 successes. There are no real-robot experiments, and the only ablation is the retrieval pool; the goal image, robot state, masking and replanning are not ablated.
Questions for authors
  • Are the retrieved demonstrations separate from the IDM training episodes and the evaluation initial states? During training only the exact trajectory-time pair is masked, so can neighboring timesteps of the same trajectory be retrieved?
  • What does the 39-D Meta-World state contain? If it includes object and goal positions, please report a proprioception-only version.
  • Can you add baselines that get the same eight demonstrations per task (e.g., DreamGen fine-tuned on them)? If GICAT still clearly wins on the unseen tasks, I would consider raising my score.
Reviewer_ZLaWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

GICAT decouples video planning from action generation. A fine-tuned Wan2.2 video model generates future frames from the current image and the instruction. The last generated frame serves as the goal image. VG-IDM then predicts an 8-step action chunk. The robot executes 4 of these steps and then replans.

VG-IDM:

  • encodes only the robot arm, using SAM3 masks and LILA features;
  • retrieves the 4 most similar expert demonstration snippets as in-context tokens;
  • generates actions with flow matching;
  • adapts to new tasks by changing the retrieval pool, with no weight updates.

All methods share the same video planner.

  • Meta-World: GICAT reaches 71.0% overall, vs. 29.7% for DreamGen and 17.7% for UniPi. On unseen tasks it reaches 51.3%, vs. 14.7% for DreamGen.
  • ManiSkill: GICAT reaches 17.8%, vs. 3.9% for both baselines.

At test time, the retrieval pool also includes demonstrations of the unseen tasks. There are no real-robot experiments.

Critical strengths

S1. Clear problem focus. The paper targets a real bottleneck: turning a video plan into executable actions. It uses a modular design that separates video planning from action generation. All methods share the same video planner, so the contribution of the IDM itself can be compared on its own.

S2. Adapts to new tasks without updating weights. Changing only the retrieval pool, with the model parameters fixed, raises the average success rate on unseen tasks from 29.3% to 51.3% (Context40 → Context80). Window-open goes from 0% to 60% and handle-press from 0% to 40%. This is useful for practical deployment.

S3. Strong results on seen tasks and good reproducibility. On Meta-World seen tasks, GICAT reaches 90.7% success, compared with 44.7% for DreamGen and 26.0% for UniPi. The paper provides full hyperparameter tables, per-task data statistics and algorithm descriptions, so the work is easy to reproduce.

Critical weaknesses

W1. The "unseen-task" evaluation is not zero-shot, and the comparison is not apples-to-apples. At test time the retrieval pool (Context80 / Context48) contains 8 expert demonstrations with action labels for every evaluation task, including the "unseen" ones. GICAT is therefore evaluated in an effectively 8-shot setting, while UniPi and DreamGen never see these demonstrations. The claim of stronger cross-task generalization should be qualified, and the baselines should receive the same data (e.g., fine-tuning on the same 8 demonstrations per task). A retrieval-only baseline is also needed: execute the action chunk of the top-1 retrieved snippet (VINN-style). This would show whether the transformer does more than a nearest-neighbor lookup.

W2. The Meta-World generalization gain comes mostly from tasks that mirror a training task.

  • Three of the five unseen tasks are reverses of training tasks on the same object: door-close / faucet-close / window-open vs. door-open / faucet-open / window-close. On these three, GICAT reaches 72.2% vs. 3.3% for DreamGen.
  • The other two unseen tasks (shelf-place, handle-press) have no same-object training counterpart. On them GICAT reaches 20.0%, below DreamGen (31.7%) and UniPi (23.3%).
  • Fig. 5 is consistent with this. With Context40, whose pool has no demonstrations of the unseen tasks, door-close and faucet-close already reach 83.3% and 63.3%.
  • On ManiSkill, PullCube (the reverse of the training task PushCube) succeeds only 1/30.

Please explain how the unseen tasks were chosen, and evaluate on held-out tasks without such counterparts (e.g., the ML10/ML45 test splits).

W3. No real-robot experiments. All results are in simulation (Meta-World and ManiSkill). The paper targets robot manipulation and builds on a large video generation model. Without real-robot validation, the practical value of the method is hard to assess. This concerns visual realism, the effect of the arm mask on real images, inference latency, and robustness to real-world dynamics. The authors list this as a limitation, but I do not consider it an acceptable one. For a robotics paper at ICLR, real-robot experiments are necessary, not optional.

W4. The core components are not ablated. At minimum, the following three ablations are needed:

  • Retrieval: a same-architecture model trained without retrieved context. The existing Context0/40/80 ablation only removes context tokens at test time from a model always trained with them. This input is out of distribution, so it does not measure the value of retrieval. The collapse of Context0 on training tasks (faucet-open 100%→0%, drawer-close 100%→≈10%) also suggests that the model relies heavily on copying retrieved actions.
  • SAM3 arm masking: with vs. without.
  • LILA features: vs. SigLIP2, which the DreamGen baseline uses.

W5. Technical issues in the method.

  • (a) Top-N selection is non-differentiable and q_t is used only for retrieval, so it is unclear how η_q is trained.
  • (b) Sec. 3.2 / Fig. 2 describe precomputed keys and frozen encoders. App. A.1.2 says these encoders are learned and the memory is rebuilt every epoch.
  • (c) The continuity regularizer (Eq. 9/18) is applied to predicted velocities, whose flow-matching targets contain i.i.d. noise across steps. It therefore conflicts with the training objective. Smoothness should be imposed on the predicted clean action.
  • (d) Training-time retrieval excludes only the exact (trajectory, t) pair. Adjacent timesteps from the same trajectory can still be retrieved, which leaks labels.

W6. Baselines appear under-tuned and under-specified. DreamGen's IDM reaches only 6.7% on the seen task faucet-open, where UniPi reaches 83.3%. On ManiSkill, UniPi and DreamGen have identical success counts on all six tasks. Please specify whether the baselines use the same inputs, chunk length, data and training budget. Please also consider stronger baselines, such as AVDC, Seer, and a BC policy trained on the same demonstrations.

W7. Cost is not reported. The 5B video model is re-queried every 4 environment steps, with 50 sampling steps on ManiSkill. Per-round latency, wall-clock time per episode and the number of planning rounds are not given.

W8. Related work is missing. The core idea is retrieval-augmented / in-context imitation, but the paper does not position itself against VINN, Behavior Retrieval, Prompt-DT, ICRT, Instant Policy, REGENT or RICL. Predictive inverse dynamics (Seer) is also not discussed.

W9. The figures are inconsistent with the method.

  • Fig. 2:
    • It feeds q_t into the transformer, whereas Eq. 15 uses z_t, z_g, z_s.
    • It retrieves by comparing q_t with context tokens that include action features, whereas the text says the keys are action-free.
    • It marks all encoders as frozen, including the state and action MLPs.
    • It omits the flow-matching inputs (noise, τ, noisy action tokens).
  • Figs. 1–2 label the demonstrations with query-time symbols and reuse the query frame for both "different" demonstrations.
  • The Meta-World "Execution" frames in Figs. 3/6/8(a) are 832×480 with no visible padding. This conflicts with App. A.1.1/B.1, which describes 256×256 renders padded to 480×832. Please clarify how these frames were produced.
  • Other presentation issues:
    • The video planner is fine-tuned on all evaluation tasks, including the "unseen" ones. The paper should state that "unseen" applies to the IDM only.
    • The notation π_θ(V_g | C_t) in Eq. 3 is non-standard.
    • Sec. 3.3 cites Fig. 2 for the closed loop.
Questions for authors

Please see above weakness.

1 comment

Thanks for the additional experiments. After looking through the new tables, I still have three main concerns:

  1. From Table 2(a), the retrained no-context model gets 90.7% on Meta-World seen tasks, which is the exact same average as full GICAT (90.7%, and 80.0% vs. 66.7% on Pick Place). So the seen-task gap over DreamGen (44.7%) does not come from retrieval, but from the extra state input, SAM3 mask, or LILA features. On the unseen tasks (Table 4), simply running the top-1 retrieved chunk (VINN-style) already gets 45.3% vs. 51.3% for GICAT (only 9 more successes out of 150 rollouts). What does VINN-style get on ManiSkill? Right now it is not clear how much the flow-matching transformer adds on top of nearest-neighbor retrieval.

  2. On W2, GICAT still averages only 20.0% on the two unseen tasks that are not reverses of training tasks (Shelf Place and Handle Press), which is lower than both 0-shot and fine-tuned DreamGen and UniPi. Meanwhile, the no-context model already gets 73.3% on Door Close without any retrieval. Also, some baseline numbers look strange: why does fine-tuning DreamGen on Context80 drop Button Press from 86.7% to 0% and Pick Place from 56.7% to 0% when seen-task demos are in that pool? Why does ICRT get 0% on Reach vs. 90% on Pick Place, and why are UniPi and DreamGen still identical on every ManiSkill task?

  3. W3 (no real-robot experiments) was not addressed, and ~12.5 s per visual plan on an H200 every 4 steps makes real-world deployment even harder to judge. On W5: (c) in flow matching, u_tau = A_1 - A_0 contains independent noise A_0 at each step, so smoothing predicted velocities conflicts with canceling that noise during integration (smoothness should be on the predicted clean action); (d) if the context pool is completely separate from the IDM training episodes, why is the exact (trajectory, t) mask in App. A.1.2 needed at all?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.