W1. The "unseen-task" evaluation is not zero-shot, and the comparison is not apples-to-apples. At test time the retrieval pool (Context80 / Context48) contains 8 expert demonstrations with action labels for every evaluation task, including the "unseen" ones. GICAT is therefore evaluated in an effectively 8-shot setting, while UniPi and DreamGen never see these demonstrations. The claim of stronger cross-task generalization should be qualified, and the baselines should receive the same data (e.g., fine-tuning on the same 8 demonstrations per task). A retrieval-only baseline is also needed: execute the action chunk of the top-1 retrieved snippet (VINN-style). This would show whether the transformer does more than a nearest-neighbor lookup.
W2. The Meta-World generalization gain comes mostly from tasks that mirror a training task.
- Three of the five unseen tasks are reverses of training tasks on the same object: door-close / faucet-close / window-open vs. door-open / faucet-open / window-close. On these three, GICAT reaches 72.2% vs. 3.3% for DreamGen.
- The other two unseen tasks (shelf-place, handle-press) have no same-object training counterpart. On them GICAT reaches 20.0%, below DreamGen (31.7%) and UniPi (23.3%).
- Fig. 5 is consistent with this. With Context40, whose pool has no demonstrations of the unseen tasks, door-close and faucet-close already reach 83.3% and 63.3%.
- On ManiSkill, PullCube (the reverse of the training task PushCube) succeeds only 1/30.
Please explain how the unseen tasks were chosen, and evaluate on held-out tasks without such counterparts (e.g., the ML10/ML45 test splits).
W3. No real-robot experiments. All results are in simulation (Meta-World and ManiSkill). The paper targets robot manipulation and builds on a large video generation model. Without real-robot validation, the practical value of the method is hard to assess. This concerns visual realism, the effect of the arm mask on real images, inference latency, and robustness to real-world dynamics. The authors list this as a limitation, but I do not consider it an acceptable one. For a robotics paper at ICLR, real-robot experiments are necessary, not optional.
W4. The core components are not ablated. At minimum, the following three ablations are needed:
- Retrieval: a same-architecture model trained without retrieved context. The existing Context0/40/80 ablation only removes context tokens at test time from a model always trained with them. This input is out of distribution, so it does not measure the value of retrieval. The collapse of Context0 on training tasks (faucet-open 100%→0%, drawer-close 100%→≈10%) also suggests that the model relies heavily on copying retrieved actions.
- SAM3 arm masking: with vs. without.
- LILA features: vs. SigLIP2, which the DreamGen baseline uses.
W5. Technical issues in the method.
- (a) Top-N selection is non-differentiable and q_t is used only for retrieval, so it is unclear how η_q is trained.
- (b) Sec. 3.2 / Fig. 2 describe precomputed keys and frozen encoders. App. A.1.2 says these encoders are learned and the memory is rebuilt every epoch.
- (c) The continuity regularizer (Eq. 9/18) is applied to predicted velocities, whose flow-matching targets contain i.i.d. noise across steps. It therefore conflicts with the training objective. Smoothness should be imposed on the predicted clean action.
- (d) Training-time retrieval excludes only the exact (trajectory, t) pair. Adjacent timesteps from the same trajectory can still be retrieved, which leaks labels.
W6. Baselines appear under-tuned and under-specified. DreamGen's IDM reaches only 6.7% on the seen task faucet-open, where UniPi reaches 83.3%. On ManiSkill, UniPi and DreamGen have identical success counts on all six tasks. Please specify whether the baselines use the same inputs, chunk length, data and training budget. Please also consider stronger baselines, such as AVDC, Seer, and a BC policy trained on the same demonstrations.
W7. Cost is not reported. The 5B video model is re-queried every 4 environment steps, with 50 sampling steps on ManiSkill. Per-round latency, wall-clock time per episode and the number of planning rounds are not given.
W8. Related work is missing. The core idea is retrieval-augmented / in-context imitation, but the paper does not position itself against VINN, Behavior Retrieval, Prompt-DT, ICRT, Instant Policy, REGENT or RICL. Predictive inverse dynamics (Seer) is also not discussed.
W9. The figures are inconsistent with the method.
- Fig. 2:
- It feeds q_t into the transformer, whereas Eq. 15 uses z_t, z_g, z_s.
- It retrieves by comparing q_t with context tokens that include action features, whereas the text says the keys are action-free.
- It marks all encoders as frozen, including the state and action MLPs.
- It omits the flow-matching inputs (noise, τ, noisy action tokens).
- Figs. 1–2 label the demonstrations with query-time symbols and reuse the query frame for both "different" demonstrations.
- The Meta-World "Execution" frames in Figs. 3/6/8(a) are 832×480 with no visible padding. This conflicts with App. A.1.1/B.1, which describes 256×256 renders padded to 480×832. Please clarify how these frames were produced.
- Other presentation issues:
- The video planner is fine-tuned on all evaluation tasks, including the "unseen" ones. The paper should state that "unseen" applies to the IDM only.
- The notation π_θ(V_g | C_t) in Eq. 3 is non-standard.
- Sec. 3.3 cites Fig. 2 for the closed loop.
Thanks for the additional experiments. After looking through the new tables, I still have three main concerns:
From Table 2(a), the retrained no-context model gets 90.7% on Meta-World seen tasks, which is the exact same average as full GICAT (90.7%, and 80.0% vs. 66.7% on Pick Place). So the seen-task gap over DreamGen (44.7%) does not come from retrieval, but from the extra state input, SAM3 mask, or LILA features. On the unseen tasks (Table 4), simply running the top-1 retrieved chunk (VINN-style) already gets 45.3% vs. 51.3% for GICAT (only 9 more successes out of 150 rollouts). What does VINN-style get on ManiSkill? Right now it is not clear how much the flow-matching transformer adds on top of nearest-neighbor retrieval.
On W2, GICAT still averages only 20.0% on the two unseen tasks that are not reverses of training tasks (Shelf Place and Handle Press), which is lower than both 0-shot and fine-tuned DreamGen and UniPi. Meanwhile, the no-context model already gets 73.3% on Door Close without any retrieval. Also, some baseline numbers look strange: why does fine-tuning DreamGen on Context80 drop Button Press from 86.7% to 0% and Pick Place from 56.7% to 0% when seen-task demos are in that pool? Why does ICRT get 0% on Reach vs. 90% on Pick Place, and why are UniPi and DreamGen still identical on every ManiSkill task?
W3 (no real-robot experiments) was not addressed, and ~12.5 s per visual plan on an H200 every 4 steps makes real-world deployment even harder to judge. On W5: (c) in flow matching, u_tau = A_1 - A_0 contains independent noise A_0 at each step, so smoothing predicted velocities conflicts with canceling that noise during integration (smoothness should be on the predicted clean action); (d) if the context pool is completely separate from the IDM training episodes, why is the exact (trajectory, t) mask in App. A.1.2 needed at all?