- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CF-WAM studies how the choice of future prediction target affects action learning. During training, it samples one of four targets: RGB, semantic features, depth, or interaction segmentation. These targets share a video representation and are specified through the language condition. The model also learns from human and robot demonstrations using a partly shared action space. Experiments on RoboCasa-GR1, LIBERO-Plus, and six real-world tasks show improvements over the reported baselines, with ablations examining the target representations and the use of human data.
Critical strengths
-
The main idea is straightforward and useful. Sampling one future representation at a time gives the model access to several types of supervision without requiring it to predict all of them in every update. Table 2 provides a useful check: success rises from 76.25% with RGB alone to 79.00% as the other projections are added. The individual gains are modest, but the trend supports the design choice.
-
The experimental results are encouraging across several settings. On RoboCasa-GR1, the full model reaches 82.50%, compared with 75.17% for WALA. I found Table 10 particularly helpful: keeping the Human-Robot action-space design fixed, replacing visual supervision with dynamic supervision raises success from 78.25% to 82.50%. This makes the benefit of the proposed training scheme easier to assess. The LIBERO-Plus and real-world results add support, and the 84.00% real-world average is achieved with a single projection, without oracle selection.
Critical weaknesses
-
My main reservation concerns the explanation for the static baseline's poor performance. Appendix A.3.1 matches total training compute, which is a sensible comparison for practical use. However, the static model processes four future streams per update, so the number of updates it receives also matters. The 68.75% versus 79.00% result supports the efficiency of dynamic supervision, but it does not establish that attention competition is the main cause of the gap. I would be more convinced by a comparison of training curves against optimizer updates. The attention plots alone do not settle this point.
-
I would also like to see a small check of what the frozen video VAE retains from the non-RGB targets. Depth maps and segmentation masks contain sharp boundaries that may not survive the shared encoding equally well. The downstream results suggest that these targets are useful, but they do not tell us which details the model can actually learn from them. A few before-and-after reconstructions, with a simple depth or boundary error measure, would help.
Questions for authors
-
Under the matched compute budget in Appendix A.3.1, how many optimizer updates does each variant receive? Do the training curves suggest that Static Multi-State is still improving when training stops?
-
How well does the frozen Wan2.2 VAE reconstruct the depth and interaction-segmentation targets? Could you show a few examples before and after encoding and decoding, particularly around object boundaries?
-
Table 4 shows that the best inference projection varies across tasks. For a new task, which projection would you use by default, and what evidence would guide a different choice without relying on test outcomes?
3 comments
Thank you for the detailed responses and the additional experiments, which largely address my concerns. In particular, on why Static Multi-State falls below Dynamic-State, the added comparison under matched updates shows that most of the compute-matched gap disappears. I think this means that training efficiency is not just one advantage of dynamic sampling over static multi-state supervision but the main one.
For the remaining 2.42 points, I agree with the response that competition among the future projection streams may play a role. However, the attention plots in Fig. 7 appear to use the compute-matched Static model, so its diffuse pattern may still reflect fewer updates. Section 4.3.3 could therefore present competition as a hypothesis instead of the cause, and report the comparison under matched updates as well.
The new results are consistent with my original assessment, so I remain positive about the paper and keep my score of 3.
We sincerely thank you for taking the time to review our responses and for your thoughtful suggestions. We appreciate your recognition of training efficiency as one of the advantages of dynamic sampling over static multi-state supervision. We will clarify this benefit in Section 4.3.3.
Please let us know if there are any specific questions or anything holding back raising the score. We would be happy to provide any additional information needed.
To Question 1 (Static Multi-State), Question 2 (VAE reconstruction) and Question 3 (Inference projection selection)
We sincerely thank you for the time and care you have devoted to reviewing our manuscript, and for your encouraging assessment of CF-WAM’s design and experimental results. We particularly appreciate your insightful perspective on both the methodological and practical implications of our approach. Your questions about training efficiency, target reconstruction, and inference-time projection selection have helped us strengthen our experimental analysis and explain our design choices more clearly. We are grateful for these thoughtful and constructive suggestions. We have carefully considered each point, added relevant experiments and clarifications, and addressed your questions point by point below.
To Question 1 (Static Multi-State):We thank the reviewer for highlighting the importance of optimizer update counts when interpreting these results. Training efficiency is one advantage of dynamic supervision. Dynamic-State predicts one future projection per sample, whereas Static Multi-State processes all four within each update. Under the original matched-compute budget, Static Multi-State received approximately 26k optimizer updates, while all other variants received 100k. The training curves plotted against optimizer updates show that Static Multi-State was still improving at 26k updates. With approximately twice the original compute budget, its success rate reached 75.92%. When trained for the same 100k optimizer updates as Dynamic-State, its success rate further increased to 76.58%, remaining 2.42 percentage points below Dynamic-State’s 79.00%, with approximately four times the total training compute. Additional updates narrowed the gap observed under the original matched-compute budget. With optimizer update counts matched, Dynamic-State still achieved a higher success rate using approximately one-quarter of the training compute. Figure 7 shows more diffuse attention from the static variant’s action branch to video tokens, suggesting competition among future projection streams for the action branch’s attention. With the batch size and video specifications held constant, CF-WAM can accommodate additional supervision modalities with almost no increase in per-update training compute. Static Multi-State must process an additional video stream at every update for each new modality, making multimodal scaling more difficult under a fixed compute budget.
To Question 2 (VAE reconstruction):We encode and decode depth maps and interaction segmentation masks using the frozen Wan2.2 VAE. The evaluation covers four subsets and 1,000 unique frames. We compare each reconstruction with its corresponding target before encoding and average the metrics across frames. Depth reconstruction achieves an MAE of 2.42 mm. For interaction segmentation, the foreground mIoU is 99.85%, and the Boundary F1 score is 99.996% with a 3-pixel tolerance, indicating high agreement between the reconstructed masks and the original targets in both foreground regions and boundaries. If the paper is accepted, we will include side-by-side comparisons of the original depth maps and interaction segmentation masks, their VAE reconstructions, and enlarged views around object boundaries in the appendix of the final version.
To Question 3 (Inference projection selection):CF-WAM’s dynamic projection sampling is primarily used during training. For each training sample, one of the four future projections is sampled. At inference, the same checkpoint predicts only the selected projection.
For new real-world tasks, we use Interaction by default. It achieves the highest average success rate among the individual projections on both the six tasks in Table 4 and the five OOD settings in Table 11, at 84.00% and 79.20%, respectively. Based on the average results in Table 4, Interaction, Visual, and Geometric perform similarly, and any of them can serve as a fixed inference projection. Semantic has a slightly lower average success rate.
Beyond the default setting, task instructions and characteristics can provide heuristic guidance if a projection is to be selected for a particular task. Geometric may be considered for tasks that depend more heavily on 3D spatial relationships. Semantic may be considered for tasks emphasizing object categories or semantic distinctions. Visual may be considered when task progress is primarily assessed through appearance or deformation, such as towel folding.
In future work, we will further explore agent-driven adaptive projection selection, allowing an agent to directly adjust the language prompt based on the task instruction, current observation, and execution stage to select the future projection to predict.