- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces QuestWAM which is a spatial-query framework for pretrained WAMs. The spatial query module is trained on 3 different supervision signals which should help the WAM model to perform better on different benchmarks (RoboTwin 2.0 and Libero-Plus): future 3D bounding boxes, interaction signals with task-relevant objects and object semantics. The overall method performs 2-3 percent better on RoboTwin 2.0 and 3-4 percent better on Libero-Plus compared to FastWAM. They further investigate their hypothesis that WAMs need a better understanding of geometry and interaction around task-relevant objects instead of reconstructing every little detail of a scene using a systematic representation study.
Critical strengths
- Real robot experiments
- Hypothesis which would be important in the field of robotics and WAMs
- The latent vs decoded readout results are a useful finding
Critical weaknesses
Weaknesses
- It is hard to follow the method section along, as it seems that the supporting figures and text are using different names for the same representations. Making this the same for both would benefit the clarity of the paper.
- It is also not understandable what the actual architecture of QuestWAM is. Is the model build upon the architecture of FastWAM and includes this query module or is it its own architecture all together, where any WAM can be plugged in? If so, then more WAM variants need to be evaluated to confirm that this additional querying process actually increases performance.
- The “Systematic Representation Study” is not yet a standalone contribution and hovers between ablations and an actual study. To make it a proper study it needs explicit hypotheses comparisons, multiple seeds per model, evaluation beyond RoboTwin Clean and a control for the extra training stage.
- There is only one real other baseline included (FastWAM), but no other models, which are also mentioned in related work, like DreamVLA, WAM4D, 4D-WAM or GaussianWAM. How do these models perform compared to the proposed method? Also is the evaluation the same for all models (e.g. same checkpoint choice, etc.)? Comparing against more baselines would strengthen the paper.
- The gain of the supervision components is not significant and it does compare the exclusion of only one at the same time, but it does not evaluate if two are missing or all 3 and we only have a plain model. The conclusion that these supervision signals help performance is therefore not conclusive. It is also not clear if the models were trained using different seeds, if only one seed was used the results and the advantages could be in the range of the variance.
- It is not clear if the stage 1 and stage 2 training is responsible for the higher success rate of QuestWAM and FastWAM should get the same training time
Minor Weaknesses
- Figure 1 is too overloaded and the right part is far too small. Only using the left part would benefit the paper much more and the right part can be an extra figure (if space allows it)
- Empirical results are not a contribution, but required in this field to prove that the proposed method is working
- Avoid empty Sections like 3 and just put the problem formulation directly below it without having the subsection there
- Figure 2 is cut at the bottom and top and has a mismatch with “…” between blocks. In general the font is also rather small.
- There is no table for the real-world robot results, instead a figure displaying the tasks. Moving this figure into the Appendix and adding a table with success rates would be more helpful.
Questions for authors
- How does the method identify task-relevant objects?
- What are semantic states?
- Where do you get the supervision information for the query model? How are interaction points defined? Are supervision information missing or complete?
- What exactly is used from the query module (because there is one arrow directly from the query model to the DiT action head and one from the outputed Queries)?
- What is RoboTwin 2.0 random? What modifications are included?
- FLEX-π uses a corrected Libero-Plus evaluation setup, where FastWAM increases performance from 49 to over 70 percent success. Could you report your results using the correct evaluation setup for FastWAM and QuestWAM?
- Why are the latency numbers for the QuestWAM model different in table 1 and 6 to 5 and 7?
Please also consider all weaknesses also as a question that should be answered.