- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces mmClaw, a synthetic radar-search testbed, and uses it to ask whether a coordinate that can be decoded from a policy’s hidden state is actually used by the policy to act. Two teachers (one steering toward a public location estimate, one toward a return centroid) train paired small imitation policies, and the teacher shifts which coordinate is relatively easier to read out (D = 0.156 [0.125, 0.236] on 48 new layouts). The authors further describe a full system from radar readings to LLM based planning, and probe behavior under different sensing conditions.
Critical strengths
- important problem in real applications
- working to release publically available simulation setup
- comprehensive from sensor to actions
- looking at a novel phenomena -- of how coordinate choices affect the infered latent representation
Critical weaknesses
- need better clarity on the assumptions, and probe them one-by-one, or state they are a mean to an end -- between probing architecture choice for a world model, probing the effect of coordinate changes, an LLM planner, or providing a good sensing simulation environment, the paper does not go deep neither in explanation nor in probing of the aspects.
Questions for authors
Clarity and accessibility. The paper is hard to follow for a reader outside its development history/context, which limits its scientific value. Key terms are used before they are defined or never defined (e.g., “request,” “this design,” and abstract jargon such as “native evidence responses”). The architecture figures show an LLM planner and tool-call interface that the experiments never execute (Appendix N). Many results the conclusions rest on (e.g., Table 33 for the action-head audit, Table 35 for the retention sensitivity, and the native-response procedure in Appendix D) are in the appendix with no pointers from the main text. The headline D = 0.156 has no table of its underlying R² values, and cohorts are labeled inconsistently (“new,” “additional,” “prior,” “development”), so values such as 0.156 and 0.140 are hard to tie to specific layouts and policy counts. Meanwhile, much of the appendix is earlier “historical” material that does not connect to the main claims. Because the central claim is a subtle one, the reader must be able to verify exactly what was tested, and the current presentation makes that impractical. I recommend a restructured version with one clearly stated hypothesis, up-front definitions, and a single evidence map linking each main claim to its supporting table, cohort and policy count, marked pre-specified or exploratory.
- Abstract - define D, LEACE before using
- Some phrases, such as “the supplied manuscript table”, “The supplied study reports..” make it seem like the authors left AI interaction without properly examining/clearing the text, beyond what is acceptable under the “The authors remain responsible for.. the final submission.” Please verify. Similarly, the “authors remain responsible for.. supplying the missing experiment sources.”, and the phrasing the experiment README.md, stating “Node dependencies, virtual environments, nested checkouts and private tool caches are not required and are not included.” seems like there wasn’t oversight of the AI results, e.g. the experiments folder
- active perception in the intro -- cite also “active vision” / John Aloimonos et al., in addition to Bajcsy’s paper, as the 2 papers were published at around the same time.
- L032, explicit - please rephrase, be specific (yes, noisy radar returns make it harder to get a good enough intermediate posterior to make active perception work, but be specific about what you say)
- L071, “This design tests spatial representations without..” - specify what about the design
- the figures mention an llm based planner, but the text doesn’t detail it, pls explain
- L111 - LTrace comment - pls explain what is it, which characteristics, it’s mentioned briefly in the text, references in the zip file, but if this is the real environment, either provide more information or provide an alternative, available/reproducible simplified setup
- section 2 - does not explain why the simulation setup is auditable
- table 1 - mention “adverse” settings, but doesn’t specify what kind of adversarial condition.
- L153 - a request -- not clear request to what from here -- sounds like this refers to the LLM policy, but this should be clear in the paper.
- “null-component projections” - cite a related domain literature that uses this, as not standard in ICLR.
- table 33 - would be good to state direction of improvement, best in column, and relate to the gap after direction
- “direction erasure” -- would be good to define it. Also, is there an equivalent of perturbing, not erasing?