- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes CEM-Agent for compositional vision-and-language navigation, where instructions require linking multiple observations (e.g., “pass the red doorway, turn left, find the chair, and return to the same doorway”). A frozen VLM (Qwen3.6-27B) serves as the planner, paired with a Compositional Evidence Memory (CEM). CEM combines past observations and motion records into evidence for the current subgoal, and reports missing evidence back to the planner to guide its next tool call. A progress gate checks both semantic evidence and execution status before accepting subgoal completion. The authors run a 2×2 comparison (fixed vs. autonomous tool scheduling × generic memory vs. CEM) on 100 navigation episodes from OmniNavBench, report that CEM’s gain is 2 percentage points larger under autonomous scheduling, and analyze the mechanism with fixed-history tests and an intervention that hides missing-evidence feedback.
Critical strengths
-
Clear experimental design idea. The 2×2 factorial comparison tests the interaction between evidence composition and tool scheduling, rather than only showing that the full system performs best.
-
Clear evidence semantics. The paper distinguishes support, refutation, and insufficiency, and explicitly treats missing support as insufficient rather than as refutation.
Critical weaknesses
-
Poor presentation. The paper is hard to follow and full of defensive, AI feedback-style statements, for example:
- "Equal interaction point estimates establish neither statistical equivalence nor a reliably positive interaction in either track." (Sec 4.4)
- "A 5% development error target is not a test-time risk guarantee." (Sec 3.5)
- "Aggregate scores alone cannot reconstruct the paired interaction uncertainty." (Reproducibility Statement)
These sentences read like pre-emptive responses to review comments rather than part of a coherent research paper.
-
The core claim is not statistically supported. The paper's core hypothesis is the interaction between CEM and autonomous scheduling, but the interaction is only +2 percentage points, based on 100 episodes with a single run each. The standard error of a single success rate is already about 5 points, so +2 is well within noise, and the authors themselves admit the result is only "descriptive." The gap-feedback intervention (47% vs. 51%) has the same problem.
-
No confidence intervals, small evaluation scale, and no external baselines. The authors say per-episode outcomes were not retained, so confidence intervals cannot be computed. But this is the most basic record of any experiment, so the explanation does not hold. The evaluation uses only 100 selected episodes from OmniNavBench, and all comparisons are internal variants, with no comparison to existing methods such as AgenticNav or NavGPT.
-
The same model plays all key roles. The planner, compiler, grounder, and verifier are all the same Qwen3.6-27B with different prompts. Their errors are likely highly correlated, so the claim of independent verification does not hold.
-
No code release and no computation cost. The Reproducibility Statement only lists what reproduction would require, without providing any materials. The authors also admit that a lower wasted-read fraction does not imply lower compute.
Questions for authors
I have nothing to ask.