acceptodds
Withdrawn from ICLR 2027

CEM-Agent: Coupling Tool Scheduling and Compositional Evidence for Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Compositional navigation requires binding observations across objects, encounters, and time before committing progress. We introduce CEM-Agent, which couples a frozen vision–language planner with Compositional Evidence Memory (CEM). CEM constructs source-linked witnesses for bounded task claims and exposes unresolved roles and relations to guide subsequent tool use. A shared gate checks semantic evidence and execution facts before accepting progress. We test whether evidence composition benefits more from autonomous than fixed tool scheduling through matched 2 × 2 comparisons on 100 compositional navigation episodes. With oracle pose, CEM raises strict episode success from 43% to 51% under autonomous scheduling, versus 34% to 40% under fixed scheduling. The resulting interaction is +2 percentage points; the estimated-pose comparison yields the same point estimate. On fixed histories, witness recovery improves from 69% to 79%. Exposing gap feedback raises autonomous CEM success from 47% to 51% and improves short-horizon gap resolution. These results support the usefulness of evidence diagnosis and its feedback to planning, while the small positive interaction remains descriptive because paired uncertainty is not established.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, LLM Agent, Agentic Planning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MfgMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CEM-Agent for compositional vision-and-language navigation, where instructions require linking multiple observations (e.g., “pass the red doorway, turn left, find the chair, and return to the same doorway”). A frozen VLM (Qwen3.6-27B) serves as the planner, paired with a Compositional Evidence Memory (CEM). CEM combines past observations and motion records into evidence for the current subgoal, and reports missing evidence back to the planner to guide its next tool call. A progress gate checks both semantic evidence and execution status before accepting subgoal completion. The authors run a 2×2 comparison (fixed vs. autonomous tool scheduling × generic memory vs. CEM) on 100 navigation episodes from OmniNavBench, report that CEM’s gain is 2 percentage points larger under autonomous scheduling, and analyze the mechanism with fixed-history tests and an intervention that hides missing-evidence feedback.

Critical strengths
  1. Clear experimental design idea. The 2×2 factorial comparison tests the interaction between evidence composition and tool scheduling, rather than only showing that the full system performs best.

  2. Clear evidence semantics. The paper distinguishes support, refutation, and insufficiency, and explicitly treats missing support as insufficient rather than as refutation.

Critical weaknesses
  1. Poor presentation. The paper is hard to follow and full of defensive, AI feedback-style statements, for example:

    • "Equal interaction point estimates establish neither statistical equivalence nor a reliably positive interaction in either track." (Sec 4.4)
    • "A 5% development error target is not a test-time risk guarantee." (Sec 3.5)
    • "Aggregate scores alone cannot reconstruct the paired interaction uncertainty." (Reproducibility Statement)

    These sentences read like pre-emptive responses to review comments rather than part of a coherent research paper.

  2. The core claim is not statistically supported. The paper's core hypothesis is the interaction between CEM and autonomous scheduling, but the interaction is only +2 percentage points, based on 100 episodes with a single run each. The standard error of a single success rate is already about 5 points, so +2 is well within noise, and the authors themselves admit the result is only "descriptive." The gap-feedback intervention (47% vs. 51%) has the same problem.

  3. No confidence intervals, small evaluation scale, and no external baselines. The authors say per-episode outcomes were not retained, so confidence intervals cannot be computed. But this is the most basic record of any experiment, so the explanation does not hold. The evaluation uses only 100 selected episodes from OmniNavBench, and all comparisons are internal variants, with no comparison to existing methods such as AgenticNav or NavGPT.

  4. The same model plays all key roles. The planner, compiler, grounder, and verifier are all the same Qwen3.6-27B with different prompts. Their errors are likely highly correlated, so the claim of independent verification does not hold.

  5. No code release and no computation cost. The Reproducibility Statement only lists what reproduction would require, without providing any materials. The authors also admit that a lower wasted-read fraction does not imply lower compute.

Questions for authors

I have nothing to ask.

Reviewer_pMfAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CEM-Agent, which integrates a frozen vision-language planner with a Compositional Evidence Memory (CEM) to address navigation tasks requiring the binding of object identities, specific encounters, temporal sequences, and spatial relationships. The system organizes evidence—traced back to its source—for each sub-goal, categorizing it as "supporting," "refuting," or "insufficient." It then feeds information regarding missing evidence back to the planner to guide subsequent tool invocations. Task progress is determined by a combination of semantic verification and execution state.

The authors evaluated the approach on a navigation subset of the OmniNavBench benchmark, comparing fixed versus autonomous tool scheduling, and a general collective memory versus CEM, across 100 identical tasks. Under conditions of ground-truth pose, CEM improved the strict success rate from 43% to 51% with autonomous scheduling, and from 34% to 40% with fixed scheduling—an interaction effect of 2 percentage points. A similar 2-percentage-point interaction effect was observed under estimated pose conditions. Furthermore, the evidence retrieval rate from the historical record increased from 69% to 79%, and providing feedback on missing evidence (filling "gaps") boosted the overall system success rate from 47% to 51%.

Critical strengths
  • The problem is clearly defined, and the semantic handling of evidence is rigorous. The paper distinguishes between "an event occurring in the real world," "the existence of sufficient evidence in the historical record," and "the sufficiency of a currently selected evidence set to support a conclusion." Notably, the authors do not equate a failure to retrieve evidence with evidence of its non-occurrence—a distinction that is practically significant for preventing the premature termination of a task. (pp. 3–5)
  • The experimental design effectively isolates specific sources of contribution. The study employs a 2×2 factorial design, with shared tools, verifiers, and progress-gating mechanisms; a "shared memory" baseline is also subjected to the same level of supervision and metadata. By utilizing fixed-history testing and interventions based on "gap feedback," the authors successfully disentangle the contributions of evidence selection from those of planning-based feedback—an approach far more compelling than a simple comparison between the full system and a basic baseline. (pp. 6–8)
  • The discussion regarding the scope and limitations of the results is measured. The authors explicitly acknowledge several nuances: the lack of paired uncertainty estimates for interaction effects; the fact that identical point estimates do not imply equivalence across different retrieval conditions; and the distinction between a reduction in "wasted" read operations and a decrease in total computational cost. These clarifications help prevent the overinterpretation of the experimental findings. (pp. 6–8, 14)
Critical weaknesses
  • The core interaction hypothesis lacks sufficient empirical support. The central research question—whether autonomous scheduling amplifies the benefits of aggregated evidence—remains inadequately addressed. In both localization conditions, the observed interaction effect is a mere 2 percentage points. Furthermore, each task was executed only once, with no data provided on task-specific paired outcomes, confidence intervals for scenario clusters, or analyses of stochasticity. The current findings suggest, at best, that individual components may be effective; they fall short of establishing a robust synergistic gain. While the authors acknowledge this limitation, such an acknowledgment cannot serve as a substitute for the missing evidence. (Tables 1 and 4)
  • Critical configuration details required for reproducibility are absent. The paper fails to provide essential specifications, including the structure of the evidence head, training hyperparameters, candidate pool size, beam search width, temperature settings, evidence cost coefficients, context/tool-use budgets, the specific workflow for fixed scheduling, the implementation details of the localization system, and the complete prompts used for various roles. Although page 9 outlines the materials necessary for reproduction, the paper does not supply the specific configurations or logs required to reconstruct the experiments.
  • Mechanistic evidence regarding feedback on "gaps" remains subject to interpretive limitations. While the distinction between hidden and open feedback represents a valuable intervention, the two strategies lead to divergent state trajectories; consequently, the "Resolve" metric may be based on sets of decision points that differ in inherent difficulty. Simply labeling instances as "recoverable" does not ensure that the underlying sets are comparable. It is recommended to conduct experiments that branch from identical initial "gap" states and to include control conditions—such as providing non-informative feedback of equal length—to better isolate the specific impact of the feedback content itself. (Pages 7–8, 13)
  • The necessity and specific contributions of individual design elements remain unverified. The study lacks targeted ablation analyses for components such as relational binding, smoothed bottleneck aggregation, gap prediction, search strategies, and the various training loss terms. The normalized smoothed minimum in Equation (3) is sensitive to both temperature and the scale of the modalities; the authors acknowledge this, yet they do not examine whether it affects cross-modality ranking or calibration. As a result, it is impossible to determine which specific design choices drive the observed performance gains. (Pages 4, 14)
  • Generalizability, cost implications, and the propagation of model errors remain unexamined. The results are confined to a select subset of simulated navigation tasks; the study lacks evaluation across a broader range of tasks, diverse base models, or varying levels of localization error. The planner, compiler, target locator, and verifier all utilize the same frozen model; therefore, the independence of prompts does not guarantee the independence of errors. Furthermore, in the absence of data regarding total call counts, latency, token consumption, or computational resource usage, it is difficult to determine whether the benefits justify the added complexity. (Pages 6, 8)
Questions for authors
  • Could you provide the per-task matching success results, scenario identifiers, and outcomes from multiple experimental runs across all conditions? Furthermore, could you report the confidence intervals for interaction effects, accounting for scenario-specific dependencies?
  • Is it possible to disclose the full prompts, model versions, training configurations, search parameters, fixed scheduling workflows, localization implementation details, and the complete list of tasks?
  • Who annotated the 4,000 evidence queries? How were the following criteria verified: that "all valid supporting evidence has been removed," that the "remaining evidence is sufficient for refutation," and that the "gap is recoverable"? Are there data on annotation consistency and the resolution of disagreements?
  • When execution branches from the same "gap" state, does explicit feedback still improve the recovery rate? How do the results change if the feedback provided is either uninformative or contains erroneous information (while maintaining the same length as the original feedback)?
  • Have you tested for malicious instructions embedded within visual signage, historical memory, or tool outputs? While source IDs allow for provenance tracking, how do you ensure that the content from these sources is treated strictly as evidence—rather than being executed by the planner or verifier as instructions with higher-level privileges?
Reviewer_gSyAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Navigation agents may mistake relevant observations for proof that a subgoal is complete. CEM-Agent addresses this problem by linking observations and action records through object identity, temporal order, and spatial relations. When evidence is incomplete, it reports the missing links to a frozen vision–language planner, which selects tools to retrieve history or gather new observations. A separate verifier and execution checks determine whether to accept progress. Experiments on 100 navigation episodes show that CEM improves both evidence verification and navigation success under autonomous tool scheduling.

Critical strengths
  • The paper identifies a subtle failure mode: individually relevant observations may not jointly justify task progress. This shifts attention from retrieving relevant memories to checking whether they establish the required identities and relations.
  • The paper treats missing evidence as a signal for further information gathering. By specifying which observation or relation is unresolved, it gives the planner a concrete basis for choosing the next tool.
Critical weaknesses
  • Although each loss has a stated purpose, the paper does not test whether all five terms are necessary or explain the choice of auxiliary weights. Loss ablations and sensitivity analysis would clarify whether a simpler objective can achieve comparable results.

  • The full system performs best, but this alone does not show that autonomous scheduling makes CEM more effective. CEM improves success by six percentage points under fixed scheduling and eight under autonomous scheduling. Without uncertainty analysis, it is unclear whether this extra two-point benefit reflects a reliable interaction or experimental variation.

Questions for authors

Which CEM components are essential to its gains? Can a simpler variant match its performance without bottleneck aggregation, structured witness search, or some auxiliary losses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.