acceptodds
Withdrawn from ICLR 2027

Resource-Aware Reasoning Orchestration for Decision-Making in Embodied Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied agents built on multimodal large language models (MLLMs) increasingly rely on explicit reasoning, such as replanning and verification, to recover from execution failures and resolve ambiguity. However, invoking such reasoning uniformly is wasteful: many decisions in a household task are unambiguous and do not benefit from additional deliberation, while others critically require it. Existing systems address this trade-off with fixed strategies, always reasoning, never reasoning, or reasoning on a static schedule, none of which adapt to the evolving execution state. We introduce RARRL (Resource-Aware Reasoning via Reinforcement Learning), an orchestration framework that learns when to invoke reasoning rather than prescribing it a priori. A lightweight policy, trained once with reinforcement learning in a Abstract Decision Environment (ADE), observes a compact state summarizing task progress, execution reliability, and remaining budget, and decides at each timestep whether to act directly or invoke a Planner-Verifier reasoning procedure before acting. Deploying this policy on a new simulated platform requires an environment-specific grounding adapter that exposes this shared state; the policy transfers zero-shot without retraining. We evaluate RARRL on EB-ALFRED and EB-Habitat, two embodied benchmarks built on distinct simulators with non-overlapping action spaces, and find that it consistently outperforms never-reasoning, always-reasoning, and fixed-schedule reasoning baselines across six task categories, while invoking reasoning substantially less often than always-reasoning agents.We further evaluate RARRL on VirtualHome and ALFWorld, achieving the best task success rate. We further perform matched-state counterfactual interventions in the target environments, showing that the value of invoking reasoning varies across execution-time decision states.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Resource-Aware, RL, MLLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9EygOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks when an MLLM-based embodied agent should spend extra reasoning. The proposed method, RARRL, is a small MLP policy trained with PPO in a synthetic Abstract Decision Environment (ADE). At each step the policy sees a 6-dimensional abstract state (progress, stagnation, remaining budget, a failure-based uncertainty estimate, and evidence and reliability from the planner/verifier) and picks ACT or THINK. Under ACT, a VLM executor acts directly. Under THINK, a planner proposes a multi-step plan, a verifier checks its first action, and the rest of an approved plan is stored in a persistent queue. THINK is capped at three calls per episode. Training needs no simulator and no LLM calls. For deployment, an environment-specific grounding map ff computes the abstract state and the policy is used frozen. On EB-ALFRED and EB-Habitat with Qwen2.5-VL-72B, RARRL reaches higher task success than never / always / fixed-interval / heuristic triggers implemented in the same pipeline, and calls THINK less often than always-think. The appendix adds ALFWorld, VirtualHome, a Gemma-3-27B backend, ablations, verifier corruption, an ADE-mismatch test, and matched-state counterfactuals.

Critical strengths
  1. The problem formulation is useful. Separating when to reason from how to reason, and treating the first as a budgeted sequential decision, is a clean way to think about agents in which every reasoning call costs real time and money. Because the scheduler is trained in a cheap abstract environment, it can in principle be attached to any reasoning backend. I expect this framing to be what others take from the paper, independent of the specific numbers.

  2. The authors are careful about what their theory does and does not show. They state that Q*-irrelevance is a conditional design rationale and not a transfer guarantee (App. E.1), that the bisimulation bound only holds within a single MDP (App. E.3), and that the analyses in App. K and O are observational. This makes the claims checkable, and it lets the evaluation focus on the empirical questions below.

Critical weaknesses

1. The efficiency claim is not established.

Doing better with fewer expensive THINK calls is the paper's central claim (title, Fig. 2, Sec. 5), but the evidence doesn't support it.

  • None of the baselines is an efficiency method. EmbodiedBench, ECoT, Plan-and-Act and KnowNo are re-implemented as Never/Always/Fixed/Heuristic trigger rules inside the authors' own pipeline (Sec. 4), so this only shows that RARRL calls THINK less often than Always-Think. Methods built to cut LLM reasoning cost (When2Ask, CogRouter, SwiftSage, System-1.x, FrugalGPT, AutoMix, Thinkless, AdaptThink; DeeR-VLA and HiRT on the robotics side) are not compared, and only SwiftSage is cited (R1, Q5).
  • Cost is not measured end to end. Never-Think sits at zero cost in Fig. 2, although it also runs the 72B model to pick actions (Q3).
  • Even inside the pipeline the advantage is unclear: RARRL uses slightly more calls ( ~4.7 vs. 4.5) and tokens ( ~7.1k vs. 6.3k) than Heuristic-Think, and every method is compared at a single cost point.
  • The per-category accuracy gains are mostly within sampling noise for 50 episodes (Q1).

Without the comparisons in R1, the claim should be narrowed to trigger efficiency within a fixed large-model pipeline.

2. The transfer is not zero-shot, and the 6-D state rules out many tasks.

  • The policy weights are frozen, but the policy only sees what the grounding map ff produces, and ff is hand-built for every environment: someone has to decide how progress, stagnation, failures and plan evidence are computed from each platform's feedback. The authors admit this ("requires code specific to that environment's API", App. F.4) but still call it "zero-shot orchestration transfer" (contribution 3). The adaptation cost is moved into manual feature design, not removed, and calling ff a "lightweight" adapter is unsupported since neither the procedure nor the effort is reported. All four environments got their own ff from the authors, so generalization to an environment nobody engineered is not shown. The only robustness test perturbs one feature in one category (Table 3; R2).
  • Even with a good ff, many tasks don't fit this state: irreversible actions (reasoning is needed before the first failure, but hth_t and utu_t only rise after it), search without milestones (ptp_t stays at 0), and the same symptoms with different causes. Grouping by zz removes only 15.7–20.8% of outcome variance (Table 6).

The "zero-shot" label should be dropped, the claim restated as transfer given a hand-designed ff, and the scope limited to tasks with milestones and observable per-action outcomes.

Questions for authors

A. Clarifications

Q1. Statistics

In Table 1, the per-category gaps between RARRL and the strongest baseline range from 1.7 to 13.8 points, and 10 of the 12 are below 8 (e.g., EB-ALFRED Common 54.5 vs. 52.8, Base 62.8 vs. 60.4). With 50 episodes per category, a 2.4-point gap is about 1.2 episodes, yet the reported confidence intervals reflect only variation across PPO seeds (App. A). Supporting the claim of the highest TSR "across all six capability categories" (Sec. 4) requires episode-level paired tests. Report paired bootstrap or exact McNemar results for Tables 1 and 2, the per-episode outcomes, and the comparison pooled over all 300 episodes. Also, why do baselines without any PPO training have intervals at temperature 0? For example, EmbodiedBench (Never-Think) is reported as 50.1 ± 2.1, although a success rate on a fixed set of 50 episodes can only move in steps of 2%.

Q2. Numbers that do not add up (a consistency check)

  • (a) Table 17: with 50 paired episodes, a net difference of 5 episodes (62% to 52%) cannot give an exact two-sided McNemar p-value below 0.0625, yet 0.044 is reported, and the text gives RARRL as 62.8%. What are the discordant counts?
  • (b) Table 1: KnowNo's PS values are identical on EB-ALFRED and EB-Habitat in five of the six categories (16.32, 15.52, 17.04, 18.16, 24.28).
  • (c) Episode 49: in Fig. 3b, one THINK approves "navigate to the sofa" and the episode succeeds. In Fig. 4, the same plan is rejected, the sofa is not visited after three THINKs, and p=0.0p = 0.0 at all seven decision points. Which one is the actual record?

Q3. Cost accounting and baselines

  • (a) Are the calls and tokens of the executor that selects actions under ACT included in the costs in Fig. 2? Never-Think is shown at zero, which suggests they are not.
  • (b) When exactly does Always-Think call THINK, and is bmax=3b_{max} = 3 applied to the baselines as well? One THINK is two LLM calls (App. B), so the roughly 8.5 calls in Fig. 2 are inconsistent with bmax=3b_{max} = 3 (at most 6 calls). And since ECoT (Always-Think) takes 15.7 to 26.3 steps on average on EB-ALFRED (Table 1), calling THINK before every action would require more than 30 calls.
  • (c) Why does Always-Think score below Never-Think in several settings (ALFWorld 29.0 vs. 38.9, Table 12; 31.83 vs. 35.67, Table 23)?

Q4. What the policy learns

Sec. 3.2 motivates the ADE by saying that the value of a THINK only shows up tens of primitive steps later, which makes credit assignment hard in the real simulator. But looking ahead to delayed effects is exactly what an RL policy is supposed to do. With only three stages per ADE episode (Table 5), while deployment episodes average 7 to 23 steps (Table 1, with a 30-step cap), has the policy solved this problem or sidestepped it, and how does it differ from a well-tuned heuristic trigger? The logs point the same way: when budget remains, the policy picks THINK 70% of the time (App. K), and its decisions are driven mainly by btb_t and hth_t (App. O). Report the effects of eplane_{plan} and κplan\kappa_{plan}, on which the motivation for the ADE relies. Finally, since changing the ADE's composition rule barely moves target performance (App. Q, 56.1 vs. 54.2), what evidence is there that the ADE's structure is what actually transfers?

Q5. Related work

When2Ask (Hu et al., RLC 2024) already learns when to query an LLM, using PPO with a penalty on unnecessary queries, and evaluates in MiniGrid and Habitat. The formulation in contribution 1 is therefore not new. What does look new is training the scheduler without any LLM calls in an abstract environment and reusing it, frozen, across environments. The contribution must be restated along those lines, and the paper must explain how RARRL differs from When2Ask, CogRouter, System-1.x, Thinkless/AdaptThink and DeeR-VLA. Also, When2Ask trains its asking policy in the target environments with real LLM queries, so the claim in Sec. 3.2 that such training is intractable needs evidence. The When2Ask-style comparison in R1 tests it directly.

Q6. Abstract state and grounding

What does it take to build ff for a new environment, and how long did it take for the environments in the paper? For tasks without intermediate milestones, or with irreversible actions, how would you define pp, hh and uu? Second, how often do states with the same zz differ in how much THINK helps? Group the 150 matched states from Sec. 5 by zz and show whether the counterfactual benefit of THINK is consistent within groups. This tests the Q*-irrelevance assumption directly with existing data.

B. Requested experiments

R1. Comparison with efficiency-oriented methods (most important)

Judging the efficiency claim requires a comparison with existing efficiency-oriented methods on a common metric. If this cannot be completed within the discussion period, the efficiency claim in this version must be narrowed to trigger efficiency within a fixed large-model pipeline.

  1. End-to-end cost. For every method, report total tokens including the executor, the number of model calls, and wall-clock time (or GPU-seconds). For existing runs this only means re-counting the logs. For RARRL, also report the effort needed to build ff for each environment (Q6).
  2. Comparison with each family of efficiency methods in the same pipeline, on EB-ALFRED and EB-Habitat:
    • SwiftSage-style fast/slow: a small VLM by default, the 72B model only when a switching rule fires.
    • Confidence-based escalation (FrugalGPT/AutoMix-style): THINK only when the confidence of the cheap output is low.
    • Model-decided thinking (Thinkless/AdaptThink-style).
    • When2Ask-style: learn when to query with RL in the real environment, with real LLM calls. This also directly tests the claim in Sec. 3.2 that training in the real environment is impractical (Q5).
  3. ALFWorld, which the paper already uses. Report RARRL's success and end-to-end tokens on the standard test split and compare with CogRouter: with the same backbone if code is available, and otherwise side by side, stating the backbone and whether each method is trained in the target environment.
  4. Matched-cost curves. Vary RARRL's budget (bmaxb_{max}) and each baseline's threshold, and report success–cost curves. This is required at least on EB-ALFRED.

R2. Grounding sensitivity and scope

Apply the same perturbation as in Table 3 to hh, uu, eplane_{plan} and κplan\kappa_{plan}, not only pp, and state explicitly that the claims are limited to tasks with milestone structure and observable per-action outcomes.

If R1 and R2 are addressed, with RARRL matching or beating these methods in success at a clearly lower end-to-end cost and the results holding under the R2 perturbations, I will raise my score.

Reviewer_pQjwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework that learns when embodied agents should invoke expensive reasoning. It trains a lightweight policy using reinforcement learning in an abstract environment. The policy then transfers to new embodied environments without retraining, although each environment requires a state-grounding adapter. The system reuses verified plans to reduce unnecessary reasoning. Experiments on four embodied benchmarks show improved task success and lower reasoning costs compared with fixed and heuristic strategies.

Critical strengths

I think learning when to reason is a useful direction for embodied agents. Training the orchestration policy in a lightweight abstract environment is particularly appealing, as it avoids expensive simulator interactions during training.

The experimental evaluation is fairly extensive. The results are consistent across several environments, and the controlled baselines help isolate the benefit of adaptive orchestration. I also appreciate the matched-state counterfactual analysis, which provides more direct evidence that reasoning decisions matter.

Critical weaknesses

I noticed several inconsistencies in the reported experimental statistics. For example, Table 18 reports 1,162 THINK decisions, which is difficult to reconcile with the three-call budget and the 212 THINK events reported in Table 15. Table 17 also reports an exact McNemar p-value of 0.044 for a five-episode success difference, which appears inconsistent with the stated statistical test. These issues raise concerns about the reliability of the analysis.

I am not fully convinced that the abstract training environment accurately captures the deployed reasoning process. The ADE models THINK as acquiring evidence about an individual decision facet, while the actual implementation invokes both the Planner and Verifier. This difference could affect the learned allocation strategy and deserves more explanation.

Questions for authors

How does the ADE's evidence-acquisition model correspond to the fixed Planner-Verifier pipeline used during deployment?

How sensitive is zero-shot transfer to the environment-specific grounding adapter? Could the authors provide more evidence about the effort required to construct it for a new environment?

Reviewer_95LHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RARRL formulates when embodied agents should invoke MLLM reasoning as a budget-constrained sequential decision problem, trains a lightweight THINK/ACT orchestration policy in an Abstract Decision Environment, and transfers it frozen to multiple embodied benchmarks, improving task success while reducing reasoning cost.

Critical strengths
  1. The problem is cleanly modeled as resource allocation for “when to reason” rather than a fixed heuristic, giving the work clear methodological value.
  2. Training once in the ADE and freezing the policy for transfer to multiple structurally different embodied environments provides convincing evidence for zero-shot orchestration transfer.
  3. The experiments are thorough, including main results, ablations, counterfactual interventions, robustness tests, and efficiency analysis, supporting the accuracy-efficiency trade-off claim.
Critical weaknesses
  1. The abstract state mapping f still requires environment-specific engineering, weakening the generality of “zero-shot” at the deployment-engineering level.
  2. Evaluation is limited to simulated benchmarks without real-robot experiments, and the Q* irrelevance condition underlying transfer is only a design rationale rather than a rigorous proof.
  3. The budget counts THINK invocations rather than exact token or latency cost, and the ADE evidence generator is only a surrogate for real reasoning, which may introduce sim-to-real bias.
Questions for authors
  1. How much environment-specific engineering does the grounding map f require for a new platform, and how sensitive is RARRL’s zero-shot transfer to errors in f?
  2. Can the paper provide empirical evidence, beyond design rationale, that the Q*-irrelevance condition approximately holds across EB-ALFRED, EB-Habitat, ALFWorld, and VirtualHome?
  3. Since the budget counts THINK invocations rather than exact token or latency cost, how well does this proxy align with real computational cost, and would the learned policy change under token- or latency-based budgeting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.