- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks when an MLLM-based embodied agent should spend extra reasoning. The proposed method, RARRL, is a small MLP policy trained with PPO in a synthetic Abstract Decision Environment (ADE). At each step the policy sees a 6-dimensional abstract state (progress, stagnation, remaining budget, a failure-based uncertainty estimate, and evidence and reliability from the planner/verifier) and picks ACT or THINK. Under ACT, a VLM executor acts directly. Under THINK, a planner proposes a multi-step plan, a verifier checks its first action, and the rest of an approved plan is stored in a persistent queue. THINK is capped at three calls per episode. Training needs no simulator and no LLM calls. For deployment, an environment-specific grounding map computes the abstract state and the policy is used frozen. On EB-ALFRED and EB-Habitat with Qwen2.5-VL-72B, RARRL reaches higher task success than never / always / fixed-interval / heuristic triggers implemented in the same pipeline, and calls THINK less often than always-think. The appendix adds ALFWorld, VirtualHome, a Gemma-3-27B backend, ablations, verifier corruption, an ADE-mismatch test, and matched-state counterfactuals.
Critical strengths
-
The problem formulation is useful. Separating when to reason from how to reason, and treating the first as a budgeted sequential decision, is a clean way to think about agents in which every reasoning call costs real time and money. Because the scheduler is trained in a cheap abstract environment, it can in principle be attached to any reasoning backend. I expect this framing to be what others take from the paper, independent of the specific numbers.
-
The authors are careful about what their theory does and does not show. They state that Q*-irrelevance is a conditional design rationale and not a transfer guarantee (App. E.1), that the bisimulation bound only holds within a single MDP (App. E.3), and that the analyses in App. K and O are observational. This makes the claims checkable, and it lets the evaluation focus on the empirical questions below.
Critical weaknesses
1. The efficiency claim is not established.
Doing better with fewer expensive THINK calls is the paper's central claim (title, Fig. 2, Sec. 5), but the evidence doesn't support it.
- None of the baselines is an efficiency method. EmbodiedBench, ECoT, Plan-and-Act and KnowNo are re-implemented as Never/Always/Fixed/Heuristic trigger rules inside the authors' own pipeline (Sec. 4), so this only shows that RARRL calls THINK less often than Always-Think. Methods built to cut LLM reasoning cost (When2Ask, CogRouter, SwiftSage, System-1.x, FrugalGPT, AutoMix, Thinkless, AdaptThink; DeeR-VLA and HiRT on the robotics side) are not compared, and only SwiftSage is cited (R1, Q5).
- Cost is not measured end to end. Never-Think sits at zero cost in Fig. 2, although it also runs the 72B model to pick actions (Q3).
- Even inside the pipeline the advantage is unclear: RARRL uses slightly more calls ( ~4.7 vs. 4.5) and tokens ( ~7.1k vs. 6.3k) than Heuristic-Think, and every method is compared at a single cost point.
- The per-category accuracy gains are mostly within sampling noise for 50 episodes (Q1).
Without the comparisons in R1, the claim should be narrowed to trigger efficiency within a fixed large-model pipeline.
2. The transfer is not zero-shot, and the 6-D state rules out many tasks.
- The policy weights are frozen, but the policy only sees what the grounding map produces, and is hand-built for every environment: someone has to decide how progress, stagnation, failures and plan evidence are computed from each platform's feedback. The authors admit this ("requires code specific to that environment's API", App. F.4) but still call it "zero-shot orchestration transfer" (contribution 3). The adaptation cost is moved into manual feature design, not removed, and calling a "lightweight" adapter is unsupported since neither the procedure nor the effort is reported. All four environments got their own from the authors, so generalization to an environment nobody engineered is not shown. The only robustness test perturbs one feature in one category (Table 3; R2).
- Even with a good , many tasks don't fit this state: irreversible actions (reasoning is needed before the first failure, but and only rise after it), search without milestones ( stays at 0), and the same symptoms with different causes. Grouping by removes only 15.7–20.8% of outcome variance (Table 6).
The "zero-shot" label should be dropped, the claim restated as transfer given a hand-designed , and the scope limited to tasks with milestones and observable per-action outcomes.
Questions for authors
A. Clarifications
Q1. Statistics
In Table 1, the per-category gaps between RARRL and the strongest baseline range from 1.7 to 13.8 points, and 10 of the 12 are below 8 (e.g., EB-ALFRED Common 54.5 vs. 52.8, Base 62.8 vs. 60.4). With 50 episodes per category, a 2.4-point gap is about 1.2 episodes, yet the reported confidence intervals reflect only variation across PPO seeds (App. A). Supporting the claim of the highest TSR "across all six capability categories" (Sec. 4) requires episode-level paired tests. Report paired bootstrap or exact McNemar results for Tables 1 and 2, the per-episode outcomes, and the comparison pooled over all 300 episodes. Also, why do baselines without any PPO training have intervals at temperature 0? For example, EmbodiedBench (Never-Think) is reported as 50.1 ± 2.1, although a success rate on a fixed set of 50 episodes can only move in steps of 2%.
Q2. Numbers that do not add up (a consistency check)
- (a) Table 17: with 50 paired episodes, a net difference of 5 episodes (62% to 52%) cannot give an exact two-sided McNemar p-value below 0.0625, yet 0.044 is reported, and the text gives RARRL as 62.8%. What are the discordant counts?
- (b) Table 1: KnowNo's PS values are identical on EB-ALFRED and EB-Habitat in five of the six categories (16.32, 15.52, 17.04, 18.16, 24.28).
- (c) Episode 49: in Fig. 3b, one THINK approves "navigate to the sofa" and the episode succeeds. In Fig. 4, the same plan is rejected, the sofa is not visited after three THINKs, and at all seven decision points. Which one is the actual record?
Q3. Cost accounting and baselines
- (a) Are the calls and tokens of the executor that selects actions under ACT included in the costs in Fig. 2? Never-Think is shown at zero, which suggests they are not.
- (b) When exactly does Always-Think call THINK, and is applied to the baselines as well? One THINK is two LLM calls (App. B), so the roughly 8.5 calls in Fig. 2 are inconsistent with (at most 6 calls). And since ECoT (Always-Think) takes 15.7 to 26.3 steps on average on EB-ALFRED (Table 1), calling THINK before every action would require more than 30 calls.
- (c) Why does Always-Think score below Never-Think in several settings (ALFWorld 29.0 vs. 38.9, Table 12; 31.83 vs. 35.67, Table 23)?
Q4. What the policy learns
Sec. 3.2 motivates the ADE by saying that the value of a THINK only shows up tens of primitive steps later, which makes credit assignment hard in the real simulator. But looking ahead to delayed effects is exactly what an RL policy is supposed to do. With only three stages per ADE episode (Table 5), while deployment episodes average 7 to 23 steps (Table 1, with a 30-step cap), has the policy solved this problem or sidestepped it, and how does it differ from a well-tuned heuristic trigger? The logs point the same way: when budget remains, the policy picks THINK 70% of the time (App. K), and its decisions are driven mainly by and (App. O). Report the effects of and , on which the motivation for the ADE relies. Finally, since changing the ADE's composition rule barely moves target performance (App. Q, 56.1 vs. 54.2), what evidence is there that the ADE's structure is what actually transfers?
Q5. Related work
When2Ask (Hu et al., RLC 2024) already learns when to query an LLM, using PPO with a penalty on unnecessary queries, and evaluates in MiniGrid and Habitat. The formulation in contribution 1 is therefore not new. What does look new is training the scheduler without any LLM calls in an abstract environment and reusing it, frozen, across environments. The contribution must be restated along those lines, and the paper must explain how RARRL differs from When2Ask, CogRouter, System-1.x, Thinkless/AdaptThink and DeeR-VLA. Also, When2Ask trains its asking policy in the target environments with real LLM queries, so the claim in Sec. 3.2 that such training is intractable needs evidence. The When2Ask-style comparison in R1 tests it directly.
Q6. Abstract state and grounding
What does it take to build for a new environment, and how long did it take for the environments in the paper? For tasks without intermediate milestones, or with irreversible actions, how would you define , and ? Second, how often do states with the same differ in how much THINK helps? Group the 150 matched states from Sec. 5 by and show whether the counterfactual benefit of THINK is consistent within groups. This tests the Q*-irrelevance assumption directly with existing data.
B. Requested experiments
R1. Comparison with efficiency-oriented methods (most important)
Judging the efficiency claim requires a comparison with existing efficiency-oriented methods on a common metric. If this cannot be completed within the discussion period, the efficiency claim in this version must be narrowed to trigger efficiency within a fixed large-model pipeline.
- End-to-end cost. For every method, report total tokens including the executor, the number of model calls, and wall-clock time (or GPU-seconds). For existing runs this only means re-counting the logs. For RARRL, also report the effort needed to build for each environment (Q6).
- Comparison with each family of efficiency methods in the same pipeline, on EB-ALFRED and EB-Habitat:
- SwiftSage-style fast/slow: a small VLM by default, the 72B model only when a switching rule fires.
- Confidence-based escalation (FrugalGPT/AutoMix-style): THINK only when the confidence of the cheap output is low.
- Model-decided thinking (Thinkless/AdaptThink-style).
- When2Ask-style: learn when to query with RL in the real environment, with real LLM calls. This also directly tests the claim in Sec. 3.2 that training in the real environment is impractical (Q5).
- ALFWorld, which the paper already uses. Report RARRL's success and end-to-end tokens on the standard test split and compare with CogRouter: with the same backbone if code is available, and otherwise side by side, stating the backbone and whether each method is trained in the target environment.
- Matched-cost curves. Vary RARRL's budget () and each baseline's threshold, and report success–cost curves. This is required at least on EB-ALFRED.
R2. Grounding sensitivity and scope
Apply the same perturbation as in Table 3 to , , and , not only , and state explicitly that the claims are limited to tasks with milestone structure and observable per-action outcomes.
If R1 and R2 are addressed, with RARRL matching or beating these methods in success at a clearly lower end-to-end cost and the results holding under the R2 perturbations, I will raise my score.