- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces EmbodiedScope, a diagnostic benchmark built on BEHAVIOR with 240 closed-loop household tasks under a humanoid setting. Agents receive egocentric visual observations and interact through a shared intermediate interface that abstracts fine-grained motor control while preserving task-level decisions. The benchmark targets four capability dimensions, including Active Information Acquisition, Memory, Physical Reasoning, and Verification and Recovery, through controlled variants that manipulate task-relevant evidence, interaction history, physical conditions, and execution outcomes. Tasks are organized into basic-interaction, capability-focused, and structured multi-capability levels. Each constructed task undergoes a two-stage validation procedure consisting of privileged-oracle execution and manual quality review, while task success during evaluation is determined by a canonical verifier. The authors evaluate representative MLLMs and agent-system configurations, including paired base and harnessed configurations for selected backbones. The results show uneven capability profiles, substantial performance degradation on structured multi-capability tasks, and selective gains from agent-system support, particularly for memory-related demands.
Critical strengths
1、The paper addresses a diagnostic limitation in existing embodied-agent evaluation. The authors identify that aggregate task success alone provides limited insight into which specific agentic capabilities account for success or failure. To address this limitation, EmbodiedScope organizes tasks into basic-interaction, capability-focused, and structured multi-capability levels, while explicitly targeting Active Information Acquisition, Memory, Physical Reasoning, and Verification and Recovery. This design supports a more diagnostic analysis of capability-specific behavior and allows the evaluation to examine whether capabilities that appear reliable in focused settings remain effective when multiple demands interact during execution. This provides information that cannot be captured by overall task success alone. 2、The paper provides a well-motivated interaction boundary between high-level reasoning and low-level motor control. The authors highlight that an overly low-level action interface can make performance depend heavily on motion planning and controller quality, whereas an overly high-level interface may perform search, grounding, or planning on behalf of the agent. To balance these two extremes, EmbodiedScope introduces a shared intermediate interaction boundary based on egocentric RGB observations, temporary visual references, local action primitives, execution receipts, and a canonical verifier, while withholding privileged simulator state. This design abstracts fine-grained motor execution while preserving the task-level spatial and physical decisions that the benchmark intends to measure, thereby making the evaluation target clearer across different agent-system configurations.
Critical weaknesses
1、The paper interprets the performance drop from capability-focused L1 tasks to structured L2 tasks as evidence that current agents struggle when multiple capability demands must persist and interact. However, L1 and L2 consist of different task sets: L1 contains 120 tasks, whereas L2 contains only 20 separately constructed tasks. The paper does not report matched task-family comparisons, analyses controlling for differences in task horizon, navigation burden, scene composition, object count, or overall difficulty, nor does it break down L2 performance by capability combination. This issue is central to my assessment because diagnosing capability interaction is a principal contribution of the benchmark. Without stronger construct validation, the observed L2 degradation cannot yet be cleanly attributed specifically to interacting capability demands. 2、The paper’s main quantitative conclusions are not supported by sufficient evidence of run-to-run stability. Table 2 reports aggregate success rates, but the paper does not state whether the full benchmark evaluation was repeated across independent runs, nor does it provide run-to-run uncertainty estimates or paired/task-level robustness analyses for the main comparisons. This is particularly consequential for L2, which contains only 20 tasks, so the outcome of a single task changes the reported success rate by five percentage points. Although Table 6 reports three independent trials for selected representative L1 instances, comparable repeated-run evidence is not provided for the main benchmark results. As a result, the stability of the model rankings, the L1-to-L2 degradation, and the reported harness effects remains unclear. Because these comparisons provide the primary empirical support for the paper’s conclusions, repeated evaluations demonstrating that the findings are stable would materially strengthen the submission.
Questions for authors
1、How do the authors validate that each controlled variant primarily changes the intended A/M/P/V demand without introducing substantial changes in the other capability demands? 2、How representative are the 20 L2 tasks of structured multi-capability execution? Which two- and three-capability combinations are covered, and does the L1-to-L2 degradation remain consistent across combinations and task families rather than being driven by sampling variability or a small subset of difficult tasks? 3、Are the main results in Table 2 based on single or repeated executions for each task-system pair? If repeated runs were conducted, are the model rankings, L1-to-L2 degradation, and Base-versus-harness differences stable across runs? 4、What evidence supports attributing the reported harness-related memory gains primarily to history or context management? Since the Base and native-harness configurations may also differ in system instructions, planning support, image representation, and interaction protocol, how are these effects disentangled? 5、What aspects of physical reasoning are actually evaluated by the L1-P tasks? Given that the default Action20 Symbolic profile disables physics stepping, which forms of physical interaction or dynamics are abstracted away? 6、What is the main empirical novelty of EmbodiedScope relative to ESI-Bench, HumanCLAW, SpatialWorld, LongAct, and CaP-X? Is the primary contribution the controlled construction of capability-specific variants and their integration into structured multi-capability tasks?