- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces TARSphere, a framework for evaluating long-horizon embodied agents in evolving physical and social environments. TARSphere-Infra combines simulated robot dynamics, embodied NPCs with constrained dialogue, and an event-driven task state machine. Task revisions become active upon actual message delivery, and evaluation checks physical and interaction evidence against the applicable task requirements. TARSphere-Harness connects frozen multimodal language models to executable skills, hierarchical memory, and execution feedback. The accompanying benchmark contains 360 episodes derived from 60 Base tasks across 30 indoor scenes. Navigate-and-report and fetch-and-deliver tasks are paired with five conditions targeting perceptual robustness, disruption adaptation, task updates, human coordination, and intent understanding. Experiments compare eight models and include harness ablations, assisted-execution interventions, skill calibration, curation ablations, and human evaluation. Results indicate substantial difficulties with task updates and human interaction, alongside frequent collisions and falls.
Critical strengths
Careful integration of physical execution, communication, and changing task requirements. The framework distinguishes tool invocation from physical success and queued speech from delivered information. Its versioned task bindings prevent later updates from retroactively authorizing earlier actions. These details are consequential: without them, an evaluation could reward unsupported completion claims or penalize an agent for information it had not received. This provides a credible foundation for studying adaptation during embodied execution.
Evaluation extends beyond a leaderboard to test mechanisms and benchmark validity. Paired variants, harness ablations, and controlled execution assistance provide complementary views of failure. In particular, Table 4 shows that the tested assistance improves completion without eliminating task-update and coordination deficits. Skill calibration and human scoring comparisons further strengthen the empirical foundation. Together, these analyses make the benchmark more informative than success rates alone.
Critical weaknesses
Small paired samples and absent uncertainty estimates limit several mechanistic conclusions. The 360 episodes represent 60 Base tasks with correlated variants. Harness ablations use only 12 Base bundles, giving 12 episodes per condition, and the intervention study uses 30 delivery bundles. Appendix D.3 explicitly describes the results as descriptive point estimates. Consequently, condition-specific gains and small differences between interventions may reflect episode variability. For example, the full-assistance change in the disruption-adaptation success gap is 3.3 percentage points—one episode at this sample size. This does not undermine the infrastructure contribution, but it limits confidence in claims about component importance, configuration trade-offs, and differences between capability gaps.
Model rankings are conditional on a substantial shared harness and differing inference configurations. The common interface is useful for controlled comparison, but it also tests compatibility with a particular prompting, memory, tool-use, and termination protocol. Table 15 gives proprietary models a 16,384-token output ceiling and local models a 2,048-token ceiling, with additional model-specific request adaptations. The failure analysis also identifies tool-interface errors and premature completion as important failure modes. These findings make it difficult to attribute the general-purpose-versus-embodied performance gap specifically to embodied reasoning ability. Moreover, harness ablations and execution interventions are conducted only with Gemini-3.8-Flash, leaving their generality across models uncertain.
Questions for authors
I am worried about the actual performance, due to lack of video and code.
Scoring validation: Table 7 reports correlations for continuous semantic scores. What is human–judge agreement on the binary final-report correctness verdict that determines navigation Success? This distinction matters because high score correlation does not necessarily establish reliable pass/fail decisions.
Curation clarification: Table 6 reports 96.7% usable episodes for the full curation pipeline, while the benchmark contains 360 episodes and all reviewed HC/IU tasks pass causal and feasibility checks. Were the unusable outputs repaired or replaced before evaluation, and what distinguishes the usability audit from the final feasibility audit?
7 comments
Response to Reviewer iDQR (part 2/3)
[Part 2/3]
(Continued from Part 1/3.)
W2: Shared harness and inference configurations
-
Identical configuration: All models share the harness interface, prompts, strict validator, and 100-decision budget. The revised Table 16 reports the endpoint settings used in the experiments, replacing the initial defaults listed in the submitted Table 15. Under the identical local configuration (same vLLM stack, 2,048 ceiling, JSON-object response mode), Gemma-4 and Qwen3.8 still lead both embodied models in Progress.
-
Unified JSON interface: All models use the same strict JSON decision format. In the recorded outputs, the longest decision from any general-purpose model contains 692 tokens, below the 2,048-token ceiling.
{"thinking": "...", "actions": [{"tool": "...", "args": {...}}],
"plan": {...}, "memory": [...], "inspection": "..."}
| Model | Mean | Median | P95 | Max |
|---|---|---|---|---|
| GPT-5.6-Terra | 126 | 124 | 154 | 561 |
| Claude-Sonnet-5 | 170 | 156 | 287 | 524 |
| Gemini-3.8-Flash | 114 | 113 | 157 | 299 |
| Qwen3.7-Plus | 150 | 145 | 224 | 465 |
| Qwen3.8-27B | 189 | 185 | 271 | 692 |
| Gemma-4-31B | 214 | 212 | 278 | 449 |
| RoboBrain2.0-32B | 452 | 241 | 2,048 | 2,048 |
| Hy-Embodied-VLM-1.0 | 302 | 271 | 399 | 2,048 |
-
Outputs at the ceiling: The observed 2,048-token outputs from RoboBrain and Hy-Embodied consist of repeated text with unclosed JSON (zlib ratio 0.023–0.029 vs. ~0.49).
-
Refined claim: The leading proprietary models (Gemini, Claude, GPT) lead the open models, and general-purpose Qwen3.8 and Gemma-4 also lead the embodied models in Progress. These results characterize the tested models on long-horizon, human-involved tasks under the shared harness.
-
Tool-interface errors: On pairs with no truncation and ≥95% parse validity, Gemma leads RoboBrain and Hy-Embodied by +16.1 and +25.4 Progress, and controlling for parse-failure and truncation rates leaves the gaps unchanged (−17.6 → −17.6; −28.6 → −29.1). Format errors and premature completion are observable failures under the shared protocol.
-
Gemini for ablations: We use Gemini as the reference agent because it succeeds on 33 of 72 subset episodes, leaving room to measure the effects of component removal. Five models score at or below 8.1% on the main benchmark, where low success can make such effects harder to resolve. The ablations characterize the harness with this reference agent, while model comparisons rest on the full eight-model evaluation.
-
Revision: We have refined this claim in the abstract, introduction, and Sec. 3.2, corrected the configuration table (now Table 16), and stated the reference-agent scope of the ablations in Sec. 3.3.1, refining their wording to be descriptive.
(Continued in Part 3/3.)
Response to Reviewer iDQR (part 3/3)
[Part 3/3]
(Continued from Part 2/3.)
Q1: Code and videos
The updated supplementary provides anonymized source code and rollout demonstrations. We commit to fully open-sourcing the codebase and benchmark upon publication.
Q2: Scoring validation
Reliable pass/fail decisions are what navigation Success depends on.
- Binary decisions in Success: Only the last of them is judged by a model; the others are rule checks on simulation evidence with no judge:
| Binary decision | Decided by | Basis |
|---|---|---|
| Target observed | Fixed threshold | Supplied RGB-D capture: at least 256 depth-supported pixels, 6×6-pixel box, 0.04 m volume tolerance |
| Recipient observed | Fixed threshold | Named recipient in the current evidence, fixed RGB-D person-visibility rule (Table 9) |
| Information receipt (inquiry) | Event state machine | Authorized answer delivered by the source |
| Addressed delivery | Event state machine | Recipient within 2.5 m and visible |
| Terminal fall or object loss | Simulation state | Physical state of the robot and objects |
| Final-report correctness | Report judge | Authored answer vs. delivered report |
-
Human pass/fail audit: One human annotator independently labelled 180 delivered navigation reports from 6 models against the authored answers using the same rubric. The human labels agree with the judge's pass/fail on 175 of 180 reports (97.2%; Cohen's κ = 0.87). The judge sees only the authored answer and the delivered report, blind to policy identity, plans, later actions, and final success.
-
Revision: We have added this audit to Sec. 3.4 and Appendix D.5.
Q3: Construction clarification
-
96.7% is the yield after the extra human review: It flagged 12 residual candidates; all 12 were repaired (none replaced), reviewed again, and entered the final 360.
-
Usability audit vs. feasibility audit: Table 6's audit checks every candidate for task coherence and completion of the required interactions and physical goals, and measures the yield of the pipeline; Sec. 3.4 separately reviews the released HC and IU tasks for causal soundness and for feasibility, i.e., whether the scene, available information, interaction conditions, and robot skills admit a task-completing sequence.
-
Revision: We have described the batch protocol and release step in Sec. 3.3.5, Appendix C.4, and the Table 6 caption.
Response to Reviewer iDQR (part 1/3)
[Part 1/3]
Thank you for the detailed review and for recognizing our infrastructure contribution. We clarify our contributions and address each point below.
TARSphere's core contribution
W1: Paired sample sizes and uncertainty
Table 22 reports sensitivity ranges that additionally use paired Gemini execution differences as a shared variability assumption (Appendix E.3):
Controlled local comparisons (Table 18, 95% scene-cluster bootstrap CIs):
(Continued in Part 2/3.)