acceptodds
Under review as a conference paper at ICLR 2027

TARSphere: A Robot-Oriented Full-Stack Framework for Long-Horizon Embodied Agents in Evolving Physical and Social Environments

Abstract

Robots must act under physical and social constraints while adapting to changes in their environment and human needs. Existing embodied-agent systems often focus on specified goals or abstract execution into symbolic actions, limiting evaluation in an evolving world. We introduce TARSphere, a robot-oriented full-stack framework for physically and socially grounded long-horizon embodied agents. TARSphere-Infra runs robot policies in a physics simulator with embodied NPCs and constrained dialogue. An event-driven finite-state machine (FSM) uses completed actions and delivered information to update task states and evaluate outcomes. TARSphere-Harness, its embodied brain harness, lets a frozen multimodal large language model (MLLM) act through multi-level executable skills, using observations, hierarchical memory, and execution feedback to revise its decisions. On this foundation, agent-in-the-loop curation aligns task semantics, causal dependencies, and scene geometry to construct TARSphere-Bench: 360 episodes across 30 indoor scenes. Five paired conditions assess perceptual robustness, disruption adaptation, task update, human coordination, and intent understanding for navigate-and-report and fetch-and-deliver tasks. Our evaluation of 8 MLLMs shows that general-purpose models can outperform embodied models, while task updates and human interaction remain difficult. Frequent collisions and falls further motivate evaluating execution safety alongside task success.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Intelligence, Long-Horizon Evaluation, Agent Harness, Human–Robot Interaction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iDQROpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TARSphere, a framework for evaluating long-horizon embodied agents in evolving physical and social environments. TARSphere-Infra combines simulated robot dynamics, embodied NPCs with constrained dialogue, and an event-driven task state machine. Task revisions become active upon actual message delivery, and evaluation checks physical and interaction evidence against the applicable task requirements. TARSphere-Harness connects frozen multimodal language models to executable skills, hierarchical memory, and execution feedback. The accompanying benchmark contains 360 episodes derived from 60 Base tasks across 30 indoor scenes. Navigate-and-report and fetch-and-deliver tasks are paired with five conditions targeting perceptual robustness, disruption adaptation, task updates, human coordination, and intent understanding. Experiments compare eight models and include harness ablations, assisted-execution interventions, skill calibration, curation ablations, and human evaluation. Results indicate substantial difficulties with task updates and human interaction, alongside frequent collisions and falls.

Critical strengths

Careful integration of physical execution, communication, and changing task requirements. The framework distinguishes tool invocation from physical success and queued speech from delivered information. Its versioned task bindings prevent later updates from retroactively authorizing earlier actions. These details are consequential: without them, an evaluation could reward unsupported completion claims or penalize an agent for information it had not received. This provides a credible foundation for studying adaptation during embodied execution.

Evaluation extends beyond a leaderboard to test mechanisms and benchmark validity. Paired variants, harness ablations, and controlled execution assistance provide complementary views of failure. In particular, Table 4 shows that the tested assistance improves completion without eliminating task-update and coordination deficits. Skill calibration and human scoring comparisons further strengthen the empirical foundation. Together, these analyses make the benchmark more informative than success rates alone.

Critical weaknesses

Small paired samples and absent uncertainty estimates limit several mechanistic conclusions. The 360 episodes represent 60 Base tasks with correlated variants. Harness ablations use only 12 Base bundles, giving 12 episodes per condition, and the intervention study uses 30 delivery bundles. Appendix D.3 explicitly describes the results as descriptive point estimates. Consequently, condition-specific gains and small differences between interventions may reflect episode variability. For example, the full-assistance change in the disruption-adaptation success gap is 3.3 percentage points—one episode at this sample size. This does not undermine the infrastructure contribution, but it limits confidence in claims about component importance, configuration trade-offs, and differences between capability gaps.

Model rankings are conditional on a substantial shared harness and differing inference configurations. The common interface is useful for controlled comparison, but it also tests compatibility with a particular prompting, memory, tool-use, and termination protocol. Table 15 gives proprietary models a 16,384-token output ceiling and local models a 2,048-token ceiling, with additional model-specific request adaptations. The failure analysis also identifies tool-interface errors and premature completion as important failure modes. These findings make it difficult to attribute the general-purpose-versus-embodied performance gap specifically to embodied reasoning ability. Moreover, harness ablations and execution interventions are conducted only with Gemini-3.8-Flash, leaving their generality across models uncertain.

Questions for authors

I am worried about the actual performance, due to lack of video and code.

Scoring validation: Table 7 reports correlations for continuous semantic scores. What is human–judge agreement on the binary final-report correctness verdict that determines navigation Success? This distinction matters because high score correlation does not necessarily establish reliable pass/fail decisions.

Curation clarification: Table 6 reports 96.7% usable episodes for the full curation pipeline, while the benchmark contains 360 episodes and all reviewed HC/IU tasks pass causal and feasibility checks. Were the unusable outputs repaired or replaced before evaluation, and what distinguishes the usability audit from the final feasibility audit?

7 comments
AuthorsOpenReview ↗

Response to Reviewer iDQR (part 1/3)

[Part 1/3]

Thank you for the detailed review and for recognizing our infrastructure contribution. We clarify our contributions and address each point below.

TARSphere's core contribution

  • The framework: TARSphere is the first robot-oriented full-stack framework for physically and socially grounded embodied agents in evolving environments: a simulation platform with embodied NPCs, an event-driven evaluator, a modular harness, and an authoring interface for developing and evaluating embodied agents and human–robot interaction.
  • Community extensibility: Each layer is open to extension, so the community can build on it rather than start from scratch.
    • Infra: richer physical and social simulation, e.g., more human-like NPCs and new robot embodiments.
    • Harness: stronger components and tools, plugged in under the same evaluator and episodes.
    • Benchmark: new tasks, scenes, and NPC knowledge authored through the interface, so task diversity can keep growing.
  • Code, videos, and open-source release: We have updated the supplementary material with anonymized code and rollout videos. We commit to fully open-sourcing the codebase and benchmark upon publication.

W1: Paired sample sizes and uncertainty

  • Main conclusions (360 episodes from 60 Base tasks): Our scene bootstrap keeps related variants together. Gemini ranks first and the embodied models 7th and 8th in all 10,000 resamples. The navigation TU, HC, and IU success gaps are −15.9 [−22.5, −9.6], −9.3 [−15.4, −3.3], and −13.0 [−19.6, −7.1], while PR and DA stay within ±3 points of Base (pooled over models), so the deficits are specific to conditions that change task information.

Table 22 reports sensitivity ranges that additionally use paired Gemini execution differences as a shared variability assumption (Appendix E.3):

ComparisonΔProgress [range]ΔSuccess [range]
Gemini vs. Claude-Sonnet-5 (closest)+16.9 [11.0, 22.8]+23.2 [13.5, 33.0]
Gemini vs. GPT-5.6-Terra+20.8 [15.8, 25.8]+30.3 [20.9, 39.6]

Controlled local comparisons (Table 18, 95% scene-cluster bootstrap CIs):

ComparisonΔProgressΔSuccess
Gemma vs. RoboBrain+17.7 [14.5, 21.1]+7.8 [4.2, 11.8]
Gemma vs. Hy-Embodied+28.7 [25.2, 32.3]+7.8 [4.2, 11.7]
  • Small studies (Tables 3 and 4): The Table 4 conclusion rests on persistence: TU and HC stay 12.0–20.0 points below Base in all six settings. For Table 3, no tested setting exceeds the full harness in average success, and the largest effect, w/o grasp skill, is −25.0 [−44.4, −6.9]. The revised appendix lists per-setting results.
  • Cost and reproducibility: For a single model, one episode takes up to 57 minutes and costs up to $2.5 (Claude-Sonnet-5); all 360 episodes cost about $890 and take 1.1 days on one 8×RTX 4090 host. All our experiments together total 4,536 runs, with a computed cost of about $3,397 and 2,222 GPU-hours (open-source models deployed on one 8-GPU host, two GPUs each, for their evaluation time):
StudyConfigurationsRunsCostRTX 4090 GPU-hours
Main experiment(4 closed-source + 4 open-source models) × 360 episodes2,880$1,9521,378
Harness ablation13 × 72 episodes936$817477
Controlled intervention6 settings × 4 conditions × 30 bundles720$628367
Total4,536$3,3972,222
  • Why 72 episodes for ablations: We share the wish for larger samples and considered a larger scale, but weighed it against evaluation cost and the community's ability to reproduce, use, and extend the benchmark: all 12 settings on the 360 episodes, with a single model, take 3,854 lane-hours, 1.9× our entire eight-model main evaluation (2,073 lane-hours). We therefore carefully selected a 72-episode miniset covering every task family, scene context, difficulty tier, and condition.

(Continued in Part 2/3.)

AuthorsOpenReview ↗

Response to Reviewer iDQR (part 2/3)

[Part 2/3]

(Continued from Part 1/3.)

W2: Shared harness and inference configurations

  • Identical configuration: All models share the harness interface, prompts, strict validator, and 100-decision budget. The revised Table 16 reports the endpoint settings used in the experiments, replacing the initial defaults listed in the submitted Table 15. Under the identical local configuration (same vLLM stack, 2,048 ceiling, JSON-object response mode), Gemma-4 and Qwen3.8 still lead both embodied models in Progress.

  • Unified JSON interface: All models use the same strict JSON decision format. In the recorded outputs, the longest decision from any general-purpose model contains 692 tokens, below the 2,048-token ceiling.

{"thinking": "...", "actions": [{"tool": "...", "args": {...}}],
 "plan": {...}, "memory": [...], "inspection": "..."}
ModelMeanMedianP95Max
GPT-5.6-Terra126124154561
Claude-Sonnet-5170156287524
Gemini-3.8-Flash114113157299
Qwen3.7-Plus150145224465
Qwen3.8-27B189185271692
Gemma-4-31B214212278449
RoboBrain2.0-32B4522412,0482,048
Hy-Embodied-VLM-1.03022713992,048
  • Outputs at the ceiling: The observed 2,048-token outputs from RoboBrain and Hy-Embodied consist of repeated text with unclosed JSON (zlib ratio 0.023–0.029 vs. ~0.49).

  • Refined claim: The leading proprietary models (Gemini, Claude, GPT) lead the open models, and general-purpose Qwen3.8 and Gemma-4 also lead the embodied models in Progress. These results characterize the tested models on long-horizon, human-involved tasks under the shared harness.

  • Tool-interface errors: On pairs with no truncation and ≥95% parse validity, Gemma leads RoboBrain and Hy-Embodied by +16.1 and +25.4 Progress, and controlling for parse-failure and truncation rates leaves the gaps unchanged (−17.6 → −17.6; −28.6 → −29.1). Format errors and premature completion are observable failures under the shared protocol.

  • Gemini for ablations: We use Gemini as the reference agent because it succeeds on 33 of 72 subset episodes, leaving room to measure the effects of component removal. Five models score at or below 8.1% on the main benchmark, where low success can make such effects harder to resolve. The ablations characterize the harness with this reference agent, while model comparisons rest on the full eight-model evaluation.

  • Revision: We have refined this claim in the abstract, introduction, and Sec. 3.2, corrected the configuration table (now Table 16), and stated the reference-agent scope of the ablations in Sec. 3.3.1, refining their wording to be descriptive.

(Continued in Part 3/3.)

AuthorsOpenReview ↗

Response to Reviewer iDQR (part 3/3)

[Part 3/3]

(Continued from Part 2/3.)

Q1: Code and videos

The updated supplementary provides anonymized source code and rollout demonstrations. We commit to fully open-sourcing the codebase and benchmark upon publication.

Q2: Scoring validation

Reliable pass/fail decisions are what navigation Success depends on.

  • Binary decisions in Success: Only the last of them is judged by a model; the others are rule checks on simulation evidence with no judge:
Binary decisionDecided byBasis
Target observedFixed thresholdSupplied RGB-D capture: at least 256 depth-supported pixels, 6×6-pixel box, 0.04 m volume tolerance
Recipient observedFixed thresholdNamed recipient in the current evidence, fixed RGB-D person-visibility rule (Table 9)
Information receipt (inquiry)Event state machineAuthorized answer delivered by the source
Addressed deliveryEvent state machineRecipient within 2.5 m and visible
Terminal fall or object lossSimulation statePhysical state of the robot and objects
Final-report correctnessReport judgeAuthored answer vs. delivered report
  • Human pass/fail audit: One human annotator independently labelled 180 delivered navigation reports from 6 models against the authored answers using the same rubric. The human labels agree with the judge's pass/fail on 175 of 180 reports (97.2%; Cohen's κ = 0.87). The judge sees only the authored answer and the delivered report, blind to policy identity, plans, later actions, and final success.

  • Revision: We have added this audit to Sec. 3.4 and Appendix D.5.

Q3: Construction clarification

  • 96.7% is the yield after the extra human review: It flagged 12 residual candidates; all 12 were repaired (none replaced), reviewed again, and entered the final 360.

  • Usability audit vs. feasibility audit: Table 6's audit checks every candidate for task coherence and completion of the required interactions and physical goals, and measures the yield of the pipeline; Sec. 3.4 separately reviews the released HC and IU tasks for causal soundness and for feasibility, i.e., whether the scene, available information, interaction conditions, and robot skills admit a task-completing sequence.

  • Revision: We have described the batch protocol and release step in Sec. 3.3.5, Appendix C.4, and the Table 6 caption.

Reviewer_UkDDOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

TARSphere is a simulation framework and benchmark for embodied agents carrying out tasks that involve people and changing requirements. It combines robot skills, interactions with simulated people, and an event-based evaluator that tracks what the agent has received and accomplished. The benchmark contains 60 base tasks across 30 scenes, each with five additional conditions, giving 360 episodes. The authors evaluate eight multimodal models and study the effects of the agent harness, execution assistance, and different sources of failure.

Critical strengths
  1. The handling of task updates is thoughtful. The framework distinguishes an update being triggered from the agent actually receiving it. Earlier actions are evaluated against the requirements active at the time, which makes the assessment of changing goals more meaningful.

  2. Completion requires evidence from execution. Calling a grasp or handover skill is not enough to establish success; the framework checks conditions such as object holding and recipient acceptance. The accompanying calibration experiments help assess how reliably the underlying skills work.

  3. The authors make a serious effort to explain the results. The matched execution interventions, harness ablations, and human checks of semantic scores provide useful evidence beyond the model rankings. The distinction between an undelivered update and a failure to respond to a received update is particularly helpful.

Critical weaknesses
  1. Figure 5 shows that the task-update and human-coordination conditions require more interaction than Base. The controls in Table 4 are useful, but the remaining gaps are still difficult to separate from this additional interaction burden.

  2. Smaller effects, such as the change from 45.8% to 41.7% success when removing spatial memory, are difficult to assess without uncertainty estimates.

  3. The relationship between the 96.7% curation usability rate in Table 6 and the final 360-episode benchmark is unclear.

Questions for authors
  1. Section 3.3.3 distinguishes failures in delivering updates from failures to respond to them. Could you report an aggregate breakdown of these cases? This would help interpret the performance gaps in conditions requiring more interaction.

  2. Could you provide paired confidence intervals or paired outcome counts for Tables 3 and 4, particularly for the smaller differences?

  3. Does Table 6 evaluate the episodes used in the final benchmark or a separate curation batch? If it concerns the final benchmark, were the candidates judged unusable subsequently repaired or replaced?

4 comments
AuthorsOpenReview ↗

Response to Reviewer UkDD (part 1/2)

[Part 1/2]

Thank you for the positive assessment. We clarify our contributions and address each point below.

TARSphere's core contribution

  • The framework: TARSphere is the first robot-oriented full-stack framework for physically and socially grounded embodied agents in evolving environments: a simulation platform with embodied NPCs, an event-driven evaluator, a modular harness, and an authoring interface for developing and evaluating embodied agents and human–robot interaction.
  • Community extensibility: Each layer is open to extension, so the community can build on it rather than start from scratch.
    • Infra: richer physical and social simulation, e.g., more human-like NPCs and new robot embodiments.
    • Harness: stronger components and tools, plugged in under the same evaluator and episodes.
    • Benchmark: new tasks, scenes, and NPC knowledge authored through the interface, so task diversity can keep growing.
  • Code, videos, and open-source release: We have updated the supplementary material with anonymized code and rollout videos. We commit to fully open-sourcing the codebase and benchmark upon publication.

W1 & Q1: Additional interaction, and delivery vs. response failures

This is an insightful question, and one we also considered when designing the benchmark.

  • Extra interaction cost is part of what we evaluate: Information seeking, communication, and replanning are part of the evaluated capability. We therefore match the underlying task while allowing each condition's required exchanges; matching interaction counts exactly would remove necessary exchanges or add unrelated ones to Base.
  • Our design: We anchor each variant to its Base task (e.g., in TU, the updated task is aligned with Base) and let the condition add the interaction it requires. Scene, objects, GT start, final task endpoint, core task structure, decision budget, and scoring stay matched; the interaction the condition requires is the intended difference.
  • Aggregate breakdown (Q1): The table separates runs that never receive the revision from those that receive it and subsequently succeed or fail, pooled over models.
TU familyValid runsUpdate not delivered (%)Delivered, succeeded (%)Delivered, failed (%)
Navigation23753.63.443.0
Delivery24080.89.69.6

Runs that never receive the revision do not directly assess post-update adaptation. Overall TU success measures end-to-end performance; success after delivery reflects both adaptation and subsequent execution.

  • Delivery: The update follows a successful grasp. The triggering grasp is never reached in 76.2% of TU runs and 76.7% of Base runs, indicating a shared upstream execution bottleneck.

  • Navigation: Among runs that receive the update, 7.3% succeed, compared with 28.8% of Base runs reaching the corresponding upstream stage. About half of the TU runs that receive the update but fail declare completion while the revised requirement remains unmet. These results show that difficulties persist after update delivery.

  • HC coordination outcomes: HC takes more decisions than Base (49.5 vs. 41.5 in navigation; 63.9 vs. 63.2 in delivery), as its required coordination implies, yet success depends on the coordination outcome: in navigation it is 23.8% when the required information is obtained and 0% without it.

  • Revision: We have stated the pairing principle in Sec. 2.3, added the TU delivery/response breakdown to Sec. 3.3.3 and Appendix E.2, and refined the task-update claim to reflect it.

(Continued in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer UkDD (part 2/2)

[Part 2/2]

(Continued from Part 1/2.)

W2 & Q2: Uncertainty and paired counts for Tables 3 and 4

  • 45.8% vs. 41.7%: The change is −4.2 points (95% CI [−12.5, +2.8]). The paper claims the pattern: without spatial memory, PR and HC Success rise from 41.7% to 58.3% while the other conditions fall, so the effect is condition-specific. The ablations use a carefully selected 72-episode miniset sized for community-affordable reruns (all 12 settings on the 360 episodes would cost 1.9× our entire eight-model main evaluation), and serve to characterize the harness.
  • Table 3: Each setting is paired with the full harness on common valid episodes: 69 pairs for Horizon 3, 71 for Horizon 5, and 72 otherwise. Valid-pair counts are in Table 20 and 95% bundle-cluster bootstrap CIs in Table 23. No setting exceeds the full harness in paired success; we interpret the differences descriptively.
SettingΔSuccess
w/o grasp skill−25.0 [−44.4, −6.9]
w/o LiDAR−12.5 [−25.0, −1.4]
w/o action residuals−11.1 [−23.6, +0.0]
w/o self-managed memory−11.1 [−23.6, +1.4]
w/o navigation skill−9.7 [−25.0, +2.8]
w/o semantic spatial memory−4.2 [−12.5, +2.8]
Text history 0−9.7 [−18.1, −1.4]
Text history 50−11.1 [−26.4, +2.8]
Visual history 2−6.9 [−18.1, +1.4]
Visual history 4−8.3 [−19.4, +2.8]
Horizon 3−10.1 [−17.5, −2.8]
Horizon 5−14.1 [−28.2, +0.0]
  • Table 4: TU and HC stay 12.0–20.0 points below Base under every form of assistance.
SettingDA − BaseTU − BaseHC − Base
Normal execution−10.0−13.3−16.7
Motion-assisted−10.0−16.7−20.0
Friction-assisted−6.7−13.3−16.7
Grounding-assisted−8.6−12.0−15.3
Geometry-assisted−6.7−13.3−13.3
Fully assisted−6.7−13.3−16.7
  • Revision: We have added paired ΔSuccess with bundle-cluster CIs for all Table 3 settings to the appendix, and refined the wording in Secs. 3.3.1–3.3.2 to describe the ablation and intervention results descriptively.

W3 & Q3: Table 6 and the final 360 episodes

  • Final benchmark or separate batch: The w/o-simulation and Full rows score Batch 4, the construction batch of the final benchmark. The other three rows use Batches 1–3, generated from the same 360 task cards, NPC profiles, and scenes for the ablations only.
  • Protocol: Removing a component means its feedback is not used for repair, while other steps run. For example, Batch 4 (w/o Simulation validation) leaves the candidates that fail simulation unrepaired, while the Full pipeline repairs the same batch with simulation feedback. A candidate is usable if it passes human review, then simulation validation.
SettingBatchAssessmentUsable / 360Usable (%)
w/o Multi-agent review1Human review → simulation16245.0
w/o Human audit2Human review → simulation23465.0
w/o SKILL.md refinement3Human review → simulation26172.5
w/o Simulation validation4Human review → simulation30083.3
Full pipeline4Same, plus an additional final human review34896.7
  • 96.7% is the yield after the extra human review: It flagged 12 residual candidates; all 12 were repaired (none replaced), reviewed again, and entered the final 360.

  • Revision: We have described the batch protocol, assessment, and release step in Sec. 3.3.5 and Appendix C.4, and refined the Table 6 reading accordingly.

My questions are resolved and thanks for the clarifying!

Dear Reviewer UkDD,

Thank you for your positive assessment of TARSphere and your thoughtful feedback. We are glad that our clarifications have addressed your concerns. We hope these clarifications and revisions further strengthen your overall assessment of our work.

Thank you again for your time and constructive discussion!

Best regards,
Authors of TARSphere

Reviewer_41v8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TARSphere, a framework for long horizon embodied agents composed of i) an infrastructure layer in the form of a physics simulator with embodied NPCs and dialogue and ii) an agent harness using frozen MLLMs to control such agents; along with a benchmark of 360 episodes across 30 indoor environments. Its main contribution is integrating physical execution, embodied NPC interactions, event-driven task updates, and progressive evaluation into a unified embodied-agent testing environment.

The authors evaluate 8 MLLMs using the framework and benchmark, covering five capability conditions, and two task families: navigate-and-report and fetch-and-deliver. The results show that even the strongest models evaluated only achieve 40.6% to 47.8% success. To illustrate the strengths of the proposed framework, the authors also include harness ablations, controlled interventions, and human assessment of selected evaluation criteria.

Critical strengths

The paper identifies a meaningful evaluation gap in that long-horizon embodied-agent evaluation frequently abstracts away execution failures and dynamic changes to task requirements. The utilized event-driven task state machine provides a relatively principled way to track task state/validity as environmental and human events occur.

The explicit separation between task progress, final success, and physical execution quality is sensible. The controlled capability variants and reasonable harness ablations also make this a more convincing evaluation-benchmark contribution than merely describing a combination of robot-agent components.

Critical weaknesses

W1. The claimed generality of evolving physical and social environments is not properly supported. Although the framework supports embodied NPCs and dynamic events, NPC behavior and task-relevant dialogue are substantially constrained, also only covering two somewhat templated task types. This makes the environment reproducible but also much easier to model than more open-ended interaction. An evaluated system could succeed by learning event and information structures rather than dealing with NPC behaviour. While the five capability conditions introduce variation, they do not fully establish generality across novel dynamics or other task requirements.

W2. The task-progress metric mixes semantic quality with actual physical accomplishment. Progress averages five stage scores, with some stages assessed semantically and others through binary "physical" evidence. This can be defensible for a diagnostic benchmark, but it creates important ambiguity: two agents with the same progress score may have achieved very different results. Couldn't an agent receive substantial credit for producing a reasonable plan without accomplishing much in the environment? The paper acknowledges that planning credit does not imply execution success, but the consequences for interpreting the core Progress metric are not sufficiently explored.

W3. The comparison between general-purpose and embodied foundation models is not sufficiently controlled. The paper states that general-purpose MLLMs can outperform embodied models. However, the evaluated models receive different provider-specific adaptations and inference configurations (like different settings and tool calling capabilities).

W4. Agent-in-the-loop curation for the 360 episodes is claimed as a major contribution of the work. But this is not properly addressed or supported in the text.

Questions for authors

Q1. Why not include held-out scene families, previously unseen combinations of events, new task structures, and compositional scenarios involving multiple simultaneous changes? This would better support any generalization claims.

Q2. Will the benchmark be released openly and be extensible? This would add great value to it as base for extensions addressing its limitations.

Q3. How much of the difference between general-purpose and embodied models persists after normalizing inference configurations and tool-call validity?

Q4. Can an agent perform well by exploiting predictable NPC/event behavior rather than maintaining "beliefs" about the simulated world?

Q5. How do safety violations correlate with successful progress, and what fraction can be attributed specifically to high-level agent decisions?

Q6. How stable are the ablation conclusions across different MLLMs rather than only the selected model?

Also:

  • The Limitations and Future Work section is too coarse. It's not clear what Mobile-Franka and using different pre-defined task-specific model backends would add to the benchmark.
  • Clarify it's a simulated Unitree G1 with Dex3-1 hands in text (add footnote/link) and better describe which policies are used for navigation/locomotion and arm/hand motion.
  • Fix citations like Gemma-4-31B (Team, 2026) and RoboBrain2.0-32B (Team et al., 2025)
  • Is there some text missing after "Gemini-3.8-Flash illustrates the contrasting profiles at identical 60.0% Base success:"?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 41v8 (part 1/4)

[Part 1/4]

Thank you for the careful review and for recognizing TARSphere's value to the community. We clarify our contributions and address each point below.

TARSphere's core contribution

  • The framework: TARSphere is the first robot-oriented full-stack framework for physically and socially grounded embodied agents in evolving environments: a simulation platform with embodied NPCs, an event-driven evaluator, a modular harness, and an authoring interface for developing and evaluating embodied agents and human–robot interaction.
  • Community extensibility: Each layer is open to extension, so the community can build on it rather than start from scratch.
    • Infra: richer physical and social simulation, e.g., more human-like NPCs and new robot embodiments.
    • Harness: stronger components and tools, plugged in under the same evaluator and episodes.
    • Benchmark: new tasks, scenes, and NPC knowledge authored through the interface, so task diversity can keep growing.
  • Code, videos, and open-source release: We have updated the supplementary material with anonymized code and rollout videos. We commit to fully open-sourcing the codebase and benchmark upon publication.

W2: Interpretation of Progress

  • Two metrics, two roles: Success is the binary end outcome; Progress measures how far a run got along the causal chain of milestones. We report both because equal Success can hide very different progress: of the failing runs, 46.2% never get the target observed (Progress 0–20%), 41.0% observe it but do not complete the onsite task (20–40%), and 12.8% get further (≥40%).
  • Semantic scoring and execution evidence: Progress combines semantic assessment of plans and task-relevant conclusions/reports with checks on physical and interaction evidence under the task's event dependencies (Sec. 3.1; Appendix D.2).
  • Execution evidence by Progress: The table relates Progress bins to recorded execution milestones across all 2,873 scored runs:
Progress (%)RunsTarget observed in RGB-D (%)Onsite task done (%)Success (%)
0–201,1710.00.00.0
20–401,039100.00.00.0
40–60147100.086.40.0
60–80171100.099.40.6
80–<10069100.0100.091.3
100276100.0100.0100.0
  • Score from a plan alone: At most 20%. Recomputing Progress from stages 2–5 alone gives the identical model ranking (Kendall τ = 1.00; episode-level Spearman ρ = 0.95).

  • Revision: We have stated in Sec. 3.1 that the plan stage contributes at most 20 points and that every later stage requires physical or interaction evidence.

AuthorsOpenReview ↗

Response to Reviewer 41v8 (part 2/4)

[Part 2/4]

W1, Q1 & Q4: Task scope, scene coverage, and NPC/event behavior

Thank you for this insightful question. We examined the logs to check whether successful runs completed the required interactions and received the episode-specific information.

  • Constrained NPC behavior: Intentional, and open to extension.
    • Reproducibility and stability: Constraining what an NPC may say and do at evaluation time keeps every model facing the same information state, so results are reproducible and comparable across models and reruns, and it prevents the LLM backend from hallucinating task facts.
    • LLM-driven backend: Our NPCs use an LLM-driven backend that supports open-ended dialogue. During evaluation, we deliberately restrict replies to authorized topics and facts (Sec. 2.2.1).
    • Semi-open by design: Within the evaluation constraint, the NPC interprets paraphrases and contextual follow-ups and provides situation-dependent answers. When asked the way to the study entrance, for example, the NPC computes the reply from the robot's live position and heading ("the corridor doorway into the study is about d m ahead-left relative to your current body heading"), so the same question gets different answers from different positions.
    • Event state: The benchmark makes task-relevant social information explicit: who knows what, when an update is delivered, and whether an interaction occurred. The agent observes these changes through images and interaction feedback.
    • Toward more open and realistic interaction: We share your view that more open and realistic human behavior is a valuable direction. TARSphere provides a foundation for it, though our claim here is the framework and its evaluation of physical and social task states rather than open-ended human behavior itself.
  • Episode-specific information (Q4): Knowing an event type does not reveal its content: the revised recipient, private location, or preference is supplied by the person in the current episode. With frozen models and fresh memory, all successful runs completed the required interaction:
ConditionRequired interactionRunsSuccessesSuccesses without it
TURevision delivered477310
HCInformation/contribution received478460
IU (clarification required)Intent clarified303100

The evaluator requires these interactions for success, so the zero-bypass counts confirm compliance with those dependencies; they do not independently rule out a fixed interaction strategy.

  • Robustness across scene subsets (Q1): TARSphere evaluates frozen models without benchmark-specific training. Removing any entire scene context from the analysis, including the ten household scenes, leaves the model ranking unchanged (Kendall τ = 1.00). Across 1,000 random 15/15 scene splits, Gemini ranks first and the embodied models last in both halves 100% of the time.
  • Task families and diversity (Q1): The two families, navigate-and-report and fetch-and-deliver, combine navigation with information gathering, reporting, object acquisition, and handover. The benchmark varies their scenes, roles, objects, routes, and interaction requirements:
DimensionCoverage
Scenes30 scenes (10 homes, 20 commercial spaces) in 8 contexts, e.g., household, restaurant, hospital, supermarket, bookstore, office
Tasks60 Base tasks (30 navigation, 30 fetch-and-deliver) with 5 paired variants each: 360 episodes in 12 authored subtypes, e.g., low light, spatial blocking, task content update, information coordination, preference confirmation
RolesRequester A, information-source NPC B, and recipient C form 4 configurations: return to A (29 Base tasks), ask B then return to A (7), fetch then hand to C (16), ask B then report to C (8)
Objects and people45 target object instances in 12 categories; 36 NPC names (72 in-scene identities); 1 to 3 NPCs per Base task
Routes and interactionPlanned Base routes average 13.35 m (4.89–31.95 m) and cross rooms and functional areas; 2 to 5 authored required exchanges per episode; 240 distinct opening utterances
Difficulty11 easy, 38 medium, 11 hard Base tasks, derived from route scale, partition crossings, occlusion, tortuosity, and clearance
  • Beyond the current scope: Unseen event combinations, new task structures, and simultaneous changes are future axes supported by the same authoring interface.

  • Revision: We have stated this scope in the limitations.

AuthorsOpenReview ↗

Response to Reviewer 41v8 (part 3/4)

[Part 3/4]

W3 & Q3: Inference configuration and output ceilings

Thank you for this careful question; a fair comparison across heterogeneous endpoints is central to our claims, and we analyzed every policy call to answer it.

  • Identical configuration (Q3): All models share the harness interface, prompts, strict validator, and 100-decision budget. The revised Table 16 reports the endpoint settings used in the experiments, replacing the initial defaults listed in the submitted Table 15. Under the identical local configuration (same vLLM stack, 2,048 ceiling, JSON-object response mode; Table 18), Gemma-4 and Qwen3.8 lead both embodied models in Progress. For Gemma, the gaps also persist on pairs with no truncation and ≥95% parse validity (95% scene-cluster bootstrap CIs):
ComparisonΔProgressΔSuccess
Gemma vs. RoboBrain+17.7 [14.5, 21.1]+7.8 [4.2, 11.8]
… both untruncated, ≥95% parse-valid+16.1 [12.3, 20.2]+3.8 [1.1, 6.9]
Gemma vs. Hy-Embodied+28.7 [25.2, 32.3]+7.8 [4.2, 11.7]
… both untruncated, ≥95% parse-valid+25.4 [22.1, 28.9]+4.7 [2.1, 7.9]
  • Unified JSON interface: All models use the same strict JSON decision format. In the recorded outputs, the longest decision from any general-purpose model contains 692 tokens, below the 2,048-token ceiling.
{"thinking": "...", "actions": [{"tool": "...", "args": {...}}],
 "plan": {...}, "memory": [...], "inspection": "..."}
ModelMeanMedianP95Max
GPT-5.6-Terra126124154561
Claude-Sonnet-5170156287524
Gemini-3.8-Flash114113157299
Qwen3.7-Plus150145224465
Qwen3.8-27B189185271692
Gemma-4-31B214212278449
RoboBrain2.0-32B4522412,0482,048
Hy-Embodied-VLM-1.03022713992,048
  • Outputs at the ceiling: The observed 2,048-token outputs from RoboBrain and Hy-Embodied consist of repeated text with unclosed JSON (zlib ratio 0.023–0.029 vs. ~0.49).

  • Tool-call validity: Controlling for parse-failure rate, truncation rate, and episode fixed effects leaves the gaps unchanged (−17.6 → −17.6; −28.6 → −29.1), and the most format-valid local model, Qwen3.8-27B (98.1%), still leads both embodied models by 10.6 and 21.7 Progress points. Format errors and premature completion are part of the capability under test.

  • Refined claim: The leading proprietary models (Gemini, Claude, GPT) lead the open models, and general-purpose Qwen3.8 and Gemma-4 also lead the embodied models in Progress. These results characterize the tested models on long-horizon, human-involved tasks under the shared harness.

  • Revision: We have refined this claim in the abstract, introduction, and Sec. 3.2, and corrected the configuration table (now Table 16).

W4: Agent-in-the-loop curation

  • Description: Curation makes task semantics, authorized NPC knowledge, event dependencies, and scene geometry agree (Sec. 2.3). The curator jointly solves bindings, dependencies, and interaction positions; review, audit, and simulation feedback drive repair.

  • In the paper: The curation workflow is described and evaluated in the paper. Sec. 2.3 and Fig. 4 give the four steps; Table 6 and Sec. 3.3.5 ablate its components (45.0 / 65.0 / 72.5 / 83.3% usable vs. 96.7% for the full pipeline); Sec. 3.4 reviews the causal soundness and feasibility of the released HC and IU tasks. The appendix gives the authoring workflow, authoring and review prompts, SKILL.md rules, and the human audit interface (App. C.3–C.4, Figs. 14–18), with matched task cases in Table 14 and Fig. 13 and the BEVs of all 60 Base tasks in Figs. 24–25.

  • Revision: We have described the curation details more fully in Sec. 3.3.5 and Appendix C.4.

Q2: Release and extensibility

Thank you for recognizing the value of TARSphere as a base for community extensions and for your expectation of its open release; we are committed to it.

  • Supplementary material: We have submitted a new supplementary package with the anonymized code and rollout videos.
  • Code and benchmark: we are committed to release the full codebase and benchmark.
  • Extensibility: As outlined at the top, every part supports extension: the Infra, the modular Harness, and the Benchmark. We will also open-source the benchmark curation pipeline (SKILL.md, prompts, authoring and validation tools), so the community can generate data and extend to new tasks, e.g., safe human–robot interaction, multi-person collaboration, and long-horizon household assistance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

TARSphere rebuttal summary

Summary of Rebuttal and Discussion

Dear Area Chair and Reviewers,

Thank you for your time and thoughtful feedback throughout the review process.

TARSphere is a robot-oriented, full-stack framework designed as a shared foundation for developing, evaluating, and extending long-horizon embodied agents in evolving physical and social environments. Grounded in physical execution and human interaction, TARSphere evaluates how agents adapt to diverse runtime changes through three modular components:

  1. TARSphere-Infra: Combines physics-based robot execution, embodied NPCs, and an event-driven evaluator that delivers dynamic task updates and verifies completion from execution and interaction evidence.
  2. TARSphere-Harness: Connects frozen multimodal models to executable robot skills, supporting online decision revision via observations, memory, and execution feedback.
  3. TARSphere-Bench: Uses agent-in-the-loop curation to construct 360 episodes across 30 indoor scenes (60 Base tasks across two service families, each with five paired variants).

Communities can swap models or replace the harness under a fixed simulator and evaluator, or reuse our curation workflow to expand the benchmark to new tasks and scenes.

We have uploaded the revised PDF, which clarifies the evaluation scope, Progress scoring, and inference settings and adds paired uncertainty estimates and a human audit of binary report scoring, along with the updated supplementary ZIP containing the revised appendix, anonymized source code, and rollout videos.

Summary of Reviewer Engagement

  • Reviewer 41v8: We clarified how Progress is scored and how inference settings affect the model comparison. We also narrowed our generalization claims as the reviewer suggested.
  • Reviewer UkDD: We provided a breakdown of task-update delivery and response failures, added paired uncertainty estimates, and clarified how candidate episodes were audited and repaired. We appreciate the reviewer’s confirmation that these responses addressed the questions raised.
  • Reviewer iDQR: We responded to the reviewer’s questions in the rebuttal and included the requested anonymized code and rollout videos in the supplement.

Thank you for considering our responses and revised materials.

Best regards,
The TARSphere Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.