acceptodds
Under review as a conference paper at ICLR 2027

TutelaBench: Evaluating the Safety of LLM Planners for Care Robots Among Simulated Humans

Abstract

Large language models are increasingly proposed as high-level planners for assistant robots, including in caregiving. Existing embodied safety benchmarks place the agent in a world without people, where safety is defined as “not doing dangerous things”. However, in a care setting, for instance, failure often looks different: something necessary is left undone, a commitment is forgotten, or the wrong person's demand prevails. To tackle this, we introduce TutelaBench, a benchmark that places an LLM-controlled agent in simulated elderly care and child care environments populated by scripted human characters who ask, insist, refuse, and need help. Requirements can come from multiple sources (deployer policies and the wishes of the persons cared for), and these sources can conflict. The world changes on its own clock, so that doing nothing can itself be unsafe, forcing the agent to revise and adapt the original plan. Each run yields a task score and an independent safety score. Safety is operationalized as compliance with explicitly stated requirements, which can be violated by action or by omission. These scores are computed from simulation state rather than judged by an LLM. On eight open-weight models and five agent architectures, TutelaBench shows task scores on the safety-critical scenarios barely separate models or architectures, whereas safety differs systematically. Stating the guideline in the prompt raises mean safety only moderately but increases the share of scenarios that are safe in every run by about half. A guideline critic raises safety further. However, it also uses about 2.5 times more LLM calls than a plain ReAct agent. Even the safest architecture violates a requirement in about a third of the scenarios over ten runs. The tested models and architectures thus often meet safety requirements, but none does so reliably across repeated runs, which leaves safe LLM-based planning for care robots an open problem.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
ai-safety, safety-alignment, agents, embodiment, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Gk74OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TutelaBench evaluates LLM-based planners in simulated elderly-care and child-care environments. It combines scripted human interactions, conflicting requirements, evolving environmental events, and safety failures caused by both actions and omissions. Task performance and safety compliance are scored separately from simulator states. Experiments with eight open-weight models and five agent architectures show that relatively high task performance does not imply consistently safe behavior, and that explicit guidelines and critics can improve repeated-run safety.

Critical strengths
  1. The benchmark is a meaningful extension of safety evaluation to care obligations. It includes omissions, proactive intervention, social pressure, and conflicting instructions. These features capture important failure modes that evaluations focused on isolated dangerous requests can miss, making the benchmark a contribution.
  2. This paper introduces a benchmark that evaluates task performance and safety compliance as separate dimensions. Separating task performance from safety compliance prevents task completion from obscuring violations. Simulator-based scoring makes outcomes inspectable, while repeated-run metrics expose inconsistency that average scores alone would hide.
Critical weaknesses
  1. Scenario and scoring validation are insufficiently established. The main text does not adequately explain how requirements are translated into scoring conditions, how acceptable alternative behaviors are handled, or how scenarios are verified to admit a fully compliant solution under their observation and timing constraints. This matters particularly for omissions and proactive behavior, where failures should reflect missed opportunities based on available information.

  2. Model comparisons are confounded by changes in the interlocutors. Section 3 states that the evaluated model also generates residents’ free-text replies. Changing the planner therefore changes part of its interaction environment. State-machine constraints improve control, but differences in dialogue clarity or informativeness may still influence outcomes. The use of LLM-based dialogue-transition classification introduces another potential source of variation. Without evidence that results are robust to a fixed interlocutor and transition classifier, the reported differences cannot be attributed solely to planner capability.

  3. The architecture comparisons do not isolate the sources of improvement. Critic-based agents receive additional inference calls, while simulated time is decoupled from computation time. Their gains therefore combine guideline auditing with additional deliberation that incurs no simulated delay. Comparisons under matched inference budgets are needed to separate these effects. Likewise, the Robot architecture jointly adds memory, review, and self-checking, leaving their individual contributions unclear.

Questions for authors
  1. What validation establishes that each scenario has a fully compliant solution under the available observations and deadlines, and how are alternative compliant behaviors recognized?

  2. Which model performs dialogue-transition classification, and is there evidence that model rankings remain stable with a fixed interlocutor and classifier?

  3. Do critic-based improvements persist under comparable inference budgets?

3 comments

Dear Reviewer,

Thank you for your thorough review and the helpful comments. We address each of your points below.

{Likewise, the Robot architecture jointly adds memory...} We agree that evaluating additional architectures would strengthen the paper. However, it is not feasible to cover all possible architectures, so we focused on those we considered most relevant and informative. We would also like to clarify that the goal of our paper is not to isolate or quantify the contribution of individual components. Instead, we aim to show that all evaluated methods, regardless of their sophistication, fail to behave safely. Since even the most elaborate configurations exhibit these failures, we do not expect other configurations to resolve them. Moreover, several components are only meaningful in combination. For example, the review step relies on the memory system to function properly, so evaluating it in isolation would not result in a sensible configuration.

{Clarify how requirements translated into scoring condition.} We are happy to clarify the benchmark setup and the scoring conditions. In TutelaBench, all safety goals are weighted equally, since the violation of any safety-critical goal indicates unsafe behavior and is thus unacceptable. Assigning them different weights would relativize the absolute importance of each. For safety scenarios, the only goal contributing to the task score is typically whether the system keeps the person involved by talking to them rather than resolving the safety issue silently. For non-safety scenarios, the task is typically more complex, and the task score is distributed across several sub-goals.

{1.} We believe that this is one of the crucial points of TutelaBench. This is why the correctness of the scenarios is verified by humans. Each scenario is also checked statically (every condition refers to an existing state, object or location). The time limits are generous: completing a task correctly typically takes only about 50% of the allotted time, so low scores are not caused by strict deadlines. To verify they are solvable for the agents we only use scenarios which least once have full score, independent on agent and model. During the design phase we tried to limit the space of alternative compliant solutions, focusing on an obvious guideline following solution.

We agree that safe behaviors other than the prescribed ones may exist. TutelaBench measures whether a planner reliably complies with the requirements it has been given, in a world with other people and a running clock. We do not claim that our rule set is the correct standard of care; standards differ between institutions and jurisdictions, which is why requirements are configurable. We do claim that reliable compliance with stated requirements is a necessary condition for deploying an LLM planner in care, because a deployer can only vet, certify, and oversee behavior that was specified in advance. This is also why we do not accept alternatives where a requirement prescribes a procedure. Contacting the prescribing physician in the medication example is a reasonable action, but it replaces the prescribed check with the agent's own judgment, and an agent that does so in one run gives no assurance about the next. Human care work has the same structure: verifying a medication against the order before administering it is mandatory however experienced the caregiver is. We will make this scope explicit in the introduction and in Section 2, and, in the released scenario files, mark each requirement as procedural or outcome-based so that users can report the two separately.

\textit{2. Which model performs dialogue-transition classification, and is there evidence that model rankings remain stable with a fixed interlocutor and classifier?}

In the first iteration the model under test performed the dialogue-transition classification. We reran the experiments with the simulated humans, both their replies and the dialogue-transition classifier, served by one fixed model (Qwen3.6-27B), independent of the model under test, and raised the runs per configuration from 10 to 15. The baseline suite is complete (8 models, 5 architectures, 14 scenarios, 15 runs: 8,400 runs) and is reported in the updated Table 3 and the appendix tables; the safety-critical suite is still running and will be added to the final version. For more see Q2 from canU.

{3.} We did not isolate these two factors, and we agree that the critic’s gain reflects both guideline auditing and the additional deliberation it allows at no cost in simulated time. We will clarify this in the results section and no longer attribute the gain to auditing alone. Importantly, this does not affect our main conclusion. Even the critic architecture, which uses roughly 2.5 times as many LLM calls as plain ReAct, still violates at least one requirement in about a third of its runs.

Thank you for the detailed response. Some of my concerns remain. The added checks do not establish that every scenario has a fully compliant solution or that reasonable safe alternatives receive credit. Also, the fixed-interlocutor rerun is complete for the baseline tasks, but not yet for the safety-critical results. I may be missing something, but based on the current evidence, I cannot rate the paper above 3.

Thank you for the follow-up. As noted in our general response, the fixed-interlocutor rerun is now complete for the safety-critical suite as well, and the conclusions hold. On your two remaining concerns:

Every scenario has a fully compliant solution. We now check the strictest criterion: a run with full task credit and no safety violation at all. Every one of the 14 baseline and 33 safety-critical scenarios has such runs, each produced by at least 7 of the 40 model–architecture configurations and at least 5 of the 8 models. These are actual benchmark runs under the real observations and deadlines, so a compliant solution demonstrably exists.

Reasonable safe alternatives receive credit. Scoring reads the simulation state and never compares against a reference trajectory. A safety requirement fails only when a stated violation occurs, e.g. medication mixed into food. Duties are checked by their effect, e.g. "the physician knows about the refusal", so it does not matter whether the agent messages him or walks over, or how it phrases the report. An alternative loses credit only where the requirement itself prescribes a procedure, such as checking the order sheet before giving a drug; there, the requirement rules it out, not the scorer. Two elderly-care practitioners who reviewed five scenarios judged none of their 21 requirements wrong (Appendix, Practitioner Review).

We hope this addresses your remaining concerns and would be grateful if you would reconsider your rating.

Reviewer_odLbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TutelaBench, a benchmark for evaluating the safety of LLM-based planners in simulated caregiving environments. The benchmark uses a text-observed 2D simulation with scripted human interactions, dynamic environmental events, and safety requirements derived from multiple sources that may conflict. It evaluates both task completion and safety compliance, including unsafe actions and failures to act when necessary. The authors evaluate 8 open-weight LLMs across 5 agent architectures. The experimental results suggest that while current LLM agents achieve relatively high task completion scores, they frequently violate safety requirements, particularly across repeated passes.

Critical strengths

None

Critical weaknesses

TutelaBench receives text descriptions rather than camera inputs. Its agent issues high-level commands in a 2D tile world, where navigation is handled by a shortest-path algorithm and atomic actions are assumed to execute safely. This aligns with text-based agent literature rather than embodied agent safety. A more appropriate novelty comparison should explain what TutelaBench adds beyond τ²-bench, SOTOPIA, and MACHIAVELLI. Also the authors fail to include related hierarchical instruction following for LLM agents like IHEval and ManyIH-Bench.

This brings us to the more critical issue with the current submission. The novelty of the proposed benchmark appears relatively limited compared to existing works on embodied agent safety especially with previous work that focus on high level planning safety. Prior benchmarks such as SafeAgentBench, IS-Bench, and SENTINEL have already investigated the evaluation of safety constraints in embodied task planning and execution; HAZARD considers dynamically changing environments with time-sensitive hazards. TutelaBench primarily combines these existing evaluation concepts within a caregiving setting, introducing scripted human interactions, conflicting requirements, and time-dependent obligations. While this combination is potentially useful, the underlying technical methodology remains largely conventional: the benchmark relies on predefined atomic actions, scripted state machines, manually specified safety conditions, and simulator-state-based evaluation, without introducing a substantially new safety specification formalism, verification method, or planning algorithm.

I am also skeptical about the scope of the current submission. In the experiment section, the authors only used 2 different scenarios to run the experiment (elderly care and childcare), and all the experimental settings seem extremely heuristic based. As the authors have noted in the Ethics section, the safety constraints in this submission is also hand crafted instead of being reflective of what the real care giving services might need to follow.

Questions for authors
  1. What safety relevant behaviors or failure modes can TutelaBench evaluate that cannot be adequately evaluated by existing benchmarks and specification-based frameworks, and can the authors empirically demonstrate this distinction? In other
  2. On line 316, the authors said that the residents’ free-text replies are, for simplicity, sampled by the same model as the agent under test. Does this not create cofounding effect as different models are getting different text responses (which will then presumably be the input for their next action)?
  3. How were the safety requirements and evaluation criteria developed and validated? Given that the authors acknowledge the lack of consultation with caregiving professionals, what evidence supports the claim that these requirements meaningfully capture safety in caregiving scenarios rather than simply compliance with predefined rules?
6 comments

Dear Reviewer, Thanks for pointing us to IHEval and ManyIH-Bench. We have added both papers to the related work and compare TutelaBench to these works. While both benchmarks evaluate the safety of LLM agents, both papers evaluate whether a model resolves conflicts between instructions of differing privilege. This is either done by message role (IHEval) or by explicit privilege annotations (ManyIH-Bench), and only the highest-privileged instruction is correct to follow. In contrast to this, TutelaBench differs in two crucial points. First, authority is not given: the agent must determine from the scenario who has the authority to impose a given action. Second, conflicts need not have a winning instruction. In many scenarios the compliant action is to escalate to a human rather than follow any party, an outcome the IHEval formulation does not represent.

{Questions} {1. What safety relevant behaviors or failure modes can TutelaBench evaluate that cannot be adequately evaluated by existing benchmarks and specification-based frameworks, and can the authors empirically demonstrate this distinction?}

SafeAgentBench and IS-Bench place the agent in a world without humans, where the hazard lies in the environment or the instruction. HAZARD adds time-sensitive hazards, but still no people. τ2\tau^2-bench and SOTOPIA include interlocutors, but no world that changes on its own clock, so failing to act in time cannot occur. In SOTOPIA neither agent acts as an assistant on behalf of the other, and their speech, gestures, and physical actions are free text that only extend the dialogue history. In TutelaBench, residents ask, insist, and refuse while the world changes, and a requirement can be violated by doing nothing. The results show why this matters: on the safety-critical scenarios, task scores barely separate models or architectures, while safety differs systematically, so a benchmark scoring task success would not detect these failures. Like SafeAgentBench and IS-Bench, we evaluate the high-level planner and assume atomic actions execute safely. This is deliberate (see our response to Reviewer MrAP on SPOC). Finally, we would like to clarify that the benchmark comprises 49 scenarios (14 baseline and 35 safety-critical, after filtering) in two domains (elderly care and childcare), not two scenarios.

{2. On line 316, the authors said that the residents’ free-text replies are, for simplicity, sampled by the same model as the agent under test. Does this not create cofounding effect as different models are getting different text responses (which will then presumably be the input for their next action)?} We agree that this might be a confound. We re-ran the experiments with the residents' replies and the dialogue-transition classifier served by one fixed model (Qwen3.6-27B), independent of the model under test. On the completed baseline suite, the model ranking is largely stable (Kendall's τ=0.78\tau = 0.78; the reordering is confined to models within 0.06 of each other) and the architecture ordering is unchanged except for one swap within 0.01. The safety-critical suite is still running and will be added to the final version. Further details can be found in our response to Reviewer canU (Q2).

{3. How were the safety requirements and evaluation criteria developed and validated? Given that the authors acknowledge the lack of consultation with caregiving professionals...}

To assess whether the scenarios are sufficiently realistic, we had a sample of the elderly care scenarios reviewed by two elderly care practitioners, and the childcare scenarios were validated by members of the author team who are parents.

TutelaBench measures whether a planner reliably complies with the requirements it has been given, in a world with other people and a running clock. We do not claim that our rule set is the correct standard of care. Standards differ between institutions and jurisdictions, which is why requirements are configurable. However, we do claim that reliable compliance with stated requirements is a necessary condition for deploying an LLM planner in real-world scenarios like caregiving, because a deployer can only vet, certify, and oversee behavior that was specified in advance. This is also why we do not accept alternatives where a requirement prescribes a procedure. Contacting the prescribing physician in the medication example is a reasonable action, but it replaces the prescribed check with the agent's own judgment, and an agent that does so in one run gives no assurance about the next. Human care work has the same structure: verifying a medication against the order before administering it is mandatory, no matter how experienced the caregiver is. We will make this scope explicit in the introduction and in Section 2, and, in the released scenario files, mark each requirement as procedural or outcome-based so that users can report the two separately.

Dear Reviewer odLb,

The discussion period closes on November 18, so we wanted to follow up on our response. We hope it has answered your questions, in particular:

Related work: we added IHEval and ManyIH-Bench to the related work and explained how TutelaBench differs. In our benchmark, a conflict does not always have one instruction that should win.

What TutelaBench adds: residents ask, insist and refuse while the world changes on its own clock, so an agent can break a requirement just by doing nothing. We also clarified that the benchmark has 47 scenarios in two domains, not two scenarios.

Confound from residents’ replies: we re-ran the experiments with one fixed model playing the simulated humans.

Validation of the requirements: two elderly care practitioners reviewed a sample of the elderly care scenarios. We also stated the scope of the benchmark explicitly: it measures whether an agent reliably follows the requirements it was given, not whether those requirements are the correct standard of care.

If anything is still unclear, or if you have further questions, we are happy to answer them before the discussion closes. If our response has resolved your concerns, we would be grateful if you would consider updating your assessment.

Sorry for the delay in response. I have reviewed the updated pdf regarding the cofounding results as well as the coverage for the benchmark. The clarifications did help with the clarity of some of the concerns I have had earlier. However, the fundamental contribution the author claimed with other existing benchmark are "residents that ask, refuse, and insist" and "a world that changes on its own clock" do not seem to be novel enough as a standalone benchmark for the following reasons.

  1. The authors emphasize the presence of simulated residents who can ask, refuse, and insist as a central distinction from existing benchmarks. However, the residents' behavior is primarily governed by predefined state machines, with LLMs generating their freetext responses and classifying certain dialogue transitions. It remains unclear what additional evaluation capabilities arise from this design compared to deterministic multi-turn interactions or existing social-agent benchmarks such as SOTOPIA and τ²-bench. It is unclear if the LLM generated free response is human like and different from fixed interactions.
  2. The world that changes on its own clock also boils down to text input changes to the evaluated models (new instructions, new hierarchy). However, dynamic environments and time-sensitive hazards have already been studied in prior work such as HAZARD. Similarly, τ²-bench already provides a shared dynamic environment in which simulated users can independently modify the world state through tool interactions. In TutelaBench, these dynamics are implemented through scheduled state changes and predefined conditions within a simplified simulation. It is also unclear how the simplifications of these cartoon like world changes are distinct modes from the existing benchmarks.
  3. The authors distinguish TutelaBench from instruction hierarchy benchmarks by emphasizing that the appropriate response to conflicting requests may be to seek clarification or escalate to a human rather than simply follow the highest-priority instruction. However, deciding when to ask for clarification or escalate to human assistance has already been considered in other agent evaluation settings like τ²-bench and HiL-Bench.

Overall, I appreciate the authors' efforts to clarify the intended scope and improve the experimental controls. Nevertheless, I remain concerned that the paper primarily combines existing evaluation concepts into a niche application domain without a clearn significant contribution other than the application domain itself is relatively new. That said, as a benchmark contribution, I remain skeptical of how useful the current scope of the benchmark's application is.

Reviewer_MrAPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark for evaluating the safety of LLM planners in elderly care and child care settings. The benchmark is implemented as an abstract simulation and includes 53 scenarios with scripted people, conflicting requests, safety rules, and events that unfold over time. Task completion and safety are scored separately from simulation states, including failures caused by failing to perform necessary actions. Experiments with eight open-weight models and five agent architectures are presented with detailed experimental results.

Critical strengths

The benchmark addresses important AI safety issues in human care. It tests not only harmful actions but also omitted responsibilities, delayed responses, and failures to resolve conflicting requests. These are important issues for AI agents that assist vulnerable people. The benchmark includes a simulation environment that supports controlled, interactive testing. Agents act in a changing world inside the simulation and receive feedback from their actions. People, objects, events, and rules are easily configurable, which makes the framework useful for testing different planners. The experiments presented in the paper cover several models and architectures. Comparisons with and without guidelines and critics help show how these components affect safety. The paper also openly discusses limits in its rules, simulation, and deployment claims.

Critical weaknesses

The authors state that the safety requirements were not derived from clinical guidelines, laws, or professional standards and were not reviewed by care practitioners. The paper should explain how scenarios were identified and selected, which risks they cover, and how their realism was checked. Without this evidence, higher scores on the benchmark primarily indicate better compliance with the authors’ rules, rather than demonstrated improvements in real-world caregiving safety. The paper should provide evidence that the measured failures reflect real caregiving risks and that higher scores indicate better safety-relevant abilities.

In the medication example, an agent's decision to contact the prescribing physician was evaluated as a failure because the benchmark solution expect the agent to read the prescription order sheet. This raises a question about whether the benchmark measures safe decisions or exact procedural compliance. Expert review of the required responses and accepted alternatives would strengthen the scoring.

The authors need to include more recent related works. For example, SPOC[1] already evaluates safety during task execution, including time-limited follow-up actions. Trinity[2] considers task planning that depends on human attributes. The paper should explain what its combination of social pressure, conflicting authority, changing events, and missed duties adds beyond the settings of these related works.

The distinction from existing simulation-based agent benchmarks is not sufficiently clear. Benchmarks such as Overcooked-AI, Watch-And-Help, and Melting Pot already study coordination and social interaction in simulated environments. The paper should explain which challenges in the proposed simulation were addressed beyond these previous works. In particular, it should show how care-specific duties, authority conflicts, and safety requirements introduce distinct evaluation challenges, rather than mainly changing the setting and rules.

The time and cost evaluation has limits. The simulation excludes model computation time. Call counts reveal added cost, but do not show whether these architectures can respond safely under real-time constraints.

[1] Kim, Hyungmin, et al. "SPOC: Safety-Aware Planning Under Partial Observability and Physical Constraints." ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2026.

[2] Kim, Hyungmin, et al. "Trinity: Human–Agent–Environment Alignment for Embodied Task Planning." Findings of the Association for Computational Linguistics: EMNLP 2026. 2026.

Questions for authors
  1. How do the scenarios and scoring rules capture real caregiving risks, and what do they test beyond existing benchmarks such as SPOC and Trinity?

  2. What evidence supports the framework’s reproducibility and ease of extension? Are the code, configurations, and evaluation scripts publicly available? How are new scenarios checked for solvability, correct state transitions, and accurate scoring?

  3. How strongly do the results support the claimed gap between task performance and safety? Can you distinguish failures that cause harm or neglect from failures to follow a required procedure? How might the findings change with a fixed model for simulated human responses and reasoning-enabled planners?

3 comments

Dear Reviewer, thanks for taking the time for reviewing and your valuable feedback!

Regarding the cost metric we agree: call counts measure compute overhead, not latency. However, leaving out wall-clock time is deliberate. The simulation advances in discrete steps, so scores reflect decision quality rather than serving hardware. Because calls within a step run sequentially, calls per step times per-call latency still gives a lower bound on real-time decision latency. However, we agree that this is indeed an interesting question for future work. We will clarify this point in the future work section.

{Questions} {1.} The scenarios target risks that are well known in caregiving scenarios: medication errors, falls, fire evacuation, acute deterioration, poisoning or allergy, and safeguarding. Where possible, they follow a documented source, such as published guidance [1,2]. In addition, two elderly-care practitioners reviewed five scenarios; both rated all as realistic, judged 14 of 21 requirements necessary to prevent harm and none wrong (Appendix A.3).

[1] K. Krieger, D. Sulmann, and D. Väthjunker, "Stärkung der Sicherheitskultur in der professionellen Langzeitpflege: Praxisleitfaden für Pflegeorganisationen," Zentrum für Qualität in der Pflege, Berlin, Germany, Mar. 2026, doi: 10.71059/HURS2317.

[2] A. Mc Gillicuddy, A. M. Crean, M. Kelly, and L. Sahm, "Oral medicine modification for older adults: A qualitative study of nurses," BMJ Open, vol. 7, no. 12, Art. no. e018151, Dec. 2017, doi: 10.1136/bmjopen-2017-018151.

To show the novelty of TutelaBench, we added SPOC to the comparison in Table 1. SPOC shares our closed-loop setup and state-based scoring. Unlike TutelaBench, it additionally introduces partial observability and physical constraints. We deliberately excluded both. They make a benchmark harder, but they also mix two different sources of failure. An agent can fail because it lacks information or cannot physically carry out a plan. Or it can fail because it chooses an unsafe action. Only the latter is a safety failure of the LLM planner, and this is what TutelaBench is designed to measure. With full observability and no physical limits, every unsafe outcome can be traced back to the planner’s own decisions. We therefore place the difficulty where safety-relevant decisions are made. These are interactions with humans, conflicting guidelines, and sudden events outside the agent’s control. Our experiments show that current agents behave unsafely in exactly these situations, even though they have full information about the environment. We see SPOC and TutelaBench as complementary. SPOC tests how well agents plan under perceptual and physical limits. TutelaBench isolates whether their decisions are safe.

Trinity appeared after the deadline and seems to be parallel work. Incorporating it required significant work, which was unfortunately not feasible during this rebuttal phase. But integrating it would be a good starting point for TutelaBench v2. It evaluates personalization rather than safety. The user’s attributes (diet, religion, allergy) are given in the instruction. The user is not present in the scene, and the world does not change during the task. TutelaBench instead tests whether agents keep people safe when the relevant facts must be acquired during the episode, under social and time pressure, and where inaction can cause harm. Trinity’s attribute-conditioned goals are a promising extension: in a future version, a combination of both would evaluate person-specific safety constraints (see further work in conclusion).

{2.}

After acceptance the code and all scenarios will be made publicly available for other researchers to extend the benchmark with their own scenarios. Each scenario is also checked statically (every condition refers to an existing state, object or location). Humans check the scenarios for solvability and correctness of the state transitions. Further, we only keep scenarios that were solved successfully at least once across all models and agents: 49 of 51 scenarios pass this filter (14 baseline and 35 safety-critical).

{3.}

Harm or neglect can be told apart from procedural failures by qualitative analysis, but the score deliberately does not do so. TutelaBench measures whether a planner reliably complies with the requirements it is given. We do not claim our rule set is the correct standard of care; standards differ between institutions, which is why requirements are configurable. Reliable compliance is, however, a necessary condition for deployment in care, because a deployer can only vet and oversee behavior that was specified in advance. Contacting the physician in the medication example is reasonable, but it replaces the prescribed check with the agent's own judgment. We will make this scope explicit in Section 2 and mark each requirement as procedural or outcome-based in the released scenarios.

The comments by the authors resolved many of my concerns, especially regarding discussions on missing recent related works and reality check on the scenarios. I maintain my score as these still do not justify the sufficient novelty of the proposed benchmark.

We thank the reviewer and are glad our response resolved the concerns on related work and realism. To address the remaining concern as precisely as possible, could the reviewer point to the work(s) they consider closest to TutelaBench, or name the specific aspects they see as not novel? For reference, we see TutelaBench as novel in the following respects:

  • Clear guidelines, precise expectations. Agents fail even though what is expected of them is explicitly stated.
  • Safety as a positive duty. Refusal or inaction fails.
  • Embodied, time-extended episodes. Agents move through a shared environment and handle stateful objects under deadlines.
  • Simulated humans. Residents, relatives and staff with their own goals and state. They apply social pressure, give or withhold consent, and may contradict the guidelines.
  • Outcome-based scoring. Success is checked deterministically against the world state rather than judged from transcripts by an LLM.
  • Reliability. Using pass^k over 15 replicates, we show that agents which are sometimes safe often fail when consistency is required.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response: fixed-interlocutor rerun and summary of revisions

We thank the reviewers for their time and their valuable feedback. We are happy to hear that the reviewers agree that TutelaBench addresses an important and underexplored AI safety problem in human care, covering failure modes such as omissions, proactive intervention, social pressure and conflicting instructions that existing embodied safety benchmarks largely miss (MrAP, canU, Gk74). The reviewers further agree on the practical value of the benchmark, which is well designed, easy to configure and extend, and useful for testing different planners with easily traceable failures (MrAP, canU).

We updated the paper, all changes are marked in blue to make them visible.

All four reviewers pointed out that the simulated humans' replies and the dialogue-transition classifier were served by the model under test, so that changing the planner also changed part of its environment. We re-ran the experiments with both served by one fixed model (Qwen3.6-27B) and raised the runs per configuration from 10 to 15. The baseline suite is complete (8,400 runs): the model ranking is largely stable (Kendall's τ=0.78\tau = 0.78), the architecture ordering is unchanged except for one swap within 0.01, and absolute task scores rise by 0.04--0.11 for models from 9B upward. The safety-critical suite is still running, with about 40% finished there are no significant changes. We will report whether those findings are equally robust in the final version.

Dear Reviewer, Thank you for your time and your careful, constructive feedback. We agree that the paper did not provide enough supporting material on our scenarios, and we have addressed this as described below.

{Q1. Will the simulator, maps, scenario and guideline configuration files, and run logs be released? Could you also add a scenario catalogue to the appendix, listing each scenario's setting, characters, targeted competence, and safety goals, together with a few visualised example trajectories, including typical failures?}

We will release the code, scenarios, and the run results after acceptance. We agree that having the code and the configuration files publicly available is essential for benchmarking the safety of the models. We hope that publicly releasing our code, motivates others to contribute new scenarios, strengthening the benchmark.

In the updated paper, we included an overview table of the scenarios as well as three additional visualised example trajectories in the appendix. Due to the overall length of all scenarios descriptions (+160 pages) we avoided to include all scenarios in the appendix. We included the complete set in the supplementary material.

{Q2. Can you re-run, at least for a subset of models, with the scripted humans' replies and the transition classifier served by one fixed model, independent of the model under test? Does the model ranking in Table 3 or the architecture ordering change? How often did the transition classifier choose a state that a human annotator would disagree with?}

We thank the reviewer for this suggestion. We reran the experiments with the simulated humans, both their replies and the dialogue-transition classifier, served by one fixed model (Qwen3.6-27B), independent of the model under test, and raised the runs per configuration from 10 to 15. The baseline suite is complete (8 models, 5 architectures, 14 scenarios, 15 runs: 8,400 runs) and is reported in the updated Table 3 and the appendix tables. The safety-critical suite is still running and will be added to the final version.

As can be seen in the results, the model ranking is largely stable: Kendall's τ=0.78\tau = 0.78 on the baseline task score, with 24 of 27 model pairs keeping their order. The strongest models (Qwen3.8-27B and gemma-4-31B-it, 0.86) and the weakest (Qwen3.5-2B, 0.28) are unchanged. The reordering is confined to the middle of the ranking, where the models were within 0.06 of each other. The largest change is Mistral-Small-3.2 (0.72 to 0.83, from sixth to fourth). The architecture ordering is also stable: ReAct remains the best and Critiqued ReAct the weakest architecture on the baseline task score. Only Robot and Critiqued Robot swap places (0.75 vs.\ 0.76).

AuthorsOpenReview ↗

Final Results

We thank the reviewers (MrAP, canU, Gk74) for their quick response. The fixed-interlocutor rerun has now finished for the safety-critical suite too. The residents are played by one fixed model (Qwen3.6-27B), as in the baseline suite. The revised PDF reports the complete results (changes in blue).

The conclusions hold. The architecture ordering is unchanged, and the model ranking is stable relative to our earlier results (Kendall's τ=0.75\tau = 0.75). Guidelines raise mean safety by 0.09 and safe$^{10}$ by 0.11, and a critic adds another 0.08 in safe$^{10}$ (all pHolm≤0.02p_\text{Holm} \le 0.02). Still, no configuration is reliable: even the best one violates a requirement within ten runs on a third of the scenarios.

We also tightened validation: every kept scenario now has at least one run with full task and safety score. This removed four scenarios leaving 47 of 51 scenarios.

Dear Area Chair,

here is a short summary of the discussion. Reviewers pointed out that the human responses, though well instructed, were not produced by the same model across all runs. We acknowledge the difference and re-ran the experiments with one fixed environment model, increasing the sample size from 10 to 15 runs. Absolute scores rise slightly, and the model ranking and architecture ordering stay largely stable.

Novelty relative to HAZARD, SOTOPIA, τ2\tau^2-bench, SafeAgentBench and IS-Bench. Each covers part of what TutelaBench combines: HAZARD has a world that changes on its own clock but no people. SOTOPIA and τ2\tau^2-bench have simulated people, but the agent acts only through dialogue and tool calls in a digital world, and the state changes only when someone acts. SafeAgentBench and IS-Bench are embodied, but have neither people nor a world that changes on its own.

We followed the practice of the other benchmarks and wrote the requirements ourselves, as TutelaBench is not a final deployment-ready check but a research benchmark that measures whether agents reliably follow the stated requirements. In addition, we asked two elderly care practitioners to review a sample of five scenarios: both rated all of them as realistic, both judged 14 of 21 requirements necessary to prevent harm, at least one of them saw each of the other seven as procedure, and neither rated any as wrong.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.