acceptodds
Under review as a conference paper at ICLR 2027

TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns

Abstract

Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focused. However, when interacting with real-world users, especially those experiencing cognitive impairments, such as people living with dementia (PLWD), the planners often make mistakes and even pose physical safety risks. We proposed TALK-Dem (Talking Attributes and Linguistic Knowledge in Dementia), the first benchmark for evaluating LLM-driven robot task planning under dementia-associated verbal communication. TALK-Dem contains 4,800 instructions and covers five typical communication patterns, including Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion, at three intensity levels. Experiments across six open-weight LLMs reveal a substantial robustness gap. Across communication patterns, open-weight models exhibited performance drops of up to 22.3 percentage points compared to ideal instructions. This revealed a critical gap and even danger for real-world applications, especially in assistive robotics, where locally deployable models are necessary due to privacy concerns and connectivity constraints. To mitigate this issue, we proposed the Context-Aware Retrieval from Experience (CARE) method, which retrieves relevant previously resolved tasks to provide task-specific interpretation and planning context. CARE generally outperformed standard prompting baselines across the six open-weight models, improving average task success by 18.1 percentage points over the vanilla prompt. These results highlighted the importance of both evaluating communication robustness and developing effective adaptation strategies for locally deployable assistive robots.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied task planning, dementia-associated communication, large language models, human-robot interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r4XQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces TALK-Dem, a benchmark for evaluating embodied task planning under dementia-associated communication patterns. Building on REI-Bench tasks, it constructs 4,800 instructions across 300 household tasks, covering five clinically motivated communication patterns at three intensity levels alongside clean controls. It also proposes CARE, a retrieval-based method that supplies relevant task interpretations and reference plans as demonstrations. The authors evaluate eight language models using SayCan in AI2-THOR, compare the synthetic instructions with DementiaBank linguistic features, and analyze planning failures and safety-relevant actions. The results show that these communication variations can degrade planning performance and that CARE generally improves task success over the evaluated prompting baselines.

Critical strengths
  1. The paper provides a benchmark and useful research directions for studying embodied intelligence under dementia-associated communication patterns, supporting research toward more inclusive assistive systems.
  2. The paper adapts retrieval-augmented generation to this setting and demonstrates its effectiveness on the proposed benchmark, providing a practical approach to improving task-planning robustness.
Critical weaknesses
  1. Despite the DementiaBank feature comparisons, it remains unclear whether the synthetic instructions and intensity levels reflect real patient interactions, limiting confidence in their real-world applicability.
  2. Stronger models already perform well on the benchmark, while CARE’s main gains are demonstrated on older open-weight models. Its benefits for newer models with comparable deployment costs remain unclear.
Questions for authors
  1. Beyond the linguistic feature comparisons with DementiaBank, is there evidence from real interaction data or expert assessments supporting the realism of the synthetic expressions and the three intensity levels?
  2. Have you evaluated newer open-weight models with comparable inference costs?
1 comment
AuthorsOpenReview ↗

Response to Reviewer r4XQ

Thank you for your constructive feedback. We have clarified the scope and limitation of our synthetic benchmark and added evaluations of newer open-weight models. All changes are highlighted in the revised PDF. Below is our detailed point-by-point response to your questions.


W1 and Q1. Do synthetic instructions and intensity levels reflect real-world dementia-associated communication?

We would like to clarify that TALK-Dem is intended as a controlled benchmark of dementia-associated communication patterns, rather than a simulation of real conversations with people living with dementia. Each pattern is grounded in clinical literature, and our comparison with DementiaBank shows that the generated instructions exhibit relevant linguistic characteristics (Table A1). The three intensity levels represent increasing degrees of each communication pattern, not clinical stages of dementia. The mapping between language patterns and dementia severity is a complicated medical topic beyond the scope of the current work. The controlled design allows us to isolate the effects of individual patterns on embodied planning. We acknowledge the limitations of synthetic instructions and the need for real-world validation. We have clarified the intensity levels in Section 3.1 and the limitations and need for real-world validation in Section 7.


W2 and Q2. Does CARE remain effective on newer open-weight models?

Following the reviewer’s suggestion, we additionally evaluated two newer open-weight models of comparable size, Qwen3-8B and gemma-4-E4B-it, using the same SayCan framework and 16 instruction conditions. CARE consistently improves task success and robustness to communication variations on both models.

ModelClean (Vanilla/CARE)Patterned Avg. (Vanilla/CARE)Retention (Vanilla/CARE)
Qwen3-8B66.7 / 82.759.0 / 80.881.8 / 89.9
gemma-4-E4B-it36.0 / 43.331.9 / 43.870.6 / 76.4

CARE improves average patterned-task success by 21.8 percentage points on Qwen3-8B and 11.9 points on gemma-4-E4B-it. Both models also show improved retention, suggesting that CARE's benefits extend to newer open-weight models. We have added the full results to Appendix B and updated the discussion in Section 5.3. We compared CARE with Vanilla methods, which is achievable within the rebuttal period.

Reviewer_e1WJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TALK-Dem, a benchmark for robot task planning under communication patterns related to dementia. It studies five patterns, such as vague references, wrong object names, topic drift, and interruptions. The paper also proposes CARE, a retrieval method that uses similar past tasks to help planning.

Critical strengths
  • The problem is important for assistive robots.

  • The benchmark is clearly designed. It has clean instructions and controlled variants.

  • The experiments cover several open and closed models.

  • CARE gives strong overall gains.

  • The paper also looks at safety-related errors.

Critical weaknesses
  1. The data is fully synthetic. The authors compare it with DementiaBank, but this only checks a few language features. It does not show that the examples match real conversations with people living with dementia.

  2. The experiments use only SayCan and AI2-THOR. It is unclear if the results will hold for other planners, other robot systems, or real robots.

  3. The paper does not compare with a clarification baseline. In many cases, a robot should ask a question instead of guessing the user's intent. This seems important for the setting of the paper.

  4. CARE uses retrieved examples with gold task interpretations and reference plans. This is helpful for the experiment, but it is a strong setup. The paper should better explain where such reliable solved examples would come from in real use. It would also be useful to test CARE with noisy retrieval or weaker memory items.

  5. The safety analysis is interesting, but it is limited. It uses a small set of hand-made rules. The paper shows that task success does not always mean safe behavior, but it does not include a safety metric or a safety-aware planning method.

Questions for authors

I like the problem and the benchmark design. The results are useful, and CARE is a promising method. However, the main evidence still comes from synthetic data in one simulator and one planner. I also think a clarification baseline is missing. These issues make it hard to judge how well the results will transfer to real assistive robots.

2 comments
AuthorsOpenReview ↗

Response to Reviewer e1WJ (Part 1/2)

Thank you for the constructive feedback and positive assessment of TALK-Dem. We have conducted additional experiments and revised the manuscript to better address the limitations of synthetic data, the role of clarification, and CARE's practical applicability. We respond to each concern below.


C1. Do synthetic communication patterns reflect real-world dementia-associated conversations?

We would like to clarify that TALK-Dem is intended as a controlled benchmark of dementia-associated communication patterns, rather than a conversation simulation of real people living with dementia. The five patterns are grounded in clinical literature, while our DementiaBank comparison examines linguistic characteristics (Table A1), rather than establishing conversational realism. This controlled design allows us to isolate the effects of individual patterns on embodied planning. We have clarified the scope of our benchmark and the role of the DementiaBank comparison in Section 3.2.


C2. Do the findings generalize to other planners, robot systems, or real robots?

We additionally evaluated LLM+P [1] across three open-weight models in AI2-THOR to test whether our findings extend beyond SayCan. We evaluated six instruction conditions: one clean condition and Level 3 of each of the five communication patterns.

ModelPlannerClean (Vanilla/CARE)L3 Avg. (Vanilla/CARE)
Llama-3.1-8B-InstructSayCan66.7 / 76.358.2 / 75.4
LLM+P72.0 / 78.365.0 / 75.3
Qwen2.5-7B-InstructSayCan28.7 / 63.718.8 / 46.9
LLM+P68.3 / 72.355.1 / 74.1
Ministral-8B-Instruct-2410SayCan73.7 / 83.756.2 / 78.3
LLM+P74.0 / 82.067.9 / 81.6

Under LLM+P, all three models exhibit clean-to-Level-3 performance degradation (6.1–13.2 percentage points), while CARE improves Level 3 task success by 10.3–19.0 points over Vanilla. These results suggest that both the communication robustness challenges and CARE's benefits extend to another planning framework. We have added the evaluation results to Appendix B.10 and discussed the need for further validation in other simulators and physical robots in Section 7.


C3. Why not use clarification as a baseline?

TALK-Dem only covers the situation when task information is sufficient to evaluate whether planners can recover user intent without additional clarification. For individuals living with dementia, if the robot does not have sufficient information to execute actions, the robot should stop instead of proceeding before reliable clarification can be acquired. Clarification is a cognitively demanding action that can cause further confusion and cognitive load for individuals living with dementia [2, 3]. How to design clarification for PLWD without triggering stigma and more cognitive load is a complex and understudied medical/cognitive science domain, which is beyond the scope of the article.

Following the reviewer's suggestion, we additionally evaluated how often models request clarification as their first action, even when the intended task remains recoverable. The results reveal substantial model-dependent differences:

ModelCleanPatterned (Avg.)
Llama-3.1-8B-Instruct19.0%77.4%
Ministral-8B-Instruct-24100.0%4.3%
Qwen2.5-7B-Instruct0.0%2.4%
Qwen3-8B (additional)1.3%6.5%

These results reveal substantial model-dependent differences in clarification seeking behavior. We have added the clarification-request analysis to Appendix B and clarified the rationale for not relying on clarification in Section 3.2.

AuthorsOpenReview ↗

Response to Reviewer e1WJ (Part 2/2)

C4. How realistic is CARE's memory setup, and how sensitive is it to retrieval quality?

CARE is designed to leverage previously resolved tasks, which could be collected from verified successful executions. To examine its dependence on memory quality, we conducted two additional ablations: (i) clean-only memory, which keeps only the clean instructions of the 100 held-out tasks, and (ii) random retrieval, which selects three demonstrations from the full memory at random instead of by semantic similarity.

  1. CARE achieves comparable task success with clean-only memory. This suggests that closely matched communication-pattern examples are not necessary for its effectiveness.

  2. Random retrieval substantially reduces CARE's task success. This highlights the importance of retrieving task-relevant demonstrations rather than selecting them randomly. | Model | Method | Clean | Patterned Avg. | Retention | |---|---|---:|---:|---:| | Llama-3.1-8B-Instruct | Vanilla | 66.7 | 64.0 | 80.4 | | | CARE | 76.3 | 77.8 | 88.0 | | | CARE (Clean Memory) | 82.3 | 79.2 | 88.0 | | | CARE (Random Retrieval) | 72.3 | 67.3 | 78.7 | | Qwen2.5-7B-Instruct | Vanilla | 28.7 | 21.7 | 54.6 | | | CARE | 63.7 | 50.6 | 67.2 | | | CARE (Clean Memory) | 66.7 | 51.0 | 65.5 | | | CARE (Random Retrieval) | 43.0 | 35.6 | 53.1 | | Ministral-8B-Instruct-2410 | Vanilla | 73.7 | 61.3 | 78.5 | | | CARE | 83.7 | 81.4 | 90.7 | | | CARE (Clean Memory) | 84.7 | 79.7 | 87.8 | | | CARE (Random Retrieval) | 74.7 | 69.4 | 81.7 |

We have explained the source of CARE's memory examples and added two ablation studies in the revised manuscript.


C5. Why not include a comprehensive safety metric or safety-aware planner?

We would like to clarify that a comprehensive safety metric is beyond the scope of this study, as it must account for both physical hazards and cognitive safety, which depends on individual users' cognitive capabilities and needs. Instead, we focus on identifying common physical hazards not explicitly evaluated by ALFRED. We identified 85 cases where hazardous actions occurred under patterned instructions but not their clean counterparts (Section 6.2), highlighting potential safety risks associated with communication variations. We agree that comprehensive safety metrics and safety-aware planning require further research and human-centered validation, and have clarified this limitation in Section 7.


References

[1] Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., & Stone, P. (2023). Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477.

[2] Pilnick, A., O'Brien, R., Beeke, S., Goldberg, S., & Harwood, R. (2021). Avoiding repair, maintaining face: Responding to hard-to-interpret talk from people living with dementia in the acute hospital. Social Science & Medicine, 282, 114156.

[3] Small, J. A., & Perry, J. (2005). Do you remember? How caregivers question their spouses who have Alzheimer’s disease and the impact on communication. Journal of Speech, Language, and Hearing Research, 48(1), 125-136.
Reviewer_KuCyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The goal of the work is to study LLM planners for embodied tasks (in this case, the ALFRED benchmark). Their specific focus is to study LLMs as planners for these tasks under dementia-related communication patterns. They do so by transforming regular task-oriented instructions into varying levels of vagueness to simulate PLWD users. As expected, they find that this perturbation reduces task performance. To mitigate this, they propose a method, CARE, which retrieves trajectories from an external task bank that are semantically related to the current query. It seems to improve performance over methods that only paraphrase the instruction to reduce the ambiguity introduced by user error.

Critical strengths
  • The paper is well written and easy to understand.
  • The problem of dealing with ambiguous instructions, for dementia and even beyond that in assistive robotics, is an important one.
Critical weaknesses
  • Although I understand the motivation for working on robot planners with ambiguous user instructions, I am unsure if a very specific case here for PLWD is a useful problem per se. It is clear from prior work (REI-Bench), as cited by the authors, that ambiguity in instructions will cause a loss in performance. Specifically, establishing it for PLWD instructions does not add extra signal to the problem, and does not justify a benchmark in itself.
    • Of course, benchmarking given different kinds of vagueness will give different trends, but how is it actually helping us make progress on a problem we are already aware of, beyond what existing benchmarks already do?
  • Next, understanding vague instructions is a general issue beyond embodied tasks. Imagine PLWD interacting with a chatbot; it faces the same challenge of building a coherent understanding of the user’s intent. If the instruction can be paraphrased into a refined one with the correct intent, that largely resolves the downstream task-planning problem. The paper’s own results support this: TOCC, which simply rewrites the instruction before planning, achieves retention close to CARE (85.8% vs 87.7%, Table B10) and outperforms it on Topic Drift and Intrusion (Table B11), and both closed-source models are barely affected by any pattern (Table B1).
    • Studying it as a language problem would also be far more scalable: instead of running a simulator or a real robot, many bad-instruction-to-refined-instruction pairs can be collected using a frontier model, which is how TALK-Dem itself was generated.
  • Lastly, the proposed CARE method goes beyond refining the instruction by retrieving resolved tasks, with task interpretations and reference plans, from an external memory bank. Its gains appear to come mostly from improving the planner rather than from handling the communication patterns: CARE raises success on clean instructions from 67.3% to 80.7% on average (Tables B4–B9), which accounts for most of its 18.1-point headline improvement. The comparison to AP, CoT, and TOCC is also uneven, since these receive no reference plans.
    • The memory bank is built from held-out tasks transformed with the same 15 pattern variants and generation prompts as the test set (Appendix D.6), so retrieved examples closely match the test instructions in style and task type. Is it feasible to expect such a well-matched memory bank in practice? An ablation using a bank of only clean instructions would help clarify this.
Questions for authors
  • By design, it seems like even with level 3 intensity, the instruction always does contain the objects needed for the task, or some reasonably close reference to it. Therefore, the perturbation is introducing mostly noise in the instruction.
    • Was there a reason that the authors didn’t erase an entire target object from the instruction? That would actually make evaluating the proposed CARE method interesting, because it can’t just do semantic matching based on target objects; it needs to realize that a user in a particular scene usually wants some objects in a given configuration, which makes the problem even more interesting. Like, instead of putting the watch on the table, the user says Put the watch, then gets distracted and never says where.
  • In Figure 1, the example used is not the best. The instruction goes from “please heat an apple and place it on the table” -> heat a tomato, sorry, an apple, and place it on the table. The failure mode depicted here is that the agent gets a knife and puts it into the microwave. The knife is not mentioned in the clean or the perturbed instruction; therefore, that’s a hallucination in the LLM policy itself, not something that’s due to the person instructing having a mental disorder. Would the same model produce the correct plan with a clean instruction? The improvement to this is a better planner, not instruction refinement. Even if retrieval-based methods like CARE curb this issue, they are fixing the LLM policy, not fixing the problem of ambiguous instructions. The idea in the figure confounds these two things. In the case that the example is made up, it’s fine, but it should be clarified in the caption.
  • Given that prior work has already worked on and argued for dealing with ambiguous instructions in embodied task planning, can the authors clarify why a dementia based angle is specifically outstanding to be studied with their separate benchmark?
  • ALFRED as a benchmark is quite outdated; any reason the authors did not experiment with more recent ones, e.g., PARTNR (Chang et al., ICLR 2025)?
  • Could the authors show a concrete example of CARE’s retrieved demonstrations for a test instruction to illustrate how closely they match it?
5 comments
AuthorsOpenReview ↗

Response to Reviewer KuCy (Part 1/3)

Thank you for your thoughtful comments. We have conducted additional experiments to address your concerns and revised the manuscript to clarify our core motivations and methodology. Below is our detailed point-by-point response.


W1 and Q3. Why is a separate benchmark needed for dementia-associated communication?

We would like to clarify that TALK-Dem is unique because existing ambiguity benchmarks are not dementia focused, as they cover very limited types of communication patterns that may overlap with dementia-associated patterns and often conflate these with other types of patterns, making it difficult to isolate and evaluate each pattern. TALK-Dem addresses these limitations in three ways:

  1. Additional patterns supported by clinical evidence. Unlike REI-Bench, which focuses on referential vagueness, TALK-Dem covers four additional patterns grounded in clinical literature that have not been addressed in any prior benchmarks. Instead of generic vagueness, these patterns capture distinct clinical phenomena: self-corrections (Object Substitution), low-information phrasing (Empty Speech), and discourse disruptions (Topic Drift and Intrusion).

  2. TALK-Dem does not require clarifications to determine the meaning of requests, which is a particular setting for dementia applications. Existing ambiguity benchmarks often assume clarification is an available strategy to improve the understanding of requests. However, clarification is a cognitively demanding action that can cause further confusion and cognitive load for individuals living with dementia [1, 2]. TALK-Dem specifically tests whether planners can recover the intended task without relying on additional user clarification, which may not be achievable for people living with dementia.

  3. TALK-Dem presents a clear and distinct mapping from a language pattern to a robot planning failure, enable targeted diagnosis and correction. In TALK-Dem, different communication patterns are associated with distinct planning failures (Table 1), highlighting the importance of pattern-specific evaluation for identifying planner vulnerabilities and guiding targeted mitigation strategies. However, in other benchmarks, the language patterns associated with dementia are often mixed, making precise failure cause diagnosis difficult.

Together, these distinctions highlight the need for a dedicated, complementary benchmark to evaluate embodied planning under dementia-associated communication patterns. We have updated Sections 1 and 2 to emphasize this motivation.


W2. Why is embodied evaluation necessary over language-only evaluation?

We choose embodied evaluation because correct instruction interpretation does not guarantee successful or safe robotic execution. While existing home-robot studies often evaluate predefined instructions in controlled settings, TALK-Dem examines how dementia-associated communication patterns affect downstream task execution and safety. Our results demonstrate why this distinction matters:

  1. Rewriting does not ensure task success. Although TOCC achieves retention close to CARE (85.8% vs. 87.7%), retention is measured relative to each method's own clean-task performance. Their actual task success differs substantially (62.8% vs. 78.1%). Even under Topic Drift and Intrusion, where TOCC has higher retention, its success remains lower than CARE (66.2% vs. 76.6% and 67.4% vs. 78.0%, respectively).

  2. Robustness of closed-source models needs to be demonstrated through robot planning. We agree with the reviewer that language refinement is the most important part, given that the same robot planner is used. Meanwhile, we would like to clarify that task success can only be evaluated through the pipeline from LLM to robot planner, as the final execution of physical embodiment. Therefore, even though closed-source models can handle language patterns well, the landing effect must be demonstrated through a robot planner. Thus, text-only evaluation alone cannot determine downstream planning success, as the pipeline is not complete yet.

  3. Planning and execution failures can only be disclosed through embodied evaluation. TALK-Dem evaluates actual task completion in robot simulator AI2-THOR, which maps language issues into interpretable robot execution failures (Table 1). 85 cases of safety-relevant actions occurred under patterned instructions but not under their clean counterparts (Section 6.2). These safety issues cannot be explicitly demonstrated without a robot simulator.

AuthorsOpenReview ↗

Response to Reviewer KuCy (Part 2/3)

W3. Clarifications on CARE's robustness gains, baseline fairness, and memory-bank dependence.

CARE improves both general planning and robustness to communication variations, without requiring a closely matched memory bank. We support this with three findings:

  1. CARE’s robustness is beyond general planning. Beyond improving clean-instruction success, CARE also reduces the clean-to-patterned performance drop from 7.3 to 2.6 percentage points and improves average retention from 78.0% to 87.7% (Table B10).

  2. All baseline comparisons contain the same reference plans at the beginning of the prompt. We apologize for not making the baseline setup clear. All methods share the same SayCan base prompt with several fixed examples (instructions and reference plans). ICL and CARE use the same demonstration format: ICL uses five fixed demonstrations, while CARE retrieves three task-relevant ones.

  3. Results of clean-only memory and random retrieval ablation. Following your suggestion, we additionally evaluated CARE with a memory bank of 100 held-out tasks and with random retrieval of three demonstrations. The results are shown below. | Model | Method | Clean | Patterned Avg. | Retention | |---|---|---:|---:|---:| | Llama-3.1-8B | Vanilla | 66.7 | 64.0 | 80.4 | | | CARE | 76.3 | 77.8 | 88.0 | | | CARE (Clean Memory) | 82.3 | 79.2 | 88.0 | | | CARE (Random Retrieval) | 72.3 | 67.3 | 78.7 | | Qwen2.5-7B | Vanilla | 28.7 | 21.7 | 54.6 | | | CARE | 63.7 | 50.6 | 67.2 | | | CARE (Clean Memory) | 66.7 | 51.0 | 65.5 | | | CARE (Random Retrieval) | 43.0 | 35.6 | 53.1 | | Ministral-8B | Vanilla | 73.7 | 61.3 | 78.5 | | | CARE | 83.7 | 81.4 | 90.7 | | | CARE (Clean Memory) | 84.7 | 79.7 | 87.8 | | | CARE (Random Retrieval) | 74.7 | 69.4 | 81.7 |

    Across the three models, CARE with clean memory achieves comparable patterned-task success to CARE (69.99% vs. 69.96%), while random retrieval reduces success to 57.4%. These results suggest that matched communication patterns are not necessary, but retrieving relevant prior tasks contributes to CARE's effectiveness.

We have added these ablations to Appendix and clarified CARE's improvements in Section 5.3.


Q1. Why does TALK-Dem preserve recoverable task information?

TALK-Dem intentionally preserves recoverable task information to isolate planning difficulties caused by dementia-associated communication patterns from those caused by missing information.

Following the reviewer's suggestion, we added a more challenging Referential Imprecision Level 4 (L4) condition that removes the descriptive clue identifying the target object.

ModelVanilla L3Vanilla L4CARE L3CARE L4
Llama-3.1-8B39.72.075.79.7
Qwen2.5-7B13.71.041.05.0
Gemma2-9B52.05.380.711.0
Ministral-8B35.02.076.012.0

Removing the identifying clue substantially reduces success, illustrating the difficulty of planning with missing information. In such cases, the robot should avoid guessing and stop execution until clarification can be made. However, designing appropriate clarification for individuals living with dementia without triggering stigma and more cognitive load is a complex and understudied medical/cognitive science domain, which is beyond the scope of the article. We have added this experiment to the Appendix and clarified our benchmark design rationale in Section 3.2.


Q2. Why does Figure 1 represent a communication robustness failure?

Figure 1 shows a real execution episode, where the same model (Llama-3.1-8B) successfully completed the task under the clean instruction condition.

While we agree that the knife is hallucinated by the LLM policy, this failure is triggered by the dementia-associated communication pattern. TALK-Dem is designed to reveal how such patterns expose downstream planning vulnerabilities. All 85 safety-relevant cases in Section 6.2 exhibited hazardous actions under patterned instructions but not their clean counterparts. Such failures may be mitigated through instruction refinement, improved LLM reasoning, or stronger planning constraints and safety checks.

We have clarified in the Figure 1 caption that this is an actual execution episode and that the same planner succeeds under the clean instruction.

AuthorsOpenReview ↗

Response to Reviewer KuCy (Part 3/3)

Q4. Rationale for choosing ALFRED over newer benchmarks such as PARTNR.

We selected ALFRED to isolate the impact of dementia-associated communication patterns on embodied planning while minimizing confounding factors like difficulty and multi-agent coordination. We highlight this choice through two key points:

  1. ALFRED provides a controlled setting for evaluating communication robustness. TALK-Dem builds on the widely cited REI-Bench's ALFRED task pool, where candidate tasks were screened for successful execution using Llama-3.1-8B + SayCan. This provides feasible seed tasks and supports methodological continuity with REI-Bench and its TOCC baseline.
  2. PARTNR contains confounding factors that make it hard to identify failures due to dementia-associated communication patterns. PARTNR introduces multi-agent coordination and more complex task constraints. These factors are blended with various communication patterns, making it hard to isolate failures caused by communication variations. PARTNR’s best LLM baseline achieves approximately 30% task success without privileged information, compared with 93% for humans [3].

Recent work such as EmbodiedBench (ICML 2025) [4] also uses ALFRED for high-level planning evaluation. We acknowledge the value of extending TALK-Dem to newer benchmarks and have clarified our choice of ALFRED in Section 3.2 and discussed future extensions to PARTNR in Section 7.


Q5. How closely do CARE's retrieved demonstrations match the test instruction?

CARE retrieves semantically relevant prior tasks as demonstrations. Below is an actual retrieval example for Object Substitution (Level 3).

Test instruction:

“Good evening. I’d appreciate your help. Put the washed plate, no, the washed bowl, sorry, the washed plate, no, the washed bowl on the wooden shelf.”

Instruction from the top-3 retrieved demonstrations:

  1. “Hi, please put the washed cup, sorry, the washed mug, no, the washed cup, actually, the washed mug on the coffee machine.”

  2. “Hey there. I need you to put a warm bowl... um, was it a warm bowl or a warm plate? A warm plate in the fridge.”

  3. “Hey there. I need you to put a warm bowl, no, a warm plate, wait, a warm bowl, sorry, a warm plate in the fridge.”

These retrieved instructions illustrate the semantic similarities between the test query and prior tasks. Our clean-memory ablation further shows that matching communication patterns are not necessary for comparable performance. We have added complete retrieval examples, including task interpretations and reference plans, to the Appendix and clarified CARE's conclusions.


References

[1] Pilnick, A., O'Brien, R., Beeke, S., Goldberg, S., & Harwood, R. (2021). Avoiding repair, maintaining face: Responding to hard-to-interpret talk from people living with dementia in the acute hospital. Social Science & Medicine, 282, 114156.

[2] Small, J. A., & Perry, J. (2005). Do you remember? How caregivers question their spouses who have Alzheimer’s disease and the impact on communication. Journal of Speech, Language, and Hearing Research, 48(1), 125-136.

[3] Chang, M., Chhablani, G., Clegg, A., Dallaire Cote, M., Desai, R., Hlavac, M., ... & Yang, T. Y. (2025, May). Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations (Vol. 2025, pp. 65205-65268).

[4] Yang, R., Chen, H., Zhang, J., Zhao, M., Qian, C., Wang, K., Wang, Q., Koripella, T.V., Movahedi, M., Li, M., Ji, H., Zhang, H. & Zhang, T.. (2025). EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:70576-70631.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewer Concerns and Rebuttal Updates

Dear Area Chair,

We summarize the main concerns raised by the reviewers and the corresponding updates made during the rebuttal.

  • CARE effectiveness and memory bank design: Reviewers questioned whether CARE's improvements primarily stem from general planning gains or reliance on closely matched memory examples. We clarified the baseline setup and added clean-only memory (using only instructions without dementia-associated communication patterns) and random-retrieval (randomly selecting memory examples) ablations, along with concrete retrieval examples. The results show that CARE improves communication robustness beyond general planning gains and does not require matching communication patterns in the memory bank.

  • Generalization across models and planners: Reviewers raised concerns about evaluation limited to a single planning framework and older open-weight models. We added evaluations with LLM+P across three models and tested two newer open-weight models (Qwen3-8B and gemma-4-E4B-it). The results show consistent improvements with CARE, suggesting that its benefits extend beyond the original models and planner.

  • Recoverable task information and clarification: Reviewers questioned the assumption of recoverable task intent and the absence of an interactive clarification baseline. We clarified that TALK-Dem intentionally preserves recoverable task information to isolate planning difficulties caused by dementia-associated communication patterns from those caused by missing information. We also conducted additional missing-information and clarification-request experiments. The results highlight the challenges of incomplete instructions and model-dependent clarification behavior, supporting our benchmark's controlled design.

  • Benchmark motivation and synthetic realism: Reviewers questioned the distinction from existing ambiguity benchmarks, the need for embodied evaluation, and the realism of synthetic instructions. We clarified that TALK-Dem is designed as a controlled, pattern-specific benchmark rather than a simulation of real-world conversations with people living with dementia. By isolating individual dementia-associated communication patterns, it enables a systematic diagnosis of embodied planning vulnerabilities that existing ambiguity benchmarks cannot capture.

We have revised the manuscript and highlighted all changes in blue in the updated PDF.

Best regards,

Authors of Submission 32159

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.