- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces TALK-Dem, a benchmark for evaluating embodied task planning under dementia-associated communication patterns. Building on REI-Bench tasks, it constructs 4,800 instructions across 300 household tasks, covering five clinically motivated communication patterns at three intensity levels alongside clean controls. It also proposes CARE, a retrieval-based method that supplies relevant task interpretations and reference plans as demonstrations. The authors evaluate eight language models using SayCan in AI2-THOR, compare the synthetic instructions with DementiaBank linguistic features, and analyze planning failures and safety-relevant actions. The results show that these communication variations can degrade planning performance and that CARE generally improves task success over the evaluated prompting baselines.
Critical strengths
- The paper provides a benchmark and useful research directions for studying embodied intelligence under dementia-associated communication patterns, supporting research toward more inclusive assistive systems.
- The paper adapts retrieval-augmented generation to this setting and demonstrates its effectiveness on the proposed benchmark, providing a practical approach to improving task-planning robustness.
Critical weaknesses
- Despite the DementiaBank feature comparisons, it remains unclear whether the synthetic instructions and intensity levels reflect real patient interactions, limiting confidence in their real-world applicability.
- Stronger models already perform well on the benchmark, while CARE’s main gains are demonstrated on older open-weight models. Its benefits for newer models with comparable deployment costs remain unclear.
Questions for authors
- Beyond the linguistic feature comparisons with DementiaBank, is there evidence from real interaction data or expert assessments supporting the realism of the synthetic expressions and the three intensity levels?
- Have you evaluated newer open-weight models with comparable inference costs?
Response to Reviewer r4XQ
Thank you for your constructive feedback. We have clarified the scope and limitation of our synthetic benchmark and added evaluations of newer open-weight models. All changes are highlighted in the revised PDF. Below is our detailed point-by-point response to your questions.
W1 and Q1. Do synthetic instructions and intensity levels reflect real-world dementia-associated communication?
We would like to clarify that TALK-Dem is intended as a controlled benchmark of dementia-associated communication patterns, rather than a simulation of real conversations with people living with dementia. Each pattern is grounded in clinical literature, and our comparison with DementiaBank shows that the generated instructions exhibit relevant linguistic characteristics (Table A1). The three intensity levels represent increasing degrees of each communication pattern, not clinical stages of dementia. The mapping between language patterns and dementia severity is a complicated medical topic beyond the scope of the current work. The controlled design allows us to isolate the effects of individual patterns on embodied planning. We acknowledge the limitations of synthetic instructions and the need for real-world validation. We have clarified the intensity levels in Section 3.1 and the limitations and need for real-world validation in Section 7.
W2 and Q2. Does CARE remain effective on newer open-weight models?
Following the reviewer’s suggestion, we additionally evaluated two newer open-weight models of comparable size, Qwen3-8B and gemma-4-E4B-it, using the same SayCan framework and 16 instruction conditions. CARE consistently improves task success and robustness to communication variations on both models.
CARE improves average patterned-task success by 21.8 percentage points on Qwen3-8B and 11.9 points on gemma-4-E4B-it. Both models also show improved retention, suggesting that CARE's benefits extend to newer open-weight models. We have added the full results to Appendix B and updated the discussion in Section 5.3. We compared CARE with Vanilla methods, which is achievable within the rebuttal period.