- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces OneDay, a benchmark and agent framework for personalized household assistance across daily schedules.
Critical strengths
- Combining personalization with temporal constraints addresses an important aspect of household assistance beyond isolated instruction following.
- Checking task outcomes in simulation provides a useful complement to scoring preference alignment and action plans.
- Evaluating 17 models, alongside profile and memory ablations, provides useful evidence about model differences and recurring failure modes.
Critical weaknesses
- The “full-day” claim exceeds the demonstrated task complexity. Each day comprises only 2-4 activities from eight templates, with globally non-overlapping windows and predefined checkpoints. The evaluation does not demonstrate handling of competing tasks, execution delays, interruptions, or changing preferences. The authors should test these temporal dependencies or narrow the claims to the evaluated setting.
- The closed-loop execution protocol remains unclear. The actual prompt in Appendix D omits current scene observations, while activity history is described using first-occurrence timestamps. It is unclear whether memory records successful completion or merely activity selection, and how failed executions trigger retries or replanning. The authors should clarify this feedback mechanism and report the stated ground-truth execution oracle, which is necessary to distinguish planning failures from limitations of the execution pipeline.
Questions for authors
Please check the weaknesses section.
Thanks and Response to Reviewer eUn7
We sincerely thank the reviewer for the thoughtful and constructive feedback. We greatly appreciate your recognition of our “combination of personalization and temporal constraints” and “analysis of recurring failure modes across models.” Below, we address your main questions and concerns in detail.
Thanks for your question.
(A) Temporal coverage. In the simulation, each day contains two to four activities in predefined, non-overlapping windows, with at least one hard deadline (Appendix E). Several days in Appendix F span 06:00 to 21:00. Day 56 in Fig. 7 illustrates the distribution with breakfast at 06:00, preparation for work at 07:30, kitchen cleanup at 12:00, and bedroom tidying at 20:30. The simulation measures timely execution and tracking of prior selections as the schedule progresses through morning, midday, and evening.
(B) Physical robot evaluation. We also evaluated day-long execution in a dynamic real-world setting, with the physical robot following daily schedules as the active task changed over time. The table below gives the same physical-robot results as our response to ks5W Q5.
Table 11. Day-long physical robot evaluation. We evaluated OneDay-Agent with MiMo-V2-Flash on a physical robot across 3 daily schedules.
OneDay-Agent completed activities on each of the three tested robot schedules, with 20 successes out of 30 evaluated activities. These trials measure execution across multiple activities within prescribed daily schedules.
Thanks for your question.
(A) Input at each checkpoint. The model receives the current environment observation in the same input as the task and time cues, daily plan, user profile, and activity history. Section 4.1 describes this observation input. We will revise the printed prompt in Appendix D to show where it is supplied.
(B) Selection history and feedback. The history records the first selection and timestamp of each activity. Habitat checks the resulting object states and records task success for scoring. The next model call is triggered by the schedule at the following checkpoint, where the model receives the current observation and updated selection history.
(C) Execution oracle. We replay the annotated
required_actionsfrom the same initial states and use the same object-state checks as in the model evaluation. The oracle success rate shows whether the execution pipeline can reach the goal states when supplied with the annotated actions.Simulator success when replaying the annotated actions.
The oracle succeeds on 288 of 300 activities (96.0%), with success rates of 94.29% for hard-deadline activities and 97.50% for flexible-window activities. The 12 failed replays show that annotated actions can still leave evaluation conditions unmet.
We hope these responses address your concerns, and we would be glad to answer any further questions.