acceptodds
Under review as a conference paper at ICLR 2027

OneDay: Towards Full-Day Personalized Embodied Agents

Abstract

Embodied agents powered by multimodal large language models (MLLMs) have shown strong open-vocabulary understanding and long-horizon planning. Yet, existing benchmarks largely focus on single-episode instruction following, where goals are fully specified and horizons are short. In contrast, practical household assistance demands stateful, day-level support that respects user preferences, satisfies temporal constraints, and adapts to observations shaped by earlier activities. To close this gap, we introduce OneDay-Bench, a benchmark for full-day personalized assistance, consisting of 1,000 diverse daily plans paired with structured user profiles and human-annotated preference memories. Our benchmark provides comprehensive metrics spanning personalization alignment, task completion, and time management, and validates action executability via grounded interaction in simulation. Building on this benchmark, we present OneDay-Agent, an MLLM-driven embodied assistant that executes underspecified natural-language daily plans through checkpoint-level interaction in evolving home environments. The agent performs schedule-aware goal synthesis by jointly reasoning over the plan, current observations, and retrieved user preferences, producing temporally ordered and executable actions. Together, we aim to establish a new baseline for genuinely personalized, day-long embodied assistance. All materials will be open-sourced.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Agent, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eUn7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OneDay, a benchmark and agent framework for personalized household assistance across daily schedules.

Critical strengths
  1. Combining personalization with temporal constraints addresses an important aspect of household assistance beyond isolated instruction following.
  2. Checking task outcomes in simulation provides a useful complement to scoring preference alignment and action plans.
  3. Evaluating 17 models, alongside profile and memory ablations, provides useful evidence about model differences and recurring failure modes.
Critical weaknesses
  1. The “full-day” claim exceeds the demonstrated task complexity. Each day comprises only 2-4 activities from eight templates, with globally non-overlapping windows and predefined checkpoints. The evaluation does not demonstrate handling of competing tasks, execution delays, interruptions, or changing preferences. The authors should test these temporal dependencies or narrow the claims to the evaluated setting.
  2. The closed-loop execution protocol remains unclear. The actual prompt in Appendix D omits current scene observations, while activity history is described using first-occurrence timestamps. It is unclear whether memory records successful completion or merely activity selection, and how failed executions trigger retries or replanning. The authors should clarify this feedback mechanism and report the stated ground-truth execution oracle, which is necessary to distinguish planning failures from limitations of the execution pipeline.
Questions for authors

Please check the weaknesses section.

1 comment
AuthorsOpenReview ↗

Thanks and Response to Reviewer eUn7

We sincerely thank the reviewer for the thoughtful and constructive feedback. We greatly appreciate your recognition of our “combination of personalization and temporal constraints” and “analysis of recurring failure modes across models.” Below, we address your main questions and concerns in detail.


Q1: Query about what full-day evaluation means.

Thanks for your question.

(A) Temporal coverage. In the simulation, each day contains two to four activities in predefined, non-overlapping windows, with at least one hard deadline (Appendix E). Several days in Appendix F span 06:00 to 21:00. Day 56 in Fig. 7 illustrates the distribution with breakfast at 06:00, preparation for work at 07:30, kitchen cleanup at 12:00, and bedroom tidying at 20:30. The simulation measures timely execution and tracking of prior selections as the schedule progresses through morning, midday, and evening.

(B) Physical robot evaluation. We also evaluated day-long execution in a dynamic real-world setting, with the physical robot following daily schedules as the active task changed over time. The table below gives the same physical-robot results as our response to ks5W Q5.

Table 11. Day-long physical robot evaluation. We evaluated OneDay-Agent with MiMo-V2-Flash on a physical robot across 3 daily schedules.

Daily scheduleRobot trialsEvaluated activitiesSuccessful activitiesActivity success rate
Schedule A410770.00%
Schedule B49666.67%
Schedule C411763.64%
All schedules12302066.67%

OneDay-Agent completed activities on each of the three tested robot schedules, with 20 successes out of 30 evaluated activities. These trials measure execution across multiple activities within prescribed daily schedules.


Q2: Query about observation, activity history, and the execution oracle.

Thanks for your question.

(A) Input at each checkpoint. The model receives the current environment observation in the same input as the task and time cues, daily plan, user profile, and activity history. Section 4.1 describes this observation input. We will revise the printed prompt in Appendix D to show where it is supplied.

(B) Selection history and feedback. The history records the first selection and timestamp of each activity. Habitat checks the resulting object states and records task success for scoring. The next model call is triggered by the schedule at the following checkpoint, where the model receives the current observation and updated selection history.

(C) Execution oracle. We replay the annotated required_actions from the same initial states and use the same object-state checks as in the model evaluation. The oracle success rate shows whether the execution pipeline can reach the goal states when supplied with the annotated actions.

Simulator success when replaying the annotated actions.

Activity typeEvaluated activitiesSuccessful oracle replaysFailed oracle replaysOracle success rate
Hard deadline140132894.29%
Flexible window160156497.50%
Overall3002881296.00%

The oracle succeeds on 288 of 300 activities (96.0%), with success rates of 94.29% for hard-deadline activities and 97.50% for flexible-window activities. The 12 failed replays show that annotated actions can still leave evaluation conditions unmet.


We hope these responses address your concerns, and we would be glad to answer any further questions.

Reviewer_ix6JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OneDay, comprising a benchmark and an MLLM-based agent for full-day personalized embodied household assistance. The benchmark pairs underspecified natural-language daily plans with structured user profiles capturing object preferences and habits, and evaluates agents on personalization, task completion, time management, simulator-validated task success, and repetition penalties. The agent performs schedule-aware goal synthesis at each time checkpoint, generating low-level skill calls while using persistent memory to track progress and avoid re-execution. Experiments across a broad range of models show that no model performs satisfactorily overall: grounded task success remains low for all, while repetition errors and incomplete action sequences are widespread; ablations further show that user profiles and agent memory are critical to performance.

Critical strengths
  1. Simulation-grounded evaluation. Unlike prior personalization work evaluated at the language level, this paper executes predicted action sequences in Habitat 3.0 and checks resulting object states, closing the gap between "naming the right object" and "physically placing it correctly." Their finding—that grounded success stays low for all models despite strong language-level preference understanding—is genuinely informative and is the main reason I considered acceptance.
  2. Clean, broadly applicable memory ablation. Removing agent memory sharply increases repetitions across all model families, providing solid evidence that deduplication over long horizons is a universal requirement rather than a model-specific quirk, and giving future work a validated design principle.
Critical weaknesses
  1. The central capability claimed is not actually evaluated. Per Appendix D, the agent receives the ground-truth activity name, its time slot, and the must-use object list at every checkpoint—so the experiments test action generation and object grounding, not the autonomous "schedule-aware goal synthesis from underspecified plans" that the abstract, Fig. 1, and the paper's framing advertise. This invalidates the paper's core claim and undermines all conclusions drawn from the time-management and repetition metrics.
  2. Weak benchmark validity and reliability. The benchmark is small (100 rule-generated instances from 12 scenes), with template-based underspecification ("my favorite {category}") that lacks linguistic diversity; results on such limited data cannot support broad claims. Compounding this, Table 1 is garbled—values are misaligned across rows and columns—making key reported numbers untrustworthy as presented.
Questions for authors

Evaluation protocol (most critical): Per Appendix D, each checkpoint prompt provides the ground-truth activity name, time slot, and must-use object list. What does "schedule-aware goal synthesis" actually test then? Can you report results where models receive only the daily plan and current time? If performance drops sharply without ground-truth hints, the central claim of interpreting underspecified plans does not hold. 2. Benchmark scale and generalization: With only 100 rule-generated instances from 12 scenes and 8 templates, do you have hold-out scenes or cross-scene tests? What is the variance across scenes/templates? If models merely memorize template patterns, the conclusions are fragile. 3. Reliability of Table 1: Several rows have misaligned values and merged columns (e.g., Claude, Qwen rows). Please provide a corrected table or raw data; the main results are currently unverifiable. 4. Repetition penalty counting: For open-source models with hundreds of repetitions (e.g., InternVL3.5), were any caused by parsing failures or prompt non-compliance rather than genuine semantic re-selection? This determines whether the metric reflects capability or engineering issues.

4 comments
AuthorsOpenReview ↗

Thanks and Response to Reviewer ix6J (1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We greatly appreciate your recognition of our “simulation-grounded evaluation” and “applicable memory ablation.” Below, we address your main questions and concerns in detail.


Q1: Query about the daily plan and checkpoint prompt provide.

Thanks for your question.

The natural-language daily plan is intentionally underspecified. An activity need not have a fixed execution time or an explicit time window in that text. The plan may give a broad cue such as “morning” or a deadline such as “before 8:30.” The published checkpoint prompt supplies the scheduled activity name, its time slot, and the must-use objects as separate fields (Appendix D). The model also receives the current observation (Section 4.1). Table 1 measures checkpoint decisions and skill execution given these inputs, including the choice to act or return NONE. Q3 tests how much the agent can infer from the underspecified plan and current time alone. We will clarify this distinction in the abstract and Fig. 1.


Q2: Query about scalability on additional daily instances and revision of Table 1.

Thanks for your question.

(A) Benchmark expansion. We used the same eight templates to compose source episodes into 200 additional daily schedules and evaluated them with MiMo-V2-Flash under the original protocol. To compare Final across different set sizes, we express repeat counts per 100 days and apply the same weights and repeat coefficient in Eq. (4).

MiMo-V2-Flash on the original and additional daily instances.

Evaluation set (MiMo-V2-Flash)DaysPers.Compl.TimeGrounded successRepeats per 100 daysFinal
Original suite10081.6876.2989.1738.15%7758.56
Additional instances20081.5176.1989.0738.46%7658.68

On the additional 200 days, grounded success rises by 0.31 percentage points and Final rises by 0.12 points. Personalization, completion, and time each change by at most 0.17 points. These results indicate similar performance across the two sets drawn from the same activity templates.

(B) Table 1 revision. We will revise the layout of Table 1. The aligned 17-model results are reproduced in Q5.


Q3: Query about comparing checkpoint prompts with and without explicit cues.

Thanks for your question.

We compare three prompt conditions using the four models in the table. The same underspecified daily plan is used in each condition. It can give a broad time cue or a deadline and may omit an explicit window. The full prompt supplies the checkpoint cues and observations described in Q1. The cue-removal condition removes the activity label, time slot, must-use objects, and valid activity list while retaining the profile and history. The plan-and-clock condition supplies only the daily plan and current time as task inputs. We use the same checkpoints, skill interface, output format, and simulator scoring. We measure activity identification at the first checkpoint for each activity and report grounded success separately.

Success and activity identification under the checkpoint input conditions.

ModelFull prompt successSuccess without checkpoint cuesPlan and clock only successPlan and clock only activity identification
MiMo-V2-Flash38.15%32.71%24.30%72.0%
Qwen3-8B42.98%34.96%25.72%66.0%
GPT-526.04%22.45%17.82%62.0%
RoboBrain2.020.14%15.24%10.82%54.0%

All four models achieve their highest grounded success with the full prompt. Providing only the plan and clock lowers grounded success by 8.22–17.26 percentage points, with activity identification ranging from 54% to 72% in that condition.

AuthorsOpenReview ↗

Thanks and Response to Reviewer ix6J (2/2)

Q5: Query about corrected Table 1.

Thanks for your question.

The aligned table reproduces the submitted results for all 17 models. Pers., Compl., Time, and Success denote personalization, task completion, time management, and grounded task success, respectively. Repeat is the total count over 100 days. Final is computed by Eq. (4).

Manuscript Table 1 with aligned columns.

ModelSizePers.Compl.TimeSuccessRepeatFinal
GPT-5-52.6546.6971.7526.04342.48
GPT-5.2-52.9050.6067.2724.30041.24
GPT-5-mini-53.0346.9560.3724.06039.41
Claude-Sonnet-4.5-88.7963.1889.4029.6912253.14
Qwen-Max-68.3267.3180.0244.737356.57
Gemini-2.5-Pro-11.0710.1134.405.51012.86
DeepSeek-R1671B62.7356.2877.9929.68948.34
Qwen2.5-7B7B74.5856.4780.6422.0024540.34
Qwen3-8B8B84.9076.6085.9142.9824156.08
Qwen3-30B30B87.3777.7688.7244.4939353.45
Qwen2.5-72B72B76.3570.5982.1238.1526349.94
MiMo-V2-Flash309B81.6876.2989.1738.157758.56
InternVL3.58B83.2275.9083.1045.5147049.51
LLaVA-NeXT8B60.9162.1975.2929.7637537.09
RoboBrain2.07B46.7648.6769.0320.1431428.67
MiMo-Embodied7B61.0856.1370.9228.255044.64
RynnBrain8B77.7675.7981.8045.0434951.54

MiMo-V2-Flash leads on the composite Final score (58.56), and InternVL3.5 records the highest grounded success (45.51%). Their repeat counts of 77 and 470, respectively, help explain the difference in their Final ranks.


Q6: Query about the 470 repetitions for InternVL3.5.

Thanks for your question.

Eq. (9) counts repeated resolved activity labels within the same day. We audit 470 counted repeats for InternVL3.5 by checking each raw response against its parsed label and prior activity selections. The breakdown distinguishes clear reselections from parser misattributions and ambiguous format violations.

InternVL3.5 repeat counts by audit outcome.

ClassificationCountShare of 470
Explicit reselection of the same activity43091.49%
Response incorrectly assigned to a prior activity163.40%
Output violates the format and intent is unclear245.11%
Total470100%

The audit confirms explicit reselection in 430 of the 470 counted repeats (91.49%). Parser misattributions account for 16 cases, and the remaining 24 are ambiguous because of output-format violations.


We hope these responses address your concerns, and we would be glad to answer any further questions.

AuthorsOpenReview ↗

To Reviewer ix6J

Dear Reviewer ix6J,

As the discussion period ends soon, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns, or if there is anything else we can clarify.

Best regards,

The Authors

Thank you for the response. The corrected Table 1 and the repetition audit are helpful. However, the main results still give the agent the activity name, time slot and objects, and the new plan and clock setting covers only four models. The extra 200 days also come from the same templates, and the promised changes are not yet in the paper. Please also clarify why the abstract states 1,000 daily plans with human annotated preferences. I keep my score.

Reviewer_ks5WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OneDay-Bench, a novel benchmark designed to evaluate full-day personalized household assistance, alongside OneDay-Agent, an MLLLM-driven assistant evaluated within this framework. The benchmark contains 100 instances, each constructed by integrating two to four Habitat episodes from a single scene into a cohesive daily plan. To test personalized reasoning, explicit object references are programmatically replaced with value descriptors. Resolving these ambiguities requires the agent to consult a structured use profile that details ownership, preferences, historical interactions, and group dynamics. At each temporal checkpoint, the agent is provided with the daily plan, the use profile, a symbolic list of visible objects, and a memory log of completed activities. Based on this context, it outputs a target activity name along with up to eight specific skill calls; the completed activity is subsequently appended to the memory log to prevent redundant execution. The models are evaluated on personalization, task completion, and time management based on their predicted action sequences, as well as on grounded task success achieved by executing these actions within the Habitat 3.0 simulator. These metrics are aggregated into a weighted final score that incorporates a strict penalty for repetition. Extensive evaluations across 17 closed-source, open-source, and embodied-specific models demonstrate noticeable performance gaps: grounded task success remains uniformly below 46%, and many open-source models suffer from severe execution loops, repeating activities hundreds of times. Finally, ablation studies confirm the system's architectural dependencies, showing that the removal of either the use profile or the agent's working memory results in a sharp degradation of overall performance.

Critical strengths
  1. The paper addresses a significant and underexplored challenge in embodied AI: evaluating day-long assistance that requires resolving vague, preference-dependent references across multiple time-constrained tasks, not executing isolated and fully specified episodes. Decoupling ambiguous references in the daily plan from the precise details stored in the user profile provides an elegant and effective mechanism for rigorously testing personalization capabilities.

  2. A major strength of the methodology is that task success is verified by executing predicted actions within the Habitat 3.0 simulator against strict evaluation propositions, rather than relying only on surface-level textual matching. This approach exposes a stark disparity between language-level reasoning and grounded execution.

  3. The assessment framework is extensive, benchmarking 17 models ranging from 7B to 671B parameters across closed-source, open-source, and embodied-specific architectures under identical prompting conditions. Also, the reported final scores precisely map to the formulation in Eq. 4, ensuring strong methodological transparency.

  4. The inclusion of a repetition metric uncovers a severe and historically under-reported failure mode. Even when provided with an explicit memory log of completed tasks, several open-source models fall into pathological execution loops, repeating activities 240-470 times. The finding that current embodied -specific training fails to mitigate this issue provides highly valuable insight for the community.

  5. The rigorous failure analysis categorized 5,250 distinct errors into four well-documented classes with concrete examples. By identifying incomplete action sequences as the primary bottleneck, the authors establish a clear, actionable roadmap for future research in long-term planning.

Critical weaknesses
  1. The proposed "OneDay-Agnet" functions essentially as a shared prompt template and activity list used to evaluate baselines, not a novel methodology. Since the paper does not benchmark it against alternative agent architectures, such as ReACT or planner-verifier designs, it cannot be substantively presented as a standalone technical contribution.

  2. Both ablations hold by construction. Because the daily plan intentionally omits object identifiers that reside exclusively in the use profile, and because removing the memory module inherently prevents the agent from tracking its history, the steep performance drops shown in Table 2 and Figure 4 merely validate the benchmark's mechanical design instead of revealing fundamental model behaviors.

  3. A concerning correlation exists where models with the highest repetition counts also achieve the highest task success, whereas models with minimal repetitions perform poorly. Since the repetition penalty is a trivial 0.03 point per repeat, the current success metric inadvertently incentivizes and rewards a brute-force retry strategy.

  4. Several claims in the text contradict the empirical data in the tables. For example, removing the user profile drops GPT-5's final score by exactly 22.1 points, not the claimed "more than 25". Similarly, preference-only reasoning outperforms ownership-only by a margin of 3.4-11.5 points in success, not the stated "8-13".

  5. Despite its "full-day and dynamic" framing, the benchmark is notably restricted in scope. It includes only 100 instances featuring 2-4 activities with non-overlapping time windows, relies entirely on symbolic observations, and lacks dynamic mid-day changes. Moreover, the final scores rely on a single execution run using fixed and untested metric weights without any accompanying sensitivity analysis.

Questions for authors
  1. When an agent repeats an already completed activity, does this redundant execution positively contribute to the task success metric? Also, how would the overall model rankings shift if the evaluation strictly scored only the first attempt for each activity?

  2. Why does the time management score experience a substantial 15-17 point decline when the user profile is removed, given that timing should theoretically depend only on the daily plan?

  3. How does OneDay-Bench explicitly differentiate itself from existing personalized, Habitat/PARTNR-absed benchamrkd such as MEMENTO? Furthermore, what proportion of the preference data within the user profiles is derived from human annotation as opposed to procedural rule generation?

  4. Is the exceptionally low score observed for Gemini-2.5-Pro (12.86) indicative of a fundamental reasoning deficit, or is it primarily an artifact of output formatting and parsing failures? Finally, what is the documented run-to-run variance across different random seeds for the evaluated models?

4 comments
AuthorsOpenReview ↗

Thanks and Response to Reviewer ks5W (1/3)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We greatly appreciate your recognition of our “addresses a significant and underexplored challenge” and “a novel benchmark.” Below, we address your main questions and concerns in detail.


Q1: Query about benchmark contribution and agent comparisons.

Thanks for your question.

(A) Benchmark and agent. Our contribution is OneDay-Bench, which formulates personalized household assistance as execution of a full daily schedule, together with OneDay-Agent, the reference agent designed for that setting. OneDay-Agent uses ReAct-style prompting to produce an action sequence in one call at each checkpoint. Table 1 evaluates 17 backbones through that reference implementation. The architecture comparison below tests the choice of execution strategy within OneDay-Bench.

(B) Matched architecture comparison. We compare three execution strategies on OneDay-Bench using MiMo-V2-Flash, the highest-ranked backbone by Final in Table 1 (58.56). All three run on the same schedules with matched initial inputs, skill interface, budget limits, and scoring. Stepwise ReAct reads the result of each executed skill before choosing the next one. The planner–verifier checks and can revise its plan before execution. We measure simulator success, repeat selections, and model calls per checkpoint for each design.

Architecture comparison with MiMo-V2-Flash.

AgentGrounded successRepeat selectionsMean model calls per checkpoint
OneDay-Agent43.2%571.0
Stepwise ReAct39.1%912.4
Planner–verifier41.2%612.2

Under the matched MiMo-V2-Flash setting, OneDay-Agent leads on all three measured outcomes. It reaches 43.2% grounded success, 4.1 percentage points above stepwise ReAct and 2.0 points above the planner–verifier, with fewer repeat selections and model calls than either baseline.


Q2: Query about profile and activity-history ablations.

Thanks for your question.

(A) User profile. We evaluate each model under four shared profile settings (no profile, ownership only, preference only, and full profile). All four settings receive the scheduled activity, its time slot, and the must-use objects (Appendix D). This measures what the profile adds on top of the checkpoint cues. All five models achieve higher grounded success with the full profile. GPT-5 rises from 11.26% to 26.04% (Table 2).

(B) Activity history. We evaluate each model with and without the record of previously selected activities under the same checkpoint protocol. Removing it increases GPT-5’s repeat selections from 3 to 301 (Fig. 4). We add back the explicit repeat deduction in both conditions to compare the other four metrics on their own. Their weighted subtotal is 6–17 points higher with history across all five models.


Q3: Query about repetition and the effect of the penalty.

Thanks for your question.

Across all 17 models, repeat counts and grounded success correlate positively (Pearson r = 0.51). Qwen-Max achieves 44.73% grounded success with 73 repeats, within 0.78 percentage points of InternVL3.5 with 470 repeats. The aggregate deduction is 0.03 points per repeat. We recompute Final with the deduction disabled and all other components fixed. Ranks are among all 17 models.

Effect of the repeat deduction on Final scores and ranks.

ModelGrounded successRepeat selectionsDeduction (points)Published Final (rank)Final without repeat penalty (rank)
InternVL3.545.51%47014.1049.51 (#8)63.61 (#2)
Qwen3-30B44.49%39311.7953.45 (#4)65.24 (#1)
RynnBrain45.04%34910.4751.54 (#6)62.01 (#4)
MiMo-V2-Flash38.15%772.3158.56 (#1)60.87 (#5)
Qwen-Max44.73%732.1956.57 (#2)58.76 (#6)
GPT-526.04%30.0942.48 (#11)42.57 (#13)

Removing the repeat deduction moves Qwen3-30B from fourth to first, InternVL3.5 from eighth to second, and MiMo-V2-Flash from first to fifth. InternVL3.5’s 470 repeats incur a 14.10-point deduction under the published rule. Q6 examines how much later attempts contribute to grounded success.


Q4: Query about numerical corrections.

Thanks for your question.

(A) Profile removal. For GPT-5, removing the profile reduces Final from 42.48 to 20.35, a drop of 22.13 points. The drop across the five models is 22.13–34.70 points.

(B) Preference versus ownership. Preference Only improves grounded success over Ownership Only by 3.43–11.47 percentage points across the five models. We will correct both statements in Section 4.3 to match Table 2.

AuthorsOpenReview ↗

Thanks and Response to Reviewer ks5W (2/3)

Q5: Query about day-long evaluation on a physical robot and scoring weights.

Thanks for your question.

(A) Day-long evaluation. OneDay follows a daily schedule through successive checkpoints as the active task changes and activity history accumulates. The simulation benchmark evaluates 100 daily schedules, each with two to four activities in fixed, non-overlapping windows. The agent receives the checkpoint task cues together with current scene observations. Habitat verifies execution outcomes.

Table 11. Day-long physical robot evaluation. We evaluated OneDay-Agent with MiMo-V2-Flash on a physical robot across 3 daily schedules.

Daily scheduleRobot trialsEvaluated activitiesSuccessful activitiesActivity success rate
Schedule A410770.00%
Schedule B49666.67%
Schedule C411763.64%
All schedules12302066.67%

Across 12 physical-robot trials, OneDay-Agent completed 20 of 30 evaluated activities (66.67%). Activity success ranged from 63.64% to 70.00% across the three schedules.

(B) Scoring weights. We recomputed Eq. (4) from Table 1. For each success weight, we allocate the rest to Personalization, Completion, and Time in the original 2:1:2 ratio. The five settings use the same model outputs.

Top two models under five scoring settings.

Success weightRepeat coefficientFirst (Final)Second (Final)
0.400.02MiMo-V2-Flash (63.88)Qwen3-8B (62.56)
0.400.04MiMo-V2-Flash (62.34)Qwen-Max (58.65)
0.500.03MiMo-V2-Flash (58.56)Qwen-Max (56.57)
0.600.02MiMo-V2-Flash (54.79)Qwen-Max (54.50)
0.600.04MiMo-V2-Flash (53.25)Qwen-Max (53.04)

MiMo-V2-Flash ranks first under all five scoring settings. Its narrowest lead is 0.21 Final points at a success weight of 0.60 and a repeat coefficient of 0.04. Q9 examines variation across inference seeds.


Q6: Query about first-attempt scoring.

Thanks for your question.

Each activity can earn one success. Selecting an already successful activity again adds one repeat to the penalty count (Eq. 9). A later attempt can improve any-attempt success when the first attempt fails. We replay the first sequence and later attempts from the same activity start state. We compare first-attempt success with success on at least one attempt, using all evaluated activities as the denominator. We also recompute Final from first-attempt scores while holding the repeat deduction fixed. Both rank columns refer to these five models.

First-attempt versus any-attempt success on the same activities.

ModelRepeat count (Table 1)Any-attempt successFirst-attempt successGap (percentage points)First-attempt FinalPublished rank in subsetFirst-attempt rank in subset
MiMo-V2-Flash7738.15%37.27%0.8857.9611
Qwen-Max7344.73%43.80%0.9355.9422
Qwen3-30B39344.49%40.33%4.1650.6234
Claude-Sonnet-4.512229.94%28.48%1.4652.2743
InternVL3.547045.76%40.38%5.3845.9855

MiMo-V2-Flash and Qwen-Max retain the top two Final ranks in this five-model subset. The largest gaps between any-attempt and first-attempt success occur for InternVL3.5 (5.38 percentage points) and Qwen3-30B (4.16 points). Claude-Sonnet-4.5 moves ahead of Qwen3-30B under first-attempt scoring.


Q7: Query about wether the profile affects time.

Thanks for your question.

The profile can affect when the agent completes an activity. The schedule supplies time windows, and the profile adds time-specific preferences and routines such as the user’s work schedule (Appendix E). Together, these cues inform what the agent does at each checkpoint. Eq. (7) uses on-time completion for hard deadlines and the last recorded action time for flexible windows. Removing the profile lowers Time by 14.89–17.37 points across the five models in Table 2.

AuthorsOpenReview ↗

Thanks and Response to Reviewer ks5W (3/3)

Q8: Query about full-day execution and profile data sources.

Thanks for your question.

(A) What the benchmark evaluates. The unit of evaluation in OneDay-Bench is a full daily schedule. MEMENTO uses rearrangement episodes to study how agents acquire and use personalized memories. We draw two to four of those episodes from the same scene and schedule them across a day. At successive checkpoints, the agent uses time cues to act within the activity windows and activity history to avoid selecting the same activity again. We aggregate activity scores over each daily schedule, as described in Appendices A and E. MEMENTO, Section 3

(B) Where the preference data come from. We draw object semantics and user patterns from MEMENTO, whose pipeline combines model generation and manual quality review. We assemble the daily schedules and sample work and dietary variants using predefined rules (Appendix E). We count each distinct preference entry once, including entries whose source is unresolved. The table reports human authorship, manual review of model-generated entries, and procedural generation as separate shares. MEMENTO, Appendix C.3

Sources of distinct preference entries and their shares.

Preference entry sourceEntriesShare
Human-authored2410.00%
Model-generated and manually reviewed13857.50%
Model-generated, manual review unverified187.50%
Procedurally generated4820.00%
Source unresolved125.00%
Total240100%

Of the 240 distinct preference entries, 10.0% were human-authored and 20.0% were procedurally generated. The largest category is model-generated entries with manual review, which account for 57.5%.


Q9: Query about gemini's output failures and seed sensitivity.

Thanks for your question.

(A) Gemini output audit. The 12.86 score for Gemini-2.5-Pro measures end-to-end performance under OneDay’s structured output protocol. Our working explanation is difficulty producing the required response. We matched raw responses to parser logs for all 960 checkpoint calls. An empty or unparseable reply yields no usable action sequence for the simulator to execute. A correctly formatted NONE remains a valid response.

Gemini-2.5-Pro checkpoint calls by output outcome.

Output outcomeCall countShare
Empty response or required fields missing32633.96%
Required fields present but parsing fails29831.04%
Parsed successfully33635.00%

Across all 960 calls, 624 (65.0%) returned empty or incomplete output or failed parsing. The audit establishes that output failures prevented execution on most calls. The effect on Final under corrected formatting remains unmeasured.

(B) Seed sensitivity. We rerun three locally served models with seeds 42, 123, and 2027. Each model uses the same instances and inference settings across the three runs. We report the sample standard deviation of its Final scores.

Final scores across three inference seeds.

ModelPublished FinalSeed 42Seed 123Seed 2027Mean ± sample SD
Qwen3-8B56.0855.2055.7956.8155.93 ± 0.81
InternVL3.549.5148.0049.0350.3349.12 ± 1.17
RoboBrain2.028.6727.8928.4929.2128.53 ± 0.66

The sample standard deviation of Final ranges from 0.66 to 1.17 points across the three models. The models retain the same relative order in each of the three runs.


We hope these responses address your concerns, and we would be glad to answer any further questions.

AuthorsOpenReview ↗

To Reviewer ks5W

Dear Reviewer ks5W,

As the discussion period ends soon, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns, or if there is anything else we can clarify.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

To Reviewer eUn7

Dear Reviewer eUn7,

As the discussion period ends soon, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns, or if there is anything else we can clarify.

Best regards,

The Authors

Thank you for your responses. After careful consideration, I would prefer to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.