acceptodds
Under review as a conference paper at ICLR 2027

Interactive Visual Gym: Evaluating visual assistants with simulated human users in interactive worlds

Abstract

Visual assistants that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual assistants through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as the Interactive Visual Evaluation (IVE) benchmark, comprising 315 interaction tasks. Evaluating state-of-the-art VLMs as visual assistants reveals substantial limitations: the best model, GPT6-Astra, completes only 44.2% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors in time. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction behavior.

Primary area:
datasets and benchmarks
Keywords:
Visual assistants, closed-loop evaluation, human_AI interaction, user simulation, vision-language models, multimodal benchmarks

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_13H4OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Interactive Visual Gym, a closed-loop testbed in which a simulated user performs a physical task in an executable environment while a visual assistant observes the user’s view and guides them verbally. Three domains are covered: cooking, household activities, and mobile-interface use. The authors build a persona-conditioned user simulator and a library of scheduled out-of-plan events that let them probe specific assistant capabilities when such events occur. The resulting IVE benchmark tests a list of models and even frontier models fall short on these interactive, visual assistance tasks.

Critical strengths
  1. Fills a real gap. The community lacks an interactive benchmark for embodied assistance across diverse tasks. Compared with text-only or digital agents, where executable environments and simulated users are relatively easier to obtain, high-quality physical simulators paired with credible user simulators remain scarce. To my knowledge this is the first work that puts a physical simulated environment, a behaviorally rich user simulator, and a visual assistant into a single closed-loop testbed.
  2. Thoughtful, comprehensive design. The scheduled out-of-plan events and the five persona axes cover many natural edge cases with impressive care, whereas prior work typically relies on three or four simplistic variants on both ends. Simulation time and wall-clock latency are properly separated, so streaming models can be tested in their native mode; ASR can be connected in, which makes the testbed even more realistic.
Critical weaknesses

The motivation for a closed-loop, simulator-driven setup is to move beyond fixed trajectories: the assistant should influence user behavior, the user alters the environment and observations, and those changes feed back to the assistant. Basically, user, assistant, and environment co-evolve. However, the paper lacks explicit analysis or evaluation on these dynamics once an out-of-plan event is triggered, nor how different models differ in handling the evolving interaction. The reported metrics capture some of it, but they do not characterize, e.g., how recovery unfolds after an error commits, how guidance quality changes under persona push-back across turns, or how an assistant’s behavior shifts as the user diverges from the plan. Evaluating these dynamic shifts is precisely where the core challenge lies, and the HCI literature on real-user interaction offers established paradigms the authors could draw on. This may be a reason to lower my score if this is not addressed to some extent, at minimum with analyses on existing traces.

Other weaknesses (not critical)

  1. In-time success entangles timeliness with correctness. “Any success” appears only in the user-model sensitivity table (Tab. 5) and not for the main body of the paper. Table 1 shows e.g., Gemini 3.8 Flash at 30.6% vs. Qwen3-Omni-30B at 26.2% on CookSim, while one cannot tell whether a model gives worse guidance or simply cannot finish within budget because it spends longer reasoning. It is recommended to report any-success (and ideally time-to-completion) alongside in-time success for all assistants in all three environments.
  2. LLM-as-a-judge reliability Several recent research has revealed that LLM-as-a-judge may not provide reliable, fine-grained judgement. It is recommended to have a small-scale human annotation of the rubric items and see whether they are aligned well. Also, binary scoring might not be appropriate, since there might be middle ground in between.

Comments

  1. Mobile interface. While mobile-interface use is important, ScreenSim sits loosely alongside cooking and household tasks: it has discrete gestures, no latency charging, and seems relatively simpler. I would consider dropping it in favor of deepening or extending the two embodied domains, though this is the authors' call.
  2. Scene-graph ablation. Since the ground-truth scene graph is available, an ablation giving the assistant partial or full environmental state would disentangle whether perception or task understanding is the bottleneck.
Questions for authors
  1. What was the reason for building CookSim from scratch rather than using existing platforms such as BEHAVIOR or CookBench? How extensible your platform is, not only for adding tasks or scenarios, but for supporting richer evaluation of interaction dynamics?
  2. What is the rationale of using Human-communicable action space instead of physical actions such as move, go to, pick up, etc. Can the verbal action always successfully translated into low-level physical actions?

Muzhen Cai et al. CookBench: A Long-Horizon Embodied Planning Benchmark for Complex Cooking Scenarios. arxiv. 2025

2 comments

Q1. How do recovery and guidance evolve after an out-of-plan event?

Thank you for highlighting this missing analysis. We now examine error recovery and multi-turn responses to user push-back, following the HCI emphasis on supporting correction and adapting to feedback (Amershi et al., 2019).

Models that prevent many mistakes can still struggle to repair committed errors. Qwen3-Omni and Muse Glimmer achieve relatively high prevention rates, but substantially lower error fix rates than Astra and Gemini 3.8 Flash.

The same assistant faces different interaction dynamics across user personas. For every model, objections persist more frequently with the expert, low-trust persona than with the novice, high-trust persona.

AssistantPrevention rate (%)Error fix rate (%)User push-back rate (%)Quality: before → after (%)Expert, low trust → Novice, high trust (%)
GPT-6 Astra31.342.97.042.9 → 67.26.8 → 0.0
Gemini 3.1 Pro20.450.034.37.3 → 34.534.4 → 26.6
Gemini 3.8 Flash28.953.311.423.6 → 49.430.8 → 13.3
GPT-Realtime 2.11.726.713.320.0 → 52.830.0 → 10.2
Gemini 3.8 Live0.90.012.322.5 → 58.722.2 → 6.2
Qwen3.8-27B27.933.318.117.4 → 35.540.7 → 9.1
Molmo2-8B11.25.628.76.6 → 13.741.4 → 32.5
InternVideo3-8B9.335.054.814.8 → 26.777.1 → 34.6
Kimi-VL-Thinking13.920.066.73.5 → 5.679.7 → 58.5
Muse Glimmer43.714.339.210.5 → 22.648.9 → 26.7
MiniCPM-o 4.538.212.547.19.4 → 17.559.4 → 33.8
Qwen3-Omni-30B43.112.537.110.8 → 25.045.2 → 22.5
Proact-VL2.616.728.611.9 → 20.938.9 → 24.9
StreamingVLM4.218.811.415.9 → 27.320.0 → 8.5

Q2. Can task correctness be separated from completion time?

We agree that correctness and timeliness should be reported separately. The table below adds any-success for every assistant: correct completion anywhere within the recorded rollout, including after the deadline. In-time success additionally requires completion within the budget.

AssistantCookSimVHSimScreenSim
GPT-6 Astra26.5 / 51.815.8 / 46.090.4 / 90.4
Gemini 3.1 Pro20.8 / 44.217.6 / 48.386.3 / 86.3
Gemini 3.8 Flash30.6 / 64.421.4 / 51.178.5 / 78.5
GPT-Realtime 2.121.2 / 37.419.0 / 47.766.7 / 66.7
Gemini 3.8 Live11.9 / 23.218.1 / 37.871.9 / 71.9
Qwen3.8-27B18.4 / 35.825.2 / 53.567.8 / 67.8
Molmo2-8B13.4 / 26.12.6 / 2.60.7 / 0.7
InternVideo3-8B14.5 / 28.718.2 / 33.70.4 / 0.4
Kimi-VL-Thinking15.2 / 30.724.3 / 35.29.6 / 9.6
Muse Glimmer23.8 / 43.615.7 / 36.044.1 / 44.1
MiniCPM-o 4.513.7 / 17.615.4 / 32.02.6 / 2.6
Qwen3-Omni-30B26.2 / 35.113.4 / 21.415.2 / 15.2
Proact-VL11.3 / 19.620.2 / 30.70.0 / 0.0
StreamingVLM10.9 / 19.419.0 / 43.50.0 / 0.0

Q3. How reliable are the judge scores, and why use binary rubrics?

The judge scores align with direct human ratings, beyond the preference-ranking agreement reported in the paper. Participants rated individual episodes for helpfulness and correctness; we compare those ratings with the corresponding judge scores below.

Judge score / human ratingEpisodesSpearman ρ [95% CI]
Interaction quality / helpfulness1890.50 [0.37, 0.61]
Factual grounding / correctness1890.64 [0.53, 0.74]
Actionable guidance / helpfulness1870.39 [0.26, 0.50]

We use binary rubric items because they express specific requirements that can be checked against the observed world state and interaction history. This follows CheckEval’s finding that decomposing evaluation into explicit criteria improves reliability and agreement with human judgments (Lee et al., 2025). Aggregating these judgments preserves differences between responses that satisfy different requirements, as in rubric-based evaluations such as HealthBench (Arora et al., 2025). We average across dimensions to avoid weighting them by their number of rubric items, and require factual grounding so that incorrect guidance cannot be compensated by presentation quality.

Q4. Why include ScreenSim alongside the embodied environments?

Thank you for the suggestion. In the camera-ready version, we will increase the difficulty of ScreenSim tasks to better match the other two simulators and improve setting consistency across different simulators.

Q5. Would access to scene-graph state clarify the bottleneck?

Access to explicit world state improves interaction quality, but does not improve task success in this ablation. Thank you for suggesting this comparison. We evaluate two assistants on ten matched CookSim cases with visual input alone or visual input plus a current-state readout, including object contents, preparation states, and cooking progress (40 episodes).

AssistantIn-time successInteraction qualityFactual grounding
GPT-6 Astra60.0 → 60.077.8 → 80.785.2 → 85.5
Gemini 3.8 Flash50.0 → 50.070.9 → 77.578.9 → 88.6

Each entry reports visual input → visual input plus state. We saw that explicit state improves factual grounding, particularly for Gemini 3.8 Flash, while task-level failures persist.

Q6. Why build CookSim from scratch, and how extensible is the simulator?

We built CookSim to provide a lightweight cooking simulator that is easy to deploy and modify. BEHAVIOR requires the OmniGibson/Isaac Sim stack and separately licensed assets, while CookBench depends on the commercial Cooking Simulator game. CookSim instead uses a Python grid-based engine and Three.js renderer, without a robotics physics stack or game license. Recipes and layouts reuse existing components; new tools or cooking processes extend object/station logic and corresponding visuals. These extensions can introduce new interaction situations, such as equipment failures requiring replanning, beyond adding task instances.

Q7. Why use human-communicable actions, and can they always be executed?

We use an abstract action space so the simulated user can plan and reason at the task level, reproducing human-like physical behavior as well as conversation. With this design, our human study observes similar task-execution patterns between simulated and real users. In contrast, prior work reports that most of VLMs struggle with long-horizon action sequencing over low-level action space (EmbodiedBench; VLM-TAMP). And yes in our case the action can always be executed when satisfying explicit syntax and preconditions. which is embedded as part of the user simulation prompt: for example, chopping requires empty hands and a raw ingredient on the facing board. When the pre-condition is satisfied, the engine executes the command through a rule-based controller.

Reviewer_snbVOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Interactive Visual Gym, a framework for closed-loop evaluation of visual assistants. Unlike offline video benchmarks, the framework simulates both the task environment and a human user, allowing assistant responses to influence subsequent user actions and task outcomes. It provides three executable environments spanning cooking, household manipulation, and mobile-interface interaction, along with persona-conditioned user simulation, controlled execution errors, and metrics for task success and interaction quality. The authors introduce the 315-episode Interactive Visual Evaluation benchmark, evaluate several frontier and open-weight VLMs, conduct a 45-participant human study, and demonstrate preliminary improvements from GRPO-based training. The best evaluated assistant achieves only 44.2% overall in-time task success.

Critical strengths
  1. This paper is a meaningful step towards evaluating assistance in closed loop. The paper tackles a real limitation of existing visual-assistance benchmarks: prerecorded trajectories cannot measure how guidance changes subsequent user behavior. Combining executable environments with simulated users is a substantial engineering and evaluation contribution.

  2. The evaluation covers a diverse collection of visual assistants, multiple task outcomes, interaction-quality metrics, and latency-sensitive behavior. I particularly appreciate the human study rather than relying exclusively on simulated users. The finding that task success and interaction quality do not always agree, along with the substantial degradation under measured latency, provides useful insights beyond model ranking itself.

Critical weaknesses

Human-simulation remains a significant concern for benchmark validity IMO. The human study shows that the simulator reproduces some aspects of user behavior, but several important dimensions differ substantially. Figure 4 reports similarities of just 0.05 for speech rate and responsiveness, and 0.0 for others. These are particularly consequential for evaluating proactive assistants, whose effectiveness depends heavily on when users respond, whether they heed warnings, and how they adapt their behavior. The reported agreement between interaction-quality metrics and human preferences is encouraging, but does not fully establish that model rankings or task-success estimates transfer to real users.

Additionally, the benchmark seems to be strongly shaped by the simulated-user policy and interaction protocol. Out-of-plan events are inserted at controlled states, user behavior is generated by an LLM-based harness, and success is evaluated within the resulting executable environments. This makes the benchmark reproducible, but also creates a risk that measured performance reflects simulator-specific behavior rather than general assistance quality.

Questions for authors
  1. How stable are assistant rankings under changes to the user simulator?
  2. For the GRPO experiment, does the learned improvement persist under a different simulated-user model or on scenarios with different error distributions?
1 comment

Q1. How well do simulated users reproduce behaviors relevant to assistance?

We thank the reviewer for highlighting this concern. We note that in fact our human study supports correspondence across most evaluated task-related behaviors, particularly physical task execution and responses to guidance. The comparisons below show how human and simulated behavior varies with user preferences (Tab. 11). These matching contrasts support our simulated users’ ability to reproduce preference-dependent changes in human task execution and responses to guidance.

Behavior / comparisonHumanSimulatedSimilarity
Action efficiency: cautious → faster0.737 → 0.7600.099 → 0.1150.82
Suggestion uptake: high → low trust32.3% → 24.3%32.9% → 20.1%0.77
Disagreement: high → low trust8.7% → 30.1%9.3% → 36.7%0.88
Guidance requests: stepwise → minimal interruptions47.9% → 20.8%44.1% → 17.9%0.98

We acknowledge that our simulated users speak and respond too frequently, and that alerting and parallelism match less well. These conversational gaps are a shared, unresolved challenge across current user-simulation frameworks, as documented by Seshadri et al. (2026) and Zhou et al. (2026). Nevertheless, such frameworks have helped drive progress across the agent community: e.g. τ-bench’s user simulation has similar conversational gap, but it is widely adopted and has been proven to successfully guide agent development in the past year.

Q2. Does the interaction protocol test general assistance quality?

We thank the reviewer for raising this concern. We do think our interaction protocol test general assistance quality for the following reasons:

  • Out-of-plan events reflect challenges that arise in human activity. Our error taxonomy follows studies of procedural mistakes, including preparation, timing, and ordering errors in CaptainCook4D (Peddi et al., 2024); eight of our error types also occur naturally in the human study. Questions and proactive requests test memory and temporal reasoning over ongoing activity, following the question types in OVO-Bench (Niu et al., 2025).

  • Our human study provides evidence beyond the simulated interaction protocol. In our human study, participants control their own actions and make naturally occurring mistakes, without injected events. In this setting, interaction-quality scores agree with human preferences in 78.5% of comparisons and correlate with participants’ helpfulness ratings (ρ = 0.50). This supports relevance to human assistance.

Q3. How stable are assistant rankings under changes to the user simulator?

Interaction-quality rankings remain consistent when switching the simulated-user model. Thank you for suggesting this direct comparison. We evaluate three assistants with Gemini 3.7 Flash and GPT-6 Astra as simulated users on the same 20 CookSim cases, holding personas and event schedules fixed (120 episodes).

Simulated userAssistantIn-time successAny successInteraction quality
Gemini 3.7 FlashGPT-6 Astra30.065.068.1
Gemini 3.7 FlashGemini 3.1 Pro30.045.032.8
Gemini 3.7 FlashGemini 3.8 Flash25.065.062.4
GPT-6 AstraGPT-6 Astra35.050.064.4
GPT-6 AstraGemini 3.1 Pro25.040.035.4
GPT-6 AstraGemini 3.8 Flash20.045.063.8

Interaction-quality rankings are identical (Kendall’s τ-b = 1.00), and any-success rankings largely agree (τ-b = 0.82). Within the rebuttal budget, we evaluate 20 cases; we will extend this comparison to the full benchmark for the camera-ready version.

Q4. Do RL gains persist with different users or error distributions?

The completed evaluation shows that interaction-quality gains extend beyond the training persona. We evaluate the novice-trained policy with persona type of expert users on 30 held-out cooking scenarios, with three runs per persona. We saw that Interaction quality improves for both two personas, similar to the claims in the paper. We will include experiments with more persona types and bigger evaluation suites in the revised version.

Evaluation personaSuccess: base → RLQuality: base → RL
Expert92.2% → 92.2%2.9% → 8.9%
Reviewer_jwaeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work designs Interactive Visual Gym, a framework for evaluating and training visual assistants. It introduces a VLM-based user simulator with a harness design. It shows its performance on an IVE benchmark.

Critical strengths

I think the problem is interesting and worth exploring. Designing and training visual assistants in the real world can be very expensive. A simulation environment could provide a scalable way to make this process more efficient. The design overall looks reasonable, and its evaluation also covers the major parts.

Critical weaknesses

My main concern is the validity of using VLMs as simulated users. While it looks like a natural way to do it, I would like to see a more rigorous way to evaluate its realism, either a performance comparison to real-world human behavior or other established evidence from the literature (e.g., an observation that the simulated users exhibit the same or similar patterns as those discovered in the existing literature). From its experiments, we can see that the simulated users diverge significantly from real humans. For example, simulated users speak much more (16.1 vs. 0.39 utterances per 100 ticks), reply to 79% of assistant utterances within ten ticks versus 2% for humans, and narrate their next action in 24% of utterances versus 1.3%. I would encourage the authors to keep diving in this direction and think about how to make the simulated users more realistic.

“Interaction quality reflects human preferences.” To me, evaluating it based on the ranking is a somewhat weak evaluation. Combining all dimensions into one result may not necessarily show the real effectiveness, as one low dimension could ruin it all. Also, a binary rubric for all the dimensions may not really reflect the usage of these aspects. Using an LLM as a judge remains uncertain in this part.

When using the five axes as user personas, what is the motivation behind them? Can all the axes be effectively reflected in the simulated human behavior? How can they be evaluated?

Other suggestions:

When RL is used to improve the assistant, the reward penalizes false flags and turn count, so the gain could come from talking less rather than from guiding better. Also, it forces the improvement on the same frozen scenarios. The authors may also consider evaluating the trained policy against a second user or, even better, with participants to show its improvement. Testing the trained policy on a different scenario could also be helpful to show its generalizability, as it is impossible to cover all the scenarios from the real-world in the simulator during training.

Questions for authors
  1. What evidence shows the VLM-simulated users behave like real users, beyond the reported divergence statistics?
  2. Why a binary rubric? How well does the LLM judge agree with human raters?
  3. What motivated the five axes, and does varying each one produce a measurable change in simulated behavior?
  4. How do you rule out that the gain comes from talking less rather than guiding better?
  5. Does the trained policy generalize to held-out scenarios, a second simulator, or human participants?
3 comments

Q1. What evidence shows the VLM-simulated users behave like real users, beyond the reported divergence statistics?

We thank the reviewer for this important question. Our human study supports correspondence across most of the evaluated task-related behavioral patterns, particularly the types of procedural errors that arise during physical task execution. Our simulated out-of-plan behaviors are also supported by studies of human procedural errors. We bring together the supporting evidence below.

1) Our simulated users reproduce most of the evaluated task-related behavioral patterns, especially those concerning physical task execution. We compare how task completion, action efficiency, and responses to guidance vary with users’ skill and preferences in the human study and simulation. Arrows indicate the comparison direction/

Behavioral measure / comparisonHumanSimulatedSimilarity
Completion: practice / skill50.0% → 79.3%25.0% → 100.0%0.56
Action efficiency: cautious → faster0.737 → 0.7600.099 → 0.1150.82
Suggestion uptake: high → low trust32.3% → 24.3%32.9% → 20.1%0.77
Disagreement: high → low trust8.7% → 30.1%9.3% → 36.7%0.88
Guidance requests: stepwise → minimal interruptions47.9% → 20.8%44.1% → 17.9%0.98
Ignored guidance: stepwise → minimal interruptions44.1% → 55.0%12.5% → 28.4%0.82

The matching trends in completion, efficiency, and responses to guidance support our claim that simulated users reproduce most evaluated task-related behaviors, particularly in physical task execution.

2) Our simulated users’ out-of-plan behaviors are grounded in studies of human procedural errors. In CookSim, our scheduled mistake taxonomy aligns with CaptainCook4D, which documents preparation, timing, missing-step, and ordering errors in recordings of people performing real-world cooking tasks. In VHSim, omissions, repeated steps, and wrong-object actions correspond to everyday action slips described by Mylopoulos (2022) and Norman (1981). In ScreenSim, changing the wrong setting while pursuing the intended goal is consistent with description errors in interface use(Norman, 1983).

3) The conversational limitations are a shared, unresolved challenge across current user-simulation frameworks. We acknowledge that our users speak and respond more frequently than humans; related gaps across LLM-based user simulators are actually well-known, documented by Seshadri et al. (2026) and Zhou et al. (2026). Nevertheless, such frameworks have helped drive progress across the agent community: e.g. τ-bench’s user simulation has similar conversational gap, but it is widely adopted and has been proven to successfully guide agent development in the past year.

Q2. Why a binary rubric? How well does the LLM judge agree with human raters?

We use binary rubric items because they express specific requirements that can be checked against the observed world state and interaction history. This follows CheckEval’s finding that decomposing evaluation into explicit criteria improves reliability and agreement with human judgments (Lee et al., 2025). Aggregating these judgments preserves differences between responses that satisfy different requirements, as in rubric-based evaluations such as HealthBench (Arora et al., 2025). We average across dimensions to avoid weighting them by their number of rubric items, and require factual grounding so that incorrect guidance cannot be compensated by presentation quality.

Our rubric scores also align with humans’ direct ratings of specific qualities of assistance. Beyond rankings, participants provided post-episode ratings of correctness and helpfulness on seven-point scales. We compare these with the existing judge scores over all episodes. The correlations below remain positive after accounting for individual rating tendencies, providing evidence beyond overall preference agreement.

Judge score ↔ human ratingSpearman ρ95% CI
Factual grounding ↔ correctness0.64[0.53, 0.74]
Actionable guidance ↔ helpfulness0.39[0.26, 0.50]
Overall interaction quality ↔ helpfulness0.50[0.37, 0.61]

Q3. What motivated the five axes, and does varying each one produce a measurable change in simulated behavior?

Our persona axes are grounded in HCI and behavioral research on how people perform tasks and respond to assistance. As described in Sec. 3.2, these cover skill level (Shneiderman, 2000), trust in AI (Lee and See, 2004), execution speed (Wickelgren, 1977), alerting preference (Meyer, 2004), and interruption preference (McFarlane, 2002). Tab. 11 in the paper evaluates the resulting behavioral differences against human participants; we summarize the supporting evidence in Q1 above.

Q4. How do you rule out that the gain comes from talking less rather than guiding better?

We examined held-out rollouts to distinguish changes in speaking frequency, response length, and guidance quality:

Additional CookSim evaluation: 30 held-out scenarios × three runs, with Gemini 3.8 Flash as the simulated user. Base model → training step 12.

MeasureBaseRL: step 12
Task success64.4%67.8%
Assistant turns / episode141.1146.6
Words / spoken utterance37.125.4
Interaction quality24.3%30.2%
Factual grounding43.0%52.2%
Actionable guidance35.9%38.8%

Interaction quality uses two judges. Overall scores average runs; category scores average conversational rounds.

After RL, factual grounding and actionable guidance improve, and the assistant produces more conversational turns with shorter utterances. The gains therefore go beyond simply reducing the number of turns.

Q5. Does the trained policy generalize to held-out scenarios, a second simulator, or human participants?

To examine dependence on the training persona, we additionally evaluate the novice-trained policy with persona type of expert users on held-out cooking scenarios: We also observe improvements in interaction quality with expert users. We will include experiments with more persona types and bigger evaluation suites in the revised version.

Evaluation personaSuccess: base → RLQuality: base → RL
Expert92.2% → 92.2%2.9% → 8.9%
(30 held-out scenarios × three runs per persona; Gemini 3.8 Flash users and a Gemini quality judge. Base model → training step 12)

Q6: Ethics concerns

Thank you for raising this. The 45 participants were recruited through an internal data-labeling team and personal referrals. Participation was voluntary. Participants received study instructions and indicated their agreement by choosing to proceed, without a separate signed consent form. Contractors participated during paid work time, and referred participants received no direct payment. Study outputs, including free-text exchanges, were stored under participant IDs in organization-managed storage. We can address the ethics-review status and provide further procedural details privately to the Area Chair to preserve double-blind review and participant privacy.

Thanks for addressing my concerns. Though I think the study could be stronger with more rigorous validation, I see the great potential of this direction. With that, I updated my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.