- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies proactive interaction decisions for multimodal assistants in livestreaming environments. The authors formulate the task as a mixed-initiative decision problem with three actions: OBS (observe), MEM (update private memory), and ANS (respond proactively). They construct structured training trajectories and a human-reviewed evaluation set from livestream audio and video, comments, and platform signals, and propose a two-stage training strategy consisting of MA-MSFT and SM-GSPO. On the proposed benchmark, the complete method achieves an Overall Average of 67.41, compared with 61.34 for ordinary multiturn SFT and 52.66 for the strongest evaluated general-purpose multimodal model. Ablations of the training stages, input modalities, semantic memory, and context length examine the contributions of individual components.
Critical strengths
- The authors construct a structured livestream-assistance benchmark that covers multimodal inputs, proactive response timing, memory updates, recipient selection, and task decisions. It provides a relatively comprehensive experimental environment for jointly evaluating these capabilities.
- The authors propose MA-MSFT to address insufficient supervision of structured decision markers and SM-GSPO to address the propagation of earlier errors during closed-loop operation. Experiments show that the complete training strategy outperforms ordinary multiturn SFT and the evaluated training variants on the proposed benchmark, providing evidence of its effectiveness in the current task setting.
Critical weaknesses
The necessity and generality of the two-stage training strategy have not been sufficiently demonstrated. Although MA-MSFT + SM-GSPO improves over ordinary SFT, the paper does not directly compare it with a conventional SFT + RL pipeline. It is therefore unclear how much additional benefit the proposed training designs provide.
Questions for authors
- Compared with existing benchmarks for streaming multimodal understanding, proactive responses, and multiparty interaction, which previously difficult-to-evaluate capabilities does LiveAssistant's joint decision setting measure, or which new model failure modes does it reveal? Could the authors also explain why jointly evaluating Whether, When, Whom, and What is necessary beyond simply broadening task coverage?
- Table 2 compares ordinary SFT, MA-MSFT, and RL strategies initialized from MA-MSFT, but does not report ordinary SFT followed by standard RL, such as conventional GSPO. Could the authors add this direct control to establish how much additional benefit MA-MSFT and SM-GSPO provide over a conventional SFT + RL pipeline?
- Could the authors clarify whether MA-MSFT and SM-GSPO are applicable to continuous multiturn decision tasks beyond LiveAssistant? In particular, are the problems they target—insufficient supervision of structured decisions and propagation of earlier errors—general beyond the current benchmark and its specific action protocol? Could the authors provide cross-task experiments or other supporting evidence?
- In Table 1, Always-ANS has zero OBS recall, zero MEM recall, and zero Host routing accuracy, yet achieves an Overall Average of 55.67, exceeding all general-purpose models. How do the authors explain this result? Could the current aggregate metric excessively reward frequent classes and therefore fail to adequately reflect selective intervention and multiparty interaction decisions? Would the relative model rankings remain stable under a class-balanced evaluation?
4 comments
Official Comment from Authors to Reviewer KC7b - Part 2
W / Q2. Contribution of the two-stage training strategy: How much do MA-MSFT and SM-GSPO add beyond ordinary SFT followed by conventional RL?
1. Direct pipeline comparison and component contributions. We added ordinary multiturn SFT + conventional turn-level GSPO, using equally weighted reward components, no state shaping, and no trajectory-level reward. This baseline improves Avg. from 61.34 to 61.93, whereas the full pipeline reaches 67.41, a 5.48-point advantage. The existing comparisons further identify a 1.67-point gain from ordinary multiturn SFT to MA-MSFT and a 2.99-point gain from turn-level GSPO to SM-GSPO under the same MA-MSFT initialization and turn-level reward, including +3.86 Recipient and +3.95 Task points. Together, these results show gains beyond simply adding RL. The conventional-to-proposed pipeline comparison changes both initialization and reward design; to separate them, we will additionally report ordinary multiturn SFT followed by GSPO with the original reward configuration.
| Method | State ↑ | Recipient ↑ | Task ↑ | Avg. ↑ |
|---|---|---|---|---|
| Ordinary multiturn SFT | 69.36 | 63.54 | 51.12 | 61.34 |
| Ordinary multiturn SFT + conventional turn-level GSPO | 69.58 | 64.31 | 51.89 | 61.93 |
| MA-MSFT | 70.05 | 66.42 | 52.55 | 63.01 |
| MA-MSFT + turn-level GSPO, original reward configuration | 69.98 | 68.81 | 54.46 | 64.42 |
| MA-MSFT + SM-GSPO | 71.14 | 72.67 | 58.41 | 67.41 |
2. Why multiturn training matters. Ordinary SFT in the original paper is already multiturn SFT. To examine the effect of removing the multiturn training formulation, we additionally evaluated single-turn SFT and subsequent single-turn RL, without trajectory-level reward. State and Recipient accuracy follow the overall definitions in Table 2; the state and role columns report per-class recall.
| Method | State Acc. ↑ | OBS ↑ | MEM ↑ | ANS ↑ | Recipient Acc. ↑ | Viewer ↑ | Host ↑ | Task ↑ | Content ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Single-turn SFT | 47.06 | 84.16 | 0.00 | 25.17 | 25.92 | 30.13 | 0.00 | 13.20 | 53.21 |
| + single-turn RL | 47.78 | 75.80 | 11.14 | 32.32 | 31.80 | 36.35 | 3.81 | 18.41 | 57.93 |
Both settings remain well below ordinary multiturn SFT in overall State and Recipient accuracy (69.36/63.54). Single-turn RL improves ANS recall, Task accuracy, and Content score, but OBS recall decreases, and MEM/Host recall remains only 11.14%/3.81%. Retaining information and deciding when to escalate to the host depend strongly on preceding interactions, supporting multiturn training in this setting. Thank you for requesting controls that distinguish these training choices.
Q3. Applicability beyond LiveAssistant: Do the methods address general problems in continuous multiturn decision-making, and what supports their applicability to other tasks?
The applicability conditions concern task structure, not the OBS/MEM/ANS labels. MA-MSFT targets outputs where sparse but important decision markers are overshadowed by longer natural-language content. SM-GSPO targets sequential decisions where earlier actions affect later histories and where both local actions and trajectories can be evaluated.
A multiplayer game assistant provides an illustrative mapping:
| Decision component | LiveAssistant | Illustrative game assistant |
|---|---|---|
| Observe or retain information | Observe the stream or store a pending need | Observe the battlefield or update an internal threat estimate |
| Whether / When to act | Respond now, wait for evidence, or defer escalation | Signal now, wait for confirmation, or remain silent |
| Whom / What | Answer viewers or notify the host with grounded content | Warn an exposed teammate or send a team-wide tactical instruction |
In this setting, explicit supervision can emphasize sparse markers such as “retreat,” “request support,” and “wait,” while self-generated trajectories expose training to the consequences of earlier decisions. Tool-using and meeting assistants have similar choices among acting, clarifying, recording information, and waiting.
Transfer requires redefining the action protocol, auxiliary targets, and local/trajectory rewards, including task-specific constraints and feedback. Our experiments establish effectiveness within LiveAssistant; the game mapping illustrates applicability. We will distinguish this transferable design rationale from cross-task performance claims, which require independent validation.
Official Comment from Authors to Reviewer KC7b - Part 3
Q4. Always-ANS and class-balanced evaluation: Why does Always-ANS score 55.67 despite failing at OBS, MEM, and Host routing? Does Avg. favor frequent classes, and are model rankings stable under class-balanced evaluation?
1. Why does Always-ANS obtain such a high score? The original metric is , where State uses all evaluation turns, whereas Recipient and Task use only reference-ANS turns (Content is excluded). Always-ANS scores 26.35, 85.58, and 55.08 on these components, giving . Its fixed ANS prediction covers all reference-ANS opportunities, while always routing to viewers and predicting a frequent task yields high scores on the ANS-conditioned components. Zero OBS/MEM recall therefore does not imply zero Recipient or Task accuracy, and zero Host accuracy can coexist with high overall Recipient accuracy when viewers dominate.
2. Does Avg. overreward frequent classes and underrepresent selective intervention? Yes. Always-ANS intervenes on 100% of reference OBS/MEM turns, but those unnecessary responses are penalized only through State accuracy (one-third of Avg.), while frequent viewer and task labels benefit the other two components. This explains how the baseline can exceed general-purpose models in Avg. without demonstrating selective intervention or balanced host/viewer routing. We will explicitly describe Avg. as a reference-agreement composite, not a standalone measure of assistance quality.
3. Do rankings remain stable under class-balanced evaluation? Not entirely, although SM-GSPO retains its lead among the methods below. We recomputed state macro recall as the equal-weight mean of OBS/MEM/ANS recalls and role macro accuracy as the equal-weight mean of host/viewer scores from Tables 1–2 (Qwen denotes Qwen3.5; SFT denotes ordinary multiturn SFT; both GSPO variants start from MA-MSFT):
| Macro metric ↑ | Always-ANS | Qwen-9B | Qwen-27B | SFT | MA-MSFT | + Turn-GSPO | + SM-GSPO |
|---|---|---|---|---|---|---|---|
| State recall | 33.33 | 48.28 | 48.93 | 64.11 | 65.19 | 64.32 | 68.77 |
| Role accuracy | 50.00 | 52.58 | 57.02 | 57.73 | 59.50 | 61.13 | 66.14 |
The recalculation gives two direct answers: SM-GSPO leads on both balanced metrics, so its advantage is not explained solely by frequent classes; other rankings change, as Always-ANS falls below both Qwen models on both metrics, and MA-MSFT exceeds turn-level GSPO on state macro recall despite having lower original Avg. Thus, the apparent superiority of Always-ANS does not persist under class balancing. We will retain Avg. for comparability and add state/role macro metrics to make both class imbalance and selective-intervention failures explicit.
Revision note. We refined the joint problem formulation and its challenges in Section 2 (with supporting case studies in Appendix D). The conventional SFT+RL comparison is summarized in Section 4.2, with detailed pipeline comparisons, single-turn controls, loss ablations, and applicability conditions in Appendix E.1. The interpretation of Avg. and its evaluation denominators are clarified in Section 4.1 and Appendix B.6, with class-balanced results in Appendix E.2 and complementary behavioral evidence in Section 4.3. Revised text is highlighted in green, and newly added tables in blue.
We hope these results and clarifications resolve the reviewer's concerns. Best wishes.
A Gentle Reminder and Invitation for Further Feedback, Looking forward to your discussion
Dear Reviewer KC7b,
Thank you again for your encouraging assessment and for the questions that helped us sharpen the contribution and strengthen the comparisons. As the discussion continues, we wanted to follow up on whether our responses have clarified the value of the joint setting, the training design, and the evaluation metrics.
To briefly summarize:
-
Why joint evaluation matters. We clarified how context-derived need discovery, history-dependent intervention, and recipient-aware assistance interact. The cases in Appendix D.3 illustrate failures such as correct but redundant answers, premature host interruptions, and source confusion leading to inappropriate escalation—showing why isolated decision scores are insufficient.
-
Gains beyond conventional SFT+RL. The added conventional pipeline reaches 61.93 Avg., compared with 67.41 for MA-MSFT + SM-GSPO (+5.48 points). Single-turn controls further support multiturn training: even after single-turn RL, MEM and Host recall remain 11.14% and 3.81%, respectively.
-
Applicability beyond LiveAssistant. We explained the general conditions targeted by the methods—sparse structured decisions and action-dependent histories—and illustrated their mapping to game and meeting assistants, distinguishing this design rationale from experimentally established cross-task transfer.
-
Class-balanced evaluation. We clarified why Always-ANS benefits from the original aggregation. Under state/role macro metrics, it scores 33.33/50.00, below every evaluated learned model, while SM-GSPO leads at 68.77/66.14. Thus, some rankings change, but SM-GSPO’s advantage persists under class balancing.
These clarifications and results are reflected in the revised manuscript, with revised text in green and new tables in blue. We would sincerely appreciate any further feedback and would be happy to discuss any points that remain unclear. Thank you once more for your careful consideration.
Best wishes,
The Authors
Official Comment from Authors to Reviewer KC7b - Part 1
Thank you for recognizing the value of our systematic evaluation of multimodal, proactive assistance and for the positive assessment of MA-MSFT and SM-GSPO. We clarify your concerns below. We appreciate your thoughtful feedback, which has helped us strengthen the experimental evidence.
1. What capabilities does the joint setting evaluate?
Consider an explicitly prompted streaming task: “Monitor event X and notify me when it occurs.” The prompt already specifies what to monitor and what should trigger a response. LiveAssistant instead requires the model to infer whether assistance is needed from heterogeneous signals: audiovisual content, such as a possible mistake in host speech or a highlight moment; audience interactions, such as repeated questions; and platform signals, such as a surge in gifts or a sudden change in viewer count. Detecting a signal is not sufficient—the model must determine whether it indicates an unmet need and which action is appropriate.
The host–viewer setting makes this decision role-dependent: answering viewers and interrupting a host conducting an ongoing broadcast have different purposes and costs. A repeated audience question may warrant a direct answer, a private memory update while awaiting evidence, or a host-facing notification at a suitable moment.
This setting therefore evaluates three capabilities in combination:
The evaluated capability is thus moving from recognizing a potentially relevant signal to selecting an appropriate assistance action in its social and temporal context.
2. What failure modes does this setting reveal?
The trajectory analyses in Appendix D.3 expose failures that individual decision scores or isolated response judgments can miss:
These cases make the interactions between grounding, prior actions, timing, and routing observable within the same trajectory.
3. Why is joint evaluation necessary beyond broader task coverage?
The four decisions are mutually dependent, so their individual correctness does not guarantee an appropriate intervention. Consider viewers repeatedly asking how much longer a dish needs to cook. If the host has already provided the answer, the assistant can respond directly to viewers rather than interrupting the host. If the answer requires host input, it may instead retain the question and wait for an appropriate opportunity. If the assistant has already answered, remaining silent may be preferable.
The same incoming question therefore calls for different actions depending on available content, intended recipient, and prior interaction history. Evaluating answer generation at a supplied time or routing a supplied message does not test whether the model can coordinate these choices itself.
LiveAssistant requires the model to make all four decisions within one evolving stream. We examine this coordination through structured decision metrics, semantic evaluation, and trajectory-level case analysis. The contribution is thus an evaluation setting for coordinated assistance decisions, rather than simply a larger collection of independent tasks.