acceptodds
Under review as a conference paper at ICLR 2027

Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams

Abstract

Livestreams are a primary form of online video: long-lasting, interactive environments in which omni-modal content, viewers, hosts, and platform signals unfold together. Yet how AI systems should act within them remains largely unexplored. The central question is how a model can move beyond perceiving what is happening and deciding when to respond proactively, toward selective participation in a shared social environment. We introduce \liveassistant, a framework for mixed-initiative, role-conditioned assistance that formulates livestream assistance as four coupled decisions over a multi-party stream: whether to act, when to act, whom to address, and what to communicate. At each incoming chunk, \liveassistant integrates native audio and video with synchronized comments, gifts, viewer dynamics, and room metadata, and chooses to continue observing, update private memory, or produce a grounded response for viewers or hosts. To learn this policy, we build a streaming trajectory engine that reconstructs replay timelines into time-aligned decision trajectories, yielding over 320 hours of training data and a 39-hour human-verified benchmark spanning 8 regions and 7 languages. We further introduce Marker-Aware Multiturn Supervised Fine-Tuning (MA-MSFT) and Streaming Multiturn GSPO (SM-GSPO) to strengthen structured decisions and improve behavior over evolving streams. \liveassistant attains the strongest joint performance in timing, routing, and content quality among evaluated models, moving streaming assistance from reactive responses toward selective participation in live social environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Live Room, Human AI Interaction

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_KC7bOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies proactive interaction decisions for multimodal assistants in livestreaming environments. The authors formulate the task as a mixed-initiative decision problem with three actions: OBS (observe), MEM (update private memory), and ANS (respond proactively). They construct structured training trajectories and a human-reviewed evaluation set from livestream audio and video, comments, and platform signals, and propose a two-stage training strategy consisting of MA-MSFT and SM-GSPO. On the proposed benchmark, the complete method achieves an Overall Average of 67.41, compared with 61.34 for ordinary multiturn SFT and 52.66 for the strongest evaluated general-purpose multimodal model. Ablations of the training stages, input modalities, semantic memory, and context length examine the contributions of individual components.

Critical strengths
  1. The authors construct a structured livestream-assistance benchmark that covers multimodal inputs, proactive response timing, memory updates, recipient selection, and task decisions. It provides a relatively comprehensive experimental environment for jointly evaluating these capabilities.
  2. The authors propose MA-MSFT to address insufficient supervision of structured decision markers and SM-GSPO to address the propagation of earlier errors during closed-loop operation. Experiments show that the complete training strategy outperforms ordinary multiturn SFT and the evaluated training variants on the proposed benchmark, providing evidence of its effectiveness in the current task setting.
Critical weaknesses

The necessity and generality of the two-stage training strategy have not been sufficiently demonstrated. Although MA-MSFT + SM-GSPO improves over ordinary SFT, the paper does not directly compare it with a conventional SFT + RL pipeline. It is therefore unclear how much additional benefit the proposed training designs provide.

Questions for authors
  1. Compared with existing benchmarks for streaming multimodal understanding, proactive responses, and multiparty interaction, which previously difficult-to-evaluate capabilities does LiveAssistant's joint decision setting measure, or which new model failure modes does it reveal? Could the authors also explain why jointly evaluating Whether, When, Whom, and What is necessary beyond simply broadening task coverage?
  2. Table 2 compares ordinary SFT, MA-MSFT, and RL strategies initialized from MA-MSFT, but does not report ordinary SFT followed by standard RL, such as conventional GSPO. Could the authors add this direct control to establish how much additional benefit MA-MSFT and SM-GSPO provide over a conventional SFT + RL pipeline?
  3. Could the authors clarify whether MA-MSFT and SM-GSPO are applicable to continuous multiturn decision tasks beyond LiveAssistant? In particular, are the problems they target—insufficient supervision of structured decisions and propagation of earlier errors—general beyond the current benchmark and its specific action protocol? Could the authors provide cross-task experiments or other supporting evidence?
  4. In Table 1, Always-ANS has zero OBS recall, zero MEM recall, and zero Host routing accuracy, yet achieves an Overall Average of 55.67, exceeding all general-purpose models. How do the authors explain this result? Could the current aggregate metric excessively reward frequent classes and therefore fail to adequately reflect selective intervention and multiparty interaction decisions? Would the relative model rankings remain stable under a class-balanced evaluation?
4 comments
AuthorsOpenReview ↗

Official Comment from Authors to Reviewer KC7b - Part 1

Thank you for recognizing the value of our systematic evaluation of multimodal, proactive assistance and for the positive assessment of MA-MSFT and SM-GSPO. We clarify your concerns below. We appreciate your thoughtful feedback, which has helped us strengthen the experimental evidence.

Q1. Capabilities, failure modes, and the necessity of joint evaluation

1. What capabilities does the joint setting evaluate?
Consider an explicitly prompted streaming task: “Monitor event X and notify me when it occurs.” The prompt already specifies what to monitor and what should trigger a response. LiveAssistant instead requires the model to infer whether assistance is needed from heterogeneous signals: audiovisual content, such as a possible mistake in host speech or a highlight moment; audience interactions, such as repeated questions; and platform signals, such as a surge in gifts or a sudden change in viewer count. Detecting a signal is not sufficient—the model must determine whether it indicates an unmet need and which action is appropriate.

The host–viewer setting makes this decision role-dependent: answering viewers and interrupting a host conducting an ongoing broadcast have different purposes and costs. A repeated audience question may warrant a direct answer, a private memory update while awaiting evidence, or a host-facing notification at a suitable moment.

This setting therefore evaluates three capabilities in combination:

  • Context-derived need discovery: identifying assistance needs without an externally specified monitoring target or response trigger.
  • History-dependent intervention: tracking unresolved needs, waiting for sufficient evidence, and recognizing when a need has already been addressed. Figure 12, for example, shows the model retaining an audience question until visual confirmation becomes available.
  • Role-conditioned assistance: choosing whether to answer viewers or notify the host, with timing and content adapted to the intended recipient.

The evaluated capability is thus moving from recognizing a potentially relevant signal to selecting an appropriate assistance action in its social and temporal context.

2. What failure modes does this setting reveal?
The trajectory analyses in Appendix D.3 expose failures that individual decision scores or isolated response judgments can miss:

Failure modeEvidenceWhat the joint analysis reveals
Correct states, ungrounded messagesFigure 16: all four states match the references, but the model misinterprets Arabic speech and comments.Structural correctness can coexist with semantic failure.
Source confusion leading to inappropriate escalationFigure 18: a viewer’s claim about a gift is treated as a confirmed platform event despite a zero gift count.A grounding error propagates into a host-directed intervention.
Correct but redundant answerFigure 17: repeated comments trigger another answer to an already-resolved question.Locally correct content can be unnecessary given the preceding interaction.
Relevant but premature host interruptionFigure 19: a pending question is escalated before a suitable interruption point.A relevant message and plausible recipient do not establish appropriate timing.

These cases make the interactions between grounding, prior actions, timing, and routing observable within the same trajectory.

3. Why is joint evaluation necessary beyond broader task coverage?
The four decisions are mutually dependent, so their individual correctness does not guarantee an appropriate intervention. Consider viewers repeatedly asking how much longer a dish needs to cook. If the host has already provided the answer, the assistant can respond directly to viewers rather than interrupting the host. If the answer requires host input, it may instead retain the question and wait for an appropriate opportunity. If the assistant has already answered, remaining silent may be preferable.

The same incoming question therefore calls for different actions depending on available content, intended recipient, and prior interaction history. Evaluating answer generation at a supplied time or routing a supplied message does not test whether the model can coordinate these choices itself.

LiveAssistant requires the model to make all four decisions within one evolving stream. We examine this coordination through structured decision metrics, semantic evaluation, and trajectory-level case analysis. The contribution is thus an evaluation setting for coordinated assistance decisions, rather than simply a larger collection of independent tasks.

AuthorsOpenReview ↗

Official Comment from Authors to Reviewer KC7b - Part 2

W / Q2. Contribution of the two-stage training strategy: How much do MA-MSFT and SM-GSPO add beyond ordinary SFT followed by conventional RL?

1. Direct pipeline comparison and component contributions. We added ordinary multiturn SFT + conventional turn-level GSPO, using equally weighted reward components, no state shaping, and no trajectory-level reward. This baseline improves Avg. from 61.34 to 61.93, whereas the full pipeline reaches 67.41, a 5.48-point advantage. The existing comparisons further identify a 1.67-point gain from ordinary multiturn SFT to MA-MSFT and a 2.99-point gain from turn-level GSPO to SM-GSPO under the same MA-MSFT initialization and turn-level reward, including +3.86 Recipient and +3.95 Task points. Together, these results show gains beyond simply adding RL. The conventional-to-proposed pipeline comparison changes both initialization and reward design; to separate them, we will additionally report ordinary multiturn SFT followed by GSPO with the original reward configuration.

MethodState ↑Recipient ↑Task ↑Avg. ↑
Ordinary multiturn SFT69.3663.5451.1261.34
Ordinary multiturn SFT + conventional turn-level GSPO69.5864.3151.8961.93
MA-MSFT70.0566.4252.5563.01
MA-MSFT + turn-level GSPO, original reward configuration69.9868.8154.4664.42
MA-MSFT + SM-GSPO71.1472.6758.4167.41

2. Why multiturn training matters. Ordinary SFT in the original paper is already multiturn SFT. To examine the effect of removing the multiturn training formulation, we additionally evaluated single-turn SFT and subsequent single-turn RL, without trajectory-level reward. State and Recipient accuracy follow the overall definitions in Table 2; the state and role columns report per-class recall.

MethodState Acc. ↑OBS ↑MEM ↑ANS ↑Recipient Acc. ↑Viewer ↑Host ↑Task ↑Content ↑
Single-turn SFT47.0684.160.0025.1725.9230.130.0013.2053.21
+ single-turn RL47.7875.8011.1432.3231.8036.353.8118.4157.93

Both settings remain well below ordinary multiturn SFT in overall State and Recipient accuracy (69.36/63.54). Single-turn RL improves ANS recall, Task accuracy, and Content score, but OBS recall decreases, and MEM/Host recall remains only 11.14%/3.81%. Retaining information and deciding when to escalate to the host depend strongly on preceding interactions, supporting multiturn training in this setting. Thank you for requesting controls that distinguish these training choices.


Q3. Applicability beyond LiveAssistant: Do the methods address general problems in continuous multiturn decision-making, and what supports their applicability to other tasks?

The applicability conditions concern task structure, not the OBS/MEM/ANS labels. MA-MSFT targets outputs where sparse but important decision markers are overshadowed by longer natural-language content. SM-GSPO targets sequential decisions where earlier actions affect later histories and where both local actions and trajectories can be evaluated.

A multiplayer game assistant provides an illustrative mapping:

Decision componentLiveAssistantIllustrative game assistant
Observe or retain informationObserve the stream or store a pending needObserve the battlefield or update an internal threat estimate
Whether / When to actRespond now, wait for evidence, or defer escalationSignal now, wait for confirmation, or remain silent
Whom / WhatAnswer viewers or notify the host with grounded contentWarn an exposed teammate or send a team-wide tactical instruction

In this setting, explicit supervision can emphasize sparse markers such as “retreat,” “request support,” and “wait,” while self-generated trajectories expose training to the consequences of earlier decisions. Tool-using and meeting assistants have similar choices among acting, clarifying, recording information, and waiting.

Transfer requires redefining the action protocol, auxiliary targets, and local/trajectory rewards, including task-specific constraints and feedback. Our experiments establish effectiveness within LiveAssistant; the game mapping illustrates applicability. We will distinguish this transferable design rationale from cross-task performance claims, which require independent validation.

AuthorsOpenReview ↗

Official Comment from Authors to Reviewer KC7b - Part 3

Q4. Always-ANS and class-balanced evaluation: Why does Always-ANS score 55.67 despite failing at OBS, MEM, and Host routing? Does Avg. favor frequent classes, and are model rankings stable under class-balanced evaluation?

1. Why does Always-ANS obtain such a high score? The original metric is Avg.=(State Acc.+Recipient Acc.+Task Acc.)/3\mathrm{Avg.}=(\mathrm{State\ Acc.}+\mathrm{Recipient\ Acc.}+\mathrm{Task\ Acc.})/3, where State uses all evaluation turns, whereas Recipient and Task use only reference-ANS turns (Content is excluded). Always-ANS scores 26.35, 85.58, and 55.08 on these components, giving (26.35+85.58+55.08)/3=55.67(26.35+85.58+55.08)/3=55.67. Its fixed ANS prediction covers all reference-ANS opportunities, while always routing to viewers and predicting a frequent task yields high scores on the ANS-conditioned components. Zero OBS/MEM recall therefore does not imply zero Recipient or Task accuracy, and zero Host accuracy can coexist with high overall Recipient accuracy when viewers dominate.

2. Does Avg. overreward frequent classes and underrepresent selective intervention? Yes. Always-ANS intervenes on 100% of reference OBS/MEM turns, but those unnecessary responses are penalized only through State accuracy (one-third of Avg.), while frequent viewer and task labels benefit the other two components. This explains how the baseline can exceed general-purpose models in Avg. without demonstrating selective intervention or balanced host/viewer routing. We will explicitly describe Avg. as a reference-agreement composite, not a standalone measure of assistance quality.

3. Do rankings remain stable under class-balanced evaluation? Not entirely, although SM-GSPO retains its lead among the methods below. We recomputed state macro recall as the equal-weight mean of OBS/MEM/ANS recalls and role macro accuracy as the equal-weight mean of host/viewer scores from Tables 1–2 (Qwen denotes Qwen3.5; SFT denotes ordinary multiturn SFT; both GSPO variants start from MA-MSFT):

Macro metric ↑Always-ANSQwen-9BQwen-27BSFTMA-MSFT+ Turn-GSPO+ SM-GSPO
State recall33.3348.2848.9364.1165.1964.3268.77
Role accuracy50.0052.5857.0257.7359.5061.1366.14

The recalculation gives two direct answers: SM-GSPO leads on both balanced metrics, so its advantage is not explained solely by frequent classes; other rankings change, as Always-ANS falls below both Qwen models on both metrics, and MA-MSFT exceeds turn-level GSPO on state macro recall despite having lower original Avg. Thus, the apparent superiority of Always-ANS does not persist under class balancing. We will retain Avg. for comparability and add state/role macro metrics to make both class imbalance and selective-intervention failures explicit.

Revision note. We refined the joint problem formulation and its challenges in Section 2 (with supporting case studies in Appendix D). The conventional SFT+RL comparison is summarized in Section 4.2, with detailed pipeline comparisons, single-turn controls, loss ablations, and applicability conditions in Appendix E.1. The interpretation of Avg. and its evaluation denominators are clarified in Section 4.1 and Appendix B.6, with class-balanced results in Appendix E.2 and complementary behavioral evidence in Section 4.3. Revised text is highlighted in green, and newly added tables in blue.

We hope these results and clarifications resolve the reviewer's concerns. Best wishes.

AuthorsOpenReview ↗

A Gentle Reminder and Invitation for Further Feedback, Looking forward to your discussion

Dear Reviewer KC7b,

Thank you again for your encouraging assessment and for the questions that helped us sharpen the contribution and strengthen the comparisons. As the discussion continues, we wanted to follow up on whether our responses have clarified the value of the joint setting, the training design, and the evaluation metrics.

To briefly summarize:

  1. Why joint evaluation matters. We clarified how context-derived need discovery, history-dependent intervention, and recipient-aware assistance interact. The cases in Appendix D.3 illustrate failures such as correct but redundant answers, premature host interruptions, and source confusion leading to inappropriate escalation—showing why isolated decision scores are insufficient.

  2. Gains beyond conventional SFT+RL. The added conventional pipeline reaches 61.93 Avg., compared with 67.41 for MA-MSFT + SM-GSPO (+5.48 points). Single-turn controls further support multiturn training: even after single-turn RL, MEM and Host recall remain 11.14% and 3.81%, respectively.

  3. Applicability beyond LiveAssistant. We explained the general conditions targeted by the methods—sparse structured decisions and action-dependent histories—and illustrated their mapping to game and meeting assistants, distinguishing this design rationale from experimentally established cross-task transfer.

  4. Class-balanced evaluation. We clarified why Always-ANS benefits from the original aggregation. Under state/role macro metrics, it scores 33.33/50.00, below every evaluated learned model, while SM-GSPO leads at 68.77/66.14. Thus, some rankings change, but SM-GSPO’s advantage persists under class balancing.

These clarifications and results are reflected in the revised manuscript, with revised text in green and new tables in blue. We would sincerely appreciate any further feedback and would be happy to discuss any points that remain unclear. Thank you once more for your careful consideration.

Best wishes,
The Authors

Reviewer_beXbOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LIVEASSISTANT, a framework for proactive, role-conditioned assistance in real-world livestreams. It jointly addresses four decisions: whether to intervene, when to intervene, whom to assist, and what to communicate. The framework integrates audiovisual content and social signals through three action states (OBS/MEM/ANS). It introduces Marker-Aware Multiturn SFT (MA-MSFT) for structured decision learning and Streaming Multiturn GSPO (SM-GSPO) for sequential policy optimization. The authors construct over 320 hours of training data and a 39-hour human-verified benchmark covering diverse languages and livestream scenarios. Experiments demonstrate improvements in decision accuracy, recipient routing, and task selection, supported by ablations on training strategies, input modalities, and semantic memory.

Critical strengths

S1. Practically meaningful problem formulation and coherent methodology.

The paper addresses an important yet underexplored problem beyond conventional streaming video understanding. Jointly modeling intervention timing, recipient selection, and response generation reflects the practical requirements of real-world livestream assistance. Although individual techniques build on existing methods, their integration is well motivated and forms a coherent solution.

S2. Valuable real-world dataset and benchmark construction.

The authors construct a substantial livestream dataset covering diverse languages, regions, and scenarios, with synchronized audiovisual and social signals. The room-level data partitioning and independent human annotation validation strengthen the credibility of the benchmark. This dataset provides a valuable foundation for studying proactive multimodal interaction.

S3. Comprehensive experimental validation and convincing capability demonstrations.

The proposed method achieves clear improvements over evaluated baselines, supported by progressive training ablations. Additional experiments demonstrate the contributions of different input modalities and temporally relevant semantic memory. The qualitative examples further illustrate meaningful proactive behaviors, including selective intervention, contextual memory, and recipient-aware assistance.

Critical weaknesses

W1. Limited methodological novelty and insufficient analysis of trajectory-level optimization.

MA-MSFT largely combines marker weighting with an auxiliary classification objective, while SM-GSPO extends existing policy optimization with multiturn credit assignment. In particular, the trajectory reward aggregates per-turn structural rewards, and its specific contribution to long-term consistency and non-repetition is not directly evaluated.

W2. Limited validation of real-world interactive effectiveness.

Despite using real livestream recordings, evaluation relies on offline replay and teacher-generated reference decisions. Independent human annotation agreement partially mitigates this concern, but does not establish actual user utility. The evaluated clips are also relatively short (2–10 minutes), limiting conclusions about sustained real-world assistance.

Questions for authors
  1. Trajectory optimization: Beyond the aggregate improvements in Table 2, does trajectory-level optimization reduce repeated responses or contradictory messages compared with turn-level GSPO?

  2. Real-world effectiveness: Is there any human preference evaluation or interactive deployment evidence demonstrating that the generated interventions are genuinely useful and appropriately timed for hosts and viewers?

6 comments
AuthorsOpenReview ↗

Official Comment from Authors to Reviewer beXb - Part 1

Thank you for recognizing our problem formulation, benchmark, human validation, and ablations. We clarify the joint optimization design and provide direct behavioral comparisons. We hope the reviewer will reconsider the assesment in light of these new results and clarifications.

W1 / Q1. Limited methodological novelty and insufficient analysis of trajectory-level optimization.

1. Methodological contribution and controlled comparison. Our contribution is structured supervision and joint turn- and trajectory-level optimization for streaming, multiparty assistance. MA-MSFT emphasizes sparse decision markers and supervises the OBS/MEM/ANS boundary. SM-GSPO combines two reward–credit branches over self-generated histories:

  • Turn-level: structural, applicable content, and state-shaping rewards produce a group-normalized turn advantage, assigned only to that turn’s assistant tokens through a turn-level sequence ratio.
  • Trajectory-level: summed per-turn structural rewards produce a group-normalized trajectory advantage, assigned to all assistant tokens in the rollout through a trajectory-level sequence ratio.

SM-GSPO jointly optimizes both branches, combining local feedback with whole-rollout credit. Its trajectory reward does not explicitly score repetition or semantic contradiction; we evaluate these separately.

Both RL variants start from the same MA-MSFT checkpoint and use the same self-generated rollout procedure. Turn-level GSPO retains only the turn branch; SM-GSPO additionally includes trajectory-level reward and its scoped credit assignment. This comparison evaluates the trajectory branch as a whole, not reward and credit assignment independently.

MethodRL reward–credit scopeState ↑Recipient ↑Task ↑MEM recall ↑Sem. ↑Avg. ↑
MA-MSFTNone70.0566.4252.5536.5079.9063.01
+ turn-level GSPOTurn only69.9868.8154.4628.5280.1964.42
+ SM-GSPOTurn + trajectory71.1472.6758.4148.2081.7867.41

SM-GSPO improves Avg., Recipient, and Task accuracy over turn-level GSPO by 2.99, 3.86, and 3.95 points, respectively. We next assess whether these gains extend to the behaviors raised by the reviewer.

2. Repetition: lower repetition with higher response coverage.

MethodRepetition rate ↓ANS recall ↑
MA-MSFT3.04%72.00%
+ turn-level GSPO2.47%74.64%
+ SM-GSPO2.11%79.26%

Repetition measures the proportion of outgoing messages repeating an earlier response within the same clip after removing protocol tags and normalizing case, punctuation, and whitespace; ANS recall measures valid same-turn responses on reference-ANS turns. Compared with turn-level GSPO, SM-GSPO lowers repetition by 0.36 percentage points while increasing ANS recall by 4.62 points. Thus, lower repetition accompanies broader reference-response coverage, rather than reduced coverage, although this text-based measure does not capture all semantic redundancy.

3. Contradictory messages: human assessment of cross-turn conflicts.

MethodContradictory messagesAudited messagesContradiction rate ↓
MA-MSFT113003.67%
+ turn-level GSPO73002.33%
+ SM-GSPO43001.33%

Two independent auditors assessed 300 sampled ANS messages per model, viewing the input prefix and actual model history with model identity and reference labels hidden. Contradictions require incompatible claims about the same entity, event, or request under unchanged temporal and contextual conditions, excluding supported updates, explicit corrections, and paraphrases; recipient changes do not exempt factual conflicts. SM-GSPO has 4 observed contradictions versus 7 for turn-level GSPO (1.33% vs. 2.33%). The blinded human audit shows that SM-GSPO produces fewer cross-turn contradictions than turn-level GSPO, demonstrating better semantic consistency across the audited responses.

4. Performance over time:

Method / Avg. ↑T1T2T3T4T5
MA-MSFT63.4363.7663.4762.9661.57
+ turn-level GSPO64.7365.0864.9464.5762.91
+ SM-GSPO67.9668.4368.1467.7664.88

Using the same five equal-duration trajectory segments as Table 5 and Avg. = (State + Recipient + Task) / 3, SM-GSPO outperforms turn-level GSPO throughout, by 1.97–3.35 points, with a narrower margin in T5. Overall MEM recall rises from 28.52% to 48.20%, indicating better identification of reference memory-update opportunities. These results show consistently higher structural decision quality, not slower temporal degradation, complementing the repetition analysis and semantic-conflict audit. The hour-long evaluation below examines longer-duration limits.

AuthorsOpenReview ↗

Official Comment from Authors to Reviewer beXb - Part 2

W2 / Q2. Helpfulness, timing, and sustained assistance: Are generated interventions useful and appropriately timed according to human evaluators? Can the evaluation extend beyond the original 2–10-minute clips?

1. Human evaluation of generated interventions. The existing benchmark review covers 14 rooms, 275 clips, and 13,781 valid turns. Two annotators independently followed the prefix-only protocol, with State/Recipient/Task agreement against teacher references of 92%/87%/83% and inter-annotator Cohen’s κ\kappa of 0.86/0.81/0.74. These results assess reference quality.

To directly assess generated interventions, we additionally report human ratings on a 1–5 scale for addressing the current need, intervening at an appropriate time, addressing the appropriate recipient, and grounding the content in the available context on 560 turns covering 14 rooms.

MethodHelpfulness ↑Timing appropriateness ↑Recipient appropriateness ↑Contextual grounding ↑
MA-MSFT4.184.064.114.02
+ turn-level GSPO4.374.174.354.41
+ SM-GSPO4.644.284.634.70

SM-GSPO receives the highest mean ratings for helpfulness, timing appropriateness, recipient appropriateness, and contextual grounding. These judgments directly support the perceived helpfulness and contextual relevance of generated interventions, beyond agreement with reference labels.

2. Hour-long continuous replay. To address the limited duration of the original clips, we evaluated the SM-GSPO checkpoint on 23 continuous one-hour livestream recordings, covering diverse regions, languages, and content types. This extends evaluation to six times the original maximum duration.

OBS/MEM/ANS are per-state recalls; Recipient All is strict routing accuracy; Avg. = (State + Recipient + Task) / 3. All metrics are percentages.

IntervalTurnsState AllOBSMEMANSRecipient AllTaskAvg.
0–10 min91169.7079.0050.0076.5767.3653.5663.54
10–20 min90072.0084.7044.6571.9564.2553.3963.22
20–30 min90269.2983.2133.9883.4173.3658.5267.06
30–40 min90267.2990.0829.6075.0063.4946.8359.20
40–50 min90569.1790.5633.8669.2957.6844.2157.02
50–60 min89767.2291.0627.1161.4446.6141.9551.93

State accuracy remains within 67.22–72.00% throughout the hour, and Avg. stays within 63.22–67.06% during the first half-hour. Avg. declines to 51.93% in the final interval, with lower memory-update recall, response recall, routing accuracy, and task accuracy. These results directly extend the evaluation beyond short clips, while identifying where performance becomes harder to sustain.

For deployment, the policy could be paired with periodic context consolidation, for example every 30 minutes: summarize relevant facts, unresolved requests, and previously communicated information, then reinitialize inference with this carryover state and recent multimodal context. This complements per-turn MEM actions without discarding interaction continuity. It is a concrete deployment option whose benefit remains to be evaluated.

Thank you for encouraging this extension. We will incorporate the hour-long results and this deployment discussion into the revision.

AuthorsOpenReview ↗

Official Comment from Authors to Reviewer beXb - Part 3

Ethics. Data use, consent, privacy, and release: What authorizes the use of real livestream material, and which assets can be redistributed?

1. Research-use basis and scope of consent. As stated in the original Ethics Statement, material was obtained under the source-platform terms and underwent institutional legal, privacy, and compliance review before annotation or training. Annotators provided informed consent for their participation and received fair compensation.

We will distinguish three scopes explicitly. Research use rests on the platform terms and institutional review; it does not imply that every livestream participant individually signed research consent. Annotator consent covers annotation work, not authorization from livestream participants to use or redistribute their material. Production deployment requires explicit user consent, human oversight, and auditing of excessive intervention and routing errors.

2. Scope of de-identification. The confirmed preprocessing removes or anonymizes personally identifiable information in comments and metadata before use. This does not by itself establish de-identification of faces, voices, or on-screen text.

The revision will describe the actual treatment of each modality separately. Material with unresolved identifiable information or without the required redistribution rights will not be included in public releases.

3. Release permissions and asset boundaries. The original manuscript plans to release an anonymized benchmark, training and evaluation code, and final checkpoints after acceptance, subject to platform terms and the Ethics Statement safeguards.

We will specify the release conditions by asset type:

AssetRelease scopeRequired checks
Raw livestream audio/videoNot included in the default public-release commitmentSeparate authorization for redistribution or controlled access
Processed mediaOnly authorized portions meeting privacy requirementsFaces, voices, on-screen text, and other identifying information
Anonymized trajectories and textWithin the applicable permissionsDe-identification of text and metadata
Training and evaluation codeRelease as plannedRemoval of credentials, internal paths, and private data links
Final checkpointsSubject to applicable permissions and safeguardsModel-release rights and training-data leakage risks

Research-use approval does not automatically authorize redistribution; controlled access also requires authorization. We will release only assets with the necessary permissions and privacy safeguards. Thank you for highlighting the need to make these boundaries explicit.

Revision note. We have clarified the joint turn- and trajectory-level optimization in Section 3.3 and Appendix B.3. Behavioral and temporal comparisons are presented in Section 4.3 and Appendix E.3; human intervention-quality ratings and the cross-turn contradiction audit appear in Appendix E.4. The hour-long replay results are summarized in Section 4.3 and detailed in Appendix E.5. Research authorization, consent, privacy handling, and release boundaries are clarified in the Ethics Statement and Appendix F. Revised text is highlighted in green, and newly added tables in blue.

If you have any further issues, please ask us at you convenience, we are willing to address your concern. Best wishes.

Reviewer_QdTQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a substantial engineering effort to extend streaming assistance to real-world social livestreams. It jointly addresses whether and when to intervene, whether to address hosts or viewers, and what to communicate based on incoming audio, video, comments, and platform events. The authors formulate assistance through OBS, MEM, and ANS actions and construct over 320 hours of training data and an approximately 39-hour benchmark with independent human validation of its annotations.

LiveAssistant adopts two-stage training. MA-MSFT strengthens supervision of state, recipient, and task markers, while SM-GSPO optimizes self-generated multi-turn trajectories using turn- and trajectory-level rewards. Experiments on a room-disjoint test set evaluate structured decisions and message quality, report improvements over standard multi-turn SFT, and examine the effects of input modalities, semantic memory, and context length.

Critical strengths
  1. A valuable task formulation and dataset grounded in real-world livestreams. The paper integrates audio-visual understanding, endogenous request discovery from social signals, intervention timing, and recipient routing to viewers and hosts within a unified streaming-assistance framework. The training data and benchmark developed for this setting provide a concrete foundation for studying multimodal, role-conditioned proactive assistance. I consider this a valuable contribution to the community.

  2. Substantial system integration supported by empirical evidence. The authors employ a prefix-only annotation protocol, room-level data splits, and independent blind human re-annotation, strengthening confidence in the data and evaluation. The full model shows a clear improvement over ordinary multiturn SFT in the aggregate structured-decision metric (Avg.). Ablations of synchronized input signals and semantic memory further indicate that the model uses multimodal signals and temporally relevant memory content under the proposed evaluation protocol, supporting the feasibility of the unified policy.

Critical weaknesses
  1. The empirical evidence for RL does not fully address its stated motivation. The authors motivate SM-GSPO through the mismatch between reference histories used in SFT and self-generated histories at deployment, highlighting error accumulation, repetitive responses, and intervention timing. Training on self-generated trajectories is a reasonable response to this mismatch. However, the trajectory-level reward primarily sums turn-level structural scores, and its connection to non-repetition, restraint, and timeliness remains insufficiently explained. The ablations demonstrate improvements in structured-decision metrics but do not directly evaluate these behaviors. Consequently, it remains unclear whether the gains reflect improvements in the specific problems motivating RL. Targeted behavioral evaluations and a comparison with supervised training on self-generated histories using appropriate supervision would help clarify RL’s contribution.

  2. The connection between the evaluation metrics and practical assistance value remains insufficiently established. The structural evaluation primarily measures agreement with a reference policy, although multiple intervention timings, recipients, and responses may be reasonable in the same context. The paper provides limited analysis of how the evaluation accommodates these alternatives. Moreover, Avg. equally averages State accuracy over all turns and Recipient and Task accuracies on reference ANS turns, while excluding message semantic quality. In Table 1, a fixed policy that always selects ANS with viewer-directed narration scores 55.67, exceeding the best-performing general-purpose model’s 52.66. This does not by itself invalidate the evaluation, but suggests that class priors and metric aggregation may substantially influence the ranking. Although the paper reports the class distribution, it does not directly discuss this result or its implications. Further validation is therefore needed to establish how improvements in Avg. translate into more useful assistance.

Questions for authors
  1. Could a small human audit conditioned on the model’s actual history assess how often reasonable actions are penalized because earlier responses or memory updates differ from the reference trajectory? Please also specify the sample size and coverage of the existing human re-annotation, and whether disagreements resulted in corrections to the final test labels.

  2. Using existing test outputs, could the authors compare MA-MSFT and SM-GSPO in response repetition, inappropriate intervention, and performance across the same chronological segments, with clearly defined metrics? A continued MA-MSFT control using the additional RL streams and a comparable training budget would help separate RL gains from additional-data and training effects. If feasible, a small-scale supervised control using self-generated histories with history-conditioned teacher relabeling would more directly test the exposure-mismatch motivation.

  3. Could a marker-weighting-only control quantify the incremental benefit of the auxiliary state loss? Please also clarify the rationale for stopping gradients through the focal weights. Is there analytical or empirical evidence supporting this choice over retaining those gradients?

  4. Beyond the weights reported in Table 7, please specify fg,tf_{g,t}, the value of λc\lambda_c, and the complete calculation of Rg,tstrR^{\mathrm{str}}_{g,t}, including conditional scoring and any normalization. In particular, how is a response scored if it matches the reference structural labels but repeats information already communicated in the model’s actual history? This would clarify the connection between the reward and the stated non-repetition and restraint objectives.

Given the substantial engineering effort and practical relevance of this work, I would consider raising my rating if the authors provide convincing clarifications and evidence addressing these concerns.

8 comments
AuthorsOpenReview ↗

Official Comment from Authors to Reviewer QdTQ - Part 1

Thank you for recognizing the value of our unified livestream-assistance formulation, data construction, and experimental analysis. Your questions motivated additional behavioral evaluations, supervised controls, a history-conditioned human audit, and targeted loss ablations. We sincerely hope the reviewer will reconsider the assesment in light of these new results and clarifications.

W1 / Q2. Behavioral improvements and supervised controls:

Behavioral Evaluation

Metrics and Results

We evaluate self-generated rollouts using five complementary metrics: Rep., repetition rate of outgoing messages after text normalization; SP, silence precision, the proportion of predicted OBS/MEM turns whose reference is also OBS/MEM; BalAcc, binary balanced accuracy, the arithmetic mean of ANS recall and non-response recall, with OBS/MEM merged as non-outgoing actions; Cov., valid same-turn ANS coverage on reference ANS turns; and Err., the proportion of turns with a recipient or task ordered-list mismatch, conditional on both reference and prediction being valid ANS. All values below are percentages.

MethodRep. ↓SP ↑BalAcc ↑Cov. ↑Err. ↓
MA-MSFT3.0489.7483.9372.0027.01
+ turn-level GSPO2.4791.4684.7674.6426.47
+ SM-GSPO2.1192.7185.7679.2625.92

Repetition and Timing

SM-GSPO reduces measured repetition from 3.04% to 2.11% while increasing response coverage from 72.00% to 79.26%: fewer repeats accompany broader coverage of reference response opportunities. Cov. captures same-turn alignment with those opportunities.

Restraint-Related Behavior

SP and BalAcc improve together, indicating more reference-consistent silence decisions and better class-balanced response selection. These complementary diagnostics assess restraint jointly with responsiveness.

Inappropriate Intervention

Conditional routing/task error decreases, including on shared reference-ANS turns where both models answer (27.06% to 25.85%). The results indicate more accurate conditional routing.

Performance over Time

Using the T1–T5 boundaries in Table 5, SM-GSPO retains an Avg. advantage throughout:

MethodT1T2T3T4T5
MA-MSFT63.4363.7663.4762.9661.57
+ turn-level GSPO64.7365.0864.9464.5762.91
+ SM-GSPO67.9668.4368.1467.7664.88

We also examine behavioral metrics in B1–B5, which divide each clip into five equal-duration bins and assign turns by their start time. Each cell reports MA-MSFT / SM-GSPO (%).

BinCov. ↑Err. ↓SP ↑BalAcc ↑Rep. ↓
B172.44 / 79.1325.42 / 24.8390.66 / 92.4583.98 / 85.202.24 / 1.44
B273.91 / 79.7827.17 / 25.6890.93 / 92.5984.45 / 85.752.71 / 1.86
B373.39 / 80.8526.49 / 26.7691.61 / 93.5784.74 / 86.463.13 / 2.06
B471.66 / 80.4227.12 / 24.5491.27 / 93.5683.98 / 86.423.41 / 2.43
B568.72 / 76.3828.91 / 29.7089.22 / 91.4082.56 / 84.963.82 / 2.84

SM-GSPO achieves higher Cov., SP, and BalAcc and lower Rep. in every bin. Err. decreases in B1/B2/B4 and increases in B3/B5. Repetition rises toward the end for both methods, with SM-GSPO remaining lower throughout.

Overall Behavioral Conclusion

Together, these evaluations connect the structural gains to lower measured repetition, broader reference-response coverage, and stronger selective-silence diagnostics across chronological segments. They also distinguish improved routing from increased response activity.

Continued-Supervision Controls

Setup and Results

All controls start from the same MA-MSFT checkpoint and use the full RL dataset, including the history-conditioned teacher-relabeling control.

Further trainingStateRecipientTaskMEM recallSem.Avg.Rep. ↓
MA-MSFT: reference history + original labels70.4467.0553.6338.4980.1463.712.83%
MA-MSFT: self-generated history + original labels70.5468.1854.2540.3380.5964.322.65%
MA-MSFT: self-generated history + teacher relabeling70.8468.9255.3542.6581.1265.042.35%
SM-GSPO71.1472.6758.4148.2081.7867.412.11%

Interpretation

The controls separate continued supervision, self-generated-history exposure, and history-conditioned teacher relabeling. All three improve performance, while SM-GSPO exceeds the strongest control by 2.37 Avg. points, with lower repetition (2.11% vs. 2.35%). This supports incremental gains beyond the tested supervised alternatives. We thank the reviewer for suggesting behavioral diagnostics and history-conditioned supervised controls to better isolate RL's contribution.

AuthorsOpenReview ↗

Official Comment from Authors to Reviewer QdTQ - Part 2

W2 / Q1. Structural agreement and practical assistance. We address alternative valid decisions, annotation reliability, class balance, and the link between structural gains and perceived assistance.

1) Reasonable actions under model histories. Our benchmark already supports joint multi-recipient/task outputs, present in approximately 2% of benchmark examples. These annotations capture cases where assisting multiple recipients or addressing multiple tasks together is appropriate, rather than restricting every response to a single recipient–task pair. At the state level, each turn currently has one reference decision.

Different earlier responses or memory updates can also make a different current action reasonable. To assess this, we audit 14 rooms × 4 clips × 10 shared target turns = 560 turns per model, using each model's actual history. One annotator, blind to model identity, reference labels, and future inputs, judges each action from the stream prefix and the model's prior ANS/MEM, with MEM treated as private context. The reference is revealed afterward to identify reasonable actions that receive structural penalties because earlier responses or memory updates differ from the reference history.

History-related false penaltyMA-MSFTSM-GSPO
Earlier response differs from reference history12 (2.14%)9 (1.61%)
Earlier memory update differs from reference history6 (1.07%)5 (0.89%)
Overlap of the two causes3 (0.54%)2 (0.36%)
Deduplicated total15 (2.68%)12 (2.14%)

History-related false penalties affect 2.68% / 2.14% of audited turns for MA-MSFT / SM-GSPO. These results quantify how often differences in prior actions lead to penalties on otherwise reasonable decisions.

We thank the reviewer for highlighting that multiple decisions may be reasonable in the same context. Beyond the joint outputs already represented, future benchmark versions will annotate alternative acceptable decisions, including states, intervention timings, recipients, and responses where appropriate, and adapt scoring accordingly to improve evaluation completeness and reliability.

2) Label reliability and correction flow. Two annotators independently re-annotated the entire benchmark (14 rooms, 275 clips, 13,781 turns; seven languages, nine categories) under the prefix-only protocol, without seeing labels or each other's decisions. Agreement with the withheld teacher labels is 92% / 87% / 83% for State / Recipient / Task, with inter-annotator Cohen's κ=\kappa= 0.86 / 0.81 / 0.74, and content scores 4.32/5 with 91% within-1 agreement. The workflow is: drafts → two independent re-annotations → agreement and κ\kappa computed before adjudication → disagreements adjudicated and labels frozen → all models evaluated on the frozen labels. 11% / 14% / 18% of applicable State / Recipient / Task entries were corrected during this process, and every result in the paper uses the finalized labels.

3) Always-ANS and class balance. Always-ANS reaches (26.35+85.58+55.08)/3=55.67(26.35+85.58+55.08)/3=55.67 because it covers all reference-ANS turns while also answering every reference OBS/MEM turn; Recipient and Task are scored only on reference-ANS turns, so the aggregate is sensitive to class priors. Re-weighting each state and role equally removes this effect:

Macro metric (↑)Always-ANSQwen-9BQwen-27BSFTMA-MSFT+ Turn-GSPO+ SM-GSPO
State macro recall33.3348.2848.9364.1165.1964.3268.77
Role macro accuracy50.0052.5857.0257.7359.5061.1366.14

SM-GSPO leads both macro metrics, and Always-ANS ranks below every learned model, so the ranking is not driven by frequent classes.

4) From structural agreement to assistance value. Structural metrics quantify agreement with one reference; to connect them with practical assistance we collect human ratings (1–5) of generated ANS actions in 560 turns on four dimensions: addressing the current need, timing appropriateness, recipient appropriateness, and contextual grounding. Raters see the stream prefix and the model's own history, with model identity hidden and order randomized.

MethodHelpfulnessTimingRecipientGrounding
MA-MSFT4.184.064.114.02
+ turn-level GSPO4.374.174.354.41
+ SM-GSPO4.644.284.634.70

Ratings improve monotonically across the three checkpoints on all four dimensions, in the same order as Avg. (63.01 → 64.42 → 67.41). The structural gains are thus accompanied by higher human-judged helpfulness, timing, routing, and grounding, not merely closer reference matching. Thank you for your insights and suggestions.

AuthorsOpenReview ↗

Official Comment from Authors to Reviewer QdTQ - Part 3

Q3. Auxiliary state loss and detached focal weights: What does the auxiliary loss add beyond marker weighting, and what supports detaching the focal weights?

Ablations. We separate ordinary multiturn SFT, marker weighting alone (α=0\alpha=0), and full MA-MSFT (retaining vs. detaching focal-weight gradients), without subsequent RL:

SFT objectiveStateOBSMEMANSRecipientTaskSem.Avg.
Ordinary multiturn SFT69.3687.7135.9968.6463.5451.1274.3261.34
Marker weighting only69.7287.3036.1970.5765.0151.8477.2562.19
Full MA-MSFT, retain focal-weight gradients69.8587.2435.9671.4165.9452.3179.4862.70
Full MA-MSFT, detach focal weights70.0587.0636.5072.0066.4252.5579.9063.01

Marker weighting adds 0.85 Avg. points by emphasizing selected targets in vocabulary-level cross-entropy; the auxiliary loss adds another 0.82 by explicitly training the OBS/MEM/ANS boundary using a restricted three-class distribution from the same logits.

Effect of detach. Let pt=πθS(zt∣ht)p_t=\pi_\theta^{\mathcal S}(z_t\mid h_t) and at=βzt/∑jβzja_t=\beta_{z_t}/\sum_j\beta_{z_j}. The detached and fully differentiated focal-loss gradients satisfy

gtdetach=−at(1−pt)γ∇θlog⁡pt,gtfull=(1−γptlog⁡pt1−pt)gtdetach.\displaystyle g_t^{\mathrm{detach}}=-a_t(1-p_t)^\gamma\nabla_\theta\log p_t,\qquad g_t^{\mathrm{full}}=\left(1-\frac{\gamma p_t\log p_t}{1-p_t}\right)g_t^{\mathrm{detach}}.

For γ≥0\gamma\ge0, the multiplier lies between 11 and 1+γ1+\gamma. Retaining focal-weight gradients preserves each example’s gradient direction but changes its magnitude according to ptp_t, altering relative example weighting and balance with the vocabulary loss (not equivalent to a single global learning-rate change).

Keeping initialization, data/order, seed, steps, optimizer, checkpoint selection, γ\gamma, α=0.1\alpha=0.1, and βOBS/MEM/ANS=3/1/1\beta_{\mathrm{OBS/MEM/ANS}}=3/1/1 fixed, stop-gradient improves Avg. from 62.70 to 63.01 and state macro recall from 64.87 to 65.19, providing modest empirical support alongside the optimization derivation.

Q4. Reward specification and historical repetition: How is reward computed, and can a structurally correct but repeated response still receive high reward?

Let ff indicate valid format, SS state correctness, and A,Q,NA,Q,N exact matches of ordered recipient lists, ordered task lists, and message count (all binary; A,Q,NA,Q,N are zero unless both states are ANS). Normalized to [0,1][0,1] with 11 rewarding valid format:

Rg,tstr={0,fg,t=0,(1+3Sg,t)/4,fg,t=1,zg,tref∈OBS,MEM,(1+3Sg,t+Ag,t+3Qg,t+Ng,t)/9,fg,t=1,zg,tref=ANS.\displaystyle R^{\mathrm{str}}_{g,t}=\begin{cases}0, & f_{g,t}=0,\\\\(1+3S_{g,t})/4, & f_{g,t}=1,\\ z^{\mathrm{ref}}_{g,t}\in\\{\mathrm{OBS},\mathrm{MEM}\\},\\\\(1+3S_{g,t}+A_{g,t}+3Q_{g,t}+N_{g,t})/9, & f_{g,t}=1,\\ z^{\mathrm{ref}}_{g,t}=\mathrm{ANS}.\end{cases}

Turn and trajectory rewards (group-normalized separately; RgR_g sums structural scores without length averaging) are

sg,t=fg,t(Rg,tstr+0.25cg,t+bg,t),Rg=∑tRg,tstr.\displaystyle s_{g,t}=f_{g,t}\left(R^{\mathrm{str}}_{g,t}+0.25c_{g,t}+b_{g,t}\right),\qquad R_g=\sum_t R^{\mathrm{str}}_{g,t}.

Here, λc=0.25\lambda_c=0.25, c∈[0,1]c\in[0,1] is the Qwen3-VL reference-semantic score (zero unless both states are ANS), and shaping bb is +0.35+0.35 for state match, −0.35-0.35 for reference-OBS mismatch, −0.60-0.60 for reference-MEM mismatch, and zero otherwise. Invalid outputs receive zero reward.

Reference content provides an implicit non-repetition signal. Reference trajectories avoid redundancy, so repeating earlier content instead of supplying current reference information lowers the semantic reward (and original SM-GSPO already reduces repetition). However, since the semantic judge does not inspect model history, a historically repeated response matching the current reference can still receive maximum reward—sg,t=1.60s_{g,t}=1.60 and a contribution of 11 to RgR_g.

Following the reviewer’s suggestion, deducting 0.20.2 from RgR_g when repetition against model history is detected (with turn rewards unchanged) reduces repetition from 2.11% to 1.37% (35.1% relative), while Avg. stays essentially unchanged (67.41 → 67.43). We will report this extension separately from original SM-GSPO.

Revision note. We incorporated the behavioral, temporal, and human evaluations into Section 4.3 and Appendices E.3–E.4; continued-supervision controls and loss ablations into Appendix E.1; benchmark re-annotation and adjudication details into Appendix C.1; metric definitions and class-balanced results into Section 4.1 and Appendices B.6/E.2; the history-conditioned audit into Appendix E.4; and reward details plus the repetition-penalty extension into Appendices B.3 and E.6. Revised text is highlighted in green, and newly added tables in blue.

Thank you again for the constructive comments. We would be glad to clarify any remaining concern. Best wishes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of Core Contributions, Empirical Findings, and New Experiments for AC and Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the AC and Reviewers (QdTQ, beXb, KC7b) for your constructive feedback and recognition of our work. Before summarizing our new experiments, we briefly recap our core contributions and empirical findings:

  • Problem Formulation & Benchmark: Moving beyond trigger-specified video QA, we formulate proactive, role-conditioned multiparty assistance in real-world livestreams—jointly deciding whether and when to act, whom to address (host vs. viewers), and what to retain or communicate across OBS/MEM/ANS states. We build 320+ hours of training data and a 39-hour human-verified benchmark (14 rooms, 275 clips, 13,781 turns; 7 languages, 9 categories).
  • Empirical Findings on Baseline Limitations: Evaluating strong general-purpose multimodal models reveals that proactive livestream assistance remains heavily underexplored: existing baselines struggle with selective intervention (peaking at 52.66 Avg. and 48.93% State Macro Recall), exhibiting severe failure modes including over-intervention on silent turns, neglect of private memory (MEM) and host escalation, source confusion between viewer comments and platform events, and redundant or premature interruptions over evolving histories.
  • Methodology & Applications: We show that open-ended streaming assistance suffers from sparse decision-marker supervision and closed-loop history drift. MA-MSFT resolves the former via marker weighting and an auxiliary state-boundary loss; SM-GSPO resolves the latter via joint turn- and trajectory-level optimization over self-generated rollouts (67.41 Avg.). This design transfers naturally to broader sequential decision applications (e.g., multiplayer game and meeting assistants).

Below we summarize the new experiments addressing your shared concerns (revised text in green, new tables in blue):

1. Behavioral Quality, Consistency & Perceived Assistance (QdTQ, beXb): To verify that structural gains reflect genuine assistance rather than label overfitting, we added rollout behavioral metrics, a 300-message contradiction audit, and 560-turn human ratings (1–5 scale). Across MA-MSFT → Turn-GSPO → SM-GSPO, repetition drops (3.04% → 2.47% → 2.11%, and to 1.37% with an explicit history-repetition penalty) and cross-turn contradictions fall (3.67% → 2.33% → 1.33%), while response coverage (72.00% → 79.26%), silence precision (89.74% → 92.71%), balanced accuracy (83.93% → 85.76%), and all four human-rated dimensions (Helpfulness 4.64, Timing 4.28, Recipient 4.63, Grounding 4.70) improve monotonically.

2. Isolating MA-MSFT & SM-GSPO Contributions (QdTQ, beXb, KC7b): We added continued-supervision controls on the full RL set (including self-generated histories + teacher relabeling, peaking at 65.04 Avg.), ordinary SFT + conventional Turn-GSPO (61.93), and single-turn SFT/RL controls (where MEM/Host recall collapses to 0.00%/0.00% and 11.14%/3.81%). MA-MSFT + SM-GSPO reaches 67.41 Avg. (+2.37 over the strongest supervised control; +5.48 over conventional SFT+RL; +2.99 Avg. and +19.68 MEM recall over Turn-GSPO), validating multiturn context and trajectory-level credit assignment. Ablations also confirm marker weighting (+0.85 Avg.), auxiliary state loss (+0.82), and focal-weight detachment (62.70 → 63.01).

3. Benchmark Reliability & Class-Balanced Metrics (QdTQ, KC7b): Full-benchmark dual re-annotation confirms high reliability (κ=\kappa= 0.86/0.81/0.74; 92%/87%/83% teacher agreement), and a 560-turn history-conditioned audit shows false penalties from diverging prior actions affect only 2.14% of SM-GSPO turns. Re-evaluating with State Macro Recall and Role Macro Accuracy removes majority-class priors: Always-ANS drops to last place (33.33 / 50.00), while SM-GSPO leads all models (68.77 / 66.14).

4. Temporal Stability & 1-Hour Evaluation (QdTQ, beXb): SM-GSPO leads across all T1–T5 / B1–B5 clip segments and maintains stable State accuracy (67.22%–72.00%) across 23 continuous 1-hour livestreams (6×6\times max clip duration; 63.22%–67.06% Avg. in 0–30 min), motivating 30-minute periodic context consolidation for long-horizon deployment.

5. Joint Formulation, Generalization & Ethics (beXb, KC7b): We detail trajectory failure modes proving decision interdependence (App. D.3), map our framework to game/meeting assistants (KC7b), and clarify research-use authorization, modality-specific de-identification, and per-asset release boundaries (beXb, App. F).

Thank you again for your time and guidance. We look forward to your feedback and will actively follow up throughout the discussion period!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.