acceptodds
Under review as a conference paper at ICLR 2027

Proact-VL2: Reinforcement Learning for Open-Ended Proactive Real-Time Streaming Interaction

Abstract

As AI companions increasingly participate in shared interactive experiences, such as co-watching and commenting on a game video with a user, they must continuously process streaming inputs to determine when and how to contribute. However, existing proactive real-time streaming models primarily optimize for predefined targets via supervised fine-tuning, such as flagging specific events or answering set queries, which falls short in open-ended scenarios, as it fails to ensure that the interaction trajectory remains contextually grounded, dynamically adapts to shifting conversational dynamics, and stays well-coordinated with others. We address this gap with reinforcement learning using a reward that assesses contextual grounding, adaptability, and human-AI coordination. Experiments demonstrate that this approach outperforms supervised learning and reference-based rewards, substantially improving proactive interaction quality and reducing multimodal grounding errors while preserving general video understanding capabilities.

Primary area:
reinforcement learning
Keywords:
proactive interaction, video multimodal large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gLjtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Proact-VL2 studies open-ended proactive video interaction, where several speaking times and contributions may be appropriate. The authors construct OpenProact across solo commentary, collaborative commentary, guidance, and collaborative presentation. A weighted SFT policy based on Qwen3-VL-4B is further trained with GRPO on causal 20–50-second trajectories, using a multimodal trajectory-quality reward, MM-TrajQ, and a per-step length penalty. Evaluation includes in-domain and held-out-domain interaction, general video understanding, behavioral diagnostics, and a paired human study. The main contribution is an empirical training framework for context-sensitive proactive behavior, rather than a new RL optimizer.

Critical strengths
  1. Quality improvements have evidence beyond the optimized judge. The paired human study compares the same 20 clips across systems, with three ratings per clip–model pair. RL improves the mean human score over SFT by 2.32/10, with a paired-bootstrap interval of [1.63, 3.02], and is rated higher on 19 of 20 clips. Appendix D explains blinding, aggregation, and agreement. This is a substantive reason to view the reported behavioral improvement as more than self-evaluation.
  2. A coherent objective with a reasonably complete experimental package. Multiple valid trajectories are a concrete motivation for moving beyond reference imitation. Same-initialization reward comparisons, length-penalty ablation, 481 held-out-domain clips, and checks on general video understanding support the practical training contribution. The appendix provides detailed prompts and optimization settings and the paper supplies anonymous code/data-construction links, although their executability was not verified in this assessment.
Critical weaknesses
  1. The distinct role of trajectory-level feedback is not isolated. Prior work already includes multi-turn RL for proactive interaction (MMDuet2) and proactive commentary/guidance (Proact-VL). Here, standard GRPO assigns one trajectory advantage to all generated tokens. Comparing PAUC against MM-TrajQ changes semantic content, visual evidence, reference dependence, and temporal scope together. Thus, Table 2 shows the usefulness of this reward, but not whether modeling cross-decision dependencies is the decisive ingredient. A stepwise or short-window semantic reward using the same judge and evidence would clarify the claimed empirical insight.
  2. The scope of interaction-quality validation remains limited. MM-TrajQ is both the training reward and a main evaluation metric, while LiveU and FinalQ use the same GPT-5.2 judge. The human study meaningfully mitigates this dependence, but contains only 20 independent clips and does not establish all reward-comparison or OOD conclusions. Collaborative evaluation also replays a fixed partner track that cannot react to the model, so it measures coordination with a scripted timeline rather than fully reciprocal interaction. The conclusions should retain this scope and acknowledge trade-offs: guidance F1 decreases from 0.638 to 0.525 after RL, and OOD LiveU decreases slightly from 5.75 to 5.70. These limitations temper, rather than negate, the positive human evidence.
Questions for authors
  1. Decisive mechanism control: Is there a comparison with stepwise or short-window semantic rewards using the same judge, multimodal evidence, rollout budget, and training initialization? How much gain is specifically attributable to whole-trajectory context?
  2. Independent validation: Which interaction categories and OOD domains are covered by the 20 human-evaluated clips? Is there human or independent-judge evidence comparing MM-TrajQ-trained RL against the strongest PAUC reward variant, rather than only SFT?
  3. Interpretation of behavior: Does RL reduce speaking coverage or frequency in guidance, and how does that relate to the F1 decrease? Please clarify which coordination claims are supported under a fixed partner track, and whether any results involve a partner that responds to the model.
2 comments

W1/Q1 (The distinct role of trajectory-level feedback is not isolated)

To address the reviewer’s concern, we added a single-step semantic-reward ablation using the same SFT initialization, four rollouts per context, the same length penalty PlenP_{\mathrm{len}}, and the same number of training steps. Each judge call scores only the model’s speech or silence at second t, with frames, reference utterances, and partner speech from t−5 to t+2 seconds. This context helps identify the event and whether speaking at t is appropriate. The per-second scores are averaged into one rollout reward for GRPO. As shown in the table below, single-step scoring can favor silence because the judge cannot see how the model will complete its utterance. For example, “Keep the lid…” does not yet convey a useful instruction; its value becomes clear only with the subsequent “slightly open to let steam escape.” Even with preceding context, the judge cannot assume this continuation. The unfinished instruction therefore receives less credit than appropriate silence, whereas trajectory-level evaluation can recognize the complete contribution.

RewardF1 ↑LiveU ↑FinalQ ↑MM-TrajQ ↑
Single-step0.00041.342.941.34
Trajectory-level (ours)0.7806.325.364.84

W2 & Q3 (The scope of interaction-quality validation remains limited)

MM-TrajQ is one of several reported metrics, and our conclusions do not rest on it alone: judge-free overall F1 remains comparable (.784→.780), while LiveU (6.06→6.32) and FinalQ (4.77→5.36) improve. Although these metrics share the GPT-5.2 judge, their prior validation against human judgments on 100 instances [1], together with our own human study favoring RL on 19 of 20 clips, provides supporting evidence beyond the training reward. For collaborative interaction, the fixed partner track provides a controlled setting in which models face the same partner utterances and timing, allowing us to assess coordination while limiting variation introduced by the partner. A reactive partner would make future inputs depend on the model’s actions, complicating credit assignment and reproducibility; jointly training both agents would additionally introduce non-stationarity as each partner’s policy changes. We therefore view this work as a first step toward RL for proactive collaboration, with demonstrated gains under a fixed-partner protocol.

Q2 (Coverage of interaction settings in human study)

The 20 clips cover all four settings. Human ratings per setting:

SettingClipsOur RLOur SFTProact-VL
Solo commentary55.732.731.73
Collaborative commentary95.563.193.22
Guidance35.563.673.44
Collab. Pres.39.448.006.44
Overall206.183.873.37

We will additionally conduct a human study with pairwise comparisons among our SFT model, our RL model, and the MMDuet2-PAUC variant.

Q3 (Why RL lowers F1 in Guidance)

We acknowledge that our RL model underperforms MMDuet2-PAUC and our SFT model on F1 in Guidance, one of our four scenarios .525 (RL) vs. .620 (MMDuet2-PAUC) and .638 (SFT) and have removed the claim of uniform improvements across all four settings. The main cause is that RL speaks less in Guidance. Compared with SFT, recall drops from .789 to .481. Guidance requires instructions before specific actions, so its timing constraints may favor SFT or PAUC’s direct optimization of reference alignment. In the other three settings, our reward improves both F1 and FinalQ over both PAUC variants. These more open-ended settings admit multiple appropriate responses and speaking opportunities, which may benefit from evaluating the quality of the complete interaction rather than agreement with a single reference trajectory. We now discuss this distinction explicitly in the paper.

Human study. As suggested, we ran a blinded pairwise human study with no rubric or judge prompt. Annotators watched the same clip with two models' commentary side by side and simply chose which side is better (20 clips, 5 annotators, 150 votes).

Our RL vs.Win rate
MMDuet2 PAUC74%
Our SFT68%
Proact-VL86%

We would be grateful to know whether these results address your concerns. If anything remains unclear, we would be happy to provide further clarification or additional analyses.

Reviewer_BJd2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper formulates open-ended proactive streaming interaction as trajectory-level behavior alignment and optimizes it with reinforcement learning. A context-based reward (MM-TrajQ) is introduced to evaluate the completeness, visual grounding, adaptability, and coordination of the interaction trajectory, complemented by a length penalty to prevent unrealistic outputs. The paper also introduces the OpenProact dataset and demonstrates that the proposed RL approach improves interaction quality and reduces grounding/tracking errors compared to SFT and reference-based RL baselines.

Critical strengths
  1. The construction of the OpenProact dataset, covering four diverse interaction settings (solo/collaborative commentary, guidance, collaborative presentation), provides a solid foundation for training and evaluating proactive streaming models.

  2. The paper clearly identifies the mismatch between frame-wise SFT objectives and the multi-faceted nature of open-ended interactions. Formulating this as trajectory-level behavior alignment is a well-motivated.

  3. The paper not only shows improvements in automated metrics but also provides human evaluation and behavioral analysis.

Critical weaknesses
  1. While the length penalty is effective, the choice of the 6-word threshold and the linear penalty scaling appears somewhat heuristic. A brief ablation or discussion on the sensitivity of the model's performance to these hyperparameters would strengthen the robustness of the method.

  2. The paper uses MM-TrajQ as both the primary semantic reward for RL training (§4.2.2) and one of the core evaluation metric for assessing final model performance (§5.1, Tables 1-3). This creates a fundamental circularity: the policy is explicitly optimized to maximize the GPT-5.2 judge's scoring preferences, so it is unsurprising that it achieves high MM-TrajQ scores on the test set.

Questions for authors
  1. Could the authors elaborate on the computational overhead of using an LLM-as-a-judge during RL training? Specifically, what is the approximate time/cost contribution of the reward computation relative to the forward/backward passes?
2 comments

W1 (Are the 6-word threshold and the linear penalty scaling heuristic? How sensitive is performance to them?)

We varied the linear penalty scale across 3, 6, 12, and 18 while keeping the six-word threshold fixed. As shown below, the scale of 6 which is used in our paper achieves the lowest Overlength rate (0.03%), while F1, LiveU, and FinalQ remain broadly comparable across settings. These results support our scaling choice for controlling overlength speech while maintaining interaction quality.

Penalty scaleOverlength (%) ↓F1 ↑LiveU ↑FinalQ ↑
30.09550.7986.285.42
6 (ours)0.03000.7806.325.36
120.20980.7905.985.34
180.05050.7976.335.49

W2 (MM-TrajQ is both the RL reward and an evaluation metric, so its gains may just reflect fitting the GPT-5.2 judge.)

MM-TrajQ is one of several metrics we report, and our conclusions do not rest on it alone. (i) Judge-free metric: F1 needs no judge and does not drop after RL (.784 vs. .780). (ii) Independently validated judge metrics: RL also improves LiveU (6.06→6.32) and FinalQ (4.77→5.36). Both of the metrics were checked against human judgments on 100 instances in [1]: excluding ties, pairwise accuracy was 84.7–100% for LiveU and 74.0–96.8% for FinalQ. LLM-as-judge evaluation is standard for streaming commentary [2]. (iii) Human evaluation: our own human study also rates the RL model higher. So the gains go beyond the RL reward itself.

Q1 (What does the LLM-as-a-judge cost during RL?)

With eight groups processed concurrently per batch, judge API calls occupy an average of 88.09 s. Compared with the average RL step time of 432.8 s, this corresponds to approximately 20.4% of the total training time per step. The API cost for one complete RL training run is approximately $37.

[1] Weicai Yan et al. Proact-VL: A Proactive VideoLLM for Real-Time AI Companions. ICML, 2026

[2] Chen et al., LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale, CVPR 2025.

Thank you for the response. I will keep my original positive score.

Reviewer_zpqzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a video MLLM for open-ended proactive streaming interaction. At each second the model stays silent or says up to six words (p.4 L187-191). It covers four settings: solo and collaborative commentary, guidance, and two-presenter slide talks. The method is GRPO from a Qwen3-VL-4B SFT checkpoint. Causal 20-50 s rollouts are scored as a whole by a GPT-5.2 rubric judge (MM-TrajQ), minus a length penalty (Eq. 3-5). The paper also introduces OpenProact: 39,156 SFT clips, 1,992 RL trajectories and 2,481 evaluation clips (Tables 6-7). RL beats SFT and two PAUC-style rewards on judge scores. It wins 19 of 20 human-rated clips against SFT. It loses 1.3 to 1.4 points on MVBench and Video-MME (Tables 1, 2, 5).

Critical strengths
  1. The SFT-to-RL gain is consistent and corroborated. It holds in every setting and OOD domain (Tables 1, 4). That includes FinalQ, a different text-only rubric with no video or persona (p.22 L1172-1174), although the same GPT-5.2 scores it (p.14 L724). A model-blinded human study agrees (six raters, paired +2.32, CI [1.63, 3.02], 19 of 20 clips, alpha 0.766; App. D). This matters because it shows RL improves behavior beyond the optimized score.
  2. The causal protocol is specified and the reporting is candid. Partner speech becomes visible only at t+1 (p.5 L231-234; B.2), and the overlength exploit and its fix are reported (Table 3; E.2, E.3). This matters because the setting is reusable and the claims are checkable.
Critical weaknesses
  1. The quality evidence is tied to the reward's own judge and rubric, and to reference-derived inputs.
    • MM-TrajQ is both the RL reward and a headline metric (p.6 L314-315; p.14 L734-736).
    • GPT-5.2 scores all three metrics (p.14 L724) and also cleaned the references, personas and dialogues (p.14 L749-755; p.15 L775, L781).
    • Raters applied the judge's exact system prompt (Figure 4, p.17), which the text calls a task-specific rubric (p.16 L848-850).
    • rho = 0.917 is pooled over three systems (p.17 L915-917), but only two system means are reported (p.16 L862-863).
    • The reward judge also sees a reference trajectory (p.13 L665-667) and a training cadence target (p.24 L1246), both omitted from Section 4.2.2 (p.5 L245-247) despite contribution (2) (p.2 L075-077).
    • Gains are largest on MM-TrajQ (+1.05 in-domain, +3.10 OOD) and small on LiveU (+0.26, -0.05) (Tables 1, 4).
    • The judge-free F1 is flat (.784 vs .780) and drops in Guidance (.638 to .525), where RL also trails Proact-VL on all quality metrics (Table 1). Pooled F1 (Eq. 6) over-weights the near-ceiling synthetic presentation setting (.978).
    • Every number in Tables 1-5 comes from one run, with no seeds or CIs (p.13 L660-661).
    • This matters because "improved interaction quality" is shown only under the reward's rubric.
  2. The distinguishing claim is not isolated, and the released recipe differs from the paper.
    • Table 2's MM-TrajQ column is the proposed reward itself. MMDuet2-PAUC wins LiveU (6.73 vs 6.32; Solo 6.46 vs 4.94) and Guidance F1 (.620 vs .525), so "improves interaction quality across the four settings" (p.7 L360-361) fails against it.
    • The variants differ only in the reward component (p.7 L356-358), so apparently without the repetition and in-span terms MMDuet2 (ICLR 2026, Sec. 5.3) calls indispensable. No step-level or reference-free judge ablation is given.
    • The released reward subtracts a partner-interruption penalty (0.5) and a length penalty of 0.5 x min(1, U/10). A unit test asserts these as the active recipes' defaults (reward_function.py L2180-2262, L2619, L2634-2636; tests/test_mllm_reward_formula.py L148-157). Eq. 3 and 5 give no interruption term and a penalty about 12 times stronger: 10 excess words cost 1.0, against 0.083 in the code (p.5 L254-259; p.13 L676-680).
    • train_grpo.sh pins the judge to gpt-5_2025-08-07 (paper: GPT-5.2, p.6 L288) and defaults to adaptive KL, LR 5e-7 and 8 rollouts (paper: KL 0.01, LR 1e-7, 4 rollouts; p.13 L656-659).
    • The recipe the code names as active, rl-multijudge.yaml, is not released (data_pipeline/README.md L160-163), so neither these defaults nor the paper's values can be tied to the tables.
    • The evaluation builder appends to each in-domain and OOD clip's persona a speaking-cadence line computed from that clip's own reference turns (build_sft_eval.py L371-377; label_talkativeness.py L46-80).
    • The evaluation MM-TrajQ judge receives that line as a "binding evaluation condition" (visual_eval_prompts.py L76-80), and the rubric caps scores for cadence mismatch (p.23 L1224-1225).
    • By default, the reference speaker's pre-clip lines are injected as "[ASSISTANT]" history (prepare_data_standard_format.py L301-341). B.2 lists neither cue and states that reference responses are excluded from assistant history (p.13 L699-701).
    • The cues reach the paper's models and the Proact-VL baseline; the other baselines have no released code.
    • This matters because the recipe is the contribution and cannot be tied to the tables. The cues are shared by SFT and RL, but they favour cue-trained systems in F1 and in Table 1.
Questions for authors

Clarification (answers could change my assessment):

  1. Which reward terms and weights, judge, KL schedule, learning rate and rollout count produced Tables 1-5? Did the evaluation prompts, for the policies and for the judge, include the cadence line and the pre-clip history? Did every baseline receive them? Please report SFT, RL and Proact-VL with both cues removed.
  2. Which settings do the 20 human-study clips cover? Please report per-setting results, the within-system judge-human correlation and Proact-VL's human mean. Please also re-score the existing outputs with a judge from another model family, or run a pairwise human study with a rubric that is not the judge prompt. The released human-review app shows the judges' scores and reasons before the human score (proactvl/eval/gaming/human_eval/README.md L27-28), whereas App. D reports no access to MM-TrajQ scores or rationales (p.16 L852). Please release the App. D interface and the ratings.
  3. How were the PAUC variants ported (scorer, reply spans, reward scale, MMDuet2's auxiliary terms), and were they tuned per reward? What explains Original PAUC's F1 of .006 and RL's lower Guidance F1?
  4. What protocol produced Table 5? Its rows are below MMDuet2's own reproduction (66.4 MVBench; 58.1 Video-MME without subtitles) and below the Qwen3-VL report for 4B-Instruct (68.9; 69.3). Are OpenProact-ID videos disjoint from RL-training videos? The released builder checks only for overlapping intervals.
  5. Table 6's RL counts (443/885/590/74) equal one epoch of the released sampler over a 1,998-row pool with 1,627 distinct trajectories (249 steps vs the reported 247, p.13 L660). Distinct or exposure?

Suggestions: report seeds and CIs; release the training recipe and splits; describe the baseline adapters (F.1 shows only the policy prompt) and how Gemini 3.8 Live (Google, 15 Sep 2026) was driven; add one public proactive benchmark; cite LiveStar (NeurIPS 2025) on response-silence imbalance; consider qualifying the only-prior-RL sentence (p.7 L352-353) in light of concurrent arXiv work (StreamPro, May 2026), which I do not hold against the paper; fix the MATTR citation (Covington and McFall 2010; p.10 L508-509); state Figure 2's episode count, Table 3's denominator and how the Overall columns are computed; state data terms and annotator consent, and reconcile "Bilibili" with the YouTube and SoccerNet sources in source_contract.py L56-68.

Rating rationale.

Not a clear rejection: no reported number is shown wrong, and the SFT-to-RL direction holds under a non-reward rubric and in a model-blinded human study. Not an acceptance: the evaluator is the reward and the raters applied its prompt, Table 2 does not isolate reward design, every table is a single run, and the released code differs in reward, judge and reference-derived test-time cues. A reconciled recipe, a cue-free re-evaluation, an independent judge or human comparison, and CIs would move this to a weak acceptance.

Comments on the paper

2 comments

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has substantially improved our paper. For the reviewers' and AC's convenience, we summarize the reviews below. All revisions are highlighted in red in the updated PDF.

ReviewerBefore RebuttalAfter RebuttalResponded
zpqz22Pending
BJd233✓
gLjt33Pending

Summary of Reviews and Our Responses

We are encouraged that the reviewers found the trajectory-level formulation of open-ended proactive interaction well-motivated (BJd2, gLjt), found the SFT-to-RL gains consistent and corroborated by a model-blinded human study (zpqz, BJd2, gLjt), valued the OpenProact dataset spanning four interaction settings (BJd2) and the complete experimental package (gLjt), and appreciated the clearly specified causal protocol and candid reporting (zpqz). We summarize the main concerns and our responses below.

ConcernReviewersOur Response
MM-TrajQ is both the RL reward and an evaluation metriczpqz, BJd2, gLjtShowed that the gains hold under the judge-free F1, under LiveU and FinalQ (both previously validated against human judgments), and in human evaluation: added per-setting ratings with within-system judge–human correlation and a blinded pairwise human study with no rubric or judge prompt
Role of trajectory-level feedback not isolatedzpqz, gLjtAdded a single-step semantic-reward ablation with the same judge, evidence, and training budget
Reward comparison against PAUC and the Guidance F1 dropzpqz, gLjtRemoved the claim of uniform improvement across settings and explained the Guidance trade-off; Sec. 5.3 now discusses when reference matching is favored; human annotators prefer our RL model over MMDuet2 PAUC
Released code vs. paper recipezpqzConfirmed the configuration behind all tables (GPT-5.2 judge, KL 0.01, LR 1e-7, 4 rollouts), released the missing config, and removed unused ablation variants from the code; clarified App. A.2
Cadence line and pre-clip history at test timezpqzClarified that both are style and context inputs without timing or content of targets, shared by SFT, RL, and all baselines
Length-penalty designBJd2Reported sensitivity to the penalty scale
Cost of LLM-as-a-judge during RLBJd2Reported judge time and API cost
Protocol for general video understanding (Table 5)zpqzSpecified the shared VLMEvalKit protocol and added Video-MME with subtitles
Data splits and RL countszpqzClarified that the ID split excludes clips overlapping any training clip while OOD uses unseen videos and domains; Table 6 reports distinct clips (1,453) and sampled episodes (1,992)
Fixed partner track in collaborative settingsgLjtClarified the scope as a controlled fixed-partner protocol and stated reactive partners as future work
Related work, citations, and data sourceszpqzCited StreamPro, qualified the prior-RL sentence, fixed the MATTR citation, and listed all Bilibili sources in App. C.1

We thank the reviewers again for their valuable feedback and would be happy to provide any further clarification.

Dear Area Chair,

Thank you for overseeing the discussion. Two reviewers highlighted the motivation for trajectory-level training and the value of OpenProact (BJd2, weak accept: 3), as well as the breadth of the evaluation and the improvement over SFT (gLjt, weak accept: 3).

The third reviewer (zpqz, weak reject: 2) framed their main questions as “Clarification (answers could change my assessment),” asking which training settings produced the results, what inputs models received at test time, and whether the gains held under independent evaluation. They stated that addressing their concerns “would move this to a weak acceptance.” We answered each question, released the training configuration, and added the requested blinded human comparison. We have yet to hear back and would be grateful for your consideration of these updates.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.