- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces meeting continuation forecasting and the Meeting Dynamic Forecasting Benchmark (MDFB), comprising 24,794 queries derived from 2,207 real-world meetings. It proposes GLARE, an adaptation of adversarial imitation learning where a discriminator ranks observed human continuations above samples from the current policy, and an actor optimizes the resulting reward with KL regularization. The experiments compare GLARE against SFT and other baselines, with reference-assisted human evaluation reporting direct GLARE–SFT win rates of 0.62 for utility and 0.69 for human-likeness.
Critical strengths
-
The focus on naturally occurring meetings provides a highly relevant and challenging setting for studying multi-speaker continuation. MDFB represents a substantial and valuable resource for this task. Furthermore, the strict splitting at the source-meeting level ensures a clean separation between training and evaluation, giving the benchmark lasting value independent of the specific training method proposed.
-
The human evaluation provides credible, grounded evidence that goes beyond learned reward metrics or model-based judging. The evaluation protocol is well-structured. Crucially, the direct comparison with the SFT baseline offers convincing evidence of improved perceived utility and human-likeness within the reference-assisted evaluation framework.
Critical weaknesses
-
The core training procedure and attribution of gains are insufficiently specified. While Appendix A.4 provides details on the SFT initialization and discriminator architecture, key hyperparameters for the actor update are missing. Specifically, the concrete actor-update algorithm, the numerical KL regularization coefficient, the candidate-set size, the actor learning rate and update schedule, and the exact SPIN tuning configuration are not clearly stated. These details are highly consequential, especially given that the SPIN baseline appears to substantially underperform SFT in Figure 6. Although the paper discusses unsuccessful fixed-reward/PPO/DPO attempts and provides frozen-discriminator diagnostics, this does not cleanly isolate the benefit of the refreshing mechanism. To properly attribute the gains, a comparison of final output quality is needed under strictly matched actor optimization and training budgets.
-
The significance analysis incorrectly treats repeated ratings as independent observations. Each system pair is evaluated on 50 instances by three raters, yet the analysis in Figure 6 (and related text) applies exact binomial tests to all 150 annotations under an independence assumption. In reality, these ratings share instance content and inherent difficulty, and queries may also share source meetings. While the point estimates of the win rates remain informative, these nominal statistical tests do not accurately establish uncertainty over new instances or meetings. Grouped confidence intervals (e.g., bootstrapping at the instance or meeting level) are necessary to assess the robustness of the human-evaluation gains.
Questions for authors
-
Clarification and focused validation: Can you provide the missing actor/SPIN configurations and a frozen-versus-refreshed discriminator comparison using the same actor optimizer, training budget, and final output-quality evaluation?
-
Focused reanalysis: What are the direct GLARE–SFT confidence intervals when ratings are grouped by instance and, where applicable, source meeting? Please clarify how instances and raters are reused across system pairs.
-
The paper reports unsuccessful fixed-reward training attempts and frozen-discriminator diagnostics. Under the same actor optimizer, training budget, and checkpoint-selection procedure, how do frozen and refreshed discriminators compare on the reported final output-quality measures? This would isolate the contribution of reward adaptation from differences in optimization.