acceptodds
Withdrawn from ICLR 2027

EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-time video understanding requires incrementally maintaining a memory of streaming content, and optimizing this requires dense process signals. On-Policy Self-Distillation (OPSD), which lets one model serve as both teacher and student with the teacher receiving additional privileged information such as the question and ground-truth (GT) answer, can supply such token-level signals. However, applying it directly to streaming video raises two problems. (1) The student cannot be optimized end-to-end, where memory is written before the question arrives, yet the teacher scores it with the question-and-GT privilege, misaligning their preferences. (2) Effective-entity memory collapses, where the question-and-GT privilege makes the teacher favor only question-relevant entities, and token-mean averaging over a memory renders its signal invariant to how many entities that memory covers, both driving memory against the streaming need for diversity. To address these issues, we propose Event-Grounded Self-Distillation (EGSD), which characterizes streaming memory as an incremental update over verifiable Events (key visual entities, actions, and details) and targets the two problems on this basis. For problem (1), we adapt the OPSD signal into a multiplicative weight combined with the outcome reward; for problem (2), we re-weight the teacher with Events as privileged information to counter its question-relevance bias, and add an entity-coverage reward to supply the coverage preference the token-mean teacher lacks. Extensive experiments on mainstream online and offline benchmarks show EGSD achieves strong performance, reaching 79.8% on StreamingBench and 73.4% on the OVO-Bench Real-Time track, while memory analysis shows effective-entity recall rises 17.4% at only 6.8% more memory length.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Video Large Language Models, On-Policy (Self-)Distillation, Reinforcement Learning Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yFCZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Event-Grounded Self-Distillation (EGSD) for streaming video understanding, addressing optimization misalignment and effective-entity memory collapse when applying conventional on-policy self-distillation to streaming memory. EGSD introduces Event-based supervision consisting of visual entities, actions, and details. It incorporates these facts into a privileged teacher and introduces an entity-coverage reward to encourage diverse and grounded memory. Experiments on five benchmarks, including StreamingBench, OVO-Bench, and Video-MME, using Qwen2.5-VL-7B and Qwen3-VL-8B demonstrate improvements in memory-dependent understanding. Memory analyses further show increased entity recall with limited memory-length overhead.

Critical strengths

S1. Well-motivated problem formulation and targeted memory supervision.

The paper identifies meaningful challenges in applying self-distillation to streaming video, particularly the degradation of effective-entity coverage. Its main contribution lies in introducing Event-grounded teacher supervision and an explicit entity-coverage reward. These designs directly address the identified problem and provide a meaningful extension to existing self-distillation methods.

S2. Comprehensive experiments with convincing empirical evidence.

The evaluation covers two backbone models and multiple online and offline benchmarks, with comparisons against relevant baselines. The ablation studies support the effectiveness of Event supervision, while the memory analyses demonstrate improved entity recall with modest additional memory overhead. These results provide evidence that the proposed method improves memory quality rather than merely increasing memory length.

S3. Clear presentation and illustrative case studies.

The paper is well organized and clearly written, with a coherent logical structure and informative, visually appealing figures. In addition, the case studies in the appendix provide intuitive explanations of how the proposed method preserves important visual information and improves downstream reasoning. These elements make the methodology and its benefits easy to understand.

Critical weaknesses

W1. Incomplete formulation of the optimization objective.

Equation (9) does not explicitly present the policy probability ratio required by the PPO/GRPO-style surrogate objective, making the actual gradient computation unclear. Additionally, the entity-coverage reward in Equation (11) differs from the precision-slack formulation in Appendix D.3. These inconsistencies affect technical clarity and reproducibility, although they appear potentially straightforward to resolve.

W2. Clarification of methodological contributions.

The multiplicative weighting mechanism has already been introduced in RLSD and therefore should not be considered a novel contribution of this paper. More precisely, the contributions of EGSD lie in maintaining non-decaying teacher guidance, modifying the content of self-distillation through Event-grounded supervision, and introducing an explicit reward for memory coverage. Nevertheless, I consider the insights and task-specific adaptations presented in this paper valuable.

W3. Ambiguity regarding question-independent Event supervision.

The paper motivates Events as question-independent facts, while the Event extraction prompt in Appendix D appears to incorporate the future question. Although this does not violate the student's causal inference protocol, it leaves some ambiguity regarding the extent to which Event supervision is independent of question relevance.

Questions for authors
  1. Could the authors provide the complete optimization objective for Equation (9), explicitly specifying the policy ratio, clipping operation, and stop-gradient treatment? Please also clarify the difference between Equation (11) and the implementation in Appendix D.3.

  2. Could the authors clarify how the proposed multiplicative teacher guidance differs from RLSD, particularly regarding the non-decaying weighting strategy, and more explicitly distinguish the inherited components from the novel contributions?

  3. To what extent does the future question influence Event extraction? How should the claim of question-independent Event supervision be interpreted given the extraction prompt in Appendix D?

1 comment

Thank you very much for the careful check of our formulation, which caught two places where the paper and the implementation do not match. The revision fixes both, and we answer each point in detail below.

W1 & Q1: the complete objective, and Eq. (11) against D.3

A. You are right that Eq. (9) omits the policy ratio and the clipping. It was written in abbreviated form. In the implementation, the teacher-weighted advantage A^t\hat A_t of Eq. (8) enters a standard PPO clipped surrogate with the same averaging as Eq. (9),

Lteacher=−E[1G∑i=1G1∣y(i)∣∑t=1∣y(i)∣min⁡(rtA^t(i), clip(rt,1−ϵl,1+ϵh)A^t(i))],rt=πθ(yt)πθold(yt)\mathcal{L}_{\text{teacher}}=-\mathbb{E}\Big[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y^{(i)}|}\sum_{t=1}^{|y^{(i)}|}\min\big(r_t\hat A_t^{(i)},\ \mathrm{clip}(r_t,1-\epsilon_l,1+\epsilon_h)\hat A_t^{(i)}\big)\Big],\qquad r_t=\frac{\pi_\theta(y_t)}{\pi_{\theta_\text{old}}(y_t)}

with ϵl=0.2\epsilon_l=0.2 and ϵh=0.28\epsilon_h=0.28 and no KL term. The coverage branch uses the same surrogate with AmemA_\text{mem} in place of A^t\hat A_t, restricted to memory-writing tokens. The total objective is Eq. (12) with λ=0.5\lambda=0.5 and βw=0.3\beta_w=0.3, and the revision writes Eq. (9) out in this form with the remaining details in Appendix D.

On Eq. (11), the ratio gkg_k of Eq. (10) is the zero-slack special case, while the implementation uses the slack precision of Appendix D.3, which exempts a small number of unmatched student entities before penalizing the rest. The slack of 0.5 printed in D.3 is a typo and the value used throughout is 0.25. The revision aligns Eqs. (10) and (11) with D.3 and corrects it.

W2 & Q2: what is inherited from RLSD and what is new

A. We agree, and the revision makes the boundary explicit. What EGSD inherits from RLSD is the sign-modulated multiplicative weight and its clipping, as noted in Sec. 3.2.2. Four things are new:

  • the diagnosis of why OPSD fails on streaming memory, with Sec. 3.1 identifying optimization misalignment and effective-entity collapse and Appendix E analyzing why a token-averaged teacher signal is insensitive to entity coverage;
  • the non-decaying gate, since the signal RLSD anneals away is exactly what refines credit onto visual-entity tokens late in training;
  • the Event-privileged teacher, which adds verifiable visual facts to the conventional question-and-answer privilege and so extends teacher guidance from question relevance toward visual grounding;
  • the entity-coverage reward, which targets the collapse directly.

Table 3 separates these increments on Qwen3-VL-8B: removing the decay raises the OVO average by 0.4, Event privilege by 2.0 and the coverage reward by a further 0.8. Only the weight form is inherited; the diagnosis, the non-decaying gate, the Event-privileged teacher and the coverage reward are this paper's contributions.

W3 & Q3: the influence of the future question on Event extraction

A. We agree that the original Event extraction uses the future question for fact prioritization, rather than being strictly question-independent. However, all extracted facts must be grounded in the observed clip, and the question must not introduce unobserved content. We have clarified this distinction and printed the prioritisation rules of both extractors in Appendix D.

To assess whether question-guided prioritization is necessary, we additionally evaluate a question-free extractor as a controlled ablation, while retaining the original configuration for the main experiments. The two variants share 91.4% of extracted entities. Rows 1-3 below are the paper's Qwen3-VL-8B results and row 4 is EGSD retrained on question-free facts.

MethodSBOVO-RTOVO-BTOVO-FWVMMELVBVHAvg.
SFT start76.9072.6052.8053.4063.4060.4043.1060.37
RLSD78.2073.4050.8055.7062.6060.4044.3060.77
EGSD79.8073.4056.2056.7066.5061.1046.2062.84
EGSD, question-free facts80.0873.4157.2356.3265.7461.4146.1562.91

Downstream performance is comparable, with every change within 1.1 and a seven-benchmark average of 62.91 against 62.84. EGSD's improvements therefore do not require question-guided Event selection. The recall analysis under question-free references is given in our response to Reviewer Svrj.

Reviewer_SEBYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

EGSD paper introduces a new distillation technique, namely Event-Grounded Self-Distillation, for video streaming, which points out the issue of directly adopting the conventional distillation method in the streaming setting, as the query is not given while the memory is constructed. Also, just adopting a priviledged teacher that already knows the question with its answer, will make the student model overfit to the question distribution/information. Hence, they carefully designed the distillation (OPSD) signal with multiplicative weight, and they also re-weight the teacher reasoning trace with Events as priviledge information to defend from the overfit.

Critical strengths

The challenge tackled in this paper is very meaningful and interesting. The core problem in the streaming setting is writing the memory before the question becomes availabe, which requires to preserve the information about the video yet should not be overfit to the (question-answer) pair. That is what makes simple distillation difficult, and the authors do tackle this idea with entity-coverage reward and adopting the event information, and the authors conducted various analyses with the memory behaviour which provides evidence of the effect.

Critical weaknesses
  1. What is the robust advantage of this distillation over existing methods? I do see that this distillation do improve other OPSD-based methods, but when compared to like other oneline methods there are mixed results across benchmarks, leaving me wondering what EGSD offers as a clear advantage over those existing online methods?

  2. The additional contribution of teacher re-weighting is not sufficiently diagnosted. The idea of re-weighting is interesting, but there aren't a experiment that kind of isolates this. Since the event references already provide the direct supervision, I wonder what the effect is if we remove the teacher re-weighting component (so what I mean is I want to see the ablation study of keeping the same event references, outcome reward, and entity -coverage reward, but remove the teacher reweighting).

Questions for authors
  1. Just from curiosity, how does the memory length evolve? Figure 3 shows that adding the entity-coverage reward increases both entity recall and memory length. What happens if we keep the same memory budget? Do the improvements in entity recall and QA performance still hold?
3 comments

Thank you very much for these constructive comments, in particular for asking where the advantage actually comes from. We have revised the manuscript accordingly, added the ablations you asked for, and go through them below with more detail in the updated PDF.

W1: the robust advantage over existing online methods

A. The mixed outcome is specific to the real-time online benchmarks. On the memory-dependent benchmarks (OVO Forward, VideoMME, LongVideoBench and VideoHolmes) EGSD leads the strongest online baseline VST at both scales. Sec. 4.3 (Main Results) states why the real-time picture is not uniform: EGSD "does not surpass VST on every StreamingBench subtask, as expected since its Event optimization targets cross-time memory, not single-frame perception." The advantage appears where a question has to be answered from what the memory retained rather than from the frames currently in view.

StreamingBench itself distinguishes these two cases by the evidence a question needs. OP, ATP, TR and SU can be answered from the current view without carrying anything forward (the single-frame group), whereas CR, CS, EU, PR and CT need evidence from several points in the stream and therefore depend on the written memory (the multi-frame group). The remaining benchmarks all sit on the memory side of that line: OVO Backward asks about events that have already streamed past and OVO Forward about evidence accumulated up to the question, while the three offline benchmarks are long videos read under the same streaming protocol.

Grouping the benchmarks by whether answering them depends on memory, the comparison with existing online methods reads as follows, with the best result in each column in bold and the second best underlined.

MethodSizeSBSB multi-frame groupSB single-frame groupOVO-RTOVO-BTOVO-FWVMMELVBVH
FluxMem (CVPR'26)7B76.4074.4279.5567.20----------
StreamForest (NeurIPS'25)7B77.3075.0879.3561.2052.0052.5061.40----
VST (ECCV'26)7B79.50‾\underline{79.50}74.8284.3467.2056.7054.0064.9058.0041.90
EGSD (Qwen2.5-VL)7B78.4078.9279.4370.00‾\underline{70.00}55.3054.50‾\underline{54.50}65.20‾\underline{65.20}60.90‾\underline{60.90}44.50‾\underline{44.50}
EGSD (Qwen3-VL)8B79.8078.61‾\underline{78.61}82.66‾\underline{82.66}73.4056.20‾\underline{56.20}56.7066.5061.1046.20

The multi-frame group is what EGSD optimizes, and it leads every online baseline there by about 4 points at 7B, with Event Understanding as its largest single gain. The single-frame group moves the other way: EGSD stays level with FluxMem and StreamForest but trails VST, whose design targets per-frame perception. This is the whole of the mixed StreamingBench picture, and it is the trade-off our claim predicts, since we optimize cross-time memory rather than single-frame recognition. The cost is small and real-time ability itself does not degrade, as EGSD holds the highest OVO Real-Time score in the table at both scales. On the memory-dependent benchmarks the advantage is consistent, with the largest margins on the offline long-video ones. The one exception is OVO Backward, whose overall score is dominated by HLD. Since the correct answer there is always "Unable to answer", that subtask measures abstention rather than recall of past evidence.

The advantage is therefore concentrated where memory is required, and it does not come at the cost of real-time perception.

W2: isolating the teacher re-weighting

A. We agree this is the ablation that isolates the mechanism. We ran exactly this setting with the same Event facts, outcome reward and rmemr_\text{mem} (βw=0.3\beta_w=0.3, βe=1.5\beta_e=1.5) but no teacher reweighting. Rows 1 and 3 are from Tables 1-2 and row 2 is this run.

ConfigurationSBOVO-RTOVO-BTOVO-FWVMMELVBVHAvg.
SFT start76.9072.6052.8053.4063.4060.4043.1060.37
+ outcome reward + rmemr_\text{mem}77.0671.8850.5853.7263.1161.6745.8860.56
+ teacher re-weighting (EGSD)79.8073.4056.2056.7066.5061.1046.2062.84

Sparse rewards alone add only 0.19 on average. They help LongVideoBench and VideoHolmes slightly but lower OVO Backward by 2.2 and OVO Real-Time by 0.7. A scalar per memory turn cannot say which tokens inside that turn deserve the credit, whereas ρt\rho_t in Eq. (8) rescales every token against the Event-privileged teacher while preserving the sign of the outcome advantage. Adding it brings a further 2.28 on average, led by OVO Backward at +5.6, VideoMME at +3.4 and OVO Forward at +3.0.

Two tables in the paper support the same reading. Table 4 reports the visual-entity share of rewarded tokens, 0.306 to 0.380 for EGSD against 0.204 to 0.300 for Vanilla RLSD, so the Event-privileged teacher does move credit onto grounded entities. Table 3 enables the components one at a time and gives the complementary direction: with the teacher kept, adding rmemr_\text{mem} raises the OVO average from 55.7 to 56.5 and VideoMME from 65.6 to 66.5. Token-level re-weighting is therefore what turns Event-grounded supervision into a usable signal, while the coverage reward adds a distinct and complementary gain.

Q: the gain at a fixed memory budget

A. This is the right control for a method that rewards coverage. We re-evaluated SFT, RLSD and EGSD with the per-turn cap set to 256, 384 and 512 tokens. The 512 cap is the paper's setting. An equal cap does not imply an equal realized length, so the table also reports the memory actually written as total characters per video, the unit of Fig. 3(b). Memory length and recall follow the measurement of Sec. 4.5.1.

CapMethodMemory charsRecallOVO-RTOVO-BTOVO-FWVMMEVHAvg.
256SFT start20240.42472.2751.9252.8863.1842.7156.59
256RLSD13510.39372.7150.4755.4961.9744.0656.94
256EGSD21840.50572.9655.3256.4565.9245.3359.20
384SFT start21170.43972.8452.3153.6662.9543.4757.05
384RLSD13720.39973.0551.3455.2162.8843.9257.28
384EGSD22710.51673.6255.8756.1366.8145.7459.63
512SFT start21560.44172.6052.8053.4063.4043.1057.06
512RLSD13670.40073.4050.8055.7062.6044.3057.36
512EGSD23030.51873.4056.2056.7066.5046.2059.80

At every cap EGSD leads both baselines on recall and on average accuracy, and since few turns reach the limit the realized lengths stay below the caps. The cleanest comparison therefore equalizes the memory actually written: EGSD at 256 writes 2184 characters against 2156 for the SFT start at 512, yet its recall is 0.505 against 0.441 and its average accuracy 2.14 higher. The same run also stays ahead of RLSD at 512, which writes the least memory of all and recalls the least.

Both the recall and the QA gain therefore survive a fixed memory budget, and the length-matched comparison shows they are not bought with longer memory.

Dear Reviewer,

We sincerely thank you for your thoughtful and constructive feedback, as well as the time and effort you have devoted to evaluating our work. Following your suggestions, we have carefully revised our manuscript and provided detailed responses to the comments raised. In particular, we have added a single- versus multi-frame subtask grouping of StreamingBench with analysis, the teacher-reweighting ablation you requested, and a fixed-memory-budget evaluation to better address your concerns.

As the author-reviewer discussion period is approaching its conclusion, we would be grateful if you could let us know whether any concerns remain. If you find that our responses and revisions have satisfactorily addressed your comments, we would greatly appreciate your consideration of them when finalizing your overall assessment.

Thank you again for your time, effort, and valuable feedback.

Sincerely,

The Authors

Reviewer_SvrjOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Event-Grounded Self-Distillation (EGSD) for learning textual memory in streaming Video-LLMs. Starting from an SFT model, EGSD combines the final-answer reward with multiplicative token-level weighting from a privileged self-teacher and adds an entity-coverage policy-gradient reward. The privileged teacher receives per-clip Event facts, the future question, and the ground-truth answer; Event facts are generated offline by a frozen Qwen3-VL-235B extractor and reviewer. Experiments with Qwen2.5-VL-7B and Qwen3-VL-8B report gains on StreamingBench, OVO-Bench, Video-MME, LongVideoBench, and VideoHolmes, together with improved entity recall in generated memory.

Critical strengths
  1. The paper identifies a meaningful optimization problem: a memory writer acts before the question is known, whereas a conventional privileged teacher evaluates memory with the future question and answer. Multiplicative modulation that preserves the outcome reward's sign is a plausible response to this mismatch.

  2. The study is unusually broad for an RL post-training paper. It uses two backbones, compares SFT/OPSD/RLSD/EGSD from a shared start, evaluates online and offline tasks, and analyzes both downstream accuracy and intrinsic memory behavior.

Critical weaknesses
  1. The main text describes Events as defined independently of any question, but Appendix D states that the 235B extractor is shown "the question that will later be asked" and is told to "extract FOR these" question types; the authors further state that the question guides prioritization. The resulting Event set is therefore another question-conditioned privileged signal. Entity coverage is then measured against this same question-conditioned set, so the reported 17.4% recall increase is partly circular and does not demonstrate broader, future-query-agnostic memory. This directly undermines the paper's proposed resolution of effective-entity collapse.

  2. The Event supervision and coverage reward are not independently validated. Facts and aliases come from a 235B extractor plus a same-model reviewer, while student entities are recovered by a heuristic whose agreement with a 1.7B extractor is only F1=0.615. Validation consists primarily of same-cache separation and synthetic perfect/verbose/wrong memories, not human checks of fact correctness, completeness, aliases, entity matching, or question dependence. The main RL results also lack multi-seed uncertainty. Given that the largest ablation gain comes from Event privilege, the observed improvement cannot yet be separated from noisy reward-model bias, training-question leakage, or run variance.

Questions for authors
  1. Please rerun the Event extractor without the future question and compare fact coverage, entity recall, memory length, and downstream accuracy. Does EGSD still improve over RLSD when the Event set is genuinely query-independent?

  2. On a blinded human audit, what fractions of extracted ENTITY/ACTION/DETAIL facts are correct and complete, and what are the precision/recall of the lexical entity matcher against human correspondences?

  3. How much of the gain remains if Event privilege receives facts but not the future question and ground-truth answer? This factorial ablation is needed to isolate the proposed grounding mechanism from conventional answer-aware privilege.

3 comments

Thank you very much for your valuable comments. We have revised the manuscript to incorporate your suggestions and ran the additional experiments they call for; we address each concern in detail below, with more detail in the updated PDF.

W1 & Q1: Question-conditioned Events and circular recall

A. Thank you for pointing out that our original statement makes the extractor's use of the question look like a question-conditioned signal, which would leave coverage measured against those facts partly circular. We first clarify what the question may do, then remove it and re-measure.

The box in Appendix D.1 is labeled "core rules" and prints only part of the extractor prompt. The omitted part is where the question's role is constrained, and we apologize for the confusion.

Two things cap how many Event facts a clip may keep. Every line is re-checked by a 235B verification pass and the set must fit the teacher prompt. The extractor therefore targets 6-18 fact lines per clip, well below what a clip can confirm. The same set supplies the denominator of rmemr_\text{mem}, so a budget spent on incidental entities makes the reward noisy. Some criterion must therefore decide what to keep, and the question was the cheapest proxy for salience. The prompt confines it to that role:

The question is given to you for PRIORITISATION ONLY: when the clip holds more verifiable facts than the line budget allows, keep the ones that could plausibly bear on it and drop the incidental ones. [...] never let it invent a fact — a clip in which nothing relevant is visible must still be described truthfully.

A further rule there forbids carrying in knowledge from the question's phrasing that cannot be seen. Appendix D.2 shows the effect. The car-wash question asks what action the person performs on the car's wheels, with ground truth "scrubs with a sponge". Chunk 0 records sponge use on the car body and spatial detail near the wheels, but never claims the wheels have been scrubbed; the wheel-scrubbing action appears only in chunks 1-2, where it becomes observable. The extractor does not copy the ground-truth answer into earlier clips.

We agree that the question is not the only possible criterion, since salience alone suffices. We therefore replaced the rule with "keep the most salient and specific ones [...] and drop the incidental ones", removed every other question-related passage and left the rest byte-identical. Re-extracting all 20,075 clips reproduces 91.4% of the original entities, with facts per clip rising from 16.52 to 17.08 and entities from 6.79 to 6.95. The revision adds this question-free extractor and reports both versions, keeping the question-conditioned results in Tables 1-2. Sec. 1 now states that every Event fact must be visible in the clip.

The recall gain is not circular. We re-score the same rollout memories against three reference sets: the original facts, facts extracted without the question, and that second set minus every entity sharing a content word with the question. Matcher and threshold follow the paper.

MethodOriginalQuestion-freeQ-free, question words removed
SFT start0.4410.4390.433
RLSD0.400 (−9.4%)0.393 (−10.5%)0.383 (−11.5%)
EGSD0.518 (+17.4%)0.513 (+16.8%)0.488 (+12.7%)

Brackets give the change over the SFT start in the same column, and column 1 is the recall reported in Sec. 4.5.1. The gain is +16.8% on facts extracted without the question and +12.7% once every entity with direct lexical overlap with it is removed, so it is not driven solely by explicitly question-matched entities. RLSD stays below the SFT start under every reference.

EGSD still improves over RLSD with query-independent facts. We retrained EGSD on them with all else fixed. Rows 1-3 are the paper's Qwen3-VL-8B results and row 4 is the retrained model.

MethodSBOVO-RTOVO-BTOVO-FWVMMELVBVHAvg.
SFT start76.9072.6052.8053.4063.4060.4043.1060.37
RLSD78.2073.4050.8055.7062.6060.4044.3060.77
EGSD79.8073.4056.2056.7066.5061.1046.2062.84
EGSD, question-free facts80.0873.4157.2356.3265.7461.4146.1562.91

Every change stays within 1.1, the largest being OVO Backward at +1.03 and VideoMME at −0.76, and the seven-benchmark average moves from 62.84 to 62.91. Recall and memory length are essentially unchanged as well, at 0.513 and 2291 characters for the question-free run against 0.518 and 2303 for the original EGSD. The question thus only chose among facts that must be visible anyway, and removing it changes neither the recall gain nor downstream accuracy.

W2 & Q2: independent validation of Event supervision

A. Thank you for pointing out that the Event facts are checked only by the 235B model that wrote them, and that the student-side heuristic agrees with a 1.7B extractor at only F1 = 0.615. Without human checks the gain could reflect reward-model bias, question leakage or variance.

Why F1 is 0.615. Appendix D.4 compares the heuristic with an offline Qwen3-1.7B extractor on 800 memory lines. Both give similar density (19.6 and 17.3 units per clip) and differ mainly in granularity, such as "bowl with spoon" against "bowl" plus "spoon". Such splits lower F1 without changing which entities are found, and the group-normalized advantages correlate at 0.985.

Human audit. We ran a blind audit with five annotators. They checked 3,910 Event facts and their aliases against the video, from 250 clips across the 12 corpora of our RL training set. They also judged 3,660 memory-entity pairs from 250 EGSD and 250 RLSD memories with method names removed.

Audited itemResult
Facts correct / wrong92.4% / 3.1%
Aliases naming the same entity94.6%
Salient entities missed per clip1.2
Annotator agreementFleiss' κ\kappa 0.74
Matcher precision / recall0.91 / 0.83
Recall from human labels, EGSD / RLSD0.551 / 0.436
Recall from matcher, EGSD / RLSD0.527 / 0.421

On these same memories the human-labeled gap of 0.115 is close to the automatic gap of 0.106.

Model extractor in training. We also retrained EGSD with Qwen3-1.7B as the training-time extractor for rmemr_\text{mem}.

ConfigurationSBOVO-RTOVO-BTOVO-FWVMMELVBVHAvg.
EGSD (heuristic)79.8073.4056.2056.7066.5061.1046.2062.84
with 1.7B extractor79.8873.0056.0656.7966.3060.9245.0662.57

Only VideoHolmes moves by more than 0.4, at −1.14, and the average shifts by 0.27. Recall on question-free facts is 0.491 with the 1.7B extractor and 0.513 with the heuristic, both far above RLSD's 0.393.

Question leakage. The extractor never receives the correct answer, and its prompt states "Do NOT answer the question. Do not state which option is correct". An automatic audit over 781 clips finds 0 answer-style matches and 2 clips with a verbatim option.

The low F1 therefore reflects granularity rather than disagreement, and the audit supports both the facts and the matcher. Replacing the heuristic with a 1.7B extractor has limited impact, so the gains are not highly sensitive to this implementation.

W2: run variance across seeds

A. Tables 1-2 report a single seed. We trained two further seeds for both methods and report the mean ± std over all three.

MethodSBOVO-RTOVO-BTOVO-FWVMMELVBVHAvg.Recall
RLSD78.53 ± 0.3273.30 ± 0.1750.57 ± 0.8755.93 ± 0.8762.63 ± 0.0660.33 ± 0.4044.53 ± 0.6860.830.402 ± 0.012
EGSD79.68 ± 0.1772.97 ± 1.3156.27 ± 0.1256.63 ± 0.1266.40 ± 0.1761.13 ± 0.1546.10 ± 0.4662.740.517 ± 0.002

Across three seeds EGSD stays ahead of RLSD everywhere except OVO Real-Time, where the two sit within 0.33, and outside that column its spread never exceeds 0.46. The entity-recall improvement is equally stable, with a gap of 0.115 that is far larger than either spread.

Q3: performance without question or answer privilege

A. We ran a 2×2 over whether the extractor saw the question and whether the teacher sees the question and answer on memory turns. The last column is the entity recall of Sec. 4.5.1.

Extractor saw QTeacher saw Q and answerSBOVO-RTOVO-BTOVO-FWVMMELVBVHAvg.Recall
yes (EGSD)yes (EGSD)79.8073.4056.2056.7066.5061.1046.2062.840.518
yesno79.2672.9756.5256.2765.4261.2145.2362.410.542
noyes80.0873.4157.2356.3265.7461.4146.1562.910.513
nono79.3872.6755.6857.1265.3561.3144.9562.350.535

Removing the question from the extractor alone leaves the average unchanged. Removing it from the teacher costs about 0.43, mostly on VideoMME and VideoHolmes where answers depend on reasoning over the question. Recall instead rises to 0.542 and 0.535 in the two cells that drop the teacher privilege, against 0.518 for EGSD. The bottom cell still sits 1.58 above RLSD. About 76% of the gain therefore remains when neither the extractor nor the memory-stage teacher sees the question or the answer, while the final-answer outcome reward is unchanged.

Most of EGSD's benefit therefore comes from grounding in visual Event facts, with the answer-aware teacher adding a smaller gain on the reasoning-heavy benchmarks. We would be happy to provide further details, and we hope this addresses your concerns.

Dear Reviewer,

We sincerely thank you for your thoughtful and constructive feedback, as well as the time and effort you have devoted to evaluating our work. Following your suggestions, we have carefully revised our manuscript and provided detailed responses to the comments raised. In particular, we have added a question-free Event re-extraction with recall re-scored against progressively stricter question-independent references, a five-annotator blind audit of the facts and the matcher and a 2×2 factorial over question and answer privilege to better address your concerns.

As the author-reviewer discussion period is approaching its conclusion, we would be grateful if you could let us know whether any concerns remain. If you find that our responses and revisions have satisfactorily addressed your comments, we would greatly appreciate your consideration of them when finalizing your overall assessment.

Thank you again for your time, effort, and valuable feedback.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.