acceptodds
Under review as a conference paper at ICLR 2027

Learning Lifetime-Aware Memory for Streaming Sports Commentary

Abstract

Streaming sports commentary requires models to identify participants, track unfolding actions, and decide when to describe an event using only the video observed so far. These decisions depend on interrelated semantic states with different useful lifetimes. Identity evidence remains relevant across events, active event states evolve as actions unfold, and completed events provide historical context. The challenge is to maintain these states over their respective lifetimes while preserving the semantic dependencies needed for accurate and timely commentary. In this paper, we propose GroundedPlay, a lifetime-aware memory framework that preserves identity evidence across events, updates active event states, and retains completed outputs as episodic context. It learns reciprocal interactions between identity and action representations, using identity evidence to refine action predictions and action context to select relevant player observations. In addition, context from the completion predictor guides evidence accumulation, while updated event states help predict whether the current event has ended. Experiments on NBA_Streaming and MatchTime-SID demonstrate that GroundedPlay outperforms state-of-the-art baselines in event response timing and commentary quality. It also achieves the highest video processing throughput among the evaluated methods. These results suggest that lifetime-aware memory and reciprocal state interactions offer a promising design approach for streaming tasks that require persistent entity knowledge, evolving event understanding, and timely responses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Sports Commentary, Lifetime-Aware Memory, Streaming Video Understanding, Online Perception

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_mKvqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes GroundedPlay, a lifetime-aware memory framework for streaming sports commentary. It maintains persistent entity memory, evolving event memory, and episodic memory to support player identification, action recognition, and timely commentary generation. Reciprocal Semantic Grounding (RSG) enables bidirectional interactions between player identities and actions, while Evidence-Completion Coupling (ECC) integrates event understanding with completion prediction. Experiments on NBA Streaming and MatchTime-SID demonstrate improvements in response timing, commentary quality, and processing throughput.

Critical strengths

The paper addresses a practical and challenging problem of real-time sports commentary requiring accurate player identification, event understanding, and timely responses.

The proposed framework integrates different semantic memory lifetimes with bidirectional interactions between identity, action, and event completion.

Extensive experiments demonstrate substantial improvements over existing baselines, particularly in CIDEr, identity accuracy, and event response timing.

Critical weaknesses
  1. Limited methodological novelty. The three-level memory organization, attention-based semantic interactions, and gated fusion mechanisms largely combine established techniques. The main contribution is their task-specific integration rather than a fundamentally new learning mechanism.
  2. Insufficient generalization. Evaluation is limited to basketball and soccer commentary, without demonstrating whether lifetime-aware memory generalizes to broader streaming video understanding tasks.
  3. Weak cross-domain evidence. Performance on MatchTime-SID remains relatively low (CIDEr 27.155, identity accuracy 5.556%, and response time difference 12.561 s), raising concerns about robustness beyond basketball.
  4. Limited analysis of memory efficiency. The paper does not sufficiently quantify memory consumption, retrieval overhead, or scalability to substantially longer video streams.
  5. Limited evidence for the episodic memory contribution. Removing episodic memory only slightly reduces CIDEr (320.016 to 312.693), suggesting that this component provides relatively modest benefits compared with entity and event memory.
Questions for authors

See the above

4 comments
AuthorsOpenReview ↗

Response to Reviewer mKvq (Part 1/2)

We thank the reviewer for the careful assessment and constructive suggestions.

W1. Limited methodological novelty.

The three-level memory organization, attention-based semantic interactions, and gated fusion mechanisms largely combine established techniques. The main contribution is their task-specific integration rather than a fundamentally new learning mechanism.

Response:

We agree that attention and gating are established building blocks. However, characterizing GroundedPlay solely as their task-specific integration overlooks how it couples memory retention, semantic grounding, and completion prediction during streaming inference. GroundedPlay uses persistent identity evidence to interpret ongoing actions, and the evolving action state helps determine when to respond. The completion predictor also guides which visual evidence is incorporated into that state.

Our methodological contribution lies in these dependencies and their associated retention rules, which determine what information persists across events and how it informs event understanding and response timing. Table 2 tests the consequences of changing the retention policies, while Table 3 examines the interaction directions. These controlled ablations support the proposed architectural and algorithmic design. The novelty claim concerns this organization of memory and inference, rather than the underlying attention or gating operators.

W2. Insufficient generalization.

Evaluation is limited to basketball and soccer commentary, without demonstrating whether lifetime-aware memory generalizes to broader streaming video understanding tasks.

Response:

This paper specifically studies streaming sports commentary, as stated in the title and contributions. We evaluate basketball and soccer to examine the proposed design across two sports, with ablations testing the effects of memory retention and semantic interactions. Evaluation beyond sports would be valuable, but would address the broader applicability of the approach rather than the task-level contribution investigated here. The abstract presents this broader applicability as a promising direction, not as an established experimental finding. Our results support lifetime-aware memory and reciprocal semantic interactions as an effective design for streaming sports commentary.

W3. Weak cross-domain evidence.

Performance on MatchTime-SID remains relatively low (CIDEr 27.155, identity accuracy 5.556%, and response time difference 12.561 s), raising concerns about robustness beyond basketball.

Response:

The absolute scores on MatchTime-SID should be considered alongside baseline performance under the same evaluation protocol. GroundedPlay achieves 3.75× the CIDEr of VideoLLM-Online, the strongest evaluated baseline on this metric. Using StreamMind as a shared reference, GroundedPlay achieves approximately 21.5× its CIDEr on NBA_Streaming and 9.4× on MatchTime-SID. These comparisons show substantial relative improvements in both sports. GroundedPlay also achieves the lowest response time difference and the highest identity and action accuracies among the evaluated methods on MatchTime-SID.

The lower absolute scores on MatchTime-SID may partly reflect differences between the two datasets. Compared with NBA_Streaming, MatchTime-SID contains more varied commentary and a larger player vocabulary. Its training and validation timestamps are automatically aligned, while test timestamps are manually aligned. These differences may contribute to the lower absolute scores. Despite the improvements over the evaluated baselines, player identification and response timing remain limitations requiring further improvement.

W4. Limited analysis of memory efficiency.

The paper does not sufficiently quantify memory consumption, retrieval overhead, or scalability to substantially longer video streams.

Response:

GroundedPlay bounds memory growth through memory-specific retention rules. Entity memory stores at most 128 entries per game. Event memory maintains a fixed-size recurrent state and action distributions, which are updated during the current event and reset at predicted completion. Episodic memory uses bounded storage and retrieves at most three completed outputs from the preceding 30 seconds.

We measured storage usage and retrieval overhead on the NBA_Streaming test set. The average retrieval time per call was 2.732 ms for entity memory and 1.155 ms for episodic memory. Their peak stored-data sizes were 0.514 MiB and 81.31 KiB, respectively. These measurements show low storage and retrieval costs on the evaluated streams. The fixed capacity limits keep memory storage bounded even for substantially longer video streams.

AuthorsOpenReview ↗

Response to Reviewer mKvq (Part 2/2)

W5. Limited evidence for the episodic memory contribution.

Removing episodic memory only slightly reduces CIDEr (320.016 to 312.693), suggesting that this component provides relatively modest benefits compared with entity and event memory.

Response:

Episodic memory complements entity and event memory by retaining completed-event history for subsequent commentary. When it is removed, the model still has access to player identity evidence and current-action information. Its ablation therefore measures the additional value of historical context.

To examine this contribution more directly, we inspected the original play-by-play records and identified 128 annotations (3.47% of the 3,693 test annotations) that were the later event in an explicit action chain, such as a steal or rebound followed by a running shot, or an offensive rebound followed by a putback. On this subset, CIDEr increases from 313.447 without episodic memory to 327.237 with it, yielding a gain of 13.790, compared with 7.323 over the full test set. Although the aggregate improvement is modest, the larger gain on explicitly linked events supports episodic memory’s complementary role in describing plays that depend on preceding actions.

The authors have addressed my concerns. Therefore, I increase my score.

Thank you for your thoughtful review and for updating your assessment. We are glad that our responses have addressed your concerns.

Reviewer_uGWfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents GroundedPlay, a memory framework for streaming sports commentary that separates persistent entity information, evolving event states, and completed commentary into entity, event, and episodic memories. Reciprocal semantic grounding and evidence-completion coupling connect identity recognition, action understanding, and commentary timing. The framework is evaluated on the NBA_Streaming basketball and MatchTime soccer benchmarks in terms of event response timing, commentary quality, and efficiency.

Critical strengths

The paper proposes a reasonable design for streaming sports commentary, a task that requires both accurate descriptions and timely responses. Separating persistent entity information, evolving event states, and completed commentary into distinct memories fits the structure of a live game, in which player identities persist while events change quickly. This is the main consideration in favor of the paper, and the evaluation on both basketball and soccer gives the design useful coverage across sports.

Critical weaknesses
  1. The central notion of the paper and the reasons its components work are not established. The term "lifetime" appears repeatedly but is never explicitly defined, so it is unclear how information lifetimes are managed beyond maintaining separate memory types with different update and retrieval rules. The ablations on memory lifetimes and on the directions of reciprocal grounding report score changes, for example higher identity and action accuracy and CIDEr when action-to-identity grounding is used, but they do not explain why these changes occur, and no identity-specific error analysis is provided. The paper is also difficult to follow overall, as its structure does not make the core mechanism clear. This is consequential because the contribution lies in why this memory design improves grounding, not only in the resulting scores.

  2. The evaluation does not yet provide convincing evidence for the reported gains. How the timing offset Δ is computed for interval-based events remains unclear even with the formula in the appendix, and precision and recall of around 70 to 80 under a 2-second tolerance are difficult to interpret given how sparse commentary-worthy events are in long-form games. The comparison includes only a few methods, and other streaming video captioning methods are not evaluated. The qualitative examples also closely match the ground truth, which is hard to reconcile with the quantitative scores, and no failure cases are shown. This is consequential because the claimed improvements in timing and commentary quality rest on these metrics and comparisons.

Questions for authors
  1. What is the operational definition of a memory "lifetime", and what are the default update and expiration rules for each memory?
  2. How is Δ computed for interval-based events, for example from the center or from both boundaries? How sensitive are precision and recall to the 2-second tolerance?
  3. Why does action-to-identity grounding improve identity accuracy, action accuracy, and CIDEr while leaving event response largely unchanged? Can the reciprocal grounding reinforce an incorrect prediction, for example when a misidentified player leads to a wrong action description?
  4. The Qwen3-VL-8B baseline runs at 12.49 fps, while GroundedPlay reaches about 240 fps. How are throughput and commentary latency each measured, and how much of the speed comes from concurrent processing rather than the proposed design?
  5. Could the authors compare with additional streaming video captioning methods?
  6. Could the authors show representative examples, including failure cases, that reflect the reported quantitative scores?
6 comments
AuthorsOpenReview ↗

Response to Reviewer uGWf (Part 1/4)

We thank the reviewer for the careful assessment and constructive suggestions.

Q1. Operational definition and rules for memory lifetimes.

What is the operational definition of a memory "lifetime", and what are the default update and expiration rules for each memory?

Response:

By “lifetime,” we mean the temporal span over which information remains relevant to its role in streaming commentary. Identity evidence can remain useful across events, active action states must track the ongoing event, and completed commentary provides context for subsequent events. Operationally, GroundedPlay reflects these different temporal roles through memory-specific rules governing when information is updated, retained, reset, or retrieved. These rules are predefined, while state updates, retrieval and fusion mechanisms, and event-completion prediction are learned.

  • Entity memory: Identity evidence persists across events without being reset at event boundaries. At each trigger, historical entries are read before eligible new observations are added, subject to redundancy filtering and capacity control. Storage is bounded to 128 entries per game and four per predicted identity. When the per-identity limit is exceeded, one entry for that identity is removed, prioritizing observations that are visually redundant and have lower quality. When the game-wide limit is exceeded, the lowest-quality entry is removed, with older entries removed first in a tie. Observation quality is the product of the highest identity prediction probability and the corresponding role-presence probability.
  • Event memory: The recurrent state and action distributions are updated with each sampled frame. When the completion predictor triggers a response, current action semantics are retained for commentary generation before active memory is reset. Its lifetime therefore follows an evidence-dependent predicted boundary rather than a fixed duration.
  • Episodic memory: Completed captions and associated metadata become available after generation finishes. At each trigger, up to three recent completed records from the preceding 30 seconds are retrieved. Records outside this window are excluded from retrieval. Thus, action information expressed in completed captions can remain useful after the active event state is reset.

We will clarify these definitions and rules in Section 3.1. “Lifetime-aware” denotes explicit modeling of these different temporal roles, rather than learning an independent expiration time for every information item.

Q2. Timing offset and tolerance sensitivity.

How is Δ computed for interval-based events, for example from the center or from both boundaries? How sensitive are precision and recall to the 2-second tolerance?

Response:

For NBA_Streaming, we use the annotated event end as the reference timestamp for interval-based events, following the benchmark’s end Δt evaluation protocol. We do not use the interval center or average the errors at both boundaries. For MatchTime-SID, we use the commentary timestamps provided by MatchTime. To compute Δt, we find the nearest predicted trigger to each reference timestamp within the same video stream, take the absolute time difference, and average over all reference events. Δt is computed over all reference events without one-to-one matching or a tolerance cutoff.

For precision and recall (P/R), we use a 2-second tolerance for one-to-one matching between predicted triggers and reference timestamps within the same video stream. A valid match allows a trigger to occur up to 2 seconds before or after the reference timestamp. Precision measures the proportion of predicted triggers that are matched, while recall measures the proportion of reference events that are matched. Unmatched predictions count as false positives, and unmatched reference events count as false negatives. To contextualize these scores, the NBA_Streaming test set contains 3,693 reference commentaries over 32.33 hours of video, averaging approximately 1.9 reference commentaries per minute. These annotations cover shots, rebounds, steals, fouls, and linked action sequences.

To assess sensitivity to the tolerance, we keep each method’s predicted triggers fixed and recompute one-to-one matching at tolerances of 1, 2, and 3 seconds. The following tables report precision and recall as percentages.

AuthorsOpenReview ↗

Response to Reviewer uGWf (Part 2/4)

NBA_Streaming:

MethodP@1sP@2sP@3sR@1sR@2sR@3s
Qwen3-VL-8B17.33629.52738.80017.05929.05538.180
Video-R29.58716.61521.5659.12515.81420.525
VideoLLM-Online28.35244.43453.70734.00253.28964.409
StreamMind20.76427.21829.90334.69945.48549.972
MATR + MatchVoice30.39745.42651.88441.29461.71170.485
MATR + LLM-IAVC30.39745.42651.88441.29461.71170.485
NBAStreaming50.07370.24075.89355.40277.71583.970
GroundedPlay60.43178.40383.18462.28080.80285.730

MatchTime-SID:

MethodP@1sP@2sP@3sR@1sR@2sR@3s
Qwen3-VL-8B4.6607.89810.60410.45117.71423.782
Video-R25.9099.89613.9629.04115.13921.361
VideoLLM-Online4.9177.4379.7114.9037.4169.684
StreamMind7.57113.59517.1048.39715.07818.970
GroundedPlay12.37319.31824.26420.47231.96440.147

Precision and recall increase as the tolerance widens, but GroundedPlay achieves the highest values for both metrics on both datasets at all three tolerances. At the stricter 1-second tolerance on NBA_Streaming, it outperforms the strongest baseline by 10.36 percentage points in precision and 6.88 percentage points in recall.

Q3. Effects of action-to-identity grounding and error propagation.

Why does action-to-identity grounding improve identity accuracy, action accuracy, and CIDEr while leaving event response largely unchanged? Can the reciprocal grounding reinforce an incorrect prediction, for example when a misidentified player leads to a wrong action description?

Response:

Action-to-identity grounding uses action context to select observations of relevant participants, helping distinguish the player performing the action from nearby players. This improves participant identification and helps the generator describe who did what. Because action accuracy is measured from generated captions, better participant grounding can also help the generator express the supplied action semantics correctly, improving action accuracy and CIDEr. Correcting participant attribution need not change the evidence for event completion, which helps explain the smaller changes in event-response metrics.

We agree that reciprocal grounding can reinforce errors. Identity-to-action refinement attends to participant features, with confidence and visual compatibility controlling the strength of the update. This limits the influence of unreliable cues, although an incorrect participant prediction made with high confidence can still mislead action prediction.

We compared GroundedPlay with the variant disabling A→I on the 2,967 NBA_Streaming test events matched by both configurations. Enabling A→I corrected 601 previously incorrect player-name predictions but also changed 262 previously correct predictions to incorrect, yielding a net gain of 339 correctly player-name predictions. For made three-pointers, 91 of 385 previously incorrect predictions were corrected (23.6%). For steals and block sequences, 27 of 98 (27.6%) and 27 of 96 (28.1%) previously correct predictions became incorrect, respectively.

The following examples illustrate a joint improvement in participant attribution and action description, an identity correction with unchanged action and timing, and an attribution error introduced when A→I is enabled.

Example 1. Correcting the shooter and assister.

Game 0022500130, fourth quarter.

ConfigurationTime (s)Caption
Without A→I27.83D. Sharpe makes a driving layup two-pointer, assisted by J. Williams.
With A→I (GroundedPlay)27.83N. Clowney makes a turnaround hook two-pointer, assisted by D. Sharpe.
Ground truth28.00N. Clowney makes a turnaround hook two-pointer, assisted by D. Sharpe.

Both configurations predict a turnaround hook before caption generation. GroundedPlay correctly identifies Clowney as the shooter and Sharpe as the assister. Its caption preserves the predicted shot type, while the ablation describes a driving layup.

Example 2. Correcting the fouler while preserving action and timing.

Game 0022500124, third quarter.

ConfigurationTime (s)Caption
Without A→I1580.00S. Fontecchio commits a personal foul.
With A→I (GroundedPlay)1580.00N. Jović commits a personal foul.
Ground truth1581.17N. Jović commits a personal foul.

Both configurations identify a personal foul and trigger at 1580.00 s. GroundedPlay correctly attributes the foul to Jović. In this example, participant identification improves while the action description and trigger time stay the same.

AuthorsOpenReview ↗

Response to Reviewer uGWf (Part 3/4)

Example 3. Failure case in steal attribution.

Game 0022500100, first quarter.

ConfigurationTime (s)Caption
Without A→I690.17M. Buzelis steals the ball from F. Wagner on a bad pass.
With A→I (GroundedPlay)690.17K. Huerter steals the ball from F. Wagner on a bad pass.
Ground truth690.50M. Buzelis steals the ball from F. Wagner on a bad pass.

Both configurations trigger at the same time and correctly describe a steal from Wagner following a bad pass. GroundedPlay incorrectly assigns the stealer role to Huerter before caption generation, and the caption carries that error forward.

Q4. Throughput, commentary latency, and concurrent processing.

The Qwen3-VL-8B baseline runs at 12.49 fps, while GroundedPlay reaches about 240 fps. How are throughput and commentary latency each measured, and how much of the speed comes from concurrent processing rather than the proposed design?

Response:

We measure aggregate throughput (FPS) on three identical GPUs, each independently processing one video stream with a batch size of 1. We divide the total number of sampled frames processed across the three streams by the total wall-clock time. The measurement includes video decoding, preprocessing, online feature extraction, trigger prediction, and complete commentary generation. We measure full-caption latency (Lat.) for each response, from the arrival of the video frame associated with the trigger to the generation of the final caption token. This includes the feature extraction, trigger computation, memory processing, and language generation required for that response.

The two methods differ in how they use the large model. The Qwen3-VL-8B baseline uses the model both to decide whether to respond and to generate commentary, so it calls the model even when no commentary is needed. GroundedPlay uses lightweight visual and semantic modules to continuously process the video, update its memories, and predict event completion. It passes scene, action, player, and historical context to the language model only when a response is triggered.

To assess the contribution of concurrent processing, we additionally evaluate Qwen3-VL-8B with concurrency enabled and GroundedPlay with concurrency disabled. On the test set, they achieve 20.63 and 167.67 FPS, respectively. GroundedPlay therefore remains approximately 8.13× faster in this comparison, indicating that its throughput advantage cannot be attributed solely to concurrent processing.

Q5. Additional streaming video captioning baselines.

Could the authors compare with additional streaming video captioning methods?

Response:

We have added comparisons with FlowNaR and ProAssist on both NBA_Streaming and MatchTime-SID. The results below cover event response, commentary quality, and inference efficiency.

DatasetMethodΔt ↓Rec. ↑Prec. ↑C ↑B-4 ↑Id. ↑Act. ↑FPS ↑Lat. ↓
NBA_StreamingFlowNaR6.40652.45139.84014.8965.6820.0799.22547.750.154
NBA_StreamingProAssist5.52958.43571.36244.92617.0180.28627.32914.821.132
NBA_StreamingGroundedPlay1.39180.80278.403320.01645.92436.59252.659239.941.060
MatchTime-SIDFlowNaR35.93912.28911.26715.5980.6150.2063.6632.770.141
MatchTime-SIDProAssist27.33029.60515.8318.2951.5700.35311.18810.731.855
MatchTime-SIDGroundedPlay12.56131.96419.31827.1554.2915.55621.471234.401.260

GroundedPlay outperforms both additional baselines on all event-response and commentary-quality metrics across NBA_Streaming and MatchTime-SID. It also achieves higher throughput in these comparisons. FlowNaR produces complete captions sooner, while GroundedPlay achieves higher commentary quality and video-processing throughput.

Reviewer_efreOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GroundedPlay, a framework for streaming sports commentary that maintains three forms of memory with different lifetimes: persistent entity memory, active event memory, and episodic memory. The method introduces Reciprocal Semantic Grounding (RSG) to connect action recognition with player identity and Evidence-Completion Coupling (ECC) to connect evolving event representations with response timing. A frozen Qwen3-VL-8B generates commentary using the resulting semantic representations and retrieved history.

Experiments on NBA Streaming and MatchTime-SID show improvements in event response, commentary quality, and player/action recognition. The authors also provide component and memory-lifetime ablations and report streaming inference efficiency.

Critical strengths

1. Meaningful problem formulation with supporting ablations.

The distinction between persistent identity evidence, evolving action states, and completed-event context is well motivated for continuous sports commentary. More importantly, the paper evaluates the consequences of these different retention policies. In Table 2, resetting entity memory after each event reduces identity accuracy from 36.592% to 30.207%, while retaining event memory across events worsens response timing from 1.391 to 1.727 seconds. These results provide useful evidence that memory retention policies should depend on the semantics being represented.

2. Substantial empirical improvements on a challenging streaming task.

On NBA Streaming, GroundedPlay improves response recall, precision, identity accuracy, and action accuracy, while increasing CIDEr from 191.525 to 320.016. The improvement is particularly notable because the system must decide when to respond rather than relying on pre-segmented events. Appendix Table 5 further compares commentary quality on commonly matched events, reducing the concern that the language-quality gains merely reflect differences in event detection. The evaluation on a second sport and the reported processing throughput add to the practical relevance of the work.

Critical weaknesses

1. The experiments do not fully isolate the benefits of the proposed memory and reciprocal interactions.

Although the lifetime ablations are useful, some component-removal results are difficult to interpret. Removing entity memory reduces identity accuracy from 36.592% to 0.388%, whereas resetting entity memory after each event still achieves 30.207%. It is unclear whether the removal ablation disables functionality beyond historical identity retention. Consequently, the large performance difference cannot automatically be attributed to persistence itself.

Similarly, Table 3 shows that the two directions of ECC contribute unequally. Event-to-trigger interaction accounts for nearly all the improvement in response time difference, while adding the reverse interaction changes it only from 1.400 to 1.391 seconds. The results support interaction between the modules, but the contribution specifically attributable to reciprocal coupling is less clear than the presentation suggests. More precise attribution would strengthen the methodological claim.

2. Response-timing evaluation and generalization require stronger qualification.

Appendix A.3 defines response time difference using the nearest prediction to each annotated event, including predictions outside the two-second matching tolerance. This is useful as a temporal proximity measure, but it is not equivalent to the delay between an event ending and the completed commentary becoming available. The handling of streams without predicted events is also unspecified.

The paper reports no game-level confidence intervals or variability across independent runs, despite evaluating NBA Streaming on 16 test games. Such estimates would help establish the reliability of the smaller timing and precision improvements.

Finally, the soccer results are substantially weaker in absolute terms, with 31.964% recall, 19.318% precision, and 5.556% identity accuracy. The improvements over the evaluated baselines remain valuable, but the evidence supports a more qualified claim of cross-sport generalization than the basketball results alone might suggest.

Questions for authors
  1. In the "w/o Entity Memory" ablation, which components are removed? Does the model retain the same current-frame identity prediction mechanism and supervision? Please clarify how this ablation isolates memory storage and retrieval from identity recognition itself.

  2. Could the authors clarify how response time difference is computed for streams without predicted events and report whether triggers tend to occur before or after the annotated endings? How does the measured timing relate to the completion time of the generated commentary, including output latency?

  3. Are paired, game-level confidence intervals or per-game results available for the main NBA comparison and the reciprocal-interaction ablations? These would help assess the reliability of the improvements, especially the smaller incremental gains.

2 comments
AuthorsOpenReview ↗

Response to Reviewer efre (Part 1/2)

We thank the reviewer for the careful assessment and constructive suggestions.

Q1. Scope of the entity-memory ablation.

In the "w/o Entity Memory" ablation, which components are removed? Does the model retain the same current-frame identity prediction mechanism and supervision? Please clarify how this ablation isolates memory storage and retrieval from identity recognition itself.

Response:

The label “w/o Entity Memory” in Table 2 did not fully convey the scope of this ablation. This setting removes the entire identity branch, including role-conditioned identity prediction from current player observations, its associated identity supervision, and the storage and retrieval of historical identity evidence. The system therefore has no dedicated player identity recognition module. The decreases in identity accuracy from 36.592% to 0.388% and CIDEr from 320.016 to 93.127 therefore reflect the removal of both identity recognition and historical memory, rather than memory storage and retrieval alone.

The “Reset Entity After Each Event” variant in Table 2 isolates cross-event identity retention by clearing entity memory at each event boundary while keeping the current-observation identity predictor and its supervision unchanged. Identity accuracy decreases from 36.592% to 30.207%, and CIDEr decreases from 320.016 to 273.662. We have clarified these definitions and revised the interpretation in Section 4.3 to distinguish the effects of removing the identity branch from those of removing cross-event retention.

Q2. Response time difference and commentary completion time.

Could the authors clarify how response time difference is computed for streams without predicted events and report whether triggers tend to occur before or after the annotated endings? How does the measured timing relate to the completion time of the generated commentary, including output latency?

Response:

We checked the complete test predictions and confirmed that GroundedPlay produced triggers in all NBA_Streaming video streams and all MatchTime-SID video streams. For a stream with annotated events but no predicted triggers, we consider the response time difference (Δt) inapplicable because there is no response whose timing can be assessed. Such streams should still be included when computing response recall (Rec.) and response precision (Prec.), with all annotated events in those streams counted as missed responses.

Among responses matched within 2 seconds, 55.36% of NBA_Streaming triggers occurred before the annotated event ending, 37.73% after it, and 6.90% at the same time. Thus, more matched triggers occurred before the annotated ending than after it. On MatchTime-SID, 46.02% of matched triggers occurred before the reference timestamp, 46.21% after it, and 7.77% at the same time, with nearly equal proportions of early and late triggers.

The response time difference (Δt) measures temporal localization error on the video timeline as the mean absolute difference between each reference event timestamp and its nearest predicted trigger timestamp. It excludes processing latency. Full-caption latency (Lat.), reported in Table 1 and defined in Appendix A.3.3, is measured from the arrival of the trigger frame until the final caption token is generated. This includes feature extraction, trigger computation, memory processing, and language generation. Thus, an early trigger does not necessarily mean that the full caption is completed before the annotated ending, since the response still requires processing time.

AuthorsOpenReview ↗

Response to Reviewer efre (Part 2/2)

Q3. Statistical reliability of the experimental results.

Are paired, game-level confidence intervals or per-game results available for the main NBA comparison and the reciprocal-interaction ablations? These would help assess the reliability of the improvements, especially the smaller incremental gains.

Response:

We provide paired analyses of the main NBA comparison and the ablations across all 16 test games. For each metric, we compute GroundedPlay minus the comparison method within each game and report the equally weighted mean difference with a two-sided 95% paired t-confidence interval. CIDEr is computed using each game’s reference captions.

GroundedPlay vs. NBA_Streaming:

Comparison MethodΔt ↓Rec. ↑Prec. ↑C ↑B-4 ↑Id. ↑Act. ↑
NBA_Streaming−0.628 [−0.975, −0.282]+3.15 [+1.53, +4.76]+7.91 [+4.58, +11.25]+118.71 [+105.03, +132.40]+11.30 [+10.06, +12.53]+18.68 [+16.12, +21.24]+11.41 [+9.25, +13.58]

Bidirectional Interaction Ablation:

Ablation ConfigurationΔt ↓Rec. ↑Prec. ↑C ↑B-4 ↑Id. ↑Act. ↑
ECC Both Off−0.597 [−0.779, −0.415]+3.70 [+1.74, +5.66]+11.67 [+9.34, +14.00]———+3.39 [+2.23, +4.55]
ECC E→T Only−0.009 [−0.054, +0.036]0.00 [−0.59, +0.59]+0.61 [−0.26, +1.48]———+0.64 [−0.27, +1.55]
ECC T→E Only−0.598 [−0.759, −0.437]+3.65 [+1.74, +5.56]+11.84 [+9.19, +14.49]———+2.91 [+1.68, +4.14]
RSG Both Off−0.019 [−0.084, +0.046]+0.08 [−0.29, +0.45]+0.64 [+0.14, +1.14]+36.62 [+25.41, +47.83]+1.83 [+0.71, +2.95]+5.53 [+3.85, +7.21]+0.46 [−0.81, +1.73]
RSG I→A Only+0.019 [−0.006, +0.044]+0.04 [−0.29, +0.37]+0.79 [+0.36, +1.22]+30.21 [+22.02, +38.40]+1.92 [+0.93, +2.91]+5.01 [+3.62, +6.40]+0.85 [−0.45, +2.15]
RSG A→I Only−0.029 [−0.092, +0.034]+0.09 [−0.07, +0.25]+0.01 [−0.42, +0.44]+12.02 [+1.68, +22.36]+0.10 [−0.90, +1.10]+0.87 [−0.26, +2.00]+0.39 [−0.99, +1.77]

Main Ablation Study:

Ablation ConfigurationΔt ↓Rec. ↑Prec. ↑C ↑B-4 ↑Id. ↑Act. ↑
w/o Entity Memory−0.023 [−0.096, +0.050]+0.17 [−0.25, +0.59]+0.65 [+0.06, +1.24]+197.08 [+173.61, +220.55]+17.20 [+15.02, +19.38]+36.22 [+31.79, +40.65]+0.41 [−0.85, +1.67]
w/o Event Memory−0.507 [−0.709, −0.305]+2.81 [+1.37, +4.25]+10.14 [+8.11, +12.17]+151.38 [+134.72, +168.04]+16.87 [+15.43, +18.31]+12.68 [+10.26, +15.10]+24.86 [+23.28, +26.44]
w/o Episodic Memory0.000 [0.000, 0.000]0.00 [0.00, 0.00]0.00 [0.00, 0.00]+7.39 [−0.43, +15.21]−0.12 [−0.89, +0.65]+0.43 [−0.49, +1.35]+0.48 [−0.46, +1.42]
Reset Entity After Each Event−0.027 [−0.175, +0.121]+0.89 [−0.10, +1.88]+0.70 [−0.42, +1.82]+41.62 [+32.63, +50.61]+2.49 [+1.44, +3.54]+6.20 [+4.68, +7.72]+0.25 [−1.33, +1.83]
Retain Event Across Events−0.336 [−0.516, −0.156]+1.79 [+0.64, +2.94]−0.12 [−1.26, +1.02]+27.58 [+19.33, +35.83]+2.01 [+1.44, +2.58]+3.12 [+1.82, +4.42]+1.72 [+0.35, +3.09]
Retrieve Earlier Events0.000 [0.000, 0.000]0.00 [0.00, 0.00]0.00 [0.00, 0.00]+15.72 [+3.49, +27.95]+1.84 [+0.37, +3.31]+2.37 [+0.61, +4.13]+1.93 [+0.36, +3.50]

The paired analysis supports GroundedPlay’s improvements over NBA_Streaming across all reported metrics. For RSG, the mean per-game paired CIDEr differences are +30.21 [22.02, 38.40] for the bidirectional model minus I→A alone, and +12.02 [1.68, 22.36] for the bidirectional model minus A→I alone. Both intervals exclude zero, supporting an additional commentary-quality benefit from reciprocal grounding.

Compared with the E→T-only variant, bidirectional ECC yields mean per-game paired differences of −0.009 [−0.054, +0.036] for Δt and +0.61 [−0.26, +1.48] for precision. These mean differences favor bidirectional ECC, although both 95% confidence intervals include zero.

Reviewer_kWWkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Streaming sport commentary requires the model to describe what happened in the correct time. To achieve this, tracking actions, keeping memories of players or other historical context is necessary. This work proposed GroundPlay, a lifetime-aware memory framework that manages memories based on the effective lifecycle of different semantic information. Player recognition, action understanding, event end detection are optimized by reciprocal interactions between identity and action representations. Experiments are done on NBA Streaming and MatchTime-SID dataset, with superior results to the baselines. Ablation studies further supports the design.

Critical strengths
  1. The paper identified a meaningful challenge in streaming sport commentary task. Different types of semantic informations have different lifetimes. The proposed method's separated memory design is well aligned with the observation and task requirements.

  2. The framework design considers the dependencies of different semantic states. RSG considers identity and action representations; ECC considers event understanding with completion prediction. These designs are reasonable.

  3. The proposed method achieves substantial improvements over existing baselines on NBA Streaming and shows promising results on MatchTime-SID. The ablation studies examine the effects of individual memory components, retention policies, and bidirectional interactions, providing empirical support for the proposed design.

Critical weaknesses
  1. The retention policies are predefined instead of learned. GroundPlay divides memory into three categories, entity, event and episodic memory. However, there are some action or information that is "in between", hard to categorize. For example, action-related context may remain relevant across multiple events, while player-related states such as possession or positioning may have dynamically changing lifetimes. It is unclear how the proposed framework handles such information without prematurely discarding useful context or retaining outdated states.

  2. The proposed framework used CLIP for visual feature extraction and Qwen3-VL-8B for commentary generation, and the experiments are conducted with one backbone configuration. It remains unclear if the proposed design generalizes to other configurations. Evaluating additional backbone configurations would strengthen the generalization claim.

Questions for authors
  1. Flexibility of memory lifetimes. The framework uses three memory types with predefined retention policies. However, some semantic information may not fit neatly into these categories. For example, action-related context may remain relevant across multiple events. How does the framework handle information with overlapping or dynamically changing lifetimes?

  2. Generalizability across backbone models. The experiments use CLIP as the visual encoder and Qwen3-VL-8B for commentary generation. How well would the proposed memory framework generalize to other visual encoders or language models? Can the authors provide additional experiments or discussion to clarify whether the improvements are backbone-dependent?

4 comments
AuthorsOpenReview ↗

Response to Reviewer kWWk (Part 1/2)

We thank the reviewer for the careful assessment and constructive suggestions.

Q1. Flexibility of memory lifetimes.

The framework uses three memory types with predefined retention policies. However, some semantic information may not fit neatly into these categories. For example, action-related context may remain relevant across multiple events. How does the framework handle information with overlapping or dynamically changing lifetimes?

Response:

The three memories have predefined retention rules, but resetting active event memory does not discard all action-related context. As described in Sections 3.1 and 3.4, current action semantics are retained for commentary generation before reset. Once generation finishes, the caption becomes available in episodic memory. At subsequent triggers, up to three completed captions from the preceding 30 seconds are retrieved as textual input to the commentary generator, making earlier action descriptions available when describing the new event. To illustrate how episodic memory preserves action information across events, we present two examples showing stored captions and comparing subsequent outputs with and without this memory.

Example 1: Steal followed by a dunk.

Game 0042500176, fourth quarter.

Previous caption stored in episodic memory

R. Sheppard misses a pullup three-pointer; A. Sengun grabs the offensive rebound after R. Sheppard's missed pullup three-pointer. - 562.83 s

T. Eason steals the ball from L. James on a bad pass. - 582.17 s

Ground truth for the subsequent event - 586.50 s

T. Eason makes a running dunk two-pointer.

Subsequent caption with episodic memory - 586.33 s

T. Eason makes a running dunk two-pointer.

Subsequent caption without episodic memory - 586.33 s

T. Eason makes a running dunk two-pointer, assisted by J. Duren.

Episodic memory retains Eason’s steal and the resulting change of possession, making this earlier action available when describing his subsequent dunk. With this action history available, the model produces the reference-matching dunk description without adding the unsupported assist attribution to J. Duren.

Example 2: Defensive rebound followed by an assist.

Game 0022500137, second quarter.

Previous caption stored in episodic memory

G. Allen misses a driving floating bank two-pointer; C. Spencer grabs the defensive rebound after G. Allen's missed driving floating bank two-pointer. - 169.00 s

Ground truth for the subsequent event - 173.50 s

J. Wells makes a running layup two-pointer, assisted by C. Spencer.

Subsequent caption with episodic memory - 172.67 s

J. Wells makes a running layup two-pointer, assisted by C. Spencer.

Subsequent caption without episodic memory - 172.67 s

J. Wells makes a running layup two-pointer, assisted by J. Williams.

Episodic memory retains Spencer’s defensive rebound and recovery of the ball, allowing the model to connect his role in the earlier possession change with the subsequent scoring play. With this action history available, the model correctly attributes the assist to Spencer rather than Williams.

Moreover, active event memory is updated through learned gates and reset at a predicted completion boundary, rather than after a fixed duration. In addition, as described in Section 3.4 and the implementation details, the scene encoder aggregates spatial features from the most recent 20 seconds. These representations are projected into continuous prompts for commentary generation, providing the model with additional visual evidence.

Nevertheless, these mechanisms do not learn an independent lifetime for every information item. Cross-event action context depends on available, retrieved captions and visual evidence within the recent window. Relevant details may become unavailable if they are absent from the retrieved captions and fall outside that window. We have explicitly acknowledged this limitation in the conclusion of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer kWWk (Part 2/2)

Q2. Generalizability across backbone models.

The experiments use CLIP as the visual encoder and Qwen3-VL-8B for commentary generation. How well would the proposed memory framework generalize to other visual encoders or language models? Can the authors provide additional experiments or discussion to clarify whether the improvements are backbone-dependent?

Response:

We have extended our experiments on NBA_Streaming to include SigLIP2 and DINOv3 as alternative visual encoders, and Llama-3-8B-Instruct and InternVL3.5-8B as alternative language models, while retaining the proposed memory architecture and interaction mechanisms.

Visual EncoderLanguage ModelΔt ↓Rec. ↑Prec. ↑
CLIPQwen3-VL-8B1.39180.8078.40
SigLIP2Qwen3-VL-8B1.54780.2980.75
DINOv3Qwen3-VL-8B1.49380.8080.45
CLIPLlama-3-8B-Instruct1.39180.8078.40
CLIPInternVL3.5-8B1.39180.8078.40
Visual EncoderLanguage ModelC ↑B-4 ↑Id. ↑Act. ↑
CLIPQwen3-VL-8B320.01645.92436.5952.66
SigLIP2Qwen3-VL-8B319.38446.38537.6652.45
DINOv3Qwen3-VL-8B319.85047.85334.4055.21
CLIPLlama-3-8B-Instruct319.88245.54336.3252.20
CLIPInternVL3.5-8B314.46545.97836.7452.22

With alternative visual encoders, response recall remains close to the original configuration and precision improves, while Δt increases from 1.391 s to 1.493-1.547 s. Commentary quality also remains similar, with SigLIP2 improving identity accuracy and DINOv3 improving action accuracy. Replacing the language model leaves event-response metrics unchanged and produces comparable commentary results. Across all tested configurations, CIDEr ranges from 314.465 to 320.016. These results support the applicability of GroundedPlay across different visual and language backbones, with some trade-offs in response timing and semantic accuracy.

Thank you for the detailed response and additional experiments.

I hope the additional experiments can also be listed in the revised version to strengthen the claims.

The clarifications have addressed my questions, and I appreciate the authors' efforts in providing further evidence to support their claims.

Thank you for your constructive feedback. We have incorporated the additional backbone experiments into Appendix A.4 and the qualitative examples illustrating cross-event action context into Appendix A.5 of the revised manuscript. We appreciate your careful review and are glad that our clarifications addressed your questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.