acceptodds
Withdrawn from ICLR 2027

TennisRAM: Advantage-Aware Tennis Commentary through Joint Perception and State Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Advances in multimodal large language models (MLLMs) have extended tennis video understanding from event recognition to match analysis and commentary generation. However, high-speed ball motion, transient racket–ball contact, and continuous player interactions pose challenges for fine-grained perception and temporal understanding. Existing methods typically rely on specialized models to detect ball locations, player positions, and stroke events, and convert these predictions into structured representations for commentary generation. Although such representations provide explicit event cues, they often overlook continuous visual evidence, such as stroke preparation, body balance, and recovery dynamics. To address these limitations, we propose TennisRAM. We first adopt high-frame-rate video sampling to preserve rapid motion information. Then, predictions from auxiliary perception heads for ball localization, player pose estimation, and stroke recognition are used to guide spatial feature compression, improving computational efficiency while enhancing fine-grained player–ball interaction representations. To model within-rally dynamics, we further introduce a relative advantage state estimation task, where the model determines which player holds the advantage and provides supporting evidence using only the video observed up to the current time step. The estimated advantage states and their rationales are incorporated with visual features as conditions for commentary generation. For training and evaluation, we annotate 1,200 tennis rallies with relative advantage states and point-formation labels, yielding 4,322 advantage-state annotations. TennisRAM achieves 66.33% accuracy on relative advantage state estimation, with ROUGE-L and CIDEr scores of 33.11 and 44.15, respectively, for commentary generation. Ablation studies demonstrate that fine-grained perceptual guidance and conditioning on both advantage evolution and point outcomes improve commentary quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Tennis Video Understanding, Sports Commentary Generation, Advantage State Modeling, Multimodal Large Language Models, Joint Perception and Temporal Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1P68OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper generates tennis commentary with a multimodal LLM (InternVL3.5-8B). The model, TennisRAM, adds four parts:

  • Auxiliary heads that predict ball position, player pose and strokes. They are trained on labels from off-the-shelf detectors.
  • A token-compression step guided by the predicted ball and pose heatmaps.
  • A GRU summarizes the visual features every second in order to predict which player holds the advantage and tag the supporting factors.
  • Commentary generation, conditioned on the visual tokens and these predictions.

The authors also annotate 1,200 TennisVL rallies with advantage and point labels.

Critical strengths
  • Tracking who holds the advantage during a rally is a sensible, well-motivated second-order (tactical) step toward commentary that explains how a point develops, not just what happened.
  • Using perception to guide token compression is an interesting idea. In the appendix it beats feeding the heatmaps as extra tokens (Table 4). Table 2 also shows large gains from guidance, but the baseline it compares against is weak (see W2).
Critical weaknesses
  1. Some main claims do not appear to be validated:
  • The paper says the auxiliary heads learn “fine-grained representations”. But the InternVL encoder is frozen in all stages (Stage III only adds LoRA to the language decoder), so the heads cannot change its features. They work as small detectors, which are never evaluated.
  • The paper says the compression improves efficiency, but no token counts, FLOPs or latency are reported.
  • High-frame-rate sampling is presented as a contribution, but the frame rate is never given or ablated.
  • Section 6.4 says advantage conditioning helps commentary, but Table 3 (which is not mentioned in the main text) shows it alone lowers BLEU and CIDEr. (Not relevant: I assume in this section when you mention Score you refer to Point(-R) ).
  • The gains in recall quoted in Section 6.3 (pose over the ball) don’t match Table 2.
  1. The experiments and method description are not solid enough to trust or reproduce.
  • The baseline without auxiliary heads in Table 2 looks too weak. Its advantage accuracy is near the majority-class rate, and its CIDEr (5.49) is far below the no-condition model in Table 3 (37.91). A fair compression baseline without perception guidance is needed.
  • The general MLLMs are zero-shot, and some score below random on a 3-class task, which suggests parsing problems. The TennisExpert commentary scores are copied from its preprint and probably come from a different test set.
  • The test set is small (219 rallies, 2 matches), with a single run and no seeds.
  • Multi-label metrics are undefined, and there’s no annotator agreement for a subjective label.
  • It is unclear whether the reference commentary is TennisVL’s original text or LLM-generated text. A generated sample in Fig. 1 mentions a soccer “penalty kick”.
  • The method lacks tensor shapes and head definitions, and the Q-Former is not cited.
  • Design choices (layers 12/6/1, 4×4 grid, 10% bound) are never justified.
Questions for authors

Q1. Is the commentary target TennisVL’s original commentary? If so, what are the GLM-5.2 descriptions in Fig. 1 used for? If they are used as targets or inputs, conditioning on the labels they were generated from would be circular. Q2. The TennisExpert BLEU/ROUGE/CIDEr scores are copied from its paper. Were they computed on the same test rallies and reference texts as TennisRAM? Q3. What is the inter-annotator agreement for the advantage labels? Q4. Adv.-R and Point-R are multi-label tasks, and Table 1 reports “accuracy” for them. Is this exact match (all labels correct), per-label accuracy, or something else? Can you also report macro-F1 and a majority-class baseline? Q5. The paper says high-frame-rate sampling and token compression help, but gives neither value. How many frames per second do you sample, and how many visual tokens reach the LLM per second (the number of Q-Former queries)? How do FLOPs or latency compare with the default InternVL3.5 input? Q6. Can you report results over several seeds? Q7. At test time, the commentary is conditioned on predicted advantage and point labels. During Stage III training, does it see ground-truth labels or predicted ones? If ground truth, how much does commentary quality drop when predicted labels replace them at test time? Q8. In the first row of Table 2, with no auxiliary branches, what does PGSP compute? Is it plain region averaging?

Suggestions:

  1. Fine-tune plain InternVL3.5-8B on the same data, to show the gains come from the new modules and not just from fine-tuning.
  2. Feed the ball & pose outputs from the third party detectors directly. This will show whether the heads add anything.
Reviewer_STJYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TennisRAM, a tennis commentary framework that uses the relative advantage state, i.e., which player holds the initiative and why, as an intermediate representation between perception and commentary. It introduces three key components: auxiliary perception heads, which learn ball localization, player pose estimation, and stroke recognition; perception-guided spatial pooling (PGSP), which uses the predicted ball and pose heatmaps to guide visual token compression; and temporal relative advantage modeling, which predicts advantage states, rationales, and point-formation causes from video prefixes to condition commentary generation. The authors also annotate tennis rallies with relative advantage states and point-formation labels. Experiments report improvements over general-purpose MLLMs and TennisExpert on advantage estimation and most commentary metrics.

Critical strengths

S1. Meaningful research problem. The paper observes that identifying strokes such as a forehand does not reveal whether a shot is attacking or defensive, and uses the relative advantage state, whose evolution captures how the advantage is gained, maintained, neutralized, or transferred, to link local player–ball interactions with process-aware commentary. This motivation is sound and could extend to other sports.

S2. Potentially valuable new annotations. The paper augments TennisVL with perceptual and analytical annotations, namely automatically generated ball and pose labels and expert annotations of relative advantage states, supporting rationales, and point-formation causes for 1,200 rallies, the latter providing process-level supervision beyond video–commentary alignment. If released, these annotations would contribute to the field.

S3. Fine-grained perception without external preprocessing at inference. The auxiliary heads are trained with ball, pose, and stroke supervision and produce these cues internally at inference to guide token compression and commentary generation, avoiding separate perception pipelines, and the ablations report gains from each auxiliary branch.

Critical weaknesses

W1. Potential contamination of within-rally advantage labels by terminal outcomes. The terminal disadvantage rationale "Point lost" (L780–781) appears 1,200 times in Fig. 3(d), matching the number of rallies, and rationales are only annotated for non-balanced states (L152–153). This suggests that about 27.7% of the 4,332 advantage-state annotations may be terminal states whose labels can be inferred from the point winner, potentially inflating results that are meant to reflect within-rally advantage (L124); moreover, multi-label accuracy is not defined (L365–366), and no trivial baselines are reported.

W2. Insufficient isolation of the perception-guided compression gains. Disabling all auxiliary branches reduces Adv. Acc. from 61.83 to 34.14 (Table 2), close to the 33.3% chance level of three-way classification, yet the paper does not explain why the no-auxiliary baseline is so weak or how much of the gain comes from PGSP rather than other aspects of the training configuration. Since Table 4 reports only commentary metrics, the effect of PGSP on advantage estimation remains unverified, and a controlled comparison with uniform pooling under identical settings is needed.

W3. Weak evidence for advantage-aware commentary. Adding Adv.(-R) alone lowers BLEU-4 and CIDEr (Table 3), and the TennisExpert commentary scores are taken from its original paper (L372–373), where they were obtained on a 20-match test split, while the paper does not establish whether TennisRAM is evaluated on the same matches. Only n-gram metrics are reported, although TennisExpert itself reports METEOR and an Expert LLM Score covering factual accuracy and professionalism, and LLM-as-a-judge evaluation is also used in prior sports commentary work [1].

W4. Related work could be expanded. F-16 [2], which combines 16 FPS sampling with per-second token compression, and SoccerMaster [3], which pretrains a soccer-specific vision encoder with spatial supervision and uses it for commentary generation, are closely related to the proposed design and could be discussed in the related work to clarify the paper's distinct contributions.

[1] Chen J, Zeng Z, Lin Y, et al. LiveCC: Learning video LLM with streaming speech transcription at scale[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[2] Li Y, Tang C, Zhuang J, et al. Improving LLM video understanding with 16 frames per second[C]//International Conference on Machine Learning. 2025.

[3] Yang H, Rao J, Wu H, et al. SoccerMaster: A vision foundation model for soccer understanding[C]//2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2026.

Questions for authors

Q1. What are Adv. Acc. and Adv.-R on non-terminal timestamps, how many terminal and non-terminal timestamps does the test set contain, and how are multi-label accuracy and the alignment between predictions and labels defined?

Q2. On which matches is commentary evaluated, and can TennisExpert be re-evaluated on the same test set with its full set of metrics? Q3. Is the point winner in the commentary prompt ground truth or predicted at test time?

Q4. How does the no-auxiliary baseline in Table 2 compute PGSP's spatial weights without ball and pose heatmaps, and is it trained with the same schedule as the full model? How does PGSP affect advantage estimation compared with uniform pooling under identical settings?

Q5. The advantage-state accuracy (66.33%) and the number of advantage-state annotations (4,322) reported in the abstract do not match Table 1 (61.83%) and the sum of the counts in Fig. 3(c) (1,595 + 1,488 + 1,249 = 4,332); which values are correct? In addition, how do the gains in L424–427 correspond to Table 2? Score-R in Table 2 is never defined in the paper; does it refer to Point-R?

Q6. Can the authors discuss the related work above and clarify how their method differs from these approaches?

Reviewer_rB4wOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for tennis commentary that models the within-rally relative advantage state, meaning which player currently holds the initiative, together with supporting rationales grounded in visual evidence.

The method uses auxiliary supervision from ball localization, player pose estimation, and stroke recognition to learn fine-grained player-ball representations without relying on external perceptual preprocessing at inference, and it uses these cues for perception-guided visual compression.

Critical strengths

The process-level formulation is a meaningful contribution. Modeling relative advantage states and supporting rationales connects local player-ball interactions to rally progression, which is more informative than event sequences alone and better aligned with analytical sports commentary.

The auxiliary perception design is sensible. Using ball, pose, and stroke supervision during training, and then using these cues to guide visual compression rather than appending heatmaps as extra tokens, is a reasonable way to preserve task-relevant evidence under a token budget. The reported comparison with direct heatmap injection supports this design choice.

The dataset construction combines large-scale automatic perceptual annotation with manual high-level annotation. The split by match identity is appropriate for reducing temporal and player overlap, and the qualitative examples illustrate how relative advantage modeling can produce commentary that better reflects initiative changes and sustained pressure.

Critical weaknesses

The manually annotated evaluation set is small and limits statistical support. Only 1,200 rallies from 10 matches are manually annotated, and the test set contains 219 rallies from two matches. This is a narrow basis for claims about relative advantage modeling and commentary generation, especially without confidence intervals, per-match variance, or cross-event generalization analysis.

Annotation reliability is not adequately reported. Relative advantage states and supporting rationales are subjective, yet the paper does not report inter-annotator agreement for these labels, the proportion of disagreements, or how often adjudication changed labels. Without this, it is difficult to judge whether the prediction targets are stable enough for the reported gains to be meaningful.

Commentary-generation evaluation is not convincing. The proposed model has a lower BLEU score than the tennis-specific baseline, and the reported ROUGE and CIDEr improvements rely on baseline scores taken from a previous paper rather than rerun under identical settings. The paper acknowledges differences in evaluation settings and pipelines, but this makes the central commentary claim weak. There is also no human evaluation or calibrated model-based evaluation of commentary quality.

Baseline comparisons are not sufficiently controlled. The tennis-specific baseline is adapted to the advantage task, and general-purpose multimodal models are evaluated zero-shot. There is no strong baseline trained with the same auxiliary perception supervision, the same compression strategy, and the same multi-task objectives but without relative advantage modeling. This makes it difficult to isolate the contribution of the proposed relative advantage component.

Ablation coverage is incomplete. The auxiliary perception ablation is described mainly in text, and the table appears incomplete or missing full numerical results. There is no systematic ablation of relative advantage state supervision, rationale supervision, point-formation supervision, perception-guided compression, temporal modeling, or the multi-stage training design. Sensitivity to loss weights and training stages is also not reported.

Multi-label evaluation is under-specified. Advantage rationales and point-formation causes are multi-label tasks, but the paper reports accuracy and recall without clearly defining multi-label accuracy and without precision, F1, per-class results, or class-imbalance analysis. This weakens the interpretation of the rationale and point-formation gains.

The quality of automatic perceptual annotations is not validated. Ball positions and player poses are generated by existing models, but the paper does not report their accuracy on tennis videos or analyze how annotation noise affects downstream relative advantage prediction. The auxiliary perception heads are also not evaluated directly, so it is unclear how reliable the perceptual supervision is.

The temporal formulation may oversimplify rally dynamics. Relative advantage is annotated at roughly two-second intervals and predicted with a one-second interval, but the paper does not explain how state transitions near annotation boundaries are handled or evaluated. A three-way state may also be too coarse to capture rapid initiative changes in tennis.

Reproducibility and data release are unclear. The paper describes a complex three-stage training pipeline with fixed loss weights and LoRA adaptation, but does not report computational cost or sensitivity to these choices. It also does not clearly state whether the annotations, code, and generated descriptions will be released, which matters for a dataset-centered contribution.

Questions for authors

Can you report inter-annotator agreement for relative advantage states and supporting rationales, including disagreement rates and adjudication outcomes?

Can you rerun the tennis-specific baseline and strong general-purpose baselines under identical input, decoding, and evaluation settings, and explain why the commentary BLEU score is lower than the baseline?

Can you provide a complete ablation that separately removes relative advantage state supervision, rationale supervision, point-formation supervision, perception-guided compression, and temporal modeling, with all metrics reported?

How exactly is multi-label accuracy defined for advantage rationales and point-formation causes, and can you add precision, F1, and per-class results?

What is the accuracy of the automatic ball and pose annotations on tennis videos, and how much does downstream performance degrade when these annotations are noisy?

Does inference truly require no external perceptual preprocessing, or are any auxiliary models still used at inference time? Please clarify the exact inference pipeline.

Can you report variance or confidence intervals across matches or players, given that the test set contains only two matches?

Can you provide human or calibrated model-based evaluation of commentary quality to support the ROUGE and CIDEr improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.