acceptodds
Under review as a conference paper at ICLR 2027

What Should a Streaming Coach Remember? Motion Evidence Across Repetitions and Transitions

Abstract

A streaming coach is a vision-language model that watches workout video as it arrives and responds proactively. After it asks for a deeper squat, judging the next repetition requires the depth that prompted the correction; once the person moves on to lunges, that squat-specific guidance must no longer apply. It therefore needs movement evidence that stays available across repetitions and whose relevance changes at exercise transitions. Existing coaching models are trained on short single-exercise segments, and processing a longer stream does not itself keep earlier movement available for later feedback. We ask what a streaming coach should remember and propose StreamingCoach, which represents movement in the terms a coach reasons in and retains it independently of the feedback it generates: Form Tokens encode each chunk in anatomical joint coordinates grounded in person-centered visual features, and a gated Motion State is updated from them at every chunk, whether or not feedback is generated. On QEVD workouts processed as continuous streams without context reset at exercise boundaries, segment-trained coaches break down beyond a single exercise, and StreamingCoach improves reference-based feedback content and response timing over a streaming backbone fine-tuned under the same protocol, with both components contributing; on separately adapted probes it also recognizes the current exercise and counts repetitions substantially better. These results expose coaching demands that exercise-segment evaluation does not capture and turn what a streaming coach should remember into a testable question for streaming vision-language models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming vision-language models, human coaching, video understanding, multimodal learning, temporal reasoning, long-horizon video understanding, personalized feedback

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XThXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies fitness coaching over a continuous workout, including changes between exercises. It adds two components to a streaming vision-language model. Form Tokens combine estimated joint angles, movement speed, acceleration, and visual features. A Motion State carries information across video chunks and updates even when the coach stays silent. The model uses both to decide when to speak and what feedback to give.

Critical strengths
  1. Coaching a whole workout without resetting at exercise changes is more demanding than coaching isolated exercises. Evaluating this setting is a worthwhile contribution.

  2. Combining joint movement with visual features makes sense for form feedback. The model improves over the fine-tuned streaming backbone, and the additional form-correction experiment provides useful evidence within fitness coaching.

Critical weaknesses
  1. The method seems closely tied to fitness, since it relies on anatomical joint features and pose-based training targets. These choices make sense for fitness, but it is unclear which parts offer a broader method for streaming understanding rather than a task-specific solution. Fit3D adds another evaluation setting, but it remains within fitness. I do not think every paper needs unrelated tasks; however, the paper should make the scope of its contribution clearer and better justify its design choices.

  2. The added contribution and the memory claim need stronger evidence. Pose-grounded coaching already exists, and the state uses an attention-based recurrent update. I am not yet convinced that their combination provides a strong enough methodological advance. More importantly, the experiments do not isolate the claimed use of earlier movement. The gains in feedback and timing over the fine-tuned streaming backbone are modest, while the large recognition gains are also present without the state. A direct test of history use would make the contribution much more convincing.

Questions for authors
  1. Which parts of the method are specific to fitness, and which could apply to other streaming tasks? Could you clarify whether the main contribution is a general memory method or a fitness-specific system and evaluation setting?

  2. Can you test whether earlier movement actually changes later feedback? For example, keep the current chunk fixed but reset or swap the previous state. Does the model correctly notice improvement after a correction or a change across repetitions?

  3. What is gained over a simpler memory, such as pooling recent motion features? A matched comparison would help show why the proposed state update is needed.

  4. Can you show examples or an evaluation that directly checks comparisons across repetitions and whether earlier advice stops applying after an exercise change? The current reference-based scores do not clearly answer these questions.

6 comments
AuthorsOpenReview ↗

Response to Reviewer XThX (1/4)

We thank the reviewer for the careful reading and the concrete suggestions. We are glad the reviewer considers evaluating "a whole workout without resetting at exercise changes" to be "a worthwhile contribution," and agrees that combining "joint movement with visual features makes sense for form feedback." The reviewer's central request is a "direct test of history use." With all current evidence held fixed, we now test what the state retains, and we apply the reset and swap interventions the reviewer proposes to the likelihood of the reference feedback (Q2). The remaining part, whether generated feedback correctly reflects a change across repetitions, is specified in Q2 and will be in the revision. Results from the submission are cited by table or section; results marked new were run for this response.

Q1—Scope: "a general memory method or a fitness-specific system and evaluation setting"?

The main contribution is a motion-grounded streaming representation: it encodes movement in coordinates that are comparable across repetitions, and it retains that movement for later responses. A streaming coach must use movement that has already left its visual window, including movement it observed without responding. Form Tokens encode joint configuration and dynamics, grounded in the person's appearance (Sec. 3.3). The Motion State carries this evidence across chunks and is updated from observations whether or not a response is generated (Sec. 3.4). Auxiliary motion-prediction objectives train both alongside the coaching objective (Sec. 3.6). Between the two descriptions the reviewer offers, it is a method, and its scope is human movement, which is broader than fitness.

PartWhat it requiresScope
Form Tokens and auxiliary motion targets (Sec. 3.3, 3.6)A visible human body and a pose estimator. No exercise labels and no exercise-specific rulesHuman movement; evaluated on fitness
Motion State update (Sec. 3.4)Per-chunk tokens and a validity score (Eq. 1). No exercise identity, repetition segmentation, or boundary signalReusable interface; evaluated with Form Tokens as its input
Decoder interface (Sec. 3.5)A decoder-based streaming VLM; not its tokenization (Sec. 5)Reusable interface; evaluated on StreamingVLM
Continuous-workout protocol (Sec. 4.1); paired-history test (Q2, new)QEVD workouts; two histories that share a current windowProtocol: fitness. Test design: reusable

The motion encoding and the state update are defined once for all exercises: they use a human-motion prior and no exercise-specific rules. The reviewer points to the "anatomical joint features and pose-based training targets." These are a prior about the human body. The same joint coordinates describe any human movement, and they place a squat and a lunge in one coordinate system. At inference the method needs no hand-coded exercise rules, no label of the exercise in progress, no repetition segmentation, and no boundary signal (Sec. 3.4); the instruction lists the planned exercises without indicating which is underway (Sec. 4.1). This is what lets one update mechanism operate across repetitions and across exercise changes. The state update and the decoder interface take tokens and a validity score and make no reference to the body; we have evaluated them with Form Tokens as input.

The experiments assess this representation in complementary ways. Coaching on QEVD evaluates the complete system (Tables 1–2). Recognition, counting, and form correction on Fit3D evaluate the representation after task-specific adaptation with low-rank adapters (Tables 1 and 3). The paired-history test freezes the coaching model and trains only a lightweight readout, which tests what the state already retains (Q2, new). EgoExo-Fitness evaluates the coaching model on a third source without training or adaptation (new; form-error F1 .114 vs. .099 for the matched backbone; protocol in our response to Reviewer g2Vs, Q2). All of this evidence is on human movement in fitness. Like the reviewer, we do not think the paper needs "unrelated tasks" to make its point, and the revision will state this scope in the introduction and the conclusion.

AuthorsOpenReview ↗

Response to Reviewer XThX (2/4)

Neither existing line has both properties: a coach that runs over a whole workout and is grounded in the person's movement. The reviewer notes that pose-grounded coaching and attention-based recurrent updates both exist. The released pose-grounded coach, BioCoach, is trained on single-exercise segments and designed to be told the exercise (Sec. 4.2). It reaches T-F .544 on such segments (Table 2) and .068 when the workout continues without resets (Table 1). The streaming backbone fine-tuned on the same workouts does sustain its responses across the workout (T-F .606), yet it names the exercise correctly in 25.6% of clips and miscounts repetitions by 3.29 on average (Table 1). StreamingCoach runs continuously like this backbone (T-F .613) and is grounded in movement (82.4% and 1.08 on the two probes). What it adds to a streaming VLM is a motion-grounded representation and a persistent state, in three respects. Movement is encoded in anatomical coordinates shared by every repetition and every person, and the exercise in progress need not be given (Sec. 3.3). The state is written from these tokens at every chunk, whether or not a response is produced, and never from generated text (Sec. 3.2). The representation is trained with motion-prediction objectives in addition to the language objective (Sec. 3.6). The streaming memories discussed in Sec. 2 do not represent movement as joints in comparable anatomical coordinates (Sec. 1).

The two design choices the reviewer names, and the results behind them. Anatomical joint features express movement in units where "a slightly shallower squat is a measurable change rather than a few pixels" (Sec. 1). Adding the Form Tokens raises exercise recognition from 25.6% to 82.2% and lowers count MAE from 3.29 to 1.52 (Table 1). Removing their derivatives or their person-region grounding lowers METEOR and T-F (.120/.583 and .118/.574 vs. .123/.589; new, same training setup). Pose-based training targets are there because the language objective "can be partly satisfied by language priors and response history alone" (Sec. 3.6). They act only in training, with weight 0.1, and are discarded at inference. The revision reports the model trained without them on the paired test of Q2.

Q2—"Can you test whether earlier movement actually changes later feedback?"

With all current evidence held fixed, the frozen coaching state retains which movement came before: 64.9% against 50% (new). The design follows the reviewer's proposal: the current input is fixed and only the previous state differs. It tests the retention part of the request; the effect on the feedback is measured by likelihood below. It is also the test that Sec. 5 of the submission calls for ("remains to be isolated by intervention").

Design. From each of the 74 test workouts we build one pair of streams, A → B → W and B → A → W. A and B are eight-second clips of two different exercises. W is an identical four-second window. It fills the visual window of the model, which holds four one-second chunks (Appendix A), and the model has to carry the earlier evidence across it. We ask which exercise immediately preceded W. A supervised two-way readout, trained on the training workouts with the coaching model frozen, reads the answer from the Motion State at the end of W. The inputs for W are computed once and reused in both streams, so that the two streams differ only in the state that enters W. A predictor that receives only the current inputs gives the same answer twice and scores 50%. This test uses the second-seed checkpoints of StreamingCoach and the GRU.

Accuracy [95% CI]Predictions differ across the two historiesBoth correctBoth wrong
Current inputs only (by construction)50.0%0 / 7400
GRU memory in place of our update59.5% [54.7%, 64.2%]16 / 74151
StreamingCoach64.9% [58.8%, 70.9%]26 / 74242

Intervals come from a bootstrap over the 74 workouts that keeps both streams of a pair together.

Earlier activity stays recoverable after it has left the visual window. StreamingCoach is 14.9 points above the reference (95% CI [8.8, 20.9]). Its prediction differs across the two histories in 26 of the 74 pairs, and both predictions are correct in 24 of these 26. The reviewer's two interventions have determined outcomes in this paired design: with the state reset before W, the prediction is the same for both histories (50%), and with the two states exchanged, the predictions are exchanged (35.1%, the mirror of 64.9%). The coaching model was never trained on this question; the readout reads what coaching training left in the state. In the other 48 pairs the prediction is the same for both histories. Retention is partial, and the revision will report both numbers.

AuthorsOpenReview ↗

Response to Reviewer XThX (3/4)

The likelihood of the reference feedback drops when the state is removed or frozen. Using the second-seed StreamingCoach checkpoint, we intervened on the state over 338 test windows, with the current chunk and the text unchanged, and measured the loss of the reference feedback. Resetting the state raises the loss by 1.9% on turns with a response and by 2.8% on turns without one. Freezing it, which stops new observations from being written, raises the loss by 1.3% and 2.2%. A state taken from another workout at the same elapsed time changes the loss little (+0.1% and +0.4%). The decoder's predictions depend on the state and on its updates. The small effect of the exchanged state means that the loss does not tell which history the state holds, and we rely on the paired test for that.

The same paired design extends to repetitions and corrections. The reviewer asks whether the model can "correctly notice improvement after a correction or a change across repetitions." The present test shows that the state carries the preceding activity across the four-second window. It does not measure how far back retention reaches, and it does not compare form across repetitions. For repetitions, the state lowers the error of the running count from 1.52 to 1.08 (Table 1), and a count cannot be read from the current window. For form, the paired design applies directly: the same current repetition, preceded by deeper or by shallower ones, with the generated feedback rated blind. QEVD does not annotate how execution changed after a correction; we will derive these labels from the joint-angle traces and report this test in the revision. The revision will claim what each test shows: retention of earlier movement now, and judgment of improvement once that test is in.

The two components have separate roles, and each has a targeted result. The reviewer observes that "the large recognition gains are also present without the state." This is the intended division, and Sec. 4.2 states it: the recognition gain comes from the motion input (25.6% to 82.2% without the state, Table 1). The role of the state is retention, and the paired test measures it with the current input fixed. The coaching scores improve over a backbone that receives the same Stage-2 training without motion modules in both seeds: +.010 METEOR and +.007 T-F (Table 1; the StreamingVLM row there also received Stage 2, which Appendix A omits and we will correct), and +.009 and +.018 (new, Q3 table). The reviewer's Q4 makes the same point about reference-based scores; the targeted tests are where the effect of memory can be read.

Q3—"What is gained over a simpler memory, such as pooling recent motion features?"

We ran the matched comparison using second-seed checkpoints. The proposed update gives the highest coaching scores of the four memories, and both learned recurrent states retain the earlier activity (new). Three simpler memories replace the gated update: an exponential moving average of the motion features (α = 0.25), a FIFO buffer that keeps one mean-pooled summary per valid chunk for the last eight, and a GRU. The EMA and the FIFO are two forms of pooling recent motion features. Everything else is shared: the backbone, the Form Tokens, the direct pathway, eight 256-dimensional slots, the 640 training windows, and the optimization budget. The table also lists two trained controls: the backbone without motion modules, and a model whose state is cleared at every chunk.

MemoryMETEORROUGE-LBERTScoreT-FPaired test (Q2)
Control: StreamingVLM + Stage 2, no motion modules.114.086.850.571n/a
Control: state cleared at every chunk (trained this way).121.094.851.57450% by construction
EMA.115.089.851.575not yet run
FIFO, last 8 valid chunks.119.094.851.572not yet run
GRU.115.086.850.57559.5% [54.7%, 64.2%]
Gated state (StreamingCoach).123.096.852.58964.9% [58.8%, 70.9%]

The cleared-state control is a separately trained model. Its state at the end of W depends on the last chunk only, which is identical in both streams; this is a different condition from resetting the state of StreamingCoach at inference (Q2).

Coaching scores: the gated state is highest on all four metrics. Its margin over the two pooling memories is .014 to .017 in T-F and .004 to .008 in METEOR. The pooling memories stay close to the control without memory: within .002 T-F of it (.575 and .572 vs. .574), and below it in METEOR (.115 and .119 vs. .121). The gated state is .015 T-F above that control. Each of these margins comes from one training run.

Reviewer_AtQYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies proactive streaming coaching over complete workouts rather than single-exercise segments. Built on StreamingVLM (Qwen2.5-VL-7B), the method adds two more modules. Form Tokens encode HSMR joint angles and their derivatives, grounded in person-region visual features. A gated recurrent motion state is updated from these tokens at every chunk and is never written from generated feedback. Under a continuous-workout protocol on QEVD, segment-trained coaches like FitCoach and BioCoach largely stop responding after the first exercise. The proposed model improves over a QEVD-fine-tuned streaming VLM modestly on content and timing metrics, and substantially on separately adapted exercise-recognition and counting probes.

Critical strengths
  1. The continuous-workout protocol is a useful contribution. It exposes a failure of existing coaching models that the segment protocol cannot reveal.
  2. The paper correctly notes that a text-based history records the correction issued by the coach but not the movement that prompted it, which is is a relevant limitation of current streaming VLMs.
  3. The state design is principled. The state is written only from observations, validity gating preserves it exactly when no valid pose is available, and write dropout removes the update count as a timing cue.
Critical weaknesses
  1. The biggest weakness: effect sizes are small and lack statistical support. The improvements over StreamingVLM are 0.007 in T-F and 0.010 in METEOR. Differences between ablations are mostly in the third decimal place. All results come from a single run. Results over multiple seeds and bootstrap confidence intervals over workouts are needed to support the conclusions.

  2. The comparison with StreamingVLM is confounded. StreamingCoach receives an additional training stage that includes long preceding context, tolerance-window masking, and further updates to the top four decoder layers. The baseline receives none of these. A stage 2 run without motion inputs is required to attribute the gains to the proposed modules. Can you also clarify whether the StreamingVLM row in Tab. 1 corresponds to the Stage 1 model?

Questions for authors
  1. Does the StreamingVLM row in Tab. 1 correspond to the Stage 1 model? If not, how do the two differ?
  2. Can the authors report results over multiple training seeds, together with confidence intervals over the 74 workouts?
  3. How are predictions matched to references when computing content metrics, and how are silent segments handled? Please report temporal precision, recall and response rate separately for each method.
  4. What accuracy does zero-shot Qwen2.5-VL-7B obtain on the exercise probe? What proportion of StreamingVLM's answers fail the name-normalization mapping? Could the authors provide additional experiments to clarify the points raised above?
4 comments
AuthorsOpenReview ↗

Response to Reviewer AtQY (1/4)

We thank the reviewer for the careful assessment and the precise questions. We are glad the reviewer finds the continuous-workout protocol "a useful contribution" and the state design "principled." The reviewer's first main request is already met by Table 1: its StreamingVLM row is itself a "stage 2 run without motion inputs" (Q1). For the second, we report a second training seed and per-workout statistics (Q2); paired bootstrap intervals over the 74 workouts will follow in the revision. Results from the submission are cited by table or section; results marked new were run for this response.

Q1—"Does the StreamingVLM row in Tab. 1 correspond to the Stage 1 model? If not, how do the two differ?"

No. The StreamingVLM row received the same Stage 2 as StreamingCoach, without the motion modules. The reviewer lists what Stage 2 adds: "long preceding context, tolerance-window masking, and further updates to the top four decoder layers." The baseline of Table 1 received all three. After first-stage fine-tuning with the Stage-1 recipe and budget (Appendix A), it was trained on the same Stage-2 supervision windows as StreamingCoach (640 windows, 160 steps), with the same preceding context, the same ±1.5 s tolerance window, and the same updates to decoder layers 24–27 and the final normalization, under the language objective alone. Appendix A describes only the first stage of this baseline, which is why it reads as a Stage-1 model. We will correct the description and name the row "StreamingVLM + Stage 2."

Relative to the Stage-1 model, the row has received the whole of Stage 2. Relative to StreamingCoach, it lacks what the paper proposes: the motion adapter and the state module, their inputs, and the auxiliary motion objective that trains them (Sec. 3.6). Table 1 therefore compares the motion design with a backbone that received the same additional training. The same holds for its probe columns: 25.6% and 82.4% are both from Stage-2 models. A run with the modules and without the auxiliary objective, which separates the two, will be in the revision.

A second seed repeats this control and reports each stage separately (new). In a second run with a different seed we report the backbone before Stage 2, the backbone after Stage 2 without motion modules, and a stricter control that retains the visual input and the added token interface and replaces the pose-derived Form and State tokens with learned constants. Within a seed, all models trained with Stage 2 share one configuration and differ only in the stated change. Across seeds, we vary initialization, training-data order, and dropout randomness.

ModelMETEORROUGE-LBERTScoreT-F
Seed 42 (submission, Table 1)
StreamingVLM + Stage 2, no motion modules.122.102.858.606
StreamingCoach.132.107.864.613
Second seed (new)
StreamingVLM before Stage 2.104.067.848.595
StreamingVLM + Stage 2, no motion modules.114.086.850.571
Constant Form and State tokens.121.093.851.574
StreamingCoach.123.096.852.589

With Stage 2 matched, StreamingCoach is higher on all four metrics in both seeds. The gains are listed in Q2. The second seed also shows what Stage 2 does by itself. In the backbone it raises the content scores and lowers timing (METEOR .104 to .114, ROUGE-L .067 to .086, T-F .595 to .571). StreamingCoach receives the same stage and ends at T-F .589: .018 above the backbone after Stage 2 and .006 below the checkpoint before it, with METEOR higher than that checkpoint by .019 and ROUGE-L by .029. Against the stricter control, which has the same added tokens without pose-derived content, StreamingCoach is again higher on all four metrics (+.002 METEOR, +.003 ROUGE-L, +.001 BERTScore, +.015 T-F). The component controls of the same run are in our response to Reviewer g2Vs (Q3); each lies between the matched backbone and the full model on METEOR, ROUGE-L, and T-F. Differences of this size need the statistical support the reviewer asks for, and Q2 addresses it.

AuthorsOpenReview ↗

Response to Reviewer AtQY (2/4)

Q2—"results over multiple training seeds, together with confidence intervals over the 74 workouts"

The gain repeats across seeds, across workouts, and across feedback types. As requested, we trained a second seed for the matched backbone, its controls, and StreamingCoach (table in Q1). The gains on the aggregate scores are small in absolute value, as the reviewer notes. We claim them as consistent, not as large, and the three paragraphs below give the evidence.

Gain over the matched backboneMETEORROUGE-LBERTScoreT-F
Seed 42 (Table 1)+.010+.005+.006+.007
Second seed (new)+.009+.010+.002+.018

Seeds. All eight differences are positive, and the METEOR gain is nearly the same in the two runs. Absolute scores differ between the seeds by more than the gain (T-F .606 and .571 for the same matched backbone). The controlled comparison is the one within a seed, where the models share the supervision windows and every training setting, and it favors StreamingCoach in both. Two seeds show the direction. They do not yet show how much the size of the gain varies (the T-F gain is +.007 in one and +.018 in the other). A third seed for both models will be posted here once it finishes, and the revision will report mean and standard deviation over the three.

Workouts. In the submitted run, StreamingCoach has the higher T-F in 63 of the 74 workouts and the higher METEOR in 50 (Sec. 4.2). As shares of the workouts these are 85% (Clopper–Pearson 95% interval: 75% to 92%) and 68% (56% to 78%). Both intervals exclude 50%, and an exact sign test that counts every other workout as a loss gives p < 10⁻⁸ and p < 0.004. These numbers show how often StreamingCoach is ahead. The size of the pooled difference and its uncertainty are what the reviewer's bootstrap measures. We will post it in the revision for every metric and both seeds: workouts are resampled with replacement, both models are scored on the same resampled workouts with the scorer of the paper, and the interval is taken on the difference. The per-workout counts of the second seed will be posted with it.

Feedback types. In the submitted run, the METEOR gain is present for each of the four types of reference feedback: announcements, counts, corrections, and praise (table in Q3).

The targeted measures show differences far beyond the third decimal. The reviewer notes that differences between ablations are "mostly in the third decimal place." On the aggregate coaching scores this is the case. The revision will no longer order the two ablations by these scores, as Sec. 4.2 currently does; it will report them with intervals and rest its statements about each module on three targeted measures. First, with the Form Tokens and no state, exercise recognition rises from 25.6% to 82.2% under the same probe adaptation (Table 1; the scoring of this probe is examined in Q4). Second, with the state added, the error of the running repetition count at the 132 probe moments (Sec. 4.1) falls from 1.52 to 1.08 (Table 1). Third, Sec. 4.2 notes that the ablation cannot tell whether the state's contribution "comes from retained history or from its encoding of the current chunk," and a new test separates the two (new). From each of the 74 test workouts we build two streams that end in an identical four-second window, which fills the model's visual window (Appendix A), and differ only in the order of the two eight-second clips before it. A readout trained on frozen second-seed coaching states is asked which exercise came immediately before. Without memory the accuracy is 50% by construction. StreamingCoach reaches 64.9%, with a 95% bootstrap interval over the 74 workouts of [58.8%, 70.9%]; its prediction differs between the two histories in 26 pairs and is correct for both in 24 of these. The state, whose design the reviewer calls "principled," does carry the preceding activity across the visual window. The two probes of Table 1 come from one run and are reported without intervals; the revision will add intervals over workouts for both.

AuthorsOpenReview ↗

Response to Reviewer AtQY (3/4)

Q3—"How are predictions matched to references when computing content metrics, and how are silent segments handled?"

Content is scored on time-matched pairs, and timing on every response of the workout. The protocol is as follows.

Matching. We use the scorer of the public FitCoach release (Panchal et al., 2024), run over each full workout. A response and a reference feedback event can match when they are less than 1.5 s apart, which is the 3-s window of Sec. 4.1. Matching is one-to-one and follows temporal order: an event is paired with the nearest event of the other kind, and the pair is accepted only if it lies within this window, does not reuse an event that is already matched, and keeps the matched pairs in chronological order. The scorer makes two such passes. The pass from the responses gives temporal precision, the share of all responses of the workout that are matched. The pass from the references gives temporal recall, the share of the 2,511 reference events (Sec. 4.1) that are matched. The two passes are separate matchings, and their numbers of matched pairs can differ. Counts are pooled over the 74 workouts, and T-F is the harmonic mean of the pooled precision and recall.

Content. METEOR, ROUGE-L, and BERTScore use the pairs of the pass from the references: each matched reference is compared with the response matched to it, and the scores are averaged over these pairs ("temporally aligned prediction–reference pairs," Sec. 4.1). Unmatched responses and unmatched references lower precision and recall and do not enter the content scores. The content scores are therefore conditional on a match and are to be read together with T-F. This matters most for FitCoach and BioCoach in Table 1 (T-F .066 and .068), whose content scores are averaged over the smaller set of references they match.

Segments and turns without feedback. Three cases arise. A turn that consists of the pause token alone is not a response (Sec. 3.5); it is not counted, and a correctly withheld response earns no credit. A segment in which a model gives no response keeps all of its reference events in the recall denominator, as unmatched, and adds nothing to the content scores. The 46 segments without reference feedback are not excluded from the evaluation: they contain no reference events, and the responses given during them are counted with all other responses in the precision denominator. Sec. 4.1 describes only the 398 segments with reference feedback as scored; this wording is imprecise, and the revision will correct it.

The reviewer asks us to "report temporal precision, recall and response rate separately for each method." The two tables below give them for the two streaming models (a new analysis of the Table 1 runs). The remaining rows of Table 1 will follow in the revision, with the share of segments in which each method responds and the number of responses that fall in the 46 segments without reference feedback.

ModelResponsesPer workoutPrecisionRecallT-F
StreamingVLM + Stage 23,20843.4.536.695.606
StreamingCoach3,27344.2.547.698.613
METEOR by type of reference feedbackAnnouncementCountCorrectionPraise
StreamingVLM + Stage 2.199.081.087.112
StreamingCoach.241.098.108.126

Precision rises by .011 and recall by .003, with 2.0% more responses. Over the 74 full workouts, StreamingCoach gives 3,273 responses and the matched backbone 3,208; the reference coach gives 2,511 feedback events (Sec. 4.1), 33.9 per workout. Most of the T-F gain is in precision.

The METEOR gain is present within every type of feedback. A score averaged over matched pairs can rise through which references a model matches, and two observations bear on this. The two models match nearly the same share of the references (69.8% and 69.5%). METEOR is higher within each type, most for announcements (+.042) and corrections (+.021). The types differ widely in level (announcements score more than twice as high as counts in both models), and a gain within every type is therefore not produced by a shift in the mix of matched types. The matched sets within a type need not be identical, and we will also report the three content scores on the references matched by both models. The count column measures agreement of the text; the count error itself is the MAE of Table 1.

AuthorsOpenReview ↗

Response to Reviewer AtQY (4/4)

Q4—"What accuracy does zero-shot Qwen2.5-VL-7B obtain on the exercise probe? What proportion of StreamingVLM's answers fail the name-normalization mapping?"

Zero-shot Qwen2.5-VL-7B reaches 20.9%, and 42.0% of StreamingVLM's answers fail the mapping (new).

ModelStrict (%)Normalized (%)Unmapped (%)
Qwen2.5-VL-7B, zero-shot14.620.945.0
StreamingVLM, adapted21.525.642.0
StreamingCoach, adapted76.282.47.0

Normalized accuracy is the probe accuracy of Table 1 for the two adapted models; an unmapped answer counts as an error. Strict accuracy uses exact name matching, without the synonym list.

Unmapped answers alone cannot account for the gap. The reviewer's question was warranted: 42.0% of StreamingVLM's answers do not map to one of the 25 classes, about as many as for zero-shot Qwen2.5-VL-7B (45.0%), against 7.0% for StreamingCoach. If every one of these answers were counted as correct, with the mapped answers left unchanged, StreamingVLM would reach 67.6% (25.6% + 42.0%), which is 14.8 points below StreamingCoach with its own unmapped answers still counted as errors. The gap is also present under strict matching (21.5% and 76.2%), and in the count probe, which involves no name mapping (MAE 3.29 for StreamingVLM, 1.52 with Form Tokens, 1.08 with the state; Table 1). Among the answers that do map, StreamingVLM is correct in about 44% and StreamingCoach in about 89%; the two sets of mapped answers differ, and we give these figures only as a description. The 67.6% covers the unmapped answers only; answers mapped to a wrong class are a separate case. We will add to the appendix a manual breakdown of the unmapped answers (synonyms the mapping misses, descriptions that name no exercise, exercises outside the 25 classes) and an audit of the mapped ones, and we will re-score all models if the mapping proves incomplete.

Under the same probe adaptation, the motion input raises recognition from 25.6% to 82.2% (Table 1). StreamingCoach, its ablations, and StreamingVLM are adapted to the probe with low-rank adapters on the training split (Sec. 4.1), with the same data, adapters, schedule, and selection rule (Appendix A). The probe shows a six-second clip in a fresh context and withholds the planned-exercise list (Sec. 4.1); the answer has to come from the clip. Within this chain, the video-only backbone reaches 25.6% and the model with Form Tokens and no state reaches 82.2%. The zero-shot numbers place the first figure: Qwen2.5-VL-7B, on which StreamingVLM builds, is at 20.9%, and four other zero-shot models of similar size are between 18.8% and 25.1% (LLaVA-OneVision-7B 18.8, Qwen2.5-VL-3B 19.1, VideoLLaMA3-7B 21.1, InternVL3-8B 25.1; new). Fine-tuning on QEVD and probe adaptation did not lower the recognition of the video-only backbone, and they left it at the upper end of this zero-shot range.

Further new experiments. Our responses to the other reviewers contain a third dataset on which no model was trained (g2Vs, Q2), a cascade of pose statistics and a language model (g2Vs, Q4), a matched comparison with simpler memories (XThX, Q3), and a transition test (XThX, Q4).

Reviewer_g2VsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses application of vision language models for workout video assessment. It aims to address the specific sub question of having vision-language models understand different motion sequences, and react correctly to these motion sequences.

Critical strengths
  • Having vision-language models understand continuous videos is a challenigng task and frontier field of research
  • Authors show promising improvements on the tested dataset vs similar baselines.
Critical weaknesses
  • While the premise of training vision language models for long, continuous video is exciting, this paper address and tests on a very small sub-domain of workout videos. This is a very specific, application-oriented solution.
  • The authors seem to only test on one dataset each for the two tasks tested. The breadth of datasets is limited.
  • the ablation study is not comprehensive, doesn't isolate the effect of each network modules.
Questions for authors
  • Vision language models excel at multi-tasking, yet you are adapting it for a specific application. What is the motivation behind this design? For pose and motion understanding, have you considered feature-extraction-based methods or other model families that predate vision language models? Perhaps if you supply their output to a language model, it will be more reactive and more accurate than a heavy VLM trained for this task.
5 comments
AuthorsOpenReview ↗

Response to Reviewer g2Vs (1/4)

We thank the reviewer for recognizing that having vision-language models understand continuous videos is a "frontier field of research," and for noting the "promising improvements on the tested dataset." We respond to the three weaknesses and the question point by point. Results from the submission are cited by table or section; results marked new were run for this response.

Q1—Scope: "a very small sub-domain of workout videos"; "a very specific, application-oriented solution."

The paper asks what a streaming VLM should retain from long, continuous video, and workouts are where this can be measured. The reviewer finds "the premise of training vision language models for long, continuous video" exciting, and the paper works on that premise. The same current movement can call for different feedback, depending on what came before. A squat of a given depth is an improvement if the earlier repetitions were shallower and a deterioration if they were deeper (Sec. 1, Fig. 1). A model that sees only the current window cannot tell these cases apart. Its text history helps only if the earlier repetitions received feedback, and a coach watches many repetitions without responding. To judge a change, the model must first have kept the earlier movement. Any assistant that watches continuously and responds occasionally faces this question. Workouts make it measurable: repetitions make earlier and current movement comparable, and exercise changes alter which earlier evidence is relevant.

The contribution is a motion-grounded streaming representation: movement encoded in comparable coordinates and retained for later responses. Form Tokens encode joint configuration and dynamics, grounded in the person's appearance, in coordinates that are the same for every repetition (Sec. 3.3). The Motion State carries this evidence across chunks and is updated from observations whether or not a response is generated (Sec. 3.4). Auxiliary motion-prediction objectives train both alongside the coaching objective (Sec. 3.6).

Two findings in the submission already concern continuous video in general. First, models trained on short segments break on a continuous stream. FitCoach and BioCoach reach T-F .560 and .544 on isolated segments (Table 2), and .066 and .068 when the same workouts are processed without resets (Table 1, Fig. 4). Second, a general streaming VLM fine-tuned on the same workouts keeps responding, yet it names the exercise in a six-second clip correctly in only 25.6% of cases, and its running repetition count is off by 3.29 on average. Under the same probe adaptation, StreamingCoach reaches 82.4% and 1.08 (Table 1). These are the weaknesses that general-purpose benchmarks report for video-language models: fine-grained motion (MotionBench) and repetition counting (PushupBench), both discussed in Sec. 2, and counting periodic events in a stream (SVCBench, arXiv:2603.12703).

We now test retention directly, with all current evidence held fixed (new). This is the test that Sec. 5 of the submission calls for ("remains to be isolated by intervention"). From each of the 74 test workouts we build one pair of streams, A → B → W and B → A → W. A and B are eight-second clips of two different exercises. W is an identical four-second window. It fills the visual window of the model, which holds four one-second chunks (Appendix A), and the model has to carry the earlier evidence across it. We ask which exercise immediately preceded W. A supervised two-way readout, trained on the training workouts with the coaching model frozen, reads the answer from the Motion State at the end of W. The inputs for W are computed once and reused in both streams, so that the two streams differ only in the state that enters W. A predictor that receives only the current inputs gives the same answer twice and scores 50%. This test uses the second-seed checkpoints of StreamingCoach and the GRU.

Accuracy [95% CI]Predictions differ across the two historiesBoth correctBoth wrong
Current inputs only (by construction)50.0%0 / 7400
GRU memory in place of our update59.5% [54.7%, 64.2%]16 / 74151
StreamingCoach64.9% [58.8%, 70.9%]26 / 74242

Intervals come from a bootstrap over the 74 workouts that keeps both streams of a pair together.

Earlier activity stays recoverable after it has left the visual window. StreamingCoach is 14.9 points above the reference (95% CI [8.8, 20.9]). Its prediction differs across the two histories in 26 of the 74 pairs, and both predictions are correct in 24 of these 26. The coaching model was never trained on this question; the readout reads what coaching training left in the state. In the other 48 pairs the prediction is the same for both histories. Retention is partial, and the revision will report both numbers.

AuthorsOpenReview ↗

Response to Reviewer g2Vs (2/4)

Both learned recurrent states retain the earlier activity. A GRU written from the same tokens reaches 59.5%, against 50% without memory. Our update is 5.4 points higher (95% CI [−0.7, 11.5]); this test does not establish an advantage over the GRU, and the revision will report it in these terms.

The motion encoding and the state update are defined once for all exercises: they use a human-motion prior and no exercise-specific rules. Form Tokens describe any human movement in the same joint coordinates. They need no hand-coded exercise rules and no label of the exercise in progress at inference; the instruction lists the planned exercises without indicating which is underway (Sec. 4.1). The state update and the decoder interface take tokens and a validity score and make no reference to the body (Sec. 3.4, 3.5), and neither depends on the backbone's tokenization (Sec. 5); we have evaluated them with Form Tokens as input. We validate on fitness, where repetitions and exercise changes provide ground truth, on three datasets (Q2). The paired test shows that the state carries the preceding activity across the visual window; a repetition-level version, which compares form across repetitions, will be in the revision.

Q2—Datasets: "only test on one dataset each for the two tasks tested."

The submission evaluates on two datasets under three settings, and we add a third dataset from an independent source.

DatasetSettingTest dataResults
QEVDCoaching over continuous workouts74 workouts, 25 exercises, 2,511 feedback eventsTable 1
QEVDCoaching on single-exercise segments398 segmentsTable 2
Fit3DForm correction, held-out subject89 repetition pairs, 10 exercisesTable 3
EgoExo-FitnessError identification in continuous streams, no training190 streams, 871 actions, 37 participantsnew, below

EgoExo-Fitness (new) combines the demands of both tasks, without any training or adaptation on this dataset. The models run over continuous multi-exercise streams, as in the coaching task, and are scored on whether their feedback identifies execution errors, as in the form-correction task. The dataset was recorded by a different group, with different participants and cameras. Each model runs without resets, with the QEVD inference settings, on the exo_l view: 190 streams, about 4.7 hours. We evaluate the second-seed checkpoints of both models.

The metric is whether the feedback finds the annotated form errors. EgoExo-Fitness has no time-aligned coaching text; it has a checklist of technical points for every executed action. We take as reference errors the items that at least half of the annotators marked as unmet. Qwen2.5-32B, which sees neither the labels nor which model produced the feedback, maps each explicit correction to checklist items. Generic reminders are not counted. A correction is assigned to the action during which it is given, and within an action each checklist item counts once, however often it is named. Precision and recall are pooled over all action–item pairs of the 871 actions, and F1 is their harmonic mean.

ModelForm-error F1 [95% CI]
StreamingVLM + Stage 2.099 [.079, .123]
StreamingCoach.114 [.095, .134]

Intervals resample participants and keep all actions of a participant together. StreamingCoach has the higher F1, by .015, on a source it was not trained or adapted on. The protocol is strict: only explicit corrections count, generic reminders are excluded, and no model is adapted to the dataset. Absolute F1 is low for both models under it and leaves room for improvement. We will post the participant-paired difference, precision, and recall in the revision. With Tables 1–3, the evidence now covers three datasets, all within fitness.

AuthorsOpenReview ↗

Response to Reviewer g2Vs (3/4)

Q3—Ablations: "doesn't isolate the effect of each network modules."

Table 1 already compares against a backbone with the same Stage-2 training and removes each motion pathway in turn; we add four targeted controls and a second seed (new). The StreamingVLM row of Table 1 received the same Stage 2 as StreamingCoach, without the motion modules, and thus controls for the additional training. Appendix A describes only its first stage, and we will correct it. The new runs make the training stages explicit (the backbone before and after Stage 2) and add one control each for the content of the added tokens, the memory of the state, the motion derivatives, and the visual grounding. The control for the state's memory addresses what Sec. 4.2 leaves open: whether the state's contribution comes from retained history or from its encoding of the current chunk. Within each seed, all rows trained with Stage 2 share one configuration and differ only in the stated change.

ConditionMETEORROUGE-LBERTScoreT-F
Seed 42 (submission, Table 1)
StreamingVLM + Stage 2, no motion modules.122.102.858.606
Without Motion State.126.104.859.609
Without direct Form Tokens.130.106.863.608
StreamingCoach.132.107.864.613
Second seed (new)
StreamingVLM before Stage 2.104.067.848.595
StreamingVLM + Stage 2, no motion modules.114.086.850.571
Constant Form and State tokens.121.093.851.574
State cleared at every chunk.121.094.851.574
Without velocity/acceleration.120.091.852.583
Without visual grounding.118.089.851.574
StreamingCoach.123.096.852.589

The backbone with Stage 2 is the Table 1 baseline, run again with the second seed. The constant-token control keeps the visual input and the added token interface and replaces the pose-derived Form and State tokens with learned constants. Clearing the state at every chunk keeps its view of the current chunk and removes its memory. The last two controls remove the derivatives and the person-region visual features from the Form Tokens.

StreamingCoach is highest on METEOR, ROUGE-L, and T-F among the Stage-2 models in both seeds, and every control falls between the matched backbone and the full model. Against the matched backbone, the gain is +.010 METEOR and +.007 T-F in the submitted run (seed 42), and +.009 and +.018 in the second seed. The first row of the second seed shows the effect of Stage 2 by itself: it raises content and lowers timing in the backbone (METEOR .104 to .114, T-F .595 to .571). With the motion modules, the model keeps the content gain and recovers most of the timing (.123 and .589). The margins on these aggregate scores are small in absolute value. Our response to Reviewer AtQY (Q2) reports their consistency across seeds and workouts, and we will post paired bootstrap intervals over the 74 workouts in the revision.

Two modules have effects that do not rest on small margins. For the Form Tokens, exercise recognition rises from 25.6% to 82.2% once the motion inputs are added, before any state is used (Table 1). For the Motion State, the repetition-count error falls from 1.52 to 1.08 when the state is added (Table 1), and in the retention test of Q1 accuracy rises from the 50% reference to 64.9% (new). The auxiliary motion objective is the one component without a control of its own so far; a run without it will be in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.