- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies fitness coaching over a continuous workout, including changes between exercises. It adds two components to a streaming vision-language model. Form Tokens combine estimated joint angles, movement speed, acceleration, and visual features. A Motion State carries information across video chunks and updates even when the coach stays silent. The model uses both to decide when to speak and what feedback to give.
Critical strengths
-
Coaching a whole workout without resetting at exercise changes is more demanding than coaching isolated exercises. Evaluating this setting is a worthwhile contribution.
-
Combining joint movement with visual features makes sense for form feedback. The model improves over the fine-tuned streaming backbone, and the additional form-correction experiment provides useful evidence within fitness coaching.
Critical weaknesses
-
The method seems closely tied to fitness, since it relies on anatomical joint features and pose-based training targets. These choices make sense for fitness, but it is unclear which parts offer a broader method for streaming understanding rather than a task-specific solution. Fit3D adds another evaluation setting, but it remains within fitness. I do not think every paper needs unrelated tasks; however, the paper should make the scope of its contribution clearer and better justify its design choices.
-
The added contribution and the memory claim need stronger evidence. Pose-grounded coaching already exists, and the state uses an attention-based recurrent update. I am not yet convinced that their combination provides a strong enough methodological advance. More importantly, the experiments do not isolate the claimed use of earlier movement. The gains in feedback and timing over the fine-tuned streaming backbone are modest, while the large recognition gains are also present without the state. A direct test of history use would make the contribution much more convincing.
Questions for authors
-
Which parts of the method are specific to fitness, and which could apply to other streaming tasks? Could you clarify whether the main contribution is a general memory method or a fitness-specific system and evaluation setting?
-
Can you test whether earlier movement actually changes later feedback? For example, keep the current chunk fixed but reset or swap the previous state. Does the model correctly notice improvement after a correction or a change across repetitions?
-
What is gained over a simpler memory, such as pooling recent motion features? A matched comparison would help show why the proposed state update is needed.
-
Can you show examples or an evaluation that directly checks comparisons across repetitions and whether earlier advice stops applying after an exercise change? The current reference-based scores do not clearly answer these questions.
6 comments
Response to Reviewer XThX (2/4)
Neither existing line has both properties: a coach that runs over a whole workout and is grounded in the person's movement. The reviewer notes that pose-grounded coaching and attention-based recurrent updates both exist. The released pose-grounded coach, BioCoach, is trained on single-exercise segments and designed to be told the exercise (Sec. 4.2). It reaches T-F .544 on such segments (Table 2) and .068 when the workout continues without resets (Table 1). The streaming backbone fine-tuned on the same workouts does sustain its responses across the workout (T-F .606), yet it names the exercise correctly in 25.6% of clips and miscounts repetitions by 3.29 on average (Table 1). StreamingCoach runs continuously like this backbone (T-F .613) and is grounded in movement (82.4% and 1.08 on the two probes). What it adds to a streaming VLM is a motion-grounded representation and a persistent state, in three respects. Movement is encoded in anatomical coordinates shared by every repetition and every person, and the exercise in progress need not be given (Sec. 3.3). The state is written from these tokens at every chunk, whether or not a response is produced, and never from generated text (Sec. 3.2). The representation is trained with motion-prediction objectives in addition to the language objective (Sec. 3.6). The streaming memories discussed in Sec. 2 do not represent movement as joints in comparable anatomical coordinates (Sec. 1).
The two design choices the reviewer names, and the results behind them. Anatomical joint features express movement in units where "a slightly shallower squat is a measurable change rather than a few pixels" (Sec. 1). Adding the Form Tokens raises exercise recognition from 25.6% to 82.2% and lowers count MAE from 3.29 to 1.52 (Table 1). Removing their derivatives or their person-region grounding lowers METEOR and T-F (.120/.583 and .118/.574 vs. .123/.589; new, same training setup). Pose-based training targets are there because the language objective "can be partly satisfied by language priors and response history alone" (Sec. 3.6). They act only in training, with weight 0.1, and are discarded at inference. The revision reports the model trained without them on the paired test of Q2.
Q2—"Can you test whether earlier movement actually changes later feedback?"
With all current evidence held fixed, the frozen coaching state retains which movement came before: 64.9% against 50% (new). The design follows the reviewer's proposal: the current input is fixed and only the previous state differs. It tests the retention part of the request; the effect on the feedback is measured by likelihood below. It is also the test that Sec. 5 of the submission calls for ("remains to be isolated by intervention").
Design. From each of the 74 test workouts we build one pair of streams, A → B → W and B → A → W. A and B are eight-second clips of two different exercises. W is an identical four-second window. It fills the visual window of the model, which holds four one-second chunks (Appendix A), and the model has to carry the earlier evidence across it. We ask which exercise immediately preceded W. A supervised two-way readout, trained on the training workouts with the coaching model frozen, reads the answer from the Motion State at the end of W. The inputs for W are computed once and reused in both streams, so that the two streams differ only in the state that enters W. A predictor that receives only the current inputs gives the same answer twice and scores 50%. This test uses the second-seed checkpoints of StreamingCoach and the GRU.
| Accuracy [95% CI] | Predictions differ across the two histories | Both correct | Both wrong | |
|---|---|---|---|---|
| Current inputs only (by construction) | 50.0% | 0 / 74 | 0 | 0 |
| GRU memory in place of our update | 59.5% [54.7%, 64.2%] | 16 / 74 | 15 | 1 |
| StreamingCoach | 64.9% [58.8%, 70.9%] | 26 / 74 | 24 | 2 |
Intervals come from a bootstrap over the 74 workouts that keeps both streams of a pair together.
Earlier activity stays recoverable after it has left the visual window. StreamingCoach is 14.9 points above the reference (95% CI [8.8, 20.9]). Its prediction differs across the two histories in 26 of the 74 pairs, and both predictions are correct in 24 of these 26. The reviewer's two interventions have determined outcomes in this paired design: with the state reset before W, the prediction is the same for both histories (50%), and with the two states exchanged, the predictions are exchanged (35.1%, the mirror of 64.9%). The coaching model was never trained on this question; the readout reads what coaching training left in the state. In the other 48 pairs the prediction is the same for both histories. Retention is partial, and the revision will report both numbers.
Response to Reviewer XThX (3/4)
The likelihood of the reference feedback drops when the state is removed or frozen. Using the second-seed StreamingCoach checkpoint, we intervened on the state over 338 test windows, with the current chunk and the text unchanged, and measured the loss of the reference feedback. Resetting the state raises the loss by 1.9% on turns with a response and by 2.8% on turns without one. Freezing it, which stops new observations from being written, raises the loss by 1.3% and 2.2%. A state taken from another workout at the same elapsed time changes the loss little (+0.1% and +0.4%). The decoder's predictions depend on the state and on its updates. The small effect of the exchanged state means that the loss does not tell which history the state holds, and we rely on the paired test for that.
The same paired design extends to repetitions and corrections. The reviewer asks whether the model can "correctly notice improvement after a correction or a change across repetitions." The present test shows that the state carries the preceding activity across the four-second window. It does not measure how far back retention reaches, and it does not compare form across repetitions. For repetitions, the state lowers the error of the running count from 1.52 to 1.08 (Table 1), and a count cannot be read from the current window. For form, the paired design applies directly: the same current repetition, preceded by deeper or by shallower ones, with the generated feedback rated blind. QEVD does not annotate how execution changed after a correction; we will derive these labels from the joint-angle traces and report this test in the revision. The revision will claim what each test shows: retention of earlier movement now, and judgment of improvement once that test is in.
The two components have separate roles, and each has a targeted result. The reviewer observes that "the large recognition gains are also present without the state." This is the intended division, and Sec. 4.2 states it: the recognition gain comes from the motion input (25.6% to 82.2% without the state, Table 1). The role of the state is retention, and the paired test measures it with the current input fixed. The coaching scores improve over a backbone that receives the same Stage-2 training without motion modules in both seeds: +.010 METEOR and +.007 T-F (Table 1; the StreamingVLM row there also received Stage 2, which Appendix A omits and we will correct), and +.009 and +.018 (new, Q3 table). The reviewer's Q4 makes the same point about reference-based scores; the targeted tests are where the effect of memory can be read.
Q3—"What is gained over a simpler memory, such as pooling recent motion features?"
We ran the matched comparison using second-seed checkpoints. The proposed update gives the highest coaching scores of the four memories, and both learned recurrent states retain the earlier activity (new). Three simpler memories replace the gated update: an exponential moving average of the motion features (α = 0.25), a FIFO buffer that keeps one mean-pooled summary per valid chunk for the last eight, and a GRU. The EMA and the FIFO are two forms of pooling recent motion features. Everything else is shared: the backbone, the Form Tokens, the direct pathway, eight 256-dimensional slots, the 640 training windows, and the optimization budget. The table also lists two trained controls: the backbone without motion modules, and a model whose state is cleared at every chunk.
| Memory | METEOR | ROUGE-L | BERTScore | T-F | Paired test (Q2) |
|---|---|---|---|---|---|
| Control: StreamingVLM + Stage 2, no motion modules | .114 | .086 | .850 | .571 | n/a |
| Control: state cleared at every chunk (trained this way) | .121 | .094 | .851 | .574 | 50% by construction |
| EMA | .115 | .089 | .851 | .575 | not yet run |
| FIFO, last 8 valid chunks | .119 | .094 | .851 | .572 | not yet run |
| GRU | .115 | .086 | .850 | .575 | 59.5% [54.7%, 64.2%] |
| Gated state (StreamingCoach) | .123 | .096 | .852 | .589 | 64.9% [58.8%, 70.9%] |
The cleared-state control is a separately trained model. Its state at the end of W depends on the last chunk only, which is identical in both streams; this is a different condition from resetting the state of StreamingCoach at inference (Q2).
Coaching scores: the gated state is highest on all four metrics. Its margin over the two pooling memories is .014 to .017 in T-F and .004 to .008 in METEOR. The pooling memories stay close to the control without memory: within .002 T-F of it (.575 and .572 vs. .574), and below it in METEOR (.115 and .119 vs. .121). The gated state is .015 T-F above that control. Each of these margins comes from one training run.
Response to Reviewer XThX (1/4)
We thank the reviewer for the careful reading and the concrete suggestions. We are glad the reviewer considers evaluating "a whole workout without resetting at exercise changes" to be "a worthwhile contribution," and agrees that combining "joint movement with visual features makes sense for form feedback." The reviewer's central request is a "direct test of history use." With all current evidence held fixed, we now test what the state retains, and we apply the reset and swap interventions the reviewer proposes to the likelihood of the reference feedback (Q2). The remaining part, whether generated feedback correctly reflects a change across repetitions, is specified in Q2 and will be in the revision. Results from the submission are cited by table or section; results marked new were run for this response.
Q1—Scope: "a general memory method or a fitness-specific system and evaluation setting"?
The main contribution is a motion-grounded streaming representation: it encodes movement in coordinates that are comparable across repetitions, and it retains that movement for later responses. A streaming coach must use movement that has already left its visual window, including movement it observed without responding. Form Tokens encode joint configuration and dynamics, grounded in the person's appearance (Sec. 3.3). The Motion State carries this evidence across chunks and is updated from observations whether or not a response is generated (Sec. 3.4). Auxiliary motion-prediction objectives train both alongside the coaching objective (Sec. 3.6). Between the two descriptions the reviewer offers, it is a method, and its scope is human movement, which is broader than fitness.
The motion encoding and the state update are defined once for all exercises: they use a human-motion prior and no exercise-specific rules. The reviewer points to the "anatomical joint features and pose-based training targets." These are a prior about the human body. The same joint coordinates describe any human movement, and they place a squat and a lunge in one coordinate system. At inference the method needs no hand-coded exercise rules, no label of the exercise in progress, no repetition segmentation, and no boundary signal (Sec. 3.4); the instruction lists the planned exercises without indicating which is underway (Sec. 4.1). This is what lets one update mechanism operate across repetitions and across exercise changes. The state update and the decoder interface take tokens and a validity score and make no reference to the body; we have evaluated them with Form Tokens as input.
The experiments assess this representation in complementary ways. Coaching on QEVD evaluates the complete system (Tables 1–2). Recognition, counting, and form correction on Fit3D evaluate the representation after task-specific adaptation with low-rank adapters (Tables 1 and 3). The paired-history test freezes the coaching model and trains only a lightweight readout, which tests what the state already retains (Q2, new). EgoExo-Fitness evaluates the coaching model on a third source without training or adaptation (new; form-error F1 .114 vs. .099 for the matched backbone; protocol in our response to Reviewer g2Vs, Q2). All of this evidence is on human movement in fitness. Like the reviewer, we do not think the paper needs "unrelated tasks" to make its point, and the revision will state this scope in the introduction and the conclusion.