- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces FitSim, a 365-day closed-loop simulator for fitness coaching that combines fitness-fatigue dynamics, an injury model, and a compliance model. It introduces nine user archetypes and the Coaching Gain Ratio (CGR) to evaluate LLM coaches using synthetic wearable data. Experiments show that the LLM coaches achieve limited coaching gains and trail Linear deload in the reported results.
Critical strengths
-
FitSim enables 365-day closed-loop evaluation without LLM role-play, avoiding persona drift over long horizons.
-
The rule-based reference coaches, oracle strategies, and separate measures of fitness gain and injury days provide useful analysis of LLM coaching behavior.
Critical weaknesses
-
Although the physiological models are grounded in existing literature, the compliance coefficients and user archetypes are hand-calibrated, and FitSim is not validated against real per-user trajectories. It remains unclear how well the simulated responses to coaching recommendations represent real users.
-
Linear deload uses the simulator's intent calendar, while LLM coaches must infer training patterns from wearable data. This difference makes the reported performance gap difficult to interpret.
Questions for authors
-
What evidence supports the simulated users' responses to coaching recommendations, particularly workout compliance and fitness changes?
-
How much do the main CGR and injury-day results depend on the chosen compliance coefficients and user archetypes?
-
Would Linear deload still outperform the LLM coaches without access to the simulator's intent calendar?
2 comments
Thank you for your additional experimental results and comments. I think that additional results address most of my concerns, such as the validity of simulated users, parameter dependence, and information gap related to intent calendar. In addition, I understand the purpose of FitSim isn't to predict human responses but evaluating the ability of the coach. Accordingly, I will change my rating from 2 to 3.
Thank you for the review! We will address the questions and concerns below.
W1 / Q1 (evidence for simulated responses).
We believe the point of the simulation is to be a good proxy for the kinds of capabilities that an agent good at coaching would need, i.e. necessary but not sufficient. As you noted, FitSim is not validated against real per-user trajectories, however, we show that it provides a realistic environment for the agent to operate in. The archetypes are calibrated to maximize the coverage of parameter space (our models have a lot of degrees of freedom, and compute is limited) including the extremities (e.g. overtrainer) -- it is extremely hard to collect real longitudinal user trajectories for such cases in an ethical manner, but exploring synthetic data could provide valuable insights about the agents under test. That said, we agree this is a limitation and discuss it as such in L465-470.
We built FitSim based on the established models in sport science literature which, in turn, were fitted based on real user data. This means, we have confidence in simulated users' responses to coaching recommendations to the same degree as we have confidence in underlying models. Compliance coefficients and archetypes are chosen to span the parameter space (App. I), not to represent a population.
Q2 (dependence on coefficients and archetypes).
Per-archetype results are in App. E. Leaving out any one archetype keeps Linear deload ahead of the best LLM. For compliance coefficients, we extended App. H to all reference coaches: under ±20% on each of ten Banister and compliance constants, the order Constant < Linear deload < partial oracle < informed oracle holds in 18 of 20 settings (in the other two, Linear deload and the partial oracle tie). We did not rerun the LLMs for every setting, but as a rough check, we fed each LLM's original prescriptions unchanged into the perturbed simulator, and the order or the models had not changed (these fixed prescriptions cannot adapt to the changed user behaviour, so this is indicative only).
W2 / Q3 (intent calendar). (copied from the response to reviewer YjTz)
Linear deload still outperforms the LLMs without the calendar. We tested several additional linear deload schedulers to achieve better parity with the LLM setting:
So the comparison holds without the privileged information as well, but we agree that it's worth reframing the scheduling claim as “inferring the user's schedule from the workout history is part of the task, and the LLMs did not do it under our default prompt”. The prompt does not name the weekdays or say that off-schedule recommendations are ignored -- both have to be inferred.