- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HealthLoopQA evaluates LLMs on diabetes monitoring questions using glucose, insulin, meal, and activity records. It contains 127 task templates and 1500+ QA instances from simulated and real data, including simulated device failures. The experiments compare direct prompting with coding agents.
Critical strengths
The problem is useful, and programmatic answers make numerical and event-level errors easier to check. Controlled fault injection is also a reasonable way to study situations that are difficult to collect from patients. I appreciate the real-data evaluation and the matched task comparison.
Critical weaknesses
-
The added value over existing CGM QA benchmarks is not clear enough. LLM-CGM (LLM-CGM: A Benchmark for Large Language Model-Enabled Querying of Continuous Glucose Monitoring Data for Conversational Diabetes Management) already includes template questions, programmatic answers, simulated and real data, and text/code-agent comparisons. HealthLoopQA mainly adds treatment/activity context and fault scenarios. These are useful directions, but there is no context ablation showing what the extra information contributes. Simple forecasting and anomaly detection baselines are also missing, making it hard to judge what the new tasks require beyond basic data processing. The practical benefit of the expanded benchmark therefore remains uncertain.
-
The underlying physiological simulator provides a reasonable starting point, but this does not validate the added fault scenarios. Real-data evaluation covers only five participants and 44 tasks, with no fault-detection tasks. Similar performance on this subset cannot establish the realism of the simulated failures. Table 9 also shows substantial distribution differences: time-in-range is 94.0% in regular simulation versus 67.3% in real data. These differences do not invalidate simulation, but the distinctive fault tasks need more evidence, such as sensitivity to fault duration and magnitude and independent checks of their labels.
-
The agent comparisons are not sufficiently controlled. It seems this work uses different frameworks and output handling across models, including an extra answer-extraction agent for GPT. Common tool-call, interaction, and retry budgets are not reported.
Questions for authors
see above