acceptodds
Under review as a conference paper at ICLR 2027

HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Care

Abstract

As medical wearables become integrated into daily chronic disease care, effectively interpreting longitudinal monitoring data is essential for patients and clinicians to understand health trends, detect safety-critical events, and make informed decisions. While large language models (LLMs) show promise for transforming this streaming physiological data into personalized health insights, evaluating their reasoning capability and analytical rigor in diverse monitoring tasks remains a fundamental challenge. Existing medical wearable question answering (QA) benchmarks primarily assess short-horizon classification or statistical summaries, largely ignoring the long-term patterns, therapeutic and behavioural contexts, and potential system failures inherent in real-world deployments. To address this, we introduce HealthLoopQA, a comprehensive diagnostic benchmark for evaluating LLM reasoning over continuous diabetes monitoring data. Grounded in a novel taxonomy of eleven atomic reasoning abilities, ranging from purely descriptive statistics to causal and inferential reasoning, HealthLoopQA comprises 127 tasks and over 1,500 QA instances spanning process mining, anomaly detection, and prediction over 30-day horizons. To systematically evaluate safety awareness, we complement real-world datasets with a fault-injected simulation testbed modeling diverse device malfunctions and cyber-physical attacks to generate physiologically plausible hazard scenarios. Evaluating state-of-the-art LLMs across prompting and agentic frameworks reveals severe limitations in complex temporal pattern mining. Furthermore, we identify a broader phenomenon of In-context Laziness under long-context prompting, where the model substitutes full computations with rough approximations and confident narrative justifications, highlighting critical open challenges in deploying LLMs for rigorous long-horizon medical reasoning.

Primary area:
datasets and benchmarks
Keywords:
large language models, medical wearables, question answering, healthcare AI, benchmark, continuous glucose monitoring, diabetes monitoring, automatic insulin delivery system

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sDr3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

HealthLoopQA evaluates LLMs on diabetes monitoring questions using glucose, insulin, meal, and activity records. It contains 127 task templates and 1500+ QA instances from simulated and real data, including simulated device failures. The experiments compare direct prompting with coding agents.

Critical strengths

The problem is useful, and programmatic answers make numerical and event-level errors easier to check. Controlled fault injection is also a reasonable way to study situations that are difficult to collect from patients. I appreciate the real-data evaluation and the matched task comparison.

Critical weaknesses
  1. The added value over existing CGM QA benchmarks is not clear enough. LLM-CGM (LLM-CGM: A Benchmark for Large Language Model-Enabled Querying of Continuous Glucose Monitoring Data for Conversational Diabetes Management) already includes template questions, programmatic answers, simulated and real data, and text/code-agent comparisons. HealthLoopQA mainly adds treatment/activity context and fault scenarios. These are useful directions, but there is no context ablation showing what the extra information contributes. Simple forecasting and anomaly detection baselines are also missing, making it hard to judge what the new tasks require beyond basic data processing. The practical benefit of the expanded benchmark therefore remains uncertain.

  2. The underlying physiological simulator provides a reasonable starting point, but this does not validate the added fault scenarios. Real-data evaluation covers only five participants and 44 tasks, with no fault-detection tasks. Similar performance on this subset cannot establish the realism of the simulated failures. Table 9 also shows substantial distribution differences: time-in-range is 94.0% in regular simulation versus 67.3% in real data. These differences do not invalidate simulation, but the distinctive fault tasks need more evidence, such as sensitivity to fault duration and magnitude and independent checks of their labels.

  3. The agent comparisons are not sufficiently controlled. It seems this work uses different frameworks and output handling across models, including an extra answer-extraction agent for GPT. Common tool-call, interaction, and retry budgets are not reported.

Questions for authors

see above

Reviewer_DwtDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HealthLoopQA, a question-answering benchmark for interpreting longitudinal diabetes monitoring data. It combines glucose measurements with insulin delivery, meal, and physical activity records, and incorporates simulated device malfunctions and cyber-physical attacks. The benchmark contains 127 task templates and 1,516 QA instances derived from 16 virtual patients and five real participants. The authors organize evaluation through a reasoning taxonomy and compare prompting-based and code-execution agent baselines. The results suggest that tools improve numerical analysis, while substantial difficulties remain in temporal pattern analysis, anomaly detection, and prediction. The paper additionally characterizes a failure pattern termed “In-context Laziness.”

Critical strengths
  1. A relevant evaluation setting that integrates physiological, therapeutic, behavioral, and operational information. The combination of glucose trajectories, insulin delivery, meals, physical activity, and system faults provides a useful setting beyond isolated signal summarization. In particular, controlled fault injection enables evaluation of hazardous situations that would be difficult to collect and label systematically in real patients. This is the benchmark’s most distinctive potential contribution.

  2. Programmatically generated reference answers and complementary evaluation settings. Using numerical, categorical, and temporal reference answers makes evaluation more verifiable than relying solely on linguistic similarity. The inclusion of both simulated and real data, together with a comparison on shared task templates, is a useful initial effort toward examining external validity. The tool-based baselines also provide practical evidence about which failures can be mitigated through explicit computation.

Critical weaknesses
  1. Overclaiming Scope and Benchmark Nomenclature. The naming "HealthLoopQA" is substantially broader than the benchmark's actual scope. The benchmark focuses exclusively on type 1 diabetes (T1D) within continuous glucose monitoring (CGM) and automated insulin delivery (AID) systems. While the authors refer to the physiological closed-loop control system as the "loop", labeling it under a generic "HealthLoop" moniker without specifying its boundary in the acronym may mislead the community into expecting a general-purpose wearable health benchmark spanning broader cardiovascular, pulmonary, or multi-disease domains.

  2. Asymmetric Baseline Comparisons and Confounded Model-vs-Agent Gains.

  • The experimental baselines lack controlled rigor: the prompting-based setup is evaluated solely on a single, lightweight model (Gemini-Flash-3.1), whereas the frontier proprietary models (GPT-5.4 and Claude-Sonnet-4.6) are only evaluated under the agentic framework. Consequently, the reported performance gap (43.7→77.2/87.143.7 \rightarrow 77.2/87.1) conflates two independent variables: the benefit of equipping models with an external Python code interpreter versus the intrinsic reasoning power of superior frontier base models. Without a zero-shot/few-shot prompting baseline on GPT or Claude, the authors cannot conclusively attribute the observed failure modes (such as "In-context Laziness") strictly to long-context prompting rather than the capacity limits of the Flash model.
  • Furthermore, constraining the reasoning effort to "low" during evaluation may have artificially exacerbated shortcut-seeking and computational laziness.
  1. Unquantified LLM Stochasticity. The paper does not report standard deviations, error bars, or multi-run consistency across different temperature samplings, leaving the statistical significance of several fine-grained rankings in doubt.

  2. Disconnect Between Problem Formulation and Evaluation Metrics.

  • In Section 3, the problem is formalized as f:(X,C,Q,I)→(A,R)f:(X, C, Q, I) \rightarrow (A, R), explicitly requiring the model to generate both a precise answer AA and a natural language reasoning rationale RR. However, Section 3.4 evaluates only AA using quantitative metrics (SMAPE, ACC, and Affinity F1). There is no automated, quantitative scoring framework for evaluating the faithfulness, clinical validity, or step-by-step logic of RR. The analysis of reasoning traces in Section 4.3 relies purely on post-hoc manual inspection of a handful of anecdotal cases (e.g., Patient 0), rendering the benchmark's diagnostic claims regarding reasoning alignment largely non-operationalized at scale.
  • Furthermore, relying on standard SMAPE for continuous glucose prediction fails to capture asymmetric clinical risks (e.g., mispredicting severe hypoglycemia carries far more life-threatening risks than equivalent errors in hyperglycemia), which are typically evaluated via clinical risk matrices such as the Clarke or Parkes Error Grid Analysis.
  1. Unjustified Equivalence Between Simulated and Real-World Evaluation. In Section 4.2, the authors claim that comparable scores between matched subsets "validate the simulated evaluation setting". This claim is mathematically and clinically ungrounded:
  • Table 9 clearly shows substantial statistical and distributional divergence between the regular simulated cohort and the real-world dataset: the real-world cohort exhibits much higher glycemic variability (CV: 0.36 vs. 0.20), lower time-in-range (TIR: 67.3% vs. 94.0%), and substantial sensor dropouts (active time: 79.0% vs. 100.0%).
  • The real-world dataset is confined to only 5 patients, spans a fundamentally different time horizon (10 months vs. 30 days), and completely omits Anomaly Detection (AD) tasks due to the lack of ground-truth malfunction/attack labels.
  • The similarity in composite scores is confounded by the fact that the real-world benchmark evaluates simpler tasks (PM/PD only, with zero fault scenarios) on noisier data. It cannot substantiate that synthetic ODE-based simulations faithfully mirror real clinical decision environments.
  1. Lack of Systematic Contextual and Feature Ablations. A central motivation of the paper is the integration of multi-modal behavioral and therapeutic context (meals, exercise logs, insulin delivery rates) alongside physiological readings. Despite this emphasis, the experimental section provides no systematic feature ablation studies (e.g., evaluating model degradation when omitting carbohydrate tags, exercise intensity, or insulin history). Aside from comparing two prompt formatting variants on a single task in Appendix C, the paper does not systematically isolate which specific contextual modalities contribute to or interfere with downstream time-series reasoning.

  2. Unsubstantiated Interpretations of "Laziness" and "Task Substitution".

  • Confounded Context Length Ablation: Table 6 claims that reducing temporal context alleviates "In-Context Laziness." However, the number of evaluated questions varies drastically across horizons (44 for 30-day, 41 for 7-day, and 34 for 1-day). Because questions requiring long-term comparisons were discarded in shorter horizons, the higher accuracy on the 1-day setting is confounded by question difficulty rather than true context-length effects.
  • Lack of Time-Series Baselines for Prediction: Section 4.3.2 characterizes models relying on historical exercise averages as a "task-substitution failure." However, historical event averaging and persistence are standard, competitive time-series forecasting heuristics. Without benchmarking against classical baselines (e.g., historical means, ARIMA, LSTM), the authors cannot substantiate whether this behavior represents a reasoning failure or a competitive predictive strategy on this dataset.
  1. How much monitoring data is actually provided for each real-world QA instance? Section 3.3 mentions ten months of monitoring data from five patients. Does this refer to the combined duration across all five patients, or ten months per patient? How many days of data does the model receive for each QA instance? In the simulation–real comparison in Table 2, are the input windows matched in duration?

  2. Why were the models evaluated with low reasoning effort? Was this choice motivated by cost constraints, deployment requirements, or preliminary results showing little benefit from higher reasoning effort? Have the authors evaluated the same questions with higher reasoning effort? If so, please report whether the scores and main error patterns changed.

  3. How were the newly injected faults validated? Using sensor bias and insulin delivery interruption as examples, please explain where each fault was injected and how its magnitude and duration were determined. Were these settings checked by relevant domain experts? For the question in Figure 3, does every reference fault interval satisfy the specified conditions concerning the magnitude of the abrupt change, the subsequent return to normal, and the associated contextual information?

Questions for authors

see Weaknesses

Reviewer_e5AgOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Problem: The paper addresses whether LLMs can interpret long-term diabetes monitoring data, specifically 30 days of CGM recordings together with insulin, meal, and exercise information. The authors propose a QA benchmark for reasoning, anomaly detection and prediction tasks. They argue that existing benchmarks do not include long-term patterns, therapeutic and behavioral context, and system failures. The problem scope of the paper matters because more people with type 1 diabetes increasingly use CGM and LLM-based monitoring assistants are being proposed to interpret this data for them. Reliable systems would enable useful insights, effective therapy implications and proactive adjustments.

Approach: As an input, each QA instance consists of physiological time series (30-day CGM data), therapy and behavioral context (insulin delivery, meal and exercise), a question about monitoring data, and answer instruction. The output is numerical, categorical, or temporal answer, and reasoning process. The authors formulate 127 tasks based on proposed novel taxonomy consisting of three analytical stages, three cognitive levels and 11 atomic skills. The authors evaluate Gemini-Flash 3.1 (prompting, and agent-based), GPT-5.4 and Claude-Sonnet 4.6 as agent-based baselines.

Claimed contributions:

  1. A novel taxonomy of three analysis stages and 11 atomic reasoning abilities for wearable data interpretation.
  2. An open-source, fault-injected AID simulation testbed with 8 device malfunctions and 10 cyber-physical attack patterns.
  3. HealthLoopQA benchmark consisting of 127 tasks with ground-truth extraction modules, reasoning instructions and taxonomy annotations.
  4. An evaluation of state-of-the-art LLMs with prompting and agent frameworks. The authors report the limitations of LLMs on temporal pattern mining and coin “In-context Laziness”, which points out that models replace exact computations with approximations under long-context prompting.

Two datasets used in the study: (i) 5 real-world patient from MetaboNet dataset with 10 months of data, yielding 44 tasks and 472 QA pairs, and (ii) 16 synthetic patients from T1DM Extended Hovorka over 30 days, yielding 127 tasks and 1044 QA pairs. The simulated dataset also contains 17 fault patterns that injected to closed-loop simulation. Baselines: Prompting-based baseline: Gemini-Flash-3.1 Agent-based baseline: Gemini-Flash-3.1, GPT-5.4, and Claude-Sonnet 4.6 Metrics:

  1. Normalized symmetric mean absolute percentage error (SMAPE) for regression tasks. Reported as 1-SMAPE/2.
  2. Accuracy for category classification.
  3. Affinity-F1 for timestamp or time-range answers (event detection). All metrics are mapped to 0-1 and averaged across questions into an overall score.

Main Results:

  1. Agent-based baselines outperform the prompting-based Gemini baseline on all datasets.
  2. GPT 5.4 Agent achieves the strongest overall performance (78.5 vs Claude-Sonnet 4-6 Agent 77.2 and Gemini-Flash-3.1 Agent 65.8 on simulated dataset and 86.1 vs Claude-Sonnet 4-6 Agent 84.1 and Gemini-Flash-3.1 Agent 78.5 on real world dataset).
  3. Anomaly Detection (AD) is the most difficult task (Gemini-Flash-3.1 Prompt achieves 21.2 and the best agent GPT 5.4 Agent achieves 72.9 on simulated dataset).
  4. Overall, the performance of the models generally declines with the increasing number of atomic reasoning steps required by each question.
Critical strengths
  1. The paper addresses an important and growing problem in digital health: testing whether frontier LLMs can correctly analyze long CGM data together with insulin, meal, and exercise information, to summarize glucose trends (latent patterns), detect safety-critical events (manipulated glucose data, treatment errors), and predict future glucose/insuline, organized through a taxonomy of task types and atomic skills.

  2. The paper constructs an open-source fault-injected AID simulation testbed to include device malfunctions (e.g., miscalibration, stop/under delivery) and cyber-physical attacks (e.g., replay, bias injection attacks), enabling an evaluation that real-world data cannot support.

Critical weaknesses
  1. The baselines are insufficient. The evaluation uses only API models, including a preview release (Gemini-3.1-Flash-Lite-Preview) and test the prompting-only setup on a single model. No open-weight model is evaluated, including the openly released time-series QA models. As a result, the reported results cannot be reliably reproduced, and the framing regarding the deploying LLMs for medical reasoning on personal health data omits the models that can run locally, which privacy constraints may require.

The results depending only on changeable APIs may not be reproducible. Leaving out open-weight models skips the setting that is relevant for personal health data and privacy concerns. A benchmark needs a wider range of models, varying in size, access (open-weight) and specialization (e.g, time-series QA models), to demonstrate applicability in broader perspective.

Recommendation: Extend the evaluation with at least one strong open-weight model, a time-series QA models such as Time-MQA, and prompting-only model. If context length is a limiting factor, report performances on the existing 7-day and 1-day settings. For reproducibility, release all raw model outputs (score verification) and reasoning traces (failure analysis), and report API access dates and decoding settings.

  1. The real-world evidence is based on five individuals, which limits the benchmark’s external validity. In addition, patient selection and time-series window criteria are not stated.

The real-world data demonstrates that the evaluation measures more than the simulator’s assumptions. With five individuals, real-world scores are not reliable enough to compare models or to validate the simulation outcomes. LLMs applicability in real-world setting requires more data for reliability.

Recommendation: Expanding the real-world split to include mode MetaboNet patients with a controlled selection procedure to incorporate sufficient diversity, and report per-patient outcomes.

Questions for authors
  1. What is the rational for the model selection (only API models, prompting on a single model) and for feeding 30 days of raw data into LLMs?

If the authors add more models (at least one open-weight model, time-series QA model and another prompting-based model), and extend the evaluation with shorter time intervals (e.g, 15 days or 7 days), I would increase my score. If they only indicate context length as problem without testing more models and shorter CGM data, my assessment remains as is.

  1. How are the five individuals selected from MetaboNet, and why only five? In addition, how are 30-day windows drawn from 10 months of data per patient? Lastly, how much do performance vary across the five individual, and does model ranking hold for each one?

If five individuals meet a defined hard requirement and reasonable justification provided, I would consider raising my score.

Clarifications:

  • Table 1 demonstrates the how 1044 questions split across PM, AD and PD, but not across the 127 individual task templates. How many questions does each template contribute, and what differs between questions generated from the same template?
  • How many questions fall on each reasoning path in Table 4?
  • Why every simulated day follows the same fixed schedule? For instance, the weekends could contain different consumption and behavioral patterns.
  • The authors indicate that HealthLoopQA contains 127 tasks that are manually curated and cross-verified in collaboration with experts from multiple domains. How does this verification process facilitated and verified?
  • Do the authors explicitly guided LLMs in their prompts to do computations for numerical outputs?

Suggestions for future work:

  • Scoring the reasoning traces would be interesting future direction.
  • Including type 2 diabetes patients.
  • Systematic evaluation on in-context laziness through rule-based heuristics or LLMs can demonstrate the failure modes, resulting in quantitative analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.