- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WearableQA, a benchmark for evaluating LLMs on questions about real-world wearable measurements and health information. It contains 4,084 ten-option multiple-choice questions constructed from 200 users’ wearable histories, blood biomarkers, and demographics. The questions cover 16 types, organized into data versus health reasoning and single-signal versus cross-signal reasoning. The authors construct reference questions through deterministic computations grounded in literature findings or population patterns, and evaluate 14 LLMs under different prompting and input settings.
Critical strengths
The paper target the important problem of longitudinal wearable QA construction and specify the detailed protocols to design the questions and provide corresponding evaluations to support the claims. Authors also provide the code and question construction example to enrich the open-source capability for the community.
Critical weaknesses
- Although the paper target the longitudinal setting, but not really specify the challenges of the longitudinal setting and not sufficiently distinguished from just a selected window. What's the key challenges or representative temporal characteristics of longitudinal setting, is still trend or anomaly? Most of the experiments are constructed using 28-day windows.
- Most the experiment and evaluation use accuracy as the metrics rather than support the claim of diagnosis of reasoning capabilities which should not only just provide the answer but the diagnosis the reasoning traces. Moreover, the question types sperates different categories from data to healthcare, but the final-answer accuracy along cannot determine whether success or failure comes from numerical computation, temporal understanding or health knowledge interpretation. More detailed error analysis beyond accuracy would strengthen the reasoning capability claims.
- The contribution of blood biomarkers and demographic context is not clearly isolated. Table 4 examines wearable measurements and history, but does not separately evaluate the additional context emphasized in the motivation.
- Important details of benchmark construction and validation remain insufficiently reported. The paper repeatedly refers to a larger cohort used for discovering population patterns and computing reference statistics, without clearly specifying its size or relationship to the 200 benchmark users. Human verification is mentioned, but its coverage, reviewer expertise, and agreement are not reported. These details are particularly important for health questions, where a deterministic labeling rule ensures reproducibility but does not by itself establish the validity or uniqueness of a clinical interpretation.
- The positioning against existing wearable QA and agentic systems needs further development. For example, WEQA includes a benchmark built from four open wearable datasets and evaluates an agentic framework. A more explicit comparison would clarify the additional task coverage and evaluation capabilities provided by WearableQA. The absence of blood biomarkers in another dataset should be distinguished from a limitation of its reasoning framework.
minor comments:
- Figure 2 is lack of clarification, is it comes from the 10-MQA accuracy?
- For the related work, in line 139-143, I am not convinced the weakness of previous methods is not incorporating complementary health information such as blood biomarkers. Not all wearable collections setting have such health information, so what's the key different or challenge of PHIA for such QA curation and answering?
Questions for authors
- Is the framework mainly use directly-LLM like HealthLLM?
- Which question types require history beyond the target window, and how many questions fall into each group? Could the authors provide complete worked examples covering the 16 types, including the evidence, computation, answer options, and justification?
- Could the authors evaluate intermediate reasoning traces or provide an error analysis separating numerical errors?
- What are the size and source of the larger reference cohort, and does it overlap with the benchmark users? What proportion of questions received human verification, and what qualifications did the reviewers have?
- in line 101-102, is the performance also comes from the MQA accuracy?
- There are already many open-source longitudinal wearable datasets such PMData, can the same QA protocols applies to other settings?
- Could the authors provide detailed examples of each question categories?
4 comments
W5/Q6/Minor 2. Positioning against existing wearable benchmarks and transferability to other datasets
We thank the reviewer for these suggestions. We agree that our original wording could incorrectly frame the absence of blood biomarkers as a limitation of prior reasoning frameworks. We have revised Sec. 2 to remove this characterization, added WEQA, and clarified the positioning of WearableQA relative to prior wearable QA and agentic systems.
- Comparison with WEQA and PHIA. WEQA primarily studies query-adaptive agentic reasoning across heterogeneous wearable and sensing datasets, whereas WearableQA is designed as a diagnostic benchmark for comparing LLM reasoning capabilities under a uniform scoring protocol. Its 16 question types explicitly separate data vs. health reasoning and single- vs. cross-signal reasoning, allowing us to identify where different models succeed or fail. We also revise our discussion of PHIA to reflect its evaluation design: PHIA combines automatically scored numerical queries with human-evaluated open-ended health queries, while WearableQA uses the same deterministic evaluation format across both data reasoning and physiological interpretation. Blood biomarkers are therefore treated as an additional source of health context in WearableQA, rather than as a limitation of prior frameworks.
- Transferability to other wearable datasets. Our QA generators are modular: each question type specifies the signals and auxiliary information required to instantiate it. Other longitudinal wearable datasets, including PMData, can therefore support the subset of question types for which the required signals and history are available, while cohort-relative or biomarker-dependent questions require the corresponding additional information.
Q1/Q5/Minor1. Clarification on evaluation protocol and Fig. 2.
Our main evaluation directly prompts each LLM with the benchmark input using Chain-of-Thought (CoT) as the default protocol, as explained in Appendix A.1. Table 3 separately studies alternative serializations of the measurements and an agentic setting with Python access as diagnostic ablations. Both Lines 101-102 and Figure 2 report accuracy on the 10-option multiple-choice questions. We revised the text to explicitly state this and updated the paper accordingly.
Q7. Detailed examples on each question category.
As suggested, Appendix A.6 now gives a worked example for each of the 16 question types. Each example lists the item ID and target window, the question, the evidence the answer depends on, the computation that fixes the label, all ten answer options, and a justification for the correct option.
We hope that the clarifications and additional experiments above have adequately addressed the reviewer’s concerns. We also note that our responses to the other reviewers contain further analyses that may be relevant to these points. We would be happy to provide any additional clarification if needed.
Thanks authors for detailed response and providing extra materials. I will increase the score.
Thank you, Reviewer PRHj, for your thoughtful feedback and for taking the time to review our response.
We thank the reviewer PRHj for the constructive feedback. Below, we address the main concerns with additional experimental results.
W1/Q2. Clarification on the longitudinal setting and history-dependent questions
We thank the reviewer for highlighting this distinction. For questions with wearable history, the default evaluation prompt includes up to approximately 500 days of daily measurements, requiring models to locate the relevant interval within a long, sparse, multi-signal record. The 28-day window specifies the target interval to be analyzed, rather than the length of the input history. More specifically, 270 questions explicitly require measurements preceding the target window and cannot be solved from the 28-day window alone, including anomaly-episode and personal-baseline questions. These questions derive a user-specific reference from prior history; removing this history reduces accuracy from 75.9% to 39.8%, while window-scoped questions remain essentially unchanged (58.1% → 58.0%).
Following the reviewer’s suggestion, Appendix A.6 now gives worked examples per type (evidence, computation, all ten options, justification).
W2/Q3. Qualitative analysis beyond accuracy.
We thank the reviewer for this suggestion. We conducted a qualitative analysis of incorrect responses across all 16 question types, comparing each reasoning trace against the reference statistics recomputed from the user’s record. Data-reasoning errors are mostly numerical or temporal (e.g., a miscomputed spread, or recovery timed from the wrong point), whereas health-reasoning errors are mostly interpretive: the trace reads the values correctly but maps them to the wrong conclusion. Appendix A.7 presents annotated traces in which the computation is correct, and the reasoning step fails, highlighting the erroneous step and the evidence that contradicts it.
W3. Ablation over all input sections of WearableQA
We appreciate the reviewer’s suggestion and have extended the ablation accordingly. We extend our context ablation on the blood biomarker panel and demographic information. We report results on Gemini 3.1 Pro, the strongest model in our main evaluation. We removed the blood panel and demographics together while retaining the cohort reference. In the no-context condition, the model received only the question and answer options
Removing all context lowers accuracy from 72.9% to 26.1%, showing that Gemini 3.1 Pro relies heavily on the supplied context. Removing blood biomarkers and demographics together yields 71.0% overall. The two inputs play different roles. Blood biomarkers are required evidence only for the blood-grounded question types, so their effect is diluted in aggregate accuracy. Demographics serve construction rather than inference: for literature-grounded items, they ensure the user falls within the population in which the anchored finding was established, so each item is a well-posed evaluation case, while the answer itself is determined by the measurements.
W4/Q4. Details on benchmark construction, human validation, and human evaluation.
We address this concern in the Global Response (GR2). In brief, we add independent expert evaluation (Appendix A.2), describe the human rule audit and source verification used during construction, and provide worked examples for all 16 question types in Appendix A.6. Appendix Table 10 also lists all 11 source publications used for literature grounding, together with the corresponding findings used in question construction. We further add annotated failure cases in Appendix A.7 to make the benchmark’s reasoning and error modes more directly inspectable.