acceptodds
Under review as a conference paper at ICLR 2027

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

Abstract

Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet the ability of LLMs to reason over longitudinal wearable data remains underexplored. Available benchmarks also largely rely on synthetic or simulated signals, leaving it unclear how well current LLMs can reason over a *real user's* longitudinal wearable measurements. We here introduce WearableQA, a benchmark comprising 4,084 ten-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, each with up to about 500 days of daily measurements. WearableQA preserves authentic wearable distributions that include device noise and inter-individual variability. To evaluate distinct reasoning capabilities, we introduce 16 question types organized along two complementary axes: data versus health reasoning, which distinguishes computation over longitudinal raw measurements from physiological interpretation; and single- versus cross-signal reasoning, which separates reasoning about individual signals from the integration of multiple signals. To construct reliable questions at scale, we adopt a dual-grounding framework that combines literature-grounded physiological findings with statistically validated population-grounded physiological patterns. This enables the capture of meaningful relationships observed in real-world wearable data. Evaluation of 14 proprietary and open-source LLMs demonstrates that WearableQA effectively differentiates model capabilities, with performance ranging from 19.6% to 72.9% against a 10% chance baseline. Moreover, WearableQA remains far from solved: most models achieve accuracies below 60%. Overall, WearableQA provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.

Primary area:
datasets and benchmarks
Keywords:
wearable health, time-series reasoning, health reasoning, benchmark evaluation, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PRHjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WearableQA, a benchmark for evaluating LLMs on questions about real-world wearable measurements and health information. It contains 4,084 ten-option multiple-choice questions constructed from 200 users’ wearable histories, blood biomarkers, and demographics. The questions cover 16 types, organized into data versus health reasoning and single-signal versus cross-signal reasoning. The authors construct reference questions through deterministic computations grounded in literature findings or population patterns, and evaluate 14 LLMs under different prompting and input settings.

Critical strengths

The paper target the important problem of longitudinal wearable QA construction and specify the detailed protocols to design the questions and provide corresponding evaluations to support the claims. Authors also provide the code and question construction example to enrich the open-source capability for the community.

Critical weaknesses
  1. Although the paper target the longitudinal setting, but not really specify the challenges of the longitudinal setting and not sufficiently distinguished from just a selected window. What's the key challenges or representative temporal characteristics of longitudinal setting, is still trend or anomaly? Most of the experiments are constructed using 28-day windows.
  2. Most the experiment and evaluation use accuracy as the metrics rather than support the claim of diagnosis of reasoning capabilities which should not only just provide the answer but the diagnosis the reasoning traces. Moreover, the question types sperates different categories from data to healthcare, but the final-answer accuracy along cannot determine whether success or failure comes from numerical computation, temporal understanding or health knowledge interpretation. More detailed error analysis beyond accuracy would strengthen the reasoning capability claims.
  3. The contribution of blood biomarkers and demographic context is not clearly isolated. Table 4 examines wearable measurements and history, but does not separately evaluate the additional context emphasized in the motivation.
  4. Important details of benchmark construction and validation remain insufficiently reported. The paper repeatedly refers to a larger cohort used for discovering population patterns and computing reference statistics, without clearly specifying its size or relationship to the 200 benchmark users. Human verification is mentioned, but its coverage, reviewer expertise, and agreement are not reported. These details are particularly important for health questions, where a deterministic labeling rule ensures reproducibility but does not by itself establish the validity or uniqueness of a clinical interpretation.
  5. The positioning against existing wearable QA and agentic systems needs further development. For example, WEQA includes a benchmark built from four open wearable datasets and evaluates an agentic framework. A more explicit comparison would clarify the additional task coverage and evaluation capabilities provided by WearableQA. The absence of blood biomarkers in another dataset should be distinguished from a limitation of its reasoning framework.

minor comments:

  1. Figure 2 is lack of clarification, is it comes from the 10-MQA accuracy?
  2. For the related work, in line 139-143, I am not convinced the weakness of previous methods is not incorporating complementary health information such as blood biomarkers. Not all wearable collections setting have such health information, so what's the key different or challenge of PHIA for such QA curation and answering?
Questions for authors
  1. Is the framework mainly use directly-LLM like HealthLLM?
  2. Which question types require history beyond the target window, and how many questions fall into each group? Could the authors provide complete worked examples covering the 16 types, including the evidence, computation, answer options, and justification?
  3. Could the authors evaluate intermediate reasoning traces or provide an error analysis separating numerical errors?
  4. What are the size and source of the larger reference cohort, and does it overlap with the benchmark users? What proportion of questions received human verification, and what qualifications did the reviewers have?
  5. in line 101-102, is the performance also comes from the MQA accuracy?
  6. There are already many open-source longitudinal wearable datasets such PMData, can the same QA protocols applies to other settings?
  7. Could the authors provide detailed examples of each question categories?
4 comments

We thank the reviewer PRHj for the constructive feedback. Below, we address the main concerns with additional experimental results.


W1/Q2. Clarification on the longitudinal setting and history-dependent questions

We thank the reviewer for highlighting this distinction. For questions with wearable history, the default evaluation prompt includes up to approximately 500 days of daily measurements, requiring models to locate the relevant interval within a long, sparse, multi-signal record. The 28-day window specifies the target interval to be analyzed, rather than the length of the input history. More specifically, 270 questions explicitly require measurements preceding the target window and cannot be solved from the 28-day window alone, including anomaly-episode and personal-baseline questions. These questions derive a user-specific reference from prior history; removing this history reduces accuracy from 75.9% to 39.8%, while window-scoped questions remain essentially unchanged (58.1% → 58.0%).

Following the reviewer’s suggestion, Appendix A.6 now gives worked examples per type (evidence, computation, all ten options, justification).


W2/Q3. Qualitative analysis beyond accuracy.

We thank the reviewer for this suggestion. We conducted a qualitative analysis of incorrect responses across all 16 question types, comparing each reasoning trace against the reference statistics recomputed from the user’s record. Data-reasoning errors are mostly numerical or temporal (e.g., a miscomputed spread, or recovery timed from the wrong point), whereas health-reasoning errors are mostly interpretive: the trace reads the values correctly but maps them to the wrong conclusion. Appendix A.7 presents annotated traces in which the computation is correct, and the reasoning step fails, highlighting the erroneous step and the evidence that contradicts it.


W3. Ablation over all input sections of WearableQA

We appreciate the reviewer’s suggestion and have extended the ablation accordingly. We extend our context ablation on the blood biomarker panel and demographic information. We report results on Gemini 3.1 Pro, the strongest model in our main evaluation. We removed the blood panel and demographics together while retaining the cohort reference. In the no-context condition, the model received only the question and answer options

ModelFull dataNo blood + demo.No context
Gemini 3.1 Pro72.971.026.1

Removing all context lowers accuracy from 72.9% to 26.1%, showing that Gemini 3.1 Pro relies heavily on the supplied context. Removing blood biomarkers and demographics together yields 71.0% overall. The two inputs play different roles. Blood biomarkers are required evidence only for the blood-grounded question types, so their effect is diluted in aggregate accuracy. Demographics serve construction rather than inference: for literature-grounded items, they ensure the user falls within the population in which the anchored finding was established, so each item is a well-posed evaluation case, while the answer itself is determined by the measurements.


W4/Q4. Details on benchmark construction, human validation, and human evaluation.

We address this concern in the Global Response (GR2).  In brief, we add  independent expert evaluation (Appendix A.2), describe the human rule audit and source verification used during construction, and provide worked examples for all 16 question types in Appendix A.6. Appendix Table 10 also lists all 11 source publications used for literature grounding, together with the corresponding findings used in question construction. We further add annotated failure cases in Appendix A.7 to make the benchmark’s reasoning and error modes more directly inspectable.

W5/Q6/Minor 2. Positioning against existing wearable benchmarks and transferability to other datasets

We thank the reviewer for these suggestions. We agree that our original wording could incorrectly frame the absence of blood biomarkers as a limitation of prior reasoning frameworks. We have revised Sec. 2 to remove this characterization, added WEQA, and clarified the positioning of WearableQA relative to prior wearable QA and agentic systems.

  • Comparison with WEQA and PHIA. WEQA primarily studies query-adaptive agentic reasoning across heterogeneous wearable and sensing datasets, whereas WearableQA is designed as a diagnostic benchmark for comparing LLM reasoning capabilities under a uniform scoring protocol. Its 16 question types explicitly separate data vs. health reasoning and single- vs. cross-signal reasoning, allowing us to identify where different models succeed or fail. We also revise our discussion of PHIA to reflect its evaluation design: PHIA combines automatically scored numerical queries with human-evaluated open-ended health queries, while WearableQA uses the same deterministic evaluation format across both data reasoning and physiological interpretation. Blood biomarkers are therefore treated as an additional source of health context in WearableQA, rather than as a limitation of prior frameworks.
  • Transferability to other wearable datasets. Our QA generators are modular: each question type specifies the signals and auxiliary information required to instantiate it. Other longitudinal wearable datasets, including PMData, can therefore support the subset of question types for which the required signals and history are available, while cohort-relative or biomarker-dependent questions require the corresponding additional information.

Q1/Q5/Minor1. Clarification on evaluation protocol and Fig. 2.

Our main evaluation directly prompts each LLM with the benchmark input using Chain-of-Thought (CoT) as the default protocol, as explained in Appendix A.1. Table 3 separately studies alternative serializations of the measurements and an agentic setting with Python access as diagnostic ablations. Both Lines 101-102 and Figure 2 report accuracy on the 10-option multiple-choice questions. We revised the text to explicitly state this and updated the paper accordingly.


Q7. Detailed examples on each question category.

As suggested, Appendix A.6 now gives a worked example for each of the 16 question types. Each example lists the item ID and target window, the question, the evidence the answer depends on, the computation that fixes the label, all ten answer options, and a justification for the correct option.


We hope that the clarifications and additional experiments above have adequately addressed the reviewer’s concerns. We also note that our responses to the other reviewers contain further analyses that may be relevant to these points. We would be happy to provide any additional clarification if needed.

Thanks authors for detailed response and providing extra materials. I will increase the score.

Thank you, Reviewer PRHj, for your thoughtful feedback and for taking the time to review our response.

Reviewer_fABcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Problem: In the current literature, it remains unclear whether current LLMs can reason about the daily longitudinal wearable history of a “real user” and determine what it implies for real health.

Approach: Developing a benchmark for data and health reasoning on real-user longitudinal wearable records. The benchmark evaluates model capabilities across 2 axes: reasoning type and signal complexity. Questions are based on two sources: literature-grounded and population-grounded. For each question, an LLM agent writes a small program that runs on the user’s measurements, and its output is the correct answer. LLM reviewers and unspecified human checks filter out ambiguous questions.

Contributions: The paper does not really claim contributions. However, if I were to name them, it would be:

  1. The benchmark: Approx. 4000 multiple-choice questions built from wearable data of 200 users.
  2. The taxonomy: 16 question types along two axes.
  3. The construction framework: “dual grounding” (literature findings plus statistically validated population pattern)

Empirical evaluation: 14 proprietary and open models, plus analyses of chain-of-thought vs. direct answering, etc.

Critical strengths
  1. The benchmark is built on real users’ longitudinal wearable data combined with blood biomarkers, not on simulated signals.

  2. The answer construction can be considered innovative (dual grounding): every question is grounded either in the biomedical literature or in a statistically validated pattern discovered directly from a large cohort.

  3. Validation of the benchmark: Two approaches are used to ensure the quality of the benchmark (Computational validity and Quality control and shortcut auditing.)

Critical weaknesses
  1. The data source and ethics approval are not reported. I would expect a statement in the paper like this one: “The study was approved by the [Institution] Ethics Committee / Institutional Review Board (protocol no. XXXX-XXXX, approved [date]). All participants provided written informed consent prior to participation. The study was conducted in accordance with the Declaration of Helsinki.”

If such information is not provided, we cannot accept the paper due to ethical concerns.

  1. The correct answers for health questions are not clinically validated. The paper does not explain how the correct answers for the health questions are decided. Section 3.5 mentions human verification; however, it does not explain how this procedure is conducted. The issue with this is that the model is scored against the author's own rules rather than clinical verification, and therefore its relevance for healthcare remains unclear. I would ask the authors to publish the rules used to assign the health labels, have clinicians review a sample of questions, and report how often they agree (inter-rater reliability).

  2. “Reasoning” is not defined, and many questions test calculation or knowledge retrieval instead. In my understanding, some question types, like signal summary and excursion count, are just calculations, not reasoning. Particularly of note is that when the model uses Python, the accuracy jumps from 51% to 71% (Table 3). This further substantiates this observation. In general, I would recommend defining the types of reasoning (e.g., inductive, deductive, abduction, etc.) and seeing what questions fit what category. This would enable us to separate reasoning from knowledge retrieval.

  3. Missing data is not addressed, although the paper claims to use realistic data. In general, wearable data is messy because it has, among other things, a lot of missing data (for instance due to non-wear phases). In general, I would expect this to be reflected or at least reported in this benchmark. If the goal is always just to operate on complete data, are we not simplifying the task and hence creating an artificial benchmark? I think it would be important to justify this decision. A possible solution would be to conduct an ablation study on how missingness influences the reasoning capabilities of models.

  4. The models used to build the benchmark are also evaluated on it. This could simply introduce a bias. Ideally, models that are used to build the benchmark should not be used to validate the benchmark. Otherwise, it would be good to demonstrate that this has no influence.

  5. The device is not named, and the results may not generalize to other wearables. Especially in wearables, device agnosticity is a main challenge. A good benchmark would incorporate different devices to investigate if the reasoning capabilities generalize across devices.

Questions for authors
  1. Was ethics/IRB approval (or a formal exemption) obtained for this use of the data, and does participant consent allow public redistribution?

  2. Which devices and source cohort does the data come from, and can you report results on data from a second device or cohort?

  3. How exactly are the health answers assigned, and have clinicians checked them? Please share the labelling rules and expert agreement on a subset.

  4. How do the models that built and validated the benchmark (Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.4) perform on the questions they generated or validated, compared with the rest?

6 comments

We thank the reviewer fABc for the valuable feedback. Below, we address the main concerns with additional experimental results.


W1/Q1. Clarification on ethics and data provenance

We address this concern in the Global Response (GR1). In brief, this study is a secondary analysis of existing de-identified records obtained from a third-party provider under a data use agreement, with no participant interaction and no access to direct identifiers or means of re-identification. We also obtained a formal determination from Bluebonnet Ethical Review, an independent institutional review board, that the work does not constitute human-subjects research under 45 CFR 46.102(e)(1). The consent and data use agreement permit research redistribution, and the released dataset is further governed by terms prohibiting re-identification and clinical, diagnostic, or treatment use.


W3. Categorization of reasoning types in WearableQA.

We thank the reviewer for this suggestion. We define reasoning in WearableQA as deriving an answer that is not explicitly stated in the record from the evidence it contains, and categorize the 16 question types by their primary inference demand:

Inference typeQuestion types
Abductionprognostic prediction, health recommendation, fitness prediction, multimodal phenotype, differential diagnosis
Inductiontrend shape, recovery time, anomaly episodes, cross-signal prediction
Computation-grounded deductionsignal summary, excursion count, signed correlation, strongest pair, strongest triplet
Knowledge-grounded deductioncohort relative risk, signal concordance

Computation-grounded items apply a specified operation to the observed record; inductive items infer a pattern or relationship from noisy longitudinal observations; knowledge-grounded items combine observed evidence with an established association; and abductive items select the best-supported explanation, prognosis, or action among plausible alternatives. The computation-grounded tier is deliberate because it isolates whether a model can correctly derive the quantities on which higher-level inference depends.

This distinction is also reflected in Table 3: Python access improves the data axis from 45.0% to 72.9%, whereas health-axis accuracy changes only from 63.5% to 68.1%, indicating that computation is a major bottleneck but does not account for the full difficulty of health interpretation. Abductive questions further help distinguish inference from knowledge retrieval because their distractors represent alternative interpretations of the same grounded evidence.

We additionally add traced failure cases for computation-grounded deduction, induction, and abduction in Appendix A.7, where the relevant evidence is correctly identified or computed but the final inference is incorrect.


W4. Real-world missingness in WearableQA

WearableQA preserves missing values in both the 28-day target window and the full measurement history (median 529 days per user). We preserve the original measurement record and perform no imputation. Among items with sensor data, 23.0% had at least one day without a resting heart rate measurement in the target window, and 9.3% had at least three consecutive days without one. The corresponding rates were 27.4% and 9.9% for steps, and 50.5% and 11.6% for sleep. Across full histories, 27.5% of users had a gap in recorded measurements lasting at least seven days, and 19.0% had a gap lasting at least 30 days. Among the 2,534 items that explicitly name a target metric, that metric was observed on 89.9% of target-window days. We will report these coverage statistics in the revision. We have not separately evaluated how model accuracy varies with the extent of missingness.


W5. Role of the three models in benchmark construction and review.

We appreciate the reviewer’s concern. The three models named by the reviewer served as validators. They reviewed the labeling criteria and checked the corresponding programs for edge cases and implementation errors. For each item, the reference answer was computed deterministically by running the program on the user’s measurements. Evaluation used fresh inference calls without access to the construction or validation responses. Our records do not identify which model reviewed each item, so we cannot compare each model’s accuracy on items it reviewed with its accuracy on the remaining items. We therefore cannot rule out the possibility that validation favored items these models find easier. We will clarify the validators’ role and acknowledge this limitation in the revision.

W5/Q3. Details on benchmark construction, human validation, and human evaluation.

We address this concern in the Global Response (GR2).  In brief, we add independent expert evaluation (Appendix A.2), describe the human rule audit and source verification used during construction, and provide worked examples for all 16 question types in Appendix A.6. Appendix Table 10 also lists all 11 source publications used for literature grounding, together with the corresponding findings used in question construction. We further add annotated failure cases in Appendix A.7 to make the benchmark’s reasoning and error modes more directly inspectable.


We hope that the clarifications and additional experiments above have adequately addressed the reviewer’s concerns. We also note that our responses to the other reviewers contain further analyses that may be relevant to these points. We would be happy to provide any additional clarification if needed.

I thank the authors for the detailed response. The IRB non-human-subjects determination addresses my main ethics concern. The reasoning taxonomy and the per-axis Python analysis are helpful, although I note that the computation-grounded tier (5/16 types) confirms that a substantial part of the benchmark tests calculation rather than inference. The missingness statistics are valuable. Since they are already computed per item, could the authors report accuracy split by missingness level? On W5, the response covers the validators but not the model(s) that generated the question programs. Please name them. A simple check would also help: do the validator models outperform their expected ranking relative to non-validator models of similar capability? Finally, W6/Q2 (device identity and generalization to other devices) remains unaddressed, and I would appreciate a response.

Reviewer_e6xYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces WearableQA, a dataset and a question-answering benchmark where questions are derived automatically and asked over data collected from real-world wearables datasets. The questions are derived either from literature or from the data itself, answers are extracted deterministically. A range of LLMs is evaluated, the best one reaches around 70% suggesting that the benchmark isn't saturated. There are also some more in-detail evaluations which I didn't find particularly surprising: CoT is better than non-CoT, tool use seems to help somewhat, and removing context (history) lowers the scores.

Critical strengths

For what it's worth I think this paper is one of the better ones of its kind that I've seen so far. The use of real data is commendable. It's clearly written, for the most part, I could (without AI) read and understand what was done.

Critical weaknesses

I think the main question here is, who is this paper positioned to be useful for, i.e. what gives? Is it for patients so they can interpret their data? Is it for general time-series reasoning? If the former, queries should come from (or at least be compared to those of) actual patients. For the latter, we need a good comparison to general-domain datasets to see why "health timeseries reasoning" is different than "timeseries reasoning". That's the objection that I have to such papers in general.

This paper, seems to suggest another alternative - it's for researchers who try to analyse wearable data, e.g. by replicating prior findings in literature. But for this, the data provenance seems somewhat murky - the supplied sample does not indicate where questions were derived from. There is no mention on disagreement between what is reported in the literature and what the sample gives (there must be at least a few such instances given the sample size). Humans seemingly validate the final output, but no details are given. I think more clarity on where the questions are coming from and what human oversight went into the benchmark curation will strengthen the credibility of the data.

A more technical point: I've ran Claude Opus 5.5 (through a Claude Code harness) on the supplied data sample and I got 30/32 questions right, which gives around 94% accuracy. Even with the most conservative CI interval, that's still above 80% and thus much higher than what the paper reports. This might suggest that the dataset is quite quickly saturated after all. By the time of submission Opus 5.5 might not have been out, but 4.6 is quite outdated - the paper should report the performance of the most up-to-date AI models available.

I think these two points should be addressed in the rebuttal, and I believe they are doable, hence I (optimistically) lean towards acceptance at this point.

Questions for authors

What data was used? What about ethics approval etc? Has user consent been sought?

The paper is very vague on human validation - how was human validation done, by whom? What was the annotation protocol, what was the inter-annotator agreement, if any?

See also my critical weaknesses points above.

Suggestions: This seems like relevant work: https://arxiv.org/pdf/2609.06976

2 comments

We thank the reviewer e6xY for the constructive feedback and address the main concerns below with additional experimental results.


W1. Target users of WearableQA and why health time-series reasoning is distinct

WearableQA is intended primarily as a research benchmark for evaluating models that reason over longitudinal wearable-health data, rather than as a simulation of questions directly asked by patients. We will make this positioning explicit.

Health time-series reasoning differs from generic time-series reasoning in several ways that are directly evaluated in WearableQA:

  1. The reference is personal. Many questions require interpreting a window relative to the user’s own longitudinal history. Removing this history reduces accuracy on history-dependent questions from 75.9% to 39.8%, while window-scoped questions remain unchanged (58.1% vs. 58.0%; Table 4).
  2. The meaning is not contained in the series alone. Models must connect patterns derived from wearable measurements to physiological or clinical evidence. Accordingly, Python access substantially improves the data axis (45.0%→72.9%) but has a much smaller effect on health reasoning (63.5%→68.1%; Table 3).
  3. Domain priors interact with observed evidence. In a controlled subset of relationship-strength questions, models score up to 51.5 points higher on familiar physiological relationships than on comparably strong associations that must be recovered from the user’s record (Table 2). This suggests that prior knowledge affects how models interpret wearable evidence.

The worked examples for all 16 question types in Appendix A.6 and the traced failures in Appendix A.7 further illustrate this distinction: successful answers require combining what the longitudinal record shows with the appropriate health interpretation, and failures can arise from either component.


W2/Q2. Details on benchmark construction, human validation, and human evaluation.

We address this concern in the Global Response (GR2). In brief, we add independent expert evaluation (Appendix A.2), describe the human rule audit and source verification used during construction, and provide worked examples for all 16 question types in Appendix A.6. Appendix Table 10 lists all 11 source publications used for literature grounding, together with the findings used in question construction. We selected only findings whose population, measurements, and criterion can be instantiated on the signals and blood panel available in our cohort. Each finding supplies a criterion, and the reference label is computed by applying it to the individual user’s record, so an item tests whether a model applies a published finding correctly to one person’s data rather than whether the cohort reproduces the population-level effect. We further add annotated failure cases in Appendix A.7 to make the benchmark’s reasoning and error modes more directly inspectable.


W3. Results on current frontier models

Following the reviewer’s suggestion, we expanded our evaluation to include several recently released frontier models on all 4,084 WearableQA questions, alongside the strongest open-weight baselines in our evaluation:

ModelAccuracy (%)ModelAccuracy (%)
Claude 5.1 Fable81.8Gemma-4-26B-A4B42.5
GPT-6 Astra79.7Mistral-Small-3.129.3
GPT-5.6 Sol78.0Llama-3.3-70B29.2
Gemini 3.1 Pro72.9Gemma-3-4B19.8

These results show that frontier performance has advanced substantially, with the strongest current models now achieving around 80% accuracy. At the same time, WearableQA continues to provide diagnostic value through its per-type breakdowns across data vs. health and single- vs. cross-signal reasoning. We also note that Claude Code is an agentic tool-use harness and is therefore not directly comparable to our standard model-only evaluation. Table 3 evaluates a related tool-use setting and shows that Python access substantially improves performance, particularly on data reasoning. We will add these frontier-model results and clarify the distinction between standard and agentic evaluation in the revision.

Q1. Clarification on ethics and data provenance

We address this concern in the Global Response (GR1). In brief, this study is a secondary analysis of existing de-identified records obtained from a third-party provider under a data use agreement, with no participant interaction and no access to direct identifiers or means of re-identification. We also obtained a formal determination from Bluebonnet Ethical Review, an independent institutional review board, that the work does not constitute human-subjects research under 45 CFR 46.102(e)(1). The consent and data use agreement permit research redistribution, and the released dataset is further governed by terms prohibiting re-identification and clinical, diagnostic, or treatment use.


Q3. Suggestion.

We thank the reviewer for pointing us to this related work. We now cite and discuss it in Sec. 2.


We hope that the clarifications and additional experiments above have adequately addressed the reviewer’s concerns. We also note that our responses to the other reviewers contain further analyses that may be relevant to these points. We would be happy to provide any additional clarification if needed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank the reviewers. We are glad the benchmark's foundations were recognised: that it is built on real users' longitudinal wearable records combined with blood biomarkers rather than simulated signals (fABc), that the dual-grounding construction is innovative (fABc), that it targets an important problem with detailed, specified protocols and releases code and construction examples to the community (PRHj), and that it is "one of the better ones of its kind" and clearly written (e6xY).


[GR1]Ethics and data provenance — (e6xY W1,Q1 · fABc W1, Q1)

We obtained a formal determination from Bluebonnet Ethical Review, an independent institutional review board, that this work does not involve human subjects as defined under 45 CFR 46.102(e)(1), and therefore does not require IRB review: we analyzed existing de-identified records, did not interact with any participant, and had no access to identifiers or any means of re-identification.

  1. Acquisition. The records were collected by a third party in the course of its own service, under terms permitting research use, and obtained by us under a data use agreement. We did not interact with any participant; this is a secondary analysis.
  2. De-identification. Each user appears only under an opaque identifier. The data contain no names, contact details, free text, or location information.
  3. Redistribution (fABc Q1). Permitted. The consent obtained at collection and the data use agreement allow research redistribution; the dataset is distributed under CC BY-NC 4.0 with additional terms prohibiting re-identification and any use in clinical, diagnostic, or treatment decisions, and the release was reviewed and approved by our institution beforehand.
  • Provider identity (fABc W1). Unfortunately, we are contractually and institutionally constrained from naming the provider in public text.

[GR2]Details on benchmark construction, human validation, and human evaluation.— (e6xY W2c, Q2 · fABc W2, Q3 · PRHj W4, Q4)

Independent expert evaluation. Five clinically trained experts (four medical doctors spanning internal medicine, cardiology diagnostics, and surgery, and a doctor of chiropractic) answered questions drawn from six of the eight health-reasoning types. They received the same evidence as the models and had no access to our labeling rules. On average, experts selected the reference answer in 77.1% of cases. The highest individual agreement was 81.1% (Appendix A.2). Recent frontier models match or slightly exceed the best expert on this subset. Even health-reasoning items require scanning up to ~500 days of data across dozens of metrics, which models process exhaustively but humans cannot, and several grounding findings come from recent wearable-specific studies outside most clinicians' daily practice. We therefore read the expert score as evidence that the reference answers are clinically recoverable, not as a performance ceiling.

Human validation during benchmark construction. Following the model-based validation pass, the authors conducted two additional checks:

  • Rule audit across all 16 question types. We reviewed the generation rules and item definitions, correcting invalid thresholds, ambiguous answer choices, and ill-defined cases. Affected items were regenerated or removed. Appendix A.6 now provides worked examples for all 16 question types, showing how each rule is instantiated from the input measurements to the derived reference answer.
  • Source verification for all literature-grounded items. We verified that each grounded finding was consistent with its source publication. Appendix Table 10 now lists all 11 source publications used for literature grounding, together with the corresponding findings used in benchmark construction.

Failure analysis beyond accuracy. Appendix A.7 now provides annotated failure cases spanning computation-grounded deduction, induction, and abduction, identifying where numerical, temporal, or health-interpretation errors arise in model responses.


We hope the additional analyses, experiments, and clarifications provided above address the reviewers’ main concerns. Several complementary results are also discussed in our responses to the other reviewers, and together they further clarify the scope, construction, and diagnostic value of WearableQA. We would be glad to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.