acceptodds
Under review as a conference paper at ICLR 2027

PULSEBench: Personalized Understanding of Longitudinal Sensor Evidence for Wearable Health Reasoning

Abstract

Answering open-set health questions from longitudinal wearable records requires identifying relevant signals, time windows, and personal reference points, yet this end-to-end evidence discovery remains insufficiently evaluated. We introduce PULSEBench, a benchmark of 3,000 questions from 120 individuals across three public wearable datasets. Questions are grounded in observed trajectories, while a five-dimensional rubric evaluates free-text answers against question-specific evidence that is withheld from answering systems. Our evaluation reveals a persistent gap in evidence localization and supported attribution: adaptive inspection improves every evaluated backbone over direct inference, but substantial weaknesses remain. Controlled experiments further show that longer inputs and uniformly finer temporal resolution do not reliably improve answer quality, highlighting the importance of temporal organization. Motivated by these findings, we introduce Personalized Hierarchical Memory (PHM), which organizes personal records across temporal scales and retrieves evidence conditioned on the question. PHM improves over direct inference with a single answer-generation pass, although iterative agents remain stronger overall in the shared-model comparison. PULSEBench provides a testbed for studying how models discover and use personal evidence, beyond their ability to reason over preselected observations.

Primary area:
datasets and benchmarks
Keywords:
large language models, benchmark, wearable health, longitudinal reasoning, temporal reasoning, agentic reasoning, memory, evidence retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P8cpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a novel benchmark for evaluating end-to-end evidence discovery in LLMs for answering open-set health questions from longitudinal wearable records. It formulates 3,000 questions from 120 individuals across three public wearable datasets, revealing the limitations of LLMs in evidence localization and attribution. Additionally, it introduces a personalized hierarchical memory system for organizing individual records. However, the validation of question design, evidence extraction, and evaluation processes remains insufficiently addressed.

Critical strengths

The topic is interesting, and overall benchmark design is good. Evidence-based reasoning is critical for healthcare applications, and this benchmark is specifically designed to evaluate the capability of LLMs in identifying query-related grounding evidence by examining free-text responses in relation to query-specific evidence specifications with a five-dimensional rubric.

Critical weaknesses

The validation of the benchmark in this paper relies on several insufficiently supported assumptions: 1) All query-related evidence can be located within the trajectory. 2) Ground-truth evidence is both valid and comprehensive. 3) The LLM-based evaluator is reliable Nevertheless, the paper fails to clearly define what constitutes “right evidence” and to specify the question design necessary to ensure all evidence is traceable. In the benchmark construction, “25 deterministic operators” and “a deterministic planner” were used to create questions and specify evidence without detailed methodology explanations. So, it is hard to evaluate the quality of the ground truth. In addition, LLM judges are used for evaluation without a comparison to human expert assessments or any validation proofs except the cross-judge consistency between 3 LLMs.

Questions for authors

How are the deterministic operators and planner implemented in the construction of the benchmark?

2 comments

Thank you for the feedback. We clarified what counts as the right evidence, how we construct and check it, and how the primary judge compares with human scoring.

What counts as the right evidence? (Sections 3.2 and 3.4, Appendix B.1) The right evidence is available at query time, traceable to source records, and relevant to the claim. The specification distinguishes observations needed to support a conclusion from observations that provide additional context. For example, establishing a deviation from a personal baseline requires both the current measurement and a comparable historical reference. The rubric also credits valid observations beyond those listed in the specification when they strengthen or qualify the interpretation.

How is the evidence constructed and checked? (Section 3.2, Appendix A.2) More than 25 deterministic operators scan wearable records for patterns and produce time-anchored seeds. Selected seeds are drafted and rephrased into questions. For each finalized question, the planner identifies relevant signals and windows, then extracts observations for the scoring specification. Human reviewers screen all candidates, and human experts review disputed cases.

Section 3.2 (“Example of benchmark construction”) uses a case from LifeSnaps to show how an unusually short sleep episode leads to a question about taking it easy that day. Appendix A.2 details how the operator flags the episode and how the planner extracts the personal reference and recovery observations.

EvidenceWhat it supportsHow to check it
4 h 45 min of sleep, about 3 h below the recent meanSleep was short relative to the person’s usual durationRecompute the episode duration and recent mean from visible sleep records
96% sleep efficiencyEfficiency remained high despite the short durationCheck efficiency for the same episode
Other available recovery observationsHow strongly the evidence supports reducing activity intensityInspect the recovery records available at query time

Together, the observations support a conditional recommendation, such as reducing activity intensity if the person also feels tired. The example demonstrates how the evidence can be traced and checked. Estimating how consistently the specifications capture all key evidence across the final set would require a separate audit.

Human scoring comparison (Section 3.4). Human experts and the primary judge scored the same answers for 100 task-stratified questions outside the development set using the same rubric. Model identities and judge scores were hidden from the experts, who could consult the model inputs and source records available at query time. The Spearman correlation between human and judge total scores was 0.774.

DimensionD1D2D3D4D5
Human and judge scores within 1 point84%96%85%98%87%

In the D1 disagreements we reviewed, human experts tended to retain more partial credit for small numerical deviations when the time window, direction of change, and main conclusion were correct. Section 3.4 (“Human scoring comparison”) gives the protocol and full results.

I thank the authors for their response. The authors explained how operators and a planner were used to construct questions and evidence, and human experts screened all candidates. However, several concerns remain. • First, since AI is used to generate all patterns and question seeds, which are then screened by humans, it is not clear whether these questions align with real-world needs and contain only patterns that AI can identify in the first place. • Second, though using AI to identify relevant signals and windows is a way to help extract evidence, it still lacks clear strategies to guarantee evidence consistency and comprehensiveness. • Third, it is good to have human expert to review all questions and evidence, but not sure whether these experts have relevant healthcare experience and knowledge to provide grounded and safe recommendations based on wearable data, and how consensus is achieved especially for vague situation. Based on this, I will maintain my original rating.

Reviewer_sjR7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PULSEBench, an open-set question-answering benchmark designed for longitudinal wearable-health records. It focuses on evaluating whether models can identify relevant signals from historical sensor records, localize appropriate time windows, and provide evidence-grounded answers by considering personal baselines. The benchmark includes 3,000 questions from 120 individuals, and a five-dimensional rubric is used to assess the quality of responses. The authors evaluated various LLMs under Direct LLM and ReAct Agent settings, and proposed Personalized Hierarchical Memory (PHM) to improve structured evidence retrieval. Experiments showed that ReAct Agent generally outperforms Direct LLM; simply increasing the length of historical input data does not necessarily improve performance, whereas multiscale temporal representations can improve performance.

Critical strengths
  1. The research topic addressed in this paper is meaningful. Existing health-related question-answering evaluations focus more on reasoning capabilities given specific information, while this paper emphasizes models’ ability to discover relevant evidence from longitudinal, multi-signal wearable records. This approach is closer to real-world scenarios of personalized health question answering.

  2. The experiments cover a wide range of settings and yield valuable insights. The authors evaluated multiple LLMs and agents, analyzing the impact of input history length, temporal representation, and structured memory. The results indicate that adaptive evidence access and proper organization of historical data can be more effective than simply increasing the amount of input data.

Critical weaknesses
  1. The data quality of the benchmark and the evidence specifications lack sufficient independent validation. The questions are generated based on predefined rules with LLM assistance, and the evidence specifications are also constructed through automated processes. Although the authors conducted automated checks and involved medical personnel in reviewing disputed cases, no systematic human validation results were reported, making it difficult to assess the naturalness of the questions or the completeness and reliability of the evidence specifications.

  2. The evaluation method does not adequately distinguish between evidence discovery and the quality of the final answers. Although evidence discovery is presented as a key contribution of this work, the final answers are primarily scored using an LLM judge, without any direct assessment of whether the model correctly identified relevant signals and time windows.

Questions for authors

Given the uneven distribution across health domains, task modes, and individuals, do the model rankings and main experimental conclusions remain consistent under task-balanced or individual-balanced evaluation?

2 comments

Thank you for the feedback. We clarified the human review process, the scope of evidence localization scoring, and the robustness of our results under different weighting schemes.

Human review of questions and evidence. Section 3.2 now describes the manual checks used during benchmark construction, with further details in Appendix A.2. Early in development, we used 300 candidate questions spanning multiple task modes and health domains to refine question generation and rubric design. After scaling up the pipeline, human reviewers screened all generated candidates, and human experts reviewed disputed cases. We corrected fixable rubric issues and excluded uninformative cases or those with unresolved concerns, yielding the final set of 3,000 questions. Section 3.2 (“Example of benchmark construction”) traces a short-sleep observation through question rewriting to its scoring evidence. Appendix A.2 shows how the sleep episode, personal baseline, and recovery observations can be checked against the source records.

Human scoring comparison (Section 3.4). For 100 task-stratified questions outside the development set, we compared human experts’ scores with the primary judge’s scores on the same answers using the same rubric. The experts could consult the model inputs and source records available at query time. Model identities and judge scores were hidden from them. The Spearman correlation between human and judge total scores was 0.774.

DimensionD1D2D3D4D5
Human and judge scores within 1 point84%96%85%98%87%

In the D1 disagreements we reviewed, human experts tended to give more partial credit for small numerical deviations when the time window, direction of change, and main conclusion were correct. Section 3.4 (“Human scoring comparison”) reports the protocol and full results.

Evidence discovery and answer evaluation (Section 3.4, Table 14). D1 evaluates evidence localization as reflected in the final answer, regardless of how the system finds the evidence. Direct models can identify evidence implicitly, PHM retrieves it, and ReAct queries records through tools. D1 applies the same question-specific criteria to the resulting claims, checking the signals, time windows, numerical values, and availability of the evidence at query time. Table 14 reports these answer-level D1 scores by task mode.

Task and individual weighting (Section 5.2, Appendix D.3). We first average paired score gains within each task mode or participant, then give each task mode or participant equal weight. The table shows mean paired gains of ReAct over Direct.

Backbone groupPer-questionTask-balancedIndividual-balanced
Open-weight+1.90+1.89+1.89
Proprietary+1.18+1.33+1.12

The rankings are stable under reweighting. On the common set of scored questions, Spearman correlations with the per-question rankings range from 0.976 to 1.000. The top rank in each setting is unchanged, with some adjacent ranks exchanged. ReAct’s gain remains positive for all ten backbones under both balanced schemes. The four methods in Table 4 also retain their ordering under equal task weights. Appendix D.3 reports their task-balanced scores.

Thank you for the detailed response and the additional analyses. I appreciate the authors' efforts to address my concerns.

The human–judge agreement study and balanced evaluation results strengthen the reliability of the benchmark. However, I still have some concerns about the validation of the ground-truth evidence specifications and the lack of a direct evaluation of evidence discovery, which is a central motivation of the work.

Overall, while the rebuttal provides useful clarifications and improvements, my main concerns have not been fully resolved. I will therefore maintain my original rating.

Reviewer_TtMZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PULSEBench, a benchmark of 3,000 open-ended questions over the records of 120 people in three public wearable datasets. The model is given a question and must generate an answer grounded in the input wearable signal. The paper evaluates an LLM's ability to ground its answer in the input signal. Specifically, an LLM judge scores each free-text answer on a five-part, 20-point rubric, checking it against a question-specific evidence spec that the answering model never sees. Across ten backbones, a tool-using agent beats direct prompting but tops out at 10.80 out of 20. In a controlled study, neither longer history nor uniformly finer history reliably helps. The authors then propose PHM, which organizes records at multiple time scales and retrieves evidence at query time. It improves single-pass answering but still falls short of the agent.

Critical strengths

The paper studies a clear problem: how to find the right evidence for a given query. The authors check the LLM judge against other LLM backbones, which is a good sanity check. It would help to also see whether the judge agrees with human annotators.

Critical weaknesses

The benchmark covers a fairly narrow set of tasks. This area has many sensor setups and task types, and it would help to see which tasks are easy and which are hard. The ReAct setup is not described clearly. If ReAct can run code in a sandbox, I would expect D1 to be high. Could the authors explain the setup and say more about why the agent still does badly on D1? PHM is tested on its own, with no comparison to other memory-retrieval methods, not even a naive RAG baseline. The writing needs more polish. Several results are stated without saying why. For example, Section 5.2 does not explain the reasoning behind Finding 1.

Questions for authors

Why are the PHM results not in Table 3?

1 comment

Thank you for the feedback, particularly your suggestion to explain the findings more clearly.

Why is PHM omitted from Table 3? (Sections 5.1 and 5.4) Table 3 presents the main benchmark evaluation of Direct and ReAct across ten backbones. ReAct improves overall scores for every backbone, yet evidence localization and attribution remain difficult. These gaps motivated us to develop PHM, which organizes longitudinal evidence across time scales for single-pass answering. Table 4 evaluates PHM alongside Direct, ReAct, and Naive RAG on two shared backbones using the same 3,000 questions. As clarified in Section 5.1, we report these results separately to show benchmark performance across backbones in Table 3 and compare answering methods on matched backbones in Table 4.

Comparison with fixed-window retrieval (Section 5.4, Table 4). Naive RAG selects calendar-day windows using signal and time cues from the question. The overall scores below give equal weight to the two shared backbones.

MethodDirectNaive RAGPHMReAct
Overall score (/20)5.916.777.728.34

Fixed-window retrieval improves on Direct by 0.86 points, and PHM gains a further 0.95 points. PHM’s largest gains over Naive RAG are in event context attribution (+1.73), anomaly detection (+1.30), and personal baseline comparison (+1.04). PHM also exceeds ReAct on these three tasks, which require event-aligned observations or comparable personal history. ReAct remains strongest overall. Section 5.4 and Table 4 provide the full task-level analysis.

ReAct setup, low D1, and Finding 1 (Section 5.2, Appendix D.1). ReAct has 12 tools and up to 20 rounds. Its tools can retrieve measurements, compute window statistics, and compare them with a personal baseline. The restricted Python tool can read CSV records available at query time. Accurate comparisons still depend on the agent choosing relevant signals and comparable windows. For example, ReAct reported the correct current HRV value but chose the wrong personal reference window, leading to an inaccurate comparison. We revised Section 5.2 to explain why D1 and D3 remain the two lowest-scoring dimensions relative to their maxima in both model groups. Appendix D.1 lists all 12 tools and their roles.

Task coverage and difficulty (Appendix A.3, Table 14). PULSEBench covers seven task modes across three wearable datasets. In both ReAct model groups, D1 is highest on Lookup / Summary and lowest on Trend Analysis and Component / Driver Analysis. These latter tasks require comparing historical intervals or tracing changes in a composite score to its underlying components. Appendix A.3 explains the evidence each mode requires.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank TtMZ, sjR7, and P8cp for their thoughtful feedback. We are delighted that the reviewers recognized the importance of finding the right evidence in longitudinal wearable records for personalized health question answering.

Strengths highlighted by the reviewers

  • A meaningful evaluation problem. The reviewers recognized the value of testing whether models can find relevant evidence in longitudinal wearable records. sjR7 noted that this setting is closer to real-world personalized health question answering.
  • Evidence-grounded scoring. P8cp highlighted the use of question-specific evidence and a five-dimensional rubric to assess free-text answers. TtMZ appreciated the cross-judge consistency checks.
  • Broad experimental coverage. sjR7 highlighted the comparisons across models and agents, as well as the experiments on history length, temporal representation, and structured memory.

Main contributions

  • PULSEBench evaluates evidence discovery and use through 3,000 questions from 120 individuals across three public wearable datasets. Each response is assessed against question-specific evidence available at query time.
  • Benchmark and controlled experiments. ReAct improves over Direct across all ten backbones, while evidence localization and attribution remain difficult. In the controlled study, extending the input from 3 to 30 days lowered scores for both tested models across all three temporal representations.
  • Personalized Hierarchical Memory (PHM) organizes longitudinal records across time scales and retrieves question-relevant evidence for single-pass answering.

Key responses and new results

  • Question and evidence verification (sjR7, P8cp). Human reviewers screened all generated candidate questions, and human experts reviewed disputed cases. The construction example traces a short-sleep observation from the initial trigger to the final question and shows how to verify the scoring evidence against source records. See Section 3.2 and Appendix A.2.
  • Human scoring comparison (TtMZ, P8cp). On 100 questions, human and judge total scores had a Spearman correlation of 0.774. Within-one-point agreement ranged from 84% to 98% across the five dimensions. In the D1 disagreements we reviewed, the human reviewer tended to give more partial credit for small numerical deviations when the time window, direction of change, and main conclusion were correct. See Section 3.4.
  • Evidence localization and ReAct (TtMZ, sjR7). D1 checks the evidence reflected in the final answer using the same criteria across methods. ReAct can retrieve records and compute statistics for a selected window. Its comparisons still depend on choosing relevant signals and appropriate reference windows. See Sections 3.4 and 5.2, Appendix D.1, and Table 14.
  • Robustness to reweighting (sjR7). ReAct’s gains over Direct remain positive for all ten backbones under task and individual weighting. The four methods in Table 4 retain their ordering under equal task weights. See Section 5.2 and Appendix D.3.
  • PHM and fixed-window retrieval (TtMZ). On two shared backbones, PHM improves single-pass answering from 5.91 for Direct to 7.72, scoring 0.95 points above Naive RAG. PHM exceeds ReAct on event context attribution, anomaly detection, and personal baseline comparison. ReAct remains strongest overall. See Section 5.4 and Table 4.

Other manuscript revisions

  • Table 1 now notes that HEARTS includes standalone event localization tasks.
  • Section 5.1 clarifies the progression from the benchmark-wide comparison in Table 3 to the focused PHM evaluation in Table 4.
  • Appendices A.3 and D.1 now describe the evidence required by each task mode and list the 12 ReAct tools, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.