- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a novel benchmark for evaluating end-to-end evidence discovery in LLMs for answering open-set health questions from longitudinal wearable records. It formulates 3,000 questions from 120 individuals across three public wearable datasets, revealing the limitations of LLMs in evidence localization and attribution. Additionally, it introduces a personalized hierarchical memory system for organizing individual records. However, the validation of question design, evidence extraction, and evaluation processes remains insufficiently addressed.
Critical strengths
The topic is interesting, and overall benchmark design is good. Evidence-based reasoning is critical for healthcare applications, and this benchmark is specifically designed to evaluate the capability of LLMs in identifying query-related grounding evidence by examining free-text responses in relation to query-specific evidence specifications with a five-dimensional rubric.
Critical weaknesses
The validation of the benchmark in this paper relies on several insufficiently supported assumptions: 1) All query-related evidence can be located within the trajectory. 2) Ground-truth evidence is both valid and comprehensive. 3) The LLM-based evaluator is reliable Nevertheless, the paper fails to clearly define what constitutes “right evidence” and to specify the question design necessary to ensure all evidence is traceable. In the benchmark construction, “25 deterministic operators” and “a deterministic planner” were used to create questions and specify evidence without detailed methodology explanations. So, it is hard to evaluate the quality of the ground truth. In addition, LLM judges are used for evaluation without a comparison to human expert assessments or any validation proofs except the cross-judge consistency between 3 LLMs.
Questions for authors
How are the deterministic operators and planner implemented in the construction of the benchmark?
2 comments
I thank the authors for their response. The authors explained how operators and a planner were used to construct questions and evidence, and human experts screened all candidates. However, several concerns remain. • First, since AI is used to generate all patterns and question seeds, which are then screened by humans, it is not clear whether these questions align with real-world needs and contain only patterns that AI can identify in the first place. • Second, though using AI to identify relevant signals and windows is a way to help extract evidence, it still lacks clear strategies to guarantee evidence consistency and comprehensiveness. • Third, it is good to have human expert to review all questions and evidence, but not sure whether these experts have relevant healthcare experience and knowledge to provide grounded and safe recommendations based on wearable data, and how consensus is achieved especially for vague situation. Based on this, I will maintain my original rating.
Thank you for the feedback. We clarified what counts as the right evidence, how we construct and check it, and how the primary judge compares with human scoring.
What counts as the right evidence? (Sections 3.2 and 3.4, Appendix B.1) The right evidence is available at query time, traceable to source records, and relevant to the claim. The specification distinguishes observations needed to support a conclusion from observations that provide additional context. For example, establishing a deviation from a personal baseline requires both the current measurement and a comparable historical reference. The rubric also credits valid observations beyond those listed in the specification when they strengthen or qualify the interpretation.
How is the evidence constructed and checked? (Section 3.2, Appendix A.2) More than 25 deterministic operators scan wearable records for patterns and produce time-anchored seeds. Selected seeds are drafted and rephrased into questions. For each finalized question, the planner identifies relevant signals and windows, then extracts observations for the scoring specification. Human reviewers screen all candidates, and human experts review disputed cases.
Section 3.2 (“Example of benchmark construction”) uses a case from LifeSnaps to show how an unusually short sleep episode leads to a question about taking it easy that day. Appendix A.2 details how the operator flags the episode and how the planner extracts the personal reference and recovery observations.
Together, the observations support a conditional recommendation, such as reducing activity intensity if the person also feels tired. The example demonstrates how the evidence can be traced and checked. Estimating how consistently the specifications capture all key evidence across the final set would require a separate audit.
Human scoring comparison (Section 3.4). Human experts and the primary judge scored the same answers for 100 task-stratified questions outside the development set using the same rubric. Model identities and judge scores were hidden from the experts, who could consult the model inputs and source records available at query time. The Spearman correlation between human and judge total scores was 0.774.
In the D1 disagreements we reviewed, human experts tended to retain more partial credit for small numerical deviations when the time window, direction of change, and main conclusion were correct. Section 3.4 (“Human scoring comparison”) gives the protocol and full results.