- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes an evidence-auditable LLM framework that produces an overall sleep-age estimate and ten organ-specific age representations from integrated sleep and clinical profiles. The framework comprises three agents. The Evidence Orchestration Agent organizes expert-registry-defined evidence into a section-wise Evidence-ID plan, which is rendered and checked for source fidelity. The Age Estimation Agent receives the resulting report and generates eleven age estimates with accompanying physiological explanations. The Consistency Audit Agent screens the frozen estimator output for two prespecified types of contradiction without modifying the age estimates. The authors evaluate associations between the resulting age gaps and mortality, incident disease, and disease burden across three cohorts.
Critical strengths
- The problem is interesting and significant. Estimating overall and organ-specific aging signatures from sleep and clinical information could support more informative health-risk assessment.
- Combining source-traceable inputs with inspectable physiological explanations is a promising direction for LLM-based age estimation. Separating input-fidelity checks from explanation-consistency checks is also a useful design choice.
- The paper provides extensive experimental results across three cohorts, including longitudinal outcome associations, organ–outcome correspondence, and comparisons with alternative LLM backbones and conventional age-prediction models.
Critical weaknesses
- The estimation procedure and the contribution of the agents require clearer explanation. Although the supplement provides prompts and output schemas, it remains unclear what justifies the specific numerical ages produced by the estimator. The authors should explain what validates these outputs as ages measured in years, beyond their associations with clinical outcomes. Furthermore, the benefit of LLM-based evidence orchestration over deterministic organization using the same registry is not established. A comparison with a simpler pipeline would help isolate the contribution of the proposed architecture.
- The LLM implementation details should be consolidated and made more explicit. The paper identifies Llama 3.3-70B as the primary backbone and evaluates six alternatives, but the model assignments for each agent and the model-based audit reference should be clearly stated. The authors should also specify whether any models were fine-tuned or otherwise adapted, provide generation settings and retry procedures, and explain how sleep-specific or medical knowledge are injected into the system beyond the prompts and expert-curated registry.
- The separation between representation generation and statistical evaluation should be clearer. The agents generate and audit the age representations, while Cox models subsequently evaluate their associations with outcomes. Although supplementary survival-analysis specifications are provided, the main text should make this separation explicit. Additional implementation details—such as the software used, proportional-hazards diagnostics, and pooling of estimates across multiple imputations—would improve reproducibility.
- The scope and methodological novelty of “evidence-auditable” need clearer defnition of range. The framework establishes input provenance and fidelity and screens generated explanations for two narrowly defined contradiction types. However, these checks do not establish that each physiological claim is supported by the participant’s evidence, that the explanation is clinically valid, or that the numerical age is justified. Input fidelity is a fundamental requirement for reliable analysis; its implementation alone does not establish a substantial methodological contribution. The authors should clarify what additional auditing capability the framework provides beyond standard provenance tracking and validation, and evaluate that capability directly.
Questions for authors
- What determines the magnitude of each numerical age adjustment, and what evidence supports interpreting these outputs in years of aging rather than as clinical risk scores?
- Which model is used for each of the three agents and for the developmental audit reference? Were these models used solely through prompting, or was any fine-tuning or other adaptation performed?
- Beyond the expert-curated registry and prompts, does the estimator receive external medical knowledge, retrieval results, or task-specific examples?
- What benefit does the Evidence Orchestration Agent in estimating the sleep age and organ specific age? Can you provide a comparison with a simpler single-estimator pipeline?
- Can you clarify the Cox-model implementation, including proportional-hazards checks, missing-data handling, and pooling across imputations?
- How much of the predictive information comes from sleep measurements versus the accompanying clinical history? Would sleep-only and clinical-context-only analyses yield similar results?
6 comments
Response to Reviewer fdjC (Part 2 of 3)
Q6: Sleep measurements and clinical context. We appreciate this suggestion and agree that comparing sleep-only and clinical-only inputs would help distinguish their respective prognostic value. In practice, overnight findings and clinical history are available together. Our aim was to determine whether the age gap decoded from this integrated sleep profile improves prediction once routine clinical risk factors are known. Across 46 organ outcomes, adding the endpoint-matched gap to each of four clinical-only models improved mean held-out C-index (Table H.1). The gain was 0.0437 over age and sex and 0.0159 (95% CI 0.0103–0.0221) over the model that also included BMI, systolic blood pressure, smoking, alcohol use, hypertension and diabetes.
To examine how the two sources of information relate to the decoded ages, we fitted a linear surrogate in each cohort to approximate the LLM's overall sleep age (Figure E.2). Across cohorts, 30 more minutes awake after sleep onset corresponded to overall sleep age estimates 0.02–0.07 years higher. A 10-mmHg higher systolic pressure corresponded to 0.10–0.15 additional years in the same models. These coefficients describe each measurement's association with decoded age after accounting for the other inputs.
Manuscript locations
The locations below refer to the revised manuscript PDF, where revisions are highlighted in blue.
-
Section 3, lines 214–219 (pp. 4–5); Appendix E.1, lines 1441–1475 (pp. 27–28) explain numerical age estimation and the surrogate analyses in clinical units (Figure E.2, p. 29).
-
Section 5.4, lines 439–443 (p. 9); Figure E.1, lines 1434–1439 (p. 27) report age-adjusted associations with frailty and clinical-deficit burden.
-
Section 3, lines 195–212 (p. 4); Appendix O, lines 2470–2479 (p. 46) describe patient-specific evidence plans, source-preserving rendering, and trace validation.
-
Appendix B.8, Table B.2, lines 1113–1133 (p. 21) consolidates model assignments, OpenRouter access, inference settings, and the clinical annotation reference.
-
Section 4.2, lines 249–280 (pp. 5–6); Appendix B.7, lines 1028–1045 (p. 20) separate age generation from Cox evaluation and document event reconciliation, software, and imputation pooling.
-
Section 4.4, lines 303–308 (p. 6); Appendix L, lines 1856–1867 (p. 35) report clinical expert annotation and audit performance for the two prespecified contradictions (Figure L.1).
-
Section 5.4, lines 434–438 (p. 9); Appendix H, Table H.1, lines 1674–1700 (p. 32) report the added discrimination of matched gaps across four clinical models.
-
Section 6.2, lines 523–529 (p. 10) discusses input comparisons, independent validation, and compatible model backbones.
Tables cited in this response
The tables below reproduce the corresponding manuscript tables. Table numbers are retained for cross-reference.
Table B.2. Audit reference, model assignments, and inference settings
Token limits are maximum completion lengths, and transport retries recover failed API requests.
| Role | Model or annotator | Procedure or settings |
|---|---|---|
| Evidence Orchestration Agent | Llama 3.3-70B Instruct | 4,096 tokens, four transport retries, up to three case attempts per run. |
| Age Estimation Agent | Llama 3.3-70B Instruct | 40,960 tokens, fourteen transport retries. |
| Consistency Audit Agent | GPT-OSS 120B | 4,096 tokens, three transport retries, medium reasoning effort. |
| Audit reference | Two clinical experts | Joint annotation of measurement-order reversal and incompatible states of the same variable in all 426 XH outputs. |
Response to Reviewer fdjC (Part 3 of 3)
Table H.1. Clinical adjustment and organ correspondence across 46 organ outcomes
| Model | Clinical C-index | + Matched gap C-index | + Non-matched gap, mean C-index | Matched gain | Matching advantage |
|---|---|---|---|---|---|
| A | 0.6639 (0.6471–0.6807) | 0.7076 (0.6926–0.7225) | 0.6758 (0.6615–0.6899) | +0.0437 (0.0286–0.0579) | +0.0318 (0.0231–0.0399) |
| B | 0.6806 (0.6649–0.6957) | 0.7072 (0.6928–0.7209) | 0.6854 (0.6713–0.6991) | +0.0266 (0.0178–0.0355) | +0.0218 (0.0154–0.0279) |
| C | 0.6823 (0.6667–0.6965) | 0.7048 (0.6900–0.7185) | 0.6849 (0.6700–0.6984) | +0.0225 (0.0157–0.0294) | +0.0199 (0.0149–0.0251) |
| D | 0.7000 (0.6855–0.7139) | 0.7159 (0.7023–0.7284) | 0.7019 (0.6881–0.7151) | +0.0159 (0.0103–0.0221) | +0.0140 (0.0094–0.0187) |
Clinical models. A, chronological age and sex. B, A plus BMI and systolic blood pressure. C, B plus smoking and alcohol use. D, C plus hypertension and diabetes history. Sex was omitted when invariant. These covariates were identical across endpoints, without substituting organ-specific histories. Evaluation. Entries average equally over the same 46 organ outcomes as Figure 6 (10 SHHS, 16 MrOS, and 20 XH). All-cause death and the XH first-new-diagnosis composite were analyzed separately. Each endpoint used identical participants and five folds for all models. Each gap was added separately. The non-matched comparator averages the C-indices of the ten alternative gaps. Matched gain is the matched-gap model minus the clinical model. Matching advantage is the matched-gap model minus the non-matched mean. Differences were calculated before rounding. Uncertainty. Parentheses give pointwise 95% percentile intervals from 2,000 paired participant bootstrap draws, retaining correlations across outcomes within each cohort and combining cohorts independently.
Response to Reviewer fdjC (Part 1 of 3)
We sincerely appreciate the reviewer's detailed comments and helpful suggestions. They have substantially improved the clarity and methodological transparency of the manuscript. We have clarified the numerical age scale, documented the agent settings and Cox analyses, and strengthened the evaluation of consistency auditing.
W1/Q1: Interpreting the numerical age scale. The prompt asks the LLM to weigh evidence of aging and resilience and estimate one overall sleep age and ten organ-specific ages (Figure M.2). The model determines the magnitude rather than adding a preset number of years per finding. Across the six joint surrogates, each additional year of chronological age corresponded to 0.97–1.03 decoded years with other inputs fixed (Figure E.2). A 10-mmHg higher systolic pressure corresponded to a cardiovascular age estimate higher by 0.69 years in SHHS, 0.34 in MrOS and 0.57 in XH. Held-out mean absolute errors were 1.49–2.20 years for overall sleep age and 2.72–3.32 years for cardiovascular age. Larger overall sleep gaps also tracked greater frailty in SHHS and MrOS and higher clinical-deficit burden in XH after adjustment for age and, where applicable, sex (Figure E.1).
W1/Q4: Function of evidence orchestration. The registry defines eligible evidence across the cohort. The orchestration agent selects nonredundant Evidence IDs for each patient and assigns them to six report sections. The renderer then copies the source labels, values and units. Compared with prompting an estimator directly, the extra step produces an inspectable record of which measurements reached the estimator and which were omitted (Figure 2, Section O).
W2/Q2–Q3: Models, medical knowledge, and inference. Section B.8 and Table B.2 consolidate role assignments and execution settings. Orchestration and estimation used Llama 3.3-70B Instruct. The Consistency Audit Agent used GPT-OSS 120B, as specified in Table B.2. Figure L.1 evaluates its decisions against clinical expert annotations for all 426 XH outputs. All agents use zero-shot prompting, with no fine-tuning, external retrieval, or labeled patient demonstrations. Medical knowledge comes from pretrained weights and the prompts, while the expert registry supplies clinical semantics. All agents were accessed through OpenRouter. The initial age-estimation response used greedy decoding (temperature 0.0, top-p 1.0), with unchanged settings for transport retries. Age vectors remain fixed after auditing. Section 6.2 discusses adoption of newer compatible backbones, including local open-weight deployment and closed models where governance permits.
W3/Q5: Representation generation and Cox evaluation. The LLM generated ages before outcome linkage. We used lifelines to fit Cox models to the fixed gaps (Sections 4.2 and B.7). In SHHS, we replaced a follow-up CVD indicator with baseline cardiovascular history and reconciled event dates against the event log and last contact. For SHHS/MrOS models with missing covariates, we pooled log HRs across five stochastic predictive-mean-matching imputations using Rubin's rules. The adjusted SHHS mortality HR per year of sleep gap is 1.048 (95% CI 1.025–1.073; Figure 4a).
W4: Additional capability of consistency auditing. Once trace checks verify input fidelity, the Consistency Audit Agent screens the generated explanations for contradictions. Two clinical experts jointly annotated all 426 XH outputs for the two prespecified contradictions. Against these clinical expert annotations, the Consistency Audit Agent's timing check detected 29/29 order reversals with no false positives. Its state check detected 4/10 incompatible states of the same variable, with 11 false positives. The agent had overall sensitivity of 86.5% and specificity of 97.4% (Figure L.1).