acceptodds
Under review as a conference paper at ICLR 2027

RubricsTree: Towards Adaptive and Scalable Open-Ended Evaluation of Personal Health Agents

Abstract

The LLM-empowered personal health agents with user health sensor metrics have offered a promising pathway to alleviate global disparities in healthcare access. However, large-scale clinical deployment remains constrained by an open-ended evaluation bottleneck: physician annotation is reliable but costly and unscalable, while LLM-as-a-judge evaluators are scalable but subjective, inconsistent, and sometimes clinically misaligned. We introduce RubricsTree, a scalable evaluation framework with an expert-aligned hierarchical taxonomy of over 100 atomic, clinically verifiable Boolean rubrics, evolving from the insights of 4,000 real user queries through an iterative human-in-the-loop curation protocol with an expertise panel led by an experienced physician. A context-aware adaptive router activates only the relevant auto-weighted rubric subset per query, providing the throughput needed for scalable evaluation with expert-aligned quality. Through a systematic meta-evaluation, we show that RubricsTree (i) substantially exceeds a strong large-scale evaluation baseline in expert alignment on challenging open-ended queries; (ii) reliably penalizes contextually degraded responses; and (iii) when used as structured instructions, text feedback, or training rewards for performance optimization, yields up to approximately 66% relative gains on HealthBench for Gemini, GPT, and Qwen model families. RubricsTree thus provides a scalable, auditable, and evolving evaluation infrastructure required for the continuous optimization of product-level personal healthcare AI.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Health, Personal Health Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DDfkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RubricsTree, an open-ended evaluation framework for personal health agents. It uses an expert-curated hierarchical taxonomy to decompose complex medical evaluation tasks into atomic Boolean rubrics, with adaptive routing to select relevant rubrics and hierarchical weighting to calculate evaluation scores. The framework was developed based on approximately 4,000 real user queries and evaluated through expert agreement, contextual perturbation tests, evaluation consistency experiments, and downstream model optimization. The results show that RubricsTree outperforms the Principle Baseline in expert agreement, perturbation detection, and evaluation stability, while also providing benefits for downstream model optimization.

Critical strengths
  1. This paper addresses the evaluation of open-ended responses generated by personal health agents, an important problem that requires consideration of users’ health backgrounds and personalized information.
  2. The authors developed fine-grained evaluation rubrics with the help of medical and health experts and evaluated the framework against an independent expert panel. The experiments also cover contextual perturbations, evaluation stability, and downstream model optimization, providing evidence of the framework’s effectiveness from multiple perspectives.
Critical weaknesses
  1. The main contribution appears to lie in the expert-curated rubrics, while the technical novelty of the hierarchical taxonomy, adaptive routing, and auto-weighting mechanisms is relatively limited. The evaluation primarily compares RubricsTree with the Principle Baseline, which differs in both the granularity of evaluation criteria and the incorporated domain knowledge. Although the supplementary material includes routing ablations, a comparison with a flat rubric-based evaluator using the same expert rubrics is missing, making it difficult to isolate the contributions of the proposed architecture.
  2. The hierarchical taxonomy and weighting scheme are not sufficiently justified, particularly given that different clinical criteria may have varying levels of importance. Moreover, adaptive routing may omit safety-critical rubrics, potentially affecting the final evaluation scores. Despite the reported routing accuracy, the impact of such omissions on evaluation reliability is not sufficiently analyzed.
Questions for authors

Q1. Could the authors compare RubricsTree with a flat rubric-based evaluator using the same expert rubrics to clarify the contributions of the hierarchical taxonomy, adaptive routing, and auto-weighting mechanisms?

Q2. How were the hierarchical taxonomy and weighting scheme validated? Why was recursive equal-weight allocation among sibling nodes chosen instead of assigning weights based on the clinical importance of different criteria?

6 comments

Thank you for the review. You raise two questions: does the tree add anything beyond the expert rubrics themselves, and can routing or weighting hide a safety failure. We answer each with evidence already in the paper, worked examples from the published rubric table, and new same-rubric experiments on public HealthBench data. In short:

  1. Same rubrics, same budget: tree routing beats a flat relevance selector by 0.0146 on HealthBench, agrees with the HealthBench grader 8.0 points more often, and is 2.8x faster.
  2. Weights: with the selected leaves and their judgments held fixed, tree weights add another 0.0190 over a plain average.
  3. Safety omissions: the tree skips 12 of 120 failed safety checks, against 32 for a flat selector. A small safety screen brings this down to 3.

W1 / Q1. Compare RubricsTree with a flat evaluator that uses the same expert rubrics.

Once the rubrics are fixed, a flat evaluator and RubricsTree differ in only two places: which leaves get checked, and how the checks are added up. The leaf text, judge and prompt are the same. We test each place separately.

Same rubrics, different selectors (Appendix D, Table 4). The per-leaf judge below is exactly a flat evaluator over our rubrics: it asks an LLM, rubric by rubric, whether each one applies. This is also how Auto-Eval (Mallinar et al., 2026) adapts its criteria.

Selector over the same rubric treeActivation accuracy vs expert labelsRouting latency per query
Flat, embedding similarity61.25%3.2 s
Flat, per-leaf LLM judge76.14%64.2 s
Tree traversal (ours)80.61%5.6 s

The tree is 4.47 points more accurate than the flat LLM judge and 11.5x faster. Embedding retrieval is faster still but 19.4 points less accurate; Appendix D shows it misses paraphrases such as "shortness of breath" vs "dyspnea".

Same rubrics, end to end on HealthBench. Setup: the 362 user-facing HealthBench-Hard queries from Figure 6, Gemini-2.5-Flash, fixed initial answers, one round of feedback and revision, scored by HealthBench's physician-written rubrics and grading prompt, with Gemini-3-Flash as the grader. All arms use the same RubricsTree leaves that apply to HealthBench, the same judge and the same prompt. Selectors see only the query and context, as in Section 2.3. Flat relevance and tree routing share an 8,000-token budget per query; "full catalog" checks every leaf and is reported at its own cost. HealthBench's own rubrics are used only for final grading. "Agrees with HealthBench grader" counts how often an evaluator prefers the same answer as the grader on 300 fixed answer pairs. Only the professional-skill leaves apply to HealthBench, so all latencies here are shorter than in Table 4.

Same rubrics, different evaluatorHealthBench after revisionGain over base (0.1944)Agrees with HealthBench graderTokens per queryLatency per query
Full catalog, plain average0.2476+0.0532219/300 (73.0%)15,24046.8 s
Flat relevance, plain average0.2518+0.0574230/300 (76.7%)7,82036.4 s
Tree routing, plain average0.2664+0.0720254/300 (84.7%)7,61013.2 s
Tree routing, tree weights (ours)0.2854+0.0910267/300 (89.0%)7,64013.2 s

What this shows:

  1. Tree vs flat at the same budget (rows 2 and 3). Same rubrics, same plain average, same token budget. Tree routing scores 0.0146 higher, agrees with the HealthBench grader 8.0 points more often (254 vs 230 of 300 pairs), and runs 2.8x faster (13.2 s vs 36.4 s).
  2. Tree vs checking everything (rows 1 and 3). Checking every leaf does not help: tree routing scores 0.0188 higher with half the tokens and 3.5x lower latency. Two likely reasons: applicability has to be judged for every leaf, so its errors add up, and the feedback becomes a long list in which the few checks that matter are easy to miss.

Why the tree helps. A flat selector judges each of the 100+ rubrics in isolation. It pays for every irrelevant one (lipid panels, tool step counts) and has no parent context when a rubric is borderline. The tree drops an irrelevant branch with one decision and judges a leaf only after its parent is known to matter. So the expert rubrics supply the content, and the tree decides which of them to spend the budget on.

Q2. How were the taxonomy and weights validated, and why split weight equally among siblings instead of by clinical importance?

Validation. The taxonomy was built by a nine-person, physician-led curation panel over about 4,000 real queries (Appendix B.2) and tested against a separate six-person evaluation panel with no shared members (Section 3.1, Figure 3):

Agreement with the evaluation panel (Figure 3)Principle BaselineRubricsTree
ICC3, overall0.2910.876
Cohen's κ, overall0.4310.787
κ, Health Data0.5671.000
κ, Action Plan0.3640.806
κ, Symptoms0.5400.724
κ, Medical Explanation0.3040.657

Equal splitting keeps a topic's weight independent of how finely it was split. Leaves have to be atomic so a judge can verify them. Some topics need several leaves for that, others need one. Equal splitting stops this bookkeeping choice from turning into importance. Example from Appendix J: hemoglobin is one leaf (weight 1/96). The white blood cell counts are five leaves, one per cell type (1/480 each, 1/96 in total). Suppose a blood test question routes to both, and the answer skips hemoglobin but lists all five white blood cell counts:

  • Plain average: the score is 5/6 = 0.83. Missing hemoglobin costs one sixth.
  • Tree weights: the score is 0.50. Missing hemoglobin costs as much as missing all five white blood cell counts.

With a plain average, importance simply follows leaf count. The same applies to the four nested length rules (under 119, 238, 476 and 714 words): together they weigh 1/40, the same as the single rule "give a short summary up front".

Clinical importance comes in through curation and routing. Curation keeps safety concepts as few, heavy leaves. From Appendix J:

Branch (Appendix J)LeavesTotal weightWeight per leaf
User Health Memory651/21/480 to 1/36
Safety & Guardrails121/81/72 (Emergency, Diagnosis); 1/144 (Risk conflicts)
Expertise101/81/120 to 1/40
Communication121/81/160 (length, structure) to 1/40
Agentic Action101/81/120 to 1/40

Each emergency and diagnosis safety leaf weighs 1/72: 2.2x a formatting leaf and 6.7x a single white blood cell count. Routing matters even more: the final score is normalized over the selected leaves only, so a safety leaf competes only with what is relevant to that query, not with the whole catalog.

Direct weighting test. We hold the active leaves and their Boolean outputs fixed and change only the aggregation, from plain average to tree weights (rows 3 and 4 of the HealthBench table above). The feedback shows each failed check with its weight, so the revising model knows which failures matter most. Tree weights raise the HealthBench score from 0.2664 to 0.2854 (+0.0190) and grader agreement from 254 to 267 of 300 pairs (+4.3 points). So the weights help, in the expected direction, without any tuning.

We did not fit a severity weight per leaf on purpose. It would add 109 free parameters, require a separate physician elicitation study in which physicians would disagree, and invite tuning toward the test set. The test above shows the untuned tree weights already beat a plain average. In the revision we will describe them as a parameter-free default that controls for granularity, not as a calibrated clinical severity scale, and handle safety at the routing step as shown next.

W2. Routing may skip safety-critical rubrics. What does that do to the score?

These errors are already inside the reported numbers. Every experiment in the paper runs the router; none uses expert-given activations (Section 3). ICC 0.876 and the perturbation results therefore already include routing mistakes. The Inappropriate Instructions setting injects unsafe instructions, which are caught mainly through safety leaves; if routing often dropped those leaves, these cases would go unpenalized. RubricsTree lowers the score in 93.5% of them, vs 71.2% for the Principle Baseline (Section 3.2).

Routing accuracy by scenario (Appendix E, Tables 5 and 6). Router selections are compared with expert selections made from scratch, and with experts who review and correct the router's suggestions:

ScenarioFrom scratch: precisionFrom scratch: recallWith expert review: precisionWith expert review: recall
Action Plan77.49%71.79%85.02%94.89%
Medical Explanation82.86%58.00%94.29%89.19%
Health Data79.17%70.37%95.83%76.67%
Symptoms50.00%75.00%76.67%92.00%
Average73.40%69.39%87.36%89.53%

The router leans toward recall where risk is highest. Symptoms, where emergency and red-flag rubrics live, has the highest from-scratch recall (75.00%) and the lowest precision (50.00%). This matches the lower trigger threshold for emergency-type queries in Section 2.3.

Measuring omissions directly. On fixed HealthBench base answers, we check every safety leaf for applicability and score it. This finds 120 cases where a safety leaf applies and the answer fails it. We then count how many each selector skips, and how much the score rises as a result. All rows use the same leaf judgments and the same tree weights, so the only difference is which leaves are selected.

SelectorFailed safety checks skippedScore inflation vs checking all
Flat relevance32/120 (26.7%)+0.048
Tree routing (as in paper)12/120 (10.0%)+0.014
Tree routing + safety screen3/120 (2.5%)+0.003

The gap between flat and tree is larger for safety (32 vs 12) than for routing overall (Table 4). One reason is that safety leaves sit under one parent. Once the tree judges Safety & Guardrails relevant, it considers all twelve children together; a flat selector judges each safety leaf on its own wording, and checks such as "did the agent ask about pregnancy before suggesting a drug" look unrelated to many queries when read alone.

The safety screen re-checks up to four unselected safety leaves per query, ranked using query and context only. It also improves the downstream result:

SettingHealthBench after revisionTokens per queryLatency per query
Tree routing (as in paper)0.28547,64013.2 s
Tree routing + safety screen0.28918,92015.8 s

So routing omissions are measurable and small: they inflate the score by 0.014 on a 0 to 1 scale. The screen removes three quarters of them for 17% more tokens and 2.6 s more latency. The reference here is an exhaustive automated check, not new physician labels: it measures what routing skips compared with checking everything.

Reviewer_2P1yOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper decomposes health-agent response quality into a tree of more than 100 evaluation criteria. It selects relevant criteria for each query, checks them individually, and aggregates the results. Its central idea is to replace a vague judgment of whether an answer is good with explicit checks of which requirements are satisfied or missing. The main contributions are an expert-curated hierarchy of evaluation criteria, a context-dependent routing mechanism, and systematic validation of the evaluator itself. The paper examines agreement with experts, sensitivity to contextual perturbations, scoring stability, and whether the resulting feedback helps improve other modelsresponses.

Critical strengths
  • Produces transparent evaluations and actionable feedback. Decomposing overall quality into explicit criteria helps identify omissions and errors and directly supports response refinement.
  • Provides broad validation of the evaluator. Experiments examine expert agreement, perturbation detection, scoring stability, and downstream optimization, with an independent expert panel used for evaluation.
Critical weaknesses
  • The independent contribution of the tree structure is insufficiently established. Improvements may primarily arise from more detailed expert criteria rather than the hierarchy or routing mechanism. Authors should use identical atomic criteria and matched budgets, to compare flat checklists, relevance-based selection, and hierarchical routing, and also if applicable, ablate hierarchical weighting.
  • The perturbation evaluation does not independently establish that response quality deteriorates. Although the rubric tree includes appropriate refusal and clarification criteria, the perturbation protocol treats score decreases as evidence of evaluator reliability without reporting paired expert judgments confirming that the resulting responses are worse. Authors should obtain blinded expert comparisons of clean and perturbed responses, including cases where appropriate clarification or refusal preserves quality, and evaluate score changes against these judgments.
Questions for authors

Missing detailed implementation settings:

  • Appendix B.7 states that the models are trained and evaluated on a user-facing subset of HealthBench-Consensus, but does not specify the split sizes or overlap checks. Could the authors clarify how training, validation, and test examples were separated and deduplicated, and how checkpoints were selected? In particular, did the test scores shown in Figure 7 inform checkpoint or hyperparameter selection?
  • Appendix B.4 reports 532 internal queries. Were all of them included in the expert-agreement analysis in Section 3.1? Please provide category-level counts of independent queries and criterion-level judgments.
  • Figure 3 reports a Health Data κ of 1.000. Could the authors provide the underlying agreement counts, pass/fail distribution, and uncertainty estimates, together with representative evaluated criteria, to help interpret this result?
  • Appendix B.3 describes binarization and averaging pairwise agreement across the panel, whereas Section 3.1 presents κ as agreement with experts. Please specify whether the pairings are model–expert or expert–expert, the binarization thresholds, and how unselected rubrics and pairwise values are handled.
6 comments

Thank you for the detailed and constructive questions. Below we give the same-rubric ablation you asked for, a test of whether perturbed answers are really worse, and the missing protocol details.

W1. Isolate the tree: same atomic criteria, matched budgets, flat vs relevance-based vs hierarchical selection, and an ablation of the weighting.

Selector only, from the appendix. Appendix D, Table 4 compares selectors over the same rubric tree against expert activation labels:

Selector over the same rubric treeActivation accuracyRouting latency per query
Flat, embedding similarity61.25%3.2 s
Flat, per-leaf LLM judge76.14%64.2 s
Tree traversal (ours)80.61%5.6 s

End to end, on public data. Setup: 362 user-facing HealthBench-Hard queries (the set in Figure 6), Gemini-2.5-Flash, fixed initial answers, one round of feedback and revision, scored by HealthBench's physician-written rubrics and grading prompt, with Gemini-3-Flash as the grader. All arms use the same rubrics, leaf judge and prompt. Selectors see only the query and context, as in Section 2.3. Flat relevance and tree routing share an 8,000-token budget per query, including routing. "Full catalog" checks every rubric for applicability and goes over that budget, so we report it at its own cost. HealthBench's own rubrics are used only for final grading. "Agrees with HealthBench grader" counts how often an evaluator prefers the same answer as the grader, on the same 300 cached answer pairs for every evaluator.

Same rubrics, different evaluatorHealthBench after revisionGain over base (0.1944)Agrees with HealthBench graderTokens per queryLatency per query
Full catalog, plain average0.2476+0.0532219/300 (73.0%)15,24046.8 s
Flat relevance, plain average0.2518+0.0574230/300 (76.7%)7,82036.4 s
Tree routing, plain average0.2664+0.0720254/300 (84.7%)7,61013.2 s
Tree routing, tree weights (ours)0.2854+0.0910267/300 (89.0%)7,64013.2 s

Rows 2 and 3 isolate selection: same rubrics, same budget, same plain average. Tree routing scores 0.0146 higher, agrees with the HealthBench grader 8.0 points more often, and runs 2.8x faster. Rows 3 and 4 isolate weighting, with the selected leaves and their judge outputs held fixed: tree weights add another 0.0190 and 4.3 points of grader agreement. Checking every rubric (row 1) is the weakest option and costs twice the tokens, so the gain does not come from using more rubrics. Our full method scores 0.2854, in line with 0.2860 for the same setting in Figure 6. We will add this table to Appendix D.

W2. A lower score after a perturbation does not prove the answer got worse. Clarifying or refusing can be the right response.

Agreed. DR measures whether the score drops, not whether quality drops. Here is the evidence that RubricsTree's drops follow real quality loss.

  1. Test against an independent quality reference. On HealthBench we keep each question and its physician-written rubric fixed, change only the context given to the model, and grade both answers with the same HealthBench grader. We then split the pairs by the grader's verdict and check how often RubricsTree lowers its score in each group:
HealthBench grader says the perturbed answer isPairsRubricsTree lowers its score
Worse244219 (89.8%)
Equal or better (includes clarifying and declining)11814 (11.9%)

RubricsTree lowers its score for 9 in 10 answers that really got worse, and for only about 1 in 8 answers that did not. It follows answer quality, not the mere presence of a perturbation.

  1. The rubric rewards good clarification. One rubric passes when the agent asks the user for missing data, and another passes when it asks the user to clear up an ambiguous question. An answer that handles missing context well loses nothing on these. This fits Table 2: DR is lowest for the two "missing" perturbations (62.9% to 88.6%), where asking is often correct, and highest for Inaccurate User Data (97.1%) and Inappropriate Instructions (93.5%), where the answer is worse by construction because it builds on fabricated values or follows an unsafe instruction.
  2. Little over-penalizing at the criterion level. When the evaluation panel marked a criterion as passed, the judge marked it as failed in only 2.7% of cases. We will add this number to Appendix B.

We could not run a new blinded expert comparison within the discussion period. In the revision we will call DR "perturbation sensitivity", remove wording that suggests every perturbation must be penalized, and add the HealthBench analysis above.

Q1. HealthBench-Consensus splits, deduplication, and whether the Figure 7 test scores informed checkpoint or hyperparameter selection.

Splits. From the 3,671 HealthBench-Consensus examples we kept single-turn, user-facing queries (dropping multi-turn and clinician-facing ones), which leaves 502. A fixed rule sends every tenth example to the test set: 452 for training and 50 for test. There is no validation split, because nothing was selected on held-out data.

Deduplication. Every example has a unique ID, no query text appears twice, and no test query shares its id or text with a training query. We also checked near duplicates: the most similar test and training queries share only 27% of their words. All examples are single-turn, so no conversation spans both splits.

Training and selection. We ran one training run per model size, all with the same configuration: GRPO with LoRA (rank 16, alpha 32), KL coefficient 0.2, 4 samples per prompt, batch size 4, at most 2,048 generated tokens, 450 steps. These settings were fixed before training and were not tuned on the test curves. The training reward is the RubricsTree score. The test score is a different signal: the physician-written Consensus criteria, applied with HealthBench's grading prompt and Gemini-3-Flash as the grader. The model is never trained on the metric we report.

We did not pick a checkpoint. Figure 7 plots the test score along the whole run, and the highlighted numbers (66.7%, 55.3%, 40.3%) are the peaks of those curves. Because peaks overstate the effect, here are the gains with no selection at all:

Qwen3Step 0Final stepGain at final stepMean of last three checkpoints (steps 300 to 420)Gain
0.6B0.1300.200+53.8%0.194+49.5%
1.7B0.1570.190+21.0%0.211+34.4%
4B0.2070.233+12.6%0.260+25.6%

All three models still score above their starting point at the end of training. We will add this table to Appendix B.7, label the highlighted values in Figure 7 as peaks, and note that the test set has 50 queries.

Q2. Were all 532 internal queries used in the expert agreement analysis? Please give per-category counts.

No. The 532 queries are the full internal evaluation set, used for the perturbation and stability experiments. For the expert agreement study, we sampled queries from the 532 so that all four scenarios are covered, and all six experts rated each sampled response. The counts below refer to distinct response–criterion pairs, counted once regardless of the number of annotators.

ScenarioRubricsTree samples (annotations)Principle Baseline samples (annotations)
Medical Explanation15378
Health Data11278
Advice / Action Plan309156
Symptoms14878
Total722390

RubricsTree counts differ by scenario because experts only label the rubrics that apply to each query. We will add this table to Appendix B.4.

Q3. Health Data κ = 1.000: counts, pass/fail rates, uncertainty and example criteria.

Figure 3 reports κ by scenario:

Scenario (Figure 3)Principle Baseline κRubricsTree κ
Health Data0.5671.000
Action Plan0.3640.806
Symptoms0.5400.724
Medical Explanation0.3040.657
Overall0.4310.787

Health Data rubrics are mostly coverage checks on named values from the user's own records, for example "Does the response cover HbA1c?" (a standard blood sugar test) or "Does the response cover the mean sleep duration and accurately reflect the real value?" (Appendix J). Whether a response states a given value is close to an objective fact, so there is little room for the judge and the experts to disagree. In this scenario, every cell rated by both the judge and an expert matched, which gives κ = 1.000. Other scenarios contain criteria that need clinical judgment, such as whether the reasoning is explained or the plan has a follow-up trigger, and there κ is lower (0.657 to 0.806). We read the 1.000 as evidence that coverage checks are easy to verify, not as a claim of perfect agreement in general, and we will add the cell counts and a bootstrap interval to Appendix B.3.

Reviewer_xBG1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RubricsTree, a hierarchical rubric-based framework for evaluating open-ended responses generated by personal health agents. The framework organizes over 100 expert-curated atomic Boolean rubrics into a clinical taxonomy and introduces context-aware adaptive routing to select relevant evaluation criteria for each query. A hierarchical auto-weighting mechanism aggregates individual rubric judgments into evaluation scores. The authors further develop a meta-evaluation protocol covering human expert agreement, robustness to contextual perturbations, evaluation consistency, and downstream optimization utility.

Critical strengths
  1. The paper addresses an important challenge in evaluating open-ended personal health agents, where traditional question-answering benchmarks and holistic LLM-based evaluators may fail to adequately assess clinical accuracy, safety, personalization, and tool-use behavior.

  2. The proposed hierarchical taxonomy covers multiple aspects of personal health evaluation, including clinical knowledge, user health data, communication, safety, and agentic actions. The involvement of domain experts in rubric construction provides a reasonable foundation for developing structured evaluation criteria.

  3. The paper investigates expert agreement, contextual perturbation sensitivity, cross-model evaluation stability, and downstream optimization. Evaluations across different LLM families and the use of HealthBench provide useful evidence of the potential applicability of the framework.

Critical weaknesses
  1. My main concern is that Fine-grained Boolean evaluation criteria and adaptive rubric selection have been explored in prior work, particularly the Adaptive Precise Boolean Rubrics framework [1]. RubricsTree extends these ideas through an expert-curated hierarchical taxonomy, context-aware tree-based routing, and broader evaluation of personal health agents. These extensions are meaningful and potentially valuable. Nevertheless, the paper would benefit from a clearer discussion of its methodological distinctions from existing adaptive rubric-based evaluators, particularly regarding the advantages of hierarchical organization and auto-weighting. Additional controlled comparisons would further strengthen the novelty claims.

[1] N. Mallinar, A. A. Heydari, X. Liu, et al., “A Scalable Framework for Evaluating Health Language Models,” npj Digital Medicine, vol. 9, article no. 437, 2026. https://doi.org/10.1038/s41746-026-02492-x.

  1. The ablation studies is inadequate to establish the contributions of individual components. The experimental comparisons primarily demonstrate that the complete RubricsTree framework outperforms a principle-based baseline. However, these approaches differ simultaneously in rubric granularity, scoring format, hierarchical organization, routing strategy, and aggregation mechanism. Therefore, the observed improvements cannot be attributed specifically to the proposed hierarchical taxonomy or adaptive routing.

  2. Although the reported expert-alignment improvements are encouraging, the evaluation protocol raises several concerns. The minimum experience requirement for the independent evaluation panel is inconsistently reported as at least five years in Section 3.1 and at least three years in Appendix B.2. The primary evaluation also relies on an internal dataset of 532 health queries that is not publicly released, limiting independent verification.

  3. Although the framework emphasizes clinically verifiable Boolean criteria, Appendix J reveals that a considerable number of rubrics evaluate superficial formatting or stylistic characteristics rather than clinical correctness. Examples include requiring Markdown headings, tables, specific reasoning connectives, empathetic expressions, and predefined response-length limits. These properties do not necessarily correlate with the medical accuracy or safety of a response. Moreover, several criteria are redundant or highly correlated, potentially introducing unintended scoring biases.

Questions for authors

See Weakness.

6 comments

Thank you for the insightful review and for pointing to specific rubrics in Appendix J. Below we explain what is new relative to Auto-Eval, give the component evidence with the same rubrics, fix the panel description, and explain how much weight the formatting rules carry.

W1. Auto-Eval (the Adaptive Precise Boolean Rubrics framework you cite) already uses Boolean criteria and adaptive selection. What do the hierarchy and auto-weighting add?

We agree that Boolean criteria are motivated from prior work, including Auto-Eval (Mallinar et al., 2026), and we do not claim them as new. We actually highlight Auto-Eval in our Introduction and would further refine the comparison with Auto-Eval: its human comparison used 20 base queries under four personalization conditions and it includes a separate study on real participant data focusing data coverage.

What RubricsTree adds:

  1. Selecting rubrics through a tree instead of one by one. Auto-Eval decides relevance for each criterion separately. Appendix D, Table 4 already runs that approach on our rubric tree: the flat per-leaf LLM judge reaches 76.14% activation accuracy at 64.2 s per query, while tree traversal reaches 80.61% at 5.6 s. Same rubrics, 4.47 points more accurate and 11.5x faster. End to end on HealthBench (table under W2), tree routing scores 0.0146 higher than this flat selection at the same token budget and is 2.8x faster.
  2. Weights that do not depend on how finely a topic is split. Hemoglobin is one leaf (weight 1/96). The white blood cell counts are five leaves, one per cell type (1/480 each, 1/96 in total). With a plain average, an answer that skips hemoglobin but lists all five white blood cell counts scores 0.83. With tree weights it scores 0.50, so missing hemoglobin costs as much as missing all five counts. Under a plain average, importance would follow leaf count. On HealthBench, tree weights add 0.0190 over a plain average with everything else fixed (table under W2).
  3. Scope. 109 published leaves (Appendix J) covering longitudinal health memory, safety, communication and multi-step tool use, curated from about 4,000 real queries by a physician-led panel. We also expand the experimental settings from internal and external benchmarks (with 3 optimization experiments). While Auto-Eval only uses 20 queries within the scope of the user-data recall binary judge based on limited Gemini models with no external benchmark verifications.

We also added new comparison experiments with Auto-Eval end to end on the Persona 2 perturbation set. Each cell gives two numbers: the average score drop after the perturbation (%), and the share of cases where the score drops at all (%). Higher is better for both.

ScenarioEvaluatorMissing instructionsMissing dataInappropriate instructionsInaccurate data
ExplanationAuto-Eval-7.8 / 25.029.4 / 100.014.6 / 75.01.2 / 50.0
ExplanationPrinciple Baseline-6.2 / 25.017.1 / 75.021.8 / 100.0-9.7 / 25.0
ExplanationRubricsTree4.8 / 75.040.2 / 100.040.9 / 100.084.3 / 100.0
Health DataAuto-Eval1.2 / 50.023.1 / 66.718.8 / 100.010.2 / 50.0
Health DataPrinciple Baseline13.0 / 100.022.5 / 66.714.4 / 100.03.3 / 66.7
Health DataRubricsTree3.9 / 66.734.5 / 66.731.4 / 100.078.5 / 100.0
Action PlanAuto-Eval-18.6 / 25.032.3 / 100.021.5 / 75.0-3.8 / 50.0
Action PlanPrinciple Baseline-15.9 / 25.034.0 / 100.030.0 / 100.00.6 / 75.0
Action PlanRubricsTree5.6 / 50.036.3 / 100.030.8 / 100.082.3 / 100.0
SymptomsAuto-Eval-4.1 / 16.720.8 / 66.76.6 / 83.3-4.9 / 16.7
SymptomsPrinciple Baseline-1.3 / 16.721.6 / 50.010.3 / 100.0-1.5 / 33.3
SymptomsRubricsTree3.6 / 83.323.2 / 100.031.1 / 100.081.1 / 100.0

RubricsTree has a larger score drop than Auto-Eval in all 16 cells, and catches the perturbation more often in 12 cells and equally often in 4. The clearest gap is Inaccurate User Data: RubricsTree catches 100% of cases in all four scenarios, Auto-Eval catches 50%, 50%, 50% and 16.7%.

Downstream use across models (Figure 6, Appendix B.6). On 362 user-facing HealthBench-Hard queries, giving RubricsTree as a prompt and as one round of feedback improves all eight models:

ModelBaseRubrics as promptRubric feedbackGain from feedback
Gemini-2.5-Flash0.19440.26330.2860+47.1%
Gemini-2.5-Pro0.23510.28640.3683+56.7%
Gemini-3-Flash0.29640.37000.3849+29.9%
Gemini-3-Pro0.25490.37210.3996+56.8%
GPT-5.4 nano0.32440.36440.3849+18.6%
GPT-5.4 mini0.20630.24600.3433+66.4%
GPT-5.40.26560.32230.3947+48.6%
GPT-5.4 Pro0.34330.42620.4499+31.0%

W2. The Principle Baseline differs from RubricsTree in many ways at once, so the gain cannot be traced to the hierarchy or the routing.

Agreed for the main comparison, so we separate the pieces with the rubrics held fixed.

Selection, from the appendix. Table 4 (above) isolates the selector on the same rubric tree. Appendix E then compares router selections with expert selections:

Router vs experts (Appendix E)AccuracyPrecisionRecallF1
Experts select from scratch (Table 5)80.61%73.40%69.39%70.35%
Experts review the router's output (Table 6)92.66%87.36%89.53%87.95%

Stability, from the appendix. Across temperatures, judge backbones, prompt roles and scenarios, the per-item variance of RubricsTree stays between 0.002 and 0.005, while the Principle Baseline ranges from 0.005 to 0.018 (Appendix F, Figure 9).

End to end on HealthBench, one change at a time. Setup: 362 user-facing HealthBench-Hard queries, Gemini-2.5-Flash, fixed initial answers, one round of feedback and revision, scored by HealthBench's physician-written rubrics and grading prompt, with Gemini-3-Flash as the grader. All arms use the same rubrics, judge and prompt. Flat relevance and tree routing share an 8,000-token budget per query; "full catalog" checks every rubric and is reported at its own cost. "Agrees with HealthBench grader" counts how often an evaluator prefers the same answer as the grader on 300 fixed answer pairs.

Same rubrics, different evaluatorHealthBench after revisionGain over base (0.1944)Agrees with HealthBench graderTokens per queryLatency per query
Full catalog, plain average0.2476+0.0532219/300 (73.0%)15,24046.8 s
Flat relevance, plain average0.2518+0.0574230/300 (76.7%)7,82036.4 s
Tree routing, plain average0.2664+0.0720254/300 (84.7%)7,61013.2 s
Tree routing, tree weights (ours)0.2854+0.0910267/300 (89.0%)7,64013.2 s

Rows 2 and 3 isolate selection: same rubrics, same budget, same plain average. Tree routing scores 0.0146 higher, agrees with the HealthBench grader 8.0 points more often, and runs 2.8x faster. Rows 3 and 4 isolate weighting, with the selected leaves and their judge outputs held fixed: tree weights add another 0.0190 and 4.3 points of grader agreement. Checking every rubric (row 1) is the weakest option and costs twice the tokens, so the gain does not come from using more rubrics. Our full method scores 0.2854, in line with 0.2860 for the same setting in Figure 6. We will add this table to Appendix D.

W3. The panel experience requirement is inconsistent, and the 532-query set is private.

Thank you for catching this. The correct values are at least five years for the curation panel and at least three years for the evaluation panel, as in Appendix B.2. The 532 queries come with each user's sensor streams and biomarker records, so we cannot release them as they are. A de-identified synthetic version is in internal review. What can be checked today: the full rubric tree with every prompt and weight (Appendix J), the code (Appendix B.1), and all HealthBench results on public data, including the new ablations above. We will also report the number of queries and labels used in the agreement study separately from the 532-query total.

W4. Many rubrics check formatting or style, and some of them overlap.

This is a fair point, and we quantify it below.

  • They carry little weight. The rules you list (four length rules, four structure rules including headings and tables, two tone rules, and the rule asking for words like "because" or "therefore") hold 1/12 of the total weight, about 8%. Safety alone holds 1/8 and health memory holds 1/2.
  • Overlapping rules do not stack. Nested rules sit under one parent, so the four length thresholds (119, 238, 476 and 714 words) together weigh 1/40, the same as the single rule "give a short summary up front". The paragraph rules (3 or 5 sentences) and tool step rules (5 or 8 steps) work the same way.
  • Why they are there. During curation, real users often complained about long and hard-to-read answers. Physicians grade this too. In the 362 HealthBench-Hard queries we use, 250 of the 4,485 physician-written criteria (5.6%) are tagged as communication quality, 170 of the 362 queries have at least one such criterion, and these criteria carry 3.5% of the positive points. Communication is part of how physicians judge a health answer, with a small share. RubricsTree treats it the same way, at about 8% of the weight.
  • The gains do not come from them. Removing all these surface rules from the feedback changes the HealthBench result only from 0.2854 to 0.2828, so 97% of the gain over the base model (0.1944) remains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.