- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RubricsTree, an open-ended evaluation framework for personal health agents. It uses an expert-curated hierarchical taxonomy to decompose complex medical evaluation tasks into atomic Boolean rubrics, with adaptive routing to select relevant rubrics and hierarchical weighting to calculate evaluation scores. The framework was developed based on approximately 4,000 real user queries and evaluated through expert agreement, contextual perturbation tests, evaluation consistency experiments, and downstream model optimization. The results show that RubricsTree outperforms the Principle Baseline in expert agreement, perturbation detection, and evaluation stability, while also providing benefits for downstream model optimization.
Critical strengths
- This paper addresses the evaluation of open-ended responses generated by personal health agents, an important problem that requires consideration of users’ health backgrounds and personalized information.
- The authors developed fine-grained evaluation rubrics with the help of medical and health experts and evaluated the framework against an independent expert panel. The experiments also cover contextual perturbations, evaluation stability, and downstream model optimization, providing evidence of the framework’s effectiveness from multiple perspectives.
Critical weaknesses
- The main contribution appears to lie in the expert-curated rubrics, while the technical novelty of the hierarchical taxonomy, adaptive routing, and auto-weighting mechanisms is relatively limited. The evaluation primarily compares RubricsTree with the Principle Baseline, which differs in both the granularity of evaluation criteria and the incorporated domain knowledge. Although the supplementary material includes routing ablations, a comparison with a flat rubric-based evaluator using the same expert rubrics is missing, making it difficult to isolate the contributions of the proposed architecture.
- The hierarchical taxonomy and weighting scheme are not sufficiently justified, particularly given that different clinical criteria may have varying levels of importance. Moreover, adaptive routing may omit safety-critical rubrics, potentially affecting the final evaluation scores. Despite the reported routing accuracy, the impact of such omissions on evaluation reliability is not sufficiently analyzed.
Questions for authors
Q1. Could the authors compare RubricsTree with a flat rubric-based evaluator using the same expert rubrics to clarify the contributions of the hierarchical taxonomy, adaptive routing, and auto-weighting mechanisms?
Q2. How were the hierarchical taxonomy and weighting scheme validated? Why was recursive equal-weight allocation among sibling nodes chosen instead of assigning weights based on the clinical importance of different criteria?
6 comments
Q2. How were the taxonomy and weights validated, and why split weight equally among siblings instead of by clinical importance?
Validation. The taxonomy was built by a nine-person, physician-led curation panel over about 4,000 real queries (Appendix B.2) and tested against a separate six-person evaluation panel with no shared members (Section 3.1, Figure 3):
| Agreement with the evaluation panel (Figure 3) | Principle Baseline | RubricsTree |
|---|---|---|
| ICC3, overall | 0.291 | 0.876 |
| Cohen's κ, overall | 0.431 | 0.787 |
| κ, Health Data | 0.567 | 1.000 |
| κ, Action Plan | 0.364 | 0.806 |
| κ, Symptoms | 0.540 | 0.724 |
| κ, Medical Explanation | 0.304 | 0.657 |
Equal splitting keeps a topic's weight independent of how finely it was split. Leaves have to be atomic so a judge can verify them. Some topics need several leaves for that, others need one. Equal splitting stops this bookkeeping choice from turning into importance. Example from Appendix J: hemoglobin is one leaf (weight 1/96). The white blood cell counts are five leaves, one per cell type (1/480 each, 1/96 in total). Suppose a blood test question routes to both, and the answer skips hemoglobin but lists all five white blood cell counts:
- Plain average: the score is 5/6 = 0.83. Missing hemoglobin costs one sixth.
- Tree weights: the score is 0.50. Missing hemoglobin costs as much as missing all five white blood cell counts.
With a plain average, importance simply follows leaf count. The same applies to the four nested length rules (under 119, 238, 476 and 714 words): together they weigh 1/40, the same as the single rule "give a short summary up front".
Clinical importance comes in through curation and routing. Curation keeps safety concepts as few, heavy leaves. From Appendix J:
| Branch (Appendix J) | Leaves | Total weight | Weight per leaf |
|---|---|---|---|
| User Health Memory | 65 | 1/2 | 1/480 to 1/36 |
| Safety & Guardrails | 12 | 1/8 | 1/72 (Emergency, Diagnosis); 1/144 (Risk conflicts) |
| Expertise | 10 | 1/8 | 1/120 to 1/40 |
| Communication | 12 | 1/8 | 1/160 (length, structure) to 1/40 |
| Agentic Action | 10 | 1/8 | 1/120 to 1/40 |
Each emergency and diagnosis safety leaf weighs 1/72: 2.2x a formatting leaf and 6.7x a single white blood cell count. Routing matters even more: the final score is normalized over the selected leaves only, so a safety leaf competes only with what is relevant to that query, not with the whole catalog.
Direct weighting test. We hold the active leaves and their Boolean outputs fixed and change only the aggregation, from plain average to tree weights (rows 3 and 4 of the HealthBench table above). The feedback shows each failed check with its weight, so the revising model knows which failures matter most. Tree weights raise the HealthBench score from 0.2664 to 0.2854 (+0.0190) and grader agreement from 254 to 267 of 300 pairs (+4.3 points). So the weights help, in the expected direction, without any tuning.
We did not fit a severity weight per leaf on purpose. It would add 109 free parameters, require a separate physician elicitation study in which physicians would disagree, and invite tuning toward the test set. The test above shows the untuned tree weights already beat a plain average. In the revision we will describe them as a parameter-free default that controls for granularity, not as a calibrated clinical severity scale, and handle safety at the routing step as shown next.
W2. Routing may skip safety-critical rubrics. What does that do to the score?
These errors are already inside the reported numbers. Every experiment in the paper runs the router; none uses expert-given activations (Section 3). ICC 0.876 and the perturbation results therefore already include routing mistakes. The Inappropriate Instructions setting injects unsafe instructions, which are caught mainly through safety leaves; if routing often dropped those leaves, these cases would go unpenalized. RubricsTree lowers the score in 93.5% of them, vs 71.2% for the Principle Baseline (Section 3.2).
Routing accuracy by scenario (Appendix E, Tables 5 and 6). Router selections are compared with expert selections made from scratch, and with experts who review and correct the router's suggestions:
| Scenario | From scratch: precision | From scratch: recall | With expert review: precision | With expert review: recall |
|---|---|---|---|---|
| Action Plan | 77.49% | 71.79% | 85.02% | 94.89% |
| Medical Explanation | 82.86% | 58.00% | 94.29% | 89.19% |
| Health Data | 79.17% | 70.37% | 95.83% | 76.67% |
| Symptoms | 50.00% | 75.00% | 76.67% | 92.00% |
| Average | 73.40% | 69.39% | 87.36% | 89.53% |
The router leans toward recall where risk is highest. Symptoms, where emergency and red-flag rubrics live, has the highest from-scratch recall (75.00%) and the lowest precision (50.00%). This matches the lower trigger threshold for emergency-type queries in Section 2.3.
Measuring omissions directly. On fixed HealthBench base answers, we check every safety leaf for applicability and score it. This finds 120 cases where a safety leaf applies and the answer fails it. We then count how many each selector skips, and how much the score rises as a result. All rows use the same leaf judgments and the same tree weights, so the only difference is which leaves are selected.
| Selector | Failed safety checks skipped | Score inflation vs checking all |
|---|---|---|
| Flat relevance | 32/120 (26.7%) | +0.048 |
| Tree routing (as in paper) | 12/120 (10.0%) | +0.014 |
| Tree routing + safety screen | 3/120 (2.5%) | +0.003 |
The gap between flat and tree is larger for safety (32 vs 12) than for routing overall (Table 4). One reason is that safety leaves sit under one parent. Once the tree judges Safety & Guardrails relevant, it considers all twelve children together; a flat selector judges each safety leaf on its own wording, and checks such as "did the agent ask about pregnancy before suggesting a drug" look unrelated to many queries when read alone.
The safety screen re-checks up to four unselected safety leaves per query, ranked using query and context only. It also improves the downstream result:
| Setting | HealthBench after revision | Tokens per query | Latency per query |
|---|---|---|---|
| Tree routing (as in paper) | 0.2854 | 7,640 | 13.2 s |
| Tree routing + safety screen | 0.2891 | 8,920 | 15.8 s |
So routing omissions are measurable and small: they inflate the score by 0.014 on a 0 to 1 scale. The screen removes three quarters of them for 17% more tokens and 2.6 s more latency. The reference here is an exhaustive automated check, not new physician labels: it measures what routing skips compared with checking everything.
Thank you for the review. You raise two questions: does the tree add anything beyond the expert rubrics themselves, and can routing or weighting hide a safety failure. We answer each with evidence already in the paper, worked examples from the published rubric table, and new same-rubric experiments on public HealthBench data. In short:
Once the rubrics are fixed, a flat evaluator and RubricsTree differ in only two places: which leaves get checked, and how the checks are added up. The leaf text, judge and prompt are the same. We test each place separately.
Same rubrics, different selectors (Appendix D, Table 4). The per-leaf judge below is exactly a flat evaluator over our rubrics: it asks an LLM, rubric by rubric, whether each one applies. This is also how Auto-Eval (Mallinar et al., 2026) adapts its criteria.
The tree is 4.47 points more accurate than the flat LLM judge and 11.5x faster. Embedding retrieval is faster still but 19.4 points less accurate; Appendix D shows it misses paraphrases such as "shortness of breath" vs "dyspnea".
Same rubrics, end to end on HealthBench. Setup: the 362 user-facing HealthBench-Hard queries from Figure 6, Gemini-2.5-Flash, fixed initial answers, one round of feedback and revision, scored by HealthBench's physician-written rubrics and grading prompt, with Gemini-3-Flash as the grader. All arms use the same RubricsTree leaves that apply to HealthBench, the same judge and the same prompt. Selectors see only the query and context, as in Section 2.3. Flat relevance and tree routing share an 8,000-token budget per query; "full catalog" checks every leaf and is reported at its own cost. HealthBench's own rubrics are used only for final grading. "Agrees with HealthBench grader" counts how often an evaluator prefers the same answer as the grader on 300 fixed answer pairs. Only the professional-skill leaves apply to HealthBench, so all latencies here are shorter than in Table 4.
What this shows:
Why the tree helps. A flat selector judges each of the 100+ rubrics in isolation. It pays for every irrelevant one (lipid panels, tool step counts) and has no parent context when a rubric is borderline. The tree drops an irrelevant branch with one decision and judges a leaf only after its parent is known to matter. So the expert rubrics supply the content, and the tree decides which of them to spend the budget on.