acceptodds
Under review as a conference paper at ICLR 2027

CARE-Bench: Evaluating Long-Horizon Patient-Advocacy Agents

Abstract

Patient advocacy requires coordinating insurers, providers, suppliers, and community organizations while discovering hidden information, satisfying authorization requirements, tracking processes, and confirming outcomes. We introduce CARE-Bench, a benchmark of 47 de-identified, expert-reviewed real-world patient-advocacy cases compiled into interactive case simulations. Evaluated agents receive a case-specific brief, general advocacy playbooks, a running case file, and eight general-purpose tools that support end-to-end care coordination work. Across 14 frontier models, Opus 5 completes every required outcome group in 81.6% of runs. Opus 5 and Grok 4.6 tie for the highest strict-pass rate at 52.5%. On hard worlds, the highest strict-pass rate is 29.0%. These results show that current agents often reach required states but do not consistently obtain required permissions, support their actions with verified evidence, or avoid prohibited actions.

Primary area:
datasets and benchmarks
Keywords:
evaluation, benchmark, healthcare, LLM, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yRUVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CARE-Bench, a benchmark with 47 patient-advocacy cases that are interacted with through simulation. Evaluated agents need to discover information, coordinate multiple parties, and complete administrative work. The evaluation covers 14 models and compares task completion as well as satisfaction with procedural standards. The main empirical finding is that agents can complete the required outcomes but often fail to meet these standards such as providing evidence and obtaining required permissions.

Critical strengths
  • The paper addresses an interesting and practically relevant evaluation setting. The open-endedness of the evaluation is particularly compelling, but has inherent limitations (see below). The simulation is intended to capture a real-world setting in which coordination across multiple providers is required.
  • The distinction between just completing a task and how the task is completed is a nice aspect of the evaluation.
  • The evaluation pipeline itself includes a number of deterministic consistency checks.
  • The paper is easy to follow and the positioning relative to other benchmarks is clear. The discussion of related work is thorough, and I have not seen a benchmark in this domain before.
Critical weaknesses
  • The benchmark itself is quite small; only 47 cases is not necessarily large enough to provide a meaningful evaluation. All cases come from the same platform as well, which may not reflect an accurate evaluation of the same task for a different platform and population.
  • While I admire how open-ended the tasks are, I have concerns that the evalution is not thorough enough. The agent interacts through natural language, while each party has a small selection of actions to respond with, and success is defined through predefined outcome groups. The paper needs more convincing evidence that reasonable interpretations of the task and reasonable alternative approaches are appropriately supported. Without this validation, it is difficult to determine whether failures reflect limitations of the agent or limitations of the simulation.
  • A human baseline using the same interface would be valuable. It would help determine whether the requirements are reasonable and whether expert judgments agree with the benchmark’s grades. The authors do acknowledge this, but this validation would strengthen the paper.
  • The expert-reviewing of cases is good, but this is left somewhat vague. Providing more detail on who these experts are and what the verification process entailed would strengthen the paper.
Questions for authors
  1. Who were the ‘expert reviewers’ (e.g., profession) and how were the scenarios verified and reviewed? Did experts validate the simulations and scoring criteria or just the cases?
  2. How can the simulation support reasonable, safe alternative workflows? Were failures or unsupported requests checked to distinguish agent errors from limitations of the simulator?
Reviewer_MuBfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARE-Bench evaluates long-horizon patient-advocacy agents through 47 interactive worlds derived from expert-reviewed real cases. It combines solvability verification with deterministic checks of outcome completion, authorization, and evidentiary support. Across 14 models and three runs per case, overall outcome completion reaches 46.7%, while strict success is only 33.8%.

Critical strengths
  • Practically meaningful evaluation target. The benchmark integrates cross-institution coordination, information discovery, and authorization constraints, capturing consequential capabilities that outcome-only evaluation can overlook.

  • Substantial solvability validation. Graph verification, public-tool replay, and gold-route experiments support task executability. The 254 outcome-complete but procedurally invalid runs demonstrate the diagnostic value of evaluating execution integrity separately.

Critical weaknesses
  • The semantic interface lacks independent validation. Requests are mapped to operations through voting by the same LLM, while conformance checks primarily cover oracle-derived requests. Without human auditing of actual model trajectories, it remains unclear how frequently translation errors contribute to failures or procedural violations (Appendices C.3 and D.4).

  • The fidelity of the final simulated worlds remains insufficiently validated. Expert review primarily concerns annotations before compilation. The paper does not report independent expert assessment after LLM compilation and friction insertion. Solvability establishes internal reachability, but does not establish that authorization rules, evidence requirements, and completion criteria faithfully preserve the source cases.

Questions for authors

These are clarification and validation questions that could change my assessment:

  • What are the translator’s false-positive and false-negative rates on independently annotated model trajectories, and how often would correcting these errors change strict-pass judgments?
  • Were the final compiled worlds independently reviewed by domain experts? What discrepancies were found in case fidelity, authorization rules, or completion criteria, and did correcting them affect the main results?
Reviewer_EGU9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors position CARE-Bench to fill a gap in existing literature. Unlike prior healthcare benchmarks (e.g., MedQA, HealthBench, SDBench) or general-purpose agent benchmarks (e.g., GAIA, SWE-bench), CARE-Bench uniquely combines real-case grounding, advocacy workflows, long horizons, complex friction, and evidence-backed operation checks. This construction prioritizes ecological validity and procedural rigor over synthetic scale. CARE-Bench is constructed from 47 de-identified, real-world patient-advocacy cases sourced from a care-advocacy platform. The benchmark simulates interactive, partially observable environments where agents begin with only a case brief and initial contacts, requiring them to discover hidden state through step-by-step interaction. Each task is structured around 8 general-purpose tools, explicit required outcomes, prohibited actions, and finally a deterministic, engine-enforced grading system that evaluates success based on the final world state rather than self-reported claims or LLM-based transcript audits

Critical strengths
  • The authors aimed to push a critical gap in the medical agents benchmark by adding a real-world gym based benchmark.
  • The paper makes a concerted effort to make sufficient use of the rather small number of use cases such as multiple seeds, Deterministinc grading, and state based evaluation
  • The authors were able to identify clear gaps in frontier models which can drive further reliable adoption of agents/LLMs for medical use-cases
Critical weaknesses
  • The authors added several guardrails such as public-action limit and maximum model response to stop the possiblity of "gaming" the benchmark. While these hard limits prevent infinite loops and control compute costs, relying solely on action/response caps is generally insufficient to capture true procedural difficulty. The engine base forbidden action rules adds procedural rigor but without explicit metrics for friction, decision branching or infromation discovery depth, this might function more as runtime constraints than meaningful proxies.
  • The authors made some clever methodological contribution to address the rather small number of base use cases. It would be useful to add confidence intervals or bootstrapped standard errors for the Pass@1/Pass3 metrics to demonstrate that trials provide adequate statistical power. Also, it would be useful to quantify how the setup compares in variance and reliability to larger benchmarks like GAIA or SWE-bench, justifying the trade-off between scale and fidelity
Questions for authors
  • how does the formulaic complexity metric interacts with the limits highlighted in weakness 1? Are the cases empirically balanced for varying friction levels?
  • Could you have added parametrized templates or controlled variation (e.g., altered authority constraints, shuffled information reveal order) derived from the 47 core cases to expand evaluation diversity without sacrificing real-case grounding
  • Can you justify the action/response limit with pilot data (e.g., human expert baselines or saturation curves) rather than treating them as arbitrary caps

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.