- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CARE-Bench, a benchmark with 47 patient-advocacy cases that are interacted with through simulation. Evaluated agents need to discover information, coordinate multiple parties, and complete administrative work. The evaluation covers 14 models and compares task completion as well as satisfaction with procedural standards. The main empirical finding is that agents can complete the required outcomes but often fail to meet these standards such as providing evidence and obtaining required permissions.
Critical strengths
- The paper addresses an interesting and practically relevant evaluation setting. The open-endedness of the evaluation is particularly compelling, but has inherent limitations (see below). The simulation is intended to capture a real-world setting in which coordination across multiple providers is required.
- The distinction between just completing a task and how the task is completed is a nice aspect of the evaluation.
- The evaluation pipeline itself includes a number of deterministic consistency checks.
- The paper is easy to follow and the positioning relative to other benchmarks is clear. The discussion of related work is thorough, and I have not seen a benchmark in this domain before.
Critical weaknesses
- The benchmark itself is quite small; only 47 cases is not necessarily large enough to provide a meaningful evaluation. All cases come from the same platform as well, which may not reflect an accurate evaluation of the same task for a different platform and population.
- While I admire how open-ended the tasks are, I have concerns that the evalution is not thorough enough. The agent interacts through natural language, while each party has a small selection of actions to respond with, and success is defined through predefined outcome groups. The paper needs more convincing evidence that reasonable interpretations of the task and reasonable alternative approaches are appropriately supported. Without this validation, it is difficult to determine whether failures reflect limitations of the agent or limitations of the simulation.
- A human baseline using the same interface would be valuable. It would help determine whether the requirements are reasonable and whether expert judgments agree with the benchmark’s grades. The authors do acknowledge this, but this validation would strengthen the paper.
- The expert-reviewing of cases is good, but this is left somewhat vague. Providing more detail on who these experts are and what the verification process entailed would strengthen the paper.
Questions for authors
- Who were the ‘expert reviewers’ (e.g., profession) and how were the scenarios verified and reviewed? Did experts validate the simulations and scoring criteria or just the cases?
- How can the simulation support reasonable, safe alternative workflows? Were failures or unsupported requests checked to distinguish agent errors from limitations of the simulator?