- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MedicalPressureBench, a benchmark for evaluating medical agents under six types of simulated workplace pressure. It includes 200 medical workflow tasks across five workflow functions, using a matched design that keeps clinical evidence, authorization, and task requirements fixed across pressure conditions. Experiments on five models show that completion quality generally decreases under pressure, while recorded risk can also decrease despite incomplete care.
Critical strengths
S1. The benchmark covers five medical workflow functions and defines explicit completion requirements and action boundaries. Importantly, the matched design keeps clinical evidence, authorization, tools, and completion requirements fixed across pressure conditions, providing a reasonable basis for controlled comparisons.
S2. The paper systematically introduces six types of workplace pressure into medical agent evaluation. Examining both unsafe execution and incomplete care is valuable, as avoiding unsafe actions does not necessarily mean that all required medical obligations have been completed.
Critical weaknesses
W1. It remains unclear whether the observed performance degradation specifically reflects workplace pressure or more general prompt-induced effects. This uncertainty affects the interpretation of the paper's central findings.
W2. The rationale for selecting the specific pressure combinations is insufficiently explained. This makes it difficult to assess the representativeness and generalizability of the composition findings.
Questions for authors
- How were the six workplace pressures constructed and validated, and how can their effects be distinguished from generic prompt-induced changes or direct behavioral guidance?
- What motivated the selection of the specific pressure combinations, and were they based on clinical plausibility or predefined hypotheses?
- Given the single rollout per model–task–condition, how robust are the reported findings, and could the authors provide uncertainty estimates or limited repeated evaluations?
- Which components of CQ and RES contribute most to the observed changes, and what specific agent behaviors or workflow failures explain these changes?
3 comments
This comment completes our response to Q3 and Q4 with the additional evaluation round, uncertainty estimates, and the behaviors behind score changes.
Q3
Thank you for suggesting repeated evaluation and uncertainty estimates. We added a second round on 20 tasks, four per workflow function, under all 17 conditions. The tasks were selected from the 117 with complete original records across all five model groups. The same deterministic scoring rules are used in both rounds, and only matching pairs with usable final states are scored. Section 5.3 (Table 4) and Appendix L describe the comparison.
The table reports additional minus original differences. SD is the sample standard deviation of paired differences. MAE is their mean absolute value, so positive and negative changes cannot cancel. Both retain the original score units. CI is the task bootstrap 95% interval for the mean difference.
| Model | Metric | Pairs | Mean | SD | MAE | 95% CI |
|---|---|---|---|---|---|---|
| GPT-5.6 | CQ | 340 | +0.62 | 22.03 | 16.26 | [-2.41, +3.42] |
| GPT-5.6 | RES | 340 | +2.23 | 14.81 | 4.73 | [+0.07, +5.32] |
| Sonnet-5 | CQ | 338 | +0.68 | 19.82 | 10.49 | [-1.20, +2.76] |
| Sonnet-5 | RES | 338 | +3.97 | 28.34 | 9.00 | [+1.18, +7.14] |
| Gemini-3.8 | CQ | 339 | +2.98 | 17.31 | 12.07 | [-0.75, +6.46] |
| Gemini-3.8 | RES | 339 | +1.03 | 36.89 | 19.42 | [-7.08, +8.82] |
| DeepSeek-V4.1 | CQ | 340 | +2.13 | 19.55 | 15.57 | [-1.66, +5.85] |
| DeepSeek-V4.1 | RES | 340 | +15.07 | 33.18 | 17.18 | [+7.11, +25.78] |
| Qwen3.8 | CQ | 339 | -23.48 | 28.06 | 29.58 | [-32.01, -15.13] |
| Qwen3.8 | RES | 339 | -54.60 | 46.76 | 54.60 | [-71.67, -37.33] |
We use 10,000 bootstrap samples over tasks, retaining all available condition pairs within each sampled task. The 17 conditions of a task are not treated as independent observations. GPT-5.6 has a mean CQ change of +0.62 but an MAE of 16.26. DeepSeek-V4.1 RES rises from 1.54 to 16.62.
We also compare the sign of each single pressure effect relative to baseline across rounds, using tasks with all four required trajectories. Agreement across six pressures is below.
| Model | CQ directions agree | RES directions agree |
|---|---|---|
| GPT-5.6 | 1/6 | 5/6 |
| Sonnet-5 | 5/6 | 6/6 |
| Gemini-3.8 | 5/6 | 6/6 |
| DeepSeek-V4.1 | 3/6 | 1/6 |
Agreement means the same negative, zero, or positive direction.
For the main experiment, Appendix H.1 reports SD of pressure minus neutral CQ and RES differences and task bootstrap intervals. Each operational efficiency bootstrap sample recomputes the pooled counts and ratios. Main experiment uncertainty and changes across rounds are therefore reported separately.
Aggregate completion quality is stable across rounds for GPT-5.6, Sonnet-5, Gemini-3.8, and DeepSeek-V4.1, with changes of 0.62 to 2.98 points and every interval including zero, while Qwen3.8 shifts by -23.48 points. Sonnet-5 and Gemini-3.8 also reproduce 5 of 6 CQ directions and all 6 RES directions. Our robustness claims therefore rest on the aggregate scores of these four models, and Section 5.3 and Appendix L report trajectory variation and effect directions separately.
Q4
We agree that score changes need to be explained through specific behaviors. Appendix F separates CQ and RES into their components. Averaged over the six single pressures, the component means give these contributions to the CQ change, in points.
| Model | TCC | FCR | PCR | Total |
|---|---|---|---|---|
| Gemini-3.8 | -0.30 | -1.35 | -0.63 | -2.28 |
| GPT-5.6 | -1.40 | -0.10 | +0.23 | -1.27 |
| Sonnet-5 | -1.98 | -1.77 | +1.08 | -2.67 |
| Qwen3.8 | -0.87 | -2.28 | +0.37 | -2.77 |
| DeepSeek-V4.1 | -2.69 | -1.05 | +0.26 | -3.48 |
Lost full completion drives the decline for Gemini-3.8 and Qwen3.8, lost end-state coverage drives it for GPT-5.6 and DeepSeek-V4.1, and Sonnet-5 loses both, while PCR rises for four models and offsets part of the loss. Unsafe successes fall for all five models, so RES declines through fewer unsafe actions while completion declines too.
The trajectory analysis connects these scores to care requirements. DeepSeek-V4.1 loses communication or ownership obligations in 20/31 eligible pairs in Appendix G. Under time pressure, Sonnet-5 loses required follow up in 5/15 pairs without unsafe execution in Table 5. Eligibility requires safely completed follow up at baseline, so this is a loss of an obligation the same model previously fulfilled.
Appendix K.2 provides a step by step pair in which the agent reads the records but repeatedly indexes, reopens, and rereads instead of drafting or publishing the reconciliation. The case stays open. Figure 7 further separates completion, process conformity, and unsafe execution. Together, the component scores and trajectories distinguish fewer unsafe actions from completed care.
Thank you for the detailed rebuttal and the additional experiments. The action-instruction control, the second evaluation round with uncertainty estimates, and the component and trajectory analyses address my questions. I will maintain my score.
Thank you for recognizing the matched design and the value of assessing unsafe execution and incomplete care together. We added 300 action instruction runs. This comment answers W1 and W2, with repeated evaluation and component analyses in a separate comment.
W1 & Q1
Thank you for asking how the pressures were constructed and validated before interpreting their effects. We first selected six categories from the healthcare research cited in Section 3.3, then wrote organizational contexts for the same medical task. Regulatory scrutiny describes audit attention, time pressure urgency, authority senior requests, peer norms team expectations, precedent earlier cases, and performance throughput or closure demands. These descriptions add workplace context without new medical evidence or authorization. The neutral baseline carries a routine case context of 37 words, and each pressure context has 37 or 38 words, so every pressure effect is measured against a matched-length context without pressure.
We checked that records, tools, action guards, and completion requirements remain fixed across conditions. Physicians reviewed the medical plausibility and care requirements of all 200 tasks and rated the six pressure contexts at 3.49 to 4.42 on a 1–5 realism scale. These checks validate task consistency across conditions. Section 3.3 now states the delivery, and Appendix D provides the actual texts, separating the category rationale from its implementation.
To distinguish context from direct guidance, we added a control on 10 tasks, two per function. Each instruction is derived from the action its corresponding pressure might encourage, then rewritten to retain the action requirement without the original pressure description or organizational demands. Paired word counts differ by at most two words. Medical evidence, authorization, tools, completion requirements, and delivery remain fixed. Five models each completed 60 additional runs, giving 300 pairs in Appendix M.
Cells give mean CQ / RES changes as instruction minus pressure over 10 task pairs. Positive CQ means higher completion quality, and positive RES higher recorded risk.
The trajectories also differ.
In 56 Sonnet-5 pairs both trajectories read all four records, yet eight change from no guarded tool call under pressure to a call under instructions, and two change in reverse. For Gemini-3.8, the corresponding changes are 14 and eight among 58 pairs. Record acquisition therefore does not account for every subsequent action difference.
In the Sonnet-5 dialysis pair, both read all four records by decision 5. Pressure leads to documenting gaps without adjustment. The instruction leads to drafting at decision 6 and adjustment at decision 8, crossing the configured boundary. Appendix M.3 provides the pair.
These score and trajectory differences show that, against a matched action instruction, pressure acts as organizational context on evidence checks and subsequent actions. Section 4 and Table 1 now report this comparison.
W2 & Q2
Thank you for asking us to clarify why these combinations were selected. Appendix D.3 explains the selection rationale. TF combines urgency with throughput or closure demands, RP combines formal oversight with team practice, and AM combines senior requests with prior case precedent. The conditions combining four pressures compare authority and performance under shared demands. Reversing R/F and T/P retains the members while testing presentation order. These are exploratory comparisons, not a sample of all combinations or clinical frequencies. Weakened and reversed effects are observations, not predefined directional hypotheses. Appendix A states the limits on coverage and formal interactions. The design keeps medical requirements and the appropriate endpoint fixed under combined demands. We do not assume that more pressure categories form a higher intensity level. Obs. 4 in Section 5.2 now shows how avoiding a premature finish can improve PCR while leaving care obligations incomplete. Weaker aggregate effects therefore require trajectory interpretation, not an assumption of psychological cancellation.