acceptodds
Under review as a conference paper at ICLR 2027

Following the Skill, Failing the Patient:Procedural Inertia in Clinical Agents

Abstract

Clinical agents are increasingly built to carry out procedures by following reusable skills, and curated skills raise average task success. A skill, however, fits the patient as they were when it was selected, and new evidence can make it wrong while the agent is still executing it. We call the tendency to keep following an invalidated skill procedural inertia. To measure it, we introduce SkillInertiaBench, a clinician-reviewed benchmark of 576 cases in six procedure families, in which each routine case is paired with three variants that differ only in the invalidating evidence and its timing: before execution, as a pre-existing fact revealed during execution, or as a change in the patient's state. Across two model families, standard skills raise routine completion from 67.7% to 80.6% but lower later-contradiction completion from 62.2% to 44.8% and raise harmful persistence from 31.2% to 49.8%; half of these harmful continuations follow the agent's own statement of the invalidating evidence. We then propose PatientGate, a patient-first execution policy in which an independent reviewer rechecks the active skill against the current patient state before each action and a gate enforces its verdict. PatientGate raises later-contradiction completion to 69.8% and lowers harmful persistence to 25.3% while keeping routine completion at 79.9%. These gains hold across both model families, two public benchmarks, and three clinical skill strategies. Our results make skill validity, not only skill quality, a first-class requirement for clinical agents, and show that it can be enforced at inference time without retraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agents, Agent Skills, Clinical Decision-Making, Agent Evaluation, Benchmarking, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_D8SDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether clinical agents keep following a skill after new patient evidence makes it inappropriate. The authors introduce SkillInertiaBench, a benchmark of 576 cases, and PatientGate, which checks whether the active skill still applies before an action is executed. Standard skills improve routine performance but can increase errors when exceptions arise. PatientGate reduces these errors through independent review and enforced action gating. The evaluation includes component ablations, two model families, and extensions of public benchmarks.

Critical strengths

(1) The paper addresses a meaningful problem. Clinical agents need to know when to stop, as well as how to follow a procedure. The benchmark makes continued execution after skill invalidation directly measurable. This adds a useful perspective beyond routine task success. (2) The evaluation includes several prompt-based controls and component ablations. Removing enforcement lowers completion on later-contradiction cases from 69.8% to 60.4%. This supports a distinct benefit from enforcing the review outcome. Extra review information alone does not explain the full gain.

Critical weaknesses

(1) Some interpretations go beyond the evidence. The paper counts mentioning an abnormal finding as “recognition.” That does not establish that the model understands why the finding invalidates the active skill. The claim that agents recognize the problem but proceed anyway therefore needs stronger support. The variants also change both the evidence and its timing. This makes it difficult to isolate the effect of having already started the procedure. These issues do not negate the observed failures, but they weaken the proposed explanation. (2) The added value of the method and the benchmark’s ability to distinguish appropriate stopping from unnecessary stopping remain unclear. The ablations show that hard gating helps. They do not establish an advantage over step-by-step action-safety checks with the same information and compute budget. The paper also gives limited results for benign late updates—cases where new patient information arrives but the skill remains valid. Such controls would help show whether PatientGate detects invalidation accurately. Conservative stopping could also explain part of the gain. This is the main reason I am not ready to support acceptance.

Questions for authors

(1) Does the benchmark include later updates that leave the active skill valid? If so, what are PatientGate’s completion and false-block rates on these cases? (2) How many failure traces explicitly state that the active skill no longer applies? Please distinguish these from traces that only mention the abnormal finding and report the two groups separately. (3) Could you compare PatientGate with a direct action-safety gate under matched information and compute budgets? Please also provide the prompts, inference budgets, and baseline adaptation details. I see promise in this work. The key is to show that the gains come from accurate applicability judgments and to clarify what PatientGate adds over existing gating methods. If the additional evidence addresses these concerns, I would reconsider my assessment and consider raising my score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer D8SD

Thank you for recognizing the importance of this problem and for your specific suggestions. We respond to each question below.

Q1: Does the benchmark include later updates that leave the active skill valid? What are the completion and false-block rates in these cases?

Thank you for this suggestion. We constructed benign late-update variants for the 144 base scenarios. Following the matched design in Section 2 and the appendix, we introduced new patient information during execution while keeping the active skill valid. This setting tests whether PatientGate blocks actions unnecessarily when new information arrives. In this experiment, the per-run success rates for No Skill, Standard Skill, and PatientGate were 67.4%, 79.9%, and 78.5%, respectively. PatientGate’s false-block rate was 5.32%. Its success rate was 1.4 percentage points lower than that of Standard Skill, but 11.1 points higher than that of No Skill. These results show that PatientGate preserves most of the task benefit provided by skills when new information does not invalidate them. We will add the case construction details, evaluation settings, and calculation of the false-block rate to the appendix.

Q2: How many failure traces explicitly state that the active skill no longer applies? Please report these separately from traces that only mention an abnormal finding.

Thank you for highlighting this distinction. Under the original definition, an R–A run is a harmful-persistence run in which the agent states the invalidating evidence before taking the irreversible action. The metric does not require an explicit judgment that the active skill no longer applies. Following your suggestion, we re-examined the R–A traces under Later Contradiction and distinguished traces that only mention abnormal findings from those that include an explicit judgment that the active skill no longer applies: Standard Skill: Of 215 traces, 114 only mentioned abnormal findings, while 101 explicitly stated that the active skill no longer applied. PatientGate: Of 74 traces, 63 only mentioned abnormal findings, while 11 included an explicit judgment that the active skill no longer applied. Independent Review: Of 87 traces, 62 only mentioned abnormal findings, while 25 included an explicit judgment that the active skill no longer applied. For PatientGate, the explicit-judgment category includes the independent reviewer’s verdict. We will make this distinction clear in the revised paper. We will also distinguish between stating invalidating evidence and explicitly judging that the evidence makes the active skill inapplicable. In the original experiments, PatientGate reduced HP from 49.8% to 25.3% and R–A from 24.9% to 8.6%. These results show that PatientGate reduces harmful persistence, including cases in which the agent states the invalidating evidence before continuing with the procedure.

Q3: Can you compare PatientGate with a direct action-safety gate under matched information and compute budgets? Please also provide the prompts, inference budgets, and baseline adaptation details.

Thank you for this suggestion. We added a direct action-safety gate baseline and evaluated it with DeepSeek-V4.1-Flash and Qwen3.5-9B. Before each action is executed, the baseline conducts an independent safety review. It uses the same patient-state summary, skill applicability and invalidation conditions, and proposed action as PatientGate. Both methods use the same base models and matched model-call and token budgets. In both methods, the gate enforces the review outcome before an action is executed. The new baseline achieved mean pass³ scores of 79.7%, 69.3%, 63.4%, and 63.2% on Routine, Initial Exception, Later Contradiction, and Temporal Change, respectively. PatientGate’s corresponding scores were 79.9%, 72.6%, 69.8%, and 68.1%. The methods performed similarly on Routine cases. PatientGate scored 3.3, 6.4, and 4.9 percentage points higher on the three skill-invalidation variants. Both methods use independent review, a patient-state summary, and an enforced gate. The direct action-safety gate judges whether the proposed action is safe. PatientGate judges whether the active skill remains applicable and uses that judgment to restrict subsequent actions. When the skill is judged to be invalid, PatientGate revokes it and triggers replanning. These results provide further support for the overall design, with gains of 6.4 and 4.9 percentage points on Later Contradiction and Temporal Change, respectively. We will include the prompt templates, inference-budget settings, and baseline adaptation details in the appendix of the revised paper.

Thank you for the additional experiments and clarifications. These alleviate my main concerns, and I am raising my score. My main remaining reservation is PatientGate’s reliance on manually specified invalidation conditions and action categories, which leaves its effectiveness in less structured clinical settings uncertain.

AuthorsOpenReview ↗

Response to Reviewer D8SD

Thank you for your recognition and for raising your score. We acknowledge this remaining limitation and will work toward evaluating PatientGate in real clinical settings, including those with less structured information.

Reviewer_zkLaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies procedural inertia when a skill-following clinical agent continuing to execute a skill after patient evidence has invalidated it. It develops SkillInertiaBench, 576 author-written, clinician-reviewed cases in six procedure families. On two models, a standard skill improves routine pass@3 from but lowers later-contradiction pass@3, and half of the harmful continuations occur after the agent has itself stated the invalidating evidence. The paper then proposes PatientGate, an inference-time policy in which a reviewer in a separate context judges whether the active skill is still supported by an explicit patient-state summary before each action. It enhances later-contradiction pass@3 while keeping routine performance.

Critical strengths
  • The studied problem is interesting and practical: whether including a skill is always helpful, and the potential harm when it is not used properly.
  • The ablation studies are thoroughly conducted to understand the contribution of adding each component.
Critical weaknesses
  • The case curation process is unclear. The paper says the authors wrote the base scenarios and two clinicians reviewed them independently, but it does not say whether the drafters are clinicians, whether the reviewers are independent of the author team, who adjudicated disagreements, how many there were, or the inter-rater agreement. The clinicians’ specialties and experience are not described.

  • All cases are synthetic vignettes with no real patient data, and it is unclear how well they reflect real clinical scenarios and evolving disease conditions. The invalidation types listed in Table 3 only covers a limited of scenarios.

  • The Standard Skill baseline is weak. In practice, a high-quality clinical skill is more comprehensive than a single rule-based description. The skill used here omits its own contraindications, which is bound to mislead the model’s decisions, and much of the reported harm appears to come from this. Also, in Table 2 the standard skill is enriched with only one additional feature at a time (generic caution, full guideline, or exception notes), and no combination of two or all three is tested, which would be the natural stronger baseline against which to measure PatientGate’s gain.

  • Only two small-scale models are evaluated (DeepSeek-V4.1-Flash, Qwen3.5-9B), described as "two model families". It's unclear whether frontier reasoning models show the same inertia, and whether the gate still helps when the reviewer is a stronger model.

Questions for authors

see comments above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer zkLa

Thank you for these thoughtful questions. We appreciate your specific suggestions and respond to each point below.

Q1.

The case curation process is unclear. The paper says the authors wrote the base scenarios and two clinicians reviewed them independently, but it does not say whether the drafters are clinicians, whether the reviewers are independent of the author team, who adjudicated disagreements, how many there were, or the inter-rater agreement. The clinicians’ specialties and experience are not described.

Scenario development and clinical review

The authors of this paper wrote the base scenarios. Our team conducts research on clinical agents and collaborates with hospitals, but the case authors are not licensed physicians. We therefore separated scenario development from clinical review.

The authors constructed the task workflows and matched variants. Two clinicians outside the author team independently reviewed the skill applicability conditions, invalidating evidence, and acceptable terminal states.

Case selection and limitations

We retained only cases approved by both clinical reviewers. Cases with disagreements were excluded from the final test set. This criterion was intended to reduce the influence of clinically ambiguous cases on the evaluation.

Human review can still involve subjective judgments and annotation noise. Clinical review supports the plausibility of the cases but does not eliminate these sources of uncertainty. We will state this limitation explicitly in the revised manuscript.

Q2.

All cases are synthetic vignettes with no real patient data, and it is unclear how well they reflect real clinical scenarios and evolving disease conditions. The invalidation types listed in Table 3 only covers a limited of scenarios.

Matched case design

SkillInertiaBench addresses a specific question under controlled conditions: does an agent continue following a previously useful skill after new evidence makes it inapplicable?

We constructed four matched variants for each of 144 base scenarios. These variants share the task goal, tool interface, and patient information unrelated to skill invalidation. We vary the evidence relevant to skill applicability and the time at which it becomes available.

Synthetic cases make this controlled comparison possible. They also avoid the data-access, privacy, and safety concerns associated with using real patient records. We therefore focused our resources on the matched case design and clinical review to characterize this failure mode and evaluate a potential intervention.

The six procedure families provide several distinct settings for this analysis. They do not exhaustively cover clinical workflows or patterns of disease progression.

Public benchmark extensions

Section 5 and Appendix D.4 also extend the evaluation to 60 MedAgentBench tasks and 40 AgentClinic tasks. We retain each original task as the routine version and construct a clinician-reviewed matched contradiction version for each task.

On these contradiction extensions, PatientGate achieves per-run success rates of 63.9% and 59.2%, respectively, compared with 43.3% and 39.2% for Standard Skill. These results provide additional evidence beyond our own benchmark cases.

The contradiction updates in these public benchmark extensions are also manually constructed. The results therefore remain evidence from simulated settings. We will clarify this scope in the revised manuscript. They should not be interpreted as estimates of real-world failure rates or effectiveness in clinical deployment.

AuthorsOpenReview ↗

Response to Reviewer zkLa

Q3.

The Standard Skill baseline is weak. In practice, a high-quality clinical skill is more comprehensive than a single rule-based description. The skill used here omits its own contraindications, which is bound to mislead the model’s decisions, and much of the reported harm appears to come from this. Also, in Table 2 the standard skill is enriched with only one additional feature at a time (generic caution, full guideline, or exception notes), and no combination of two or all three is tested, which would be the natural stronger baseline against which to measure PatientGate’s gain.

Information provided by Standard Skill

Standard Skill includes the procedure and its applicability conditions. It omits the additional, explicit list of invalidating conditions I recorded in the case specification. This setting provides a reference for measuring how reusable skills affect agent behavior in routine cases and after invalidation.

Controls with additional information

To examine the role of missing invalidation information, we included Exception Notes, which explicitly provides the invalidating conditions to the executor. Full Guideline also supplies the complete clinical guideline from which the skill was derived.

As shown in Table 2, adding Exception Notes raises completion (pass³) on later-contradiction cases from 44.8% to 60.1% and reduces harmful persistence (HP) from 49.8% to 33.8%. Explicit invalidation information clearly helps.

PatientGate nevertheless reaches 69.8% completion and 25.3% HP under the same case setting. Thus, the absence of explicit invalidation conditions in Standard Skill alone does not explain the remaining gap.

Section 5 and Table 12 also compare PatientGate with CodeClinic, GRASP, and MediSkill-Evo. Each comparison uses the same subset of cases representable by the corresponding method. These experiments provide additional comparisons beyond the basic skill prompt.

Combined prompts

Regarding combined prompts, we evaluated Generic Caution, Full Guideline, and Exception Notes separately to assess the effect of each addition. We agree that these individual controls do not replace an evaluation of their combinations.

We have not tested combined prompts, so the current results do not establish an advantage over the strongest combination of these additions. We will clarify this limitation in the revised manuscript.

Q4.

Only two small-scale models are evaluated (DeepSeek-V4.1-Flash, Qwen3.5-9B), described as "two model families". It's unclear whether frontier reasoning models show the same inertia, and whether the gate still helps when the reviewer is a stronger model.

Model coverage

The original evaluation used DeepSeek-V4.1-Flash and Qwen3.5-9B. These models allowed repeated evaluations across two model families, but they do not provide broad coverage of model sizes or reasoning capabilities.

Additional evaluation

We extend the evaluation to GPT-6.1 sol and Qwen3.5-27B using No Skill, Standard Skill, and PatientGate in the appendix. For each model, we will report pass³ across all four case variants, together with HP and R–A on later-contradiction cases.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer zkLa:

Thank you again for your time and feedback. We would like to follow up on our rebuttal.

As the deadline for open discussion is soon,if our responses and additional experiments have addressed your concerns, we would be grateful if you could consider raising your score. If you have any remaining questions, please let us know. We would be happy to discuss them further.

Thanks again for your time.

Best,

Authors

Reviewer_cytYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a limitation of reusable clinical skills: a procedure that initially fits a patient may become inappropriate after additional evidence arrives, yet an agent may continue executing it. The authors introduce procedural inertia and construct SkillInertiaBench, comprising 144 scenarios with four variants each, to evaluate routine execution, initial exceptions, later contradictions, and temporal changes. They propose PatientGate, which combines a current-state summary, an independent applicability reviewer, and an enforced action gate. Experiments with two model families show that standard skills improve routine completion but worsen performance under invalidation, while PatientGate reduces harmful persistence. Ablations and author-constructed extensions of public benchmarks provide additional validation.

Critical strengths
  1. The motivation identifies an important distinction between following a procedure correctly and determining whether it still applies. The problem is not merely failure to obey a safety instruction: the same reusable skill can help under routine conditions and hurt after invalidation.

  2. The intervention is aligned with the observed failure, and the ablations test its central design choice.

Critical weaknesses
  1. Limited architectural isolation. Without specification-matched runtime guards, the results establish the value of enforcement more clearly than that of skill-applicability review.

  2. Dependence on structured annotations. PatientGate uses explicit invalidation conditions and manually specified action categories; obtaining these outside the benchmark remains unclear.

  3. Timing is not independently controlled. Variants change both the content and timing of invalidating evidence, limiting attribution to timing-dependent inertia.

  4. Recognition does not establish understanding. Stating an invalidating fact does not necessarily mean understanding why the proposed action is inappropriate.

  5. Unreported inference overhead. Equal action budgets do not control for reviewer calls, token usage, or latency.

  6. Overstated oracle interpretation. Oracle Routing bounds the tested skill-relative gate, not what any applicability check could achieve.

Questions for authors
  1. How are invalidation conditions and action categories obtained outside the authored benchmark? Are the reviewer’s conditions generic skill metadata or case-specific annotations? Please clarify precisely which of these fields are available to each baseline.

  2. Could the authors compare PatientGate with a deterministic guard where conditions are machine-checkable, and with a direct per-action safety reviewer using the same patient summary, invalidation rules, and call budget? This would establish whether skill-relative applicability checking adds value beyond structured action shielding.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cytY

Thank you for recognizing the importance of the problem and for your thoughtful questions. We appreciate your suggestions and respond to each point below.

Q1.

How are invalidation conditions and action categories obtained outside the authored benchmark? Are the reviewer’s conditions generic skill metadata or case-specific annotations? Please clarify precisely which of these fields are available to each baseline.

Source of the annotations

Thank you for this question. In SkillInertiaBench, the authors explicitly wrote the invalidation conditions and action categories during case construction. Two senior clinicians independently reviewed them. The reviewer uses structured conditions specified for the corresponding skill in each case specification. These conditions are not automatically retrieved from a general skill library.

Appendices A.3–A.4 describe the case specifications and clinical review process. Two clinicians independently checked the skill’s applicability conditions, invalidating evidence, and acceptable terminal states. The cases were finalized before model evaluation.

Scope of the current setup

Our goal is to measure continued execution after skill invalidation under controlled conditions and to test an intervention at the point of action. We therefore hold these reviewed conditions fixed to reduce confounding between errors in obtaining the rules and errors in execution.

We agree that this setup makes PatientGate dependent on structured information supplied in advance. Our experiments do not establish whether this information can be obtained automatically in open clinical environments. We leave this question to future work.

Beyond the benchmark, one possible approach is to derive applicability and invalidation conditions from reviewed clinical guidelines and local operating protocols. Action categories could then be defined based on what the tool interfaces actually do, followed by clinical review. This approach still requires validation, and we will make this scope limitation clear.

Information available to each method

Section 2.1, Appendix C, and Table 5 specify the information available to each method. Irrelevant Context adds only an unrelated passage of the same length as the skill.

AuthorsOpenReview ↗

Response to Reviewer cytY

Q2.

Could the authors compare PatientGate with a deterministic guard where conditions are machine-checkable, and with a direct per-action safety reviewer using the same patient summary, invalidation rules, and call budget? This would establish whether skill-relative applicability checking adds value beyond structured action shielding.

Thank you for this suggestion. We agree that additional comparisons would help assess the value of skill-applicability review beyond structured action shielding.

Existing ablation

In our existing ablation, making the gate advisory reduces later-contradiction completion from 69.8% to 60.4% and increases HP from 25.3% to 36.2%. These results support the importance of enforcing the reviewer’s verdict. On their own, however, they do not establish that reviewing skill applicability is better than other safety checks before execution.

Matched action-safety baseline

We have therefore added a direct action-safety gate baseline and evaluated it on DeepSeek-V4.1-Flash and Qwen3.5-9B. This baseline conducts an independent safety review before each action is executed. It receives the same patient-state summary, skill applicability and invalidation conditions, and proposed action as PatientGate.

Both methods enforce the review outcome through a gate. They use the same base models, separate contexts for review, and matched model-call and token budgets.

Table R1. Mean pass³ scores on the four case variants

MethodRoutineInitial ExceptionLater ContradictionTemporal Change
Direct action-safety gate79.7%69.3%63.4%63.2%
PatientGate79.9%72.6%69.8%68.1%

The two methods perform similarly on routine cases. On the three skill-invalidation variants, PatientGate scores 3.3, 6.4, and 4.9 percentage points higher, respectively.

Difference between the two gating designs

This comparison keeps independent review, the patient-state summary, and enforced gating in both methods. It varies the design of the check: direct action safety versus skill applicability.

The main distinction is that the direct safety gate judges whether the proposed action is safe. PatientGate judges whether the active skill still applies and restricts subsequent actions accordingly. When the skill is judged invalid, PatientGate revokes it and triggers replanning.

The new results provide further support for this overall design, with gains of 6.4 and 4.9 percentage points on Later Contradiction and Temporal Change, respectively.

Interpretation of Oracle Routing

We also agree with your comment on the interpretation of Oracle Routing. It should be described as an oracle reference that replaces the reviewer’s prediction with the ground-truth applicability label while keeping the executor, action categories, and gate structure fixed.

Its remaining errors do not establish a performance ceiling for all applicability-checking methods. We will narrow the corresponding claims in the revised paper.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer cytY:

Thank you again for your time and feedback. We would like to follow up on our rebuttal.

As the deadline for open discussion is soon,if our responses and additional experiments have addressed your concerns, we would be grateful if you could consider raising your score. If you have any remaining questions, please let us know. We would be happy to discuss them further.

Thanks again for your time.

Best,

Authors

Reviewer_cytYOpenReview ↗

response to rebuttal

Thank you very much for your carefully written responses. The rebuttal meaningfully improves the empirical comparison, particularly through the matched action-safety baseline. However, my remaining concern is the incremental nature of the contribution. PatientGate combines applicability monitoring, enforced gating, and replanning, and its distinction from established execution-monitoring approaches remains insufficiently articulated. Moreover, the evaluation assumes manually authored and clinically reviewed invalidation conditions and action categories, leaving robustness to imperfect specifications untested. While the failure mode is practically relevant, I am not yet convinced that the current evidence establishes sufficient methodological novelty or a sufficiently distinct characterization of procedural inertia.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to Reviewers

Dear Area Chair and Reviewers,

Thank you for your careful reviews and constructive feedback. Reviewer cytY recognized the motivation for distinguishing between following a procedure correctly and determining whether it still applies, and the connection between our method and the observed failure mode. Reviewer zkLa found the problem interesting and practically relevant and noted the thorough component ablations. Reviewer D8SD highlighted the importance of knowing when clinical agents should stop, and the benchmark’s direct measurement of continued execution after skill invalidation. We also thank Reviewer D8SD for confirming that the additional experiments and clarifications alleviated their main concerns and for raising their score. Below, we summarize our core contributions and the main revisions.

Core Contributions

  • Procedural inertia. We identify and systematically analyze a safety-critical failure of skill-following clinical agents: skills that help on routine cases hurt once the patient changes, and most of the harm they add follows the agent’s own statement of the invalidating evidence.

  • A timing-controlled benchmark. We introduce SkillInertiaBench, 576 clinician-reviewed cases in six procedure families whose matched variants control when and how a skill becomes invalid.

  • Patient-first execution. We propose PatientGate, a training-free policy that lifts completion on later contradictions from 44.8% to 69.8% while keeping the routine benefit of skills, with consistent gains across model families, public benchmarks, and clinical skill strategies.

Main Revisions

  • Direct action-safety gate with matched information and compute budgets (Appendices D.2 and E.1, Table 7). In response to Reviewers cytY and D8SD, we added a baseline that independently reviews safety before each action. Both methods receive the same patient-state summary and skill applicability and invalidation conditions. They use the same base models, matched model-call and token budgets, and enforced review outcomes. Performance is similar on Routine cases. On Initial Exception, Later Contradiction, and Temporal Change, PatientGate achieves mean pass³ scores 3.3, 6.4, and 4.9 percentage points higher, respectively. These results add a comparison with direct action-safety checks and support PatientGate’s overall execution policy.

  • Benign late-update control (Appendix E.5, Table 11). To address Reviewer D8SD concern about false blocks, we constructed variants of the 144 base scenarios in which new information arrives but the active skill remains valid. PatientGate achieves a per-run success rate of 78.5%, close to Standard Skill’s 79.9% and above No Skill’s 67.4%. Its false-block rate is 5.32%. This control directly tests cases in which execution should continue and extends the evaluation beyond cases of skill invalidation.

  • Failure-trace analysis and clarification of R–A (Appendix E.3, Table 9). Following Reviewer D8SD request, and addressing Reviewer cytY concern that recognition does not imply understanding, we separately report traces that only mention abnormal findings and those that explicitly judge the skill to be invalid. Of the 215 Standard Skill R–A traces, 114 only mention abnormal findings, while 101 explicitly state that the skill no longer applies. The revised manuscript also states that PatientGate’s explicit-judgment category includes the reviewer’s verdict. These counts should therefore not be directly interpreted as differences in the executor’s understanding across methods.

  • Broader model coverage (Appendix E.2, Table 8). In response to Reviewer zkLa, we extended the comparison of No Skill, Standard Skill, and PatientGate to GPT-6.1 sol and Qwen3.5-27B, in addition to DeepSeek-V4.1-Flash and Qwen3.5-9B. We report pass³ across all four case variants to examine skill benefits and failures across a broader set of models.

  • Clarification of clinical review and structured annotations (Appendices A.3–A.4 and D.1). In response to Reviewer zkLa, we clarified that the authors wrote the scenarios and two clinicians outside the author team independently reviewed them. Only cases approved by both clinicians were retained. In response to Reviewer cytY, we clarified that the invalidation conditions and action categories are manually specified, clinically reviewed case annotations. We also detailed the information available to each baseline.

We hope these additional experiments and clarifications address your main concerns. We would appreciate your reassessment of the paper in light of the revisions and ask you to consider updating your scores if your concerns have been resolved. We will continue to pursue validation in settings closer to clinical practice and welcome further discussion of any remaining questions.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.