- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether clinical agents keep following a skill after new patient evidence makes it inappropriate. The authors introduce SkillInertiaBench, a benchmark of 576 cases, and PatientGate, which checks whether the active skill still applies before an action is executed. Standard skills improve routine performance but can increase errors when exceptions arise. PatientGate reduces these errors through independent review and enforced action gating. The evaluation includes component ablations, two model families, and extensions of public benchmarks.
Critical strengths
(1) The paper addresses a meaningful problem. Clinical agents need to know when to stop, as well as how to follow a procedure. The benchmark makes continued execution after skill invalidation directly measurable. This adds a useful perspective beyond routine task success. (2) The evaluation includes several prompt-based controls and component ablations. Removing enforcement lowers completion on later-contradiction cases from 69.8% to 60.4%. This supports a distinct benefit from enforcing the review outcome. Extra review information alone does not explain the full gain.
Critical weaknesses
(1) Some interpretations go beyond the evidence. The paper counts mentioning an abnormal finding as “recognition.” That does not establish that the model understands why the finding invalidates the active skill. The claim that agents recognize the problem but proceed anyway therefore needs stronger support. The variants also change both the evidence and its timing. This makes it difficult to isolate the effect of having already started the procedure. These issues do not negate the observed failures, but they weaken the proposed explanation. (2) The added value of the method and the benchmark’s ability to distinguish appropriate stopping from unnecessary stopping remain unclear. The ablations show that hard gating helps. They do not establish an advantage over step-by-step action-safety checks with the same information and compute budget. The paper also gives limited results for benign late updates—cases where new patient information arrives but the skill remains valid. Such controls would help show whether PatientGate detects invalidation accurately. Conservative stopping could also explain part of the gain. This is the main reason I am not ready to support acceptance.
Questions for authors
(1) Does the benchmark include later updates that leave the active skill valid? If so, what are PatientGate’s completion and false-block rates on these cases? (2) How many failure traces explicitly state that the active skill no longer applies? Please distinguish these from traces that only mention the abnormal finding and report the two groups separately. (3) Could you compare PatientGate with a direct action-safety gate under matched information and compute budgets? Please also provide the prompts, inference budgets, and baseline adaptation details. I see promise in this work. The key is to show that the gains come from accurate applicability judgments and to clarify what PatientGate adds over existing gating methods. If the additional evidence addresses these concerns, I would reconsider my assessment and consider raising my score.
3 comments
Thank you for the additional experiments and clarifications. These alleviate my main concerns, and I am raising my score. My main remaining reservation is PatientGate’s reliance on manually specified invalidation conditions and action categories, which leaves its effectiveness in less structured clinical settings uncertain.
Response to Reviewer D8SD
Thank you for your recognition and for raising your score. We acknowledge this remaining limitation and will work toward evaluating PatientGate in real clinical settings, including those with less structured information.
Response to Reviewer D8SD
Thank you for recognizing the importance of this problem and for your specific suggestions. We respond to each question below.
Q1: Does the benchmark include later updates that leave the active skill valid? What are the completion and false-block rates in these cases?
Thank you for this suggestion. We constructed benign late-update variants for the 144 base scenarios. Following the matched design in Section 2 and the appendix, we introduced new patient information during execution while keeping the active skill valid. This setting tests whether PatientGate blocks actions unnecessarily when new information arrives. In this experiment, the per-run success rates for No Skill, Standard Skill, and PatientGate were 67.4%, 79.9%, and 78.5%, respectively. PatientGate’s false-block rate was 5.32%. Its success rate was 1.4 percentage points lower than that of Standard Skill, but 11.1 points higher than that of No Skill. These results show that PatientGate preserves most of the task benefit provided by skills when new information does not invalidate them. We will add the case construction details, evaluation settings, and calculation of the false-block rate to the appendix.
Q2: How many failure traces explicitly state that the active skill no longer applies? Please report these separately from traces that only mention an abnormal finding.
Thank you for highlighting this distinction. Under the original definition, an R–A run is a harmful-persistence run in which the agent states the invalidating evidence before taking the irreversible action. The metric does not require an explicit judgment that the active skill no longer applies. Following your suggestion, we re-examined the R–A traces under Later Contradiction and distinguished traces that only mention abnormal findings from those that include an explicit judgment that the active skill no longer applies: Standard Skill: Of 215 traces, 114 only mentioned abnormal findings, while 101 explicitly stated that the active skill no longer applied. PatientGate: Of 74 traces, 63 only mentioned abnormal findings, while 11 included an explicit judgment that the active skill no longer applied. Independent Review: Of 87 traces, 62 only mentioned abnormal findings, while 25 included an explicit judgment that the active skill no longer applied. For PatientGate, the explicit-judgment category includes the independent reviewer’s verdict. We will make this distinction clear in the revised paper. We will also distinguish between stating invalidating evidence and explicitly judging that the evidence makes the active skill inapplicable. In the original experiments, PatientGate reduced HP from 49.8% to 25.3% and R–A from 24.9% to 8.6%. These results show that PatientGate reduces harmful persistence, including cases in which the agent states the invalidating evidence before continuing with the procedure.
Q3: Can you compare PatientGate with a direct action-safety gate under matched information and compute budgets? Please also provide the prompts, inference budgets, and baseline adaptation details.
Thank you for this suggestion. We added a direct action-safety gate baseline and evaluated it with DeepSeek-V4.1-Flash and Qwen3.5-9B. Before each action is executed, the baseline conducts an independent safety review. It uses the same patient-state summary, skill applicability and invalidation conditions, and proposed action as PatientGate. Both methods use the same base models and matched model-call and token budgets. In both methods, the gate enforces the review outcome before an action is executed. The new baseline achieved mean pass³ scores of 79.7%, 69.3%, 63.4%, and 63.2% on Routine, Initial Exception, Later Contradiction, and Temporal Change, respectively. PatientGate’s corresponding scores were 79.9%, 72.6%, 69.8%, and 68.1%. The methods performed similarly on Routine cases. PatientGate scored 3.3, 6.4, and 4.9 percentage points higher on the three skill-invalidation variants. Both methods use independent review, a patient-state summary, and an enforced gate. The direct action-safety gate judges whether the proposed action is safe. PatientGate judges whether the active skill remains applicable and uses that judgment to restrict subsequent actions. When the skill is judged to be invalid, PatientGate revokes it and triggers replanning. These results provide further support for the overall design, with gains of 6.4 and 4.9 percentage points on Later Contradiction and Temporal Change, respectively. We will include the prompt templates, inference-budget settings, and baseline adaptation details in the appendix of the revised paper.