acceptodds
Under review as a conference paper at ICLR 2027

Refuse Without Abandoning: Circuit Coordination for Safe and Helpful Medical LLMs

Abstract

An ideal medical large language model should refuse harmful requests while preserving useful answers to legitimate needs. Yet existing models tend to achieve only one of these goals at a time. This paper systematically studies the safety-helpfulness imbalance in medical LLMs and identifies two recurring failure modes: models frequently discard information that could be safely provided when refusing harmful content, and safety warnings largely disappear when requests are rewritten as role-play or indirect prompts, with some rewritten requests receiving outright harmful responses. To quantify these behaviors, we propose HELP and CAUTION, two seven-dimensional rubrics that measure useful content and safety guidance respectively. Building on this analysis, we introduce MedCircuit, a method grounded in the observation that models already possess both behaviors but fail to combine them on sensitive inputs. MedCircuit extracts activation difference vectors corresponding to each behavior, constructs two fixed circuits, and trains a lightweight coordinator to jointly control their activation strengths, enabling the model to produce both helpful content and appropriate safety guidance during generation. Experiments across three medical LLMs and two benchmarks show that MedCircuit is the only defense that improves both safety and helpfulness simultaneously across all models, and maintains consistent advantages under role-play and indirect prompt rewriting attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Interpretability, medical large language model, safety, helpfulness, circuits

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XT7qOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the difficulty of balancing safe refusal and helpful responses in medical large language models. It proposes two fine-grained evaluation metrics, HELP and CAUTION, and designs MedCircuit. By identifying Attention Heads and FFN Neurons related to helpfulness and safety warnings, it constructs two behavioral circuits, and then uses a lightweight Coordinator to jointly adjust their activations. The method achieves simultaneous improvements in helpfulness and safety guidance ability on three medical LLMs.

Critical strengths
  1. The problem is clearly defined, and the medical application has practical significance.

  2. The experimental details are clearly described.

Critical weaknesses
  1. The novelty of the paper is very limited. The problem it focuses on is essentially the trade-off between safety and helpfulness when considering model safety. This is a very common and typical problem, and it is almost a necessary consideration for any model safety work. This paper merely applies this problem to medical LLMs. In essence, the paper focuses on the balance between over-refusal and safety.

  2. The method has no novelty. In essence, it balances the activation content of safe responses against that of normal responses. Such an approach is still balancing safety and helpfulness, rather than truly improving helpfulness while improving safety.

Questions for authors
  1. The HELP circuit and the CAUTION circuit may overlap. In addition, how the vector extraction accurately obtains the corresponding helpful and safe vector pairs is also a question. The paper does not clearly consider these issues. Moreover, whether the constructed paired data is too template-based is also a concern.

  2. Is the method general? Conceptually, it can be fully transferred to various non-medical models. If so, is the method effective on those models? Why must this approach be applied to medical models?

  3. Based on the results in Table 1, why do other safety methods such as SafetySFT show decreases in both helpfulness and safety compared with the base model? What is the reason? What is the purpose of these methods?

  4. The content of Fig. 4 is confusing, and it is hard to understand what the authors intend to convey. What do the boxes in Fig. 4(a) represent?

  5. The paper uses its own defined metrics for all evaluation and testing. The definitions of HELP and CAUTION have clear limitations, and the score improvements are not fully equivalent to improvements in the safety and effectiveness of medical responses. The paper is aware of this to some extent, but higher HELP and CAUTION scores should not be simply interpreted as the model being able to stably provide safe help.

  6. The gains of MedCircuit may mainly come from supervised learning on joint safety and helpfulness responses, rather than necessarily from first separating the two circuits and then coordinating them.

  7. The paper should pay attention to some organizational details. For example, there is an obvious issue that "AI USE STATEMENT" appears in two sections.

3 comments

Thank you for the comments.

W1 — Novelty of the problem

We disagree. A familiar safety problem is not necessarily trivial in medicine. First, detailed studies and targeted solutions remain scarce (L46–L48). Second, medical questions may look dangerous yet reflect legitimate needs (L34–L37). Refusal can therefore obstruct help, while medical jailbreaks can harm users’ health. Both sides of the balance are especially urgent here. We will clarify this distinction.

W2 — Novelty of the method

First, MedCircuit improves both safety and helpfulness, rather than sacrificing one for the other. Calling it “merely balancing” them misrepresents our results.

Circuit methods exist in general settings. However, their use in medical models remains rare. MedCircuit also requires domain adaptation: our medical evaluation criteria are essential to extracting the vectors and constructing the circuits. This is not a mechanical transfer.

More importantly, Tables 1–2 show greater efficiency and effectiveness than prompting or training-based defences. Circuit intervention is therefore both novel and useful in medicine.

Q1 — Overlap and vector extraction

First, the claim that we ignored overlap is incorrect. In Section 4.3, the coordinator is trained to allocate circuit strengths and handle interference between shared components.

Second, “vector pairs” is unclear. HELP and CAUTION vectors only serve their respective circuits. They are not paired, and Section 4.1 explains their extraction. If the concern is classification accuracy, we provide these separability results:

ComparisonBioMistral-7BMed42-8BMedGemma-4B
HELP+instruction vs instruction97%96%99%
CAUTION+instruction vs instruction100%100%97%
HELP+instruction vs CAUTION+instruction99%77%94%

The table gives linear regression probe accuracy on the sample pairs. High separability supports the extracted vectors.

Q2 — Is the method general?

We do not rule out other applications. However, applicability depends on the task. Jointly improving safety and helpfulness fits our approach, but this does not establish that it works in every domain. We focus on medicine and leave broader applications to future work.

Q3 — Why do other methods lower both scores?

HELP and CAUTION assess information that is actually provided. A refusal or generic template can therefore score poorly on both (Figure 9). Table 5 supports this: several methods have low Harm. and low SCR, meaning they provide neither harmful nor useful information. Their goal is to prevent harmful output; this does not necessarily mean providing helpful or cautionary content. We will clarify the distinction.

Q4 — Figure 4

The reviewer does not specify what is confusing. Section 5.3 explains the figure and how MedCircuit addresses the previously identified imbalance. We are also unsure what “boxes” refers to. Does the reviewer mean the bars in Figure 5?

Q5 — Metric limitations

First, the reviewer names no specific limitation or evidence. If the concern is coverage, our 14 criteria are already more comprehensive than binary or opaque LLM judgments in existing research.

Besides, scores reflect actual response content, checked against Table 3 by the judge. Higher scores mean more helpful or cautionary information. Denying improvement in those behaviours without identifying a flaw in these checks seems contradictory.

Q6 — Do the gains come from supervised learning?

We tested two controls variants: residual-stream steering without circuit selection, using the same training targets, and direct prompting. Below are HELP / CAUTION scores on 450 MedSafetyBench direct requests; steering uses a 50-question subset.

MethodBioMistral-7BMed42-8BMedGemma-4B
Undefended28.5 / 33.944.4 / 45.928.6 / 49.7
Residual-stream offset40.9 / 44.354.6 / 59.7 (n.s.)67.1 (n.s.) / 70.9 (n.s.)
Combined prompt (C.1)25.3 / 42.352.0 / 60.442.9 / 73.8
MedCircuit55.1 / 60.471.1 / 65.169.4 / 71.0

Unless marked n.s., differences from MedCircuit are significant.

Although both controls help, neither reproduces MedCircuit’s overall performance. Steering trails it by over 10 points on both metrics for BioMistral and on HELP for Med42. We attribute this to its uniform residual shift, which cannot adjust individual modules as MedCircuit does.

Prompting, meanwhile, varies across models, possibly because instruction-following ability differs. Optimisation may help, but gains remain uncertain and it lacks MedCircuit’s combination of interpretability and effectiveness. Searching for a stable optimum is outside our scope.

Q7 — Organisation

Thanks for raising this. We will remove the duplicated “AI Use Statement” and correct other typos in our revision.


If you find the response address major concerns, we would be grateful if you consider raising your score.

We thank the reviewer for the comments. We have uploaded the revised PDF, with all changes in blue.

  • W1 (novelty of the problem). The first paragraph of Section 1 now explains the medical consequences of both failures. Refusal can withhold help that a patient needs, while a successful jailbreak can directly harm health. Few defences address both risks.
  • W2, Q6 (balancing versus improving; do the improvements come from supervision?). Section 5.4 ("The source of the improvements") and Appendix F.1 (Table 10) now report both controls. The first is residual-stream steering with the same training targets but no circuit selection. The second is direct prompting with the combined instructions. Neither reproduces MedCircuit's overall performance.
  • Q1 (overlap and vector extraction). Section 4.2 now gives the circuit overlap and explains why the coordinator is trained on their union. Appendix F.2 (Table 11, Figure 10) reports the full overlap results. Appendix F.4 (Table 13) gives the probe accuracies for the activations used to extract the vectors.
  • Q2 (generality). Appendix A ("Limitations") now explains that applying the procedure beyond medicine requires criteria for that domain, since our rubrics and elicitation instructions are specific to medicine. We leave these applications to future work.
  • Q3 (why other defences lower both scores). Section 5.2 now explains that both rubrics score content present in the response. A bare refusal or generic template provides little of this content and therefore scores low on both, consistent with the low safe-completion rates in Table 5.
  • Q4 (Figure 4). The caption of Figure 4 now explains the colours for HELP and CAUTION and the horizontal axis. It also explains the solid and dashed lines in (a).
  • Q5 (metric limitations). Section 5.1 ("Evaluation criteria") now states that the rubrics measure response content without establishing clinical outcomes. Harmful assistance and safe completion are judged separately (Appendix E.1, Table 5).
  • Q7 (organisation). We removed the duplicated "AI use statement" heading and corrected the judge name and section reference in that statement.

As the discussion period is ending soon, we would be grateful if you could take a look at our responses and the revised paper. If they address your concerns, we would appreciate it if you could consider raising your score. Thank you very much.

Thank you for the author's response. I keep my score unchanged

Reviewer_2th5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper studies the safety–helpfulness imbalance in medical LLMs, where models may either over-refuse sensitive but legitimate requests or lose safety boundaries under rewritten harmful prompts.
  • It introduces HELP and CAUTION, two fine-grained evaluation rubrics, and proposes MedCircuit, which extracts behavior-specific circuits and learns a lightweight coordinator to jointly steer helpfulness and caution during generation.
  • Experiments on three medical LLMs show consistent improvements in both dimensions across over-refusal and harmful-request benchmarks, including role-play and indirect jailbreak rewrites.
Critical strengths
  • Important problem formulation. The paper targets a meaningful form of over-refusal in medical settings, where safe responses often require rejecting harmful details while still addressing legitimate underlying needs.

  • Fine-grained evaluation. HELP and CAUTION provide a more informative view than binary refusal or attack-success metrics, and help distinguish “safe but unhelpful” from “helpful but unsafe” responses.

  • Strong empirical results. MedCircuit consistently improves both HELP and CAUTION across three medical LLMs and multiple prompt conditions. The ablations also show that independently combining the two circuits is suboptimal, while the coordinator improves the balance between the two behaviors.

Critical weaknesses
  • The HELP representation may be overly simplified. HELP combines heterogeneous behaviors such as explanation, alternatives, specificity, actionable guidance, and reliable sourcing, yet the method compresses them into a single average activation direction. It is unclear whether this direction represents a coherent notion of helpfulness or mainly captures the effect of a compound instruction.
  • Limited evidence for true OOD generalization. Training questions already cover the same harm levels and prompt styles used at evaluation, including direct, role-play, and indirect prompts. The experiments therefore demonstrate held-out question generalization, but provide limited evidence that the extracted circuits generalize to unseen attack types, domains, languages, or interaction settings.
Questions for authors
  • What is the exact source of the training questions used for circuit extraction and coordinator training? Were they sampled from existing datasets, manually constructed, or LLM-generated? Clarifying their construction and distribution is important for assessing the claimed generalization.
  • How much do the extracted HELP and CAUTION circuits overlap? Please report the overlap ratio for both FFN neurons and attention heads, ideally by layer. This would help clarify whether HELP and CAUTION are represented by largely distinct circuits or by substantially shared components.
2 comments

We thank the reviewer for the thoughtful comments.

W1 — Is a single HELP direction overly simplified?

Our aim is to strengthen HELP and CAUTION overall, rather than model each criterion separately. Our experiments show that the individual criteria improve together, which supports this choice.

We also checked whether the direction depends on the wording of the instruction. We paraphrased the HELP instruction five times, extracted a direction from each version, and compared its top-4,096 neurons with the original:

Neuron overlapBioMistral-7BMed42-8BMedGemma-4B
Five paraphrases39–52%62–72%64–73%
Random<1%<1%<1%

The overlaps are well above chance, suggesting that the direction is not simply tied to one instruction’s wording.

We also tested whether one circuit per criterion would work better, using 14 circuits and a shared coordinator. We used the same pipeline and judge, and compared it with MedCircuit on 50 MedSafetyBench direct requests from our 200-question subset. Although the coordinator was trained on the target answers, many outputs became repetitive or garbled. The numbers of repetitive responses out of 50 were:

MethodBioMistral-7BMed42-8BMedGemma-4B
MedCircuit111
14 circuits + coordinator50407

The HELP / CAUTION scores also fell:

MethodBioMistral-7BMed42-8BMedGemma-4B
Undefended29.7 / 34.349.7 / 46.032.3 / 54.0
MedCircuit61.7 / 60.076.0 / 66.376.9 / 74.6
14 circuits + coordinator0.0 / 5.112.9 / 30.629.7 / 58.6

HELP fell below the base model on all three models. These results suggest that coordinating 14 circuits is harder and does not necessarily improve performance. This supports our choice of two broader circuits.

Finally, MedCircuit does more than reproduce the extraction prompts. Using those prompts directly as a system instruction does not reproduce its overall gains. On 450 MedSafetyBench direct requests, HELP / CAUTION are:

MethodBioMistral-7BMed42-8BMedGemma-4B
Combined prompt (C.1)25.3 / 42.352.0 / 60.442.9 / 73.8
MedCircuit55.1 / 60.471.1 / 65.169.4 / 71.0

W2 — OOD generalisation

The generalisation we claim is within medical QA. We train on 126 CARES-18K questions and evaluate on 1,655 questions from two different benchmarks, MedSafetyBench and Health-ORSC, without overlap. The gains hold across four harm levels, three prompt styles (direct, role-play and indirect), and three models. These results support our claim. Generalisation to unseen attack families, interaction settings, languages or non-medical domains is a broader question, which we acknowledge in the limitations and leave to future work.

Q1 — Where do the training questions come from?

All 126 questions were sampled from the public CARES-18K dataset. We did not write or generate new questions for training.

Paper · Dataset

Q2 — How much do the circuits overlap?

Each circuit contains 4,096 FFN neurons and 192 attention heads. We measure overlap as shared units divided by the union of both circuits (Jaccard). For comparison, we randomly select units while keeping each circuit’s per-layer counts unchanged. All observed overlaps are above chance (permutation p < 2e-4).

UnitsBioMistral-7BMed42-8BMedGemma-4B
Neurons1,570/6,622 = 0.242,093/6,099 = 0.342,455/5,737 = 0.43
Random neurons0.0060.0060.008
Heads126/258 = 0.49115/269 = 0.43172/212 = 0.81
Random heads0.120.120.67

To compare models with different depths (32, 32 and 34 layers), we group layers by relative depth. Each cell below shows neuron overlap / head overlap:

DepthBioMistral-7BMed42-8BMedGemma-4B
0–10%0.09/0.560.11/0.500.85/0.75
10–20%0.35/0.500.14/0.780.11/0.60
20–30%0.41/0.710.40/0.730.59/0.44
30–40%0.34/0.590.40/0.690.56/0.81
40–50%0.25/0.450.41/0.580.54/0.88
50–60%0.26/0.490.37/0.410.49/0.83
60–70%0.24/0.530.35/0.370.52/0.94
70–80%0.18/0.400.30/0.430.45/0.96
80–90%0.21/0.420.33/0.330.40/0.91
90–100%0.20/0.500.33/0.150.21/0.67

The circuits share many components, but they are not identical: 38%, 51% and 60% of each circuit’s neurons also belong to the other circuit. This helps explain why simply combining them can cause interference and why we train the coordinator on their union. MedGemma’s head overlap is less informative: each circuit selects 192 of only 272 heads, so even random selection gives 0.67 overlap. We will add the full per-layer results in the revision.


We would be grateful if you consider raising your score, if you find the response address major concerns.

We thank the reviewer for the thoughtful comments. We have uploaded the revised PDF, with all changes in blue.

  • W1 (is a single HELP direction too simple?). Appendix F.3 now includes both checks from our response.
    • Instruction wording. Appendix F.3 reports the neuron overlap between the original HELP instruction and five paraphrases. It is 39–52%, 62–72% and 64–73% on the three models, and less than 1% for random selection.
    • One circuit per rubric item. Table 12 compares 14 circuits with a shared coordinator against MedCircuit. Many outputs become repetitive, and HELP falls below the undefended model on all three models.
    • Section 5.4 and Appendix F.1 (Table 10) report the combined-prompt control. Direct use of the extraction prompts does not reproduce MedCircuit's overall improvements.
  • W2 (OOD generalisation). Appendix A ("Limitations") now states that our results concern held-out medical questions. Training uses CARES-18K questions, and evaluation covers two other benchmarks, four harm levels, three prompt styles and three models. We have not tested other attack families or interaction settings, other languages or non-medical domains.
  • Q1 (source of the training questions). Appendix C.5 now states that all 126 training questions were sampled from the public CARES-18K dataset. None were written or generated for training.
  • Q2 (overlap between the circuits). Section 4.2 now gives the share of each circuit's neurons that also belongs to the other circuit (38%, 51%, 60%). Appendix F.2 gives the full results. Table 11 compares neuron and head overlap with random selection, and Figure 10 shows overlap by relative depth to compare models with 32 and 34 layers.

As the discussion period is ending soon, we would be grateful if you could take a look at our responses and the revised paper. If they address your concerns, we would appreciate it if you could consider raising your score. Thank you very much.

Reviewer_QyjWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

MedCircuit aims to improve useful medical assistance while retaining safety guidance. It identifies model components linked to HELP and CAUTION, fits activation offsets, and learns a fixed joint correction over the selected components. The base model stays fixed. The authors test three medical language models on harmful requests, two types of jailbreak rewrites, and sensitive but answerable questions. They report higher HELP and CAUTION scores. A separate judge assesses harmful assistance and safe completion. Ablations test the circuits, coordinator, and intervention settings.

Critical strengths
  1. The paper addresses a useful problem: refusing harmful details should still leave room for safe assistance. The reported rubric gains over undefended models hold across three models and all main test conditions. Appendix E also reports lower harmful assistance and higher safe completion on harmful requests than the undefended models. This gives support beyond the new rubric scores.
  2. The coordinator ablation is informative. In Table 8, Med42 HELP rises from 49.71 without the coordinator to 67.00 with it. CAUTION changes from 65.71 to 61.71, so the result shows a trade-off. The duration study also documents repetition and truncation when intervention lasts too long.
Critical weaknesses
  1. The comparisons do not isolate the benefit of the selected circuits. MedCircuit learns from guided answers screened for HELP and CAUTION. SafetySFT uses different safe-response targets, while Guard and GoalPrompt emphasize refusal or safety. The module ablation shows that joint fitting helps, but does not separate circuit selection from the constructive supervision. A learned residual-stream offset using the same targets would be a useful control. Direct prompting with the combined HELP and CAUTION instructions would also test a simpler explanation for the gains.
  2. The reported results are not clearly linked to the supplied checkpoints. Section 5 of the supplement README says the checkpoints were refit after the stored expected and verification outputs were generated. Those records therefore do not verify the current checkpoints. Table 5 also needs its counts and averaging rule clarified. For example, 59.7% harmful assistance cannot be a rounded binary count out of 450 responses. A different denominator or weighting could explain this. These gaps limit confidence in the evidence for the stated method.
Questions for authors
  1. How does a learned residual-stream offset perform with the same training targets and tuning budget? What happens when the HELP and CAUTION instructions are combined directly?
  2. Which code version, objective, and checkpoint produced each main and ablation table? Can the authors provide the evaluated version or matching results for the supplied refit?
  3. What counts, denominators, and averaging rules produce Table 5? In particular, how is BioMistral's 59.7% harmful-assistance rate obtained for the 450 direct requests?
3 comments

We thank the reviewer for the careful reading and the concrete suggestions. Our responses follow.

W1 + Q1 — Do the improvements come from the selected circuits?

Why the baselines in Table 1 lean towards safety? Existing medical defences put safety first and often over-refuse; methods targeting safety and helpfulness together are rare (L13–L15). Table 1 asks whether MedCircuit improves helpfulness while increasing caution relative to these defences. The baselines therefore focus on safety: MedSafetyBench has its own safety training set, which SafetySFT uses directly. Table 2 and Figure 4 examine the balance on sensitive questions. We acknowledge that this motivation was not sufficiently clear and will clarify it in the revision.

The two requested experiments. We agree that baselines with other training targets cannot isolate the circuits, so we ran both controls the reviewer proposed, with the paper's models, questions, decoding and judge. The first keeps MedCircuit's training data, targets, layers and parameter budget but drops circuit selection: a learned offset on the whole residual stream. The second needs no training: the two App. C.1 instructions used to build the circuits, concatenated verbatim as the system prompt. HELP / CAUTION on the 450 MedSafetyBench direct requests, best per column in bold; the offset row is scored on a 50-question subset of them.

BioMistral-7BMed42-8BMedGemma-4B
Undefended28.5 / 33.944.4 / 45.928.6 / 49.7
Learned residual-stream offset40.9 / 44.354.6 / 59.7 (n.s.)67.1 (n.s.) / 70.9 (n.s.)
Combined prompt (C.1 )25.3 / 42.352.0 / 60.442.9 / 73.8
MedCircuit55.1 / 60.471.1 / 65.169.4 / 71.0

Except where marked (n.s.), every difference from MedCircuit is significant under the per-question paired bootstrap of Table 5.
Neither control reproduces MedCircuit’s overall performance. With the same supervision, the residual-stream offset trails MedCircuit by 14 / 16 points on BioMistral and 17 points of HELP on Med42, supporting an important contribution from the circuits. We attribute this to the offset’s uniform shift of the residual stream, whereas MedCircuit weights and adjusts individual modules. Direct prompting varies across models, potentially because of differences in instruction following; it lowers HELP on BioMistral and yields a much smaller HELP gain than MedCircuit on MedGemma. Further prompt optimization may help, but random variation cannot be excluded, and it lacks a comparable framework combining interpretability and effectiveness. Searching for a consistently optimal prompt is outside our scope. Overall, the controls support the contribution of circuit selection and coordination to MedCircuit’s joint gains.

W2 + Q2 — Checkpoints and reported results

We apologise for our version-management error, which led us to upload an earlier code package. An AI assistant identified the version mismatch but recorded it in the README instead of correcting it. We have replaced the supplement with the current package. The reported results were produced by the method described in the paper and can be verified with the updated package.

Q3 — Counts and averaging in Table 5

We thank the reviewer for raising the point. Table 5 is computed over the responses with a valid judge label. A label is valid only if it meets the scoring criteria: correct format, values in range, and evidence quoted from the response. Responses whose label does not meet these criteria are excluded from both numerator and denominator, due to the reliability issue. For BioMistral on the 450 direct requests, 6 responses were excluded, giving 59.7 % = 265 / 444. Exclusions per model:

excludedshare of all 24,825 responses
BioMistral-7B1860.7 %
Med42-8B3491.4 %
MedGemma-4B1,7867.2 %

The exclusions are concentrated on MedGemma, which writes much longer answers, and do not change the conclusions. Counting every excluded response against MedCircuit, its harmful rate stays below Undefended on BioMistral and Med42 under all three attack styles, and its safe-completion rate stays above Undefended in 7 of the 9 model–attack cells. The revision will state this rule, and the excluded responses are listed in the supplement.


Your feedback has helped us improve the paper, and we would be grateful if you would consider raising your score in light of our responses, clarifications, and additional results.

We thank the reviewer for the careful reading and concrete suggestions. We have uploaded the revised PDF, with all changes in blue.

  • W1, Q1 (do the improvements come from the selected circuits?).
    • Motivation for the baselines. We agree that this was not clear. Section 5.1 ("Baselines") now explains why the baselines follow existing medical defences, which put safety first. It also points to the controls that use MedCircuit's supervision.
    • The two requested controls. Section 5.4 ("The source of the improvements") now summarises both controls. Appendix F.1 and Table 10 give the setup and the results. The learned residual-stream offset uses the same training data, targets, layers and parameter budget, but no circuit selection. The combined prompt joins the two instructions of Appendix C.1. Table 10 marks differences that are not significant, including the MedGemma scores where the offset is comparable to MedCircuit.
  • W2, Q2 (checkpoints and reported results). The Reproducibility statement now states that the updated supplementary package contains the code version and checkpoints used for the reported results.
  • Q3 (counts and averaging in Table 5). We agree that the counting rule was missing. The new "Counting rule" paragraph in Appendix E.1 defines a valid judge label and explains that responses without one are excluded from both numerator and denominator. It gives the BioMistral example (6 of 450 excluded, 265/444 = 59.7%), the exclusions per model and the worst-case check. The supplementary material lists the excluded responses.

As the discussion period is ending soon, we would be grateful if you could take a look at our responses and the revised paper. If they address your concerns, we would appreciate it if you could consider raising your score. Thank you very much.

I would like to thank the authors for their detailed response and revision of the paper. I will take these comments into account during the discussion period amongst the reviewers and the area chair, but choose to keep my score as is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.