acceptodds
Under review as a conference paper at ICLR 2027

Criteria as Code: When Written Rules Beat Learned Ones, and When They Don't

Abstract

Written clinical criteria offer an explicit alternative to learning every diagnostic decision from annotated images. But when does protocol-derived decision logic transfer better than logic learned from data? We study this question with CriteriaCode, which separates multiple sclerosis (MS) lesion segmentation into learned candidate detection, 34 radiological attributes, and an executable criteria program with calibrated thresholds. The main comparison uses a hand-written program and a shallow decision tree trained on the identical clinically informed attributes. Both expose their decision logic, while its source differs: the annotation protocol or the training labels. In-domain, the tree exceeds the program from 32 annotated patients, and the end-to-end network remains more accurate. Under transfer to MSLesSeg, the program achieves lesion Dice of 0.587 versus the tree's 0.550 with FLAIR input, matching the source-trained network's mean Dice. On Shifts, the ordering reverses. Controlled noise and bias-field experiments do not reproduce this reversal as a function of severity alone. Attribute analysis instead identifies greater orientation and multi-contrast drift on Shifts. Removing the associated rules and recalibrating closes 70% of the FLAIR program-tree gap there, but reduces multi-contrast performance on MSLesSeg. This intervention supports an attribute-specific explanation rather than a general robustness advantage for written rules. A threshold sweep also exposes a mismatch between a literal 3 mm size rule and its implementation on thick-slice images. LLM-generated programs are less reliable on real candidates than the hand-written reference. Together, these results clarify the benefits and limits of protocol-derived structure: its transfer value depends on the attributes affected by the target shift, while explicit rules make those dependencies inspectable and editable. We also correct two label-orientation errors in the public MS3SEG release; anonymized code and every generated program accompany this submission.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
neurosymbolic AI, interpretability, distribution shift, LLM program synthesis, medical image segmentation, clinical decision rules, auditable machine learning, multiple sclerosis lesion segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Uy7JOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript investigates the use of combining explicit rule-sets aside to an end-to-end image segmentation model for the medical application of multiple sclerosis lesion segmentation. The motivation for this resides in the more explainable nature of the rules in the form of a decision tree or python code which then leads to the assignment of normal / abnormal white matter hyperintensities in magnetic resonance imaging. The methodological contribution summarizes to a workflow how rules can be integrated into neural network-based multiple sclerosis lesion segmentation. Experiments are carried out on three small-scale datasets (MS3SEG, MSLesSeg and Shifts) where a variety of alternatives for the integration of rules into the workflow are compared to each other and to end-to-end trained neural networks. The results of these experiments are extensively described and a broad set of method- and data-variations are explored.

Critical strengths
  1. Critical strength: The area of research and experimental exploration of combining sub-symbolic models that excel at pattern recognition and representation learning with symbolic methods that can be verified, audited and changed at test-time is valuable. Why it is consequential to my overall assessment: High-stakes domains such as the medical imaging domain where patient health and safety are directly implicated by a system's assessment, e.g., through a wrong diagnosis, require measures to audit and interpret how an assessment came about. Research that provides explanation on the decision making process thereby can have a big impact on practical application of medical image analysis systems. This research is also timely with the availability of effective image segmentation models and the availability of large language models for parsing medical knowledge into program code.
Critical weaknesses
  1. Critical weakness: The structure and presentation of the manuscript is in a severely rough state such that readers can not easily follow the strain of thought while major inconsistencies are present. Why it is consequential to my overall assessment: Without clearly conveying the methodological contributions, results and their interpretation in a concise, well thought through, curated manner, the manuscript is substantially less accessible to its intended audience, limiting its potential impact.

(a) The structure of the manuscript makes it hard to follow the line of thought in multiple sections, as there are constant references to content beyond the main manuscript, i.e., the appendix. This makes the main body of text not self-contained. Here is a list of references from the main body of text to results in the appendix (marked in bold).

  • (line 259) "the best real method(the learned classifier) reaches only CV 0.648/0.693 (FLAIR)"

→\to From the main body this references results from Table 6 in Appendix C.

  • (line 282) "the classifier still reaches only CV 0.656/0.699 (FLAIR) and 0.649/0.697 (multi) and the program only 0.634/0.669 and 0.632/0.666 (Table 6)—5–7 and 7–9 points below the network’s own native decision on those same pixels "

→\to references results from Table 1 (main paper) and Table 6 in the Appendix, interrupting reading flow, and using the appendix as extension to circumvent the page limit as stated in the caption of Table 6 in the appendix: "Rows omitted from Table 1 for space".

  • (line 428) "and is not overtaken until 32 (Table 7)"

→\to references results in the appendix

  • (line 449) "Table 4 (Appendix C) removes one attribute group"

→\to references results in the appendix

  • (line 455) "accuracy conditioned on which rule fired ranges from 42.6% to 82–84% (Table 5, Appendix C)—decision agreement"

→\to references results in the appendix

  • (line 457) "audited the same way, is at least as uneven (16 leaves, 19.2–93.0%, Table 11)"

→\to references results in the appendix

  • (line 463) "built from it is the program’s least trustworthy (Table 5)"

→\to references results in the appendix

  • (line 466) "A small prompt-content ablation is mixed (Appendix C): dropping the protocol for general knowledge yields more valid programs"

→\to references the appendix generically

  • (line 471) "Iterative refinement (Appendix B) did not work reliably."

→\to appendix is refereneced for evidence for the statement made

Referring to the appendix is of course no issue in itself when doing so sparringly. In this case, the appendix is used readily as extension to the main text for circumventing the page limit and the appendix is referred to so often that it severely impacts readability.

Further, multiple times, the text refers to future sections as if the reader already is familiar with their content, which also shows the issues in the structure of the manuscript.

  • (line 269, 277, 286) →\to references into future section 4.3

(c) The text includes sudden shifts in settings when describing results, which are not communicated, leaving the reader wondering in which Table to find the stated numbers. An example of this can be found in (line 405) where the manuscript reads "[...] recalibrate, and re-run Table 2. In-domain this costs nothing (CV 0.622/0.651 vs. 0.620/0.654), matching Table 4.". Here, it is completely opaque, in which Table the reader needs to look for the results. Rather than in Table 2 of the main manuscript or Table 4 of the appendix which are mentioned in the section, the results are actually found in Table 7 of the appendix, which is never mentioned. Such inconsistencies and missing contextualization for better readability can be found throughout the text. This missing contextualization extends to method names, which are often not clearly introduced, or inconsistently referred to, which makes it hard to find numerical results of a method from the text in the respective table. E.g., "Decision tree unmodified", "Learned classifier", "GBDT" and "Decision tree(depth 4, same attributes)" all seem to refer to variants of decision trees, some refer to the exact same trees.

(e) Some sentences in the manuscript are semantically unclear and it is not clear what is actually meant, as their content is not introduced graciously. Examples for such sentences are the following:

  • (line 152) "[...] and we hold the program’s structural origin fixed as a variable against a matched-interpretability tree"
  • (line 271) "Naming a split “periventricular” rather than “dist_to_ventricle_mm < 2.1” does not, by itself, buy accuracy a data-fit split of the same depth does not already provide; whatever value the program’s language-derived structure has over data-derived structure at matched capacity is not in Dice."
  • (line 429) "A zero-shot VLM judge needs no target-domain patients either (Section 4.4)"

(f) Inconsistencies between stated contributions and what is presented in the main manuscript can be found. For example, the fourth contribution mentions "a zero-shot VLM judge baseline" (line 89), which is only mentioned in line 428 - 431 in the result section, without substantial evaluation or insights.

  1. Critical weakness: The experimental setup and exploration domain is too narrow. Why it is consequential to my overall assessment: The manuscript exclusively explores an application-specific subproblem in medical image segmentation, namely the issue of obtaining interpretable predictions for multiple sclerosis lesion segmentation. The generality of the proposed workflow is never evaluated on a different medical image segmentation problem, thereby severely limiting the contribution and potential impact of the proposed workflow. Further, the manuscript does not compare the proposed workflow to prior art in the field of program synthesis, while it also barely evaluates the stated implications on interpretability of the resulting system. Even though it is stated in the second contribution that "matched interpretability" is investigated, in the respective section in the manuscript there is almost no information on interpretability, which makes the contribution statement misaligned with the actual content in the manuscript.
Questions for authors

What could be an explaination of the worse results on multi-contrast evidence vs. FLAIR only evaluation in Table 2, as the former seems to include FLAIR.

Could the authors clarify what is meant by the statements referred to in weaknesses 1 (e).

1 comment

We accept the presentation criticism and restructured the paper (changes in blue).

1(a) Too many appendix references; appendix used to extend the main text. The numbers behind the paper's main argument are now in main-text tables:

  • Same-region rows (old Table 6) moved into Table 1 (p. 6 L. 282); L259/L282 now cite Table 1 only.
  • Table 2 (p. 7) now holds every external comparison.
  • VLM judge: full results in §4.4 (p. 9 L. 453) and Table 2.

Remaining pointers (Tables 4, 5, 7, 11; prompt ablation; refinement) lead to supporting detail. Their key numbers are now stated in the text (e.g. per-rule agreement, p. 9), so §4 reads without the appendix. The 10-page limit does not leave room to move those tables as well.

Forward references (L269, 277, 286). §4.1 no longer cites §4.3 results; its one remaining pointer is to Figure 2 (data efficiency), which we keep because it shows the same tree-vs-program comparison across budgets.

1(c) Unannounced setting changes; inconsistent method names. Quoted numbers now name their source (e.g. the L405 pruning sentence now cites App. G). One name per method, defined once in §3 (p. 5 L. 228): network (end-to-end four-class U-Net), program, tree (depth-4, ≤16 leaves; previously also "decision tree unmodified"), GBDT (previously also "learned classifier"), VLM judge, oracle.

1(e) Unclear sentences.

  • L152 (now p. 3 L. 144): "we do not assume that the program's language-derived structure is what makes it useful: we test this against a tree of the same size and against established rule learners, all built from the same attributes but with every split learned from data."
  • L271 (now p. 6 L. 292): "writing a rule as 'periventricular' instead of learning the split dist_to_ventricle_mm < 2.1 from data does not improve in-domain Dice: a learned tree of the same depth is at least as accurate. Any benefit of writing the rules from clinical text must therefore appear elsewhere—under shift, with no labels, or in auditability."
  • L429 (now p. 9 L. 433): "The program is not the only method that needs no annotated patients: the zero-shot VLM judge needs none either, but is 7–9 Dice points worse on both external datasets."

1(f) VLM contribution is thin. The VLM judge is now thresholded once on source candidates and scored as a complete segmentation system on all three datasets: lesion Dice 0.580 / 0.288 / 0.488 (MS3SEG test / Shifts / MSLesSeg). The program is significantly better on each (p ≤ 2e-4). Contribution list updated (p. 2 L. 77).

2. No comparison to prior art in program synthesis or rule learning. We add five established interpretable rule learners, each fit on exactly the same 34 attributes and source candidates: RIPPER (Cohen 1995), a greedy rule list, FIGS (Tan et al.), RuleFit (Friedman & Popescu 2008) and boosted rules. We also add L1 logistic regression. LLM program synthesis, the paradigm of VisProg and INFORM-CT applied to our attributes, is compared in §4.4: LLM-compiled programs are weaker than the hand-written one on real candidates. Results (Table 2 p. 7; text p. 7 L. 363; FLAIR, patient level):

  • The program clearly beats RIPPER (MSLesSeg 0.573 vs 0.295) and the greedy rule list (0.540).
  • It is level with FIGS, RuleFit and boosted rules on Shifts.
  • It is 1.5–2 points behind them on MSLesSeg.

So we narrowed the claim: a program written once without labels transfers on par with data-fit rule learners, not better.

2. Interpretability barely evaluated. §4.5 (p. 9 L. 464–482) now reports in the main text:

  • Rule-level agreement with the reference: 42.6–84% per fired rule, versus 19.2–93.0% for the tree's leaves.
  • A parameter-edit sweep that exposed a concrete mismatch between a literal 3 mm protocol rule and 6 mm slices.
  • Model size against the rule learners: 10 named rule units, versus FIGS 14 leaves, boosted rules 13 stumps, RIPPER 16 conjunctions and RuleFit 30 terms.

We call these auditability proxies, not clinician benefit; a clinician audit study is planned for the final version.

2. Single application. A fair limitation, now stated in Limitations (p. 10 L. 499). MS lesion vs normal-WMH discrimination is where a written, attribute-based protocol exists (McDonald/MAGNIMS) and public multi-scanner data allow real external tests. We test three datasets from different scanners and sites, five data budgets and two input settings, and do not claim generality beyond MS.

Q1. Why is multi-contrast worse than FLAIR? T1 and T2 add evidence that the target scanners shift more than FLAIR. On Shifts the T1 contrast of true lesions moves by 2.4 source SDs, so the T1 "black hole" rule fires on 47% of true lesions instead of 4%. On MSLesSeg, T2 confirmation fires on 18% instead of 31%. Almost every system loses Dice with three modalities (Table 2), e.g. GBDT 0.389 → 0.270 on Shifts. Added to §4.2 (p. 8 L. 394).

Reviewer_VZTwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CriteriaCode separates MS lesion segmentation into learned candidate detection, 34 radiological attributes, and an executable criteria program with calibrated thresholds. Its main comparison is between a hand-written program and a shallow decision tree using identical attributes. The program loses to learned alternatives in-domain, exceeds the tree on MSLesSeg with FLAIR input, and loses on Shifts. Synthetic perturbations, attribute-drift analysis, and rule pruning investigate this reversal. Additional experiments examine candidate quality, annotation budgets, rule-level decision agreement, and LLM-generated programs, which perform worse on real candidates than the hand-written reference. The main contribution is an empirical account of when explicit criteria help, rather than a claim of uniformly superior segmentation.

Critical strengths

Giving the criteria program and shallow tree the same clinically informed attributes avoids attributing every benefit of those attributes to written rules. The end-to-end, learned-classifier, oracle, and same-predicted-region comparisons also examine whether performance differences arise from candidate localization or candidate-level classification. The paper reports that the decomposition does not beat the network in-domain rather than obscuring this result. These controls are central to the value of the study. The analysis moves beyond a blanket robustness claim. The program–tree ordering reverses across the external datasets, and the controlled noise/bias experiments do not reproduce a simple severity explanation. The subsequent attribute analysis and pruning intervention provide a concrete, inspectable account of a possible failure mechanism, including the fact that pruning helps one target while hurting another. Patient-level testing of the primary external comparisons also appropriately accounts for repeated timepoints.

Critical weaknesses

The transfer and mechanism claims need to remain conditional on the preprocessing and intervention design. External 1 mm volumes are converted into approximately 6 mm slabs, and the corresponding lesion masks use a majority rule across the contributing slices. This changes the geometry of the evaluated images and labels, including properties related to the size, shape, and orientation attributes under investigation. Applying the same transformation to all methods preserves the within-protocol comparison, but it does not establish that the same ordering or mechanism holds at native resolution.The result supports an attribute-specific explanation, but it does not separately identify the effects of rule removal, recalibration, and target-informed intervention selection. A recalibration-only control and a clearer account of the selection boundary would strengthen the mechanistic claim.

Questions for authors

Were the rule-pruning decision and recalibration procedure fixed without target labels or target performance feedback? How much improvement remains relative to an unpruned program given the same recalibration opportunity? How many lesions or lesion voxels are removed or substantially changed by slab averaging and majority-mask formation? Does the program–tree reversal persist under an alternative evaluation that better preserves native lesion geometry?

1 comment

We thank the reviewer for a careful and constructive review. It asked for exactly the right controls. We ran all of them; one result changes a claim, and we report it plainly.

Q1. How many lesions or voxels are removed or changed by slab averaging and majority masks? We rebuilt each external mask at native 1 mm resolution, verified that its majority slab exactly reproduces the evaluated label, and matched native lesions to the slab labels (new Appendix F, p. 24; pointer in §4 p. 5 L. 243):

MSLesSegShifts
Native lesions2,5293,117
Lost entirely518 (20.5%)1,201 (38.5%)
Median size of lost lesions21 voxels11 voxels
Native voxels missed / added18.4% / 24.6%25.9% / 27.2%
Median per-scan Dice, slab vs native0.7500.705

Slabbing mostly removes very small lesions, and it changes Shifts considerably more than MSLesSeg.

Q2. Does the program–tree ordering persist at native resolution? Yes. We carried every system's decisions back to the 1 mm grid and scored them against the native masks (patient-level lesion Dice; program averaged over 5 calibration seeds as in Table 2; FLAIR / three modalities):

ProgramTreeGBDTNetwork
MSLesSeg0.496 / 0.4730.465 / 0.4720.501 / 0.4790.504 / 0.473
Shifts0.298 / 0.2220.318 / 0.2320.321 / 0.2250.255 / 0.212

The program stays ahead of the tree on MSLesSeg FLAIR (p = 2e-4), tied with three modalities (p = 0.59), and behind it on Shifts (p ≤ 6e-4). The three significant program–tree differences also survive Holm correction in Table 2. The paper now reports both protocols (§4.2 pp. 7–8 L. 366–394; Appendix F).

Q3. Were pruning and recalibration fixed without target labels or target performance? Pruning used no target labels and no target performance, and recalibration used only source data. However, the five pruned rule units were chosen from label-free target attribute-drift statistics, so the choice is target-informed. The submitted text did not make this clear; the revision states it explicitly (§4.2 p. 8 L. 411; Appendix G p. 24).

Q4. How much improvement remains with the same recalibration opportunity? (recalibration-only control and selection null)

  • Recalibration only. We recalibrated the unpruned program with five TPE seeds. On Shifts FLAIR its patient-level Dice ranges 0.306–0.376 (mean 0.359). The single seed in the submission was the least favourable, so most of the apparent pruning gain was seed variance. Seed-averaged, pruning still adds +0.012 (0.371 vs 0.359, p = 0.001) and reduces seed variance (SD 0.007 vs 0.027).
  • Selection null. We recalibrated all C(10,5) = 252 five-unit prunings identically. The drift-selected pruning ranks 230/252 on Shifts FLAIR and 155/252 on MSLesSeg FLAIR. Random prunings do as well or better.

We therefore withdraw the claim that pruning confirms an attribute-specific failure mechanism. The attribute-drift analysis remains as a description of what the shift does to each attribute. The pruning experiment is reported with its null, as a negative result about target-informed rule selection. We are grateful the reviewer pushed on this before publication.

Consequences for the main comparison. Table 2 (p. 7 L. 324) is now patient-level, averages five calibration seeds and adds five interpretable rule learners. The program's advantage over the tree on MSLesSeg FLAIR holds (0.573 vs 0.536, p = 2e-4, Holm-corrected 7e-4): across seeds, at native resolution, when all our numeric defaults are redrawn (0.590 ± 0.003), and even with no calibration at all (0.564, p = 0.02). The tree's advantage on Shifts holds as well. Against stronger rule learners (FIGS, RuleFit, boosted rules, L1 logistic regression), the program is on par on Shifts (only logistic regression is significantly ahead, by 0.009) and 1.5–2 points behind on MSLesSeg. We have therefore narrowed the contribution: a program written once without labels transfers on par with data-fit interpretable models, but not better. We believe this conditional, fully controlled account is the analysis the reviewer valued, and that it is now on firmer ground.

Reviewer_dSPQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks when a written clinical protocol, compiled into code, transfers better than a classifier fitted to data. The paper uses three datasets - MS3SEG (in-domain) and Shifts and MSLesSeg (domain-shifted/unseen). They show that in the in‑domain dataset the end‑to‑end deep‑learning pipeline works well, but in unseen contexts, especially with small training dataset, a pipeline that predicts potential lesions and uses interpretable, rule‑based criteria derived from lesion features such as geometry and proximity to anatomical landmarks based on qualitative protocol in the clinical literature is useful.

Critical strengths

I think the problem that's been picked and the proposed idea is reasonable (see below for execution issues).

Critical weaknesses
  1. Missing details, Experimental Setup issues
  • The hand-written reference program seem incomplete. How does one go from segmentation mask of hyperintensities to "periventricular horn cap within 3 mm of the anterior or posterior horn"? This would probably require non-trivial geometric heuristics that may sometime break (or fine grained anatomy segmentation which does not seem available). These geometric heuristics are either swept as one-line prose that are far from reproducible/replicable.
  • The purpose of the zero-shot VLM as judge experiment is unclear. What question does it answer, specially since it reports AUC (different from Dice score that all other experiments report) and is reported on in-domain only?
  • §4.2 defines "non-reasoning baseline" as a baseline that "calls every detected hyperintensity a lesion". Is that reasonable? This seems like a very weak baseline specially since the underlying segmentation model is trained on merged lesion class (both normal and abnormal WMH are merged as lesion).
  • The hand-written program is presented as protocol-derived, but there are thresholds such as "rim orientation > 60" and increment the additive score by 0.15 whose provenance/source is unclear and seems like author judgement. Please provide either a provenance table or clinician review.
  • The paper cites anonymized code release, but is not available anywhere (either as supplement or link).
  1. lacks Clinical Evaluation The central stated benefit is that clinicians can inspect and edit the rules, yet no clinician is involved anywhere. A user study showing how interpretability allows flagging false negatives or improved end to end performance or some human centered metric as a proxy for trust or usability would provide better support for the claim that the structure is clinician-editable and auditable.
Questions for authors
  • Clinical evaluation. The central claim is that clinicians can inspect and edit the rules, yet no clinician is involved. Can you provide a user study (e.g., whether the rules help flag false negatives, improve end-to-end performance, or raise trust and usability) to support the claim that the structure is clinician-editable and auditable?
  • Provenance of the rules. The program is presented as protocol-derived, but thresholds (e.g., rim orientation ≥ 50°) and additive score increments (e.g., +0.15) have no cited source and appear to be author judgement. Please provide a provenance table or a clinician review.
1 comment

We thank the reviewer. Every point led to a change in the revised PDF (changes in blue).

1. Geometric heuristics are not reproducible (e.g. horn caps). Every attribute is computed per slice from only two masks predicted by the perception network: ventricles V and brain B. No fine-grained anatomy is used. New Appendix E (p. 24) gives pseudo-code for every geometric attribute; §3 points to it (p. 4 L. 212). Horn regions: the anterior (posterior) horn is the part of V within the most anterior (posterior) 15% of the ventricle's AP extent on that slice. Horn distance is the Euclidean distance transform to that region, in mm. A horn cap fires if horn distance ≤ 3 mm and mean thickness ≤ 3 mm. Orientation is the angle between the candidate's major axis and the gradient of the ventricle distance transform at its centroid (0° = perpendicular, Dawson's finger; 90° = parallel, rim/cap). These are approximations, and we say so; Table 5 reports how often each rule's decision agrees with the reference. The code is released (link below).

2. Purpose of the VLM judge; AUC; in-domain only. It answers this question: with the same zero target data, can a general VLM replace written criteria by looking at the candidate? We now threshold its score once on source candidates and score it as a complete segmentation system with the same lesion Dice as every other row (§3 p. 5 L. 219; Table 2 p. 7; §4.4 p. 9 L. 453):

Lesion DiceMS3SEG testShiftsMSLesSeg
VLM judge0.5800.2880.488
Program0.6920.3590.573

The program is significantly better on all three (p ≤ 2e-4, patient level; program averaged over 5 calibration seeds on the external sets, Table 1 seed in-domain). Compiling the protocol beats asking a VLM to apply it: even the program's uncalibrated defaults, which use no labels at all, beat the VLM on MSLesSeg (0.564 vs 0.488, p < 1e-6).

3. The call-everything baseline is too weak. Agreed. We add (§3 p. 5 L. 223; Table 2 p. 7; text p. 7 L. 350) a source-calibrated size filter (the most common post-hoc rule), a contrast filter, and L1 logistic regression on all 34 attributes. The program beats the size filter by 0.11–0.37 Dice on every dataset (Shifts 0.359 vs 0.248, MSLesSeg 0.573 vs 0.408, in-domain 0.682 vs 0.314; p < 1e-5), and the contrast filter (in-domain 0.459, Shifts 0.000, MSLesSeg 0.083) by more still on the external sets. We also report honestly that on Shifts, calling every candidate a lesion (0.386) remains above every rule set that removes candidates, including the program; it is below the program on MSLesSeg (0.487). Logistic regression is a learned model and is reported with the other learners (it is 1 point ahead of the program on Shifts and 2 points ahead on MSLesSeg).

4. Provenance of thresholds and increments. New Appendix D (p. 23, Table 12) gives a provenance table: for each rule, the clinical source of the rule (Fazekas 1987, STRIVE, McDonald 2017, MAGNIMS 2016, MS3SEG protocol) and, separately, the source of the number. Stated plainly: the rule structure and the sign of every evidence weight come from the protocol and literature. Only one numeric default comes from the literature (the 3 mm minimum lesion size). All others, including the 50° rim orientation and the ±0.1–0.35 increments, are our choices that operationalise qualitative text. The paper now says this in §3 (p. 4 L. 212). We tested whether these choices matter:

FLAIR lesion DiceMS3SEGShiftsMSLesSeg
Our constants0.6920.2770.587
All weights equal (±0.2)0.6910.3450.547
All defaults redrawn ×U[0.5,2], 20 draws0.683±0.0070.349±0.0110.590±0.003

In-domain and on MSLesSeg, results do not depend on our constants. On Shifts, our constants were among the least favourable choices.

5. Code. We apologise that the link was missing. Anonymized code, every program and every rebuttal script: https://anonymous.4open.science/r/criteriacode-4685/

6. Clinician study. We agree this is the key missing evidence for the auditability claim, and we no longer overstate it. The revised paper says that it measures auditability proxies: rule-level agreement (Table 5), a parameter-edit sweep that exposed a concrete protocol-to-acquisition mismatch, and model size against the rule learners (§4.5 p. 9 L. 472). It does not claim a clinician benefit (Limitations, p. 10 L. 488). For the final version, we plan a study in which two neuroradiologists audit the program on held-out scans, recording flagged errors, the edits they make, Dice after editing, and a standard trust/usability questionnaire.

Comments on the paper

1 comment

We thank the reviewers. Their requests led to eight new experiments on the same pipeline, data and splits. The revised PDF marks every change in blue (main text within the 10-page limit); each reply below gives the page (p.) and line (L.) of every change.

New experiments

  1. Stronger non-reasoning baselines: a source-calibrated size filter, a contrast filter and L1 logistic regression (dSPQ).
  2. Five established interpretable rule learners on the same 34 attributes: RIPPER, a greedy rule list, FIGS, RuleFit and boosted rules (Uy7J).
  3. The VLM judge as a complete segmentation system, scored with lesion Dice on all three datasets (dSPQ, Uy7J).
  4. A provenance table for every rule and constant, plus sensitivity to all author constants: 20 random redraws, equal weights and calibrated weights (dSPQ).
  5. A slab-geometry audit and a native 1 mm re-evaluation of every system (VZTw).
  6. Five calibration seeds and an exhaustive pruning null over all 252 five-rule prunings (VZTw).
  7. A seed-averaged, patient-level Table 2 with all comparators (all reviewers).
  8. Attribute pseudo-code, including the horn regions (dSPQ), and an analysis of why multi-contrast input is worse than FLAIR (Uy7J).

What the evidence supports (revised Table 2, p. 7 L. 324: patient-level lesion Dice, FLAIR, program averaged over 5 seeds)

ShiftsMSLesSeg
Size filter0.2480.408
VLM judge (zero-shot)0.2880.488
Program, uncalibrated defaults (no labels at all)0.3200.564
Program (rules written without labels; thresholds calibrated on source)0.3590.573
Depth-4 tree0.3820.536
FIGS / RuleFit / boosted rules / L1 logreg0.345–0.3680.588–0.592
Four-class network0.3120.582

A criteria program written once from the protocol, without seeing any label, transfers on par with interpretable models whose structure is fit to source labels. It is ahead of the matched tree on MSLesSeg and behind it on Shifts. It is significantly above calibrated filters and the zero-shot VLM judge on both datasets, and it is the only candidate-level method usable with zero annotated patients.

Two controls narrow our claims, and we report them openly:

  • Not more accurate than data-fit rule learners. The submitted claim that written structure transfers better under moderate shift holds only against the depth-4 tree. Modern rule learners are 1.5–2 points ahead on MSLesSeg. We withdraw that claim (abstract; contributions p. 2 L. 77–92; §4.2 p. 7 L. 366; conclusion p. 10 L. 502).
  • No confirmed mechanism. Pruning the rules built on the most-drifting attributes does no better than random prunings (rank 230/252). We withdraw the claim that pruning confirms a mechanism, and keep attribute drift as a description (p. 8 L. 411; Appendix G p. 24).

What a reader learns that was not known before (two reviewers rated informativeness 1):

  1. On real multi-scanner MRI, decision logic written once from a clinical protocol, with zero labels, matches interpretable learners fit to labels on identical attributes. Even its uncalibrated defaults, which use no labels from any dataset, reach 0.564 on MSLesSeg, above a depth-4 tree fit to 80 labelled patients (0.536, p = 0.02). Prior work on compiled symbolic programs (e.g. Wang et al. 2025) reports an advantage over a prototype tree; against eight data-fit learners we find parity, not superiority.
  2. Given the same protocol and zero target data, compiling it into code beats asking a VLM to apply it, by 7–11 Dice points.
  3. Two natural explanations of where written rules fail, shift severity and drift-guided rule pruning, fail controlled tests. These are negative results with their controls, useful to anyone proposing either.

We believe the revised paper is a more useful contribution: a fully controlled account of what compiled clinical criteria buy (inspectability, editability, a zero-label starting point, transfer on par with data-fit interpretable models) and what they do not (an accuracy advantage). Reviewer VZTw identified this kind of honest, controlled analysis as the paper's main value.

Code (anonymized, including every rebuttal script): https://anonymous.4open.science/r/criteriacode-4685/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.