- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This manuscript investigates the use of combining explicit rule-sets aside to an end-to-end image segmentation model for the medical application of multiple sclerosis lesion segmentation. The motivation for this resides in the more explainable nature of the rules in the form of a decision tree or python code which then leads to the assignment of normal / abnormal white matter hyperintensities in magnetic resonance imaging. The methodological contribution summarizes to a workflow how rules can be integrated into neural network-based multiple sclerosis lesion segmentation. Experiments are carried out on three small-scale datasets (MS3SEG, MSLesSeg and Shifts) where a variety of alternatives for the integration of rules into the workflow are compared to each other and to end-to-end trained neural networks. The results of these experiments are extensively described and a broad set of method- and data-variations are explored.
Critical strengths
- Critical strength: The area of research and experimental exploration of combining sub-symbolic models that excel at pattern recognition and representation learning with symbolic methods that can be verified, audited and changed at test-time is valuable. Why it is consequential to my overall assessment: High-stakes domains such as the medical imaging domain where patient health and safety are directly implicated by a system's assessment, e.g., through a wrong diagnosis, require measures to audit and interpret how an assessment came about. Research that provides explanation on the decision making process thereby can have a big impact on practical application of medical image analysis systems. This research is also timely with the availability of effective image segmentation models and the availability of large language models for parsing medical knowledge into program code.
Critical weaknesses
- Critical weakness: The structure and presentation of the manuscript is in a severely rough state such that readers can not easily follow the strain of thought while major inconsistencies are present. Why it is consequential to my overall assessment: Without clearly conveying the methodological contributions, results and their interpretation in a concise, well thought through, curated manner, the manuscript is substantially less accessible to its intended audience, limiting its potential impact.
(a) The structure of the manuscript makes it hard to follow the line of thought in multiple sections, as there are constant references to content beyond the main manuscript, i.e., the appendix. This makes the main body of text not self-contained. Here is a list of references from the main body of text to results in the appendix (marked in bold).
- (line 259) "the best real method(the learned classifier) reaches only CV 0.648/0.693 (FLAIR)"
From the main body this references results from Table 6 in Appendix C.
- (line 282) "the classifier still reaches only CV 0.656/0.699 (FLAIR) and 0.649/0.697 (multi) and the program only 0.634/0.669 and 0.632/0.666 (Table 6)—5–7 and 7–9 points below the network’s own native decision on those same pixels "
references results from Table 1 (main paper) and Table 6 in the Appendix, interrupting reading flow, and using the appendix as extension to circumvent the page limit as stated in the caption of Table 6 in the appendix: "Rows omitted from Table 1 for space".
- (line 428) "and is not overtaken until 32 (Table 7)"
references results in the appendix
- (line 449) "Table 4 (Appendix C) removes one attribute group"
references results in the appendix
- (line 455) "accuracy conditioned on which rule fired ranges from 42.6% to 82–84% (Table 5, Appendix C)—decision agreement"
references results in the appendix
- (line 457) "audited the same way, is at least as uneven (16 leaves, 19.2–93.0%, Table 11)"
references results in the appendix
- (line 463) "built from it is the program’s least trustworthy (Table 5)"
references results in the appendix
- (line 466) "A small prompt-content ablation is mixed (Appendix C): dropping the protocol for general knowledge yields more valid programs"
references the appendix generically
- (line 471) "Iterative refinement (Appendix B) did not work reliably."
appendix is refereneced for evidence for the statement made
Referring to the appendix is of course no issue in itself when doing so sparringly. In this case, the appendix is used readily as extension to the main text for circumventing the page limit and the appendix is referred to so often that it severely impacts readability.
Further, multiple times, the text refers to future sections as if the reader already is familiar with their content, which also shows the issues in the structure of the manuscript.
- (line 269, 277, 286) references into future section 4.3
(c) The text includes sudden shifts in settings when describing results, which are not communicated, leaving the reader wondering in which Table to find the stated numbers. An example of this can be found in (line 405) where the manuscript reads "[...] recalibrate, and re-run Table 2. In-domain this costs nothing (CV 0.622/0.651 vs. 0.620/0.654), matching Table 4.". Here, it is completely opaque, in which Table the reader needs to look for the results. Rather than in Table 2 of the main manuscript or Table 4 of the appendix which are mentioned in the section, the results are actually found in Table 7 of the appendix, which is never mentioned. Such inconsistencies and missing contextualization for better readability can be found throughout the text. This missing contextualization extends to method names, which are often not clearly introduced, or inconsistently referred to, which makes it hard to find numerical results of a method from the text in the respective table. E.g., "Decision tree unmodified", "Learned classifier", "GBDT" and "Decision tree(depth 4, same attributes)" all seem to refer to variants of decision trees, some refer to the exact same trees.
(e) Some sentences in the manuscript are semantically unclear and it is not clear what is actually meant, as their content is not introduced graciously. Examples for such sentences are the following:
- (line 152) "[...] and we hold the program’s structural origin fixed as a variable against a matched-interpretability tree"
- (line 271) "Naming a split “periventricular” rather than “dist_to_ventricle_mm < 2.1” does not, by itself, buy accuracy a data-fit split of the same depth does not already provide; whatever value the program’s language-derived structure has over data-derived structure at matched capacity is not in Dice."
- (line 429) "A zero-shot VLM judge needs no target-domain patients either (Section 4.4)"
(f) Inconsistencies between stated contributions and what is presented in the main manuscript can be found. For example, the fourth contribution mentions "a zero-shot VLM judge baseline" (line 89), which is only mentioned in line 428 - 431 in the result section, without substantial evaluation or insights.
- Critical weakness: The experimental setup and exploration domain is too narrow. Why it is consequential to my overall assessment: The manuscript exclusively explores an application-specific subproblem in medical image segmentation, namely the issue of obtaining interpretable predictions for multiple sclerosis lesion segmentation. The generality of the proposed workflow is never evaluated on a different medical image segmentation problem, thereby severely limiting the contribution and potential impact of the proposed workflow. Further, the manuscript does not compare the proposed workflow to prior art in the field of program synthesis, while it also barely evaluates the stated implications on interpretability of the resulting system. Even though it is stated in the second contribution that "matched interpretability" is investigated, in the respective section in the manuscript there is almost no information on interpretability, which makes the contribution statement misaligned with the actual content in the manuscript.
Questions for authors
What could be an explaination of the worse results on multi-contrast evidence vs. FLAIR only evaluation in Table 2, as the former seems to include FLAIR.
Could the authors clarify what is meant by the statements referred to in weaknesses 1 (e).
We accept the presentation criticism and restructured the paper (changes in blue).
1(a) Too many appendix references; appendix used to extend the main text. The numbers behind the paper's main argument are now in main-text tables:
Remaining pointers (Tables 4, 5, 7, 11; prompt ablation; refinement) lead to supporting detail. Their key numbers are now stated in the text (e.g. per-rule agreement, p. 9), so §4 reads without the appendix. The 10-page limit does not leave room to move those tables as well.
Forward references (L269, 277, 286). §4.1 no longer cites §4.3 results; its one remaining pointer is to Figure 2 (data efficiency), which we keep because it shows the same tree-vs-program comparison across budgets.
1(c) Unannounced setting changes; inconsistent method names. Quoted numbers now name their source (e.g. the L405 pruning sentence now cites App. G). One name per method, defined once in §3 (p. 5 L. 228): network (end-to-end four-class U-Net), program, tree (depth-4, ≤16 leaves; previously also "decision tree unmodified"), GBDT (previously also "learned classifier"), VLM judge, oracle.
1(e) Unclear sentences.
1(f) VLM contribution is thin. The VLM judge is now thresholded once on source candidates and scored as a complete segmentation system on all three datasets: lesion Dice 0.580 / 0.288 / 0.488 (MS3SEG test / Shifts / MSLesSeg). The program is significantly better on each (p ≤ 2e-4). Contribution list updated (p. 2 L. 77).
2. No comparison to prior art in program synthesis or rule learning. We add five established interpretable rule learners, each fit on exactly the same 34 attributes and source candidates: RIPPER (Cohen 1995), a greedy rule list, FIGS (Tan et al.), RuleFit (Friedman & Popescu 2008) and boosted rules. We also add L1 logistic regression. LLM program synthesis, the paradigm of VisProg and INFORM-CT applied to our attributes, is compared in §4.4: LLM-compiled programs are weaker than the hand-written one on real candidates. Results (Table 2 p. 7; text p. 7 L. 363; FLAIR, patient level):
So we narrowed the claim: a program written once without labels transfers on par with data-fit rule learners, not better.
2. Interpretability barely evaluated. §4.5 (p. 9 L. 464–482) now reports in the main text:
We call these auditability proxies, not clinician benefit; a clinician audit study is planned for the final version.
2. Single application. A fair limitation, now stated in Limitations (p. 10 L. 499). MS lesion vs normal-WMH discrimination is where a written, attribute-based protocol exists (McDonald/MAGNIMS) and public multi-scanner data allow real external tests. We test three datasets from different scanners and sites, five data budgets and two input settings, and do not claim generality beyond MS.
Q1. Why is multi-contrast worse than FLAIR? T1 and T2 add evidence that the target scanners shift more than FLAIR. On Shifts the T1 contrast of true lesions moves by 2.4 source SDs, so the T1 "black hole" rule fires on 47% of true lesions instead of 4%. On MSLesSeg, T2 confirmation fires on 18% instead of 31%. Almost every system loses Dice with three modalities (Table 2), e.g. GBDT 0.389 → 0.270 on Shifts. Added to §4.2 (p. 8 L. 394).