- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MIRROR, a neuro-symbolic architecture that formulates clinical decision-making as an iterative perception–reasoning process. MIRROR employs sparse attention to determine the next clinical inquiry and executes symbolic Horn clauses through a Straight-Through Estimator (STE) combined with differentiable SoftMin and soft-OR operations. It maintains a signed state matrix to represent accumulated evidence and evolving diagnostic hypotheses across reasoning iterations, with an optional VQ-based knowledge-region module for conditioning rule reliabilities. Experiments on three medical datasets show that MIRROR achieves competitive diagnostic accuracy while providing explicit traces of the cognitive inquiry process underlying its decisions.
Critical strengths
-
The full patient record ablation is highly compelling. Table 3 shows that providing all available evidence at can actually degrade performance; for example, DiReCT drops from 50.46 to 38.43. This is a strong empirical observation. It indicates that the iterative perception-reasoning loop is not merely a conceptual addition, but a necessary mechanism for filtering noisy evidence and approximating the sequential nature of clinical inquiry.
-
The integration of discrete inquiry actions with differentiable logic is also well executed. The paper addresses a genuine architectural challenge: how to propagate diagnosis-level supervision through discrete inquiry selection. The Straight-Through Estimator, or STE, in Eq. 7 provides a practical bridge between the diagnosis loss and the sparse attention-based inquiry scorer. Meanwhile, the SoftMin and soft-OR relaxations in Eqs. 10–12 make rule execution differentiable. As a result, symbolic forward chaining is not simply attached to a neural model as a post-hoc component; it is incorporated into the trainable computation graph.
Critical weaknesses
-
The VQ-based “Knowledge Regions” module introduces substantial complexity, but its empirical benefit appears limited and may come at the cost of reasoning stability. The stated motivation is to simulate clinicians with different clinical experiences. However, Table 7 shows that the simplest setting, , which removes the VQ mechanism, achieves 71.69% accuracy, while the default setting reaches 72.46%. This is an improvement of less than one percentage point. More importantly, increasing the number of regions to reduces test accuracy to 71.11% and substantially destabilizes the reasoning trace, with State F1 dropping to 84.26 and a large standard deviation of . Given the additional architectural overhead introduced by the VQ-VAE codebook in Eqs. 14–15, as well as the added hyperparameter sensitivity, the current evidence does not clearly justify the necessity of this module.
-
Training stability under STE is another significant concern. Although the end-to-end formulation is conceptually appealing, the DiReCT results suggest notable instability. For example, Table 3 reports a standard deviation of for the “No implication” ablation on DiReCT, while the hyperparameter sensitivity analysis in Table 9 shows variations as large as . These results suggest that the surrogate gradients induced by discrete query selection may have high variance, making the model sensitive to initialization, rule-bank coverage, or other training details.
Questions for authors
1.Given that and yield nearly identical test accuracy in Table 7, and that leads to a large drop in State F1 with high variance , what is the strict justification for the VQ module? At present, the VQ routing appears to introduce instability in the reasoning trace without clearly improving final diagnosis accuracy. Could the authors provide codebook utilization statistics, such as assignment perplexity or cluster occupancy, to show that the model learns distinct “doctor profiles” rather than using the codebook as a noisy regularizer?
-
How do the authors mitigate the high variance observed in STE-based training? The reported standard deviations on DiReCT are substantial, including in the hyperparameter sweeps and in the Table 3 ablations. Have the authors considered alternatives such as REINFORCE with a learned baseline, Gumbel-Softmax relaxation, or a curriculum over query length to stabilize discrete action learning?
-
The rule bank appears to be strictly frozen. How does the model behave when a patient presents symptoms that trigger no rules, or when activated rules are mutually conflicting? Since is updated through rule inference in Eq. 13, missing rules could leave the state matrix close to a zero-information state, which may cause the attention mechanism to fail when selecting the next useful inquiry. How does the model handle out-of-distribution cases or rule-coverage failures at inference time?
3 comments
Weakness 2 / Question 2
Q: How do the authors mitigate the high variance observed in STE-based training? The reported standard deviations on DiReCT are substantial...
A: MZ-4 and DDXPlus also use STE, yet their main-result standard deviations are small ( and percentage points, respectively). This suggests that STE does not invariably cause unstable training. We therefore consider DiReCT's small sample size and severe class sparsity ( training notes over diagnoses) likely major contributors to its variability. Its test set contains only cases, so a single case changes accuracy by percentage points.
On MZ-4 and DiReCT, inquiry receives two complementary training signals: diagnosis loss through STE and direct supervision from the differentiable process loss. These signals can be used jointly. DDXPlus has no process supervision () and uses the diagnosis-to-inquiry STE signal alone.
Q: Have the authors considered alternatives such as REINFORCE with a learned baseline, Gumbel-Softmax relaxation, or a curriculum over query length to stabilize discrete action learning?
A: We appreciate your suggestion and we implement all three discrete-action learning alternatives across five seeds: (1) Gumbel-Softmax (straight-through), (2) Plackett–Luce REINFORCE (for top- sampling without replacement ( in the current paper), using an action-preceding learned baseline), and (3) Query-Budget Curriculum (linearly annealing query horizon from to ):
The comparison is shown below (see revised draft, Appendix D, Table 14). Alternatives use five seeds.
| Dataset | Main model | Gumbel-ST | REINFORCE | Budget curriculum |
|---|---|---|---|---|
| MZ-4 | 72.46 ± 0.29 | 71.36 ± 0.43 | 70.67 ± 1.29 | 71.07 ± 0.38 |
| DDXPlus | 99.20 ± 0.80 | 95.84 ± 2.60 | 83.84 ± 7.28 | 84.12 ± 8.22 |
| DiReCT | 50.46 ± 2.89 | 37.22 ± 6.62 | 17.78 ± 4.33 | 42.22 ± 14.08 |
Overall, no alternative consistently improves both accuracy and stability across datasets.
Question 3
Q: How does the model behave when a patient presents symptoms that trigger no rules?
A: We discreate the work flow. The state is never zero-information, because the chief complaint initializes with observed signed facts. Attention queries come from occupied slots , so the model can always rank unread predicates from the observed facts alone. If no diagnostic rule fires, the logits are all zero, the prediction is uniform, and confidence stopping never triggers, so the model keeps asking until the budget is exhausted.
Q: How does the model handle mutually conflicting rules?
A: MIRROR handles conflicting rules through signed aggregation. For each predicate or diagnosis class, it first combines all rules supporting the positive conclusion and all rules supporting the negative conclusion using two separate weighted soft-OR operations. It then subtracts the negative support from the positive support to compute the implication value or diagnostic logit . For an empty rule set, we define . Conflicting rules therefore offset one another and reduce the magnitude of the final score. (Please refer to Eqs. 12 and 16.)
Q: How robust is the model empirically to rule-coverage failures, conflicts, or rule-bank perturbations?
A: Thanks for your question. We add some statistic results to further explain the questions (see revised draft, Appendix D, “Rule coverage, conflict, and perturbation,” and Table 15).
- (1) Unfired & conflicting rule statistics: Zero diagnostic rules fire in only of MZ-4 cases, of DDXPlus cases, and of DiReCT cases. On MZ-4, positive and negative rules for the same diagnosis co-fire in of test cases, yet accuracy on these conflicting cases () matches overall accuracy ().
- (2) Rule deletion and noise injection: We andomly deleting 10% / 30% / 50% of rules and injecting 10% random noisy rules. Mean diagnosis accuracy is shown below. Perturbations use five seeds:
| Dataset | Full model | Delete 10% | Delete 30% | Delete 50% | Inject 10% noisy rules |
|---|---|---|---|---|---|
| MZ-4 | |||||
| DDXPlus | |||||
| DiReCT |
Deleting more rules generally lowers accuracy. Injecting noisy rules changes MZ-4 accuracy only slightly, but lowers DDXPlus to and DiReCT to . Thus, robustness to rule noise varies across datasets, with DiReCT showing the largest degradation.
Future extension to OOD rule learning: While the rule bank is frozen in this study as we focus on behavior modeling, our MIRROR is easy to learn the rule structure, which is an advantage of our nerual symbolic design. Representing predicates in a shared continuous anchor space provides a natural foundation for differentiable Horn-clause structure learning, which we list as future work.
We warmly thank the reviewer for the positive assessment and the constructive questions.
Weakness 1 / Question 1
A: The VQ knowledge-region module is motivated by behavioral modeling of bounded, heterogeneous human knowledge rather than solely maximizing aggregate classification accuracy on unlabelled encounters. Its empirical behavior is explained directly by clinician-profile identifiability without doctor IDs:
Controlled synthetic results: To verify that the VQ module learns rule profiles rather than acting as a noisy regularizer, we simulated clinicians drawn from H⋆ ground-truth rule profiles and trained MIRROR without doctor IDs: When case context carries signal about clinician type, the VQ module recovers the latent clinician profiles without ID supervision (ARI 0.72 at H⋆=2, 0.74 at H⋆=3) and boosts accuracy by +4.5 and +6.2 points over H=1, closely approaching an ID-oracle. (Please see revised draft, Section 4.3)
A: Thanks for your suggestion. We add three metrics: assignment perplexity, active-code count, and diagnosis NMI. Under H=2, perplexity is the effective number of codes used (1 means complete collapse to one code; 2 means balanced use of both codes); Active-code count tests whether either code is unused; Diagnosis NMI measures whether routing merely reproduces the disease label (lower is better for ruling out trivial disease-based routing). Results across five seeds (see revised draft, Section 4.3 and Table 4):
Both profiles are active on every dataset and seed. MZ-4 and DiReCT use them nearly evenly; DDXPlus is more imbalanced but does not collapse to one code. The low diagnosis NMI (0.00–0.24) further shows that the profiles are not simply copies of disease labels. Together, these results show that the compact H=2 codebook is used non-trivially.