- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper builds primarily on the work of van Krieken et al. (2022), who observed that incorporating fuzzy logical constraints does not necessarily improve neural network training performance on semi-supervised MNIST and attributed this phenomenon to the gradient properties of different implication operators, particularly the Gödel and Goguen implications. Building on this work, the present paper applies the known derivatives of these operators to seven ALC axiom structures. It proves that, for certain right-quantified axioms, the two operators yield identical gradients with respect to role parameters, while for certain left-quantified axioms, there is a quantifiable difference in their first-order descent behavior. The paper refers to this analysis as "Routing". Furthermore, building on prior research into gradients vanishing and operator-related issues associated with softmax outputs, the paper introduces a "Survival" analysis to investigate, from the perspectives of gradients and loss functions, why certain logical operators can cause models to converge to incorrect solutions. As a methodological improvement, the paper replaces logical operators that tend to trap models in incorrect solutions with existing operators that have better gradient properties.
Critical strengths
- The research problem is reasonable, and the paper generally provides a fair distinction between its own contributions and prior work's.
- The experiments are comprehensive, and the results provide a coherent account that supports the paper's claims.
Critical weaknesses
- The main shortcut mitigation experiments in Section 5 are directly based on Survival. The authors do not demonstrate a complete pipeline in which Routing is first used to analyze a new task, followed by automatic operator selection and successful shortcut mitigation. Routing appears to be an independent theoretical contribution, but it has not yet been integrated as an essential algorithmic step in the paper's shortcut mitigation experiments.
- I personally find the derivations of Theorems 2 and 3 relatively straightforward, as they can be derived directly from van Krieken et al. (2022).
- Prior work from 2022 has already demonstrated that replacing operators can improve training performance. The methodological contribution of this paper still primarily relies on manually selecting and replacing operators, which limits its novelty.
- Writing Problem: a) The abstract begins by presenting the authors' own experimental findings, which may give the misleading impression that these findings reflect the established state of research. b) The paper uses fancy emphasis on its theoretical framing, while some of the theorems are relatively straightforward. The overall presentation lacks clarity.
Questions for authors
- Under the same training data and computational budget, can replacing the Gödel/Goguen operators with alternatives such as KD achieve both higher concept accuracy and higher logical constraint satisfaction rates, compared to retaining the Gödel/Goguen operators and adjusting the weights of the logic loss?
- Compared with the operator gradient analysis presented by van Krieken et al. (2022), can the proposed Routing × Survival framework predict which operator will perform best on a completely new neuro-symbolic learning task before training?
6 comments
Q1. With the same data and budget, does KD give higher concept accuracy and higher constraint satisfaction than Gödel/Goguen with a tuned weight?
Yes on both, by a small margin, with one exception (Goguen at 100 labels, where the intervals include zero). Each implication at its best weight, chosen on a held-out split; KD minus the other, in points, 95% paired intervals → App. L.1
| Labels | Concept vs. Gödel | Concept vs. Goguen | Constraint vs. Gödel | Constraint vs. Goguen |
|---|---|---|---|---|
| 100 | +4.4 [3.4, 5.5] | +0.4 [−0.3, 1.1] | +6.0 [4.8, 7.1] | +0.6 [−0.3, 1.6] |
| 300 | +1.8 [1.4, 2.3] | +0.9 [0.5, 1.3] | +3.5 [2.7, 4.3] | +1.7 [1.0, 2.5] |
| 3,000 | +0.25 [0.12, 0.38] | +0.24 [0.18, 0.30] | +0.45 [0.21, 0.68] | +0.53 [0.40, 0.66] |
At 10 labels KD is 10.3 and 9.9 concept points above Gödel and Goguen. Tuning the weight removes most of the gap; Section 5 now reports this.
The gap is large only where the cause is routing: on MNIST-Addition in clause form, no weight helps Gödel or Goguen, and KD at its worst weight is 46.7 points [45.6, 47.7] above their best with 10 labels (8.7 [8.0, 9.5] with 100). → App. L.2
Q2. Can the framework predict the best operator on a completely new task before training?
No. It predicts which operators fail and which form of a rule trains; it does not rank the remaining operators. The abstract and Section 6 now say so.
To test what it does predict, we added MNIST-Max (two images, labeled with the larger digit) and wrote six predictions before any run → App. L.2
| Prediction | Outcome |
|---|---|
| Gödel and Goguen in clause form equal supervision alone | Held: run for run (50.1% with 10 labels) |
| Łukasiewicz, KD, Reichenbach in clause form and all five in contrapositive form learn | Held: 97.8–97.9% with 10 labels, +47.7 points [46.7, 48.8] |
| These combinations coincide (two predictions: clause form among Łukasiewicz, KD, Reichenbach; contrapositive form among all five) | Held only with labels |
| SP with 10 labels ends lower | Held (65.7%), but it stayed above supervision alone, against our expectation |
| The Łukasiewicz t-norm without labels does not learn | Held (11.1%) |
Overall: three held fully, two only with labels, one in part (Section 6).
Other predictions:
| Prediction | Outcome | Where |
|---|---|---|
| Three new aggregation benchmarks | CelebA and bAbI as predicted; ABSA against | Table 12 |
| Tolerated weight ordered Gödel ≤ Goguen ≤ KD | Held on digit-sum, FashionMNIST, target sums 5 and 13 | App. L.1 |
| Contrapositive encoding shifts the trained Łukasiewicz–Gödel gap in four quantified structures | Held in two; within 0.004 in the other two | App. E.2 |
| Smoothing Gödel's branch indicator lets it escape | Failed: 13 of 15 runs leave the shortcut and all return | App. L.3 |
Thanks for the additional experiments.
I agree that the revised experiments provide stronger evidence for the proposed analysis. However, the routing theorems remain fairly direct consequences of known operator derivatives. The weight-tuning results also show that much of KD's advantage over Gödel/Goguen disappears with proper tuning.
Also, the authors have clarified that the framework cannot predict the best operator or provide an automatic selection procedure. Given this narrower scope, I still find the methodological novelty limited.
I will keep my original score of 2.
Thank you for the fair summary and for the two questions, which led to new experiments.
W1. Routing is not part of the shortcut pipeline.
Agreed for the submission. Routing and survival are two axes of one analysis, not two stages, and the paper proposes no automatic selection procedure; the abstract and Section 6 now say so. The revision adds experiments in which routing decides the outcome of the shortcut task:
W2. Thms. 2 and 3 (now 1 and 2) follow directly from van Krieken et al.
Their per-operator analysis does not contain the per-axiom statement: on which of the seven forms two implications give identical role gradients, and the exact difference on the others. This depends on where the role sits in the axiom and which quantifier branch is selected. We also report that Thm. 2 is first-order only: left-quantified axioms have no measurable effect on the trained gap (coefficient 0.000 [−0.021, 0.021]). → Section 3.1, App. H
W3. Operator replacement was already shown to help in 2022.
Yes; we do not claim the replacement itself. The additions are:
W4. Writing.