- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes to study neuroSymbolic models whose supplied background knowledge (a table mapping concept pairs to labels) differs from the intended , while leaving training labels intact. The proposed investigation spans three layers: compatibility, repair identifiability and neural recovery. Compatibility asks whether there is a concept remapping reconciling the supplied knowledge with the true knowledge . Repair identifiability refers to the possibility of finding the unique repair of the corrupted knowledge at minimum cost and, finally, neural recovery to whether the training actually recovers this repair. The authors define a certified edit margin , prove that for 2-slot addition over digits the nearest competing zero-loss table is edits distant from , and report MNIST/SVHN experiments showing that the repair is not always recovered by end-to-end training.
Critical strengths
The problem the paper addresses is interesting per se and understudied, relative to the reasoning-shortcut literature that assumes . The results in Table 2 are clear in this sense: they show how, under the permutation corruption of the knowledge, the label accuracy stays high while concept recovery is severely affected. The characterization of the minimum-repair in Theorem 3 is correct and tight (for the case of a sum between two digits). The paper also comes with an extensive reproducibility package.
Critical weaknesses
W1: Theorem 3 is the paper’s only non-trivial theoretical result and it already fails to carry over to the comparison task.
The guarantee it provides is addition (2 digits) specific; already for the case of comparison in Proposition 6 a single corrupted cell can create ties. So outside of addition, the framework provides no guarantee and everything falls back to CP-SAT that solves using anyway (the ground truth).
W2. Certificates have limited practical value.
They provide a quantitative measure of how different and are, but they need to be computed using , which is not available in practice. Given Eq.(5), stating that implies is the unique minimum-edit explanation follows by definition; the only substantive content is computing .
W3. Theory is narrow and trivially follows from the assumptions.
Theorem 3 is the main theoretical result, but it is very narrow in scope: it only covers the case of addition between two digits. There is no result beyond two inputs, n-ary programs and logic programs as DeepProbLog. Many results, like Proposition 1 and 2, are elementary derived from the assumptions and do not carry significant insight.
W4. Contribution 3 is not supported by the experimental design.
The main failure mode is attributable to the encoder choice, as Table 3 shows that SlotCNN recovers the intended knowledge 46/50 times on both MNIST and SVHN, while a SmallResNet succeeds only 10/50 times on SVHN. Then the surrogate recovery is measured only in terms of the objective in Eq. (6) (apart from an ABL adaptation in App E.5), and this objective is also not properly motivated nor justified. So general conclusions about the recovery under neural training are not supported by the experiments.
W5. Experiments are narrow both in the scope of the data (simple MNIST and SVHN datasets) and the knowledge (2 digits addition, comparison).
The paper mentions [1], and yet never considers more challenging tasks from that benchmark suite.
W6. Severe problems with the writing and presentation.
- , and are used at (071–077) before is defined (214–219).
- Several assertions are made without either a justification or a reference. As an example, consider the Jensen argument (178) and lines (233-234). These are just two of several others I found throughout the paper.
- Proposition 1 bundles together four different statements at once.
- The experimental section simply reads the data tables and provides little discussion about the meaning of the results.
W7. Related work is neither complete nor accurate.
Semantic Loss is cited as a reasoning shortcut mitigation, but it is not. Conversely, several mitigation strategies from [2] are not reported. Also the paragraph does not include a relevant method for addressing the issue in ABL as [3].
The paper picks an interesting problem, but the weaknesses in theoretical scope, applicability, experimental design, writing and presentation are major. I recommend rejection.
References
[1] Bortolotti, S., Marconato, E., Carraro, T., Morettin, P., van Krieken, E., Vergari, A., Teso, S., and Passerini, A. (2024). "A neuro-symbolic benchmark suite for concept quality and reasoning shortcuts." Advances in Neural Information Processing Systems 37 (Datasets and Benchmarks Track).
[2] Marconato, E., Teso, S., Vergari, A., and Passerini, A. (2023). "Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts." Advances in Neural Information Processing Systems 36.
[3] Huang, Y.-X., Dai, W.-Z., Cai, L.-W., Muggleton, S. H., and Jiang, Y. (2021). "Fast abductive learning by similarity-based consistency optimization." Advances in Neural Information Processing Systems 34.
Questions for authors
Have you considered extending the framework to more than 2 concept inputs? In particular, does the permutation argument underpinning Theorem 3 still hold for k > 2 slots?