- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a two-stage initialization method to improve the semantic alignment of learned intermediate concepts in Abductive Learning (ABL). In particular, the BASIL (Binding-Alignment for Semantic Initialization via Logic-consistency) approach reformulates ABL through an explicit semantic alignment comprising two parts: one abstraction — mapping inputs to logical constants — and one interpretation, which assigns pseudo-labels to concept predicates. More in details, BASIL carries out the following initialization stages:
- Stage I: trains a contrastive clustering model and assigns one logical constant per cluster ("strong binding");
- Stage II: searches for a bijective cluster-to-predicate mapping (interpretation) maximizing logic consistency with the underlying knowledge base. The search uses Gumbel-perturbed proposals and cross-entropy updates, followed by a final decoding step to assign pseudo-labels to the concept predicates.
The aligned initialisation can then be used to support learning in different abductive frameworks, including the original ABL, A3BL or DeepStochLog. Under an idealised "perfect strong binding" assumption in which the inputs are perfectly clustered as the ground truths, a Reasoning Shortcut (RS) problem can be expressed as a mapping between concepts that preserve the reasoning result. Bijective mappings define shortcuts called benign, and non-injective ones collapsing multiple concepts into one are called malignant.
The main theoretical results cover score separation under cluster errors (Theorem 2) and the correctness of the final decoding (Theorem 3). Experiments cover four CIFAR-10/MNIST arithmetic tasks (Addition, Decipher, Addition-Half and Addition-EvenOdd) and one synthetic symbolic task (Large-K). The results show that BASIL consistently improves the recovery of intermediate concepts with the intended semantics, up to benign relabelling, according to the reported metrics.
Critical strengths
The paper addresses a relevant and overlooked problem in the field of weakly supervised neuro-symbolic learning and it does so in a principled way. In particular, separating the abstraction from the interpretation in the initialization stage models explicitly where the ambiguity enters the learning process, which is an interesting perspective per se. I found the proposed method to be well motivated and structured: it plugs into the existing ABL framework easily without disrupting the original learning procedure. The theoretical results are sound and the proofs are clear. The baselines selected for the experiments are also appropriate, as the ablations are effective in isolating the contribution of each stage of the initialization on the full model, along with the comparison with the pretrained models.
Critical weaknesses
Please consider the following points.
- W1) Defining bijective shortcuts "benign" goes beyond the identifiability framework the paper cites. The effect compounds over the experimental analysis.
The paper treats any bijection on concept values preserving the knowledge labels as a benign shortcut and counts the recovery of the intended semantics up to a mapping as a success. However, [1] defines intended semantics in a different way. Their Definition 3.3 allows a permutation of concept variables and element-wise invertible relabellings of their values, but only when the inference layer undoes them. This is what guarantees ultimately the correct reasoning result. When the knowledge is fixed, as in the current formulation of the framework, the identity is the only intended solution. Also, the shift-by-five on the Addition-EvenOdd task preserves only the training support, confirming that the intended semantics is not recovered as the framework promises: as stated below Table 11, BASIL recovers the intended semantics 10/20 times under identity only evaluation. Please report OOD RA, CA and Success under identity only, using a single relabelling per run. This leads to the second weakness, that is a problem of how realistic the evaluation is.
- W2) The evaluation requires knowing in advance the reasoning shortcuts.
The evaluation for the core metrics 'Success', 'CA' and 'SSV' rests on the best case scenario defined over the set of benign shortcuts. For every task, the authors derive this set by exact search (App. D.1) or by hand (App. E.3). For a new task the set is unknown, it may be very large or impractical to compute in advance. On Kand-Logic from [2], for example, every permutation of colours, every permutation of shapes, and the shape↔colour swap would all count as benign. Even when the set is known, choosing among benign interpretations requires labels. For example, on Addition-EvenOdd, the protocol (App. E.1, l.1193–1198; l.348–349) applies both identity and shift-by-five to each run and keeps whichever scores higher against held-out sums and concept labels, separately for each metric. This is an oracle choice that a deployed model cannot make, and it is what turns 10/20 into the 20/20 reported in Fig. 2.
- W3) The experiments put little pressure on BASIL in terms of the shortcut reasoning.
Three out of the five tasks proposed do not admit malignant reasoning shortcuts. This is unexpected for a paper whose main contribution is reasoning shortcut mitigation. Also, while Addition-Half does admit a malignant collapse, Stage II excludes it by construction (bijective interpretation with k = |P|): collapse is prevented by a hard constraint and not avoided by learning. Whether it re-emerges downstream is barely tested: visual ABL training is short, and the failing DSL+BASIL-init run on Addition-Half (19/20, SSV 10.8 ± 19.9, Table 10) is not analysed. Conversely, the challenging tasks proposed in [2] like Kand-Logic and BDD-OIA, highly favour the use of few concepts only through a knowledge signal that is not only ambiguous but also prone to reintroduce the collapse the initialization is trying to avoid. The open question is whether downstream ABL training re-introduces collapse when the knowledge strongly favours it, and this is not tested. This is fundamental to prove that the proposed initialization is effective in practice, as the paper claims.
- W4) No in-the-wild, real-world evaluation is provided.
All visual tasks render digit concepts with MNIST or CIFAR-10 images, which are small, object-centred and low-resolution. It would noticeably strengthen the paper's claims if the authors provided a real-world, in-the-wild evaluation over a task prone to shortcut reasoning. I suggest the BDD-OIA task from [2] and, if not applicable, please explain which assumption prevents using BASIL for it.
- W5) The framework is formulated in a limited way and the consequences are not discussed.
The predictor assigns each input exactly one of k mutually exclusive unary predicates. However, this excludes objects with several attributes (such as shape and colour in Kand-Logic or multi-label concepts such as in BDD-OIA) and relational concepts learned from perception. Only the single-object restriction is acknowledged and the other consequences are not discussed.
- W6) The assumptions of Theorem 1 are stronger than intended and they do not match the experiments.
The second condition in Theorem 1 requires to preserve the feasibility of all the candidate groundings. This is stronger than what Definition 2 asks for and it fails to hold for shift-by-five on Addition-EvenOdd. For example, given the label 6, the pair (2,4) can be shifted to (7,9) and yet the sum 16 is not feasible. So Theorem 1 does not currently cover Addition-EvenOdd. For the same reason, calling these conditions "mild" is misleading. Finally, the core assumptions of the Theorem should be stated in the main text, as they are central to the paper's claims.
- W7) Stage I operates under the hidden assumption of knowing the class of invariances that are relevant to the task.
Stage I also builds in assumptions about which variations are irrelevant, through its augmentations. The CIFAR pipeline (Table 4) includes colour jitter and grayscale conversion, which would remove colour as a concept in tasks where colour matters.
- W8) The method relies on several hyperparameters and the paper does not provide a clear guidance on how to set them.
Stage II has , , , , and , on top of the Stage I settings. The sensitivity study reported in Figure 5 varies each parameter by only about around its default value, and appears to cause failures already (16/20 on Large-K). This is particularly concerning for the cases of real-world tasks with in-the-wild data, where the tuning of hyperparameters is not trivial. Also the paper does not explain how the defaults are chosen in the first place.
- W9) There is no systematic count of the number of reasoning shortcuts admitted by each task.
App. E.3 characterises shortcuts task by task, by hand. The central claim is that strong binding reduces ambiguity, and this would be much better supported by counting shortcuts. This is now standard practice in the RS literature, cf. [3, 4].
- W10) Presentation, positioning and related work are not polished enough.
The introduction to reasoning shortcuts is brief, and a recent survey [4] on the topic is missing. Strong binding is also closely related to cluster-separability assumptions in [5] and the similarity-based consistency in ABL [6]: these connections should be adequately discussed.
Overall, the proposed method is well structured in spirit, but there are major weaknesses (W1-W6) which need to be addressed before the paper meets the ICLR expected standards. I am open to discussing the points above with the authors and I am happy to raise my score if they can be addressed in a revision.
References
[1] S. Bortolotti, E. Marconato, P. Morettin, A. Passerini, S. Teso. Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lens. NeurIPS 2025.
[2] S. Bortolotti, E. Marconato, T. Carraro, P. Morettin, E. van Krieken, A. Vergari, S. Teso, A. Passerini. A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts (rsbench). NeurIPS 2024, Datasets and Benchmarks Track.
[3] E. Marconato, S. Teso, A. Vergari, A. Passerini. Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts. NeurIPS 2023.
[4] E. Marconato, S. Bortolotti, E. van Krieken, P. Morettin, E. Umili, A. Vergari, E. Tsamoura, A. Passerini, S. Teso. Symbol Grounding in Neuro-Symbolic AI: A Gentle Introduction to Reasoning Shortcuts. arXiv:2510.14538, 2025.
[5] L. Andolfi, E. Giunchiglia. Right for the Right Reasons: Avoiding Reasoning Shortcuts via Prototypical Neurosymbolic AI. NeurIPS 2025.
[6] Y.-X. Huang, W.-Z. Dai, L.-W. Cai, S. H. Muggleton, Y. Jiang. Fast Abductive Learning by Similarity-based Consistency Optimization. NeurIPS 2021.
Questions for authors
-
Can the framework express Kand-Logic? If so, how would you construct the clusters, how many bijections would be benign under your definition, and how would Stage I augmentations handle colour as a concept?
-
What happens if is unknown?
3 comments
Response to Reviewer aaWc (2/3)
Q3 (Weakness 6). Does Theorem 1 cover the EvenOdd shift? Which assumptions are needed?
A3. Theorem 1 does not cover the shift. The identifiability limit on EvenOdd holds independently of that theorem. As you point out, is a feasible correction for sum 6 and its image sums to 16. Section 3 now states the conditions for equality in distribution of the ABL trajectories. The runs share the abstraction and initial states. Relabeling must preserve correction loss and every candidate's feasibility, abduction including tie-breaking must respect it, and updates must be independent of predicate names.
The two namings give identical training observations. With no side information distinguishing the two worlds, any learner's probability of recovering the original names averages at most one-half across them (Proposition B.1). The namings disagree on 50 of 84 unseen pairs. On a uniform draw from those pairs, any predictor incurs expected error of at least 25/84 in at least one world (B.2). For the trajectory question, B.3 shows that shared updates give identical trajectory laws if the correction-target distributions agree at the states training actually visits.
Q4 (Weakness 3). Does downstream training undo the initialization? What happened in the unsuccessful DeepStochLog run?
A4. On the two original tasks that admit exact concept mergers, every initially successful ABL run was also successful after downstream training. The single DeepStochLog failure started with two digits already confused.
For ABL on Addition-Half, 18/18 initially aligned runs retained all five digits on the ID test after 20 update loops, and the other two recovered. On EvenOdd, BASIL scored 20/20 at both endpoints on the symmetry-aware OOD test, versus 0/20 at both endpoints with predicate reuse. Bijectivity rules out mergers during search, but only one of the 120 bijections on Half satisfies all seven training sums. Reasoning supervision selects among them, and downstream learning is unconstrained.
For DeepStochLog on Half, 18/18 initially aligned runs retained the five digits at epoch 20, and one of the other two recovered. In the final failure two Stage I clusters already had the same majority digit. In the first epoch ID reasoning accuracy rose from 49.0% to 88.0% while digit accuracy fell from 63.7% to 45.2%. The learner reinforced a merger that fit every training sum. Final success was 19/20 on both ID and OOD tests (Appendix F.2.1).
On Kand-Logic, aligning the same pretrained models raised mean test reasoning accuracy from 60.2% to 99.9% before any downstream update. All 20 models retained the nine categories at epoch 40 under the naming fixed at initialization. All 17 entropy-only controls reaching at least 99% test accuracy used just three labels (Appendix F.6).
Q5 (Question 1 and Weakness 7). How does BASIL ground Kand-Logic, handle its symmetries and preserve color?
A5. Stage I uses contrastive pretraining to group the released object crops into nine clusters. Stage II uses the training outcomes to align them to the nine shape–color categories. Summing the probability table along its axes gives the official reasoner shape and color probabilities. Stage I preserves color with mild geometric transformations and padding, omitting grayscale conversion and color jitter.
Permuting shape names, permuting color names and exchanging the two attributes gives global bijections. All 72 preserve the OR answer on every configuration. The released examples also admit 72 non-injective maps that copy one attribute into both outputs. Each fails somewhere on the full rule. BASIL's unrelabeled test reasoning accuracy was 100.0%. Joint concept accuracy was 98.6% under one validation-selected symmetry and 15.3% under the original names (0/20 original-name successes). The global symmetries explain why the choice of names does not change the reasoning answer (Appendices E.3 and F.6).
Response to Reviewer aaWc (3/3)
Q6 (Weaknesses 4 and 5 and Question 2). Can BASIL handle BDD-OIA, relations learned from perception, or an unknown vocabulary size?
A6. BDD-OIA's 21 binary concepts can co-occur. BASIL currently aligns mutually exclusive categories. We therefore did not evaluate BDD-OIA. A joint vocabulary could require up to categories. A multi-label output could represent the co-occurring concepts directly. Kand-Logic demonstrates composition of a small joint vocabulary under a supplied rule. Learning the relations themselves from raw scenes would also require grounding those relations (Appendix E.4).
When the knowledge base lists the elementary categories, it supplies . If is unknown, the current method cannot infer it. An extension could use reasoning outcomes to decide which groups in a finer partition should be merged.
Q7 (Weakness 8). Why does a temperature of 1.1 affect Large-K, and how should users choose the search parameters?
A7. At , is the high-temperature extreme of our sweep. The 10% increase from the default is a substantial push toward exploration. It cuts the maximum odds of a cluster first proposing one predicate over another from 451:1 to 259:1 (43%). In the single-mode limit those odds likewise fall from 316:1 to 187:1. The 15/20 benign Stage II outputs and 16/20 downstream successes were therefore obtained at a setting that sharply limits how strongly a cluster can favor its preferred predicate (Appendix F.5).
We recommend the shared defaults as a starting point. During development we chose , and together on Decipher and Large-K using semantic recovery, and retained and with 50,000 candidates. We transferred these settings to Kand-Logic without retuning Stage II. In Figure 5, 54 of the 60 displayed task–parameter points gave 20/20 downstream successes, with default runs reused across sweeps. At a fixed partition and budget, held-out outcomes can reveal losses in reasoning accuracy, although equal accuracy can conceal different concepts.
Q8 (Weakness 9). How much ambiguity does binding remove, and how many shared maps remain on each task?
A8. On Addition-Half, strong binding reduces 576 feasible occurrence-level assignments to three shared maps. Bijectivity leaves one. Take one copy of each of the seven training pairs. With 14 independent occurrences a digit can change its name from one pair to the next. Tying repeated occurrences forces a shared name, and two of the three surviving maps merge distinct digits.
Table 8 counts shared deterministic maps preserving the stated training support under perfect binding. Bijective counts include identity. Large-K counts apply to the evaluated relations.
| Task | Bijective | Non-injective |
|---|---|---|
| Addition | 1 | 0 |
| Decipher | 1 | 0 |
| Addition-Half | 1 | 2 |
| Addition-EvenOdd | 2 | 34 |
| Large-K | 0 | |
| Kand-Logic | 72 | 72 |
Q9 (Weakness 10). How does strong binding relate to cluster separability and similarity-based ABL?
A9. BASIL uses similarity to form groups and reasoning outcomes to name them before downstream training. Andolfi and Giunchiglia rely on separable concepts and use concept-labeled support examples to give prototypes their meanings. Huang et al.'s ABLSim uses similarity to select consistent abductions during training. BASIL pools reasoning constraints across each group to obtain a classifier that can initialize other learners. Section 1 illustrates shortcut collapse and cites the survey by Marconato et al. Appendix A discusses these connections and the role of concept-preserving augmentations.
We welcome further discussion of the evaluation and the scope of the theory.
Response to Reviewer aaWc (1/3)
We thank Reviewer aaWc for recognizing that we address an overlooked problem in weakly supervised neuro-symbolic learning and that separating abstraction from interpretation makes the source of ambiguity explicit. We also appreciate your assessment of the clarity of the proofs and of the ablations that isolate each stage's contribution.
Summary of the changes.
New or rewritten main-text passages and the titles of new appendix subsections are marked in blue.
Q1 (Weaknesses 1 and 2). What does Figure 2's 20/20 mean on EvenOdd, and what are the OOD results under the original digit names?
A1. For each learner, BASIL preserved all ten digit distinctions in 20/20 runs and recovered the original names in 10/20. All three improved on unseen sums under the original names. Figure 2 requires each digit's most frequent prediction to be correct under either identity or a global shift by five modulo ten. Both namings preserve all 16 training sums.
Original-name evaluation. Table 13 uses the identity for every example and metric. RA is reasoning accuracy, CA is digit accuracy, and SSV is the largest fraction of any digit predicted as one wrong digit (mean ± standard deviation over 20 runs).
ABL's raw OOD reasoning accuracy rises from 5.2% to 53.8%. Definition 3.3 of Bortolotti et al. allows permutations of concept variables and invertible relabelings of their values when the inference map compensates. Our rule is fixed. “Benign” means the bijection preserves concept distinctions and training outcomes. The EvenOdd shift can change unseen sums, which the original-name evaluation measures (Section 3). Bortolotti et al., Definition 3.3
Q2 (Weakness 2). Does BASIL need concept labels or the shortcut set? Does using one map per run change the OOD results?
A2. Stage II needs neither concept labels nor the shortcut set. For DeepStochLog with BASIL-init, fixing one map reproduced all four aligned metrics in every run. The original evaluation chose a naming per metric using held-out labels. In the replay, concept accuracy on the existing in-distribution split determined one map for all OOD predictions.
DeepStochLog obtained 20/20 successes, 80.6% RA, 89.6% CA and 13.7% SSV (Table 14). The same replay for A³BL gave 20/20, 89.4% CA and 14.1% SSV. Its saved outputs lack operand pairings for shifted RA, and the original ABL cohort lacks the per-example predictions for replay. Table 13 gives uncalibrated reasoning performance for all three.
Even the offline Success metric needs at most one candidate check given a labeled confusion matrix covering every class. Each row must have a unique most frequent prediction, distinct across rows. These determine the sole candidate, whose admissibility still has to be verified on the task support.