- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper proposes a neural-symbolic architecture, Frozen Holographic Memory (FHM), for compositional reasoning. Building on the holographic representation framework of Plate (1994), FHM uses fixed algebraic operations to store, update, and compose facts, while relying on lightweight neural controllers to learn task-specific decisions. Across three state-tracking and compositional reasoning benchmarks (Possession, CLUTRR, and NoRA), FHM demonstrates strong generalization performance when evaluated on reasoning depths and relation compositions beyond those encountered during training, outperforming the compared transformer-based baselines.
Critical strengths
- The experimental results are promising. FHM demonstrates strong extrapolation performance on possession chains and CLUTRR relation compositions beyond the training depths, and substantially outperforms the Edge Transformer on the evaluated NoRA splits, particularly on off-path reasoning.
Critical weaknesses
-
Limited baseline comparisons. The compared baselines are limited, making it unclear whether the strong empirical results come from the proposed holographic memory design or primarily from the predefined symbolic reasoning operations. The authors mainly compare FHM against transformer-based models, while prior work has shown that incorporating explicit structural or algebraic inductive biases can substantially improve compositional generalization [1,2,3]. I believe it is important to include comparisons with other symbolic or neuro-symbolic approaches, particularly a symbolic baseline using the same reasoning procedures but without the holographic memory representation.
-
Heavy reliance on task-specific design. The proposed FHM architecture heavily relies on task-specific knowledge and manually designed reasoning procedures, which raises concerns about its applicability to new tasks. For example, the rule templates used for NoRA are selected based on the rule structures provided by the benchmark, while CLUTRR relies on predefined generation-count operations and relation composition procedures. In my view, a major limitation of such approaches is their dependence on domain-specific structural priors, which may not be readily available in realistic applications. The paper lacks a discussion of this limitation and how FHM could be applied to tasks where the underlying reasoning structures are unknown.
-
Presentation needs to be improved. Section 2 is difficult to follow, partly because substantial space is devoted to introducing existing holographic memory operations. I would suggest more clearly separating the established algebra from Plate 1994 from the novel design choices of FHM, and adding an overview figure illustrating the complete reasoning pipeline.
[1] Herzig, J., & Berant, J. (2021, August). Span-based semantic parsing for compositional generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 908-921).
[2] Yao, Y., & Koller, A. (2022, December). Structural generalization is hard for sequence-to-sequence models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 5048-5062).
[3] Liu, C., An, S., Lin, Z., Liu, Q., Chen, B., Lou, J. G., ... & Zhang, D. (2021, August). Learning algebraic recombination for compositional generalization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (pp. 1129-1144).
Questions for authors
Do the authors have any insights into how FHM or similar symbolic methods could be applied to more realistic state-tracking or compositional reasoning tasks, where task-specific reasoning structures and operations may not be known in advance?
Response to Reviewer 599y
Thank you for your time and for highlighting FHM's off-path reasoning on NoRA.
"The compared baselines are limited, making it unclear whether the strong empirical results come from the proposed holographic memory design or primarily from the predefined symbolic reasoning operations."
The symbolic operations produce the scores, while the holographic memory trains the decisions on possession and CLUTRR. Part 2/5 reports the baseline you requested, where Python dictionaries replace FHM's stores and the reasoning procedures stay unchanged. On every benchmark, this baseline executes FHM's trained decisions and scores the same as FHM. With the dictionary replacing the store during training, no gradient path exists, so we use REINFORCE. The controller then learns to write on an observation verb in some possession seeds and does not fit CLUTRR's net generation count in four seeds of five.
"The authors mainly compare FHM against transformer-based models, while prior work has shown that incorporating explicit structural or algebraic inductive biases can substantially improve compositional generalization [1,2,3]."
Besides the symbolic baseline, we added two baselines that are not transformers. DeltaNet fits its training regime on possession and reaches 98.7% at depth 40 but 84.8–89.7% across depths with 100 items. An LSTM does not fit (Part 3/5). We did not run the methods of the three papers you cite, which study semantic parsing. The revised related work discusses them (Part 5/5).
"For example, the rule templates used for NoRA are selected based on the rule structures provided by the benchmark, while CLUTRR relies on predefined generation-count operations and relation composition procedures."
The revised experiments drop NoRA's rule templates and train a controller on CLUTRR's products. In the submission, NoRA's candidate if-parts took their shapes from the benchmark's rules. Now, controllers decide which if-parts of one to three facts to keep as rules. On CLUTRR, a controller learns each pair's product from the answer alone, whereas the submission wrote it into the product store from the training questions. The net generation count still runs on a predefined count store, but its controller chooses among 19 offsets, up from five. Learning these decisions costs accuracy: CLUTRR's full label falls from 98.3% to 69.9% at length 10, while NoRA scores lower on Test-BL, Test-OPEC and Test-W's hard questions (Parts 3/5 and 4/5).
"I would suggest more clearly separating the established algebra from Plate 1994 from the novel design choices of FHM, and adding an overview figure illustrating the complete reasoning pipeline."
The revised Section 2 separates Plate's algebra from what FHM adds and includes an overview figure of the full pipeline (Part 5/5).
"Do the authors have any insights into how FHM or similar symbolic methods could be applied to more realistic state-tracking or compositional reasoning tasks, where task-specific reasoning structures and operations may not be known in advance?"
The revised experiments are a step toward such tasks. Controllers now learn NoRA's rules and CLUTRR's products, whereas before, FHM induced the rules in its rule store and wrote the products from the training questions. The store's operations and each task's procedure are still fixed in advance. Applying FHM where those are unknown is outside this paper's scope, as the new Limitations section states (Part 1/5).