- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies inference about interacting latent variables in Cognitive Gridworld. It defines Factorization Regret (FR) as the KL divergence from the joint Bayesian observer's goal-variable posterior to the naive observer's. In Experiment 1, networks receive the true interactions and are trained against Bayesian posteriors; FR helps account for differences in accuracy and calibration between fully trained networks and networks with frozen recurrent weights. Representation Classification Chains (RCCs) then learn interaction embeddings from binary outcome feedback and observation prediction, and generalize to unseen combinations of known variables. Finally, a controller trained in the learned generative model achieves control without interacting with the target environment during controller training.
Critical strengths
-
S1. Cognitive Gridworld allows a precise analysis of inference errors. For the small contexts used in the experiments, exact Bayesian posteriors provide a reference for separating observational uncertainty from approximation error (Secs. 3–4.1, pp. 3–4). The inference tasks exclude exploration and planning, so the analysis can focus on how agents use latent interactions. The same setting also supports controlled tests on unseen variable combinations (Fig. 5, p. 6).
-
S2. The network results make a useful case for FR as an inference diagnostic. With Bayesian posterior supervision in Experiment 1, fully trained networks approach Joint Bayes performance, while networks with frozen recurrent weights resemble Naive Bayes in both performance and overconfidence (Figs. 3–4, pp. 5–6; Fig. 11, p. 22). The performance gap persists when controlling for the number of trainable parameters and when ESNs output updates to joint beliefs (Fig. 9, p. 21). Even at comparable mean accuracy, the networks differ in the correlation between FR and accuracy (Fig. 15d–e, p. 28). This gives information that average accuracy alone would miss.
-
S3. RCC shows that interactions can be learned from the available feedback and used in new combinations. Without access to the true interaction parameters, it learns from binary outcomes and observation prediction, then succeeds on unseen variable combinations (Sec. 4.3, Fig. 6c, p. 7). Test variables appear during training as non-goal variables, at most one per episode; their combinations and use as goals are held out (p. 6, lines 294–298). This is an existence proof under the specified architecture and feedback conditions.
Critical weaknesses
- W1. I find the evidence connecting FR to learning compositional representations less direct than the evidence for its value as an inference diagnostic. The Discussion presents RCC as an existence proof of learnability (p. 8, lines 422–424), then suggests that greater FR creates stronger pressure to develop compositional representations (p. 9, lines 469–472). The RCC experiment does not directly test this proposed relationship. Figure 12 (p. 23) shows that generalization depends on the total number of available variables, but does not relate that dependence to FR. This leaves open how far FR explains the acquisition of transferable representations, beyond the inference benefit of using interactions (Q1–Q2).
Questions for authors
-
Q1 (W1). What theoretical or empirical evidence supports the suggestion that greater FR creates stronger pressure to develop compositional representations? How, if at all, does this account for the dependence on the number of variables in Fig. 12?
-
Q2 (W1). What do the RCC experiments establish about learned compositional structure?
-
Q2(a). The environment and RCC both use shared embeddings and a specified pairwise interaction rule (p. 6, lines 289–298; p. 23, lines 1194–1198). Figure 6c shows that learning within this structure supports transfer to unseen combinations. Is there evidence linking the acquisition of these embeddings to FR beyond the improvement over Naive Bayes in Fig. 6c?
-
Q2(b). The main text cites Fig. 13 as evidence that routing embedding gradients through the Classifier impairs learning (p. 7, lines 339–340). The appendix tests this in a simplified setting with ground-truth latent realizations and architectural modifications (p. 24, lines 1242–1250). What can this comparison tell us about the Generator's role in learning representations under the original feedback conditions?
-
2 comments
Thank you for addressing my questions. The new Fig. 13 resolves Q2(b): the gradient routes are now compared under the original feedback conditions. I also found the analysis of the generative model's errors in Fig. 14 helpful, along with the explanation in Section B.5 of how transfer depends on the number of variables. Treating that dependence separately from FR answers this part of Q1. Together with the clearer scope, these results address much of W1 (p.s. the additional experiments support the authors’ hypothesis, but do not directly show that higher FR makes it easier to learn transferable embeddings.) and provide the additional evidence requested in Q2(a).
Thus I have raised my score to reflect these improvements.
Thank you for your careful reading of the main text and supplementary analyses. Your summary describes the scope of the work as we intended it, and your questions about representation learning happen to have been our research focus since the initial submission. In the revised PDF, former Figs. 12 and 15 are now Figs. 15 and 19.
W1. “I find the evidence connecting FR to learning compositional representations less direct than the evidence for its value as an inference diagnostic.”
We agree. FR is a statistic of the environment, not of a network; as you put it, an inference diagnostic: it identifies when latent structure is non-trivial. We have (i) emphasized FR’s scope (Discussion, lines 444–459) and (ii) run more focused analyses and experiments on how FR shapes learning and generalization (B.1–B.2; new Figs. 14, 17 and 18; revised Figs. 19–20; three supplementary animations, see the general comment).
Q1. “What … evidence supports the suggestion that greater FR creates stronger pressure to develop compositional representations? How … does this account for the dependence on the number of variables in Fig. 12 [now Fig. 15]?”
In theory, FR bounds the accuracy lost by ignoring interactions (B.3). Practical applications are outside our scope, but a new analysis and a new experiment show its potential usefulness. Fig. 17: FR predicts the order and speed in which regions of latent space are learned: low-FR regions rapidly, high-FR regions slowly and later. Fig. 18: Learning from high-FR experience transfers immediately to low-FR settings, whereas the reverse transfer is slow (animated in
Extrapolation_from_FR.gif; B.1).FR does not account for the ∣S∣ dependence of transfer (see the new B.5 and Fig. 23). That said, the ∣S∣-dependent generalization of a Generator trained on goal values alone, or on Naive Bayes classifications, drops with the FR of the testing environment (new Fig. 14). The ∣S∣ at which transfer begins also depends on the learner: an improved Generator architecture lowers it from 500 to 250 for the RCC (Fig. 15), though still not quite as low as for a Generator trained directly on Joint Bayes classifications (Fig. 14c).
Q2(a). “Is there evidence linking the acquisition of these embeddings to FR beyond the improvement over Naive Bayes in Fig. 6c?”
A Generator (and thus its embeddings) trained on goal realizations alone, or on the Naive observer’s actions, has a generalization error that grows with the FR of the novel environment (Fig. 14a–b). This dependence disappears when the Generator is conditioned on the RCC’s Classifier, which learns how latent variables interact (Fig. 14d). In summary, whether the embeddings’ generalization error is invariant to FR depends on whether the Generator is conditioned (during training) on actions from a policy that accounts for how latent variables interact.
Q2(b). “What can this comparison tell us about the Generator’s role in learning representations under the original feedback conditions?”
The original Fig. 13 removed the challenge of classification, largely to speed up data collection: training the Classifier with naturalistic feedback takes very many episodes, whereas a Generator conditioned on optimal actions learns extremely fast. This let us iterate faster and ultimately discover improvements to the Generator architecture (having repeated several experiments with it, we updated Fig. 15; see Q1).
We realize that introducing a new gradient routing and a new feedback condition at once in the original Fig. 13 created confusion, and that the resulting hyper-efficient Generator training oversimplified the comparison with the Classifier-routed gradient. The new Fig. 13 trains RCCs with naturalistic feedback, and for much longer: Classifier-routed gradients can learn beyond Naive Bayes, but much less reliably, with dramatic variance across seeds since learning is all or none. Our conclusion on the Generator’s role in learning transferable representations under naturalistic feedback is therefore: it dramatically boosts training efficiency, and makes classifier accuracy a heavily limiting factor (Fig. 14c–d). Its role is equally important for control, as internal generation (Fig. 7) avoids relearning action values from scratch whenever the value function, the action space or the relevant latent space changes. The Generator thus speeds up both representation learning and behavioral adaptation to new objectives.
Future work could use a high-dimensional observation space, where generative models are particularly effective (e.g., Ha & Schmidhuber, 2018; Hafner et al., 2025), to emphasize the Generator’s contribution. Here, we aimed to abstract away all structure from the observation space, reducing compositionality to simple and explicit vector products, and to avoid the confound of observations containing structure that is easier to learn with a generative model’s inductive biases.