- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a novel framework for learning modular neural networks by learning how components within a structurally modular architecture should communicate. Taking inspiration from the mechanism of synchrony, they separate the information encoded in neural modules into content and phase, and use phase to restrict communication in a shared channel by having modules share their content modulated by phase. The authors test their method on two relational visual datasets (SQOOP and Sort-of-CLEVR) and compare their results with several baseline models, as well as performing ablations within their method.
Critical strengths
- This work introduces an incredibly creative and well justified approach to learning modular systems, and makes the reasoning behind this approach evident and easy to parse. The method presented clearly separates itself from the existing literature, and justifies each decision taken with a specific problem to solve. Furthermore, despite having little experience with phase based methods for communication, I was able to follow and understand the ideas and implementation of the method with little additional reading. The authors provided formal and diligent explanations for every step, making the reasoning easy to follow and a pleasure to read.
- The testing and particularly the ablation is quite exhaustive, with several combinations of the different parts of the method being evaluated in order to isolate contribution. The results are discussed in detail, and the authors provide justified explanations and conclusions for all empirical values. I especially commend the authors for prominently featuring the not so flattering Sort-of-CLEVR results and discussing them. This and the transparency of the training and testing regimes shows a commendable commitment to scientific integrity.
Critical weaknesses
- I feel the underperformance in the Sort-of-CLEVR task should be discussed in more detail. Though still performing comparably to the top models, the method does fall slightly short. The current explanation does not give a clear reasoning as to why this might be, and I think a deeper discussion would make for a stronger submission.
Questions for authors
Again, I commend you for a very interesting and entertaining work. I have a few questions:
- Regarding the weakness I mentioned, I would be happy to review my score with some additional explanation. In the paper you attribute this performance to the cost of using a single shared channel. However, if this were the case, one would expect that increasing d would help increase performance, but Figure 4 does not suggest this. Is there a particular difference between the baseline models and this method which would warrant worse performance? (This discussion could be added to appendix E for the sake of space and referenced in the main text).
- You mention that for specific baselines, you had to change the training hyperparameters for stability. Even so, the RelNet variance is very large, and clearly shows failed runs. Is this the case after tuning the hyperparameters, or does the plot include the previous version? Can you give some intuition as to why this model is so unstable to train?
Thank you for your generous feedback and for your appreciation of our architectural motivation and ablation design.
Why does increasing d not close the Sort-of-CLEVR gap? (W1, Q1) Please refer to our General Comment, which provides the detailed breakdown you requested. In short, the phase dimension d governs how many senders a module can distinguish, rather than how much information each read carries. Whatever d is set to, each module reads a single vector of dimension dm=4 per step, which explains why increasing d beyond 4 yields no further gains. Regarding the architectural differences from the baselines, SyncNet's head module never accesses the image feature map directly, and each module summarises its assigned region via a weighted spatial average before communicating through narrow dm-dimensional messages. By contrast, the strongest baselines attend directly to the unpooled spatial feature grid. As detailed in the General Comment, the subtype breakdown indicates that this bottleneck reduces fine geometric precision rather than the ability to aggregate across multiple objects, pointing to the read dimension dm as the primary limiting factor. Following your suggestion, we will add this full analysis to Appendix E and reference it in the main text of the revised manuscript.
RelNet variance and training instability (Q2). Figure 2 reports only the tuned configuration, meaning that the failed runs visible in the plot are those that remain even after hyperparameter tuning. In those unsuccessful runs, training accuracy stays at chance level and parameter gradients remain near zero throughout optimisation, which points to an optimisation failure rather than a capacity limitation. On SQOOP's 8×8 feature grid, RelNet aggregates over all 4,096 pairs of spatial cells equally, even though the correct answer depends on only a small subset of locations. Early in training, the informative gradient signal from the relevant object pair is heavily diluted by the remaining background pairs. Extending the training schedule and adjusting the learning rate help the model escape this plateau in most seeds, though not in every run.
One plausible explanation for why this pathology is much more severe on SQOOP than on Sort-of-CLEVR, although we have not tested it directly, lies in how objects are identified. In SQOOP, all objects share the same colour and must be distinguished by shape. To isolate the relevant pair of cells, the relational MLP must already recognise object shapes, yet the convolutional encoder only receives strong gradients to learn shape representations once the relational module attends to the right pair. Adjusting the grid resolution offers no easy escape: a coarser grid blurs the character shapes further, whereas a finer grid quadratically increases the number of cell pairs. On Sort-of-CLEVR, which uses a 5×5 grid with 625 pairs, RelNet trains reliably across seeds, likely because questions reference objects by distinct colours that are immediately linearly separable in the early convolutional features. Notably, when SyncNet is trained without its spatial preference, it suffers from the exact same failure mode on SQOOP. We will include this discussion in the revised paper to give readers clearer intuition for the baseline variance.
We hope these explanations clearly address both of your questions.