- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how convolutional neural networks learn global visual connectivity rules, with particular attention to the distinction between having access to relevant information and actually utilizing that information.
The authors introduce a minimal-pair evaluation framework in which single-pixel interventions outside a restricted observation window alter the connectivity between a seed and a query point. Size-matched placebo interventions are further employed to distinguish genuine connectivity-related responses from sensitivity to changes in connected-component size.
The experiments identify a three-stage progression in the studied LocalCNN: initial reliance on seed-position information, followed by sensitivity to distant structural changes that is largely associated with component removal size, and eventually connectivity-specific discrimination.
The authors also argue that path-conditioned AUC can produce a misleading impression of competence expanding from shorter to longer paths, whereas minimal-pair analyses reveal that nonlocal sensitivity can emerge across path lengths before becoming sufficiently strong to affect aggregate discrimination metrics.
The study includes controlled interventions, multiple random initializations, within-run training trajectories, regression analyses, and supplementary comparisons involving alternative neural architectures and input distributions.
Critical strengths
1. Clear research motivation and a carefully constructed evaluation methodology
The paper addresses a meaningful distinction between information accessibility and actual information utilization. The minimal-pair construction, together with restricted-access controls and size-matched placebo interventions, provides a relatively rigorous experimental framework for separating nonlocal sensitivity from a particular class of shortcut-based responses.
2. Coherent experimental analysis of learning dynamics
The experimental workflow is reasonably comprehensive within the primary setting. The authors combine multiple random initializations, checkpoint-level analyses, regression-based decomposition, and propagation-depth comparisons to support their observations.
The discrepancy between path-conditioned AUC and minimal-pair accuracy is particularly informative: the results provide useful evidence that weak but systematic responses can remain obscured by aggregate performance measurements.
Critical weaknesses
1. Limited methodological novelty and an insufficiently established general contribution
The proposed evaluation framework is carefully implemented, but its conceptual contribution reads as relatively narrow given the surrounding literature. Minimal-pair evaluation, controlled counterfactual interventions, and contrastive examples for separating target behavior from shortcut-based predictions have each been explored in prior work the paper itself acknowledges — including contrast sets, counterfactual data augmentation, and earlier studies of neural connectivity and algorithmic reasoning. The contribution is therefore best characterized as the particular combination of single-pixel connectivity interventions, restricted observation windows, and component-size-matched placebo controls on a specific spatial reasoning task. A useful experimental design, but one that does not clearly rise to a general new methodology or theoretical account.
A second interpretive gap concerns the inference from sensitivity to competence. Detecting a consistent directional response to an intervention is not necessarily equivalent to demonstrating reliable competence on the underlying task, and the discrepancy between minimal-pair accuracy and path-conditioned AUC is informative precisely because the two metrics measure different properties rather than providing competing estimates of the same capability.
Consequently, the methodological implications seem more limited than the broader framing suggests.
2. Insufficient experimental breadth and limited generality of the central learning-dynamics findings
The three-stage progression is established primarily on LocalCNN under a fixed 64 × 64 near-critical Bernoulli connectivity setup. Although U-Net, dilated CNN, recurrent CNN, Transformer, and MPNN variants are included, the principal minimal-pair analyses and detailed checkpoint-level trajectories are not systematically reapplied across these architectures. This is a material limitation: U-Net reaches low error with substantially fewer training examples, while recurrent models exhibit different dependencies on computational steps, so it remains unclear whether the reported progression is a general phenomenon or a LocalCNN-specific training-dynamics artifact.
Three concrete empirical caveats further narrow the generality claim:
- Specificity gap at 256k: pair accuracy of 0.759 coexists with branch specificity of only 0.560, indicating that much of the early response is not connectivity-specific; the later improvement at 1M is encouraging but does not on its own rule out alternative structural proxies.
- Selection effects in the branch-placebo analysis: the procedure operates on a qualifying subset of constructed pairs, and the resulting subset composition is not fully quantified.
- Trajectory and budget confounds: the within-run progression is demonstrated with only two training trajectories, and cross-pool-size comparisons involve different realized optimization budgets rather than controlled ablations.
I consider the empirical observations credible within the investigated configuration, but the experimental evidence does not yet establish that the identified progression is a broadly applicable characteristic of neural networks learning global visual rules.
Questions for authors
1. Methodological contribution and positioning
Could the authors clarify what they consider the primary methodological advance beyond existing minimal-pair, contrast-set, and counterfactual evaluation approaches?
In particular, is the intended contribution a reusable general evaluation framework, or a carefully controlled empirical diagnosis of one specific global connectivity task? The paper currently sits between these two readings, and making the positioning explicit would directly determine how reviewers should weigh the significance and generality claims.
Clarifying this distinction up front would also help subsequent work decide whether to adopt the protocol as-is or treat the current results as a case study.
2. Generality across architectures
Could the authors provide evidence that the sensitivity-before-connectivity progression replicates in at least one additional successful architecture — e.g. U-Net, dilated CNN, or recurrent CNN?
These variants are already listed in the experiments, but the decisive metrics (pair accuracy and branch specificity tracked across training checkpoints) appear to be reported only for the main LocalCNN configuration. Extending the same intervention-based trajectory analysis to one architecturally distinct baseline would substantially narrow the alternative explanation that the three-stage pattern is a LocalCNN-specific optimization artifact.
A minimal viable extension would be: same checkpoint grid + same qualifying-pair selection pipeline, even on a single held-out architecture.
3. Robustness of connectivity-specific discrimination
How stable are the branch-specificity conclusions under:
- alternative placebo constructions (beyond matching connected-component removal size), and
- additional controls for confounding topological proxies such as bottleneck geometry and path redundancy?
The 256k checkpoint already signals a non-trivial gap (pair accuracy 0.759 vs. branch specificity 0.560), and the branch-placebo subset is necessarily filtered. It would be useful to know whether (i) varying the candidate-selection procedure changes the magnitude of that gap, and (ii) adding a bottleneck- or redundancy-matched placebo condition shifts the later-stage specificity numbers.
This is the key test for whether the model is tracking connectivity itself versus a correlated structural feature of the generated graphs.
Dear reviewers,
The discussion phase is nearing its end. If you haven't already, could you please read the author's rebuttal and respond?
Thank you & kind regards,
Your AC