- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents a framework to describe neural architectures as directed compositions of two operations: align (exposing patterns in shared coordinates) and fold (combining input and identifying their distinctions). Using this framework, the authors describe a range of neural architectures (CNN, Transformer, and State space mixer). The paper then presents two empirical principles, based on which it introduces a measure to characterize architectural flexibility before training. Combined with computational cost, this measure is used to diagnose task realizability and architectural selection.
Critical strengths
- The authors present a unified framework to analyze the capacity of neural network architecture across CNNs, Transformers, and SSMs, allowing diagnosis before training. This is a useful tool for model design and model selection.
Critical weaknesses
-
The novelty and theoretical justification of the proposed combination remain insufficiently established. The align and fold framework resembles existing views of neural networks as compositions of feature extraction, transformations, and aggregation [1, 2]. While the unified formulation is interesting, it is unclear what new theoretical or practical capabilities it enables beyond existing frameworks.
-
It remains unclear why the capacity measure L should reliably predict task coverage or architecture suitability, as it is just an upper bound rather than the actual number of realizable transformations.
-
Direct comparisons with existing training-free architecture selection methods under matched search spaces and evaluation protocols are missing.
Ref:
[1] Lad et al. The Remarkable Robustness of LLMs: Stages of Inference?. NeurIPS 2025.
[2] Tuan and Tong. Analyzing spectral information of transformers. In Proceedings of the Korea University AI Research Workshop, 2025.
Questions for authors
-
What can the align-and-fold framework characterize or predict that existing approach (eg, architectural abstractions, expressivity measures, or structural analyses) cannot? Could the authors identify a concrete example?
-
Could the authors compare the proposed capacity measure L against simpler architecture-only statistics, as well as representative training-free NAS baselines, on the same search spaces? Such comparisons would help establish whether the proposed measure provides independent predictive value.
2 comments
Update on the revision
The revision is posted (see our general comment). We added both works to the related work (App. A.11). We also added the comparison and the NAS-Bench-201 analysis (App. A.12, Tables 13 and 14).
Choosing only by simple architecture-only statistics (parameters, MACs, depth) is worse than our method in every family we test, by up to 28 points on NAS-Bench-201 (Table 13).
We have now trained all 140 residual-block schemas on CIFAR-100 and CIFAR-10. Among blocks of equal cost, every block our method accepts is more accurate than every block it rejects (Table 12). We have also trained all 24 ViT-Lite configurations (Table 11). All four that our method accepts reach the accuracy of the reference model (within one standard deviation), and our method picks ViT-Lite-6/4.
Thank you for the constructive review.
We thank the reviewer for pointing us to [1] and [2]; both are excellent works. While both articles analyse the layers of a network after training, ours works before training: it decides which architectures are feasible for the task. We see the two lines as perfectly complementary, and would add both to the related work in the upcoming version.
W1 and Q1: the insight. Any architecture is all about whether the right amount of possibility can appear given a limited cost. Both are counted from one syntax shared by convolution, attention and state-space layers, so all architectures can be compared in one currency before training.
To show that our framework gives the true insight, we verify this on familiar families:
On residual blocks, our framework gives a rule that coincides with a known one: keep a nonlinearity between the two convolutions. At matched cost on CIFAR-100, the blocks it admits reach 64.1–66.4%, those it excludes 62.0–63.2% (Sec. 7.3).
On MobileLLM's nine 125M language models of equal size, every configuration our framework excludes trains below any of the admitted ones (42.3–43.3% against 43.9–44.8%).
On vision transformers, without training, it selects a subnet that already matches AutoFormer-T (75.35% against 75.3%) with 98% of its parameters; AutoFormer needed supernet training and evolutionary search to reach a comparable result.
For unfamiliar families and new assemblies, where no experience exists, L gives a safe zone to build in (see W2).
W2: how much L can do.
Within the framework, L has one job: to give a safe zone. Above the L of a model known to work, comparing costs becomes meaningful.
In a new analysis of NAS-Bench-201, every one of the 9,544 cells in the zone reaches at least 79.08% on CIFAR-10.
When a model fails, L tells what to do: outside the zone, increase L; inside it, add training budget (EP2, Sec. 7.2).
W3 and Q2. We directly compare our method against NASWOT, SynFlow, Zen and ZiCo on NAS-Bench-201 and -101, under the same reference and selection procedure:
We will add this comparison and the NAS-Bench-201 analysis to the revised version.
Edited: the Zen numbers now follow the published Zen-NAS procedure (Gaussian re-initialization and residual removal).