acceptodds
Under review as a conference paper at ICLR 2027

Aligning and Folding: What Networks Do Before They Learn

Abstract

Architectures are usually organized by name, which hides an elementary question: what are the reusable pieces from which they are assembled? We answer that a network is an alternation of two processes. An align exposes patterns on arbitrary supports in comparable coordinates; a fold combines responses and may identify distinctions. Convolutional networks, transformers, and state-space mixers are sub-families of this single class: residual blocks, a pre-norm transformer block, and the mixer of an official Mamba-130M agree with their translated programs to numerical precision on the order of in forward values and per-parameter gradients, and covering the state-space mechanism costs one additional align primitive. The class is built to answer two questions before training: whether a given architecture can realize a task within a declared budget, and which architecture in a predeclared family should be chosen. Two empirical principles make this precise. Information is pattern: a task is generated by finitely many admissible patterns of a label-independent grammar. Failure is cost: once an architecture can jointly expose those patterns, an exact realization of the task is reachable from some admissible initialization whenever the declared budgets suffice. Each has a stated falsifier. For diagnosis, a representation admits an unrestricted deterministic readout precisely when its fibers refine the label partition, and cross-label collisions are irreversible downstream. We define a capacity that counts, as a formal upper bound, the ways an architecture can transform the basis of the patterns it jointly exposes. Predictions made before training hold on controlled tasks, including a certified impossibility and a targeted repair. For selection, the same capacity and cost are computed for every member of a predeclared family without training candidates, returning a finite cost-Pareto set; the framework finds a vision transformer with 25% fewer parameters than ViT-Lite-7/4 that matches ViT-Lite-6/4, and, among inherited-weight subnets of AutoFormer's ImageNet search space, one with fewer parameters matching the reported accuracy of AutoFormer-T.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
deep network architecture, expressivity, architecture design, pre-training diagnosis, architecture selection, capacity, convolution, attention, state space models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NWXiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a framework to describe neural architectures as directed compositions of two operations: align (exposing patterns in shared coordinates) and fold (combining input and identifying their distinctions). Using this framework, the authors describe a range of neural architectures (CNN, Transformer, and State space mixer). The paper then presents two empirical principles, based on which it introduces a measure to characterize architectural flexibility before training. Combined with computational cost, this measure is used to diagnose task realizability and architectural selection.

Critical strengths
  • The authors present a unified framework to analyze the capacity of neural network architecture across CNNs, Transformers, and SSMs, allowing diagnosis before training. This is a useful tool for model design and model selection.
Critical weaknesses
  • The novelty and theoretical justification of the proposed combination remain insufficiently established. The align and fold framework resembles existing views of neural networks as compositions of feature extraction, transformations, and aggregation [1, 2]. While the unified formulation is interesting, it is unclear what new theoretical or practical capabilities it enables beyond existing frameworks.

  • It remains unclear why the capacity measure L should reliably predict task coverage or architecture suitability, as it is just an upper bound rather than the actual number of realizable transformations.

  • Direct comparisons with existing training-free architecture selection methods under matched search spaces and evaluation protocols are missing.

Ref:

[1] Lad et al. The Remarkable Robustness of LLMs: Stages of Inference?. NeurIPS 2025.

[2] Tuan and Tong. Analyzing spectral information of transformers. In Proceedings of the Korea University AI Research Workshop, 2025.

Questions for authors
  • What can the align-and-fold framework characterize or predict that existing approach (eg, architectural abstractions, expressivity measures, or structural analyses) cannot? Could the authors identify a concrete example?

  • Could the authors compare the proposed capacity measure L against simpler architecture-only statistics, as well as representative training-free NAS baselines, on the same search spaces? Such comparisons would help establish whether the proposed measure provides independent predictive value.

2 comments

Thank you for the constructive review.

We thank the reviewer for pointing us to [1] and [2]; both are excellent works. While both articles analyse the layers of a network after training, ours works before training: it decides which architectures are feasible for the task. We see the two lines as perfectly complementary, and would add both to the related work in the upcoming version.

W1 and Q1: the insight. Any architecture is all about whether the right amount of possibility can appear given a limited cost. Both are counted from one syntax shared by convolution, attention and state-space layers, so all architectures can be compared in one currency before training.

To show that our framework gives the true insight, we verify this on familiar families:

  • On residual blocks, our framework gives a rule that coincides with a known one: keep a nonlinearity between the two convolutions. At matched cost on CIFAR-100, the blocks it admits reach 64.1–66.4%, those it excludes 62.0–63.2% (Sec. 7.3).

  • On MobileLLM's nine 125M language models of equal size, every configuration our framework excludes trains below any of the admitted ones (42.3–43.3% against 43.9–44.8%).

  • On vision transformers, without training, it selects a subnet that already matches AutoFormer-T (75.35% against 75.3%) with 98% of its parameters; AutoFormer needed supernet training and evolutionary search to reach a comparable result.

For unfamiliar families and new assemblies, where no experience exists, LL gives a safe zone to build in (see W2).

W2: how much LL can do.

Within the framework, LL has one job: to give a safe zone. Above the LL of a model known to work, comparing costs becomes meaningful.

In a new analysis of NAS-Bench-201, every one of the 9,544 cells in the zone reaches at least 79.08% on CIFAR-10.

When a model fails, LL tells what to do: outside the zone, increase LL; inside it, add training budget (EP2, Sec. 7.2).

W3 and Q2. We directly compare our method against NASWOT, SynFlow, Zen and ZiCo on NAS-Bench-201 and -101, under the same reference and selection procedure:

  • On NAS-Bench-201, no proxy pick is both cheaper and more accurate than ours in any of the 36 settings. The cheaper picks (NASWOT, Zen) fall outside our safe zone and lose 3.7 to 10.1 points; the costlier ones (SynFlow, ZiCo) lie inside it and gain at most 3.0 points at 1.2 to 2.9× our parameters, and in 8 settings SynFlow's costlier pick is even less accurate.
  • On NAS-Bench-101, SynFlow and ZiCo pick cells with fewer parameters and higher accuracy (+2.7 and +1.2 points), but 45 and 27 layers deeper, and depth is a real cost in latency; NASWOT's pick gains 2.0 points at 1.8× the parameters, and Zen's is at chance (10.0%).
  • On the controlled tasks of Sec. 7.2, SynFlow scores a feasible and a provably infeasible architecture identically, and ZiCo prefers both broken models to the repaired one. Our framework separates every case successfully and even indicates the repair.
  • LL is about 1,300× cheaper to compute than SynFlow and 36,000× than ZiCo.
  • LL needs no seed, while changing only the seed flips 64 of 300 NASWOT decisions, 19 for ZiCo, 4 for SynFlow and 1 for Zen.

We will add this comparison and the NAS-Bench-201 analysis to the revised version.

Edited: the Zen numbers now follow the published Zen-NAS procedure (Gaussian re-initialization and residual removal).

AuthorsOpenReview ↗

Update on the revision

The revision is posted (see our general comment). We added both works to the related work (App. A.11). We also added the comparison and the NAS-Bench-201 analysis (App. A.12, Tables 13 and 14).

Choosing only by simple architecture-only statistics (parameters, MACs, depth) is worse than our method in every family we test, by up to 28 points on NAS-Bench-201 (Table 13).

We have now trained all 140 residual-block schemas on CIFAR-100 and CIFAR-10. Among blocks of equal cost, every block our method accepts is more accurate than every block it rejects (Table 12). We have also trained all 24 ViT-Lite configurations (Table 11). All four that our method accepts reach the accuracy of the reference model (within one standard deviation), and our method picks ViT-Lite-6/4.

Reviewer_ScLAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors come with a unified framework for convolution, transformer and state-mixer models for assessment of their ability to realize a given task and to choose optimal architecture from a given search space. They do so by using empirical principles (information is pattern and failure is cost) and defining combinatorial freedom of given model. This allows us to assess models from a given model size budget before training them (i.e. a training-free method). The authors evaluated their method by finding a model with similar accuracy on ImageNet as previous method in AutoFormer search space with (slightly) less parameters and by finding efficient ViT for CIFAR-10.

Critical strengths

The paper considers interesting question on how do different model size definitions (combinatorial freedom, number of parameters and number of operations) interact. Well explained and evaluated results in this direction are always good for the community as typically studies consider just one of the definitions neglecting the others.

Critical weaknesses

The evaluation is poor as the method finds mostly models on a small scale datasets where finding just a single model with competetive performance is a very small contribution as all the models are very easily trainable and such task is prune to target-leaking (i.e. it is easy to precompute the accuracies of all models and then adjust the method accordingly). For small scale tasks we are rather interested in correlation of the method with the target accuracies (which is typically done in other training-free neural architecture search studies). The only exception of large dataset is a model found for ImageNet classification, where however the improvement over Autoformer-T is very limited (the same performance with 2% less parameters).

The paper is very hard to follow. Some of the contributions are very vaguely formulated and described. The method is based on a definition of combinatorial freedom which is very poorly specified (Table 4 from Supplementary materials would deserve detailed explanation), now it looks rather arbitrarily defined scores for which we lack some insights and motivations on why the authors chose them the way they did.

Questions for authors

How does your method compare to existing training-free methods? You mention them as previous work but then you never empirically compare in a unified setting.

How does your method rank models within a specified search space (given e.g. by number of parameters budget) and how does this ranking correlate to the model accuracies? This correlation should be compared to existing training-free methods.

Can you explain in more detail how did you define the values in Table 4? This is needed so that the method is not arbitrary but rather based on some motivation (there is a large gap between the theory in your case and application of the theory to actually define LL).

4 comments

Thank you for the careful review. We understand the paper is hard to follow, and would like to explain in simple words here.

On W2 and Q3: what the paper says, and where Table 4 comes from

The main finding is a simple law: an architecture is all about whether the right amount of possibility can appear given a limited cost. Possibility here means the ways an architecture can combine the patterns it exposes. The paper makes both possibility and cost countable from the architecture's syntax, so that whether a candidate can realize a task, and which qualified candidate is the cheapest, are decided before training.

What is counted. At each stage, the channel responses, traced back to the input, are patterns. But there are too many possible patterns exposed. To grasp this, we notice that the patterns span the exposed pattern space, so we work on the basis of the space: each channel is treated as one basis element, and each input channel of a layer is one source for that layer.

Possibility grows only when a layer combines different sources; this is when it transforms the basis. Whatever acts inside a single source (spatial pooling, softmax reweighting, normalization) only changes that one basis element by itself, and thus won't create more possibilities, counted as 0. A combination of different sources takes one of two forms:

  • selection of one among nn alternatives: log⁡2n\log_2 n bits;
  • joint composition of mm independent sources, each taking part or not: 2m2^m possibilities, mm bits.

Summed over every layer FF and every scalar output qq, L(E)=∑F∑q∈out(F)ℓF(q).L(E)=\sum_{F}\sum_{q\in\mathrm{out}(F)}\ell_F(q).

Three examples in Table 4.

  • A ReLU passes or zeroes each scalar: a selection between two, 1 bit.
  • A residual addition of two independent branches allows four formal combinations (neither, first, second, both): 2 bits.
  • Pooling or softmax reweighting acts inside one channel map and combines no different sources: 0. For the same reason, two linear layers with nothing in between count as one, and a branch that only re-expresses another linearly is not a new source.

Every entry of Table 4 is one of these two rules applied to a primitive. Under the framework, convolution, attention and state-space layers are all unified, so they can be combined freely and still be counted in the same way. We are happy to go through any row here if you are interested.

On a family. The framework's guidance can be checked against experience. On the familiar residual-block family, it turns out to be equivalent to a well-known rule: keep a nonlinearity between the two convolutions.

On W1, Q1 and Q2

Comparison with training-free methods (Q1). We compared with NASWOT, SynFlow, Zen and ZiCo in one setting: all 15,625 cells of NAS-Bench-201 and the cells of NAS-Bench-101, the same reference for every measure, and the same selection procedure with each measure in place of LL.

  1. Selection. On NAS-Bench-201, no procedure beats ours on both cost and accuracy. On NAS-Bench-101, the proxy picks that use fewer parameters and score higher are 27–45 layers deeper, and depth is a cost.
  2. Cost. LL is read from the architecture alone: per candidate, about 1,300× cheaper than SynFlow and 36,000× cheaper than ZiCo.
  3. Determinism. LL has no seed. Changing only the seed flips 64 of 300 NASWOT admit/reject decisions.

We will add this comparison to the revised version.

Ranking (Q2). We use LL as a threshold: it tells us whether a model has enough possibility for the task. Among the models that pass, we rank by cost.

What we see, in every benchmark we report, is that the models passing the threshold land in a range of accuracy, and inside that range the differences follow cost. Two of them at the same cost differ little in the residual-block family: by at most 1.2 points on CIFAR-10 and 2.4 on CIFAR-100. In benchmark samples the spread at equal cost is larger (median 2.9 points on NAS-Bench-201, CIFAR-10).

Thus the framework makes cost comparison meaningful: every model left to compare already has enough possibility, so the cheaper one is the better choice under the declared cost.

Target leaking (W1). We fixed the thresholds and selection rules before training any candidate, and we trained every one of the 33 admissible residual schemas (Table 8). To leave no room for choosing what to report, we are now training the rest of the 140 schemas on both datasets, and all 24 ViT-Lite configurations. We will post the results here.

Scale (W1). The AutoFormer experiment was designed to show that, without any training, the framework accurately recovers what they found by supernet training and evolutionary search. Our evaluation also reaches larger scales: language models (891 decoder configurations; the 125M and 350M MobileLLM grids), the Mamba-130M mixer, matched to 10−1010^{-10}, and the NAS-Bench-201 and -101 spaces of the new comparison above.

AuthorsOpenReview ↗

Results of the promised experiments

The revision is posted (see our general comment). All 140 residual-block schemas are now trained on CIFAR-100 and CIFAR-10 (Table 12): at equal cost, every admitted schema scores above every excluded one on both datasets. All 24 ViT-Lite configurations are trained as well (Table 11): the four above the threshold reach the reference's accuracy within its seed spread, and the procedure returns ViT-Lite-6/4. The comparison with training-free methods is in Table 14, and the simpler selectors are in Table 13.

Thank you for your response.

The explanation of how the values in Table 4 are defined is not satisfactory in my point of view and the method, as it relies on those values, seems arbitrary without a clear explanation.

Therefore, I do not change my rating.

Thank you for the reply.

For clarity, here is the idea in one sentence: each operation contributes to LL if and only if it adds to the formal count of patterns that the network can detect jointly.

Every value in Table 4 follows from the two counting rules of Sec. 4.2.

App. A.9 "Reference capacity audit and calibration" contains an example of how things are put together on a standard ViT block.

We would be glad to clarify any specific step if you could point us to it.

Reviewer_sogCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes describing any neural architecture as alternating two operations: "align", which exposes patterns on some support in shared coordinates, and "fold", which combines responses and may collapse distinctions between them. Convolutions, transformer blocks and a Mamba mixer are shown to be instances, with numerically identical translations. The authors define a formal count L of how many ways a construction can transform the basis of the patterns it exposes, and propose screening candidates before training by requiring L to exceed a threshold calibrated on a reference model, subject to support, width and cost constraints. The fiber partition of a representation is then used to diagnose when an architecture has already discarded information the task needs. Experiments range from exhaustive certificates on 3×3 binary images and a cost-matched toy task, through an attempted falsification of the "failure is cost" principle, to residual-block variants on CIFAR-100, the ViT-Lite family on CIFAR-10, and subnets of AutoFormer's ImageNet search spaces.

I found the diagnosis much more convincing than the selection part. The controlled examples show clearly how an architectural choice can make a task impossible while every standard cost measure stays fixed. I am not convinced that L adds predictive value for selection beyond simpler rules, and my doubts come from the paper's own experiments.

Critical strengths
  1. The diagnosis of information loss is clean and well supported. Separating loss of task information (cross-label collisions) from a missing readout for information that is still present is a useful distinction, and Table 1 (512 images) reports collisions separately from whether an affine readout is feasible. The far-pair vs. local-pair experiment (Sec. 7.2) is well controlled: the two architectures match in parameters, MACs, depth and encoder, and differ only in which input pairs are exposed. Pairing an impossibility certificate with a constructive witness is much stronger than comparing training accuracy.
  2. The reporting is careful. The implementation audits check parameter gradients as well as forward values, and the authors report results that don't fit their story, including the failed Tiny-MLP budget extrapolation. Training all 33 admitted residual schemas with three seeds (Table 8) shows variation that the six-model comparison in Table 7 hides.
Critical weaknesses

1. L is not shown to predict whether an architecture can solve a task, and it is not compared with simpler selection rules. L counts formal combination labels. By construction it does not say whether the patterns a task needs are jointly realizable, and the paper's own controlled example shows this: under the stated counting rules, the far-pair and local-pair architectures have the same L and both reach full support span, yet only one can realize the task. A successful reference shows that one architecture with a given L works, not that a different architecture with a larger L has the right structure. The paper calls the screen an empirical prediction "within a declared family" but does not say which properties of a family make that prediction reasonable. That makes comparison with simpler rules essential, and it is missing. In the residual-block experiment the split coincides with a one-line rule (a nonlinearity between the two convolutions). On AutoFormer-B, the 163 subnets below, 799 between and 39 above the L thresholds have mean accuracies of 82.08%, 82.07% and 82.05%, and all 1,001 subnets span 0.56 points, so L is uninformative there. Cost-only and random selection under the same resource bounds, and the nonlinearity rule, can mostly be computed from tables the authors already have. A training-free NAS proxy from Appendix A.11 would also help place the method.

2. The reported selection wins depend on choices that were not fixed in advance.

  • Residual blocks: Table 8 lists five 3x3→1x1 schemas that pass the screen and tie on cost, spanning 64.08–66.43%. Appendix A.9 says the 66.43% block in Table 2 has "the highest mean", which suggests it was picked after accuracies were known. The four lowest admitted schemas (62.7–63.2%) also overlap the excluded blocks, so what counts as a successful screen needs stating.
  • ViT: the headline 2.77M-parameter configuration (Table 11) has L = 0.998 L_target, so it fails the method's own L ≥ L_target rule. The cheapest member that passes is the calibration reference ViT-Lite-6/4. With five of 24 priced configurations trained, the tolerance and the choice of which five to train carry the result. The ~25% saving in the abstract is relative to ViT-Lite-7/4 (93.30%); at matched accuracy the saving is ~12% relative to ViT-Lite-6/4.
  • EP2: the frozen budget rule failed on Tiny-MLP, and success is then reported at four times each optimizer's extrapolated budget. That shows more training helped, not that a sufficient budget can be declared in advance. The extrapolation function, step counts and optimizer settings for the 30-seed experiment are missing, and it is unclear what outcome would have counted as a counterexample rather than a calibration failure.
Questions for authors
  1. What selection rules were fixed in advance for the residual and ViT experiments, including any threshold tolerance, the cost objective, and how ties among equal-cost candidates are broken?
  2. How does the method compare with cost-only selection, and with the rule requiring a nonlinearity between the two convolutions, using the candidate results already available?
  3. What accuracy tolerance defines a successful selection, and what fraction of admitted candidates meets it?
  4. What independently specified budgets were used in the 30-seed EP2 experiment, and what outcome would have falsified the principle?
  5. Are the frozen protocols and registry records listed in Appendix A.14 in the supplement? For example, in the jigsaw experiment, which representation does the affine readout receive, and are the source images disjoint between training and evaluation? (clarification)
4 comments

Thank you for the insightful and detailed review. You read the diagnosis exactly as we intended. On selection, we answer with the comparisons you suggested and with new experiments.

In the framework, the certificates decide whether a structure can realize a task, LL marks a safe zone above a model known to work, and cost decides inside that zone. The far/local pair is a case for the certificates: equal LL, opposite verdicts, both proven. The selection experiments test the safe zone and the cost choice together, so we compare that procedure with the simpler ones you list.

W1 / Q2: simpler rules, on our own candidates. Each comparison replaces our admission rule by the one named, keeps the same cost bound, and picks the cheapest admitted candidate; numbers are the accuracy of our pick minus theirs.

  • Cost-only. Our procedure is higher in every family: residual blocks +1.6 (CIFAR-100) and +1.05 (CIFAR-10), ViT-Lite +0.4, AutoFormer +0.1 to +0.3, MobileLLM +0.6 and +1.6, NAS-Bench-201 +18 to +28, NAS-Bench-101 +0.5.
  • Random in budget, the expected accuracy of picking any candidate within the same cost bound: NAS-Bench-201 +2.0 to +3.4, residual blocks +2.5 and +0.74, AutoFormer-T space +1.5, AutoFormer-B space +0.17, ViT-Lite +0.03, MobileLLM 125M −0.01 and 350M −0.25, NAS-Bench-101 −3.0. Except on ViT-Lite and MobileLLM 350M, our pick also costs less than the average random draw on Par, MAC and depth. If the random pick is restricted to candidates that cost no more than ours, it scores 10–13 points lower on NAS-Bench-201, 1.0–1.5 lower on residual blocks, 0.1–0.2 lower on AutoFormer, and within 0.1 on NAS-Bench-101 and MobileLLM 350M.
  • The nonlinearity rule ("keep a nonlinearity between the two convolutions"). On all 140 schemas it admits exactly the set LL admits.
  • AutoFormer-B. All 1,001 subnets lie in the safe zone and span only 0.56 points, so LL has nothing to exclude there and the choice falls to cost, as designed.

Note that we regard agreeing with simple rules in familiar families as an advantage of the framework. Without any training, it arrives at a rule that practice reached only through long experience, and it gives the insight behind the rule: without a nonlinearity, the two convolutions compose into one linear fold. So for unfamiliar families, its guidance is promising.

Training-free proxies (A.11). NASWOT, SynFlow, Zen, and ZiCo:

  • On the 14 residual blocks, NASWOT's gate selects the same five schemas as LL, ZiCo's is 0.05 higher, SynFlow's and Zen's 1.5 lower.
  • On the controlled tasks of Sec. 7.2, SynFlow scores the far and local architectures identically, and ZiCo prefers both broken Drop models to the repaired one.
  • On all 15,625 NAS-Bench-201 cells, in 36 settings, no proxy pick is both cheaper and more accurate than ours: the cheaper ones (NASWOT, Zen) fall outside the safe zone and lose 3.7–10.1 points; the costlier ones (SynFlow, ZiCo) lie inside it and gain at most 3.0 at 1.2–2.9× our parameters. On NAS-Bench-101, SynFlow and ZiCo find cells with fewer parameters and +2.7 and +1.2 points, 45 and 27 layers deeper.

W2 and Q1: rules fixed in advance.

Residual blocks. Fixed before training: admission as L≥LtargetL\ge L_{\rm target} with no tolerance, a cost bound of Par and MAC at most the standard block's, terminal width k=64k=64, and the order used to pick the cheapest admitted candidate, (Dgeo,Par,MAC)(D_{\rm geo},\mathrm{Par},\mathrm{MAC}). No tie-break was fixed. On Table 2: the cheapest admitted cost holds five tied schemas, and Table 2 showed the best of them (66.43%). The other four score 64.08–65.31%, all above every excluded schema of the same cost (at most 63.22%). The revision reports the whole tie group.

To remove any room for choice, we trained the remaining 100 of the 140 schemas after submission, judging each by the submitted rules first. On CIFAR-100, three seeds each: in every group of identical Par, MAC and depth, every admitted schema scores above every excluded one, 360 pairs and no inversion. Across costs, the 33 admitted average 64.99% against 61.06% for the 47 excluded within the bound.

Edited: the Zen numbers now follow the published Zen-NAS procedure (Gaussian re-initialization and residual removal).

Q3: what counts as success. Success is defined first, and LL is then found for it. By EP1, a selection succeeds when the selected architecture exposes every pattern the task needs at one stage, so that the declared readout decides the task exactly. But natural tasks usually cannot reach that accuracy, so success is declared as an accuracy line. In the experiments in the paper, we draw the line in the most conservative way: a reference that meets it gives the threshold, so the line is the reference's accuracy (within its seed spread) and the threshold is its LL.

If the line is loosened by a tolerance tt, trading accuracy for cost, each tt gives a looser success line, Acc(E)≥Acc(reference)−t\mathrm{Acc}(E)\ge\mathrm{Acc}(\text{reference})-t, under which cheaper admitted candidates also count. To make this concrete, we just computed, over the 80 CIFAR-100 schemas within the cost bound (33 admitted, 47 excluded), the fraction of each that meets the line, for 101 values of tt from 0 to 10. The admitted fraction is strictly higher at every tt. At t=3t=3, 18% of the admitted reach the line and none of the excluded; at t=5t=5, 82% and 13%. The overlap you note occurs only across costs; at equal cost, as in our first comment, every admitted schema scores above every excluded one.

W2 and Q1: ViT-Lite. Nice catch. But we did not select the configuration after training. The abstract's claim comes from an early implementation bug that charged GELU 0 bits instead of 1. Under the correct rule, the configuration lies at 0.998 of the threshold and the procedure returns ViT-Lite-6/4 itself, as Sec. 7.3 reports. We will correct the abstract. All 24 configurations are being trained, and we will post them here.

Q4: EP2. EP2 is a basic assumption (postulate); we do not claim to prove it, and it can be refuted only at a budget declared sufficient in advance. EP2 does not predict the budget: the first MLP budget was a prediction from an extrapolation rule, never declared sufficient, and its miss is a prediction error of that rule. Before the 30-seed runs, we declared four times the extrapolated budget for each optimizer sufficient (Adam 260 and 336 steps, SGD 212 and 272, at 10 and 12 inputs), and fixed the test: failure of all 30 seeds would reject a success probability of at least 0.1, at level 0.930≈0.0420.9^{30}\approx0.042. SGD succeeded in 28/30 and 26/30 runs, Adam in 30/30 at both sizes. This test is only an example of how EP2 could be refuted; passing it does not prove EP2. The extrapolation formula and all optimizer settings will be added to the appendix.

Q5. To clarify: the readout receives 28 scalars computed from the 16 declared patterns: seam continuity (4), seam-gradient agreement (4), tile orientation (8, two per slot) and per-slot mean colour (12, three channels per slot). Training puzzles use the 50,000 CIFAR-10 training images and evaluation puzzles the 10,000 test images, so the source images are disjoint.

AuthorsOpenReview ↗

Update on the revision

The revision is posted (see our general comment). All 24 ViT-Lite configurations are now trained (Table 11): the four above the threshold reach the reference's accuracy within its seed spread, two of the 20 below it do, both at 0.95–0.998 of the threshold, and the procedure returns ViT-Lite-6/4. All 140 residual-block schemas are trained on CIFAR-100 and CIFAR-10 (Table 12), and at equal cost every admitted schema scores above every excluded one on both. The whole tie group is in Table 12, the corrected abstract on page 1, and the settings of the budget tests in App. A.9.

We thank the authors for the additional experiments and the explanations they have provided. They address several of our original concerns, and the results on all 140 residual schemas give much stronger evidence than the original subset. However, the nonlinearity rule admits exactly the same residual schemas as L, so this experiment doesn't establish an advantage over the simple rule. The paper should also state what accuracy is assigned to a method when several candidates tie on its selection objective: comparing the best member of one method's tied set with an average or arbitrary member of another wouldn't isolate the benefit of the admission rule. If all 1,001 AutoFormer-B candidates pass the L screen, removing that screen should leave the same selection, so it is unclear where the reported gain over cost-only selection comes from. Finally, a higher success fraction among admitted models shows that screening improves the candidate set, but not that admission predicts reference-level accuracy. Our remaining concern is how much benefit can be attributed specifically to L, so we keep our score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revision update

Changes are marked in blue.

  • Abstract and Sec. 7.3. Corrected the ViT-Lite sentence: the procedure recovers ViT-Lite-6/4 itself, as Sec. 7.3 already reported. Sec. 7.3 now states that all 24 configurations are trained.
  • Rule (ii) (App. A.6). Completed: a merged branch obtained from another branch through linear operations counted 0 alone is not independent. Note that we do it only for logical consistency and this does not change any conclusion in the paper.
  • New results (App. A.12):
    • all 140 residual-block schemas on CIFAR-100 and CIFAR-10, including the five tied at the cheapest cost (Table 12);
    • all 24 ViT-Lite configurations (Table 11);
    • the safe zone on NAS-Bench-201;
    • comparison with cost-only, random and equal-cost random selection (Table 13);
    • NASWOT, SynFlow, Zen and ZiCo on NAS-Bench-201, NAS-Bench-101 and the residual blocks, with the NAS-Bench setting (Table 14).
  • Budget tests (App. A.9). A corrected sentence on the blind prediction, and the extrapolation rule and optimizer settings of the EP2 experiment.
  • Related work (App. A.11). The two works suggested by Reviewer NWXi.

With all candidates now trained, the comparisons in our replies are recomputed in Tables 13 and 14. The Zen numbers now follow the published Zen-NAS procedure, and our replies are edited accordingly.

The authors have posted their responses to the reviews. Could I please encourage the reviewers to read these responses and reply to the authors.

AC

AuthorsOpenReview ↗

Closing remarks

In our framework, the quality of an architecture comes down to one question: can it offer the right amount of possibility within a limited cost? The framework counts both from the architecture's syntax, and answers the question before training. It also gives one language for convolution, attention, state-space layers, and even for architectures that no one has built yet. This language serves both to diagnose why an architecture fails a task and to construct candidates that can succeed.

Following the suggestions of the reviewers, our revision further shows that selection by the framework is reliable, especially on residual blocks.

We truly believe in the value and the potential of this work. We hope to see the community put our framework to practical use soon.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.