acceptodds
Under review as a conference paper at ICLR 2027

When is a Representation Sufficient? Probe -Information for Neural Architecture Search

Abstract

When is a neural representation sufficient for a task? A representation Z is sufficient if I(X;Y | Z)=0, meaning no task-relevant information remains in the input. This provides a natural criterion for deciding when additional model capacity is unnecessary. However, estimating such information-theoretic quantities in high-dimensional neural networks is intractable in practice. We introduce V-NAS, a progressive architecture search method that replaces intractable Shannon information with probe-based V-information, a directly computable measure of extractable predictive information. Using the cross-entropy of a restricted probe, our method determines whether increasing width or depth yields meaningful gains, enabling principled capacity control. Across multiple datasets and architecture families, V-NAS discovers compact architectures that match the performance of significantly larger baselines at substantially lower cost. These results demonstrate that representation sufficiency can serve as a practical and interpretable foundation for neural architecture design.

Primary area:
learning theory
Keywords:
Representation Learning, Neural Architecture Search, V-Information, Mutual Information

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zu2uOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper discussed the approach of using Shannon-mutual information as a criteria to expand width/depth of a neural network. This information has several limitations like being difficult to estimated in practice hence the authors proposed v-NAS, based on v-Information (Xu et al. 2020.) The main practical implementation is to limit the classifier each layer as linear functions, train them and get useful information from the CE loss of this classifier, to tell whether the width/depth is saturated. This also a general progressive architecture search method.

Experiments are conducted on several datasets on CIFAR and ImageNet.

Critical strengths

The usage of v-information into the arch search/parameter tuning works is interesting. To my best knowledge, this is prior NAS work that tries to approach the problem from such a perspective.

Critical weaknesses

It remains questionable that in 2026, doing NAS works under a fixed search dimension (e.g., width or depth) is still meaningful. In practice, this is the local optimal solution which likely cannot generalize to the global one, especially given the large model complexity people generally feel interested now. Even if we do not consider the general scaling law literature, there was also related works like "Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design (NeurIPS 2023)" that studies the joint scaling of width and depth of ViT under a compute budget. The authors seem not reporting any comparison towards it even not citing.

The small scale of the datasets used in the experiments is not enough to remove the above concern as well. A more comprehensive experiments would be helpful to answer a fundamental question: whether the v-information based approach can provide a better capacity allocation principle than prior works under a fixed computation budget.

Questions for authors

Check the weakness part

Reviewer_cTAWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript asks when a learned representation is sufficient for a task, formalizes sufficiency as the vanishing of the conditional mutual information between input and label given the representation, and argues that Shannon estimators are unusable at high dimension. It proposes to replace Shannon quantities with probe-based V-information, computed as the cross-entropy of a linear probe trained on frozen features, and to grow networks until the marginal reduction in that quantity is no longer statistically and economically justified. Experiments cover ImageNet-1K, CIFAR-10, CIFAR-100, and Tiny-ImageNet across five architecture families, with protocol-matched retraining of discovered architectures, an eight-way ablation of stopping rules, an information-plane analysis, and an empirical check of the representation complexity identity.

Critical strengths
  1. The paper is unusually open about negative results: Appendix F reports the collapse of the unstandardized probe signal, instability of the MLP probe, failure of KSG above 50 dimensions, and the cost of joint width–depth search.
  2. The split protocol is stricter than usual: there are four separate splits with reported sizes; the probe split is never used for gradient updates; the selection split is used only to choose seeds; and the test split is held out until final evaluation.
  3. Discovered architectures are retrained from scratch under the same protocol as the fixed baselines. Table 1 reports standard deviations, so the comparison with ResNet-18/50/101 can be checked rather than taken on trust
Critical weaknesses
  1. Probe loss does not rank architectures, and the central empirical premise is contradicted by the paper's own table. Section 4.1 and Figure 1 claim that V-entropy is an architecture-agnostic predictor of generalization, but in Table 1 the protocol-matched ViT-B/16 has the lowest V-entropy of the fixed baselines at 0.82 bits and the lowest accuracy of the larger models at 72.8 percent, while ResNet-50 at 0.88 bits reaches 76.1 and ResNet-101 at 0.80 bits reaches 77.4. Figure 1 measures the correlation along training epochs within one family, where accuracy and probe loss both improve with training time, so the reported anti-correlation is a time confound rather than evidence about architecture quality. Linear probes are an established diagnostic of intermediate representations[1], but using them to rank architectures requires evidence that the ranking survives across families at fixed training, and that evidence is negative here.

  2. The stopping rule is a significance heuristic rather than a sufficiency test, and the paper's own residuals show that the stopped representations are not sufficient. At the V-NAS stopping point, the FCN retains 2.80 bits of residual information, 27 percent of the complexity functional, and the CNN retains 0.42 bits, while ResNet, DenseNet, and ViT are near 0.12 to 0.18. A method framed as answering when a representation is sufficient thus stops while substantial task-relevant information remains, and its threshold, 0.2 times one plus the log parameter ratio, comes from a small-effect convention rather than an error or cost model. V-information itself is prior work[1], so the novel element is precisely this threshold, and it is presented without derivation or sensitivity analysis.

  3. The comparison with the NAS literature is not protocol-matched, and the compactness claim does not survive it. V-NAS/ResNet reaches 75.5 percent with 14.2M parameters, while Table 1 lists PC-DARTS at 75.8 percent with 5.3M, P-DARTS at 75.6 with 4.9M, and DrNAS at 76.2 with 5.7M. Section 4 states that prior architectures were retrained under the authors' protocol, but cell-based architectures from the CIFAR space cannot be inserted into this stage-wise family, and the values coincide with published ImageNet numbers, so the provenance is unclear. In addition, the GFLOPs column duplicates the parameter column for several NAS rows, which suggests a transcription error. Either the NAS rows should be reproduced under the same 300-epoch protocol or the compactness claim should be limited to the ResNet baselines.

  4. Section 4.1 calls 6.5 GPU-hours "on par with training-free proxies", yet Table 1 lists ZiCo at under five minutes and SWAP-NAS at fifteen minutes, which are 26 to 78 times cheaper; parity holds only with TE-NAS at one GPU-hour, which is six times cheaper. In fact, the proposed approach is neither cheaper nor performance better.

[1]Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A Theory of Usable Information Under Computational Constraints. arXiv preprint, 2020. arXiv:2002.10689.

Questions for authors

N/A

Reviewer_hcSVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes V-NAS, which leverages the normalized gain of linear probe loss and parameter cost adjustment to guide the progressive growth of network width and depth. The method focuses on the automatic selection of model capacity, features a relatively concise algorithm, and is evaluated on four datasets and five types of architectures.

Critical strengths

The topic is suitable. The manuscript is well written.

Critical weaknesses
  1. Equation (15) only judges whether the normalized gain of a candidate expansion exceeds the threshold; it cannot infer that the current representation satisfies sufficiency, nor can it rule out improvements brought by other architectural adjustments or more adequate training. It is necessary to clarify whether the method judges the saturation of local gains or the strictly defined representation sufficiency, and provide supporting evidence.

  2. Section 2.1 states that the residual quantity is non-decreasing with depth in a deterministic network chain, yet Figure 4 and its accompanying discussion interpret the decline of this quantity with depth as a gradual achievement of sufficiency. It should be clarified whether these curves correspond to intermediate layers of a single network or separately trained networks with different depths, and explain how the corresponding theory applies.

  3. Theorem 2.2 links the minimization of differential entropy to searching for the minimal sufficient representation. Nevertheless, invertible scaling preserves representation sufficiency while altering the differential entropy value. Without scale constraints, differential entropy can be arbitrarily reduced without decreasing model capacity or predictive information. Necessary constraints should be supplemented, or the relevant conclusions should be qualified.

  4. As stated in Section 2.2, this baseline actually adopts unnormalized probe loss difference instead of Shannon CMI estimation. Therefore, its inferior performance cannot be attributed to the failure of Shannon information estimation. The effects of normalization, cost penalty and threshold selection should be evaluated separately, and the explanations in the main text and appendix must be unified.

  5. Algorithm 1 only allows keeping the current width or doubling it, while the ViT architectures in Table 15 contain width changes that deviate from this rule. In addition, according to the ImageNet input size and the first-layer width of FCN described in H.2–H.3, the weights of the first layer alone contain approximately 38.5M parameters, which exceeds the reported total model size of 4.2M. The actual candidate search space, input preprocessing pipeline, model configurations and parameter statistics should be provided to resolve these discrepancies.

  6. Algorithm 1 uses a held-out split for final retraining, whereas Table 5 employs the same split for model selection after retraining, with the claim that this split is never involved in gradient updates. The purpose of each data partition during search, model selection and final training should be explicitly specified to guarantee the independence of metrics used for evaluation and model selection.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.