acceptodds
Under review as a conference paper at ICLR 2027

Capacity Floors and Readout Limits for Bottlenecked Medical Image Classifiers

Abstract

Medical image classifiers often route every input through a bounded store, a code- book, a concept layer, or a prototype bank, chosen for interpretability or for de- ployment cost. When such a model falls short, the cause may be a store too small to separate the classes, an encoder that underuses the store it has, or a decision rule that discards information the store holds, and each calls for a different fix. A vali- dation curve reports the total error alone and leaves the binding cause unidentified. This paper decomposes the error into one term for each of these three causes. The first is a capacity floor, the smallest error any sampled positive quadratic readout attains on a store of dimension m, equal to one minus the sum of the m largest prior masses and therefore fixed by the architecture and the prior alone. The sec- ond is the representation gap: the error of the best readout in that class on the learned store, obtained as a semidefinite program with a dual certificate, minus the floor. The third is the readout gap: the attained error minus that best error, the excess due to the decision rule. The floor is tight and holds for every decoder on a discrete store and for the positive-readout class on a continuous store, and a two- dimensional counterexample shows that this decoder boundary is sharp. Inverting the floor sizes a store before training: the prior and a target error fix the narrow- est bottleneck that could reach it, so a proposed architecture can be ruled out on paper. Across four benchmarks, five bottleneck families, and an unbottlenecked reference, the decomposition separates certified capacity limits from errors car- ried by the representation or the readout. On discrete stores and our own register, where it is exact, the representation gap dominates the readout gap in all 12 con- figurations, and in the 12 diagnostic ones continuous heads deployed with argmax beat the best positive readout of their store, as the decoder boundary predicts.

Primary area:
learning theory
Keywords:
bottlenecked classifiers, capacity bounds, error attribution, representation learning, model diagnostics, medical image classification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QWGBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes decomposing classification error into a capacity floor, a representation gap, and a readout gap. The capacity floor depends on class priors and either discrete-store cardinality or continuous-state dimension under a restricted sampled positive-readout model. The best positive readout is computed through semidefinite optimization, and the resulting quantities are used to assign diagnostic verdicts and suggest interventions. Experiments cover three MedMNIST subsets and CIFAR-100. The paper also introduces BornReg, whose readout satisfies the assumptions by construction. My main concern is that the mathematical results have a substantially narrower scope than the diagnostic and practical interpretations attached to them.

Critical strengths

The distinction between sampled positive readouts and argmax decisions is meaningful, and the counterexample makes the scope limitation explicit. The authors also report numerical certificates for the restricted readout optimization and disclose unfavorable intervention results. These are useful elements, even though they do not establish the broader diagnostic contribution.

Critical weaknesses

The claimed diagnostic conclusions do not follow from the bounds without additional assumptions. For continuous representations, the capacity floor applies to sampled positive quadratic readouts, rather than ordinary argmax classification. The paper’s own counterexample demonstrates this limitation, and Table 2 contains argmax models whose errors fall below the stated floor. Nevertheless, such models can still receive a “capacity” verdict. A limitation of the restricted readout class does not establish a capacity limitation of the deployed classifier.

There is also a distinction between a lower bound permitting a target and an architecture being capable of learning that target on a particular task. The statement that a zero floor in binary classification rules out capacity as a source of error—and the claim in Appendix E that widening can therefore be ruled out—overinterprets the result. A zero lower bound alone cannot establish that widening the trained architecture would be unhelpful.

Finally, Section 5.2 acknowledges that the representation gap includes irreducible ambiguity, label noise, and input overlap, alongside encoder inefficiency. Assigning an “encoder-limited” verdict does not distinguish these causes. The mathematical quantities may be well defined within the stated model, but their interpretation as identifying the component responsible for failure remains insufficiently justified.

The experiments do not establish the usefulness of the proposed diagnosis. The most direct evaluation is the intervention study: the verdict selects the best repair in only 7 of 23 decisive cases, compared with 9 of 23 for always widening, as reported in Appendix G. The utilization regularizer increases error on average. These findings do not demonstrate that the diagnostic procedure provides an actionable advantage.

The authors appropriately acknowledge these limitations, but acknowledging them does not supply the missing validation. Appendix G also reports a training-schedule mismatch between the intervention study and the main experiments, and describes implementation changes whose corresponding intervention results have not been rerun. Consequently, the central practical claim remains unresolved.

If the intended contribution is instead a restricted theoretical characterization, the manuscript needs to state that contribution more precisely and distinguish its novelty from the counting bound and minimum-error discrimination machinery that it explicitly attributes to prior work. The current combination of theory, model construction, and unsuccessful intervention selection does not establish a sufficiently clear contribution.

Questions for authors

What conclusion does a “capacity” verdict establish for a continuous model deployed with argmax, when that model can outperform the positive-readout floor? Why should this verdict inform architectural changes to the deployed model?

How does the framework distinguish encoder inefficiency from irreducible uncertainty or label noise within the representation gap? What additional assumptions justify the “encoder-limited” interpretation?

What evidence supports actionable diagnostic value beyond always widening or refitting a classifier? Can the intervention study be rerun with consistent training and the final implementation?

Please clarify which claims concern only necessary feasibility conditions and which establish that a proposed bottleneck size is sufficient for the actual learning problem. In particular, how is ruling out widening in binary classification justified?

6 comments
AuthorsOpenReview ↗

Response to Reviewer QWGB, 1 of 2: scope of the guarantee

We thank the reviewer for a careful and demanding review. We agree with its main point: the bounds have a narrower scope than some of the diagnostic language of the submission. The revision aligns every claim with the result that supports it, and OpenReview's PDF diff shows every change.

Q1. The capacity verdict for a continuous model deployed with argmax. For such a model the verdict is diagnostic, and the revision names it so. A floor εcap=1−Σq(π)\varepsilon_{\rm cap} = 1-\Sigma_q(\pi) above the target means that no positive readout of a store of dimension qq reaches the target, for any encoder (Lemma 1). It proves a limitation of the deployed classifier only when the deployed rule lies in that class: any decoder of a discrete store's symbol, argmax included (Corollary 2), or a sampled positive readout such as BornReg (Section 5.4). Theorem 3 shows the restriction is needed: on a two-dimensional store the sampled readout sits on the floor while the argmax over the same scores is exact. The revision carries the distinction through. Algorithm 1 returns a status, certified or diagnostic, on every branch; its capacity branch returns "capacity limited" only when the status is certified and "positive-readout floor active" otherwise. Section 5.3 states that the second outcome does not show that the argmax head needs a wider store. Table 2 has a Status column and the verdict "positive floor" on its diagnostic rows, for example the CIFAR-100 projection and prototype bank at q=32q = 32, whose argmax risks of 0.484 and 0.607 lie below the floor of 0.680. The revision also characterizes the argmax separately, without extending the floor to it. Proposition 9 (Appendix D.1) bounds the argmax risk of a positive readout by the per-class mean and variance of its margin, through Cantelli's inequality. On four trained registers, with moments taken on the test split, it gives 0.300 to 0.451 against argmax errors of 0.104 to 0.270 on that split, where the Markov bound built on the column margin of Proposition 4 is vacuous (Table 11). That is a statement about the sample; Appendix D.1 also estimates the moments on validation data and shows where that estimate transfers and where it does not.

Q4 and W2. Necessary conditions, and the binary case. Every claim derived from the floor is a necessary condition: if εcap>ε⋆\varepsilon_{\rm cap} > \varepsilon^\star, no model with that store and a rule in the stated class reaches ε⋆\varepsilon^\star. Section 4.4 now says that a floor below the target, zero included, says nothing about whether training reaches the target or whether widening would help. The reviewer is right about the binary case: the sentence that a zero floor rules out capacity as a source of error, and the Appendix E statement that widening can then be ruled out, are removed. For C=2C = 2 the floor is zero at every m≥2m \ge 2 and rules out no store size.

Q2 and W3. Encoder inefficiency against irreducible uncertainty. The class states alone cannot separate the two, and Appendix A.6 proves it with two problems that share one prior, one set of class states and one deployed rule, hence one decomposition, while their Bayes errors are 0 and ee: a better encoder removes the whole gap of the first and none of the second. The revision adds a bound that separates them from one side (Proposition 10, Appendix I.3). For any classifier gg of the input with error εg\varepsilon_g, the encoder that emits the basis state of gg's prediction among the qq heaviest classes has a best readout with error at most εcap+εg\varepsilon_{\rm cap} + \varepsilon_g. So the part of a representation gap that the inputs force is at most εg\varepsilon_g, and εreg−εcap−εg\varepsilon_{\rm reg} - \varepsilon_{\rm cap} - \varepsilon_g lower-bounds the risk that another encoder into the same store removes. Both are differences of risks, not fractions of the gap. The bound needs no estimate of the Bayes error, a stronger reference tightens it, and the two problems above attain it. With the unbottlenecked linear head as gg, averaged over its runs, the risk that some encoder into the same store removes is at least 0.029 to 0.321 for the released register on four of the five datasets (Table 25); on DermaMNIST the gap exceeds the reference by 0.010, within its sampling error. The bound is an existence statement: the encoder of its proof need not be one the register's parameterization represents, so it bounds the risk some encoder removes, and not the gain from retraining the register. A stabilized register trained longer has a smaller gap on all five, and it exceeds the reference on HyperKvasir alone, by 0.024; on the other four this reference no longer separates the remaining gap from input ambiguity. run_reference.py reproduces the table from the saved diagnostics.

AuthorsOpenReview ↗

Response to Reviewer QWGB, 2 of 2: value beyond always widening

Q3, W4 and W5. Value beyond always widening or refitting. The general response gives the completed study. On selecting the repair the verdict does not beat always widening: 16 against 22 of the 23 decisive cases, exact McNemar p = 0.031 in favor of widening, and the paper states this. Always refitting is right in 1 case and always the encoder objective in none. The value we can show lies in the capacity branch, which is fixed before training: it marks the stores where widening pays most, and the general response gives the counts (Table 4). The contrast holds within each head family, with any one dataset left out, and in three comparisons new in the revision that narrow the room the base error has to account for it (Table 20). Over the range the two groups share, 0.091 to 0.482, the six named stores inside it reduce the error by 0.065 against 0.006 for the nineteen others; regressed on the base error alone the gain is almost unexplained (fit 0.017), and adding the branch raises the fit to 0.143 with a branch coefficient of 0.038 against a standard error of 0.015, while the base error keeps a coefficient within its own standard error of zero. Pairing each named store with the unnamed store closest in base error leaves eleven pairs within 0.05, whose mean difference is 0.017 with the named store ahead in six, the weakest of the three. Together they put the contrast beyond the base error alone and leave the size of that share open. They leave the dataset free, and the stabilized grid adds the comparison inside one dataset: on OrganAMNIST and HyperKvasir, the two datasets that widen stores of both branches, every capacity pair gains more than every representation pair, 9 of 9 on each (Appendix I.2). run_repair_policies.py prints all three, and Appendix G gives the tests at each level. The gain survives the matched continuation: 0.027 over it on average, while the encoder objectives and refitting stay within 0.001 of it. An encoder objective that works within that budget is open, and the conclusion says so.

W6. Novelty. The counting bound (Frenkel & Weiner, 2015; Shah, 2025) and minimum-error discrimination (Helstrom, 1969; Yuen et al., 1975; Eldar et al., 2003; Ježek et al., 2002) are credited as prior work in Sections 2 and 5.1, as are semidefinite duality and the Holevo bound. The new contribution is a statement of which decoder class such a bound covers for a bottlenecked classifier, and how to size and attribute inside that class. Theorem 3 separates the two decoder classes as far apart as the floor: for every C≥3C \ge 3 there is an encoder into a store of dimension 2 and a positive readout whose sampled risk equals the floor 1−2/C1-2/C, while the argmax over those same scores is exact, so no capacity bound of this kind reaches argmax without a further assumption. Sampling is a mixture of decoders, so the direction of that separation needs no construction; the witness fixes its size, which dominance does not give. That floor is the smallest risk Lemma 1 allows at m=2m=2, so the separation equals the floor at every class count and approaches one as CC grows (Appendix A.4). Proposition 4 says which quantity each rule depends on, the weighted diagonal for sampling and the column margin for argmax. Proposition 9, new in the revision, bounds the argmax risk of a trained readout from the mean and variance of its margin, a characterization of the deterministic rule that is separate from the floor. Proposition 6 is a tight total-variation bound that keeps the sizing rule valid under an estimated prior. Proposition 10, also new, bounds the part of a representation gap that the inputs force by the error of any reference classifier. Eq. (4) states when the three-term attribution is exact, and BornReg is an architecture whose deployed readout lies in the certified class by construction. The anonymous supplementary code carries every script these answers name. verify_theory.py checks Lemma 1, Corollary 2, Theorem 3 for C=3C = 3 to 4040, Propositions 4, 6, 8 and 10, and the scope of Eq. (4) on the Theorem 3 witness. run_intervention.py --controlled is the runner of the repair study, and run_margin.py evaluates Proposition 9 on any saved model.

On the scope of the claim, the revision does not claim that the class states identify the cause of a representation gap, which Appendix A.6 proves they cannot; Proposition 10 bounds the encoder's share from a reference classifier instead. Nor does it claim that its verdicts select a repair better than a fixed policy. It states which decoder class a capacity bound for a bottleneck can cover and when the three-term attribution is exact, with Theorem 3 showing that the first question has a sharp answer.

Thanks for the detailed response. The distinction between certified and diagnostic conclusions is now clearer. Removing the claims about the binary case and acknowledging that the representation gap cannot identify encoder inefficiency address several of my concerns.

However, my main concern remains the usefulness of the framework. For continuous models with argmax, the “positive floor” does not establish a capacity limitation of the deployed classifier. The reference-classifier bound is an existence result and does not establish what retraining the actual encoder can achieve. These are meaningful qualifications, but they leave the practical interpretation quite limited.

The completed intervention study also does not support the framework for choosing a repair: always widening succeeds in 22 of 23 cases, compared with 16 for the proposed diagnosis. Identifying where widening tends to help more is a narrower result, and the additional analyses do not yet establish a clear decision-making benefit.

The revised theoretical positioning is more defensible. Still, I find the overall framework complicated relative to the conclusions it supports, and the significance of the remaining contribution is not sufficiently clear to me. I therefore maintain my recommendation to reject.

Reviewer_vq53OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a theoretical framework for analysing errors in bottlenecked medical image classifiers. It derives a capacity floor and decomposes classification error into representation and readout gaps under specific assumptions. The authors introduce BornReg to enable exact error attribution for continuous stores. Experiments on three MedMNIST datasets and CIFAR-100 show that the representation gap dominates in the configurations with an exact decomposition. An additional experiment evaluates whether the resulting diagnoses can guide model improvements.

Critical strengths
  1. Theoretical Framework: Separates capacity limits from representation and readout errors. It offers a pre-training bottleneck sizing rule and clearly identifies the limitations of its theoretical guarantees.

  2. Certified Optimisation and Evaluation: Uses semidefinite optimisation and dual certificates to compute and verify optimal positive readouts. Experiments across multiple bottleneck architectures illustrate the proposed error decomposition under its stated assumptions.

Critical weaknesses

1. Unclear Decomposition Validity:

The claim that the full error decomposition (εcap≤εreg≤εact\varepsilon_{\mathrm{cap}}\leq\varepsilon_{\mathrm{reg}}\leq\varepsilon_{\mathrm{act}}) applies to arbitrary decoders on discrete stores appears to be contradicted by the construction in Theorem 3 under the definition in Section 3. While the capacity floor itself holds, this inconsistency raises significant questions about the scope of a central theoretical result.

2. Limited Practical Effectiveness:

The empirical results do not establish the diagnostic's reliability in guiding model improvements. It identifies the best tested intervention in only 7 of 23 decisive cases, while the proposed LfillL_{\mathrm{fill}} regulariser increases average error. The training-protocol mismatch acknowledged in Appendix G further limits the conclusions. Moreover, the requirement for sampled positive readouts restricts the applicability of the exact decomposition to conventional continuous-store classifiers, where argmax often achieves lower error.

Questions for authors
  1. Equation (4) Soundness: How is εreg≤εact\varepsilon_{\mathrm{reg}}\leq\varepsilon_{\mathrm{act}} guaranteed for arbitrary discrete-store decoders when εreg\varepsilon_{\mathrm{reg}} is optimised only over positive readouts? Under the discrete-store definition in Section 3, the construction in Theorem 3 appears to contradict this inequality. Does the result require additional assumptions about how discrete states are represented?

  2. Intervention Validation: Given the 7/23 success rate in identifying the best tested intervention, the average error increase from LfillL_{\mathrm{fill}}, and the acknowledged training-protocol mismatch, what evidence supports the diagnostic's reliability? Are results using the corrected protocol, including LsepL_{\mathrm{sep}}, available?

  3. Practical Applicability: Since the full continuous-store guarantees do not extend to conventional argmax decisions, which frequently outperform sampled readouts in the reported experiments, what practical benefit does the exact decomposition offer for conventional classifiers?

  4. Gap Estimation Reliability: While dual certificates establish numerical optimality for the estimated states, how sensitive are the representation gaps and diagnostic verdicts to finite-sample estimation uncertainty, particularly for underrepresented classes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer vq53, 1 of 2: the Eq. (4) correction and the decoder class

We thank the reviewer. The point about Eq. (4) is correct, and its correction is the main change to the theory.

Q1. Eq. (4) for discrete-store decoders. Under the submitted Section 3 a store was discrete when its states take finitely many values, and εreg\varepsilon_{\rm reg} was the positive-readout optimum on those states. The witness of Theorem 3 is then a discrete store of CC states with εreg=1−2/C\varepsilon_{\rm reg} = 1-2/C and an exact argmax, so εreg≤εact\varepsilon_{\rm reg} \le \varepsilon_{\rm act} fails for that decoder. Corollary 2 is unaffected; the middle term was in the wrong class, and the missing assumption is the one the reviewer asks about, that the classifier acts on the symbol. Correction (Sections 3, 5.1 and 5.2, Algorithm 1):

  • A store is discrete with cardinality MM when the encoder emits one of MM symbols S(x)S(x) and the classifier acts on that symbol, so every decoder is a function of SS. A store given by states in Hm\mathcal{H}_m is continuous of dimension mm, however many distinct states it takes.
  • Discrete store: εregdisc=1−∑smax⁡cPr⁡(S=s,Y=c)\varepsilon_{\rm reg}^{\rm disc} = 1-\sum_s \max_c \Pr(S=s,Y=c), the maximum a posteriori risk, and 1−ΣM(π)≤εregdisc≤εact1-\Sigma_M(\pi) \le \varepsilon_{\rm reg}^{\rm disc} \le \varepsilon_{\rm act} for every decoder of SS, by Corollary 2 and the optimality of that rule.
  • Continuous store: εregpos=1−max⁡{∑cπcTr(Ecρc):Ec⪰0,∑cEc=I}\varepsilon_{\rm reg}^{\rm pos} = 1-\max\{\sum_c \pi_c {\rm Tr}(E_c\rho_c) : E_c \succeq 0, \sum_c E_c = I\}, and 1−Σm(π)≤εregpos≤εact1-\Sigma_m(\pi) \le \varepsilon_{\rm reg}^{\rm pos} \le \varepsilon_{\rm act} for a deployed sampled positive readout. For a continuous store deployed with argmax only the first inequality is guaranteed, and Algorithm 1 marks that case diagnostic.
  • The witness in both classes (Section 5.2): read as a discrete store, whose classifier receives the symbol naming which of its CC states was emitted, it has cardinality CC and εregdisc=0\varepsilon_{\rm reg}^{\rm disc} = 0, and the chain holds for its argmax; read as a continuous store of dimension 2, its positive optimum is 1−2/C1-2/C, which its argmax beats.
  • A binary bottleneck whose classifier takes the concept probabilities is not a decoder of its code, so its rows in Table 2 are diagnostic (double dagger). Table 8 lists the decoder class of each kind of head, and Table 9 the measured chain of the same head trained on its thresholded code, where it is certified.

The released code already computed the discrete term by the closed-form rule, so no reported number changes. verify_theory.py checks the witness in both classes.

Q3. Practical benefit for conventional argmax classifiers. The certified decomposition covers two families used in practice: classifiers that decode a discrete code with any rule, argmax included (a codebook classifier on its selected entries, a concept layer on its thresholded code, a prototype bank under hard assignment), and continuous heads deployed as sampled positive readouts, such as BornReg. For a conventional continuous argmax head the revision offers four things.

  • A feasibility answer: whether any rule of the certified class reaches the target on this store.
  • A measured contrast: its argmax risk sits below the best positive readout of its lifted store in all 15 such configurations of Tables 2 and 13. That comparison is diagnostic: the lift keeps the direction of the feature and drops its magnitude. The same-score comparison is now measured as well: run_same_score.py compares each saved store's own argmax with a positive readout fitted to it: over the 63 stores where the two rules differ the argmax errs less in 62 (Section 6.4, Appendix F). Theorem 3 shows the separation can occur, not that it occurs on every model.
  • A bound on its own rule: Proposition 9 (Appendix D.1) bounds the argmax risk of a positive readout from the mean and variance of its margin. With moments from the test split it gives 0.300 to 0.451 against argmax errors of 0.104 to 0.270 on four trained registers (Table 11). With moments from validation data it covers the test error on DermaMNIST and HyperKvasir and falls below it on OrganAMNIST and PneumoniaMNIST, whose validation split is easier, so it is a tool for the distribution it is estimated on.
  • A certified evaluation of the same store. A positive readout fitted on one half of the test states and taken to the other, a cross-fitted evaluation, puts the prototype and projection heads inside the certified class. Over their 30 runs the chain holds up to the solver's tolerance, the readout gap is at most 0.004 and the representation gap, 0.140 to 0.627 per cell, carries the excess error. Each head's own argmax errs 0.306 less on average, as Theorem 3 allows. For both heads on DermaMNIST and the prototype head on PneumoniaMNIST the best positive readout returns the majority class (Table 10, Appendix D). A prospective deployment fits the readout off the test split instead, which Q4 below reports for five registers.
AuthorsOpenReview ↗

Response to Reviewer vq53, 2 of 2: intervention validation and finite samples

Q2. Intervention validation. The general response gives the completed study. The submitted 7 of 23 came from a faulty stage. In the controlled rerun the verdict names the winning repair in 16 of 23 decisive cases against 22 for always widening (exact McNemar p = 0.031), so it does not select repairs better than that fixed policy. The part the study supports is the capacity branch: the 26 stores it selects gain 0.039 from widening on average, against 0.006 for the other 19, and the gain holds against a matched continuation (0.027 over it). The branch follows the dataset, so we report the contrast per dataset, with the tests at each level in Appendix G, and inside OrganAMNIST and HyperKvasir, which widen stores of both branches, every capacity pair gains more than every representation pair (Appendix I.2). The encoder objectives stay within 0.001 of the continuation, and the paper states that such an encoder objective is open.

Q4. Finite-sample reliability. The dual certificate answers a numerical question, not a statistical one. It brackets the optimum of the semidefinite program on the estimated states, within 2.2×10−62.2 \times 10^{-6} over the 96 runs on continuous stores of the four medical datasets and 2.9×10−62.9 \times 10^{-6} over the 24 on CIFAR-100, and it says nothing about new patients. Appendix B separates the two and names three sources of sampling error.

  • The prior. Proposition 6 bounds the floor error by the total variation distance, tightly. On HyperKvasir the two official folds differ by 0.0016, and the largest floor shift over every qq is 0.0015.
  • The class states. εreg\varepsilon_{\rm reg} is the best value on the very states it is scored on, so it is optimistic on average, as Appendix B derives. Table 7 measures this on 150 of the 180 models of the main grid (the 30 linear-head records at q=256q = 256 are left out for cost). Each test split is cut into two halves with the same class proportions; the readout, or the maximum a posteriori decoder of a discrete store, is fitted on one half and scored on the other, both ways, over five cuts. The optimism averages 0.0008 to 0.0022 on the four medical datasets and 0.016 on CIFAR-100, where each class has 50 images in a half. The share of 20 stratified resamples that return the verdict of the full split is 1.00 on every dataset; this is stability under resampling of the evaluation split of one trained model, not across training seeds, whose verdicts can differ (Table 2 marks them mixed); Table 26 gives both gaps as mean and standard deviation over seeds for the 14 certified configurations, where the representation gap exceeds the readout gap in 40 of their 42 runs. The smallest class has 1 image in a half on HyperKvasir and 11 on DermaMNIST, where a class state is estimated least well, and the linear head at q=256q = 256, where the optimism should be largest, is outside the table, 30 records in all. The direction of the bias runs against the reading that it inflates the dominance finding. The reported εreg\varepsilon_{\rm reg} is the optimum on the class states of the split it is scored on, and Appendix B derives that it is biased low in expectation as an estimate of the optimum on the population states. In expectation the population representation gap is therefore at least the reported one and the population readout gap at most the reported one, so the margin of 0.025 by which the representation gap leads across the 14 certified configurations, all of which the table covers, is conservative. The held-out column measures how a readout fitted elsewhere generalizes, the quantity a deployment faces, and the in-class optimum remains the quantity that defines the certified decomposition. Appendix B now states both points.
  • The Holevo term enters through a stratified bootstrap upper estimate over 400 resamples, an empirical estimate kept apart from the numerical certificates.

A deployed readout is fitted off the test split. Appendix I.2 does this on five registers: the best readout of the validation states, solved under the evaluation prior and scored on test, has a sampled risk within 0.004 of εreg\varepsilon_{\rm reg} on each (Table 24), and the readout solved on the training states within 0.007 (Table 22).

Q5. Utilization. Appendix I states that the true ratio χ/log⁡2q\chi/\log_2 q reaches 1 when every class state is pure and their mixture is maximally mixed, which a full-rank mixture alone does not give, and that the reported u=χU/log⁡2qu = \chi_U/\log_2 q estimates that ratio: χU\chi_U is a bootstrap upper estimate of χ\chi, so uu can sit above the ratio and measures a store without certifying it. The correlation with the representation gap is reported as an association.

Thanks to the authors for the detailed response and substantial revision, which address most of my concerns, in particular the issue with Equation (4).

However, I remain concerned about the practical utility of the proposed diagnostic. In the controlled intervention study, it selects the best repair in 16/23 cases, compared with 22/23 for always widening. While I acknowledge the positive results for identifying where widening is beneficial, I remain unconvinced by the overall practical contribution.

I appreciate the improvements but I prefer to keep my initial ratting of 2.

Reviewer_5RmhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the performance limits of medical image classifiers with bottleneck representations. The authors propose a theoretical framework that divides classification error into three parts: capacity floor, representation gap, and readout gap. The method aims to identify whether the performance limitation comes from the bottleneck size, learned representation, or classifier. Experiments on MedMNIST and CIFAR-100 are conducted to validate the proposed analysis.

Critical strengths
  1. The paper studies an interesting problem. Understanding the performance limits of bottleneck models can be useful for model design and optimization.

  2. The theoretical analysis provides a clear way to separate different sources of classification errors.

  3. The paper considers several types of bottleneck models, including codebooks, concept representations, and prototype-based models.

  4. The experiments include different datasets and class distributions, which help demonstrate the proposed theoretical framework.

Critical weaknesses
  1. Limited applicability to practical classifiers. The theoretical guarantees are mainly established for specific readout functions. However, many practical neural networks use deterministic argmax predictions. The authors should better explain how the results can be applied to commonly used classifiers and provide more practical examples.

  2. The practical benefits are not sufficiently demonstrated. Although the proposed method can identify possible sources of errors, it is unclear whether this analysis can effectively guide model improvement. In Table 4, the proposed diagnostic method selects the best intervention in only 7 out of 23 cases. More experiments are needed to demonstrate its practical usefulness.

  3. Limited evaluation on medical imaging tasks. The experiments use three relatively small MedMNIST subsets. More challenging medical image datasets with different class numbers and imbalance levels would strengthen the experimental validation.

  4. The proposed bounds may not always be informative. When the bottleneck dimension is larger than the number of classes, the capacity floor becomes zero. The authors should discuss when the proposed bound provides useful information and compare it with other methods for evaluating representation capacity.

  5. Computational complexity needs further discussion. The proposed analysis involves semidefinite optimization, which may become expensive for larger models. The authors should report the computational cost and study how it scales with the number of classes and bottleneck dimensions.

Questions for authors
  1. How can the proposed theoretical framework be applied to standard classifiers using argmax predictions?

  2. Table 4 shows limited success in selecting the best model improvement strategy. How can the authors explain this result?

  3. Can the authors provide experiments on more challenging medical image datasets?

  4. Under what conditions is the proposed capacity floor informative for practical model design?

  5. How does the computational cost scale with the bottleneck size and number of classes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5Rmh, 1 of 2: positioning, clarity, and the certified class

We thank the reviewer for the clear summary and questions.

Q1. Standard classifiers with argmax. The guarantee depends on the decoder class, and the revision marks every verdict certified or diagnostic (Algorithm 1, Table 2); Table 8 in Appendix D lists each kind of head and the class it falls in. Three practical cases:

  • Discrete bottlenecks. On a store that emits one of MM symbols, the floor bounds every decoder of the code, argmax included (Corollary 2): a vector-quantized classifier on its selected entries, a concept bottleneck on its thresholded concepts, a prototype bank under hard assignment. Before training, M⋆M^\star gives the smallest code whose floor admits a target error under any decoder, for example 9 symbols on OrganAMNIST and 90 on CIFAR-100 at a target of 0.10; after training the attribution is certified (codebook rows of Table 2).
  • Continuous heads deployed with argmax. The floor binds the sampled rule, and Theorem 3 shows that the argmax over the same scores can be exact on a store where the sampled rule sits on the floor, so no capacity bound of this type covers argmax without further assumptions. The attribution of such a head is diagnostic, and its capacity outcome is "positive floor" in Table 2. In Tables 2 and 13 the argmax beats the best positive readout of its lifted store in all 15 such configurations, by up to 0.55 for the prototype bank at q=4q = 4 on OrganAMNIST. For the argmax itself the revision adds Proposition 9 (Appendix D.1), a bound on the argmax risk of a positive readout from the mean and variance of its margin: 0.300 to 0.451 on four trained registers with moments from the test split, against argmax errors of 0.104 to 0.270 on that split (Table 11). It characterizes the deterministic rule separately and does not extend the floor to it.
  • Certified continuous heads. A head whose deployed rule is a sampled positive readout, such as BornReg, has a certified attribution (Section 5.4). The readout the diagnostic returns is itself deployable: fitted on validation states and scored on test, it brings the sampled risk of five registers within 0.004 of εreg\varepsilon_{\rm reg} (Appendix I.2, Table 24).

Q2. Table 4 and the 7 of 23. The submitted count came from a repair stage that ran with no learning-rate schedule and at batch size 2, and the revision replaces it with a controlled rerun of 60 cases, described in the general response. Its outcome has two parts. On choosing among repairs the verdict is right in 16 of 23 decisive cases and always widening in 22 (exact McNemar p = 0.031), so the diagnostic does not beat that fixed policy, and the paper says so. On locating where capacity pays most it helps: the 26 stores its capacity branch selects, fixed before training, gain 0.039 from widening on average, against 0.006 for the other 19. The branch follows the dataset, so we report the contrast per dataset; it holds within each head family, with any one dataset left out, and inside OrganAMNIST and HyperKvasir, where every capacity pair gains more than every representation pair (Appendix I.2). The revised Table 4 stratifies the benefit of widening by that branch, Table 16 gives the comparison with fixed policies, and Table 19 every case with its matched continuation.

AuthorsOpenReview ↗

Response to Reviewer 5Rmh, 2 of 2: harder data, utilization, and the floor

Q3. A more challenging medical dataset. We added HyperKvasir, and its trained results are in the paper: gastrointestinal endoscopy, 10,662 labeled images in 23 classes at 128 by 128 pixels, trained on the first official fold and tested on the second, with every bottleneck family at store sizes 4 and 12. Its prior ratio between the heaviest and lightest class is 191, against 58 for DermaMNIST, so m⋆(0.10)=12m^\star(0.10) = 12 against 21 under a uniform prior over 23 classes. Table 2 shows the floor's regimes on one dataset: at q=4q = 4 the floor is 0.608 and the codebook is certified capacity limited; the register at m=12m = 12, where the floor is 0.086, is certified with a representation gap of 0.357 and a readout gap of 0.205; the projection and prototype bank at q=4q = 4 have the positive-readout floor active, while their argmax risks, 0.290 and 0.571, lie below it, as Theorem 3 allows. Table 27 gives the accuracies, Table 15 both decision rules, and Figure 4 the utilization analysis, all with HyperKvasir. The split lists images and not patients, and no examination identifier is distributed, so a patient-level split is not available. The code counts duplicates across the folds before any number is used: no pair shares a byte, and 3,835 pairs lie within 4 bits of a 64-bit difference hash, over 2,222 images; 3,091 of those pairs share a label and 744 do not. A small hash distance shows that two images look alike, not that they come from one patient, so these pairs neither prove nor rule out leakage. Appendix E states the consequence: the floor depends on the evaluation prior alone and is unaffected, while the learned representation, εreg\varepsilon_{\rm reg}, both gaps and the accuracy may be optimistic against a split that separates examinations.

Q4. When the floor is informative. The floor 1−Σq(π)1-\Sigma_q(\pi) is positive exactly when the store size qq is below the number of classes with positive mass; it grows as qq falls and as the prior flattens. Three cases follow (Table 1, Section 6, Appendix E):

  • q≥Cq \ge C, including every binary task at m≥2m \ge 2: the floor is zero and rules out no store.
  • q<Cq < C with a skewed prior: the floor is small and m⋆m^\star can sit far below CC, for example 4 of 7 classes on DermaMNIST and 12 of 23 on HyperKvasir at a target of 0.10.
  • q<Cq < C with a flat prior: the floor is large and m⋆m^\star close to CC, for example 90 of 100 classes on CIFAR-100.

A floor decides feasibility only where it exceeds the target. One observation from a register trained longer at one seed: the classes its argmax never returns are the lightest ones, exactly the 3 on DermaMNIST whose mass of 0.059 equals the floor at q=4q = 4, and the 11 on HyperKvasir whose mass of 0.086 equals the floor at q=12q = 12, plus one more (Appendix I.2, Table 24). Lemma 1 does not require this, and it does not bind the argmax; we report it as an empirical tendency of a frequency-weighted objective. Comparison with other capacity measures: the entropic floor of Section 4.3 depends on the learned states and never exceeds the counting floor at the dimension-only ceiling (Proposition 8); mutual-information bounds need estimators that are unreliable in high dimension; linear probes describe a representation after training, while the floor bounds every encoder of a given store size before training (Section 2).

Q5. Computational cost. Appendix B and Table 6, measured on 2 cores of a 2.1 GHz Xeon:

  1. Floor and sizing rule: one sort of the prior, O(Clog⁡C)O(C \log C), about 0.2 ms at C=10,000C = 10{,}000.
  2. Certified readout: a discrete store takes the closed-form maximum a posteriori rule, and the two-class case one eigenvalue decomposition. Otherwise each iteration costs O(Cm3)O(Cm^3), and so does the dual certificate. Measured: 0.02 s at (m,C)=(4,7)(m, C) = (4, 7), 0.28 s at (4,11)(4, 11), 21.2 s at (32,100)(32, 100), 92.6 s at (90,100)(90, 100), and 4.0 s per iteration at (256,100)(256, 100), where a first-order semidefinite solver becomes the better choice.
  3. Training dominates: every configuration trains a ResNet-18 end to end with three seeds.

Limitations. The main text now ends Section 6 with a Limitations paragraph on page 10: the decoder class the guarantee covers, the optimism of εreg\varepsilon_{reg} that Table 7 measures by cross-fitting, the input ambiguity the representation gap holds, the exploratory repair study with its dependence on the dataset and its result inside one dataset, and the HyperKvasir split by image.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Global response 1 of 4: the stabilized register

Every reviewer asked whether the central finding survives training the register properly. We reran the register over the same 3 seeds and the same 15-epoch protocol as the main grid, changing the register alone: it layer-normalizes the backbone feature, draws its generators at scale 0.1, adds the sampled risk to the log loss at weight 1, and deploys the readout solved on its training states.

Accuracy. The main paper now carries the matched 60-epoch comparison (Table 5); the 15-epoch grid Table 2 decomposes is Table 27. There the stabilized register improves on all five datasets: OrganAMNIST 0.885 to 0.941, DermaMNIST 0.702 to 0.732, PneumoniaMNIST 0.817 to 0.853, CIFAR-100 0.273 to 0.467, HyperKvasir 0.670 to 0.833. Under the reported 15-epoch settings it has the highest argmax accuracy among the bottlenecked heads on 4 of the 5 datasets, all but CIFAR-100, where the projection head reaches 0.516, and it is above the unbottlenecked linear head on OrganAMNIST, 0.941 against 0.938, and on HyperKvasir, 0.833 against 0.832. Tuning the baselines. The register was revised after the first round while the comparison heads kept the released settings, so we retrained the two discrete heads with the one option the released code carries for each, the codebook's dead-code restart and the binary head's logit normalization, at the store sizes, protocol and seeds of Table 27, each head at one setting of its own released configuration (Table 21, Appendix I.1). The codebook gains 0.007 to 0.143 on all five datasets; the binary head gains 0.048 to 0.077 on three, holds on DermaMNIST and falls 0.075 on PneumoniaMNIST, spread 0.131 over seeds. The stabilized register keeps the highest bottlenecked accuracy on OrganAMNIST, DermaMNIST and HyperKvasir. On PneumoniaMNIST the restarted codebook reaches 0.855 and the stabilized register 0.853, a difference inside the seed spread of both, 0.004 and 0.007; Table 27 bolds the best of its own rows. The prototype and projection heads carry no such option. The ordering holds on all five datasets (Tables 22 and 23). CIFAR-100 is no longer the exception, and Table 22 now carries its released against stabilized comparison too: at q=90q = 90 the stabilization lowers εreg\varepsilon_{reg} from 0.848 to 0.704, εact\varepsilon_{act} from 0.884 to 0.711 and the readout gap from 0.036 to 0.007. We now have the stabilized register's certified decomposition at three seeds on every dataset and at every store size of the grid, 42 runs, and the representation gap exceeds the readout gap in all 42, by 0.027 to 0.599. At the admissible size, five datasets at three seeds, the 15 runs lead by 0.054 to 0.599. On CIFAR-100 at q=90q = 90 the floor is 0.100, εreg\varepsilon_{reg} is 0.704 and εact\varepsilon_{act} is 0.711, a representation gap of 0.604 against a readout gap of 0.007.

Which decoder each number uses. These are not interchangeable and we have separated them in both captions. On OrganAMNIST at q=9q = 9 the stabilized register has a floor of 0.088, an εreg\varepsilon_{reg} of 0.145 and an εact\varepsilon_{act} of 0.145 under its sampled positive readout; the argmax of that same readout errs 0.121; and the decoder trained beside it errs 0.059, which is the 0.941 of Table 27. The certified chain covers the sampled rule. The decoder is an argmax rule the floor does not bind, as Theorem 3 permits, and Table 27 reports every head under argmax so the column is comparable.

The finite-sample check covers the widest certified store. The CIFAR-100 register at q=90q = 90, the one certified configuration missing from Table 7 for cost, is now in: optimism 0.0119, 0.0165 and 0.0191 on the three seeds, 0.0158 on average, with the full-split verdict returned by every resample. The table covers 150 of the 180 primary-grid models, every certified one among them; the 30 linear-head records at q=256q = 256 are out for cost. Under a readout cross-fitted between the two halves of the test split, the representation gap leads the readout gap on all 14 certified configurations and in 52 of their 54 records, by 0.194 on average, the records being 6 for each of the 4 codebook configurations whose two requested widths realize one store and 3 for each of the other 10. Resampling the configurations with their seeds inside gives a configuration-level interval of 0.124 to 0.284, which holds the test split fixed; the smallest configuration margin is 0.026 (Appendix B). Section 6.3 states the interval and Section 6.4 the 66 certified runs; Algorithm 1 moved to Appendix H and Table 3 lists each head as certified or diagnostic. Since εreg\varepsilon_{reg} is fitted on the states it is scored on, it is optimistic, an effect this table measures by cross-fitting, and Appendix B derives the direction: in expectation the population representation gap is at least the reported one and the population readout gap at most it, so the dominance finding is conservative.

AuthorsOpenReview ↗

Global response 2 of 4: paired widening and a certified head outside our family

A paired test of the capacity branch. The grid now trains three store sizes per dataset, two on CIFAR-100, so widening can be measured inside one dataset, one head and one seed, which holds the dataset and the seed fixed, while the base error of the two widths differs. Over the 27 adjacent pairs, the 9 whose narrower store carries a capacity verdict lower the sampled risk by 0.268 on average, from 0.072 to 0.377; the 18 whose narrower store carries a representation verdict lower it by 0.024, from -0.015 to 0.078. The groups of pairs are disjoint and their gain ranges overlap: the three CIFAR-100 pairs gain 0.072 to 0.078 and the OrganAMNIST pairs from q=9q = 9 to q=16q = 16 gain 0.076 to 0.078. Of the 162 cross comparisons the capacity-branch gain is the larger in 155, and the two group means differ by 0.245. Both branches now sit inside one dataset on three of the five, and OrganAMNIST and HyperKvasir widen stores of both, so there the pairing holds the dataset fixed across the verdict as well as across the size: every capacity pair gains more than every representation pair, 9 of 9 on each, at 0.333 or more against at most 0.078. The three seeds of a pair of sizes share their configuration, so we report the contrast per run and per dataset (Table 23, Appendix I.2).

The branch locates the binding constraint, and the size of the gain depends on how close the wider store trains to its floor. On CIFAR-100 widening from q=45q = 45 to q=90q = 90 drops the floor by 0.450, from 0.550 to 0.100, and lowers the sampled risk by 0.074, because the representation gap grows from 0.227 to 0.604 over the same step. The capacity verdict states that widening is necessary, and Appendix I.2 reports the size of each gain.

A certified head outside the register family. The sharpest objection was that the certified class holds little besides our own construction. Table 8 already states that a concept layer whose classifier takes the thresholded code is certified for any decoder, and we have now trained that head: the same backbone with a hard concept bottleneck, five datasets, three seeds, concept counts from three to nine, 36 runs. The certified chain holds in all 36. A code of nn concepts carries at most 2n2^n symbols, so the floor becomes active once 2n2^n drops below the class count, and it does: 0.360 on CIFAR-100 at six concepts, where 64 symbols serve 100 classes, 0.262 on HyperKvasir at three and 0.133 on OrganAMNIST at three. The floor is active in 9 of the 36 runs and each returns a capacity verdict, so that branch is reachable on a head that is not ours, and OrganAMNIST, HyperKvasir and CIFAR-100 each carry both branches. The store accounts for more of the excess error than the decoder in 35 of the 36. The exception is HyperKvasir at three concepts on seed 1, where 8 symbols for 23 classes leave a representation gap of 0.189 under a readout gap of 0.267, and we report it. The paired widening result replicates on this head: of the 21 adjacent pairs, the 9 narrower codes under a capacity verdict cut the risk by 0.216 on average against 0.022 for the 12 under a representation verdict, the larger figure in 98 of the 108 cross pairs. It costs accuracy against the stabilized register at nine concepts on three of the four datasets they share, and matches it within 0.007 on PneumoniaMNIST. This is a standard interpretable architecture and not ours (Table 9, Appendix D).

Scope of the stabilized results. The readout gap falling to at most 0.007 is expected, since a readout solved on the training states already sits near the optimum of the test states; that number measures how far the solve carries across splits. No step of the stabilization sets εreg\varepsilon_{reg} directly, so the representation result is a measurement of the trained store. At 15 epochs the stabilized register reaches 0.467 on CIFAR-100 and the projection head 0.516; a search over four configurations at seed 0 varied the generator rank, the depth, the loss weight and the learning rate, and the best of them, a rank-16 variant, reached 0.4784 against the 0.4655 of the same seed. The training budget is the binding constraint on that column: at 60 epochs the register gains 0.034 and the projection head gives up 0.015, so the two meet at 0.501 (Table 5). Table 27 compares every head at the 15-epoch protocol they share.

On the protocol question we retrained every head of Table 27 for 60 epochs on all five datasets, three seeds and five on DermaMNIST and PneumoniaMNIST, each at its last epoch with its Table 21 option. Under that one protocol the register holds the highest bottlenecked accuracy on each of the four medical datasets, by 0.002 to 0.010, and ties the projection head on CIFAR-100 at 0.501; on three of those four it is also above the unbottlenecked linear head, by 0.002 to 0.005, and comes within 0.001 of it on DermaMNIST; its own spread stays at 0.006 or below everywhere (Table 5, Appendix I.5).

AuthorsOpenReview ↗

Global response 3 of 4: the repair study, rerun and audited

All three reviewers asked whether the verdict picks a better repair than a fixed policy. The controlled rerun we announced is complete, and the revised paper reports it alone (Table 4, Tables 16, 18 to 20, Appendix G). In short: among the study's repairs the verdict is not a better selector than widening everywhere; its capacity branch marks the stores where widening pays most, at every target from 0.05 to 0.30; and on configurations paired at one seed, following it between widening and a repair at the same size beats widening everywhere. The submitted 7 of 23 came from a repair stage with two verified defects: it ran with no learning-rate schedule, and at batch size 2. An intermediate corrected study widened by an amount that depended on the verdict, had no matched continuation and scored the register by argmax. Neither is used.

Protocol. 60 cases: five datasets, four continuous heads, three seeds. Every arm starts from the same base weights, with the same extra epochs, one tenth of the base learning rate and one order of the training examples. The capacity arm widens to max⁡(m⋆,2m)\max(m^\star, 2m) whatever the verdict, so that always widening is a policy fixed in advance. A matched continuation trains for the same epochs with no change. The register is scored by its sampled positive readout and the other heads by argmax. The linear head has no store to widen, so the counts below cover the 45 stores of the three bottlenecked heads. A case is decisive when one repair leads the next by 0.005, which leaves 23.

Result 1, selecting the repair. Among policies fixed in advance, always widening is right in 22 of the 23 decisive cases, always refitting in 1 and always the encoder objective in 0. The verdict is right in 16 (95% bootstrap interval 12 to 20). The two disagree on 6 cases, all won by widening (exact McNemar p = 0.031). The paper states that the verdict is not a better selector among repairs (Section 6.4, Table 16). All 16 cases it gets right lie in its capacity branch. The counts move with the margin and the pattern stays: at margins of 0.0025, 0.01 and 0.02 the capacity branch names the winner in 18 of 19, 14 of 14 and 9 of 9 decisive cases, and the encoder objective wins none (Appendix G).

Result 2, where widening pays most. The capacity branch of Algorithm 1 follows from the prior and the store size, so it is fixed before any repair is trained. It is certified for the register and diagnostic for the argmax heads. The 26 stores it selects gain 0.039 from widening on average and the other 19 gain 0.006, so 91% of the total reduction from widening falls on 58% of the stores (Table 4). Widening is still the best repair in 6 of the 7 decisive cases among the 19. The branch follows the dataset, and 3 seeds share each dataset and head. A one-sided Mann-Whitney test gives p = 0.004 over the 45 stores and p = 0.034 over the 15 means by dataset and head, and with the dataset as the unit, 2 of the 10 ways of choosing 3 of the 5 datasets give a difference at least as large, so we report the contrast per dataset. The revision adds comparisons that hold the error of the base model fixed, since the stores the branch names start with more of it, 0.572 against 0.268: inside the range the two groups share the six named stores still gain 0.065 against 0.006, the base error alone explains almost none of the gain, and the branch keeps its coefficient when both are in the model. These comparisons put the contrast beyond the base error alone (Table 20 and Appendix G), and the stabilized grid holds the dataset fixed: inside OrganAMNIST and HyperKvasir every capacity pair gains more than every representation pair (Appendix I.2). The contrast also holds within each head family and with any one dataset left out. Table 18 gives the means by dataset: 0.031 on OrganAMNIST, 0.080 on HyperKvasir and 0.002 on CIFAR-100 at m=45m = 45 under the capacity branch, against 0.011 on DermaMNIST and 0.001 on PneumoniaMNIST.

Result 3, the control. Against the matched continuation, widening lowers the error by a further 0.027 on average over the 45 stores, ahead by the margin on 23 of them and behind on 3. The two encoder objectives and readout refitting stay within 0.001 of the continuation on average over the 60 cases. So widening reduces the error beyond continued training alone, and an encoder objective that removes the gap within that budget remains open, as the conclusion states.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.