- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes decomposing classification error into a capacity floor, a representation gap, and a readout gap. The capacity floor depends on class priors and either discrete-store cardinality or continuous-state dimension under a restricted sampled positive-readout model. The best positive readout is computed through semidefinite optimization, and the resulting quantities are used to assign diagnostic verdicts and suggest interventions. Experiments cover three MedMNIST subsets and CIFAR-100. The paper also introduces BornReg, whose readout satisfies the assumptions by construction. My main concern is that the mathematical results have a substantially narrower scope than the diagnostic and practical interpretations attached to them.
Critical strengths
The distinction between sampled positive readouts and argmax decisions is meaningful, and the counterexample makes the scope limitation explicit. The authors also report numerical certificates for the restricted readout optimization and disclose unfavorable intervention results. These are useful elements, even though they do not establish the broader diagnostic contribution.
Critical weaknesses
The claimed diagnostic conclusions do not follow from the bounds without additional assumptions. For continuous representations, the capacity floor applies to sampled positive quadratic readouts, rather than ordinary argmax classification. The paper’s own counterexample demonstrates this limitation, and Table 2 contains argmax models whose errors fall below the stated floor. Nevertheless, such models can still receive a “capacity” verdict. A limitation of the restricted readout class does not establish a capacity limitation of the deployed classifier.
There is also a distinction between a lower bound permitting a target and an architecture being capable of learning that target on a particular task. The statement that a zero floor in binary classification rules out capacity as a source of error—and the claim in Appendix E that widening can therefore be ruled out—overinterprets the result. A zero lower bound alone cannot establish that widening the trained architecture would be unhelpful.
Finally, Section 5.2 acknowledges that the representation gap includes irreducible ambiguity, label noise, and input overlap, alongside encoder inefficiency. Assigning an “encoder-limited” verdict does not distinguish these causes. The mathematical quantities may be well defined within the stated model, but their interpretation as identifying the component responsible for failure remains insufficiently justified.
The experiments do not establish the usefulness of the proposed diagnosis. The most direct evaluation is the intervention study: the verdict selects the best repair in only 7 of 23 decisive cases, compared with 9 of 23 for always widening, as reported in Appendix G. The utilization regularizer increases error on average. These findings do not demonstrate that the diagnostic procedure provides an actionable advantage.
The authors appropriately acknowledge these limitations, but acknowledging them does not supply the missing validation. Appendix G also reports a training-schedule mismatch between the intervention study and the main experiments, and describes implementation changes whose corresponding intervention results have not been rerun. Consequently, the central practical claim remains unresolved.
If the intended contribution is instead a restricted theoretical characterization, the manuscript needs to state that contribution more precisely and distinguish its novelty from the counting bound and minimum-error discrimination machinery that it explicitly attributes to prior work. The current combination of theory, model construction, and unsuccessful intervention selection does not establish a sufficiently clear contribution.
Questions for authors
What conclusion does a “capacity” verdict establish for a continuous model deployed with argmax, when that model can outperform the positive-readout floor? Why should this verdict inform architectural changes to the deployed model?
How does the framework distinguish encoder inefficiency from irreducible uncertainty or label noise within the representation gap? What additional assumptions justify the “encoder-limited” interpretation?
What evidence supports actionable diagnostic value beyond always widening or refitting a classifier? Can the intervention study be rerun with consistent training and the final implementation?
Please clarify which claims concern only necessary feasibility conditions and which establish that a proposed bottleneck size is sufficient for the actual learning problem. In particular, how is ruling out widening in binary classification justified?
6 comments
Response to Reviewer QWGB, 2 of 2: value beyond always widening
Q3, W4 and W5. Value beyond always widening or refitting.
The general response gives the completed study. On selecting the repair the verdict does not beat always widening: 16 against 22 of the 23 decisive cases, exact McNemar p = 0.031 in favor of widening, and the paper states this. Always refitting is right in 1 case and always the encoder objective in none.
The value we can show lies in the capacity branch, which is fixed before training: it marks the stores where widening pays most, and the general response gives the counts (Table 4). The contrast holds within each head family, with any one dataset left out, and in three comparisons new in the revision that narrow the room the base error has to account for it (Table 20). Over the range the two groups share, 0.091 to 0.482, the six named stores inside it reduce the error by 0.065 against 0.006 for the nineteen others; regressed on the base error alone the gain is almost unexplained (fit 0.017), and adding the branch raises the fit to 0.143 with a branch coefficient of 0.038 against a standard error of 0.015, while the base error keeps a coefficient within its own standard error of zero. Pairing each named store with the unnamed store closest in base error leaves eleven pairs within 0.05, whose mean difference is 0.017 with the named store ahead in six, the weakest of the three. Together they put the contrast beyond the base error alone and leave the size of that share open. They leave the dataset free, and the stabilized grid adds the comparison inside one dataset: on OrganAMNIST and HyperKvasir, the two datasets that widen stores of both branches, every capacity pair gains more than every representation pair, 9 of 9 on each (Appendix I.2). run_repair_policies.py prints all three, and Appendix G gives the tests at each level. The gain survives the matched continuation: 0.027 over it on average, while the encoder objectives and refitting stay within 0.001 of it. An encoder objective that works within that budget is open, and the conclusion says so.
W6. Novelty.
The counting bound (Frenkel & Weiner, 2015; Shah, 2025) and minimum-error discrimination (Helstrom, 1969; Yuen et al., 1975; Eldar et al., 2003; Ježek et al., 2002) are credited as prior work in Sections 2 and 5.1, as are semidefinite duality and the Holevo bound. The new contribution is a statement of which decoder class such a bound covers for a bottlenecked classifier, and how to size and attribute inside that class. Theorem 3 separates the two decoder classes as far apart as the floor: for every there is an encoder into a store of dimension 2 and a positive readout whose sampled risk equals the floor , while the argmax over those same scores is exact, so no capacity bound of this kind reaches argmax without a further assumption. Sampling is a mixture of decoders, so the direction of that separation needs no construction; the witness fixes its size, which dominance does not give. That floor is the smallest risk Lemma 1 allows at , so the separation equals the floor at every class count and approaches one as grows (Appendix A.4). Proposition 4 says which quantity each rule depends on, the weighted diagonal for sampling and the column margin for argmax. Proposition 9, new in the revision, bounds the argmax risk of a trained readout from the mean and variance of its margin, a characterization of the deterministic rule that is separate from the floor. Proposition 6 is a tight total-variation bound that keeps the sizing rule valid under an estimated prior. Proposition 10, also new, bounds the part of a representation gap that the inputs force by the error of any reference classifier. Eq. (4) states when the three-term attribution is exact, and BornReg is an architecture whose deployed readout lies in the certified class by construction.
The anonymous supplementary code carries every script these answers name. verify_theory.py checks Lemma 1, Corollary 2, Theorem 3 for to , Propositions 4, 6, 8 and 10, and the scope of Eq. (4) on the Theorem 3 witness. run_intervention.py --controlled is the runner of the repair study, and run_margin.py evaluates Proposition 9 on any saved model.
On the scope of the claim, the revision does not claim that the class states identify the cause of a representation gap, which Appendix A.6 proves they cannot; Proposition 10 bounds the encoder's share from a reference classifier instead. Nor does it claim that its verdicts select a repair better than a fixed policy. It states which decoder class a capacity bound for a bottleneck can cover and when the three-term attribution is exact, with Theorem 3 showing that the first question has a sharp answer.
Thanks for the detailed response. The distinction between certified and diagnostic conclusions is now clearer. Removing the claims about the binary case and acknowledging that the representation gap cannot identify encoder inefficiency address several of my concerns.
However, my main concern remains the usefulness of the framework. For continuous models with argmax, the “positive floor” does not establish a capacity limitation of the deployed classifier. The reference-classifier bound is an existence result and does not establish what retraining the actual encoder can achieve. These are meaningful qualifications, but they leave the practical interpretation quite limited.
The completed intervention study also does not support the framework for choosing a repair: always widening succeeds in 22 of 23 cases, compared with 16 for the proposed diagnosis. Identifying where widening tends to help more is a narrower result, and the additional analyses do not yet establish a clear decision-making benefit.
The revised theoretical positioning is more defensible. Still, I find the overall framework complicated relative to the conclusions it supports, and the significance of the remaining contribution is not sufficiently clear to me. I therefore maintain my recommendation to reject.
Response to Reviewer QWGB, 1 of 2: scope of the guarantee
We thank the reviewer for a careful and demanding review. We agree with its main point: the bounds have a narrower scope than some of the diagnostic language of the submission. The revision aligns every claim with the result that supports it, and OpenReview's PDF diff shows every change.
Q1. The capacity verdict for a continuous model deployed with argmax. For such a model the verdict is diagnostic, and the revision names it so. A floor εcap=1−Σq(π) above the target means that no positive readout of a store of dimension q reaches the target, for any encoder (Lemma 1). It proves a limitation of the deployed classifier only when the deployed rule lies in that class: any decoder of a discrete store's symbol, argmax included (Corollary 2), or a sampled positive readout such as BornReg (Section 5.4). Theorem 3 shows the restriction is needed: on a two-dimensional store the sampled readout sits on the floor while the argmax over the same scores is exact. The revision carries the distinction through. Algorithm 1 returns a status, certified or diagnostic, on every branch; its capacity branch returns "capacity limited" only when the status is certified and "positive-readout floor active" otherwise. Section 5.3 states that the second outcome does not show that the argmax head needs a wider store. Table 2 has a Status column and the verdict "positive floor" on its diagnostic rows, for example the CIFAR-100 projection and prototype bank at q=32, whose argmax risks of 0.484 and 0.607 lie below the floor of 0.680. The revision also characterizes the argmax separately, without extending the floor to it. Proposition 9 (Appendix D.1) bounds the argmax risk of a positive readout by the per-class mean and variance of its margin, through Cantelli's inequality. On four trained registers, with moments taken on the test split, it gives 0.300 to 0.451 against argmax errors of 0.104 to 0.270 on that split, where the Markov bound built on the column margin of Proposition 4 is vacuous (Table 11). That is a statement about the sample; Appendix D.1 also estimates the moments on validation data and shows where that estimate transfers and where it does not.
Q4 and W2. Necessary conditions, and the binary case. Every claim derived from the floor is a necessary condition: if εcap>ε⋆, no model with that store and a rule in the stated class reaches ε⋆. Section 4.4 now says that a floor below the target, zero included, says nothing about whether training reaches the target or whether widening would help. The reviewer is right about the binary case: the sentence that a zero floor rules out capacity as a source of error, and the Appendix E statement that widening can then be ruled out, are removed. For C=2 the floor is zero at every m≥2 and rules out no store size.
Q2 and W3. Encoder inefficiency against irreducible uncertainty. The class states alone cannot separate the two, and Appendix A.6 proves it with two problems that share one prior, one set of class states and one deployed rule, hence one decomposition, while their Bayes errors are 0 and e: a better encoder removes the whole gap of the first and none of the second. The revision adds a bound that separates them from one side (Proposition 10, Appendix I.3). For any classifier g of the input with error εg, the encoder that emits the basis state of g's prediction among the q heaviest classes has a best readout with error at most εcap+εg. So the part of a representation gap that the inputs force is at most εg, and εreg−εcap−εg lower-bounds the risk that another encoder into the same store removes. Both are differences of risks, not fractions of the gap. The bound needs no estimate of the Bayes error, a stronger reference tightens it, and the two problems above attain it. With the unbottlenecked linear head as g, averaged over its runs, the risk that some encoder into the same store removes is at least 0.029 to 0.321 for the released register on four of the five datasets (Table 25); on DermaMNIST the gap exceeds the reference by 0.010, within its sampling error. The bound is an existence statement: the encoder of its proof need not be one the register's parameterization represents, so it bounds the risk some encoder removes, and not the gain from retraining the register. A stabilized register trained longer has a smaller gap on all five, and it exceeds the reference on HyperKvasir alone, by 0.024; on the other four this reference no longer separates the remaining gap from input ambiguity.
run_reference.pyreproduces the table from the saved diagnostics.