acceptodds
Under review as a conference paper at ICLR 2027

Do Functionally Equivalent Representations Fail Alike? Task-Consistent Gauges under Partial Availability

Abstract

Intermediate representations are often only partly available in split, adaptive, or progressive inference. If two coordinate systems implement exactly the same complete predictor, should they also fail alike when some representation blocks are missing? We show that they need not: function-preserving rotations can leave every complete prediction unchanged yet produce sharply different partial predictions. Under fixed-cardinality availability, partial logits and pairwise class margins are unbiased estimates of their complete counterparts. Their variance separates how much representation remains from how task-relevant evidence is distributed across coordinates. This explains why reducing global logit-estimation variance can still hurt classification. The nearby margins that determine the decision may become less stable. We therefore introduce BoundaryGauge, which preserves the complete predictor while selecting coordinates that stabilize those margins, without ground-truth labels or training on partial masks. At 50% availability, the same fixed Hadamard gauge changes accuracy by −13.5 points for one ResNet50 predictor but +24.9 points for another. BoundaryGauge adapts successfully to both. Across four architectures and 48 frozen-predictor cases, each optimized gauge improves mean, P05, and exact worst-subset accuracy across all three tested availability levels. Partial robustness therefore depends not only on the complete function, but on how its decision geometry is distributed across coordinates.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, partial availability, function-preserving transformations, representation geometry, margin stability, split inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bMhxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The writing of this draft requires improvement, I will try to follow the notations with my understanding to summarize this study.

This draft attempts to solve a very special task(I highly doubt this falls into communication) where the representation of a text or image, assuming this is the embedding of one data sample, will be transformed and split into sub-blocks, each block is a fraction of the transformed embedding Eqs(1) and (2). If all of the blocks are available, then the final prediction can be done flawlessly. If some of the blocks are missing(during transmission?), then the final prediction can only be made based on the rest of the blocks S, Eq(3). The loss for the prediction can be computed on the blocks S, Eq(5). Assuming each block is included or missed at the same probability, uniform distribution, the expectation of the partial logits ai, (the output of weight and embedding Wh) is the same as the full version(no missing blocks). The variance of ai can be used as a gauge to choose the coordinate system, propositions 1, 2 and Eq(6).

Section 3.1, this draft claims that a simple gauge to choose Q would be minimizing the expected squared distance, Eq(7). My understanding in computer science is finding a transform matrix Q based on the least squares method. This gauge could be problematic when there exist missing blocks, the classification boundary could be changed, Eqs(8-10).

The newly proposed gauge is introduced in Section 4, the logit of the full version is kept and frozen is used as a gauge. The logit score differences between the highest class and the top-5 neighbour classes are used to choose coordinates, defined as margins, Eq(11) and Figure 2. Then the detailed design of the coordinates is very unclear, only the short paragraph between lines 231-236. Based on Figure 2 bottom right, I would guess the proposed gauge intends to minimize the margin variance.

The coordinates designed based on this proposed new gauge is claimed to be more robust under different missing rates compared against other gauges, Identity, fixed Hadamard, GlobalVAR and MaskedCE.

Critical strengths

The proposed new gauge might be important to solve the problem of missing blocks in communication.

Critical weaknesses
  1. One main weakness of this draft is readability. The first sentence in abstract “Intermediate representations are often only partly available in split”. The first sentence in Section 1, “Modern inference systems often operate with incomplete intermediate representations”. After finishing the summary, I am still not sure if my guesses are correct about those terms in computer science, what intermediate representations? What modern systems? I will leave the details to the question section below.

  2. Based on my understanding, this study intends to solve missing packets in communication, which does not fit into this field. Some references about quantization for LLMs also utilized matrix coordinate transforms, but they are not proposed for missing blocks. The research direction is completely different.

  3. Assuming my understanding on Section 4 is correct, the new gauge is based on the soft output logits, or the differences(margins). This trick is widely used in computer science for distillation, calibration, or test-time adaption. I cannot see a clear contribution of this draft for computer science.

  4. Assuming this new gauge indeed is important in communication, a more formal and scientific setup would be finding those related studies in communication and reproducing their experiments(making predictions based on missing packets?). Then applying this new gauge to compare against those studies for an apple-to-apple comparison. The other gauges used for comparison, line 256, Figures 4 and 5, have no references in the field. They look self-defined by this study, random Q? GlobalVar(Eq7) and supervised version of the proposed method, line 235. These baselines are not convincing(I could be wrong in communication literature).

  5. Some terms are used without introduction, e.g. line 232, what is A?

Questions for authors

I try to summarize my questions in this section, maybe some of them are due to my ignorance in communication.

  1. First sentence in abstract, “Intermediate representations”, what is it in this context? I guess its visual features before the classifier. First sentence in Section 1, “Modern inference systems often operate with incomplete intermediate representations”. Not sure which systems are referred here, most systems in CS do not consider missing packets.

  2. Line 41, “if only part of a hidden representation is available, does the choice of representation coordinates matter?”, line 43, “For the complete predictor, the usual answer is no.”, this could be minor but a little confusing, the answer is outside the scope of the assumption, right?

  3. Line 105, “(1)We show that representations that are exactly equivalent for complete inference can fail differently”, if my understanding about the missing packet is correct, then this is not surprising that different fractions could lead to different outcomes. Based on the claim, I wonder which experiments show this claim, Fig 1?

  4. Line 123, Q ∈ RB×B, line 128, function-preserving representation gauges. I suppose Q is the gauge used to choose coordinates, and the newly proposed gauge is a new Q?

  5. Line 130, “The gauge becomes visible only after some blocks are unavailable.” Could you explain what this means? I thought the gauge is designed, e.g., the proposed BoundaryGauge or MaskedCE(BoundaryGauge with label), if all of the blocks are available, no missing packets, then the gauge is not visible? I admit I don’t have a background in communication.

  6. Line 147, “For a fixed gauge, let ci = ci(Q) denote the contribution of block i”, this seems to prove my guess of question 4, Q is a gauge, then what does the equation mean? Simply omitting the term Q? This is confusing without explanation. Based on my guess, based on Eq(1), ci seems to be the logit from block i. What does “contribution” mean, the weight for the logit? Please correct me if I am wrong.

  7. Following Q6, line 162, Proposition 2, “For any u ∈ RC”, what is u here? Any explanation, what is the “the scalar projection” used for? The resulting ai seems to be important, but it is unclear what it is without knowing the context. Ai looks like linearly transformed logits?(I could be wrong). Then what does the alpha mean, “through the positive scalar α(B, k).”? The definition in Eq(6) has no explanation, I can see alpha is proportional to the missing rate.

  8. I suppose the proposed gauge is based on the top-5 margins from Figure 2, how were the margins used for the new gauge? The only explanation is a short paragraph, lines 231-235. Then Q seems to be an operator this time Q(A) (vs ci(Q))? Then what is A in this context? The new coordinates, alternative to T in Eq(2)??

  9. Based on Figure 2, I suppose the proposed gauge is looking for equal margins between the highest class and the other top-5. I see the overall entropy will remain the same, but the partial entropy (among the rest) could be higher. In other words, the proposed gauge is optimized to make the prediction EQUALLY wrong to the top-5 or the rest of the classes. Not exactly sure if this is the best way to retain information(from a perspective of distillation).

  10. Following the above question, it could be verified by a list of solid and convincing experiments in the same field, missing packets for model prediction. However, I did not see related works in the experiment comparison. The benchmarked gauges are also from this study, and look like ablated design. I wonder what the state-of-the-art studies in this field and how does this draft compare to those literatures.

Reviewer_bDk3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper deals with the context of a distributed neural network, where in a feed-forward NN a set of activations are transmitted, but some activation sets, or blocks, are lost before processing continues. The question of how to minimize loss of performance when some blocks are lost is asked. The idea of a linear transform on the blocks is proposed, rotating the coordinate system so that if blocks are lost, the importance of all blocks are "equal", minimizing potential performance loss from some blocks being absent. Experiments are provided showing the advantages of this method.

Critical strengths

Paper is straightforward and easy to follow. Paper seems to have modest but real advantaged in experimentation.

Critical weaknesses
  • The paper's method seems a bit simple. Correct me if I am wrong, but the method simply uses a calibration step where no blocks are missing to find a linear transform that spreads variance equally among all blocks, yes?
  • I would recommend more orientation on split inference and the datasets/experimental setup, for the reader.
Questions for authors

Can the authors address the seeming low novelty/impact of the proposed algorithm?

Reviewer_HGi8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

The authors study what happens when only part of a hidden representation is available at inference time, e.g., when some blocks of features are lost during transmission in split inference. They split the penultimate representation into blocks and apply an orthogonal transformation that mixes these blocks together with a compensating change to the linear classification head, such that predictions using all representational blocks are unchanged. They also show that such transformations (which they call “gauges”) can nonetheless lead to very different predictions when some blocks are missing. Under uniformly random subsets of fixed size, they show that partial logits and class margins are unbiased. Moreover, their variance depends on how evenly the evidence for a class is spread across blocks. Based on this observation, they propose BoundaryGauge, which learns an orthogonal transformation that reduces the variance of the margins between the predicted class and its top competitors. This is done without labels or training on partial masks. The authors evaluate BoundaryGauge on frozen CIFAR-100 probes and native ImageNet classifiers across four architectures.

Critical strengths
  1. The core observation itself is interesting. Representations that are exactly equivalent under complete inference, and that rotation-invariant similarity measures (e.g., CKA, Procrustes) would treat as identical, behave differently when part of the representation is missing. This gives a concrete functional consequence of the choice of basis, which relates to recent work arguing that neural representations have privileged axes (Khosla et al., 2024).
  2. Reframing partial prediction as a sampling without replacement problem is a clever way to frame the problem. The variance decomposition (Prop. 2) is neat -- it allows one to separate how much of a representation remains from how the evidence is distributed across blocks. The proofs presented in the appendix are short and correct.
  3. Despite using neither labels nor partial masks during optimization, BoundaryGauge comes close to MaskedCE on native ResNet50, which is trained with labels and masked samples.

References:

  1. Khosla, M., Williams, A. H., McDermott, J., & Kanwisher, N. (2024). Privileged representational axes in biological and artificial neural networks. bioRxiv.
Critical weaknesses
  1. I found the paper difficult to follow, starting with the abstract, where unfortunately I understood almost nothing without giving the entire paper a first read. A lot of terms here and in the main text are used before they are defined (or not at all), for instance "Hadamard gauge", "hard native ResNet50", "block-axis gauge" etc. to name a few. A "gauge" is only named at L128, without any explanation. For readers unfamiliar with the term, it would be beneficial to add a sentence of clarification that it refers to a function preserving change of coordinates, and more specifically, here an orthogonal mixing of blocks compensated in the head. Next, Fig. 1 introduces much of this jargon before the text does, and it doesn't help orient the reader (e.g., "critical-margin contributions", "decision evidence").

  2. Next, I found Sec. 2 hard to follow. It is not stated what rr is -- I assume the block size, or what ⊗\otimes denotes (I assume the Kronecker product). As far as I understand, the gauge is an orthogonal transform that mixes whole blocks, whose effect is reversed by applying its transpose to the linear head. Hence, the invariance under complete inference is immediate. The interesting part is only what happens once the masking is inserted between the two, and I think the paper would benefit from saying this plainly. Similarly, the margin decomposition in Eq. 8 would be much easier to follow with a simple example, e.g., a margin that comes entirely from one block versus a margin spread evenly across blocks. The notation overall is dense and non-standard.

  3. My main substantive concern is the choice of baseline. The headline claim that BoundaryGauge (BG) improves mean, 5th5^\text{th} percentile and worst-subset accuracy across all 4848 frozen predictor combinations is measured against the paired random gauge that was used for initialization. I see random rotations as a weak baseline, since, as the authors show, generic mixing such as Hadamard can be harmful. Relative to an identity transform (Fig. 3a), BG gives essentially no improvements on the frozen ResNet50 probe. For the native ResNet50, the Hadamard line in Fig. 3b appears to be hidden behind BG (see question 6), which would suggest that BG recovers the Hadamard solution. If this is indeed the case, BG matches the better of two fixed training-free gauges in each of the highlighted regimes. I also think identity and Hadamard should be included in Table 1 and Fig. 6 for all architectures, together with a simple baseline that picks between Identity and Hadamard using the BoundaryGauge objective on the calibration set.

  4. Relatedly, the claimed gains seem to hinge on one model. Native ResNet50 is the only "hard" case, while ConvNeXt-T, ViT-B/16 and Swin-T barely change (Appendix C.3). The claim that minimizing global logit variance can move in the wrong direction (Sec. 6.2) is also only shown for native ResNet50, whereas Appendix C.4 reports that GlobalVar improves accuracy in all 144144 frozen-probe combinations. Also, GlobalVar is not shown in Fig. 6. It is also not explained why the identity basis performs so poorly for native ResNet50 in the first place, which I think would be the most informative analysis to add.

  5. Last, I am also not sure how well the method fits the settings which are used to motivate it. Since each transformed block is a combination of all original blocks, the complete representation has to be computed before the gauge can be applied. This seems to rule out progressive or adaptive pipelines where the representation itself is only partially computed (L40-41), thus leaving mainly loss in transmission. The stated theory also assumes that the retained blocks are a uniformly random subset of fixed size, which I imagine does not hold for ordered progressive transmissions (per my understanding, one of the practical use cases of the paper). The note at L141 that "systems-level compute savings depend on how those blocks are produced" also highlights that there is no systems-level evaluation, despite the split-inference motivation.

Questions for authors
  1. Can the authors add Identity and Hadamard to Table 1 and Fig. 6 for all architectures, and compare BoundaryGauge to simply choosing between Identity and Hadamard on the calibration set?
  2. Do the authors have an idea as to why the identity basis performs so poorly for native ResNet50? What do the block contributions look like under identity?
  3. Which 1414 gauges are included in Fig. 4? Also, is margin variance computed on the same held-out images and subsets as the flip probability? If so, I would imagine that the two quantities are closely linked by design. Perhaps a Pearson correlation / within-method correlations would help show whether the relationship holds beyond the separation between methods.
  4. L231: what is Q0\boldsymbol{Q}_0? How is A\boldsymbol{A} updated? I assume this is an exponential-map parametrization of orthogonal matrices with Q0\boldsymbol{Q}_0 as a random orthogonal initialization (Appendix B.3). If this is indeed the case, it should be stated in the main text. Also, since exp⁡(A−A⊤)\exp{(\boldsymbol{A} - \boldsymbol{A}^\top)} has determinant +1+1, the reachable gauges retain the determinant of Q0\boldsymbol{Q}_0. If Q0\boldsymbol{Q}_0 is sampled uniformly from O(B)O(B), then det⁡(Q0)=−1\det{(\boldsymbol{Q}_0)} = -1 with probability 1/2, in which case the identity gauge cannot be reached. Is this accounted for?
  5. In which availability regime is BoundaryGauge learned? As far as I can tell, the objective only uses complete representations and the same gauge is evaluated at all 33 availability levels. If this is the case, it should be stated clearly in the main text.
  6. Fig. 3b: why is the Hadamard line not visible for the native ImageNet predictor? Is it overlapping with BoundaryGauge?
  7. The abstract compares “−13.5−13.5 points for one ResNet50 predictor” with “+24.9+24.9 points for another,” but from what I can tell, these results are for a CIFAR-100 linear probe and the ImageNet classifier. Since both are evaluated on different datasets, the difference is not only due to the predictor. As currently written, the abstract makes it sound as if the difference comes from the predictor alone

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.