- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The writing of this draft requires improvement, I will try to follow the notations with my understanding to summarize this study.
This draft attempts to solve a very special task(I highly doubt this falls into communication) where the representation of a text or image, assuming this is the embedding of one data sample, will be transformed and split into sub-blocks, each block is a fraction of the transformed embedding Eqs(1) and (2). If all of the blocks are available, then the final prediction can be done flawlessly. If some of the blocks are missing(during transmission?), then the final prediction can only be made based on the rest of the blocks S, Eq(3). The loss for the prediction can be computed on the blocks S, Eq(5). Assuming each block is included or missed at the same probability, uniform distribution, the expectation of the partial logits ai, (the output of weight and embedding Wh) is the same as the full version(no missing blocks). The variance of ai can be used as a gauge to choose the coordinate system, propositions 1, 2 and Eq(6).
Section 3.1, this draft claims that a simple gauge to choose Q would be minimizing the expected squared distance, Eq(7). My understanding in computer science is finding a transform matrix Q based on the least squares method. This gauge could be problematic when there exist missing blocks, the classification boundary could be changed, Eqs(8-10).
The newly proposed gauge is introduced in Section 4, the logit of the full version is kept and frozen is used as a gauge. The logit score differences between the highest class and the top-5 neighbour classes are used to choose coordinates, defined as margins, Eq(11) and Figure 2. Then the detailed design of the coordinates is very unclear, only the short paragraph between lines 231-236. Based on Figure 2 bottom right, I would guess the proposed gauge intends to minimize the margin variance.
The coordinates designed based on this proposed new gauge is claimed to be more robust under different missing rates compared against other gauges, Identity, fixed Hadamard, GlobalVAR and MaskedCE.
Critical strengths
The proposed new gauge might be important to solve the problem of missing blocks in communication.
Critical weaknesses
-
One main weakness of this draft is readability. The first sentence in abstract “Intermediate representations are often only partly available in split”. The first sentence in Section 1, “Modern inference systems often operate with incomplete intermediate representations”. After finishing the summary, I am still not sure if my guesses are correct about those terms in computer science, what intermediate representations? What modern systems? I will leave the details to the question section below.
-
Based on my understanding, this study intends to solve missing packets in communication, which does not fit into this field. Some references about quantization for LLMs also utilized matrix coordinate transforms, but they are not proposed for missing blocks. The research direction is completely different.
-
Assuming my understanding on Section 4 is correct, the new gauge is based on the soft output logits, or the differences(margins). This trick is widely used in computer science for distillation, calibration, or test-time adaption. I cannot see a clear contribution of this draft for computer science.
-
Assuming this new gauge indeed is important in communication, a more formal and scientific setup would be finding those related studies in communication and reproducing their experiments(making predictions based on missing packets?). Then applying this new gauge to compare against those studies for an apple-to-apple comparison. The other gauges used for comparison, line 256, Figures 4 and 5, have no references in the field. They look self-defined by this study, random Q? GlobalVar(Eq7) and supervised version of the proposed method, line 235. These baselines are not convincing(I could be wrong in communication literature).
-
Some terms are used without introduction, e.g. line 232, what is A?
Questions for authors
I try to summarize my questions in this section, maybe some of them are due to my ignorance in communication.
-
First sentence in abstract, “Intermediate representations”, what is it in this context? I guess its visual features before the classifier. First sentence in Section 1, “Modern inference systems often operate with incomplete intermediate representations”. Not sure which systems are referred here, most systems in CS do not consider missing packets.
-
Line 41, “if only part of a hidden representation is available, does the choice of representation coordinates matter?”, line 43, “For the complete predictor, the usual answer is no.”, this could be minor but a little confusing, the answer is outside the scope of the assumption, right?
-
Line 105, “(1)We show that representations that are exactly equivalent for complete inference can fail differently”, if my understanding about the missing packet is correct, then this is not surprising that different fractions could lead to different outcomes. Based on the claim, I wonder which experiments show this claim, Fig 1?
-
Line 123, Q ∈ RB×B, line 128, function-preserving representation gauges. I suppose Q is the gauge used to choose coordinates, and the newly proposed gauge is a new Q?
-
Line 130, “The gauge becomes visible only after some blocks are unavailable.” Could you explain what this means? I thought the gauge is designed, e.g., the proposed BoundaryGauge or MaskedCE(BoundaryGauge with label), if all of the blocks are available, no missing packets, then the gauge is not visible? I admit I don’t have a background in communication.
-
Line 147, “For a fixed gauge, let ci = ci(Q) denote the contribution of block i”, this seems to prove my guess of question 4, Q is a gauge, then what does the equation mean? Simply omitting the term Q? This is confusing without explanation. Based on my guess, based on Eq(1), ci seems to be the logit from block i. What does “contribution” mean, the weight for the logit? Please correct me if I am wrong.
-
Following Q6, line 162, Proposition 2, “For any u ∈ RC”, what is u here? Any explanation, what is the “the scalar projection” used for? The resulting ai seems to be important, but it is unclear what it is without knowing the context. Ai looks like linearly transformed logits?(I could be wrong). Then what does the alpha mean, “through the positive scalar α(B, k).”? The definition in Eq(6) has no explanation, I can see alpha is proportional to the missing rate.
-
I suppose the proposed gauge is based on the top-5 margins from Figure 2, how were the margins used for the new gauge? The only explanation is a short paragraph, lines 231-235. Then Q seems to be an operator this time Q(A) (vs ci(Q))? Then what is A in this context? The new coordinates, alternative to T in Eq(2)??
-
Based on Figure 2, I suppose the proposed gauge is looking for equal margins between the highest class and the other top-5. I see the overall entropy will remain the same, but the partial entropy (among the rest) could be higher. In other words, the proposed gauge is optimized to make the prediction EQUALLY wrong to the top-5 or the rest of the classes. Not exactly sure if this is the best way to retain information(from a perspective of distillation).
-
Following the above question, it could be verified by a list of solid and convincing experiments in the same field, missing packets for model prediction. However, I did not see related works in the experiment comparison. The benchmarked gauges are also from this study, and look like ablated design. I wonder what the state-of-the-art studies in this field and how does this draft compare to those literatures.