- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues whether nearly lossless representation of the current ViT state in a subspace guarantees that the remaining layers can continue computing within that same subspace. Further, the authors introduce the representation–continuation gap. Even when the entry state is preserved almost exactly, restricting subsequent residual updates to the same subspace can degrade performance. Further, to reduce the gap, authors propose FSC, which learns a few complementary directions while keeping the original model frozen. They demonstrate that FSC improves performance across 8 .
Critical strengths
-
Well-controlled experiments. The paper isolate entry reconstruction errors and providing convincing evidence for the representation–continuation gap.
-
Thorough evaluation. Across 8 ViT settings, FSC keeps the dominant entry directions and learns the shared source of complementary directions while keeps parameter efficient. Results on extensive experiments demonstrate the effectiveness of their models.
Critical weaknesses
- Poor writing and presentation. The overall writing quality is poor, making the paper’s main ideas unnecessarily difficult to follow. Several terms and expressions in the introduction are insufficiently explained. Here are some examples: “representation sufficiency” and “continuation sufficiency” (lines 53 and 74–76), “current entry support” (line 99), and (line 97) and " define and test continuation sufficiency"(line 111). The paper needs substantial revision to improve its clarity, organization, and explanation of key concepts before acceptance.
Questions for authors
See weakness
Response to Reviewer E2n9
Thank you for the review, and for finding the experiments well controlled and the evaluation thorough. We are sorry that the submitted text was hard to follow. Your list of the exact lines where it failed guided our rewriting of the introduction.
In short. Every term you list is now defined in the introduction, on one running example. The abstract, the introduction and the conclusion are rewritten, the main text names four comparison arms, and the changed text is in blue in the revised PDF.
Weakness. Terms in the introduction were not explained (lines 53, 74–76, 97, 99 and 111 of the submission). The running example is an image entering the last two blocks of a ViT-B, where the state is a matrix of 197 tokens by 768 channels. The revised introduction defines the terms on this example (p. 1, lines 48–53, and pp. 2–3, lines 104–108).
Comparison arms. The main text compares four arms (p. 2, lines 73–78). Each starts from the same projected entry.
Organization. The main text now follows one line of argument. The introduction states the question on the running example. Section 5.1 (p. 6) shows that a subspace containing the whole entry still fails the last blocks. Section 5.2 (p. 7) shows that eight directions decide most of the outcome. The new Section 5.3 (pp. 7–8) shows that the loss comes from holding the subspace fixed across depth, and Section 6 (pp. 8–10) presents FSC.
Soundness. The revision adds four controls that test the result further. Refitting the subspace after every update brings accuracy within 0.6 points of free continuation in seven of eight settings, so the gap comes from holding the entry subspace fixed. In a ViT sliced by SliceGPT at 25%, the gap of the complete entry basis grows from 16.25 to 21.75 points on AugReg-B and from 33.25 to 64.40 on CLIP-FT. With more tokens than channels, the comparisons at 64 directions keep their sign. On GPT-2 and GPT-2 medium with 256 tokens, the complete entry basis moves the output by 1.06 and 1.80 nats of KL.
We hope the revised text is now easy to follow. If any passage is still unclear, please tell us where and we will fix it. If the revision resolves your concern about the presentation, we would be grateful if you would reconsider your rating.