- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper examines an adaptive running average of some constant input, of the form
.
It shows several convergence results and properties of the converged solutions. It then examines these in some practical benchmarks.
Critical strengths
None.
Critical weaknesses
- This paper is very dense and hard to read, as can be observed already from the abstract and introduction. The main issue for me is that many sentences (like the first sentence in the abstract: "Positive averaging can converge while amplifying input changes.") are difficult to interpret unless you understand the context for them, which appears only later (this context-awareness is something the authors can do easily, but not a new reader). There are many other issues stemming from this. For example, the proof of the theoretical results is hard to find: there is no clear place that is labeled as "Proof of Theorem 1" in the paper or appendix, and the proof sketch that appears immediately after the Theorem is not identified as such. Moreover, the motivation for each result is not stated clearly, so the significance or novelty of the different results is not clear.
- Unless I missed something, the convergence result in Theorem 1 follows from standard contraction arguments. The authors also show the sharpness of this result using a simple (N=2,d=1) example, but I don't understand: what is the new theoretical insight or the practical significance here?
Questions for authors
None.
2 comments
Response to Reviewer M8M5 (2/2): the two-token example, practical significance, and soundness
Why a two-token scalar example. A sharpness statement needs one counterexample, and the smallest is the most transparent: the threshold already fails with two scalar tokens. It is not special to that size. Appendix A.2.2 extends it to every N ≥ 2 and d ≥ 1 by embedding the two tokens along a unit vector and giving the remaining tokens identity weights and zero anchors. The same failure appears with a published rule on real data. In Fig. 1b, non-local means inside RED reaches two different stable limits from one observed photograph. In the revision we also add noise to that photograph: the two limits persist in all 100 noisy copies at each of four noise levels, and at the three larger levels the noisy image and the clean solution, used as starts, reach different limits in 22, 57 and 92 of 100 draws (Appendix A.7.3).
Practical significance. Weighted averaging with weights computed from the data is the core of kernel denoisers, of graph propagation and of attention, and the belief that such averaging is safe is what the paper tests. The margin m = β − αRL_P sets how firmly the input is anchored against the gain of the path through the weights. Wherever this structure appears, a positive margin guarantees one limit and a bounded response, and no better guarantee follows from α, R and L_P alone.
The converged average with its weights held fixed can never amplify. It is also the object analyzed when kernel weights are frozen to prove convergence. Gavaskar and Chaudhury (2020), for plug-and-play ISTA with kernel denoisers, compute the kernel during the first iterations and then freeze it. Our results show that this frozen object is the wrong one for sensitivity. In Fig. 1c its gain is 0.999, and the gain of the loop that actually runs is 19.12 along a selected direction.
That direction is selected. We therefore also measured unstructured noise, and we report it as it came out (Section 4, Appendix A.7.3). With independent random noise in place of the selected perturbation:
- The root-mean-square gain is 0.38 for the rebuilt loop and 0.155 for the frozen one.
- Over four noise levels, the largest pixel change is 0.92 to 1.51 times the largest input change in median, against 0.53 frozen. It exceeds the largest input change in 27 to 73 of 100 draws. With frozen weights it never does.
So the 19-fold figure is a worst-direction statement, relevant to a feedback loop or an adversary that selects the direction. Under random noise the loop attenuates on average, at 1.6 to 2.6 times the response of its frozen average. The revision says this in the abstract and the introduction.
Three further additions bear on significance. The loss of predictions recurs for a supervised ViT-S/16 and a supervised ResNet-50, and on CIFAR-100 (Appendix A.4.10). A 48-setting grid shows that this loss is governed by the mixing weight and begins near α = 0.8 (Appendix A.4.9). A standard transformer is not this loop, but it shares the ingredient: in a trained ViT-S/16 the derivative through the attention pattern is 0.44 to 1.07 times the fixed-pattern part in every layer (Appendix A.7.1, Table 16).
Soundness. The review marks soundness 1, but its text raises presentation and novelty only. If a step seems wrong or unsupported, please point us to it and we will address it in this discussion.
Response to Reviewer M8M5 (1/2): readability, proofs, and what the paper claims
Thank you for the review. We agree that the paper was dense and that the abstract assumed context that appears only later. Your example, the first sentence of the abstract, is fair. Line numbers refer to the original submission. Section numbers refer to the revised PDF, where changes are marked in blue.
Readability. The abstract now opens: "Many iterative systems repeatedly replace each token by a weighted average of all tokens, recompute the weights from the current state, and blend in the original input." The introduction displays the iteration before any result. It then lists the contributions, each with the reason it matters. The supplementary material also holds an optional six-minute narrated overview of the paper, built from its own figures and data; it adds no result.
Where the proofs are. Full proofs were in Appendix A.2 of the supplementary PDF: A.2.1 for the contraction half of Theorem 1, A.2.2 for sharpness, A.2.5 for Proposition 1, and A.2.6 for the all-budget bound. You are right on two counts. The two paragraphs after Theorem 1 are proof sketches and were not labeled as such, and the appendix headings did not name the statements they prove. In the revision the sketches are labeled, every statement points to its full proof, each proof heading in Appendix A.2 names the statement it proves, and the appendix follows the references in the same PDF.
Theorem 1 and standard contraction arguments. We agree, and the paper says so: "The sufficient contraction argument is classical" (lines 45–47). We state it because its margin m = β − αRL_P is the quantity in which the new results are expressed. The revision separates the two parts: Theorem 1 is the classical certificate, marked as classical, and Theorem 2 is the sharpness result. The paper claims four results. They test two beliefs that the revised introduction names: that an average cannot amplify a change (result 2), and that a loop that converges is stable (results 1, 3 and 4). Both beliefs hold when the weights are fixed.