acceptodds
Under review as a conference paper at ICLR 2027

When Averaging Amplifies: Stability and Sensitivity in Adaptive Mixing

Abstract

Positive averaging can converge while amplifying input changes. We study adaptive mixing that retains the original input, as in non-local-means RED denoising and related graph and equilibrium architectures. A class-sharp convergence certificate gives an optimal input-response bound: smooth stochastic mixers approach that bound while their frozen averages (realized weights held fixed) approach identity, and cross-token responses can be negative. A photographic RED/NLM example and ordinary softmax kernels exhibit amplification. Convergence also need not preserve predictions. At a prespecified strong-mixing setting, adaptive mixing of full-dimensional DINO features in 32-image contexts reduces accuracy on 512 matched CIFAR-10 queries from 88.09% to 35.87%; freezing input weights retains 88.02%. An exact score decomposition and context removal identify recoverable interference: the original classifier recovers 87.96%. Finally, intermediate iteration budgets can destabilize downstream feedback even when a longer solve is stable. We give a sufficient condition covering every budget. These results separate guarantees for convergence, input response, task fidelity, and downstream use.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
adaptive mixing, fixed-point iterations, stability analysis, input sensitivity, self-attention, graph neural networks, oversmoothing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_M8M5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines an adaptive running average of some constant input, of the form

Xt+1=αP(Xt)Xt+βAX_{t+1}=\alpha \mathcal{P}(X_t)X_t + \beta A.

It shows several convergence results and properties of the converged solutions. It then examines these in some practical benchmarks.

Critical strengths

None.

Critical weaknesses
  1. This paper is very dense and hard to read, as can be observed already from the abstract and introduction. The main issue for me is that many sentences (like the first sentence in the abstract: "Positive averaging can converge while amplifying input changes.") are difficult to interpret unless you understand the context for them, which appears only later (this context-awareness is something the authors can do easily, but not a new reader). There are many other issues stemming from this. For example, the proof of the theoretical results is hard to find: there is no clear place that is labeled as "Proof of Theorem 1" in the paper or appendix, and the proof sketch that appears immediately after the Theorem is not identified as such. Moreover, the motivation for each result is not stated clearly, so the significance or novelty of the different results is not clear.
  2. Unless I missed something, the convergence result in Theorem 1 follows from standard contraction arguments. The authors also show the sharpness of this result using a simple (N=2,d=1) example, but I don't understand: what is the new theoretical insight or the practical significance here?
Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Response to Reviewer M8M5 (1/2): readability, proofs, and what the paper claims

Thank you for the review. We agree that the paper was dense and that the abstract assumed context that appears only later. Your example, the first sentence of the abstract, is fair. Line numbers refer to the original submission. Section numbers refer to the revised PDF, where changes are marked in blue.

Readability. The abstract now opens: "Many iterative systems repeatedly replace each token by a weighted average of all tokens, recompute the weights from the current state, and blend in the original input." The introduction displays the iteration before any result. It then lists the contributions, each with the reason it matters. The supplementary material also holds an optional six-minute narrated overview of the paper, built from its own figures and data; it adds no result.

Where the proofs are. Full proofs were in Appendix A.2 of the supplementary PDF: A.2.1 for the contraction half of Theorem 1, A.2.2 for sharpness, A.2.5 for Proposition 1, and A.2.6 for the all-budget bound. You are right on two counts. The two paragraphs after Theorem 1 are proof sketches and were not labeled as such, and the appendix headings did not name the statements they prove. In the revision the sketches are labeled, every statement points to its full proof, each proof heading in Appendix A.2 names the statement it proves, and the appendix follows the references in the same PDF.

Theorem 1 and standard contraction arguments. We agree, and the paper says so: "The sufficient contraction argument is classical" (lines 45–47). We state it because its margin m = β − αRL_P is the quantity in which the new results are expressed. The revision separates the two parts: Theorem 1 is the classical certificate, marked as classical, and Theorem 2 is the sharpness result. The paper claims four results. They test two beliefs that the revised introduction names: that an average cannot amplify a change (result 2), and that a loop that converges is stable (results 1, 3 and 4). Both beliefs hold when the weights are fixed.

  1. Sharpness (Theorem 2). Just past m = 0, a smooth row-stochastic weight rule can have two stable fixed points. The classical condition is therefore not an artifact of a loose proof: nothing weaker by a constant factor in α, R and L_P guarantees uniqueness.
  2. Positive averages approach the response bound (Proposition 1). Inside the certified regime the fixed point moves by at most β/m per unit of input change. We construct smooth positive weight rules that approach β/m, with negative derivatives across tokens, while the converged average with weights held fixed approaches the identity.
  3. Convergence does not preserve predictions (Section 5). On identical inputs, with the same kernel and α, weights built once retain 88.02% accuracy and weights rebuilt to convergence give 35.87%. Removing the shared context term recovers 87.96%. In the revision the loss recurs for two further encoders and a second dataset, at 36.6 to 58.6 points in five new encoder–dataset pairs.
  4. A finite budget can destabilize downstream use (Section 6). An outer loop is unstable at K = 16 inner updates and stable at K = 128. The condition γ < m/β suffices for every budget.
AuthorsOpenReview ↗

Response to Reviewer M8M5 (2/2): the two-token example, practical significance, and soundness

Why a two-token scalar example. A sharpness statement needs one counterexample, and the smallest is the most transparent: the threshold already fails with two scalar tokens. It is not special to that size. Appendix A.2.2 extends it to every N ≥ 2 and d ≥ 1 by embedding the two tokens along a unit vector and giving the remaining tokens identity weights and zero anchors. The same failure appears with a published rule on real data. In Fig. 1b, non-local means inside RED reaches two different stable limits from one observed photograph. In the revision we also add noise to that photograph: the two limits persist in all 100 noisy copies at each of four noise levels, and at the three larger levels the noisy image and the clean solution, used as starts, reach different limits in 22, 57 and 92 of 100 draws (Appendix A.7.3).

Practical significance. Weighted averaging with weights computed from the data is the core of kernel denoisers, of graph propagation and of attention, and the belief that such averaging is safe is what the paper tests. The margin m = β − αRL_P sets how firmly the input is anchored against the gain of the path through the weights. Wherever this structure appears, a positive margin guarantees one limit and a bounded response, and no better guarantee follows from α, R and L_P alone.

The converged average with its weights held fixed can never amplify. It is also the object analyzed when kernel weights are frozen to prove convergence. Gavaskar and Chaudhury (2020), for plug-and-play ISTA with kernel denoisers, compute the kernel during the first iterations and then freeze it. Our results show that this frozen object is the wrong one for sensitivity. In Fig. 1c its gain is 0.999, and the gain of the loop that actually runs is 19.12 along a selected direction.

That direction is selected. We therefore also measured unstructured noise, and we report it as it came out (Section 4, Appendix A.7.3). With independent random noise in place of the selected perturbation:

  • The root-mean-square gain is 0.38 for the rebuilt loop and 0.155 for the frozen one.
  • Over four noise levels, the largest pixel change is 0.92 to 1.51 times the largest input change in median, against 0.53 frozen. It exceeds the largest input change in 27 to 73 of 100 draws. With frozen weights it never does.

So the 19-fold figure is a worst-direction statement, relevant to a feedback loop or an adversary that selects the direction. Under random noise the loop attenuates on average, at 1.6 to 2.6 times the response of its frozen average. The revision says this in the abstract and the introduction.

Three further additions bear on significance. The loss of predictions recurs for a supervised ViT-S/16 and a supervised ResNet-50, and on CIFAR-100 (Appendix A.4.10). A 48-setting grid shows that this loss is governed by the mixing weight and begins near α = 0.8 (Appendix A.4.9). A standard transformer is not this loop, but it shares the ingredient: in a trained ViT-S/16 the derivative through the attention pattern is 0.44 to 1.07 times the fixed-pattern part in every layer (Appendix A.7.1, Table 16).

Soundness. The review marks soundness 1, but its text raises presentation and novelty only. If a step seems wrong or unsupported, please point us to it and we will address it in this discussion.

Reviewer_954eOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies adaptive mixing systems that repeatedly average evolving states with an original input while rebuilding the mixing weights from the current state. It shows that convergence of the inner iteration and positivity of the averaging weights do not guarantee low sensitivity to input changes, preservation of predictions, or stability of a downstream feedback loop. The paper gives a contraction certificate with a sharp boundary, derives an input response bound, and constructs smooth positive mixers whose frozen averages are nearly identity while their adaptive responses are strongly amplified. Experiments on denoising and image classification demonstrate amplification, class margin reversal, recoverable context interference, and instability at some finite iteration budgets even when the final equilibrium is stable.

Critical strengths
  1. The paper clearly separates several notions that are often conflated: convergence of the state iteration, sensitivity of the equilibrium to the input, preservation of task predictions, and stability of downstream feedback. This conceptual separation is useful and is supported by distinct mathematical objects and sufficient conditions.

  2. The theory is paired with targeted controls rather than only aggregate performance results. The sharpness construction, adaptive versus frozen response comparison, context removal experiment, score decomposition, and finite budget feedback analysis provide coherent evidence for the central claims.

Critical weaknesses
  1. The main empirical validation is narrow. The classification study uses one pretrained feature encoder, fixed affine heads, one data set, and a selected strong mixing regime. The denoising and softmax examples establish possibility, but they do not show how common or consequential the effect is in practical trained architectures.

  2. The convergence certificate relies on a common invariant domain and a global Lipschitz bound for the adaptive weight builder. These quantities may be difficult to verify for realistic kernels and attention systems, and the paper acknowledges that the certificate can be conservative. The sharpness result is over a broad constructed builder class, so its practical relevance to particular architectures remains uncertain.

Questions for authors
  1. Can the authors evaluate the classification effect across additional encoders, feature dimensions, data sets, classifier heads, and mixing architectures? This would clarify whether the observed margin reversal is specific to the chosen DINO feature setting or is a broader consequence of adaptive mixing.

  2. How can the Lipschitz constant, invariant radius, and contraction margin be estimated or certified for realistic non local means kernels, attention mixers, and graph propagation rules? Please distinguish computable sufficient bounds from empirical estimates based on sampled trajectories.

  3. Can the authors quantify the prevalence of amplification outside the selected response maxima? A distribution of response gains over randomly sampled contexts, inputs, and perturbation directions would help distinguish a generic risk from an adversarial possibility.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 954e (1/3): the rating, and empirical scope

Thank you for the careful reading. Your summary states the paper's claims exactly. We agree with both weaknesses as statements of scope, and we ran the experiments they call for. Line numbers refer to the original submission. Section and table numbers refer to the revised PDF, where changes are marked in blue.

A question about the rating. Your review lists two substantive strengths. It rates presentation "clear and easy to follow" and informativeness "a substantially deeper understanding", and it marks the soundness concerns minor or fixable. The overall rating is 1, clear rejection. The form is new this year, so we ask only to be sure we answer your main concern. Does the rating reflect your assessment, and if so, which weakness is decisive?

Narrow empirical validation (weakness 1, question 1): other encoders and a second dataset. We repeated the matched-query protocol unchanged for three encoders on CIFAR-10 and CIFAR-100: the DINO ViT-S/16 of the submission, a supervised ViT-S/16 of the same architecture, and a supervised ResNet-50. The design was recorded before any outcome was computed, and every pair and setting is reported (Appendix A.4.10, Tables 13 and 14). At the same prespecified setting (α, s) = (0.95, 30) with uniform companions, accuracy (%) on 512 queries is:

EncoderDatasetInputFrozenAdaptiveContext removed
DINO ViT-S/16CIFAR-1088.0988.0235.8787.96
DINO ViT-S/16CIFAR-10069.9269.7928.7869.79
Supervised ViT-S/16CIFAR-1096.0996.0937.5095.70
Supervised ViT-S/16CIFAR-10084.3884.1144.1484.38
Supervised ResNet-50CIFAR-1090.0489.6536.6590.04
Supervised ResNet-50CIFAR-10070.7070.3134.1170.70

The first row re-extracts the submission's features on different hardware and reproduces Table 2 (Table 1 of the submission). In the five new pairs the adaptive loss is 36.6 to 58.6 points, and every paired 95% interval excludes zero. Frozen weights and context removal stay within 0.4 points of the input.

Nearest companions are less uniform across pairs. The adaptive loss is 2.0 points for the supervised ViT-S/16 on CIFAR-10 and 9.6 to 23.8 points for the other four new pairs. On CIFAR-100, context removal with nearest companions leaves 2.9 to 4.1 points unrecovered.

All three encoders are fixed and pretrained, and none is trained here. The study extends the result to other feature banks and a second label set. Trained mixing layers remain outside the scope, as the Scope paragraph says (lines 475–480).

Why this setting, and a first step toward trained architectures. Mixing test examples with attention weights is used at inference in transductive few-shot classification, where one pass helps and repeating it lowers accuracy (Cui and Guo, 2021; now cited in Section 5). We hold the encoder and head fixed so that every change in a prediction is the mixing's doing. A standard transformer is not an instance of the loop, but it shares the ingredient that drives our results, weights computed from the state. We therefore measured that ingredient in the DINO ViT-S/16 itself (Appendix A.7.1, Table 16). Along random directions the derivative through the attention pattern is 0.44 to 1.07 times the fixed-pattern part in every layer, and rebuilding the patterns raises the end-to-end largest singular value by a median factor of 1.38. We observe no amplification of the kind in Fig. 1 there, and we claim none.

AuthorsOpenReview ↗

Response to Reviewer 954e (2/3): the selected regime, and the certificate

Narrow empirical validation, continued: the selected regime. A grid of 48 settings uses the same queries and contexts, with α in {0.3, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95} and s in {0.3, 1, 3, 10, 30, 100} (Appendix A.4.9, Tables 11 and 12). Its four published settings reproduce the recorded accuracies exactly. With uniform companions:

  • The adaptive loss is governed by the mixing weight α. At α = 0.95, adaptive accuracy is 35.9% to 36.5% for every s ≤ 30. At s = 3 it is 86.2%, 79.8%, 72.1%, 58.9% and 36.5% for α = 0.7, 0.8, 0.85, 0.9 and 0.95. The loss begins near α = 0.8 and stays under 2.1 points for α ≤ 0.7.
  • At s = 100 the weights stay near the identity and accuracy is unchanged.
  • Frozen and adaptive weights differ only where the input kernel is selective. At α = 0.95, frozen weights retain 88.0% for s = 30 and 64.9% for s = 10, but 36.6% to 38.0% for s ≤ 3, where the weights are already diffuse at the input.

The last point narrows a reading that the submission invited, as stated in the revision. The frozen control isolates rebuilding in the selective regime. With soft logits the same near-uniform mixing, and the same loss, arise without rebuilding. Context removal recovers 87.3% to 88.5% in all 48 uniform settings.

The headline claims remain separations: convergence does not imply a small input response, and neither implies preserved predictions. We do not claim that adaptive mixing typically harms trained models.

Verifiability of the certificate (weakness 2, question 2). These bounds are available in closed form, each on a stated domain. The revision collects them in Table 3 (Appendix A.2.3).

  • Fixed weights, as in APPNP: L_P = 0 and m = β. Always certified.
  • Dot-product softmax with inverse temperature s on the ball of radius R: L_P ≤ 2sR, so 2αsR² < β certifies (line 131).
  • Hull refinement: with input initialization, R·L_P can be replaced by D·L_P/2, where D is the diameter of the input hull (line 135). This certifies 1,349 of 4,608 image configurations (line 138).
  • Gaussian weights with adaptive bandwidth: L_P ≤ s(4R/h_min² + 16 c_h R³/h_min⁴). This bound was already in Appendix A.2.3 of the supplement. It is inactive at every nonzero setting we tested.
  • Gaussian weights with a fixed bandwidth h, including the non-local-means kernel of Fig. 1 (new): L_P ≤ 4D/h², where D bounds pairwise feature distances, so 2αD² < βh² certifies on the hull. A numerical search found no violation.

The following are empirical diagnostics, and the paper never reports them as certificates: the spectral radius of J at a computed equilibrium, finite-difference and re-solved responses, and Lipschitz estimates from sampled trajectories. The paper calls these numerical settling (lines 139–140). Table 3 separates the two kinds.

The certificate is very conservative for sharp kernels, and the revision says where it is vacuous. It does not certify the strong-mixing classification setting, where 2αsR² = 57 against β = 0.05, or the softmax response maximum (lines 137–138). For Fig. 1 the hull margin is β − 2αD²/h² ≈ −1415 with pixel range D = 0.682, and certification would need h > 4.21 against the h = 0.025 used. No uniqueness certificate can cover that image in any case, since it has two stable limits.

In this paper the certificate fixes the reference quantities: the response bound β/m and the all-budget condition γ < m/β. Theorem 2 then shows that no condition weaker by a constant factor, stated in α, R and L_P alone, can guarantee uniqueness over row-stochastic weight rules. Kernel-specific thresholds remain open (line 475).

AuthorsOpenReview ↗

Response to Reviewer 954e (3/3): prevalence outside the selected maxima

Prevalence (question 3). Fig. 2c plots all 360 configurations, not only the maximum. Each point is one configuration's worst-direction gain, adaptive against frozen. Of the 900 configurations, 92 reach distinct endpoints from the two starts (line 224). The distribution over directions was missing, and we thank you for pointing this out. We have added two measurements.

Random directions (Appendix A.3.4, Table 4). For each of the 360 configurations we drew 1,000 random unit directions. Across the 288 configurations with α > 0:

  • The median direction has gain 0.48 with rebuilt weights against 0.46 frozen, and the 95th-percentile direction 0.55 against 0.51 (medians over configurations).
  • The median direction is amplified in 8 configurations and the 95th-percentile direction in 21. No frozen average amplifies either.
  • In the response-maximum configuration the worst direction has gain 18.875. Its median random direction has gain 1.56 against 0.67 frozen, and 84% of random directions are amplified.

Random noise on the Fig. 1 photograph (Appendix A.7.3, Table 18). We replaced the selected probe by independent Gaussian noise and kept the photograph, the setting and the stopping rule.

  • To first order, the root-mean-square gain is 0.38 with rebuilt weights against 0.155 with frozen weights. The rebuilt loop raises the noise level at 38 of 1,024 pixels, by up to a factor of 2.8. A frozen average cannot raise it at any pixel.
  • Re-solving 100 noisy images at each of four noise levels, from 1e-5 to 10/255 per pixel, gives median maximum-pixel gains of 0.92 to 1.51 against 0.53 frozen. The rebuilt gain exceeds one in 27 to 73 of 100 draws. The frozen gain never does.
  • The two stable limits of Fig. 1b persist in all 100 noisy copies of the photograph at every noise level. The dependence on the start is not special to the mean start: at the three larger noise levels the same noisy image reaches different limits from the clean solution and from itself in 22, 57 and 92 of 100 draws.

So typical directions are amplified far less than the worst one. The 19-fold figure of Fig. 1 is a worst-direction statement, and under unstructured noise that loop attenuates on average, at 1.6 to 2.6 times the response of its frozen average. The revision says this in the abstract, the introduction and Section 4.

Which statistic matters depends on the use. For unstructured noise it is the typical direction. For a feedback loop it is the worst one: at K = 16 the first sampled perturbation shrinks by a factor of 0.512, and repeated outer updates then grow it (lines 425–428).

These measurements describe the designed configurations and one photograph at one screened setting. They are not an estimate of how often amplification occurs in tuned pipelines or trained models, which the paper leaves open.

Reviewer_XMhEOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies adaptive mixing with input retention, where the averaging weights are recomputed from the evolving state at each iteration. It derives a sufficient condition for convergence to a unique fixed point and bounds the fixed point’s sensitivity to changes in the input. The paper argues that positive averaging and convergence do not, by themselves, guarantee a small input response or preservation of downstream predictions.

Critical strengths
  • The convergence analysis gives an explicit sufficient condition for a unique fixed point and geometric convergence, together with a bound on the fixed point’s input sensitivity.
Critical weaknesses
  • I found the Introduction difficult to follow. It reads like a compressed combination of a problem statement, related work, and a results summary: it moves from an abstract claim about averaging to a list of architectures, theoretical contributions, classification results, and downstream feedback before clearly explaining the operation being studied. The paper also has no dedicated Related Work or Preliminaries section. Such sections are not required, but their absence makes the current presentation especially difficult to navigate: the Introduction introduces several unfamiliar methods and technical claims without first providing enough context to understand how they fit together. As a result, I found it difficult to understand the paper's central problem and contributions from the Introduction.

  • What is the reason for using "maximum-row norm"?

  • What are the dimension of XtX_t P(⋅)\mathcal P(\cdot ) near line 95? Is it a matrix because of "whose rows sum to one"?

  • What is the definition of LPL_P in Eq. (2) near line 104?

  • How do we know the sharpness of Theorem 1? In other words, what factor indicates "class sharpness"?

Questions for authors

Refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Thank you for your review

Response to Reviewer XMhE

Thank you for the candid review. The difficulty you describe is a fault of the writing, and we have rewritten the introduction and setup in response. We answer your questions first. Line numbers refer to the original submission.

Dimensions (Eq. 1). Yes, P(X) is a matrix. There are N tokens: pixels, patches, graph nodes, or images in a batch. Each has a d-dimensional representation. The input A and the state X_t are N×d matrices with one row per token. P(X) is an N×N matrix with nonnegative entries whose rows each sum to one, so row i of P(X)X is a weighted average of the rows of X.

Why the maximum-row norm. The norm of a state is the length of its longest row. We use it because averaging cannot increase it: when the weights are nonnegative and each row sums to one, every output row is a convex combination of input rows. In this norm, "an average cannot amplify" is a theorem. That makes it the right yardstick for our question, which is whether recomputing the weights can amplify anyway. In the Euclidean operator norm even a fixed average can exceed one (line 229).

Definition of L_P (Eq. 2). L_P is a Lipschitz constant of the weight rule. It is any number such that the weights change by at most L_P times the change in the state, for all states of norm at most R. Weight matrices are compared by their largest absolute row sum. Fixed weights have L_P = 0, and dot-product softmax with inverse temperature s has L_P ≤ 2sR (line 131).

What indicates class sharpness (Theorem 1). The theorem guarantees a unique fixed point when αRL_P < β. Sharp means this inequality cannot be weakened using only α, R and L_P. For every ε > 0 there is a smooth weight rule with αRL_P between β and (1 + ε)β that has two distinct stable fixed points. The factor you ask about is that 1 + ε, and it can be as close to one as desired. "Class" means the statement concerns the family of weight rules: some rule fails just past the threshold, not every rule. In the revision the classical guarantee is Theorem 1, this sharpness statement is Theorem 2, and one sentence between them defines "class-sharp".

Introduction, Related Work, Preliminaries. You are right on all three. In the revision:

  • The introduction opens with the iteration itself, displayed as Eq. (1), before any claim.
  • A contributions paragraph follows, one item per result, and Table 1 lists the four questions the paper answers.
  • Related work is Section 2.
  • A "Setting and notation" paragraph opens Section 3, before the theorems, and contains the four answers above.
  • Internal labels are replaced by descriptive names in the main text.

The new opening reads: "Consider N tokens, each a vector, and a step that replaces every token by a weighted average of all tokens, with weights computed from the current tokens, before blending in the original input: X_{t+1} = α P(X_t) X_t + β A. The rows of A ∈ R^{N×d} are the original inputs, the rows of X_t are the current tokens, and P(X_t) is an N×N matrix of nonnegative weights whose rows sum to one. Non-local-means denoising inside regularization by denoising (RED) is exactly this loop. [...] Every step is an average with positive weights, and under a classical contraction condition the loop converges to a single fixed point. Two beliefs about such loops are tempting. The first is that averaging is safe: an average never leaves the convex hull of its inputs, and a fixed average never enlarges the largest change across tokens. The second is that convergence implies stability, in the sense of a single limit that moves little when the input does. Both hold when the weights are fixed." The next paragraph says why these beliefs matter and where they fail.

The supplementary material now also holds an optional six-minute narrated overview that walks through the setting and the main results with the paper's own figures and data. It adds no result.

If these answers and the revision resolve the difficulty, we would be grateful if you would revisit your scores. If something is still unclear, please tell us where you got stuck.

Thank you for your detailed responses to my questions. After reviewing the revisions, I have a few additional questions .

First, regarding LpL_p, wouldn't it be more appropriate to explicitly state the corresponding condition as an assumption rather than describing it in the main text? Such theoretical conditions are typically formalized as assumptions, and I find the current presentation somewhat unconventional.

Second, I would encourage the authors to reconsider how the Introduction is presented. I believe an Introduction should be written not only for researchers familiar with the topic but also for readers encountering the research area for the first time. However, the current Introduction begins directly with NN tokens and mathematical notation without sufficient explanation of what these tokens or the corresponding vectors represent.

For example, even within the first few lines, several questions arise: What does dd, first introduced in line 39, represent? Is it the feature dimension? What is the role of α\alpha? What exactly does P\mathcal P represent? Although its dimensions are specified, its purpose remains unclear. Is it simply a weight matrix? These questions concern the basic notation and problem formulation rather than the core ideas or technical contributions of the paper. I believe readers should not have to resolve so many questions about notation before they can understand the motivation and main contributions of the work.

I understand that presentation styles may vary across research areas. Nevertheless, I would suggest looking at how influential papers such as DDPM, Attention Is All You Need, and Distilling the Knowledge in a Neural Network structure their introductions. These papers generally establish the research context, motivate the problem, and introduce the key ideas before presenting detailed mathematical formulations. I believe adopting a similar approach would significantly improve the clarity and accessibility of this paper.

I therefore encourage the authors to revise the Introduction from the perspective of a reader unfamiliar with the specific topic, with particular attention to the logical flow, motivation, and clarity of notation.

AuthorsOpenReview ↗

Response to Reviewer XMhE (follow-up)

Thank you wholeheartedly for reading the revision and for these further clear and candid comments. Everything you have said is fair. We have made further changes in line with your comments. We realize this would have been a frustrating paper to read based on the initial submission copy, and we appreciate your efforts to provide useful feedback, and then again when further improvement was needed after revision. Realistically, we don't expect you to engage further with the paper since you have already spent a lot of time, and helped us to better communicate some of the ideas. Thank you for your suggestions and for spending your time to help us understand what we can do better. We expect that this helps the current paper, but also will help more in making sure the technical pieces of our work are communicated in a way that ensures the value in the work isn't lost going forward.

L_P as an assumption. We agree that this is the conventional form. The Lipschitz condition on the weight rule is now Assumption 1 (lines 167–169), placed directly before Theorem 1, and Theorem 1 invokes it by name. Equation (2) and all theorem numbers are unchanged.

The Introduction. You are right that the revised opening still asked the reader to parse notation before saying what the paper is about. We have reordered the introduction along the lines you describe, so that context, motivation and the key idea come before the formulation.

  1. Context and the two beliefs, in words (lines 037–044). "In kernel denoisers, graph propagation and attention, each item, whether an image patch, a graph node or the feature vector of an image, is updated with a weighted average of all items, with positive weights. Many systems repeat this operation in a loop, blending the original input back in at every step." The paragraph then defines frozen and adaptive weights and states the two beliefs.
  2. An example and the key idea (lines 046–053). The denoising loop of Figure 1 is described in words, followed by the reason for its behaviour: "A change to the input changes the values being averaged and also the weights; the changed weights move the result, which changes the weights again. A frozen average has only the first path."
  3. The formulation (lines 082–093). Eq. (1) now appears here, with its symbols explained beside it: "Formally, let there be N items, called tokens, each a vector of d features. [...] The rows of A ∈ R^{N×d} are the original inputs, the rows of X_t are the current tokens, and P(X_t) is an N×N matrix of nonnegative weights whose rows sum to one (row-stochastic). Row i of P(X_t)X_t is the weighted average for token i. The mixing weight α sets how much of each update comes from this average, and β how much from the input."
  4. Why the beliefs matter in practice, and the scope of our examples (lines 094–100).
  5. Contributions (from line 102). The margin is introduced in words, as "the weight β on the original input minus the gain of the path through the weights". L_P and R no longer appear in the introduction, and Table 1 names its quantities in words.

This answers your three questions in the text itself: d is the feature dimension, P(X_t) is the weight matrix, rebuilt from the current tokens at every step, and α is the share of each update that comes from the average.

The abstract follows the same order. Its first sentence is unchanged. It then states the two beliefs, says in one sentence why they can fail, and groups the results under the belief each one tests.

To stay within the page limit, Figure 3 is about a tenth narrower, one footnote is folded into its sentence, and two sentences in Sections 4 and 6 are a few words shorter. No result, number or proof has changed. The only change to a theorem statement is the reference to Assumption 1, and all section, equation, table and figure numbers are the same.

Terms defined where they are first used. With your comments in mind we also read the whole paper, appendices included, as a newcomer would. Sections 4 to 6 now say which belief they test; the bound (3), the response formula (5) and Table 1 explain their symbols in place; and the appendix defines its working terms (for example the two kinds of frozen weights, belief states, and the grid of designed point clouds) at first use and cites every table from the text. The appendix is one page longer as a result.

We hope this addresses both comments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

The big picture (or a post-mortem?)

Thank you to all of the reviewers for providing invaluable feedback on this paper, which we feel has significantly improved the paper in it's structure, writing, and how well the findings are communicated.

Having reflected deeply on this work since the initial reviews were received, we realize there were two significant overlapping problems with the initial presentation:

  1. This is a clear example of "losing the forest for the trees" - the extent of time spent focused on the details showed up in a presentation that drifted away from sharing adequate context.
  2. We aimed to share a story with a beginning, middle and ending, and this paper only communicated the middle. It's probable that most readers might have benefited more from reading either the beginning or the ending instead.

We don't have any illusions about the likely outcome of this paper and the initial presentation fell short. Having even one or two reviewers raise their score by a single point is already a tall order, and that would almost certainly not be enough. However, we felt compelled to engage in the rebuttal process anyway, mainly because we felt the reviewers deserve to take more away for their efforts than the initial disjointed presentation offered. In this regard, perhaps the 6 minute video added to the supplemental material provides the quickest avenue to context around the purpose of the examples presented in the paper.

Again, thank you to everyone for your helpful feedback which has undoubtedly improved this paper, but also more broadly for the opportunity to learn what we can do better going forward.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.