acceptodds
Withdrawn from ICLR 2027

GRASP: Graph-Refined Anchored Soft Predictions for Training-Free Vision-Language Model Adaptation Across Stream Structures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-free test-time adaptation of vision-language models must operate on test streams whose local structure can vary substantially in deployment. Samples observed together may share useful temporal or semantic context, yet heterogeneous batches can make the same relational evidence unreliable. This creates a central challenge for relational adaptation: exploiting local structure without assuming that it is always informative. We demonstrate this failure mode with a simple batch-local graph refinement, which performs strongly on class-correlated streams but degrades sharply under IID ordering. We introduce GRASP (Graph-Refined Anchored Soft Predictions), which retains this local relational structure while controlling its influence. GRASP anchors graph refinement to a cache-refined posterior through a fidelity-regularized objective and attenuates edges whose endpoint posteriors disagree. The graph is restricted to the current batch and discarded after inference; only the bounded cache persists across the test stream. GRASP requires no target labels, gradient updates, prompt optimization, or knowledge of the stream regime. Across 11 image-classification benchmarks and five stream regimes, GRASP achieves the highest cross-regime mean among evaluated methods at . Relative to TDA, which provides its default prediction anchor, GRASP trails by percentage points under IID ordering but improves by points under class-sequential ordering. These results show that local relational evidence can provide substantial test-time gains while limiting degradation under weak local structure.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Training-Free Test-Time Adaptation, Vision-Language Models, Relational Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_WnPJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors observe that when test samples arrive grouped by class, exploiting relationships among samples can yield substantial gains; however, when samples are randomly mixed, graph propagation may lead to significant performance degradation. Therefore, relationships among samples cannot be assumed to always be beneficial. To address this, the authors propose GRASP, adjusts the connection weights between samples according to the agreement between their predictive distributions, and uses a fidelity constraint to limit the changes that graph propagation makes to the original predictions. The aim is to exploit helpful relational information while reducing interference from unreliable relationships.

Critical strengths

The authors observe that when test samples arrive grouped by class, exploiting relationships among samples can yield substantial gains; however, when samples are randomly mixed, the same propagation mechanism may lead to severe performance degradation. This observation has practical significance because, in real-world deployment, it is generally not possible to determine in advance whether local relationships among samples are reliable.

Critical weaknesses

The ablation results for the key design components and the method’s performance under different settings still require further analysis. These issues directly affect my assessment of the necessity of the method’s design.

Questions for authors
  1. Have the authors considered how prediction uncertainty affects JS? The ablation experiments in the appendix show that the average gain from JS is relatively small.

  2. Based on Table 1, the method appears to have limited performance stability under IID and in the Cross-regime(Worst), with relatively large performance fluctuations across different settings.

  3. The ablation experiments show that the anchored random-walk yields the largest improvement under IID. Could the authors provide further analysis of this result?

  4. The paper appears to involve quite a few hyperparameters. Is the method somewhat engineering-heavy?

  5. Some symbols are explained only after they first appear. Even if readers can infer the meanings of the symbols in the equations, these meanings should still be explicitly defined at their first occurrence. In terms of presentation, the figures should be made more consistent. The fonts and font sizes used in the figures throughout the paper are quite inconsistent, which reduces readability.

2 comments

Thank you. We address each point below, with new results or changes to the paper. Several answers compare methods at the same IID cost: we vary each method's strength and compare them where each loses the same IID accuracy, so that design is not confused with strength (see the general response).

Q1: Uncertainty and JS.

When both predictions are confident, JS reliably tells same-class from different-class neighbours; when both are uncertain, the gate steps back and GRASP stays safe (App. A.6).

  • Confident pairs. In IID batches most nearest neighbours belong to different classes, so most confident neighbour pairs predict different classes. JS separates these cases cleanly:
Confident neighbour pairs (IID)Share of pairsActually the same class
Predictions disagree → gate down-weights the edge87%0.3%
Predictions agree → gate keeps the edge13%82%
  • Uncertain pairs. Between two flat predictions JS is small whatever the classes, so the gate keeps most edges (only 16% are down-weighted). GRASP then acts as the anchored feature graph, whose output is still a weighted average of the anchor predictions, so it stays bounded. The revision states this.
  • Effect on accuracy: at equal IID accuracy, JS improves every stream type by 0.2 to 0.7 pp (Sec. 5.3, App. A.17). At the default strength it halves the IID cost (0.84 vs 1.70 pp).

Q2: IID stability

GRASP's IID loss is small and measured on every dataset.

  • On every dataset and every stream type (10 seeds), GRASP is never more than 0.82 pp below zero-shot CLIP. For γ ≤ 0.01 it is above its TDA anchor on every dataset.
  • In IID it is 0.82 pp below the anchor on average (worst dataset −1.49). On its worst dataset, LAME on the same anchor drops by 4.11 pp and official LAME by 7.0 pp.
  • With a stronger anchor the worst case improves further: ADAPT+GRASP has the best worst-stream accuracy of all methods (70.51).

Q3: Why the anchored random walk helps IID most? That row lowers the propagation strength, compared at equal IID cost, each component still contributes.

  • Lower strength keeps IID accuracy (67.6 at strength 1, 59.7 at 10) and higher strength helps Separate (72.1 vs 78.0). Table 2 is replaced by an ablation at equal IID cost, and Fig. 3 shows the strength trade-off.
  • At equal IID cost (new Table 2), each component matters: 3 iterations instead of the exact solve cost 1.16 pp, the random-walk Laplacian 0.94 pp, and removing the gate 0.43 pp.

Q4: Is the method engineering-heavy?

No: one linear solve per batch, three effective hyperparameters, and one setting for everything, not tuned on the test streams.

  • GRASP uses (μ, λ, k, τp) = (10, 10, 5, 1) for all 11 datasets, 5 stream types, 3 backbones and every anchor. Only λ/μ matters, so there are effectively three.
  • Where they come from: all values were fixed before the frozen test streams were built and were not tuned on them; μ = λ gives the anchor and graph terms equal weight (η = 1); k = 5 is a small neighbourhood, common for batch kNN graphs; τp = 1 is the unit temperature.
  • Fixing one setting for all experiments, without target labels, is common in training-free adaptation; TransCLIP, for example, fixes its weight once for all zero-shot experiments.
  • After the main experiments, we checked how sensitive this choice is: on 5 datasets, we varied each hyperparameter around the default, one at a time (App. A.7, Fig. 4). Each one trades IID against Separate accuracy, so no single value is best for both, and the default sits in between.
  • In the same check, if k is instead chosen on held-out datasets (leave one dataset out), the choice is k=10, a neighbouring value. The comparisons with LAME and with the no-JS variant reach the same conclusions there, so we kept the original default
  • Revision: Sec. 5.1 now states where the values come from, and App. A.7 reports the sensitivity check.

Q5: Presentation.

Thank you, both points are fixed. Every symbol is now defined at first use: Fig. 1's caption points to Eq. (13) and names η and L, and the background defines λ, W, D, B, C and P_cache. All plots now use one font family and matching sizes. Table 2 is replaced as described above.

Dear Reviewer WnPJ,

Thank you again for your review. With the discussion period closing on November 18, we would be grateful to know whether our responses and the revised PDF resolve your concerns.

Specifically, we explained how the JS gate behaves when predictions are uncertain, reported IID accuracy on every dataset, and added an ablation of each component at the same IID cost, which shows that GRASP's gain is not only a strength effect.

We are happy to clarify anything further if that would help.

Reviewer_S3DXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles TDA: a setup where adaptation is performed whilst keeping encoder frozen, by maintaining a cache, and an evolving graph. The core argument is that class sequential ordering leads high performance as compared to IID ordering (line 57 of paper).

Critical strengths
  • I really liked the idea that graph is not grown across batches, and only the cache propagates.
  • fig 1 looks cool quite informative, the accuracy of GRASP (green curve) seems to be better than TDA, zero-shot clip baseline
Critical weaknesses
  • Equation 10 assumes a symmetric constraint on weights of the graph. Perhaps, an explanation could be provided behind this choice.
    • Equation 14 solves for F* , which in turn requires one to solve for inverse . In practice, taking inverses takes time as opposed to multiplying matrices together. I was able to find a latency analysis in Table 13, but an alternative procedure not to take inverses might help speed it further.
Questions for authors
  • Will GRASP work for test-time-training, i.e. only one test-sample is given at a time, and information is not allowed to be leaked.
  • How can one measure compatibility? (line 484). Some starting points might be helpful.
2 comments

Thank you, we are glad you liked that the graph is discarded after each batch while only the cache persists.

Symmetric weights (Eq. 10). Symmetry is what makes the refinement an optimization problem with a cheap, well-understood solution.

  • With W symmetric, the refined predictions are exactly the minimizer of the objective in Eq. 12, and μI + λL is symmetric positive definite, so one Cholesky solve gives the unique solution.
  • Its rows sum to μ and it is an M-matrix, so each refined prediction is a weighted average of the anchor predictions: a valid probability vector, never more confident than the most confident input.
  • Symmetry also gives L real, non-negative eigenvalues, which is what the displacement bound in Prop. 1 is stated in.

Inverse (Eq. 14). No inverse is formed; one small linear solve per batch.

  • Per batch of 128 (10 to 1,000 classes), V100 / CPU: LU 0.65–0.71 ms / 8–14 ms; Cholesky 0.86–1.16 ms / 12–17 ms.
  • The whole GRASP step takes 3.1–4.9 ms on GPU, similar to TDA's cache step (2.4–12.4 ms). On CPU most time goes to building the graph, not the solve. Appendix A.11 now reports this breakdown.

One sample at a time. GRASP is designed to use neighbouring samples, but it does not collapse without a batch. Each sample is first added to the TDA cache and predicted, as in official TDA; it is then refined with the ≤ 127 samples before it and output immediately. No future sample enters the cache or the graph, and no label is used (App. A.18, Table 20):

IIDγ=0.1γ=0.01γ=0.001Sep
TDA, one at a time67.6467.8267.2767.2066.94
GRASP, one at a time66.8468.5271.1073.4174.35
GRASP, batched (reference)66.7869.2372.5974.4475.11

For γ = 0.01, γ = 0.001 and Separate it beats TDA by +3.8, +6.2 and +7.4 pp (on 10, 11 and 11 of the 11 datasets), keeping 72–88% of its batched gain, and it matches the batched version in IID.

Measuring compatibility (line 484). Posterior agreement is a strong, label-free proxy for same-class edges.

  • Edges the gate down-weights (r < 0.6) link same-class samples only 0.5% of the time in IID, vs 46% for kept edges.
  • Overall the gate keeps about 90% of same-class edge weight but only about 62% of cross-class weight. It is soft by design (r ∈ [0.5, 1]), so it never disconnects the graph.
  • A simple diagnostic for practitioners: the share of edge weight the gate keeps (61% in IID vs 89% in Separate) tracks how much useful structure a batch has.

Dear Reviewer S3DX,

Thank you for your review. As the discussion period ends on November 18, we wanted to check whether our responses and the revised PDF address your concerns.

We added a strictly causal evaluation that processes one sample at a time and clarified the symmetric weights, the linear solve (no inverse is formed) and how posterior agreement measures compatibility. If any point is still unclear, and requires further explaination, please let us know and we will reply promptly.

Reviewer_vrjnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GRASP addresses training-free adaptation of vision-language models across test streams with varying class correlations. It refines cache-based posteriors through a batch-local graph, combining fidelity regularization with posterior-consistency edge weighting.

Critical strengths
  1. Extensive evaluation supports the usefulness of relational refinement. Applying the unchanged operator to DOTA and ADAPT improves cross-regime accuracy with small IID losses, suggesting benefits beyond the default TDA anchor.
  2. The method is simple and computationally practical, with bounded persistent state. Its mathematical analysis establishes a well-posed, simplex-preserving solution and appropriately distinguishes prediction-displacement bounds from accuracy guarantees.
Critical weaknesses
  1. Component attribution remains unclear. Eqn. (36) makes converged propagation equivalent to an anchored random-walk solve at matched regularization strength, but Table 2 reports substantially different results. A coefficient change may explain the improvement. Similarly, the JS ablation lacks uniformly rescaled or strength-matched controls, leaving selective weighting confounded with weaker smoothing.
  2. The evidence for deployment robustness is limited. The default method trails ADAPT under IID ordering. The natural temporal experiment lacks frame-sampling details, temporal-averaging controls, and comparisons with the strongest adapter combinations, limiting conclusions about practical robustness. Thus, it is difficult to distinguish the value of graph construction from ordinary aggregation of correlated frames. The additional-backbone comparison similarly omits DOTA and ADAPT. These gaps limit how strongly the results support the practical significance of the proposed method.
Questions for authors
  1. What coefficients and convergence criteria were used in each Table 2 row? Why do mathematically equivalent variants differ, and what happens at matched regularization strength?
  2. Does JS weighting outperform uniformly rescaled graphs or appropriately tuned no-JS refinement? Please provide paired uncertainty and explain hyper-parameter selection.
  3. Please specify the temporal experiment’s frame sampling and batch construction, and compare against temporal averaging and DOTA/ADAPT with and without GRASP under matched conditions.
2 comments

Thank you for the detailed questions. We address each question below with new experiments. Several answers compare methods at the same IID cost: we vary each method's strength and compare them where each loses the same IID accuracy, so that design is not confused with strength (see the general response).

Q1: Table 2. You are right: the two rows are the same operator at different strengths. Compared at equal IID cost, each GRASP component still matters.

  • Converged GRASP-Base and the anchored random walk differ only in strength (λ=10 vs η=1); at matched strength their outputs are identical. Their gap in the old Table 2 was therefore a strength effect.
  • We replaced Table 2 with an ablation in which every variant is compared at the same IID cost (table in the general response). Each component matters: 3 iterations instead of the exact solve cost 1.16 pp, the random-walk Laplacian 0.94 pp, and removing the gate 0.43 pp. GRASP-Base as a whole is 1.09 pp below GRASP.
  • Coefficients: App. A.15 lists the strength of every variant at the matched cost. All variants except "3 iterations" use an exact linear solve, so no convergence criterion is involved.

Q2: Is JS just a rescaling? No.

  • We sweep the strength of both GRASP and GRASP without JS and compare them at equal IID accuracy, which removes the strength difference: JS is ahead in every stream type (+0.2 to +0.7 pp; Sec. 5.3 and App. A.17, Table 19). Fig. 3 now plots both sweeps. The result also holds at k=10 (App. A.7).
  • Other gates at equal IID cost: JS beats a confidence-only gate and a feature threshold with the same edge weight. The benefit comes from agreement between predictions, not from removing edge weight.
  • Paired uncertainty: all comparisons use paired 95% bootstrap CIs over (dataset, seed), reported in App. A.15–A.17. Hyperparameter selection is answered in our response to WnPJ (Q4).

Q3: Temporal experiment on UCF-101. GRASP's gain on natural video comes from its graph step, not from averaging.

  • Setup (App. A.10): we take 8 evenly spaced frames from each of the 3,783 UCF-101 test videos and stream them in batches of 128 (4 seeds). Natural: videos in random order, each video's frames kept together in time order. Shuffled: the same frames in random order.
  • Is the gain just averaging neighbouring frames? No. Even a temporal smoother tuned on this natural stream (an upper bound) recovers only 0.96 of GRASP's 3.91 pp gain over TDA (Table 13):
MethodAccuracy
TDA69.07
TDA + temporal smoothing (best window)70.03
GRASP72.98
  • With stronger adapters: DOTA and ADAPT gain nothing from frame order on their own (natural vs shuffled: −0.39 and −0.49 pp). Adding GRASP raises both on the natural stream and barely changes the shuffled one:
AnchorNatural: anchor → +GRASPShuffled: anchor → +GRASP
DOTA74.13 → 76.24 (+2.12)74.52 → 74.08 (−0.44)
ADAPT70.55 → 72.36 (+1.81)71.04 → 70.84 (−0.20)

W2: Stronger adapters and backbones. GRASP improves DOTA and ADAPT, unchanged. (11 datasets, 3 seeds)

IIDγ=0.1γ=0.01γ=0.001SepMeanWorst
DOTA69.5269.1968.7268.3268.1268.7768.12
DOTA+GRASP69.0570.0472.6974.4175.4872.3369.05
ADAPT70.5770.1669.3568.5568.8669.5068.55
ADAPT+GRASP70.5170.8271.8772.4173.3171.7870.51
  • IID cost: −0.47 pp (DOTA), −0.06 pp (ADAPT, not significant). Gains of +2.5 to +7.4 pp for γ ≤ 0.01 on 11/11 datasets.
  • GRASP's IID level follows its anchor: on ADAPT it keeps ADAPT's IID accuracy and adds the correlated-stream gains.
  • On ViT-L/14 (settings unchanged), ADAPT+GRASP keeps ADAPT's IID accuracy and raises the mean from 75.85 to 77.46 and the worst case from 75.05 to 76.70 (App. A.13).

Dear Reviewer vrjn,

Thank you for your review. As the discussion period ends on November 18, we wanted to check whether our responses and the revised PDF address your concerns. In particular, we added an ablation of each GRASP component at the same IID cost, results with GRASP added unchanged to DOTA and ADAPT, and an analysis of the UCF-101 video experiment with natural and shuffled order.

If any point is still unclear, or you would like an additional result, please let us know and we will reply promptly.

Reviewer_WjD9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GRASP, a training-free test-time adaptation method for vision-language models such as CLIP that improves predictions on unlabeled test images without gradient updates. It addresses the challenge of unknown test-stream ordering, where images may arrive randomly or in class-grouped sequences. The authors introduce GRASP-Base, which smooths predictions from the cache-based TDA method across visually similar images. While effective on class-sequential streams (77.1%), this baseline performs poorly on random streams (52.6%), even falling below zero-shot CLIP (65.3%).

To address this limitation, GRASP introduces a constrained refinement mechanism that balances information from neighbouring images with the original TDA predictions. It also adaptively weakens connections between images whose TDA predictions disagree, reducing the risk of propagating incorrect predictions. The neighbour graph is reconstructed for every batch, requiring no persistent state beyond TDA's existing cache. The authors theoretically establish that refined predictions remain valid probability distributions and that deviations from the original TDA predictions are bounded.

Experiments across 11 classification datasets and five stream orderings show that GRASP achieves the best average accuracy of 71.6%, outperforming 10 comparison methods. Compared with TDA, it improves accuracy by 8.4 percentage points on class-sequential streams, with only a 0.8-point drop on random streams. Further experiments demonstrate consistent behaviour across three additional backbones, when combined with DOTA and ADAPT, and on naturally ordered UCF-101 video streams.

Critical strengths

1 The paper evaluates 11 datasets ×\times 5 regimes ×\times 10 seeds against 10 comparison methods (8 published training-free methods, zero-shot CLIP and the authors' GRASP-Base) on identically ordered streams, with full per-dataset tables. Table 1 shows that method rankings flip with stream structure: ADAPT is best under i.i.d. but declines as structure increases, StatA improves, and ECALP falls below zero-shot CLIP under Separate. This is useful for practitioners and for how this sub-field evaluates methods.

2 GRASP needs one B×BB\times B linear solve per batch and uses one configuration for all datasets and regimes. For γ≤0.01\gamma\le 0.01 and Separate it improves over TDA by 5.3 - 8.4 , with no dataset below TDA in these regimes. The same mean-level trend appears on four backbones (Figure 5, which compares five methods and omits DOTA and ADAPT) and when the operator is stacked on DOTA or ADAPT (Table 16, three seeds; DOTA+GRASP: mean 72.34%, worst regime 69.05%). This is the main value in the paper.

Critical weaknesses

1. LAME (Boudiaf et al., CVPR 2022) already performs training-free, batch-local, output-only adaptation: a KL fidelity term to the frozen model's predictions plus a Laplacian term over a kkNN graph of the current batch, designed for robustness across i.i.d., label-shifted and non-i.i.d. streams. It reports the same qualitative pattern as GRASP: no gain (a slight loss) under balanced i.i.d. streams and large gains on class-correlated streams. GRASP's operator (I+ηL)−1Pcache(I+\eta L)^{-1}P^{\mathrm{cache}} is the classical regularized-Laplacian / graph-Tikhonov smoother (Zhou et al., 2003; Belkin et al., 2004; Smola & Kondor, 2003), applied post hoc to a base predictor's soft outputs as in Correct & Smooth (Huang et al., ICLR 2021). Without a LAME comparison it is unclear what GRASP contributes.

2. The ablations do not separate the claimed mechanisms from propagation strength.

  • Cache anchoring. By the paper's own Eq. (34), converged GRASP-Base is already anchored: (I+λ(I−D−1W))−1Pcache\big(I+\lambda(I-D^{-1}W)\big)^{-1}P^{\mathrm{cache}} with λ=10\lambda=10. The "converged →\to anchored, random-walk" step in Table 2 therefore appears to lower the strength from 10 to 1 (since the paper does not state I am assuming the default η=1\eta=1) rather than change the operator, contrary to Appendix A.13's "modify the refinement operator itself". GRASP itself also collapses at high strength (Table 9: i.i.d. 75.34% at η=1\eta=1, 55.79% at η=20\eta=20). The size of GRASP-Base's i.i.d. collapse therefore looks like a hyperparameter choice, although λ\lambda is never swept, TT also matters (+4.20+4.20 pp i.i.d. from T=3T=3 to convergence, Table 2), and an i.i.d.–Separate trade-off remains.

The paper notes that Proposition 1's displacement bound does not guarantee accuracy, so the theory gives no account of why GRASP keeps i.i.d. accuracy while GRASP-Base does not.

Questions for authors

Clarifications

  1. Which strength does each anchored row of Table 2 use? Given Eq. (34), do "converged" and "anchored, random-walk" differ only in strength (λ=10\lambda=10 vs. η=1\eta=1)? If so, please report the operator change and the strength change separately.
  2. How were the defaults (η,k,τp)=(1,5,1)(\eta,k,\tau_p)=(1,5,1) and B=128B=128 chosen, and on which data? If they were selected using test accuracy on these benchmarks, the "single fixed configuration" claim is weaker than stated.

Additional evidence that would change my score

  1. LAME (Boudiaf et al., 2022) applied to zero-shot CLIP and to TDA posteriors, under the same five regimes and BB. What does GRASP add over it?
  2. A strength-matched JS ablation: sweep η\eta for both GRASP and no-JS and plot i.i.d. against Separate accuracy. Does JS move the frontier, or only the operating point?

If (4) and (5) show a clear advantage over LAME and over a strength-matched no-JS variant, I would raise my score.

4 comments
AuthorsOpenReview ↗

Comparison with LAME (W1, Q4)

Thank you for a precise review. We have run the comparisons you asked for.

Summary: official LAME is below GRASP in every stream type. With LAME run on the same TDA predictions and compared at the same IID cost, GRASP is better on class-sequential streams at every cost and ahead or tied elsewhere near GRASP's default.

1. Official LAME (direct comparison; Table 1, 10 seeds). As published, on CLIP's predictions, LAME is below GRASP by 4.64 / 3.64 / 1.44 / 0.70 / 0.47 pp (IID / γ=0.1 / γ=0.01 / γ=0.001 / Separate).

2. LAME on the same TDA predictions as GRASP. This separates the refinement rule from the input predictions. LAME and GRASP both have a setting (α and η) that controls how strongly neighbours influence each prediction. More influence buys accuracy on correlated streams at the cost of IID accuracy (Fig. 3), and the two defaults sit at different points of this trade-off: LAME's default smooths more, ending 1.3 pp lower in IID accuracy than GRASP's default on the same anchor. Comparing the defaults would therefore mix the method with how hard each smooths. α and η are on different scales, so equal values would not mean equal smoothing. We instead swept both settings (k = 3, 5, 10) and compared the methods where they give up the same IID accuracy, much like comparing classifiers at the same false-positive rate; any remaining difference then comes from the refinement rule itself.

GRASP − LAME at equal IID cost (TDA anchor, k=5). In each row, GRASP's strength η and LAME's strength α are chosen from their sweeps so that both give up the same IID accuracy; positive means GRASP is ahead on that stream type:

IID accuracy given up by each methodγ=0.1γ=0.01γ=0.001Separate
0.1 pp+0.11+0.57+0.82+0.91
0.6 pp (near GRASP's default)−0.11+0.39+0.86+1.19
1.6 pp−0.56−0.02+0.91+1.83
  • Separate: GRASP is ahead in every row, and its lead grows as more IID accuracy is given up (up to +2.7 pp across k; App. A.16, Table 18).
  • γ=0.001: GRASP is ahead in every row; LAME catches up only at about 4.6 pp of IID cost (Table 18).
  • γ=0.01: GRASP is ahead until about 1.6 pp, where the two tie.
  • γ=0.1: the two are essentially tied near GRASP's default; LAME is ahead once more IID accuracy is given up.

3. Why GRASP is safer than LAME. The difference lies in how each method combines a sample's prediction with its neighbours' predictions, not in the neighbour graph: GRASP's update applied on LAME's kNN graph changes its accuracy by at most 0.4 pp.

  • GRASP's output is a weighted average of the anchor predictions, so it can never be more confident than its most confident input.
  • LAME's output becomes more confident than its input in 22–60% of batches.
  • This matters in IID batches, where at k=5 only 19% of nearest-neighbour edges link same-class samples on average (1.5% on ImageNet). An update that can sharpen predictions amplifies these wrong links; an average of the anchors cannot.
AuthorsOpenReview ↗

JS gate (Q5), ablation and strength (Q1, W2), defaults (Q2)

Q5: Does JS move the whole curve, or only the operating point? It moves the whole curve.

  • Fig. 3 now plots the strength sweeps of GRASP and of GRASP without JS, as you asked. The two curves meet next to the anchor; once more than a few tenths of IID accuracy are spent, the gate lifts the curve by 0.4–0.8 pp on Separate. The table gives the Separate accuracy from the Fig. 3 sweeps (η for GRASP with and without JS, α for LAME, λ for GRASP-Base), read at the same IID loss for every method (k=5):
IID accuracy given up by each methodGRASPGRASP, no JSLAMEGRASP-Base
0.6 pp74.173.773.073.3
1.6 pp76.976.275.075.1
2.6 pp77.877.275.975.7
  • At equal IID accuracy, GRASP with JS beats GRASP without it in every stream type (+0.2 to +0.7 pp; Sec. 5.3, App. A.17 Table 19).
  • It is not a rescaling. At equal IID cost, JS beats a confidence-only gate (by 1.09 pp) and, near the default, a feature-similarity threshold that keeps the same total edge weight (by +0.4 to +2.6 pp). What helps is agreement between predictions.

Q1 and W2 (ablation vs. strength): You are right that two rows of the old Table 2 differ only in strength. We replaced it with an ablation at equal IID cost.

  • Converged GRASP-Base, [I + λ(I−S)]⁻¹P with λ=10, and the anchored random walk, [I + η(I−S)]⁻¹P with η=1, differ only in strength (S = D⁻¹W).
  • The question behind Table 2 is which components matter. In the new Table 2 every variant is compared at the same IID cost, and each component matters: 3 iterations instead of the exact solve cost 1.16 pp, the random-walk Laplacian 0.94 pp, and removing the gate 0.43 pp. GRASP-Base as a whole is 1.09 pp below GRASP.
  • Fig. 3 shows the strength trade-off itself and now also sweeps GRASP-Base's λ: at every IID cost, GRASP-Base stays below GRASP.

Q2: How were the defaults chosen? Once, before the test streams were built, and not tuned on them.

  • (μ, λ, k, τp) = (10, 10, 5, 1) for all datasets, streams, backbones and anchors. μ = λ gives the two terms equal weight; k = 5 is a small neighbourhood, common for batch kNN graphs; τp = 1 is the unit temperature. B = 128 was fixed with them; App. A.9 sweeps B from 8 to 1024 (B = 64 / 256 change IID by −0.36 / +0.18 pp).
  • After the main experiments, we checked that the default is not a lucky choice. If k is instead chosen on held-out datasets (leave one dataset out), the choice is k=10, a neighbouring value, and the comparisons with LAME and with the no-JS variant reach the same conclusions there (App. A.7; details in our response to WnPJ, Q4).

W2 (theory): Why does GRASP keep IID accuracy while GRASP-Base does not? GRASP-Base’s collapse comes from its high strength; at the same strength, GRASP’s gate halves the IID cost.

  • Both are anchored and both outputs are weighted averages of the anchors. Prop. 1 bounds how far either can move through the strength; as the paper says, it does not by itself predict accuracy.
  • GRASP-Base’s IID collapse comes mainly from its strength (λ=10). At strength 1 the same operator keeps 67.6% IID but reaches only 72.1% on Separate, versus 75.1% for GRASP.
  • At equal strength, the difference is which neighbours receive the weight. In IID batches most kNN edges join different classes. The gate removes about 38% of cross-class weight but only about 10% of same-class weight, so at the default strength GRASP loses 0.84 pp IID with the gate and 1.70 pp without it.
  • The remaining gap at equal IID cost comes from the design: the gate, the unnormalized Laplacian and the exact solve (Q1).

W1 (relation to prior smoothers): we agree on the lineage, and the revision states it. The solve is the regularized-Laplacian smoother of Zhou et al. and the "Smooth" step of Correct & Smooth. What GRASP adds:

  • Setting: a label-free graph over the current test batch only, discarded after it and anchored to an online adapter (Zhou et al. and Correct & Smooth use a fixed graph with training labels).
  • Edge weights: attenuated by disagreement between the anchor's predictions, which lifts the IID–Separate curve (Q5).
  • Evidence: at equal IID cost it beats LAME on class-sequential streams, and on DOTA or ADAPT it gives the best mean and worst-case accuracy tested.
AuthorsOpenReview ↗

Your two conditions (Q4, Q5)

You asked for a clear advantage over LAME and over a strength-matched no-JS variant.

  • Over the strength-matched no-JS variant (Q5): away from the anchor, GRASP's curve lies above the no-JS curve along the whole sweep (Fig. 3; +0.4 to +0.8 pp on Separate), and at equal IID accuracy JS is ahead in every stream type (+0.2 to +0.7 pp; Sec. 5.3, App. A.17, Table 19).
  • Over LAME, on both inputs you asked for (Q4):
    • Zero-shot CLIP (official LAME): GRASP is ahead in every stream type, by up to 4.64 pp (Table 1).
    • TDA predictions, at equal IID cost: GRASP is ahead on class-sequential streams at every cost and every k (up to +2.7 pp), and ahead or tied on every stream type near its default (Fig. 3; App. A.16, Table 18).
    • One setting per method, selected on held-out datasets and applied unchanged: GRASP is 1.21 pp better on the mean over the five stream types and better on all 11 datasets (App. A.16).
    • Behaviour: each GRASP output is a weighted average of the anchor predictions in the batch, so it can never be more confident than the most confident anchor (App. A.2). LAME's output becomes more confident than its input in 22–60% of batches (Sec. 5.3).

We believe this meets both conditions.

Dear Reviewer WjD9,

Thank you again for your review. As the discussion period ends on November 18, we wanted to check whether our responses and the revised PDF address your concerns. In particular, we compared GRASP with LAME on both zero-shot CLIP and TDA predictions, compared GRASP with a no-JS variant at the same IID cost, and explained where the defaults come from. We believe these results meet the two conditions you set out.

If any point is still unclear, or you would like an additional result, please let us know and we will reply promptly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: GRASP’s gains come from its design, not from strength or the anchor

We thank the reviewers for detailed reviews.

The question shared across the reviews is: does GRASP's gain come from its design, or from the TDA anchor, the propagation strength, or a missing baseline such as LAME? We answered it with new controlled experiments (same frozen streams, 11 datasets × 5 stream types, official LAME / DOTA / ADAPT code, paired 95% bootstrap CIs). The revised PDF is uploaded, with changes in blue.

How we compare. Every graph method has a strength setting, and more strength trades IID accuracy for accuracy on correlated streams. Comparing methods at fixed settings therefore mixes design with strength. We compare them at the same IID cost instead. We sweep the strength of every method, GRASP included, and compare them where each loses the same IID accuracy relative to the TDA anchor. All main results still use GRASP's single fixed default.

1. Every GRASP component matters (WjD9, vrjn, WnPJ). Each row replaces one GRASP component and is compared with GRASP at the same IID cost, close to that of our default. Every one is worse than GRASP in every non-IID stream type:

Replace in GRASPMean, 4 non-IID streamsSeparate
exact solve → 3 iterations (as in GRASP-Base)−1.16−1.29
JS gate → confidence-only gate−1.09−0.98
unnormalized → random-walk Laplacian (as in GRASP-Base)−0.94−0.79
JS gate → no gate−0.43−0.45
unnormalized → symmetric Laplacian−0.40−0.50

GRASP-Base itself, compared at the same IID cost, is 1.09 pp below GRASP on average, so GRASP's gain over GRASP-Base is not only a strength effect.

2. LAME (WjD9). Official LAME, as published on CLIP's predictions, is below GRASP in every stream type (by 4.64 pp under IID). To compare the two refinement rules on equal footing, we also run LAME on the same TDA predictions GRASP uses. There, at the same IID cost, GRASP is better on class-sequential streams at every cost (up to +2.7 pp) and ahead or within 0.13 pp on every stream type near GRASP's default.

3. Plug-in on stronger adapters (vrjn, WnPJ). Added unchanged to DOTA or ADAPT, GRASP costs at most 0.5 pp IID and adds +2.5 to +7.4 pp for γ ≤ 0.01, on 11/11 datasets. DOTA+GRASP gives the best average and ADAPT+GRASP the best worst-case accuracy of all methods tested.

4. Deployment (S3DX, vrjn). GRASP also works one sample at a time, with only past samples in both the cache and the graph. It still beats TDA, also run one sample at a time, by +3.8 to +7.4 pp for γ ≤ 0.01, keeping 72–88% of its batched gain, and matches the batched version under IID. The solve takes under 1 ms per batch of 128 on a GPU.

5. Hyperparameters (WjD9, vrjn, WnPJ). One default for every dataset, stream and anchor, fixed before the test streams were built and never changed. As an extra check after the main experiments, we asked what selection on held-out datasets would have chosen: a neighbouring setting (k=10), and the LAME and JS conclusions hold there too.

Revisions: the old Table 2 is replaced by the equal-IID-cost ablation (new Table 2) and the strength sweeps of GRASP, GRASP without JS, LAME and GRASP-Base (Fig. 3); LAME added as a baseline (Table 1); plug-in results on DOTA and ADAPT (Table 3); relation to Zhou et al. and Correct & Smooth; worst-case IID numbers; where the hyperparameters come from; symbols defined at first use and figures with consistent fonts; new appendix sections with full results, including one-sample-at-a-time inference and the video experiment with DOTA and ADAPT.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.