acceptodds
Withdrawn from ICLR 2027

Persistent-State Management for Streaming Test-Time Adaptation in Open-Vocabulary Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming test-time adaptation (TTA) changes the model while it is being evaluated, so deployment quality depends on the adaptation state carried from earlier samples. We study this persistent state as a first-class object in open-vocabulary semantic segmentation and introduce Profiled State Recovery, a practical framework for managing it. Profiled State Recovery uses a shared monitor–trigger–recovery–resume lifecycle around the native TTA update, while specializing recovery semantics to each adaptation mechanism. A profile is calibrated once on a labeled development stream, frozen, and then deployed label-free without target-stream retuning. Across MLMP, SAR, and TENT, three datasets, two ViT scales, and a ConvNeXt backbone, frozen profiles improve four of five systems, and all four improve on both fresh validation seeds, with mean gains of +4.204, +0.316, +0.491, and +2.881 mIoU points for S1, S2, S3, and S5. All three tested cross-dataset transfers are positive without target-stream retuning: +0.448 (MLMP), +0.602 (SAR), and +0.739 (TENT). The same frozen MLMP profile improves both ViT-L/14 and ViT-B/16. Mechanistic analysis shows that recovery granularity is a central design factor: local/progressive, selective late-state, and global recovery regimes emerge for MLMP, SAR, and TENT, respectively. These results establish persistent adaptation-state management as a structured design axis for streaming TTA and provide a compact, transferable implementation framework.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, open-vocabulary semantic segmentation, streaming adaptation, persistent state, state recovery, continual adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Vu6FOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how accumulated updates affect streaming test-time adaptation (TTA) for open-vocabulary semantic segmentation. The authors argue that these updates can both help and hurt performance over a long test stream. They propose Profiled State Recovery (PSR), which monitors the adaptation state and restores some or all adaptable parameters when a trigger condition is met, before resuming adaptation.

I have several major concerns about the paper.

First, the problem setup is difficult to follow. It took me considerable effort to understand how “persistent state” differs from the familiar distinction between episodic TTA, where the model is reset, and continual TTA, where updates carry over. The paper introduces too much terminology before explaining the basic setting, what existing methods fail to address, and what new research question it aims to answer.

Second, the motivation is not convincing. The argument seems to be that parameters may drift during continual adaptation, so a recovery mechanism is needed. However, the paper does not clearly demonstrate a reproducible failure of existing methods or explain why state recovery is a distinct research problem rather than an engineering extension of continual TTA.

Overall, the paper needs clearer writing, stronger justification for the method, and more convincing experiments. The terminology makes a relatively straightforward setting harder to understand, and the current evidence does not adequately support the framework or its conclusions.

Critical strengths

None

Critical weaknesses
  1. The writing and logical organization are highly confusing.

  2. The proposed method is technically very naive, while the manuscript substantially overcomplicates its presentation.

  3. The lack of qualitative segmentation results is a major experimental omission.

  4. The figures are difficult to interpret and their presentation quality is concerning.

Questions for authors

I do not believe the current manuscript meets the standard for acceptance at ICLR.

The underlying topic—managing accumulated adaptation state in long-running TTA systems—may be worth studying. However, the current paper does not clearly establish a sufficiently important new research problem, the proposed method appears technically limited, the presentation substantially overcomplicates a simple mechanism, and the experimental evidence is not convincing. In particular, for a paper centered on open-vocabulary semantic segmentation, the complete absence of qualitative segmentation results is difficult to justify.

Reviewer_3PmpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies streaming test-time adaptation (TTA) for open-vocabulary semantic segmentation (OVSS). In this setting the adaptation state persists from one sample to the next. The authors propose Profiled State Recovery, a monitor–trigger–recover–resume wrapper around the native TTA update:

  • Monitor: Adaptable state is partitioned into four ordered groups G1G_1–G4G_4. PSR computes the source-relative drift dg=∥θg−θg0∥2/∥θg0∥2d_g = \|\theta_g-\theta_g^0\|_2 / \|\theta_g^0\|_2 for each group, aggregates it as d=(∑gdg2)1/2d=(\sum_g d_g^2)^{1/2}, and standardizes it with frozen constants (cm,sm)(c_m, s_m).
  • Trigger: Recovery fires when z>τz > \tau for qq consecutive steps, subject to a cooldown.
  • Recover: A specific subset of groups is reset to its source weights (optimizer state, method history and RNG state are kept).
  • Profiling: The profile is selected from three manuallt chosen candidates per method on a labeled seed-42 development stream. It is then frozen and deployed without labels.

Claimed contributions:

  1. Treating persistent adaptation state as a first-class design object.
  2. The shared PSR lifecycle with frozen, label-free deployment.
  3. Method-specific recovery regimes:
    • local/progressive for MLMP (G1→G1+G2G_1 \to G_1{+}G_2)
    • selective late-state for SAR (G4→G1+G4G_4 \to G_1{+}G_4)
    • global for TENT (FULL)

Validation:

  • Five systems built from MLMP, SAR and TENT on NACLIP ViT-L/14, NACLIP ViT-B/16 and SED ConvNeXt-B backbones, using YTVIS, COCO and ACDC streams.
  • Two test seeds- the only comparison is against unmanaged persistent TTA.
  • mean Δ\Delta foreground mIoU: +4.20 (S1), +0.32 (S2), +0.49 (S3), −0.16 (S4), +2.88 (S5).
  • Cross-dataset transfer gives +0.45 (MLMP), +0.60 (SAR) and +0.74 (TENT). The same MLMP profile improves both ViT-L/14 and ViT-B/16.
Critical strengths
  • The frozen MLMP profile improves S1 by +4.54 / +3.87 mIoU and S5 by +2.90 / +2.86 mIoU
  • The identical profile works for both ViT-L/14 and ViT-B/16.
  • The negative result is reported rather than dropped: S4 is −0.15 / −0.17 mIoU, and Appendix F diagnoses it.
Critical weaknesses

1. The only baseline is unmanaged persistent TTA, so the advantage of PSR is not established.

  • Wang (2026, Freeze-to-Confirm). This work already studies persistent MLMP on YTVIS and COCO. Protocols may differ, but the paper presents itself as broadening that study, so F2C is the most natural baseline, and it is absent.

  • Lim et al. (ICLR 2026, ASR). ASR adaptively decides when and where to reset, motivated by the same premise as PSR: periodic resets ignore collapse risk, and full resets discard useful knowledge. The paper mentions this in intro but shows no results.

  • RDumb, CoTTA's stochastic restoration and PeTTA are not compared either.

  • Not cited at all: RDumb++ (drift-triggered full and soft resets), AMROD, PETAL and EATA.

  • SAR's built-in model-recovery reset is not discussed. Please compare against the closest feasible alternatives and explain any incompatibilities

  • The monitor adds little. In the WHEN rows of Table 3, state-drift timing differs from matched-count periodic resets by only −0.065-0.065 to +0.074+0.074 mIoU. The authors note that this control is not deployable, because it gets its intervention count from the drift run. That is why it needs a tuned-period periodic baseline.

2. The central claim that recovery granularity follows the adaptation mechanism cannot be identified from the experimental design.

  • Candidate families are shaped in advance (App. A):

    • every MLMP candidate starts at G1G_1
    • every SAR candidate starts at G4G_4
    • TENT's candidates are FULL / FULL / G3+G4G_3{+}G_4

    The regimes are therefore largely fixed beforehand. For example, G4G_4 was never available to MLMP.

  • Method and dataset are fully confounded in the main systems (MLMP–YTVIS, SAR–COCO, TENT–ACDC). The cross-dataset transfers uses the same source profile, so they test portability, not whether the same granularity would be selected on the target. The within method evidence that does exist points the other way:

    • S3 and S4 share the TENT profile but fall in different regimes (Sec. 7, Fig. 3).
    • App. D reports a cross-dataset action-rank correlation of ρ=−0.657\rho = -0.657 for TENT, and SAR changes regime across datasets.
  • Statistical support is thin.

    • There are two seeds and no confidence intervals.
    • SAR and TENT gains are 0.3–0.5 mIoU, and S2 on seed 47 gains only +0.045.
    • It is unclear whether the seed-42 calibration stream and the seed-47/48 test streams share images. This seems likely for ACDC's recurrent blocks and COCO's 800-image pool. If they do, labels of the test images informed profile selection.
Questions for authors

Clarifications that could change my assessment:

  1. Baselines: For S1–S5 on seeds 47/48, please report:

    • the source model (no adaptation)
    • episodic adaptation (MLMP's native per-batch reset)
    • RDumb with a tuned reset period
    • Lim et al.'s ASR
    • for S1/S5, Freeze-to-Confirm (Wang, 2026)
    • for S2, SAR with its native model recovery on vs. off

    Does persistent + PSR beat the best of these?

  2. Do the seed-42 calibration stream and the seed-47/48 test streams share any images? If so, what fraction, per dataset?

  3. How many recoveries fire per stream for each system, and at what intervals? Appendix F indicates S3 and S4 trigger only once per seed.

  4. What does the read-only routing forward cost per sample in latency and memory? Is it an extra forward pass? Please provide the details of this.

  5. Wang’s Freeze-to-Confirm; This work already reports that persistent vs. reset protocols change MLMP's method ordering, and it proposes LayerNorm restoration after confirmation. Section 2 repeats the ranking claim without numbers. What exactly is new here, and do any S1 settings or results overlap with that work?

Reviewer_Qs3tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates persistent adaptation state in streaming test-time adaptation (TTA) for open-vocabulary semantic segmentation. Rather than assuming stateless or periodically reset evaluation, the paper frames persistent adaptation state as a first-class managed deployment object. Then, it introduces Profiled State Recovery (PSR), a shared lifecycle for persistent-state management: monitor, trigger, method-conditioned recovery, and resume around the native TTA update. PSR is evaluated on 3 TTA methods, three datasets, two ViT scales, and a ConvNeXt backbone. The results show that four of five primary systems improve, and all four improve on both fresh validation seeds.

Critical strengths
  • The paper addresses a realistic operational deployment setting in streaming vision-language segmentation by treating long-horizon persistent adaptation state as a first-class object rather than assuming stateless or segment-reset evaluation.
  • The paper decomposes adaptable parameters into ordered structural groups (G1G_1–G4G_4), providing a useful alternative to coarse, binary (full-reset vs. continue) interventions.
  • The experiments show promising results where four of five primary systems improve.
Critical weaknesses
  • While dynamic monitor to trigger lifecycle is central claim of the paper, it does not clearly gain over fixed periodic resets. This means that such dynamic mechanizm is not necessary.
  • The experiments only compare the proposed PSR and vanilla TTA while missing some important baselines as in the related work such as Rdum, WHEN/WHERE, Freeze-to-Confirm, PeTTA.
  • Though investing all system combinations is heavy, only evaluating five combinations cause a doubt on effectiveness of PSR in general.
  • In addition, only using two seeds in the experiment is not convincing.
Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.