acceptodds
Under review as a conference paper at ICLR 2027

EvoCDS: Unifying Video Context-Dependent Concept Segmentation via Evolving Concept Discriminative Subspaces

Abstract

Unified image context-dependent (ICD) concept segmentation has made substantial progress, yet extending it to video requires more than temporal modeling. Across natural videos, medical videos, and 3D medical scenarios, video context-dependent (VCD) concept segmentation must handle heterogeneous concepts whose discriminative evidence evolves with the visual context, together with severe cross-task imbalance in video and frame counts. We characterize this temporal evolution as concept drift and propose EvoCDS, the first unified framework specifically designed for VCD concept segmentation. Motivated by the compact low-dimensional structure of concept-specific discriminative information, EvoCDS constructs an evolving concept discriminative subspace that preserves reliable historical directions while adapting to current evidence. We further introduce temporal clip balancing to stabilize joint learning across heterogeneous sequence datasets. Across eight tasks spanning natural and medical scenarios, EvoCDS consistently outperforms existing unified models and achieves competitive performance against specialized methods. We also construct Video-CD, a benchmark of 201 videos and 34,562 frames for evaluating robustness to concept drift. Together, EvoCDS and Video-CD provide a unified framework and benchmark for studying evolving CD concepts in dynamic visual scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Context-Dependent Concept Segmentation, Concept Drift, Unified Segmentation, Discriminative Subspace, Temporal Clip Balancing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_s38LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoCDS, a unified framework for context-dependent concept segmentation across six video tasks and two 3D medical segmentation tasks. It constructs a low-dimensional Concept Discriminative Subspace (CDS) from foreground–background prototype differences derived from annotated concept-definition examples. During sequential inference, high-confidence predictions update the subspace, while reconstruction error triggers the removal of outdated historical information. Temporal Clip Balancing supports joint training across imbalanced datasets. The paper evaluates eight tasks and introduces Video-CD, a benchmark containing drift-containing sequences.

Critical strengths

S1. EvoCDS provides a coherent and empirically motivated framework for unified segmentation. It combines shared visual features with task-specific foreground–background discriminative subspaces and adapts these subspaces during sequential inference. Appendix C’s spectral analysis and the rank ablations support the compact representation. This makes the approach a meaningful contribution toward handling heterogeneous segmentation tasks within one model.

S2. The breadth of evaluation and controlled experiments supports the method’s practical value. Results across eight natural and medical tasks demonstrate competitive performance, while joint/separate training, static/evolving subspaces, and historical/current-information ablations examine important design choices. These results are the main reason I lean toward acceptance despite the remaining questions about mechanism attribution.

Critical weaknesses

W1. The contribution of the specific evolution mechanism is not sufficiently isolated from simpler temporal adaptation. Static-subspace and component-removal ablations establish the usefulness of adaptation, but do not show whether repeated SVD and drift-aware resetting outperform moving-average prototype updates or fixed-length feature memory under matched conditions. This matters because subspace evolution is the central methodological contribution, and its advantage over simpler alternatives remains uncertain.

W2. The connection between the concept-drift formulation and its evaluation needs stronger validation. Appendix B explains the intended distinction from ordinary temporal variation, but Video-CD provides sequence-level annotations and evaluates entire drift-containing sequences. Consequently, the results do not isolate whether improvements specifically occur when discriminative evidence changes. This limits how strongly the experiments substantiate the paper’s central explanation, although the overall segmentation gains remain useful.

Questions for authors
  1. Were simpler temporal prototype-update or fixed-length-memory baselines evaluated under the same backbone and training protocol? If such results are available, how do they compare with the proposed evolution mechanism?
  2. Is there an existing analysis separating performance on relatively stable portions of sequences from portions with substantial changes in discriminative evidence? Such an analysis would help distinguish general segmentation improvements from improved drift robustness.
  3. What are the historical-memory capacity and retention policy, how are reconstruction-error statistics initialized and maintained, and when are these states reset? Please also report inference speed and memory usage if available; these details would help assess the practicality of repeated subspace updates.
6 comments
AuthorsOpenReview ↗

Response to Reviewer s38L (Part 1/3: W1, Q1)

Thank you very much for your insightful and constructive feedback on our submission. In response to the feedback from all three reviewers, we have revised the paper, and the changes are indicated in blue text within the manuscript. In the following, we provide a point-by-point response to your concerns. Our responses use the revised manuscript’s numbering unless otherwise noted. Quoted reviewer comments retain the original numbering.

Weakness 1 and Question 1: Comparison with Temporal Alternatives

W1: The contribution of the specific evolution mechanism is not sufficiently isolated from simpler temporal adaptation. Static-subspace and component-removal ablations establish the usefulness of adaptation, …

Q1: Were simpler temporal prototype-update or fixed-length-memory baselines evaluated under the same backbone and training protocol? If such results are available, how do they compare with the proposed evolution mechanism?

Thank you for pointing this out. We added two temporal-update baselines in Table 5 using the foreground–background representation of CDS.

  • EMA Baseline. Maintains an exponential moving average of unnormalized foreground–background prototype difference vectors with β=0.5\beta=0.5. Under identical initialization and valid update steps, averaging the difference vectors is equivalent to separately averaging the foreground and background prototypes and then taking their difference. Normalization is applied after averaging.

  • FIFO Baseline. Uses the same history buffer as EvoCDS, retaining the three most recent valid foreground–background discriminative directions while disabling reconstruction-error-based reset.

  • Matched Experimental Settings. Both baselines retain SVD-based subspace construction and disable reset. FIFO, EMA, and EvoCDS share the same Swin-B backbone, training schedule, concept-definition sampling protocol, subspace rank, confidence thresholds, and segmentation head architecture. Each method is retrained under its respective update rule and evaluated using the same protocol, allowing different temporal update strategies to be compared within a common discriminative representation.

  • Comparison Results. Table 5 shows that EvoCDS outperforms both FIFO and EMA across all eight tasks and both reported metrics. For example, shadow BER is 0.103 for EvoCDS, compared with 0.118 for FIFO and 0.116 for EMA, while polyp mDice is 0.848, compared with 0.838 and 0.825, respectively.

TaskMetricsFIFOEMAEvoCDS
CamouflagedMAE / SmS_m0.030 / 0.7980.031 / 0.7910.028 / 0.802
SalientMAE / SmS_m0.080 / 0.7380.076 / 0.7390.070 / 0.742
ShadowMAE / BER0.028 / 0.1180.029 / 0.1160.026 / 0.103
TransparentMAE / BER0.072 / 0.0710.074 / 0.0720.069 / 0.069
Thyroid NodulemDice / mIoU0.818 / 0.7190.820 / 0.7250.824 / 0.731
PolypmDice / mIoU0.838 / 0.7750.825 / 0.7610.848 / 0.783
COVID-19mDice / mIoU0.967 / 0.9460.959 / 0.9390.970 / 0.951
Liver TumormDice / mIoU0.868 / 0.8620.870 / 0.8670.876 / 0.870

These results demonstrate that the complete EvoCDS evolution strategy outperforms the tested fixed-length history retention and moving-average update strategies.

AuthorsOpenReview ↗

Response to Reviewer s38L (Part 2/3: W2, Q2)

Weakness 2 and Question 2: Benefits Under Discriminative Changes

W2: The connection between the concept-drift formulation and its evaluation needs stronger validation. Appendix B explains the intended distinction from ordinary temporal variation, …

Q2: Is there an existing analysis separating performance on relatively stable portions of sequences from portions with substantial changes in discriminative evidence? …

We agree that sequence-level improvements alone cannot distinguish general segmentation gains from improved robustness to concept drift. To address this concern, we added a within-sequence analysis that evaluates both effects separately.

  • Analysis Protocol. Using a frozen encoder and ground-truth (GT) masks, we measure adjacent-frame changes in foreground–background discriminative evidence through the cosine distance between their mean-feature difference vectors. Low-change portions comprise temporally contiguous runs selected from the lowest 30% of change scores, while high-change portions comprise seven-frame windows centered on frames with the highest 20% of change scores, including two preceding and four subsequent frames. Static and EvoCDS are evaluated on identical selected frames. GT masks are used only for post-hoc analysis and metric computation, not during inference.

The results reveal two complementary effects.

  • Benefits in Low-Change Portions. First, EvoCDS consistently improves segmentation even in low-change portions. For example, shadow BER decreases by 0.009, and polyp mDice increases by 0.005. These improvements indicate a general segmentation benefit of subspace evolution that is not exclusively attributable to drift adaptation.

  • Additional Benefits in High-Change Portions. Second, the gains become substantially larger in high-change portions across all eight tasks and both metrics. Shadow BER improvement increases from 0.009 to 0.018, polyp mDice improvement from 0.005 to 0.013, and COVID-19 mDice improvement from 0.004 to 0.009.

  • Temporal Behavior Around Changes. The temporal analysis further shows that improvements peak at the high-change centers and remain elevated afterward. For example, shadow BER improvements are 0.008, 0.029, and 0.021 before, at, and after substantial changes, respectively.

Overall gains in low- and high-change portions(Gain of EvoCDS over static (positive = better))

TaskMetricsLowHigh
CamouflagedMAE / SmS_m0.003 / 0.0050.006 / 0.016
SalientMAE / SmS_m0.002 / 0.0030.004 / 0.006
ShadowMAE / BER0.002 / 0.0090.004 / 0.018
TransparentMAE / BER0.002 / 0.0020.004 / 0.004
Thyroid NodulemDice / mIoU0.005 / 0.0050.011 / 0.012
PolypmDice / mIoU0.005 / 0.0020.013 / 0.004
COVID-19mDice / mIoU0.004 / 0.0040.009 / 0.009
Liver TumormDice / mIoU0.004 / 0.0050.009 / 0.011

Temporal gains within high-change windows(Gain of EvoCDS over static (positive = better))

TaskMetricsBeforeCenterAfter
CamouflagedMAE / SmS_m0.002 / 0.0060.010 / 0.0250.007 / 0.018
SalientMAE / SmS_m0.002 / 0.0030.005 / 0.0100.004 / 0.007
ShadowMAE / BER0.003 / 0.0080.005 / 0.0290.004 / 0.021
TransparentMAE / BER0.002 / 0.0020.006 / 0.0050.004 / 0.004
Thyroid NodulemDice / mIoU0.005 / 0.0060.017 / 0.0170.012 / 0.013
PolypmDice / mIoU0.004 / 0.0020.020 / 0.0050.015 / 0.004
COVID-19mDice / mIoU0.003 / 0.0040.014 / 0.0140.010 / 0.011
Liver TumormDice / mIoU0.004 / 0.0050.014 / 0.0160.011 / 0.012
  • Interpretation and Scope. These findings distinguish the general segmentation benefits from the additional gains associated with stronger discriminative changes. While the change scores provide a proxy rather than explicit annotations of drift events, the consistent high-change gains and their temporal concentration support the contribution of subspace evolution to improved robustness under changing foreground–background evidence. We therefore attribute the overall improvement to both effects, rather than to drift adaptation alone.
AuthorsOpenReview ↗

Response to Reviewer s38L (Part 3/3: Q3)

Question 3: Historical State and Computational Cost

Q3: What are the historical-memory capacity and retention policy, how are reconstruction-error statistics initialized and maintained, and when are these states reset?

  • Memory Capacity and Retention. All eight tasks use the same configuration, with subspace rank r=8r=8. Each sequence maintains a FIFO buffer containing at most three valid 64-dimensional historical directions. For each valid frame, the current foreground–background difference vector dcd_c is constructed from the preliminary logits yy and detached query features, as described in Sec. 3. The same direction is used for the current subspace update and subsequently retained in the FIFO buffer for future frames, rather than being recomputed from the final refined prediction. Directions from annotated concept-definition samples are maintained separately and are unaffected by history replacement or clearing.

  • Reconstruction-Error Statistics. The error window stores the eight most recent valid reconstruction errors. After at least four preceding valid errors have accumulated, their mean and population standard deviation define the threshold: τrec=μrec+2σrec\tau_{\mathrm{rec}}=\mu_{\mathrm{rec}}+2\sigma_{\mathrm{rec}}. The current error is compared with this threshold before being inserted into the window, so it does not influence its own decision threshold.

  • Reset Behavior and Invalid Updates. When the reconstruction error of the current valid direction exceeds the threshold, both the historical-direction buffer and the reconstruction-error window are cleared. The concept matrix is rebuilt from the annotated concept-definition directions and the current valid direction dcd_c, and SVD is recomputed. Subsequent valid frames rebuild the history and accumulate new reconstruction-error statistics. If a frame lacks valid high-confidence foreground or background regions, it does not contribute to the historical-direction buffer or reconstruction-error statistics.

  • Sequence Boundaries. During inference, both the historical-direction buffer and the reconstruction-error window are initialized as empty at the start of each video or CT volume. No historical directions or error statistics are transferred across sequences.

Q3 (continued): Please also report inference speed and memory usage if available; these details would help assess the practicality of repeated subspace updates.

  • Measurement Protocol. Table 6 reports inference cost on an NVIDIA RTX 5090 using FP32, batch size one, and 512×512512\times512 inputs. Memory measurements exclude initialization and warm-up.

  • Inference Speed and Memory. For Swin-B, static and evolving CDS take 26.63 and 35.84 ms/frame, respectively, giving an additional 9.21 ms/frame, while the evolving model achieves 27.90 FPS. Independent-process memory measurements give peak allocations of 904.577 MiB for static CDS and 904.583 MiB for evolving CDS. Across four backbones, evolution adds 3.96–10.87 ms/frame, and the full model achieves 27.15–29.10 FPS with 822–1826 MiB peak allocated GPU memory. The measured peak-memory increase is 6.5 KiB for each backbone.

Latency per frame

BackboneStatic (ms)Evolving (ms)Added (ms)FPS
Swin-B26.6335.84+9.2127.90
ConvNeXt-B23.4934.36+10.8729.10
Swin-L32.0536.84+4.7927.15
ConvNeXt-L32.5436.50+3.9627.40

Peak allocated GPU memory

BackboneStatic (MiB)Evolving (MiB)Added
Swin-B904.577904.583≈0
ConvNeXt-B822.174822.180≈0
Swin-L1825.9401825.947≈0
ConvNeXt-L1736.9141736.920≈0
Reviewer_zTgDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EvoCDS, a unified framework for video context-dependent concept segmentation across natural and medical scenarios. The authors identify concept drift as an important challenge in video segmentation, where the visual evidence used to distinguish foreground from background changes over time. To address this, they introduce an evolving concept discriminative subspace that combines foreground-background differences from concept definition samples with historical and current-frame information. A temporal clip balancing strategy is also proposed to support joint training across different tasks. The method is evaluated on eight segmentation tasks and shows strong performance compared with existing unified models. The authors additionally construct Video-CD, a benchmark containing 201 videos for evaluating segmentation robustness under concept drift.

Critical strengths
  • The paper addresses an interesting and meaningful problem. Extending context-dependent concept segmentation from images to videos is a natural but nontrivial direction, particularly when considering the diversity of concepts across natural and medical scenes. I appreciate the attempt to study these tasks within a unified framework rather than designing separate solutions for each one.

  • The proposed method is reasonably well motivated. Representing concept-specific information through foreground-background discriminative directions is an interesting approach, and evolving the subspace using both historical and current information provides a natural way to handle changing visual contexts. The overall framework is also relatively straightforward without relying on complicated task-specific architectures.

  • The experimental results are convincing overall. EvoCDS achieves strong performance across eight tasks, and the improvements over existing unified methods are fairly consistent. The authors evaluate several backbone architectures and provide extensive ablations to examine the contributions of the main components. These results suggest that the proposed framework is effective across different segmentation scenarios.

  • The Video-CD benchmark is a useful addition. The construction process combines multiple drift-related criteria with human verification, and the benchmark provides a more focused evaluation of segmentation performance on challenging video sequences. I also appreciate the additional analysis of the low-dimensional concept representations and their temporal behavior.

Critical weaknesses
  • The connection between concept drift and the proposed improvements could be explained more clearly. The paper groups several temporal challenges, including appearance variation, occlusion, deformation, and scene changes, under the notion of concept drift. While this perspective is reasonable, some of these challenges are also commonly studied in conventional video segmentation. It would be helpful to further clarify what distinguishes the proposed concept drift formulation from general temporal appearance changes, and which aspects of the method specifically address this distinction.

  • The contribution of the subspace evolution mechanism could be further analyzed. The ablation studies show that evolving the subspace performs better than keeping it fixed, which supports the proposed design. However, it is less clear how much of the improvement comes from the reconstruction-error-based reset mechanism itself. A comparison with simpler temporal update strategies, such as maintaining a fixed-size history or using moving-average prototypes, would help better establish the benefit of the proposed evolution strategy.

  • The evaluation of concept drift could be more fine-grained. Video-CD contains sequences with substantial concept drift, but the benchmark does not provide frame-level annotations of when drift occurs. Consequently, the reported segmentation metrics are averaged over entire sequences. Although the results demonstrate good performance on challenging videos, they do not directly show how the method behaves before, during, and after significant concept changes. A more detailed temporal analysis would make the conclusions about drift adaptation stronger.

  • Some practical aspects deserve further discussion. The subspace is updated using high-confidence predictions from the current frame, which may introduce errors when the model produces inaccurate masks under severe occlusion or abrupt scene changes. Although the authors acknowledge this limitation, an analysis of how such errors affect subsequent predictions would be useful. It would also be helpful to report the inference-time overhead of the evolving subspace, particularly the repeated SVD operations.

Questions for authors
  • Could the authors further clarify the distinction between concept drift and general temporal appearance variation? In particular, what types of changes are expected to benefit most from the proposed evolving subspace?

  • How important is the reconstruction-error-based reset mechanism compared with simpler temporal update strategies? Would continuously updating the subspace without resetting historical information lead to substantially different results?

  • Could the authors provide a more detailed analysis of performance under different degrees of concept drift? For example, does the advantage of EvoCDS over a static subspace become more pronounced in sequences with stronger appearance or contextual changes?

  • What is the additional inference cost introduced by the subspace evolution mechanism? Also, how sensitive is the method to errors in the high-confidence masks used for updating the current concept representation?

6 comments
AuthorsOpenReview ↗

Response to Reviewer zTgD (Part 1/4: W1, Q1)

Thank you very much for your insightful and constructive feedback on our submission. In response to the feedback from all three reviewers, we have revised the paper, and the changes are indicated in blue text within the manuscript. In the following, we provide a point-by-point response to your concerns. Our responses use the revised manuscript’s numbering unless otherwise noted. Quoted reviewer comments retain the original numbering.

Weakness 1 and Question 1: Concept drift and appearance variation

W1: The connection between concept drift and the proposed improvements could be explained more clearly. The paper groups several temporal challenges, ….

Q1: Could the authors further clarify the distinction between concept drift and general temporal appearance variation?

  • Discriminative Evidence. Our concept-drift formulation focuses on changes in foreground–background discriminative evidence while the segmentation concept remains fixed. The key distinction lies in what changes: general temporal appearance variation concerns changes in visual appearance across frames, whereas our formulation concerns changes in the discriminative evidence required to separate foreground from background.

  • Relationship to Appearance Variation. Thus, appearance variations such as illumination changes, deformation, or occlusion become relevant to concept drift only when they alter this discriminative evidence. For example, a new background distractor may weaken previously effective discriminative directions, while a shared feature shift in foreground and background may leave their discriminative difference unchanged.

  • Subspace Evolution and Adaptive Reset. EvoCDS addresses these changes through discriminative subspace evolution and adaptive reset. Subspace evolution incorporates current foreground–background directions estimated from high-confidence predictions alongside historical and annotated concept-definition directions, adapting feature projection and prototype matching. When reconstruction error exceeds the adaptive threshold, reset removes accumulated historical directions while retaining the annotated definition directions and the current valid direction, reducing reliance on outdated discriminative evidence.

Question 1: Expected benefits

Q1: In particular, what types of changes are expected to benefit most from the proposed evolving subspace?

  • Comparison Protocol. Table 4 compares static and evolving CDS on low- and high-change portions within each sequence. Low-change portions comprise contiguous runs selected from the lowest 30% of valid ctc_t values; high-change portions comprise seven-frame windows centered on frames with the highest 20% of scores, including two preceding and four subsequent frames.

  • Observed Benefits. Evolution improves both subsets, with larger overall gains on high-change portions across all eight tasks and both metrics. Shadow BER improvement increases from 0.009 to 0.018, and polyp mDice improvement from 0.005 to 0.013. These results support greater benefits when discriminative directions change more strongly. Full results appear in W3 & Q3.

AuthorsOpenReview ↗

Response to Reviewer zTgD (Part 2/4: W2, Q2)

Weakness 2 and Question 2: Temporal updates, history, and reset

W2: …, it is less clear how much of the improvement comes from the reconstruction-error-based reset mechanism itself. A comparison with simpler temporal update strategies, …

Q2: How important is the reconstruction-error-based reset mechanism compared with simpler temporal update strategies?

  • Temporal Baselines. We added FIFO and EMA baselines. FIFO retains the three most recent valid foreground–background directions, while EMA maintains an exponential moving average. Both retain subspace updates and omit reconstruction-error-based reset.

  • Matched Experimental Settings. Each baseline is retrained under its respective update rule, with the same Swin-B backbone, training schedule, concept-definition sampling protocol, subspace rank, confidence thresholds, and segmentation head.

  • Comparison Results. Table 5 shows that EvoCDS outperforms both baselines across all eight tasks and both metrics. Shadow BER is 0.103, compared with 0.118 for FIFO and 0.116 for EMA; polyp mDice is 0.848, compared with 0.838 and 0.825, respectively.

TaskMetricsFIFOEMAEvoCDS
CamouflagedMAE / SmS_m0.030 / 0.7980.031 / 0.7910.028 / 0.802
SalientMAE / SmS_m0.080 / 0.7380.076 / 0.7390.070 / 0.742
ShadowMAE / BER0.028 / 0.1180.029 / 0.1160.026 / 0.103
TransparentMAE / BER0.072 / 0.0710.074 / 0.0720.069 / 0.069
Thyroid NodulemDice / mIoU0.818 / 0.7190.820 / 0.7250.824 / 0.731
PolypmDice / mIoU0.838 / 0.7750.825 / 0.7610.848 / 0.783
COVID-19mDice / mIoU0.967 / 0.9460.959 / 0.9390.970 / 0.951
Liver TumormDice / mIoU0.868 / 0.8620.870 / 0.8670.876 / 0.870
  • Contribution of Historical Information. We further examine the contributions of historical information and reset. Table 7(b) compares EvoCDS with a variant that retains annotated concept definitions but uses only the current frame for adaptation. EvoCDS improves both metrics across all eight tasks, reducing shadow BER from 0.105 to 0.103 and increasing polyp mDice from 0.843 to 0.848, supporting the benefit of incorporating historical information.

  • Contribution of Reset. To isolate reset, Table 16 evaluates the same trained checkpoint with reset enabled or disabled on Video-CD, keeping frame-by-frame updates and the three-direction history capacity unchanged. Enabling reset improves both metrics across all eight tasks, reducing camouflaged MAE from 0.023 to 0.020 and shadow BER from 0.157 to 0.148, while increasing polyp mDice from 0.851 to 0.856.

TaskMetricsw/o ResetEvoCDS
CamouflagedMAE / SmS_m0.023 / 0.7530.020 / 0.759
SalientMAE / SmS_m0.081 / 0.7360.079 / 0.739
ShadowMAE / BER0.033 / 0.1570.031 / 0.148
TransparentMAE / BER0.079 / 0.0840.077 / 0.081
Thyroid NodulemDice / mIoU0.826 / 0.7390.832 / 0.743
PolypmDice / mIoU0.851 / 0.7850.856 / 0.789
COVID-19mDice / mIoU0.968 / 0.9490.970 / 0.951
Liver TumormDice / mIoU0.837 / 0.8270.841 / 0.832

Question 2: Continuous updating without reset

Q2: Would continuously updating the subspace without resetting historical information lead to substantially different results?

  • Comparison Outcome. Continuous updating remains effective, but consistently underperforms EvoCDS in Table 16.

  • Historical Evidence Management. Without reset, historical directions remain available until replaced through FIFO, regardless of their compatibility with current evidence. Reset excludes historical directions from the triggered reconstruction while retaining the concept-definition directions and the current valid direction dcd_c.

  • Additional Benefit of Reset. The gains are moderate but consistent across tasks, showing that reset provides an additional benefit beyond continuous subspace updating.

AuthorsOpenReview ↗

Response to Reviewer zTgD (Part 3/4: W3, Q3)

Weakness 3 and Question 3: Performance under discriminative changes

W3: The evaluation of concept drift could be more fine-grained. Video-CD contains sequences with substantial concept drift, …

Q3: Could the authors provide a more detailed analysis of performance under different degrees of concept drift? For example, does the advantage of EvoCDS over a static subspace become more pronounced in sequences with stronger appearance or contextual changes?

We analyze temporal performance and different degrees of discriminative change on Video-CD (Table 4).

  • Independent Change Measurement. Using a frozen encoder and ground-truth masks, we compute foreground–background mean-feature differences and measure their cosine distance between adjacent frames. This measure is used only for post-hoc analysis and does not rely on model predictions, reset events, or reconstruction errors.

  • Frame Selection and Temporal Windows. Within each sequence, frames with the highest 20% of change scores serve as centers for complete seven-frame windows comprising two preceding frames (Before), the center frame (Center), and four subsequent frames (After). Low-change portions comprise temporally contiguous runs selected from the lowest 30% of change scores, while high-change portions use the seven-frame windows defined above.

  • Matched Frame Evaluation. Static and EvoCDS are evaluated on identical selected frames, with positive differences indicating improvements.

Overall gains in low- and high-change portions(Gain of EvoCDS over static (positive = better))

TaskMetricsLowHigh
CamouflagedMAE / SmS_m0.003 / 0.0050.006 / 0.016
SalientMAE / SmS_m0.002 / 0.0030.004 / 0.006
ShadowMAE / BER0.002 / 0.0090.004 / 0.018
TransparentMAE / BER0.002 / 0.0020.004 / 0.004
Thyroid NodulemDice / mIoU0.005 / 0.0050.011 / 0.012
PolypmDice / mIoU0.005 / 0.0020.013 / 0.004
COVID-19mDice / mIoU0.004 / 0.0040.009 / 0.009
Liver TumormDice / mIoU0.004 / 0.0050.009 / 0.011

Temporal gains within high-change windows(Gain of EvoCDS over static (positive = better))

TaskMetricsBeforeCenterAfter
CamouflagedMAE / SmS_m0.002 / 0.0060.010 / 0.0250.007 / 0.018
SalientMAE / SmS_m0.002 / 0.0030.005 / 0.0100.004 / 0.007
ShadowMAE / BER0.003 / 0.0080.005 / 0.0290.004 / 0.021
TransparentMAE / BER0.002 / 0.0020.006 / 0.0050.004 / 0.004
Thyroid NodulemDice / mIoU0.005 / 0.0060.017 / 0.0170.012 / 0.013
PolypmDice / mIoU0.004 / 0.0020.020 / 0.0050.015 / 0.004
COVID-19mDice / mIoU0.003 / 0.0040.014 / 0.0140.010 / 0.011
Liver TumormDice / mIoU0.004 / 0.0050.014 / 0.0160.011 / 0.012
  • Before, Center, and After Changes. The temporal analysis shows that EvoCDS achieves the largest gains at Center, with improvements remaining higher in After than Before across all eight tasks and both metrics. For example, shadow BER improvements are 0.008, 0.029, and 0.021 for Before, Center, and After, respectively; polyp mDice improvements are 0.004, 0.020, and 0.015; and COVID-19 mDice improvements are 0.003, 0.014, and 0.010.

  • Low- and High-Change Portions. Comparing different degrees of change, EvoCDS consistently achieves larger gains in high-change portions across all eight tasks and both metrics. Shadow BER improvement increases from 0.009 to 0.018, thyroid-nodule mDice from 0.005 to 0.011, and polyp mDice from 0.005 to 0.013. Similar trends appear in COVID-19 mDice (0.004 to 0.009) and liver-tumor mIoU (0.005 to 0.011).

These results indicate that subspace evolution provides greater benefits when foreground–background discriminative evidence changes more strongly, with the strongest relative gains occurring at high-change frames and extending into subsequent frames.

Reviewer_8EiWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EvoCDS, a single model for six video and two CT-sequence segmentation tasks that represents each concept by a rank-8 SVD basis of foreground–background prototype differences from 12 definition samples, updated online from confident predictions and reset when reconstruction error is high. It also introduces temporal clip balancing for joint training on imbalanced datasets and Video-CD, a drift-focused test subset of 201 videos and 34,562 frames, and reports the best unified-model results in Table 2.

Critical strengths
  1. Modeling each concept as a low-rank subspace of foreground–background differences is an elegant and easily inspected idea, and the rank analysis in Appendix C makes the choice of eight directions convincing.

  2. With matched backbones the method consistently improves on SR-ICL and Spider, for instance lowering shadow BER from 0.159 to 0.103 against SR-ICL.

Critical weaknesses
  1. The core mechanism, a subspace basis updated incrementally and adapted from the model's own confident pixels, is closely related to incremental subspace tracking in IVT by Ross et al. 2008 and to online-adapted VOS in OnAVOS by Voigtlaender and Leibe 2017, yet neither is discussed. Beyond CamSAM2 as a camouflage-only specialist, memory-based SAM2-family segmenters such as Medical SAM 2 and MedSAM2, which already treat CT volumes as videos, are not positioned, so it is unclear what the method adds beyond relabeling occlusion, deformation and scene change as concept drift.

  2. The online update is self-referential, because dcd_c comes from pixels the model itself predicts above 0.7 or below 0.3 in Eq. 8 and a reset rebuilds DD from the definition samples plus this same dcd_c, so a confidently wrong direction survives the step meant to discard outdated evidence. The reset rule, which compares the unnormalized error of Eq. 9 against μrec+2σrec\mu_{rec}+2\sigma_{rec}, is a heuristic never validated against actual drift because Video-CD has only video-level labels per Appendix I.4, and the error propagation acknowledged in Appendix J is left unanalyzed.

  3. Removing subspace evolution costs only 0.005 camouflaged MAE and 0.011 liver mDice in Table 4b, with similarly small gaps on Video-CD in Table 7, no seed variance, and rr, λ\lambda and the confidence thresholds tuned on the test sets per Tables 4c, 9 and 10. The comparison with specialized models also mixes protocols, since SODA at BER 0.136 and Vivim at mDice 0.815 fall below their published ViSha and VTUS results of 0.091 and 0.832, which would beat or tie the best EvoCDS entries of 0.103 and 0.833.

Questions for authors
  1. The definition prototypes come from a frozen encoder in Eqs. 1–3, while dcd_c and the projected queries come from the learnable stream's QtQ_t in Eqs. 5 and 8, so how do the two share one space when dcd_c is appended to DD and QtQ_t is projected onto UrU_r? Does the low-rank evidence in Appendix C, gathered on frozen Swin features with C=1024C=1024 and ground-truth masks, carry over to the C=64C=64 space where the CDS operates per Appendix E, and do gradients flow through the SVD during training?

  2. Since CT slices are ordered spatially rather than temporally in Sec. 4.1, what does concept drift mean along the slice axis, and are the history, μrec\mu_{rec} and σrec\sigma_{rec} cleared at video and volume boundaries at inference, given that training clips may span several videos? What prevents a confidently wrong dcd_c from persisting after a reset, and how often are resets triggered per sequence?

  3. Which baseline numbers in Tables 2–3 did you retrain and which come from prior papers, and why do SODA and Vivim fall below their published results while SR-ICL differs between Tables 2 and 8, with shadow BER at 0.159 against 0.152? Since the COVID-19 columns of Table 3 match Table 2 for every method, including 0.970 and 0.951 for EvoCDS with Swin-B, does Video-CD select any drift-specific subset for that task?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8EiW (Part 1/4: W1)

Thank you very much for your insightful and constructive feedback on our submission. In response to the feedback from all three reviewers, we have revised the paper, and the changes are indicated in blue text within the manuscript. In the following, we provide a point-by-point response to your concerns. Our responses use the revised manuscript’s numbering unless otherwise noted. Quoted reviewer comments retain the original numbering.

Weakness 1: IVT and OnAVOS

W1: The core mechanism, a subspace basis update dincrementally and adapted from the model's own confident pixels,…

We acknowledge that the confident-pixel selection follows OnAVOS, adopting its idea of using high-confidence predictions for online adaptation. There are three key distinctions between EvoCDS and IVT/OnAVOS:

  • 1.Representation Learning (vs. IVT). IVT learns a generative model of target appearance through incremental PCA. In contrast, EvoCDS constructs a concept-conditioned foreground–background discriminative subspace, initialized from annotated concept-definition samples across videos.

  • 2.Adaptation Strategy (vs. IVT). IVT downweights older observations with a forgetting factor. Rather than continuously discounting older observations, EvoCDS resets historical directions when the current direction's reconstruction error exceeds an adaptive threshold.

  • 3.Parameter Updating (vs. OnAVOS). OnAVOS adapts network parameters through test-time backpropagation, whereas EvoCDS keeps network parameters fixed and computes a compact 64×864\times8 basis using SVD for feature projection and prototype matching.

  • Key Distinction. Thus, EvoCDS differs in both the representation being adapted and its role in segmentation: it evolves discriminative directions for a fixed segmentation concept, rather than tracking target appearance or updating network parameters. These distinctions are clarified in the expanded Related Work (p. 3).

Weakness 1: Medical SAM 2 and MedSAM2

W1: Beyond CamSAM2 as a camouflage-only specialist, …

  • Memory Representations. Medical SAM 2 selects informative memory embeddings and supports transfer from prompted templates to unrelated 2D images, while MedSAM2 uses streaming memory to condition segmentation through memory attention. Both methods incorporate historical information through memory representations.

  • Discriminative Subspace. In contrast, EvoCDS explicitly evolves a foreground–background discriminative subspace initialized from annotated concept-definition samples. Historical and current discriminative directions determine the subspace used for feature projection and prototype matching, while network parameters remain fixed.

  • Role in Segmentation. Thus, the key distinction lies in what is adapted and how it influences segmentation: memory representations for conditioning predictions versus discriminative directions for projection and matching.

  • Evaluation Protocols. MedSAM2’s reported 3D evaluation uses a bounding-box prompt on the target’s middle slice, whereas EvoCDS relies on external concept-definition samples without query-volume annotations. Their reported results are therefore not directly comparable. Medical SAM 2’s one-prompt setting is closely related to our external-reference setting. We additionally evaluated a self-sorting variant adapted to use 12 reference samples. The 12 reference image–mask pairs jointly initialize the memory bank, and we ensure a consistent feature layout for similarity computation. | Concept | mDice (%) | mIoU (%) | |---|---:|---:| | COVID-19 | .918 | 901 | | Liver Tumor | .836 | .832 |

AuthorsOpenReview ↗

Response to Reviewer 8EiW (Part 2/4: W1 (continued))

Weakness 1: Concept Drift and Appearance Variation

W1: … so it is unclear what the method adds beyond relabeling occlusion, deformation and scene change as concept drift.

  • Conceptual Distinction. Our concept-drift formulation focuses on changes in foreground–background discriminative evidence while the segmentation concept remains fixed. We do not equate appearance variations, such as illumination changes, deformation, or occlusion, with concept drift. They become relevant to our formulation when they alter the evidence used to distinguish foreground from background.

  • Discriminative Change Measurement. To examine this distinction, we measure changes in foreground–background discriminative directions using frozen features and ground-truth masks: gt=μtfg−μtbgg_t=\mu_t^{\mathrm{fg}}-\mu_t^{\mathrm{bg}},ct=1−gt⊤gt−1∥gt∥2∥gt−1∥2c_t=1-\frac{g_t^\top g_{t-1}}{\lVert g_t\rVert_2\lVert g_{t-1}\rVert_2}. An identical additive shift in foreground and background features cancels in gtg_t, illustrating that appearance changes do not necessarily alter discriminative evidence.

  • Supporting Evidence. The within-sequence analysis in Table 4 further supports this distinction (Gain of EvoCDS over static (positive = better)) of EvoCDS over static CDS, with positive values indicating improvement. For the low- and high-change partitioning, please refer to our response to Reviewer zTgD (Part 3/4: W3 and Q3).). Although EvoCDS improves performance in both low- and high-change portions, its gains are consistently larger in high-change portions across all eight tasks and both metrics. These results indicate that subspace evolution is particularly beneficial when foreground–background discriminative evidence changes, directly supporting the motivation for adapting discriminative directions.

SubsetCamouflagedSalientShadowTransparentThyroid NodulePolypCOVID-19Liver Tumor
MetricsMAE/SmS_mMAE/SmS_mMAE/BERMAE/BERmDice/mIoUmDice/mIoUmDice/mIoUmDice/mIoU
Low0.003/0.0050.002/0.0030.002/0.0090.002/0.0020.005/0.0050.005/0.0020.004/0.0040.004/0.005
High0.006/0.0160.004/0.0060.004/0.0180.004/0.0040.011/0.0120.013/0.0040.009/0.0090.009/0.011

Question 1: Feature-Space Alignment

Q1: …, so how do the two share one space when dcd_c is appended to DD and QtQ_t is projected onto UrU_r?

  • Definition Representations. The definition encoder is frozen, and its backbone features are mapped to 64 dimensions by a trainable linear projection followed by LayerNorm.

  • Query Representations. The segmentation stream produces 64-channel decoder features as query representations.

  • Joint Optimization. The linear projection and segmentation stream are jointly optimized by the segmentation loss through subspace projection and prototype matching. Definition prototypes, current-frame directions, and historical directions are constructed in this learned 64-dimensional space. The concept matrix therefore has 64 rows, and its retained basis is Ur∈R64×8U_r\in\mathbb{R}^{64\times8}.

Question 1: Low-Rank Evidence and Gradient Flow

Q1: .., carry over to the C=64C = 64 space where the CDS operates per Appendix E, and do gradients flow through the SVD during training?

  • Low-Rank Analysis.

To examine the representation used by CDS, Table 8 analyzes the learned 64-dimensional discriminative directions from trained EvoCDS. We sample 128 examples per task from the training sets and report the mean R8R_8 over their corresponding CDS matrices. We use the same column normalization as CDS before SVD. The cumulative energy retained by the first eight components,, R8R_8, is:

ConceptCamouflagedSalientShadowTransparentThyroid NodulePolypCOVID-19Liver Tumor
R8R_8 (%)93.9492.9794.6497.5495.4697.5099.6096.82

The first eight directions retain 92.97%–99.60% of the energy across all eight tasks, providing direct evidence of compact discriminative structure in the space where CDS operates. This analysis uses no additional training or test-set annotations.

  • Gradient Flow.

The final SVD remains in the training computation graph. Gradients reach the definition-side linear projection through differentiable subspace construction and prototype matching, and reach the segmentation stream through query-feature projection and prototype matching. The definition encoder remains frozen. The current update direction dcd_c is constructed from detached predictions and detached query features; historical directions are also detached. Gradients therefore do not propagate through prediction-based update construction, historical states, or the discrete reset decision. During inference, the discriminative subspace is updated while network parameters remain fixed.

AuthorsOpenReview ↗

Response to Reviewer 8EiW (Part 3/4: W2, Q2)

Weakness 2 and Question 2: Retention of Current Directions

W2: The online update is self-referential, …

Q2: What prevents a confidently wrong dcd_c from persisting after a reset, …

  • Retention of Current Evidence. Reset removes accumulated historical directions while retaining annotated concept-definition directions and the current valid dcd_c. The definition directions preserve the concept reference, while the current direction allows the reconstructed CDS to immediately incorporate current discriminative evidence.

  • Bounded Historical Retention. To limit the persistence of potentially incorrect directions, the three-entry FIFO buffer removes each stored direction after three subsequent valid updates, or earlier if another reset occurs. Thus, reset removes outdated historical evidence while FIFO limits the retention of individual update directions.

  • Remaining Limitation. Since reconstruction error measures subspace consistency without assessing prediction correctness, subsequent incorrect predictions may still introduce erroneous updates.

  • Reset Ablation. Table 16 evaluates the same trained EvoCDS checkpoint with reset enabled or disabled during inference, keeping frame-by-frame updates and the three-direction history buffer unchanged. Enabling reset improves both reported metrics across all eight Video-CD tasks, reducing shadow BER from 0.157 to 0.148 and increasing polyp mDice from 0.851 to 0.856.

MethodCamouflagedSalientShadowTransparentThyroid NodulePolypCOVID-19Liver Tumor
MetricsMAE/SmS_mMAE/SmS_mMAE/BERMAE/BERmDice/mIoUmDice/mIoUmDice/mIoUmDice/mIoU
w/o Reset0.023/0.7530.081/0.7360.033/0.1570.079/0.0840.826/0.7390.851/0.7850.968/0.9490.837/0.827
EvoCDS0.020/0.7590.079/0.7390.031/0.1480.077/0.0810.832/0.7430.856/0.7890.970/0.9510.841/0.832

Weakness 2 and Question 2: Reset Statistics and Update Errors

W2: The reset rule, …

Q2: … and how often are resets triggered per sequence?

  • Reset Frequency. We additionally measured reset frequency across tasks and used the reset statistics to examine the association between reset events and foreground–background discriminative changes.

  • Association with Discriminative Changes. Based on the independently defined ctc_t grouping, the highest-change 20% of frames contain 59%–80% of all resets, compared with 18%–22% in randomly sampled 20% subsets. These results demonstrate a strong association between reset events and discriminative changes. Although the unnormalized reconstruction residual is sensitive to feature magnitude, the observed concentration indicates that reset events are closely associated with changes in discriminative evidence.

The last two rows report the percentage of all resets occurring in the highest-change or randomly sampled 20% of frames/slices.

MetricCamouflagedSalientShadowTransparentThyroid NodulePolypCOVID-19Liver Tumor
Resets/seq.: median [min,max]3 [1, 14]7 [2, 25]7 [3, 10]4 [1, 12]12 [5, 28]7 [2, 10]10 [2, 17]13 [7, 20]
Resets/100 frames/slices77777842
Resets in top 20% (%)7263797862725980
Resets in random 20% (%)1822222122191820
  • Sensitivity to Update Errors. Table 12 compares predicted masks, GT masks, and randomly perturbed GT masks as update inputs. Replacing predicted masks with GT masks improves all reported metrics by no more than 0.004, providing an empirical upper bound on the performance loss due to error propagation under this evaluation setting. When 50% of pixel labels in GT-derived update masks are randomly flipped, polyp mDice decreases from 0.859 to 0.851 and liver tumor mDice from 0.845 to 0.837. These results further characterize the impact of random update noise on overall segmentation performance.
Update maskCamouflagedSalientShadowTransparentThyroid NodulePolypCOVID-19Liver Tumor
MetricsMAE/SmS_mMAE/SmS_mMAE/BERMAE/BERmDice/mIoUmDice/mIoUmDice/mIoUmDice/mIoU
GT0.019/0.7620.077/0.7420.030/0.1450.075/0.0790.835/0.7460.859/0.7930.973/0.9540.845/0.836
10%0.020/0.7600.078/0.7410.031/0.1470.076/0.0800.834/0.7440.858/0.7920.971/0.9520.844/0.834
30%0.023/0.7570.079/0.7380.032/0.1530.076/0.0810.830/0.7440.853/0.7860.969/0.9500.841/0.830
50%0.023/0.7540.080/0.7360.033/0.1570.079/0.0800.828/0.7400.851/0.7840.968/0.9470.837/0.827
EvoCDS0.020/0.7590.079/0.7390.031/0.1480.077/0.0810.832/0.7430.856/0.7890.970/0.9510.841/0.832

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their constructive feedback and recognition of our work. Their comments have been instrumental in improving the clarity and completeness of our paper. We also extend our gratitude to the Area Chair for the time and careful oversight of this submission.

Below, we summarize the strengths highlighted by the reviewers and our responses to their main concerns.

Consensus on Strengths:

  • Elegant and well-motivated framework (R#8EiW, R#zTgD, R#s38L). All reviewers recognize the foreground–background discriminative subspace formulation as an intuitive and meaningful approach to unified concept segmentation.

  • Strong and consistent empirical performance (R#zTgD, R#s38L). Both reviewers acknowledge the improvements over existing methods across diverse tasks, supported by controlled comparisons and extensive ablations.

  • Well-supported low-rank representation (R#8EiW, R#s38L). Reviewers appreciate the compact subspace design, with R#8EiW and R#s38L explicitly highlighting the supporting spectral analysis and rank ablations.

  • Well-motivated temporal adaptation (R#zTgD, R#s38L). Both reviewers recognize the motivation for evolving subspaces using historical and current information.

  • Valuable Video-CD benchmark (R#zTgD). R#zTgD highlights the benchmark's contribution to evaluating challenging video sequences with concept drift.

Main Concerns:

We have addressed the reviewers’ central concerns through the following clarifications and additional evidence:

  • Methodological Contribution (R#8EiW). We clarified how EvoCDS differs from existing subspace-tracking, online-adaptation, and memory-based methods. Its contribution is a concept-specific foreground–background discriminative subspace that evolves during inference, enabling unified segmentation across eight natural and medical tasks while keeping network parameters fixed.

  • Effectiveness of Evolution and Reset (R#8EiW, R#zTgD, R#s38L). We added matched comparisons with retrained FIFO and EMA baselines and a fixed-checkpoint reset ablation (Tables 5 and 16). EvoCDS outperforms both temporal alternatives across all eight tasks, and reset provides additional gains beyond continuous subspace updates. These results directly support the proposed evolution strategy.

  • Evidence for Drift Adaptation (R#8EiW, R#zTgD, R#s38L). We added a within-sequence analysis that measures discriminative changes independently of model predictions and reset decisions (Table 4). Evolution yields larger improvements on high-change portions across all eight tasks, connecting the performance gains to the changes in foreground–background evidence that motivate our method.

  • Validity of the Low-Rank Representation and Hyperparameter Selection (R#8EiW). We clarified feature alignment and gradient flow and added spectral analysis in the learned 64-dimensional space (Table 8). The first eight directions retain 92.97%–99.60% of the discriminative energy, supporting the compact representation used by EvoCDS. The subspace rank, reset multiplier, and confidence thresholds were selected based on prior experience and fixed before test-set evaluation.

  • Update Reliability and Computational Cost (R#8EiW, R#zTgD, R#s38L). Additional perturbation experiments demonstrate tolerance to the tested random update-mask errors (Table 12). Runtime measurements quantify the cost of evolution: Swin-B EvoCDS achieves 27.90 FPS, with an additional 9.21 ms/frame on an RTX 5090 (Table 6).

We appreciate the reviewers' time and consideration in reviewing our responses. We hope that the additional experiments and clarifications have addressed the concerns raised, and we welcome any further questions or discussion.

Best regards,

All #10001 Authors

AuthorsOpenReview ↗

Final Remarks

We sincerely thank all reviewers for their thoughtful evaluations, constructive feedback, and active engagement throughout the rebuttal period. We are particularly grateful that all reviewers have expressed positive and supportive views of our work, and we greatly appreciate their constructive discussions and continued engagement.

We are grateful for the reviewers' careful consideration of our responses and additional experimental results. We are pleased that our clarifications and supplementary analyses have helped address the concerns raised and provide a clearer understanding of our work and its contributions.

We thank all reviewers and the Area Chair again for their valuable time and careful consideration.

Best regards,
All #10001 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.