- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoCDS, a unified framework for context-dependent concept segmentation across six video tasks and two 3D medical segmentation tasks. It constructs a low-dimensional Concept Discriminative Subspace (CDS) from foreground–background prototype differences derived from annotated concept-definition examples. During sequential inference, high-confidence predictions update the subspace, while reconstruction error triggers the removal of outdated historical information. Temporal Clip Balancing supports joint training across imbalanced datasets. The paper evaluates eight tasks and introduces Video-CD, a benchmark containing drift-containing sequences.
Critical strengths
S1. EvoCDS provides a coherent and empirically motivated framework for unified segmentation. It combines shared visual features with task-specific foreground–background discriminative subspaces and adapts these subspaces during sequential inference. Appendix C’s spectral analysis and the rank ablations support the compact representation. This makes the approach a meaningful contribution toward handling heterogeneous segmentation tasks within one model.
S2. The breadth of evaluation and controlled experiments supports the method’s practical value. Results across eight natural and medical tasks demonstrate competitive performance, while joint/separate training, static/evolving subspaces, and historical/current-information ablations examine important design choices. These results are the main reason I lean toward acceptance despite the remaining questions about mechanism attribution.
Critical weaknesses
W1. The contribution of the specific evolution mechanism is not sufficiently isolated from simpler temporal adaptation. Static-subspace and component-removal ablations establish the usefulness of adaptation, but do not show whether repeated SVD and drift-aware resetting outperform moving-average prototype updates or fixed-length feature memory under matched conditions. This matters because subspace evolution is the central methodological contribution, and its advantage over simpler alternatives remains uncertain.
W2. The connection between the concept-drift formulation and its evaluation needs stronger validation. Appendix B explains the intended distinction from ordinary temporal variation, but Video-CD provides sequence-level annotations and evaluates entire drift-containing sequences. Consequently, the results do not isolate whether improvements specifically occur when discriminative evidence changes. This limits how strongly the experiments substantiate the paper’s central explanation, although the overall segmentation gains remain useful.
Questions for authors
- Were simpler temporal prototype-update or fixed-length-memory baselines evaluated under the same backbone and training protocol? If such results are available, how do they compare with the proposed evolution mechanism?
- Is there an existing analysis separating performance on relatively stable portions of sequences from portions with substantial changes in discriminative evidence? Such an analysis would help distinguish general segmentation improvements from improved drift robustness.
- What are the historical-memory capacity and retention policy, how are reconstruction-error statistics initialized and maintained, and when are these states reset? Please also report inference speed and memory usage if available; these details would help assess the practicality of repeated subspace updates.
6 comments
Response to Reviewer s38L (Part 2/3: W2, Q2)
Weakness 2 and Question 2: Benefits Under Discriminative Changes
W2: The connection between the concept-drift formulation and its evaluation needs stronger validation. Appendix B explains the intended distinction from ordinary temporal variation, …
Q2: Is there an existing analysis separating performance on relatively stable portions of sequences from portions with substantial changes in discriminative evidence? …
We agree that sequence-level improvements alone cannot distinguish general segmentation gains from improved robustness to concept drift. To address this concern, we added a within-sequence analysis that evaluates both effects separately.
- Analysis Protocol. Using a frozen encoder and ground-truth (GT) masks, we measure adjacent-frame changes in foreground–background discriminative evidence through the cosine distance between their mean-feature difference vectors. Low-change portions comprise temporally contiguous runs selected from the lowest 30% of change scores, while high-change portions comprise seven-frame windows centered on frames with the highest 20% of change scores, including two preceding and four subsequent frames. Static and EvoCDS are evaluated on identical selected frames. GT masks are used only for post-hoc analysis and metric computation, not during inference.
The results reveal two complementary effects.
-
Benefits in Low-Change Portions. First, EvoCDS consistently improves segmentation even in low-change portions. For example, shadow BER decreases by 0.009, and polyp mDice increases by 0.005. These improvements indicate a general segmentation benefit of subspace evolution that is not exclusively attributable to drift adaptation.
-
Additional Benefits in High-Change Portions. Second, the gains become substantially larger in high-change portions across all eight tasks and both metrics. Shadow BER improvement increases from 0.009 to 0.018, polyp mDice improvement from 0.005 to 0.013, and COVID-19 mDice improvement from 0.004 to 0.009.
-
Temporal Behavior Around Changes. The temporal analysis further shows that improvements peak at the high-change centers and remain elevated afterward. For example, shadow BER improvements are 0.008, 0.029, and 0.021 before, at, and after substantial changes, respectively.
Overall gains in low- and high-change portions(Gain of EvoCDS over static (positive = better))
| Task | Metrics | Low | High |
|---|---|---|---|
| Camouflaged | MAE / | 0.003 / 0.005 | 0.006 / 0.016 |
| Salient | MAE / | 0.002 / 0.003 | 0.004 / 0.006 |
| Shadow | MAE / BER | 0.002 / 0.009 | 0.004 / 0.018 |
| Transparent | MAE / BER | 0.002 / 0.002 | 0.004 / 0.004 |
| Thyroid Nodule | mDice / mIoU | 0.005 / 0.005 | 0.011 / 0.012 |
| Polyp | mDice / mIoU | 0.005 / 0.002 | 0.013 / 0.004 |
| COVID-19 | mDice / mIoU | 0.004 / 0.004 | 0.009 / 0.009 |
| Liver Tumor | mDice / mIoU | 0.004 / 0.005 | 0.009 / 0.011 |
Temporal gains within high-change windows(Gain of EvoCDS over static (positive = better))
| Task | Metrics | Before | Center | After |
|---|---|---|---|---|
| Camouflaged | MAE / | 0.002 / 0.006 | 0.010 / 0.025 | 0.007 / 0.018 |
| Salient | MAE / | 0.002 / 0.003 | 0.005 / 0.010 | 0.004 / 0.007 |
| Shadow | MAE / BER | 0.003 / 0.008 | 0.005 / 0.029 | 0.004 / 0.021 |
| Transparent | MAE / BER | 0.002 / 0.002 | 0.006 / 0.005 | 0.004 / 0.004 |
| Thyroid Nodule | mDice / mIoU | 0.005 / 0.006 | 0.017 / 0.017 | 0.012 / 0.013 |
| Polyp | mDice / mIoU | 0.004 / 0.002 | 0.020 / 0.005 | 0.015 / 0.004 |
| COVID-19 | mDice / mIoU | 0.003 / 0.004 | 0.014 / 0.014 | 0.010 / 0.011 |
| Liver Tumor | mDice / mIoU | 0.004 / 0.005 | 0.014 / 0.016 | 0.011 / 0.012 |
- Interpretation and Scope. These findings distinguish the general segmentation benefits from the additional gains associated with stronger discriminative changes. While the change scores provide a proxy rather than explicit annotations of drift events, the consistent high-change gains and their temporal concentration support the contribution of subspace evolution to improved robustness under changing foreground–background evidence. We therefore attribute the overall improvement to both effects, rather than to drift adaptation alone.
Response to Reviewer s38L (Part 3/3: Q3)
Question 3: Historical State and Computational Cost
Q3: What are the historical-memory capacity and retention policy, how are reconstruction-error statistics initialized and maintained, and when are these states reset?
-
Memory Capacity and Retention. All eight tasks use the same configuration, with subspace rank . Each sequence maintains a FIFO buffer containing at most three valid 64-dimensional historical directions. For each valid frame, the current foreground–background difference vector is constructed from the preliminary logits and detached query features, as described in Sec. 3. The same direction is used for the current subspace update and subsequently retained in the FIFO buffer for future frames, rather than being recomputed from the final refined prediction. Directions from annotated concept-definition samples are maintained separately and are unaffected by history replacement or clearing.
-
Reconstruction-Error Statistics. The error window stores the eight most recent valid reconstruction errors. After at least four preceding valid errors have accumulated, their mean and population standard deviation define the threshold: . The current error is compared with this threshold before being inserted into the window, so it does not influence its own decision threshold.
-
Reset Behavior and Invalid Updates. When the reconstruction error of the current valid direction exceeds the threshold, both the historical-direction buffer and the reconstruction-error window are cleared. The concept matrix is rebuilt from the annotated concept-definition directions and the current valid direction , and SVD is recomputed. Subsequent valid frames rebuild the history and accumulate new reconstruction-error statistics. If a frame lacks valid high-confidence foreground or background regions, it does not contribute to the historical-direction buffer or reconstruction-error statistics.
-
Sequence Boundaries. During inference, both the historical-direction buffer and the reconstruction-error window are initialized as empty at the start of each video or CT volume. No historical directions or error statistics are transferred across sequences.
Q3 (continued): Please also report inference speed and memory usage if available; these details would help assess the practicality of repeated subspace updates.
-
Measurement Protocol. Table 6 reports inference cost on an NVIDIA RTX 5090 using FP32, batch size one, and inputs. Memory measurements exclude initialization and warm-up.
-
Inference Speed and Memory. For Swin-B, static and evolving CDS take 26.63 and 35.84 ms/frame, respectively, giving an additional 9.21 ms/frame, while the evolving model achieves 27.90 FPS. Independent-process memory measurements give peak allocations of 904.577 MiB for static CDS and 904.583 MiB for evolving CDS. Across four backbones, evolution adds 3.96–10.87 ms/frame, and the full model achieves 27.15–29.10 FPS with 822–1826 MiB peak allocated GPU memory. The measured peak-memory increase is 6.5 KiB for each backbone.
Latency per frame
| Backbone | Static (ms) | Evolving (ms) | Added (ms) | FPS |
|---|---|---|---|---|
| Swin-B | 26.63 | 35.84 | +9.21 | 27.90 |
| ConvNeXt-B | 23.49 | 34.36 | +10.87 | 29.10 |
| Swin-L | 32.05 | 36.84 | +4.79 | 27.15 |
| ConvNeXt-L | 32.54 | 36.50 | +3.96 | 27.40 |
Peak allocated GPU memory
| Backbone | Static (MiB) | Evolving (MiB) | Added |
|---|---|---|---|
| Swin-B | 904.577 | 904.583 | ≈0 |
| ConvNeXt-B | 822.174 | 822.180 | ≈0 |
| Swin-L | 1825.940 | 1825.947 | ≈0 |
| ConvNeXt-L | 1736.914 | 1736.920 | ≈0 |
Response to Reviewer s38L (Part 1/3: W1, Q1)
Thank you very much for your insightful and constructive feedback on our submission. In response to the feedback from all three reviewers, we have revised the paper, and the changes are indicated in blue text within the manuscript. In the following, we provide a point-by-point response to your concerns. Our responses use the revised manuscript’s numbering unless otherwise noted. Quoted reviewer comments retain the original numbering.
Weakness 1 and Question 1: Comparison with Temporal Alternatives
Thank you for pointing this out. We added two temporal-update baselines in Table 5 using the foreground–background representation of CDS.
EMA Baseline. Maintains an exponential moving average of unnormalized foreground–background prototype difference vectors with β=0.5. Under identical initialization and valid update steps, averaging the difference vectors is equivalent to separately averaging the foreground and background prototypes and then taking their difference. Normalization is applied after averaging.
FIFO Baseline. Uses the same history buffer as EvoCDS, retaining the three most recent valid foreground–background discriminative directions while disabling reconstruction-error-based reset.
Matched Experimental Settings. Both baselines retain SVD-based subspace construction and disable reset. FIFO, EMA, and EvoCDS share the same Swin-B backbone, training schedule, concept-definition sampling protocol, subspace rank, confidence thresholds, and segmentation head architecture. Each method is retrained under its respective update rule and evaluated using the same protocol, allowing different temporal update strategies to be compared within a common discriminative representation.
Comparison Results. Table 5 shows that EvoCDS outperforms both FIFO and EMA across all eight tasks and both reported metrics. For example, shadow BER is 0.103 for EvoCDS, compared with 0.118 for FIFO and 0.116 for EMA, while polyp mDice is 0.848, compared with 0.838 and 0.825, respectively.
These results demonstrate that the complete EvoCDS evolution strategy outperforms the tested fixed-length history retention and moving-average update strategies.