- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents DuSAM, a hierarchical visual backbone that combines image-local working memory with a persistent bank of class-specific prototypes collected from annotated training data. At deeper stages, tokens retrieve semantic prototypes, bind them to transported working states, and read the states back to refine image features. The evaluation covers ImageNet classification, ADE20K semantic segmentation, and COCO detection and instance segmentation, along with memory ablations, high-resolution profiling, and targeted memory removal and restoration.
Critical strengths
-
Binding retrieved semantic prototypes to transported, spatially anchored working states gives the two memory stores a clear role within the backbone. The retrieved information modifies intermediate features rather than only a task-specific prediction head.
-
The component ablations are informative. In particular, keeping the semantic-address supervision while disabling persistent memory helps distinguish retrieval gains from an auxiliary classification loss.
-
The memory interventions make the stored content inspectable. The class-targeted removal and restoration examples show that predictions can change through memory updates without changing the network weights.
Critical weaknesses
-
I am not convinced that the experiments establish DuSAM as a competitive general-purpose backbone. Tables 1 to 3 compare many older architectures, but omit strong published efficient backbones. In particular, MSVMamba-Micro [1] reports 79.8% ImageNet accuracy with 12M parameters, 45.1 single-scale mIoU with a 42M-parameter UPerNet, and 43.8 box AP / 39.9 mask AP with a 32M-parameter Mask R-CNN. DuSAM reports 77.8% with 20.5M parameters, 42.90 mIoU with 49.9M, and 41.9 / 38.1 AP with 39.5M, respectively. These are published results with potentially different training recipes, not controlled head-to-head measurements. Still, excluding a smaller, already published backbone with stronger results on all three tasks leaves the proposed architecture's advantage unclear. A matched comparison or a careful account of the protocol differences is needed. The omission also extends to more recent backbone families such as MambaVision.
-
The transfer evidence is narrower than the general-purpose framing suggests. Appendix A.4 states that persistent memory is rebuilt for each downstream task's label vocabulary. Reusing pretrained weights across several supervised recognition tasks is demonstrated, but transfer of the stored semantic experience is not. I would like to see a frozen-feature or memory-free transfer analysis, or a more limited claim.
-
The benefits of the specific dual-store binding over existing visual memory mechanisms are not isolated. Associative Transformer [2] already uses explicit associative memory for visual processing, while MCIBI [3] retrieves class-level dataset memory using predicted semantic addresses. Table 4 establishes gains over local-only variants, but gives no matched comparison to either memory-based alternative.
-
The efficiency comparison in Fig. 2 is against global-attention DeiT-S alone, leaving the advantage over modern efficient backbones unresolved. The targeted restoration results in Figs. 6, 10, and 11 are qualitative. Average recovery across classes is not reported.
[1] Shi et al., Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model, NeurIPS 2024.
[2] Sun et al., Associative Transformer, CVPR 2025.
[3] Jin et al., Mining Contextual Information Beyond Image for Semantic Segmentation, ICCV 2021.
Questions for authors
-
When transferring from ImageNet to ADE20K or COCO, which PM writer, reader, and semantic-address parameters are retained, and which are reinitialized along with the class-specific prototype bank (Appendix A.4)?
-
Can the authors clarify the pretrained checkpoints and augmentation protocols behind the Table 2 and 3 baselines? Are results available under a matched setting for a more recent efficient backbone such as MSVMamba-Micro?
6 comments
W1/Q2 (Recent efficient backbones, matched setting, and baseline protocol).
Matched comparison. As requested, we added MSVMamba-Micro and MambaVision-T on all three tasks. For ADE20K and COCO, we trained both ourselves under the same protocol as every other baseline, so Tables 2–3 now provide the matched setting you asked about rather than published numbers. (Tables 1–3, pp. 5–7 of the revised manuscript; highlighted in red.)
Baseline protocol (Q2). Table 1 lists published ImageNet-1K results with each model’s augmentation recipe. For Tables 2–3, baselines are initialized from publicly released ImageNet-1K checkpoints and trained by us under the same segmentation and detection settings as DuSAM, including the newly added MambaVision-T and MSVMamba-Micro. (Appendix A.1.2, p. 15 of the revised manuscript; highlighted in red.)
Stronger DuSAM results. With the same augmentation recipe as these baselines, DuSAM improves to 79.1% top-1 (from 77.8%), 43.33/44.50 SS/MS mIoU on ADE20K, and 42.6/38.9 box/mask AP on COCO (DuSAM⋆).
Where DuSAM stands. DuSAM is, to our knowledge, the first general-purpose visual backbone to build its feature hierarchy on explicit memory, with no conventional self-attention or state-space blocks. In this first instantiation, under the matched protocol, it already matches Swin-T in COCO box AP (42.6) with 17% fewer parameters, exceeds DeiT-S, PVT-S, and PoolFormer-S24 by 2.2–2.5 box AP, exceeds PVT-S and PoolFormer-S24 on ADE20K by 0.9 and 1.6 mIoU, and scales to high resolution as efficiently as MSVMamba-Micro (both retain 12.3% of their throughput from 512² to 2048²). On small objects, its box and mask AP are the second highest among all compared models, above ConvNeXt-T, Swin-T, and MSVMamba-Micro (Table 10). MSVMamba and MambaVision build on several generations of refinement of visual state-space models and currently lead in accuracy; DuSAM reaches the range of established backbones in its very first form, with clear room for the same kind of refinement. Beyond accuracy, it offers what these backbones do not: semantic evidence stored explicitly, which can be inspected, diagnosed, and replenished from five annotated regions without changing weights (Sec. 4.4, p. 10).
W2/Q1 (Transfer: retained parameters and scope of the general-purpose claim).
Retained and reinitialized parameters (Q1). When transferring from ImageNet-1K to ADE20K or COCO:
-
Retained: all task-independent backbone parameters, including WM construction, cross-stage transport, and readback; the PM reader (content query) and PM-to-WM binding projections; the task-independent layers of the semantic-address heads; and the PM writer’s descriptor construction and EMA update rule.
-
Reinitialized: the class-specific prototype bank, which is rebuilt from downstream annotated regions (ADE20K) or instances (COCO); the semantic-address output layers for the new label vocabulary (150 or 80 classes); and the task-specific segmentation and detection heads.
(Sec. 3.2, pp. 4–5, and Appendix A.1.2, pp. 14-15 of the revised manuscript; highlighted in red.)
Scope of the claim. DuSAM is a new memory-native backbone, not a memory module added to an existing CNN, Transformer, or SSM. Our general-purpose claim concerns this architecture and its pretrained representations, which transfer across classification, segmentation, and detection, as with other general-purpose backbones. The learned memory operations are reused across tasks, while persistent semantic content is accumulated for each task’s label vocabulary, as stated in Appendix A.4.
Frozen-feature and memory-free transfer. We added the analysis you asked for on ADE20K. Frozen-feature: with all backbone weights frozen and PM disabled, DuSAM reaches 36.70/37.92 SS/MS mIoU, so its pretrained representations support dense prediction on their own. Memory-free: with fine-tuning and no PM, it reaches 41.75/42.97. Task-adapted PM then adds a further 1.15 points on both metrics (42.90/44.12). The full analysis, including a preliminary test of directly reusing the ImageNet prototype bank, is given in Table 12. Together, these results show that DuSAM learns transferable visual representations as a memory-native backbone, with persistent memory supplying additional task-relevant semantic evidence. (Appendix A.3.6 and Table 12, p. 22 of the revised manuscript; highlighted in red.)
Response to Reviewer mEh3
Dear Reviewer mEh3, thank you sincerely for your valuable review, and for pointing us to specific recent work. We are truly glad that the paper deepened your understanding of the problem and that you saw a clear role for each memory store within the backbone. Your comments helped us sharpen how this new architecture is positioned and presented. We have addressed every point below and updated the revised manuscript accordingly, and we hope this answers your questions.