acceptodds
Under review as a conference paper at ICLR 2027

Patchwise 3D Context Embedding: Equipping Efficient 2D Foundation Models With 3D Perception For Brain Hemorrhage Segmentation

Abstract

Brain hemorrhage segmentation is clinically important on non-contrast computed tomography (CT), but current methods face a trade-off: task-specific 3D models capture volumetric context at high computational cost, while 2D foundation models offer efficiency and strong pretrained representations but lack cross-slice context. Here, we ask whether a 2D foundation model can be efficiently equipped with the missing 3D prior for brain hemorrhage segmentation, without relying on computationally expensive dense 3D or video-style volumetric processing. We propose Patchwise 3D Context Embedding (P3CE), an efficient approach for introducing 3D prior into 2D foundation models. Instead of densely processing the full volume, P3CE represents local 3D neighborhoods as sparse point-cloud tokens, allowing information from adjacent axial slices to be captured efficiently. We first pre-trained a brain-specific Point-MAE on CT volumes to capture volumetric anatomical context. The resulting frozen encoder generates attributed 3D patch tokens once for each case, providing reusable 3D context for subsequent slice-wise segmentation. A slice-conditioned retrieval module then selects relevant tokens for each axial slice and injects them into the 2D model through lightweight cross-attention and residual fusion. Segmentation remains slice-wise, while 3D context is encoded only once. We evaluated P3CE on Seg-CQ500 and CT-ICH. P3CE achieved the highest overlap accuracy among the compared methods (0.5686/0.4773 IoU, 0.7078/0.6343 Dice), improving Dice by 9.25 points over the SAM-Med2D baseline on Seg-CQ500 while adding only a single frozen pointencoding pass per volume, with notable improvements on small and low-contrast hemorrhages and competitive inter-slice continuity. Ablations confirmed the contributions of both the 3D point-cloud representation and point attributes. These results show that 2D foundation models can benefit from volumetric information without dense 3D computation. By separating the computation unit from the context unit, P3CE provides an efficient interface for introducing sparse 3D information into slice-wise foundation models, suggesting a potential route for extending them to other medical imaging tasks where 3D context is important, pending broader validation. This study also provides a pre-trained, brain-specific Point-MAE for head CT.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Image Segmentation, Foundation Models, Point Cloud, 3D Context, Cross-Attention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dEMLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Patchwise 3D Context Embedding (P3CE), a method for introducing 3D priors into 2D foundation models for brain hemorrhage segmentation. P3CE converts each volume into an attributed point cloud and uses a brain-specific pretrained Point-MAE to encode 3D anatomical representations. Then the Point-MAE encoder is frozen and applied only once per CT volume. For each axial slice, a slice-conditioned retrieval module selects relevant tokens and injects them through lightweight cross-attention and residual fusion before mask decoding. Experimental results on Seg-CQ500 and CT-ICH demonstrate that P3CE achieves the best IoU and Dice among the compared methods, while adding a small computational overhead over SAM-Med2D.

Critical strengths

The method design is intuitive, forming a reasonable compromise between pure 2D segmentation and expensive dense 3D methods.

Critical weaknesses
  • The use of GT-derived point prompts during inference weakens the validity of the reported experimental results. Specifically, P3CE receives a positive point sampled from the ground-truth hemorrhage mask and a negative point sampled from the surrounding brain region during testing. This provides explicit lesion-localization information at inference time, making the reported setting closer to GT-guided segmentation than automatic hemorrhage segmentation. Furthermore, it is unclear whether all compared baselines use exactly the same GT prompts.
  • Critical point attributes are insufficiently defined, raising reproducibility and potential information-leakage concerns. Each 3D point contains one-hot brain-tissue, source-state, and health-state attributes. The paper does not clearly describe the origins and definitions of these attributes.
  • The information provided in the paper is insufficient to ensure reproducibility. Neither source code nor supplementary material detailing the implementation is provided. Although Section 4.1 outlines basic training settings, several critical details of the overall pipeline lack adequate description (e.g., the generation of the attributed point-cloud features, Point-MAE pretraining details). Furthermore, the Related Work section is relatively brief. The authors need to provide a more comprehensive discussion of closely related work to better position P3CE and more robustly substantiate its innovative contributions.
Questions for authors

See Weaknesses.

Reviewer_7sJAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Patchwise 3D Context Embedding (P3CE) for brain hemorrhage segmentation. It represents a head CT volume as an attributed sparse point cloud, uses a pretrained and frozen Point-MAE to extract 3D patch tokens, and injects slice-relevant tokens into SAM-Med2D through slice-conditioned retrieval, cross-attention, and residual fusion. Experiments on Seg-CQ500 and CT-ICH report improved overlap accuracy over the evaluated baselines, with ablations examining the point-cloud branch and point attributes.

Critical strengths
  1. The point encoder is executed once per volume, while the resulting sparse tokens are reused across slices. This provides a simple mechanism for augmenting a pretrained 2D model with 3D information without introducing a dense 3D backbone.

  2. P3CE improves Dice over MedSAM2 by 2.62 and 2.31 percentage points on Seg-CQ500 and CT-ICH, respectively.

Critical weaknesses
  1. The origin of key point attributes is unclear. Each point contains a brain-tissue label, source-state indicator, and health-state indicator, but the paper does not explain how these attributes are obtained at test time. In particular, it is unclear whether the health-state attribute contains information correlated with the target hemorrhage annotation. The scale and composition of the Point-MAE pretraining data, and its independence from the downstream test sets, are also unspecified. These omissions make it difficult to exclude unintended supervision or data leakage.

  2. The evaluation does not sufficiently isolate the contribution of the proposed 3D representation. Testing uses one positive and one negative point sampled from the ground-truth mask for every slice, but it is not stated whether all prompt-based baselines use exactly the same prompting protocol. Moreover, the ablations remove either the complete point branch or all additional point attributes; they do not include a coordinate-plus-HU variant or a matched 2.5D adjacent-slice baseline. Therefore, it remains unclear how much of the improvement comes from sparse 3D geometry, semantic attributes, or additional supervision.

Questions for authors
  1. How are the brain-tissue, source-state, and health-state attributes constructed? Are all of them available at inference time without using hemorrhage annotations or derived lesion labels? What data were used to pretrain Point-MAE, and were they strictly disjoint from the downstream validation and test subjects?

  2. Did every prompt-based baseline use the same ground-truth positive/negative point protocol during testing? How were prompts generated for slices without hemorrhage?

  3. Could the authors report matched comparisons using: (a) coordinates and HU only, and (b) a neighboring-slice 2.5D model with the same 2D backbone, training setup, and prompting protocol?

  4. Only 64 patch centers are assigned to slices by the rounded center coordinate, with at most eight tokens per slice and zero-padding when fewer are available. How many slices receive no valid token, and how are padded tokens masked in cross-attention?

Reviewer_SBduOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes P3CE, which equips a slice-wise 2D segmentation foundation model with volumetric context for brain hemorrhage segmentation on non-contrast CT, without dense 3D computation. Each CT volume is converted once into a sparse attributed point cloud and encoded by a frozen, brain-specific pretrained Point-MAE into a small set of 3D patch tokens; a slice-conditioned routing module assigns relevant tokens to each axial slice, and lightweight cross-attention with residual fusion injects them into the 2D features of a SAM-Med2D branch before mask decoding. This separates the computation unit (2D slice) from the context unit (3D patch): the volumetric branch runs once per case and its tokens are reused across all slices. Experiments on two public hemorrhage CT datasets report the best overlap accuracy among six compared supervised and foundation-model baselines with marginal computational overhead, and ablations examine the point-cloud branch and point attributes. The claimed contributions are the carrier-swap framework (dense voxels → sparse 3D patch tokens), the pretrained brain-specific Point-MAE, the P3CE retrieval-and-fusion design, and the empirical validation.

Critical strengths

1.Separating the computation unit (2D slice) from the context unit (sparse 3D patch tokens) — encoding the volume once per case and reusing tokens across all slices — is an appealing middle path between slice-wise 2D foundation models and dense 3D networks, decoupled from inference order and at marginal overhead. If validated properly, this framing would be useful beyond this paper. 2.Because FPS centers and kNN groups are formed in the full 3D coordinate space, each patch token inherently aggregates cross-slice information, and routing (Eq. 6–7) assigns a token's destination slice without confining its content to that slice ("the routing operation is slice-specific, whereas the routed representation remains volumetric"). Volumetric context is obtained with no adjacent-slice image encoder and no sequential propagation. 3.The brain-specific Point-MAE demonstrates that anatomy-specific 3D priors can be learned separately from the downstream 2D foundation model; with the frozen SAM-Med2D branch (LoRA/adapters) and a small trainable fusion module, the overall PEFT recipe is coherent and in principle reproducible — a transferable pattern and a potentially useful released artifact.

Critical weaknesses

1.The evaluation rests on two very small datasets with an undisclosed protocol and no uncertainty estimates. Seg-CQ500 contains only 51 volumes and CT-ICH 75 with 36 positive; after patient-level splitting, test partitions are tiny, and metric differences of the reported magnitude are well within split-to-split and seed-to-seed noise — yet the paper reports no standard deviations, confidence intervals, multiple seeds, or significance tests. This instability is not hypothetical: independent reports on these same datasets diverge substantially for identical or comparable models (e.g., SAM-family baselines evaluated on CT-ICH by Tao et al., Sci. Rep. 2025; the dataset publishers' own baselines in Spahr et al., Front. Neuroimaging 2023), and Spahr et al. further demonstrate that the same model's Dice can swing severalfold depending on evaluation aggregation (all volumes vs. hemorrhage-only volumes). This paper discloses neither its metric aggregation level, test-set sizes, baseline prompt configuration, nor whether CT-ICH is evaluated in-domain or cross-dataset, so its numbers cannot be audited or meaningfully situated against the literature. 2.The claimed general design principle is validated on a single target type — hemorrhage, a blob-like structure friendly to sparse sampling — using two small, single-source datasets; CT-ICH's 5 mm inter-slice spacing, which its publishers describe as unsuitable for 3D algorithms (Spahr et al. 2023), weakens it as evidence of 3D-context benefit. To be fair, the framework is re-instantiable per domain: anatomical attribute taxonomies, region-of-interest masks, and organ-specific Point-MAE pretraining all have plausible analogs beyond the brain. But each such re-instantiation is non-trivial (a label source for the taxonomy, an ROI extractor, a fresh point-cloud pretraining), and none is demonstrated — so whether the gain survives re-instantiation is unknown, which matters because the gain is driven mostly by these domain-bound attributes rather than by geometry. The HU attribute also has no direct counterpart outside CT, making MRI the harder and more informative test of the design principle. No cross-domain (abdominal CT, MRI) or larger-benchmark (e.g., BHSD, INSTANCE) evidence is provided, so both applicability and effectiveness beyond this narrow setting remain unverified. 3.The routing and representation mechanisms have unexamined structural gaps. (i) Routing coverage is neither guaranteed nor analyzed — with 64 patches over ~32 slices routed by patch-center z, some slices plausibly receive zero tokens and silently degrade to pure 2D while others are truncated at M_max, and no per-slice token distribution is reported. (ii) Sparse sampling combined with max-pooling over k=16 groups is structurally hostile to small targets — a small hemorrhage may contain no sampled point, or a single point drowned among normal neighbors — making the reported gains on small and low-contrast hemorrhages mechanically surprising. (iii) Within-slice spatial correspondence is weak — a token summarizes a multi-slice neighborhood but is offered to all image tokens of the slice, leaving in-plane correspondence to be inferred by attention from appearance alone.

Questions for authors

1.Define health-state and source-state precisely: how are they obtained, and do they encode any information derived from hemorrhage annotations? Please provide a per-attribute ablation (coordinates / HU / tissue / source / health), including a "health-state only" condition. 2.What data was the brain-specific Point-MAE pretrained on (size, composition, mask ratio), and can you confirm no overlap with the Seg-CQ500 / CT-ICH test cases? 3.Does the reported 1.9 GFLOPs point-branch cost include FPS and kNN grouping? Please report end-to-end latency, peak memory, and trainable/total parameter counts. 4.Why are N=8192, G=64, k=16, M_max=8 chosen? How sensitive is performance to these, especially for small hemorrhages, given that uniform sampling of ~8k points covers <0.2% of brain voxels? 5.What is the empirical distribution of retrieved tokens per slice (min / mean / max across the test sets)? How many slices receive zero patches, and how does the model behave on those slices? Is per-slice coverage correlated with lesion location or size?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.