acceptodds
Under review as a conference paper at ICLR 2027

LLM-RegProto: LLM-Enabled Cross-Modal Prototype Regularizer for Weakly Supervised Event Semantic Segmentation

Abstract

Event cameras offer microsecond temporal resolution and high dynamic range, making them attractive for autonomous perception. However, the sparsity and texture-free nature of event data render pixel-wise annotation prohibitively expensive, bottlenecking fully supervised semantic segmentation. We present LLM-RegProto, a weakly supervised framework requiring only extremely sparse point annotations (as few as 1 class 1 point, or a standard 1 class 10 points) to achieve competitive segmentation. The core innovation repurposes a frozen, pretrained text-only LLM layer as a cross-modal prototype regularizer: visual tokens from the event-stream encoder are projected into a high-dimensional linguistic manifold, where the frozen transformer layer induces semantically structured class prototypes via its pretrained attention mechanism. Unlike multimodal LLM approaches requiring paired vision-language training or text prompts, our framework feeds visual features directly into a pure-text LLM layer, with class prototypes formed autonomously through learnable class query tokens without textual supervision. To bridge the visual–linguistic modality gap, we design (i) a cross-modal adapter with 2D spatial positional encoding and bidirectional attention, enabling the autoregressive LLM backbone to process non-sequential spatial tokens; and (ii) a dual-stream prototype alignment network with background-noise suppression, jointly training two encoders on temporally complementary event windows and filtering ego-motion-induced clutter via contrastive prototype alignment. On DDD17 and DSEC benchmarks under the 1C10C budget, LLM-RegProto achieves mIoU gains of +2.16% and +2.04% over prior state-of-the-art weakly supervised methods, demonstrating that LLM pretraining weights encode universal structured priors transferable across modalities without any fine-tuning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
weakly supervised semantic segmentation, autonomous driving perception

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EvuwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLM-RegProto addresses weakly supervised event-camera semantic segmentation using sparse point annotations. Its central component projects event features and learnable class queries through a frozen text-only LLM layer to construct class prototypes without text prompts. A spatial adapter enables bidirectional token interactions, while a dual-stream network aligns foreground prototypes across complementary temporal windows and suppresses background noise. Experiments on DDD17 and DSEC evaluate one-point and ten-point annotation budgets per class. Under the ten-point setting, the method achieves 62.55% and 53.23% mIoU, improving over EV-WSSS by 2.16 and 2.04 points. Component ablations, pretraining controls, and prototype visualizations support the reported improvements.

Critical strengths
  1. The framework presents an innovative combination of frozen LLM layers, learnable class queries, and temporal prototype alignment, requiring neither textual prompts nor paired multimodal training.

  2. The method consistently outperforms EV-WSSS across both datasets and annotation budgets, delivering gains of 2.16 and 2.04 mIoU points under ten-point supervision on DDD17 and DSEC.

Critical weaknesses
  1. Comparisons omit modern WSSS approaches leveraging CLIP, DINO, or SAM spatial priors. Adapting such methods to events under matched supervision would better establish the proposed language-based regularizer’s distinctive contribution.

  2. Projection into an LLM’s hidden space does not guarantee semantic alignment. Although Figure 3 and Table 3 show benefits, they do not validate the claimed universal, modality-independent semantic clustering geometry.

  3. The LLM layer remains active during inference, yet latency, memory consumption, and computational overhead are unreported. Its practical value for low-latency event-based perception therefore remains unclear despite the accuracy improvements.

Questions for authors

Please see weaknesses.

Reviewer_wjajOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents LLM-RegProto for weakly supervised event-based semantic segmentation using sparse point annotations. Its Frozen LLM Prototype Extractor (FLPE) jointly processes event-derived visual tokens and learnable class query tokens through a single frozen, pretrained LLM layer to generate class-specific prototypes for segmentation. The framework further combines Foreground-Only Prototype Alignment (FOPA) with bidirectional pseudo-label supervision to align prototypes and encourage prediction consistency across complementary temporal windows. Under the 1C10C annotation setting, LLM-RegProto achieves mIoU of 62.55% and 53.23% on DDD17 and DSEC, outperforming EV-WSSS by 2.16 and 2.04 percentage points.

Critical strengths
  • Practical significance. The method constructs class-specific prototypes without updating LLM weights or using class-name prompts. The reported performance gains on both datasets under the 1C1C and 1C10C settings highlight its potential to reduce reliance on dense annotations for event-based semantic segmentation.
Critical weaknesses
  • Novelty. EV-WSSS already employs asymmetric dual-stream learning and cross-branch prototype mechanisms, while using frozen LLM layers to process visual features has also been explored in prior works. The main increment of this paper lies in combining explicit class-query prototypes with FOPA. Despite the controls in Tables 3 and 9, the results do not fully isolate the contributions of prototype formation, auxiliary supervision, and inference-time fusion, or sufficiently support the stronger interpretation of a “universal semantic clustering geometry.”

  • Writing and presentation. Several inconsistencies affect readability. The main text defines streams A/B as using short/long windows, but the appendix reverses these assignments (p. 4, lines 195–198; p. 13, lines 698–701). The stated gains do not match the values in Table 7 (p. 14, lines 719–720). Class indices are inconsistent between (1,…,K)(1,\ldots,K) and (0,…,K−1)(0,\ldots,K-1) (p. 3, line 161; p. 6, lines 279–280), and the similarity-map resolution after pooling is unclear (p. 4, line 211; p. 5, lines 242–263). Some mechanistic claims also appear overstated.

Questions for authors

I am not very familiar with this field, and the following questions may help me better understand the paper:

  1. Loss functions. The paper specifies the weights of the three auxiliary losses but does not sufficiently explain their selection. Did the loss magnitudes and gradient scales inform these choices? Why is only the primary loss used during the first two epochs, and were other warm-up durations evaluated?

  2. Language model selection. The comparison between LLaMA and DeepSeek is largely limited to layer selection on DDD17 under 1C10C. Could the authors provide results for both models on DSEC and under 1C1C? What motivated the choice of these two models.

  3. Mechanism of background exclusion. The benefits of background exclusion in FOPA are attributed to suppressing ego-motion noise. However, “background” is a semantic class in these datasets and does not directly correspond to noise. Could the authors clarify this relationship and report per-class IoU, including the background class?

Reviewer_hY1uOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLM-RegProto addresses event semantic segmentation with sparse point annotations. Pooled event features receive 2D positional encoding and a linear projection, then enter a frozen LLM layer alongside learnable class queries to form class prototypes through bidirectional attention. Training combines two temporal streams, foreground-only prototype alignment, and pseudo-label cross-supervision. Inference fuses the forward stream with prototype similarities. Evaluation covers DDD17 and DSEC under two annotation budgets. Under 1C10C, the reported mIoU scores are 62.55 and 53.23, improving over EV-WSSS by 2.16 and 2.04 percentage points, respectively.

Critical strengths

1 A useful sparse-supervision setting with consistent reported gains. Dense annotation is costly for event semantic segmentation, making point supervision practically relevant. Tables 1 and 2 report improvements over EV-WSSS on both datasets under both annotation budgets, including gains of 2.16 and 2.04 mIoU percentage points under 1C10C. Using a single frozen LLM layer and retaining only the forward event stream at inference is also a worthwhile architectural direction. These results support the practical value of the method.

2 Targeted controls for pretrained weights and prototype formation. Table 3 includes architecture-matched random frozen, random trainable, and shuffled-pretrained-weight controls, all below the full model. Table 8 reports a FOPA variant without FLPE, and Table 9 compares feature transformation with explicit prototype formation. These experiments provide stronger evidence than a comparison against the base network alone and support contributions from pretrained weights and class-query design.

Critical weaknesses

1 Effective pretraining does not establish the proposed linguistic-manifold mechanism. Equations 5 and 6 on p. 5 assume ideal within-class and between-class distances and extend this interpretation to projected visual features without directly verifying it. The adapter and class queries receive point supervision, which could itself produce class separation. The pretrained frozen layer outperforms random frozen and shuffled-weight controls in Table 3 (62.55 versus 60.41 and 60.53), supporting learned organization but not separating linguistic semantics from general pretrained interaction priors. A t-SNE plot cannot establish correspondence to a linguistic manifold in the original high-dimensional space. This central interpretation requires direct evidence or narrower wording.

2 Foreground exclusion and prediction operations are not uniquely specified. Section 3.1 defines K classes, but Section 3.3 lists foreground classes 0 through K−1 and adds background K. The excluded class IDs and their inclusion in mIoU must be specified for each dataset. Table 7 treats background handling as one switch, without separating prototype-loss exclusion from pseudo-label exclusion or establishing ego-motion noise suppression as the cause of improvement. In addition, p. 5 pools spatial tokens to 16×16, whereas p. 4 describes an H/8×W/8 similarity map without the restoration step; similarity scaling at training versus inference is also unclear. These definitions affect losses and predictions and require consistent equations and implementation details. They do not, by themselves, establish a code error.

Questions for authors

1 Additional validation Beyond random and shuffled controls, how does an equal-budget non-language pretrained block perform? What quantitative evidence in the original high-dimensional space supports a linguistic-manifold or universal-clustering interpretation beyond separability learned from point labels?

2 Clarification and additional validation Please specify class IDs, excluded background classes, and mIoU rules for DDD17 and DSEC, including whether prototype alignment masks classes absent from a sample. What are the results of background exclusion only in prototype alignment, only in pseudo-labeling, and in both?

3 Clarification Please give the full shape sequence from 16×16 spatial tokens to the similarity map and specify inference temperature or fusion weights in Eq. 4. How are RoPE, normalization, feed-forward, and residual operations handled, and are gradients preserved to the adapter and class queries?

4 Clarification Do EV-WSSS and all controls use identical point labels, temporal windows, training budgets, and validation sets? What statistic does ± denote in Tables 1 and 2, over how many runs? Please provide actual configurations or anonymous code and reconcile window lengths between the main text and appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.