acceptodds
Under review as a conference paper at ICLR 2027

MambaFourier: A Physics-Inspired Spectral Fusion Model for Low-Light Event–RGB Semantic Segmentation

Abstract

Low-light perception remains a fundamental bottleneck, as insufficient brightness causes texture to collapse and appearance-based segmentation to fail catastrophically. Event cameras remain informative in these regimes, yet existing RGB-Event fusion modules are generic spatial or token mixers that ignore the modalities' spectral asymmetry: events are sparse high-frequency temporal derivatives of log intensity, whereas frames carry reliable low-frequency structure but noise-dominated high frequencies under low light. We introduce MambaFourier, a dual-branch state-space segmentation network whose fusion operator is physics-inspired. At each scale, a Wave-Fourier Fusion (WFF) block treats the two feature maps as coupled damped wave fields. Since the Laplacian is diagonal in the Fourier basis, the damped wave equation has a closed-form per-wavenumber solution: WFF performs one real FFT, applies a learnable per-channel wave propagator, a band-limited free-form filter, and a reciprocal 2x2 cross-modal coupling, then one inverse FFT. Spectral parameters are indexed by mode, enabling resolution transfer, and WFF is an exact identity at initialisation. MambaFourier reaches 79.39% mIoU on DDD17 (+1.83% over MambaSeg), 75.65% on DSEC-Semantic (+4.61% over ESC), 70.40% on DERS-XS (+3.30% over ESC), and 56.51% on DSEC-Xtrm (+5.64% over the previous best). Under synthetic rain and fog on DSEC, it attains 73.7% mIoU versus 72.5% for MambaSeg.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Event camera, RGB-event fusion, semantic segmentation, state space models, Mamba, Fourier neural operator, spectral fusion, wave equation, low-light vision, robust perception

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_naxEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses a relevant problem and reports encouraging results. However, the technical contribution seems somewhat incremental, and the evidence for the physics-inspired design is not fully convincing. In particular, it is not clearly shown why wave-based fusion is better than simpler spectral alternatives. The experimental comparisons and robustness analysis also leave important questions unanswered.

Still, I recognize the potential value of this work for event-based vision. I am open to raising my score if the authors can adequately address the above concerns, especially with stronger evidence for the physical motivation, more controlled comparisons, and additional robustness experiments.

Critical strengths

Low-light Event–RGB semantic segmentation is a meaningful problem, and combining the two modalities is reasonable because their information is complementary.

The WFTF module has a clear mathematical formulation. Using the analytical solution of a damped wave equation in the Fourier domain is an interesting design choice.

The method unifies frequency filtering and cross-modal coupling, which offers an alternative to common attention-based and spatial fusion methods.

Experiments cover four benchmarks and show consistent improvements, with large gains under extreme degradation. The paper also provides ablations and computational cost analysis, and the model reports fewer parameters and FLOPs than MambaSeg.

Critical weaknesses

Technical novelty seems limited. The framework follows a common dual-branch architecture with pretrained encoders and multi-scale fusion. The main contribution, WFTF, combines existing ideas from Fourier filtering, wave propagation, and cross-modal interaction. It is not clear whether it gives a fundamental advantage over simpler frequency-domain fusion methods.

The physical motivation is not sufficiently supported, I am afraid. The method assumes event features mainly provide high-frequency information and RGB features provide reliable low-frequency information. However, event cameras measure temporal brightness changes, which do not necessarily correspond to high spatial frequencies. The paper gives no detailed frequency analysis of real RGB and event features, and the learned wave parameters and coupling weights are not visualized. Thus, it is unclear whether the module learns the expected physical behavior.

The ablation study is insufficient to justify the design. Table 3 shows that removing WFTF components reduces performance, but their individual contributions are not fully isolated. For example, removing the propagator still leaves the Fourier filter and cross-modal coupling active. The paper also lacks controlled comparisons with standard learnable Fourier operators under the same parameter budget. The 0.64 mIoU gain over the DCT-based wave gating baseline is relatively small, and no statistical analysis is given.

Experimental comparisons are not sufficiently controlled. MambaFourier and MambaSeg use different backbones and fusion architectures, so the improvements cannot be fully attributed to WFTF. Table 1 mainly uses results reported by ESC, while the proposed model is trained separately. The paper should clarify whether training settings, pretrained weights, and computational budgets are comparable. Multiple random seeds are also needed.

Robustness and generalization claims need stronger evidence. Most challenging conditions are synthetic. In the additional rain, fog, and dark-night experiments, the event stream remains unchanged, which does not fully reflect real event sensor behavior under low illumination and noise. The self-collected dataset reports only the proposed method without baseline comparisons, and cross-resolution transfer is not directly evaluated.

Questions for authors

Why is the physics-based wave propagator necessary? Could the authors compare WFTF with simpler Fourier filtering and spectral fusion modules under exactly the same backbone, training settings, and parameter budget? Visualizing the learned spectral responses and coupling weights would also help show whether they follow the proposed physical interpretation.

How well does the method generalize to real low-light conditions, especially when the event stream contains realistic noise, missing events, or reduced activity? Could the authors also provide baseline comparisons on the self-collected dataset?

1 comment
AuthorsOpenReview ↗

We thank the reviewer for recognizing the importance of low-light event–RGB segmentation and the potential of WFTF. We have expanded the evidence with multiple spectral alternatives, supplementary coupling experiments, and repeated WFTF training runs.

1. Technical contribution and physical motivation

The contribution is the WFTF fusion operator. The dual-branch framework and GroupMamba encoders provide its host architecture. WFTF combines:

  1. Analytical propagation of independently projected displacement and velocity features.
  2. A learnable spectral correction.
  3. Reciprocal frequency-wise cross-modal exchange.

The wave parameterization supplies a compact family of frequency-dependent attenuation and oscillatory responses. Its role is to provide a structured inductive bias for representation learning. The exact solution applies to the propagator; the subsequent filtering and coupling provide learned flexibility. This defines the mathematical contribution without requiring encoder features to obey a literal physical wave law.

2. Additional comparisons with simpler spectral alternatives

We implemented and trained DCT, asymmetric DCT, Fourier, and Gaussian controls on DERS-XS, sharing the dual GroupMamba-Small encoders, pretrained weights, decoder, sample splits, and augmentation.

FusionEpochsBest validation mIoUFusion parameters
DCT10068.717.846M
Fourier10068.747.846M
Gaussian10068.727.849M
Asymmetric DCT20070.248.924M
WFTF20070.277.849M

The matched 200-epoch comparison shows comparable accuracy with approximately 1.08M fewer fusion parameters for WFTF. This supports the parameter efficiency of its structured design. The other controls are reference results under their stated 100-epoch schedules.

Across WFTF seeds 0/1/2, best validation scores within the common first 177 epochs of the planned 200-epoch schedule are 70.27/69.47/69.72, yielding 69.82±0.4169.82\pm0.41 mIoU (sample SD). Seed 2 currently reaches epoch 177. Separate Test evaluations for seeds 0/1 yield 68.25/68.62. This quantifies WFTF variability and is distinct from a paired significance test against SWP.

3. Component contributions and controlled attribution

The shared-architecture DDD17 study gives:

VariantmIoUChange from full WFTF
Full WFTF79.39—
Without propagation77.93−1.46
Without coupling77.78−1.61
Without free-form filtering77.68−1.71

These interventions address whether each component improves the complete architecture while the others are retained. All three contribute positively under this protocol. Their interpretation is conditional component utility, rather than independent or additive effects.

Additional DDD17 logs record 78.8266 for symmetric DCT, 78.8496 for unconstrained DCT, and 79.0150 for kappa-zero. These are supplementary single-seed best evaluation results using ten event bins and LR 3×10−43\times10^{-4}, reported separately from the manuscript recipe.

We distinguish the comparison levels: Table 1 and the MambaSeg comparison evaluate complete systems, whereas shared-backbone studies assess fusion choices within a common architecture. This avoids attributing backbone or training differences exclusively to WFTF.

4. Spectral interpretation

We clarify that temporal contrast and spatial frequency are different quantities. The motivation is feature-dependent spectral complementarity, rather than an assumption that events contain only high spatial frequencies.

DSEC coupling support expands from 5.7% at stage 1 to 100% at stage 4, enabling selective exchange across multiple representation scales. These percentages describe architectural support. Learned spectral maps and band interventions would quantify a trained model’s actual frequency preferences and their causal contributions; support coverage alone is not such a measurement.

5. Robustness, real data, and resolution

The evaluation covers complementary conditions: RGB-only rain/fog corruption, joint frame/event degradation in DSEC-Xtrm, and low-light frames with noisy synthetic events in DERS-XS.

Under corruption-aware rain/fog training, MambaFourier achieves 73.7 mIoU, compared with 72.5 for MambaSeg and 71.5 for EISNet. DSEC-Xtrm and DERS-XS extend the evaluation to settings with degraded event information.

The campus dataset supplies additional real-data evidence: 68.1 mIoU and 95.95% pixel accuracy after training on its designated split. We present this as additional real-data evaluation, rather than a comparative campus ranking or zero-shot transfer result.

Mode-indexed parameters also permit operation at different feature-map sizes. This architectural property is distinguished from measured cross-resolution transfer accuracy. Our robustness conclusions correspond to the evaluated protocols, including synthetic event noise, rather than extrapolation to all real-sensor noise, missing-event, or reduced-activity conditions.

Reviewer_psGtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MambaFourier for semantic segmentation from paired frames and event voxel grids, with an emphasis on low-light conditions. Two GroupMamba encoders are connected by stage-wise WFTF blocks. Each block predicts displacement and velocity features, applies an analytical damped-wave propagator in Fourier space, adds band-limited learnable complex filtering and reciprocal cross-modal mixing, and returns the results through residual connections. A SegFormer-style decoder is trained with pixel-wise cross-entropy. The paper reports 79.39%, 75.65%, 70.40%, and 56.51% mIoU on DDD17, DSEC-Semantic, DERS-XS, and DSEC-Xtrm, respectively. Validation also includes fusion/component ablations, computational-cost measurements, synthetic corruption experiments, and a self-collected campus dataset.

Critical strengths
  1. The paper offers an interesting way to combine spectral processing with event–frame fusion.
  2. The experimental results are solid and the ablations give useful evidence for the fusion block.
Critical weaknesses
  1. The explanation of why the method works needs more support. The wave equations describe each modality separately. The two streams are then mixed by a learned matrix in Eq. (8). The paper does not derive this matrix from a coupled wave system. I therefore see this as a physics-inspired fusion block, rather than an exact solution for the full interaction between two wave fields. I also find the high-frequency argument incomplete. Spectral coupling is zero outside the retained low-frequency blocks, and Table 5 shows only 5.7% coverage at DSEC stage 1. Later stages may still help with fine details, so this does not mean the design cannot work. But the paper needs to show where the claimed high-frequency transfer actually happens. The current ablations show that the parts are useful, but not why they help under low light.
  2. Section 4.2 says the model is trained on both DSEC-Semantic and DSEC-Xtrm. It is unclear whether Table 1 uses the same checkpoint for both. If separate models are trained, the score gap is not simply the effect of degrading the inputs to one model. The rain/fog and dark-night tests also use corrupted training data and leave events unchanged. They do not test a clean-trained model on unseen corruptions. 3.The controlled comparisons in Table 3 are useful, but they are only on DDD17. Table 1 takes baseline results from Bao et al. and compares full systems, rather than fusion blocks on the same backbone. Using Mamba in both models does not isolate fusion, especially since Section 4.4 notes that the backbones differ. I would like to see the same ablations on DSEC-Xtrm or DERS-XS. Results across several runs would also help judge smaller gains, such as the 0.64-point improvement over SWP.
Questions for authors
  1. Do the DSEC-Semantic and DSEC-Xtrm results use the same checkpoint? If not, can you separate performance after retraining from the effect of corruption on a fixed model?
  2. Which stages and frequency bands carry the claimed high-frequency transfer? Could you show the learned coupling weights and test what changes when selected bands are disabled? Please also clarify which part of the full block is covered by the exact wave solution.
  3. Can you repeat the key fusion ablations on DSEC-Xtrm or DERS-XS using the same backbone and training setup? Please report results across several seeds, including the no-propagator and no-coupling variants. Also state the effective batch size, gradient accumulation, and which settings match the Table 1 baselines.
2 comments
AuthorsOpenReview ↗

We thank the reviewer for the constructive questions. We address them through explicit protocol clarification, a precise account of the analytical operator, and additional spectral-control and repeated-seed experiments on DERS-XS.

Q1. DSEC-Semantic and DSEC-Xtrm evaluation protocols

Table 1 reports benchmark-specific training and evaluation. The 75.65 mIoU on DSEC-Semantic and 56.51 on DSEC-Xtrm therefore describe performance under their respective training protocols, rather than evaluation of one unchanged clean-trained checkpoint.

The relevant within-benchmark comparisons are +4.61 points over ESC on DSEC-Semantic and +5.64 points on DSEC-Xtrm. These establish competitive performance under standard and severely degraded benchmark conditions. We will replace the interpretation of the cross-benchmark 19.14-point difference with this protocol-specific interpretation.

Likewise, rain/fog and simulated dark-night results use corruption-aware training. DSEC-Xtrm additionally includes degradation of both frames and events. These settings are distinguished from fixed-checkpoint testing on unseen corruptions.

Q2. Exactness and the location of spectral exchange

The exact solution applies to the within-modality propagator. Equation (6),

P^=AU^+B(V^+γU^),\widehat P=A\widehat U+B(\widehat V+\gamma\widehat U),

corresponds to the analytical solution with effective initial velocity

\qquad \alpha_k=\gamma\|k\|^2.$$ The subsequent Fourier filter and reciprocal mixing matrix are learned operators. Thus, **the propagator is analytical, while the complete fusion block is physics-inspired**. The spectral support is explicit: DSEC stages 1–4 have coupling coverage of **5.7%, 22.7%, 76.2%, and 100%**, respectively. Propagation acts over the whole spectrum. Outside the retained coupling bands, coupling is zero and filtering is identity. This provides a multiscale pathway: early stages exchange selected feature modes, later stages exchange a larger fraction of their spectra, and fused features feed subsequent encoding and decoding. Coarse-stage frequencies should not be equated directly with high-frequency input-image detail. The support analysis establishes where exchange can occur; learned coupling maps and band-disable measurements would establish its empirical importance at each stage. ### Q3. Additional controls, seeds, and training settings We extended the study to DERS-XS with common dual GroupMamba-Small encoders, pretrained weights, decoder, sample splits, and augmentation. | Fusion | Epochs | Best validation mIoU | |:--|--:|--:| | DCT | 100 | 68.71 | | Fourier | 100 | 68.74 | | Gaussian | 100 | 68.72 | | Asymmetric DCT | 200 | 70.24 | | WFTF | 200 | **70.27** | Under the matched 200-epoch schedule, WFTF achieves comparable accuracy with **7.85M versus 8.92M fusion parameters**. The shorter-schedule controls are reported under their stated budgets. For WFTF seeds 0/1/2, best validation scores within the common first 177 epochs are **70.27/69.47/69.72**, or **$69.82\pm0.41$** (sample SD). All use a planned 200-epoch schedule; seed 2 currently ends at epoch 177. Validation-selected checkpoints for seeds 0/1 achieve **68.25/68.62 on Test**. These new-run results are reported separately from the original Table 1 result. The DERS-XS configuration uses: - Batch size **6**, accumulation **2**, and **effective batch size 12**. - AdamW, initial LR **$6\times10^{-5}$**, and weight decay **0.01**. - **500** warmup updates and polynomial decay with power **0.9**. - Five event bins and **384×640** inputs. - Joint scaling [1.02,1.20], cropping, and horizontal flipping. - Validation-based checkpoint selection and separate Test evaluation. Additional DDD17 logs record best evaluation scores of **79.0150** for kappa-zero, **78.8266** for symmetric DCT, and **78.8496** for unconstrained DCT. These use seed 1, 100 epochs, batch 12 without accumulation, LR $3\times10^{-4}$, and ten event bins. They are supplementary recipe-specific results; their logged Test stream is used for checkpoint selection. The completed DERS-XS additions examine alternative spectral operators and WFTF repeatability. The original Table 3 supplies no-propagator/no-coupling evidence on DDD17; it is distinct from the requested multi-seed DERS-XS component study. Finally, Table 1 imports competing scores from Bao et al. and evaluates complete systems. We distinguish those comparisons from shared-architecture fusion studies rather than assuming identical backbones, pretraining, effective batches, or training budgets.
Reviewer_psGtOpenReview ↗

Concerns addressed

I thank the authors for these additional experiments and explanations. The reply addressed my concerns on the evaluation setting, and the effectiveness of the WFTF blocks. I have no more questions.

Reviewer_RCBCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MambaFourier, a frequency-domain fusion network for low-light Event–RGB semantic segmentation. Its central component, Wave–Fourier Transform Fusion (WFTF), combines an analytical damped-wave propagator, a learnable Fourier filter, and frequency-wise reciprocal cross-modal coupling to fuse RGB and Event features across four encoder stages. The method uses dual GroupMamba backbones and reports 79.39% mIoU on DDD17, 75.65% on DSEC-Semantic, 70.40% on DERS-XS, and 56.51% on DSEC-Xtrm. Component ablations and complexity comparisons are provided to support the claimed benefits.

Critical strengths
  1. Clear motivation and distinctive fusion design. The method is motivated by the spectral complementarity between RGB and Event modalities and formulates a structured fusion operator using a damped wave equation. Compared with generic spatial mixing or attention-based fusion, it offers a relatively clear mathematical design and a task-relevant inductive bias.
  2. Competitive empirical results and useful component ablations. The method performs well across four segmentation benchmarks, with particularly notable gains on degraded inputs such as DSEC-Xtrm. Ablations show performance drops when the propagator, learnable Fourier filter, or cross-modal coupling is removed. The reported parameter count, FLOPs, and training throughput are also favorable relative to MambaSeg, although the backbones differ. Together, these results indicate practical potential.
Critical weaknesses
  1. The proposed physical interpretation and spectral mechanism are not sufficiently validated. The authors model abstract encoder features as damped wave fields without adequately explaining why this particular physical model should hold for RGB–Event representations. More importantly, while the narrative emphasizes using high-frequency Event information to compensate for degraded RGB information, learnable cross-modal coupling is restricted to a limited set of low-order Fourier modes (only 5.7% of rFFT2 coefficients at the first DSEC stage, according to Table 5). The paper does not show learned spectral responses, coupling coefficients, or modality-specific frequency distributions. Consequently, the central claimed mechanism remains insufficiently supported.
  2. The necessity of the damped-wave formulation is not established by sufficiently controlled comparisons.

The ablations confirm that WFTF works, but do not establish that this specific PDE-based propagator is needed rather than a simpler frequency-domain filter. DCT-based wave gating already reaches 78.75% mIoU on DDD17, only 0.64 percentage points below the full method. In addition, some SOTA comparisons use different backbones, model capacities, and training configurations; analogous controlled comparisons on the other datasets and repeated-run statistics are lacking. These gaps weaken the attribution of the gains to the claimed core innovation and are the main reason for my weak-reject recommendation.

Questions for authors

Q1. [Mechanism clarification]

Why is the selected damped wave equation a suitable model for intermediate GroupMamba features? Could the authors provide learned spectral responses, |κ(k)| maps across stages, and RGB/Event feature-frequency distributions to verify the proposed frequency-selective information exchange?

Q2. [Assessment-critical comparison]

With identical backbones, training settings, and comparable parameter budgets, how does WFTF compare with simpler spectral baselines (e.g., DCT-based gating, Gaussian filtering, or unconstrained learnable Fourier masks)? Is the 0.64-point margin over DCT wave gating consistent across random seeds?

Q3. [Robustness-scope clarification]

Do the robustness findings extend to unseen degradations and cases in which the Event stream is also degraded? In the reported rain/fog and simulated dark-night studies, Events remain unchanged. Please clarify the scope of the robustness claims; additional joint-degradation results would materially affect my assessment.

1 comment
AuthorsOpenReview ↗

Thank you for recognizing the motivation and practical potential of WFTF. We address the mechanism, spectral controls, and robustness scope separately.

We thank the reviewer for recognizing the distinctive fusion design, competitive performance, and practical potential of MambaFourier. We have expanded the experimental analysis with additional spectral controls on DERS-XS, repeated WFTF training runs, and supplementary DDD17 coupling experiments.

Q1. Physical motivation and frequency-selective exchange

We clarify that the wave formulation supplies a structured spectral inductive bias for learned representations. For q=∥k∥q=\|k\|, the analytical responses are

B(k)=e^{-\gamma q^2\tau}\frac{\sin(cq\tau)}{cq}.$$ The exponential envelope controls frequency-dependent attenuation, while the oscillatory terms combine independently projected displacement and velocity features. Compared with Gaussian smoothing, this parameterization additionally controls propagation through learned wave speed, damping, and evolution time. The exact solution applies to **within-modality propagation**. WFTF then combines this analytical component with a learned Fourier correction and reciprocal cross-modal mixing. Equation (6) corresponds to the effective initial velocity $\widehat V+(\gamma-\gamma q^2)\widehat U$, as detailed in Appendix C.1. Thus, the contribution is an analytical propagator embedded within a trainable spectral fusion operator. We also clarify the frequency support. On DSEC, coupling covers **5.7%, 22.7%, 76.2%, and 100%** of the respective stage spectra. Propagation operates across all frequencies, while learned exchange occurs within the retained bands. This enables selective early-stage exchange and broader relative spectral exchange at deeper stages. The appropriate interpretation is **multiscale, frequency-selective exchange in learned representations**. Temporal event contrast does not imply universally high spatial-frequency features. The coverage figures specify where exchange is possible; learned response maps and band interventions would quantify which supported frequencies a trained model actually uses. ### Q2. Additional spectral controls and repeatability We implemented and trained DCT, asymmetric DCT, Fourier, and Gaussian controls on DERS-XS using common dual GroupMamba-Small encoders, pretrained weights, decoder, data splits, and augmentation. | Fusion | Epochs | Best validation mIoU | Fusion parameters | |:--|--:|--:|--:| | DCT | 100 | 68.71 | 7.846M | | Fourier | 100 | 68.74 | 7.846M | | Gaussian | 100 | 68.72 | 7.849M | | Asymmetric DCT | 200 | 70.24 | 8.924M | | WFTF | 200 | **70.27** | **7.849M** | The direct 200-epoch comparison demonstrates **comparable validation accuracy with approximately 1.08M fewer fusion parameters for WFTF**. The shorter-schedule controls provide reference results under their stated budgets; the matched-schedule conclusion applies to WFTF versus asymmetric DCT. Across WFTF seeds 0/1/2, best validation scores within the common first 177 epochs of the planned 200-epoch schedule are **70.27, 69.47, and 69.72**, yielding **$69.82\pm0.41$ mIoU** (sample standard deviation). Seed 2 currently extends through epoch 177. Separate Test evaluations of validation-selected checkpoints yield **68.25 and 68.62** for seeds 0/1. These additional experiments extend the analysis beyond DDD17 and quantify WFTF variability. They are distinct from a paired multi-seed test of the original 0.64-point SWP margin. The original controlled DDD17 study also supports complementary component utility: **79.39** for full WFTF, **77.93** without propagation, **77.78** without coupling, and **77.68** without free-form filtering. ### Q3. Robustness with degraded events Our evaluation covers complementary degradation settings. Under corruption-aware RGB rain/fog training, MambaFourier achieves **73.7 mIoU**, compared with **72.5** for MambaSeg and **71.5** for EISNet. Joint degradation is represented by **DSEC-Xtrm**, which combines degraded frames with injected event shot noise, and **DERS-XS**, which contains low-light frames and noisy synthetic events. Therefore, the evidence extends beyond RGB-only corruption. We distinguish these benchmark and corruption-aware results from clean-trained, fixed-checkpoint evaluation on unseen corruptions. This ties each robustness conclusion to its actual experimental protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.