acceptodds
Under review as a conference paper at ICLR 2027

Spk2ImgMoE: Heterogeneous Mixture-of-Experts for High-Fidelity Spike-to-Image Reconstruction

Abstract

Spike-to-image reconstruction remains challenging due to the strong heterogeneity of spiking data across space, time, and noise conditions. Most existing reconstruction methods process all features with uniform computation, which is inefficient and limits reconstruction quality. We propose Spk2ImgMoE, a Mixture-of-Experts-based framework for high-fidelity spike-to-image reconstruction. Spk2ImgMoE integrates Mixture-of-Experts at a mid-level semantic feature stage, enabling adaptive feature refinement with conditional computation. In particular, Spk2ImgMoE incorporates a Heterogeneous Mixture-of-Experts (HMoE) module that models the intrinsic heterogeneity of spiking data through structurally diverse experts with different computational capacities. In addition, a bias-driven routing mechanism introduces a learnable bias into the gating logits, serving as a dynamic load balancer that regulates expert utilization and prevents expert collapse without auxiliary loss constraints. Extensive experiments on multiple datasets show that Spk2ImgMoE improves reconstruction quality while maintaining a favorable balance between performance and computational efficiency, confirming the effectiveness of heterogeneous capacity allocation for spiking data.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spike Image Reconstruction, Heterogeneous Mixture‑of‑Experts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3ndVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Spk2ImgMoE, a mixture-of-experts framework for spike-to-image reconstruction. The idea is that different regions of a spike stream have different spatial, temporal, and noise characteristics, and may therefore need different amounts of computation.

The model uses heterogeneous experts with different computational capacities and introduces bias-driven routing to improve expert utilization without an auxiliary load-balancing loss. Experiments on synthetic SREDS and several real-captured datasets show improvements over existing methods. The authors also study MoE integration, expert heterogeneity, and module placement through ablations.

Critical strengths
  1. I like the idea of using heterogeneous experts for this task. Different regions of a spike stream can vary quite a bit in complexity, so using the same amount of computation everywhere may not be ideal. Table 3 provides some good evidence for this design. The heterogeneous configuration reaches 37.76 dB PSNR, outperforming both the all-simple and all-complex configurations while keeping the parameter count moderate. I think this is an interesting result, since it suggests that combining experts with different capacities can be more effective than simply making all experts more complex.

  2. The reconstruction results are also quite good, especially considering the evaluation on both synthetic and real-captured datasets. On SREDS, Spk2ImgMoE improves PSNR from 37.46 dB to 37.76 dB over Spk2ImgMamba, with improvements in LPIPS and no-reference image-quality metrics as well. I also appreciate the real-captured experiments, since synthetic data may not fully reflect the characteristics of real spike cameras.

Critical weaknesses
  1. Section 3.3 argues that adding a learnable bias to the routing logits automatically compensates for under-utilized experts and prevents expert collapse. However, the formulation (h(x)=W_gx+b) alone does not establish this behavior. A conventional trainable bias is not necessarily updated according to expert utilization, and the paper does not provide a mechanism or theoretical argument showing why under-utilized experts should consistently receive larger bias updates.Table 2 demonstrates an improvement in reconstruction accuracy after adding the bias, but this is not direct evidence of improved load balancing. The paper would benefit from reporting expert selection frequencies, load distributions, or utilization changes during training. A comparison against an explicit load-balancing strategy would also help establish the claimed advantage.This concern matters because bias-driven routing is presented as one of the paper's main technical contributions.

  2. Compared with Spk2ImgMamba, the proposed method improves PSNR by 0.30 dB, but the parameter count increases from 0.282M to 0.364M and FLOPs increase from 0.099T to 0.126T.The result is encouraging, but it is difficult to determine how much of the improvement comes from the proposed expert allocation rather than additional model capacity. The homogeneous-expert ablations are helpful, but they do not replace a carefully controlled comparison with a non-MoE backbone at a similar computational budget.Furthermore, the paper does not report practical inference latency or a quantitative uncertainty analysis for the main benchmark comparisons. These would help assess whether the proposed architecture provides a robust efficiency advantage.

Questions for authors
  1. Could the authors provide expert selection frequencies or load distributions with and without the learnable bias? I am particularly interested in whether the bias actually improves expert utilization and reduces expert collapse, or mainly makes the gating function more flexible.

  2. How does Spk2ImgMoE compare with a parameter-matched or FLOP-matched non-MoE backbone? If such experiments are not available, how can we distinguish the benefit of heterogeneous expert allocation from the effect of increased model capacity?

1 comment

Thank you for your constructive feedback. Supporting figures are included in the Appendix (Supp. Figs. 6–8); we cite them below.

Q1. Expert utilization and collapse.

We compare three variants: no balancing (no routing bias or auxiliary balancing loss), standard auxiliary balancing (Baseline + MoE), and load-feedback balancing (our full model).

Our routing bias is updated using expert-load feedback. After each optimizer update, we compare each expert's normalized EMA load with the uniform target:

bi(t+1)=bi(t)+ηb(1N−f^i(t)).b_i^{(t+1)}=b_i^{(t)}+\eta_b\left(\frac{1}{N}-\hat{f}_i^{(t)}\right).

Here, f^i(t)\hat{f}_i^{(t)} is expert ii's normalized EMA load, NN is the number of experts, and ηb=0.001\eta_b=0.001. The EMA tracks actual Top-2 assignment shares with a decay of 0.95. Underused experts receive positive bias adjustments and overloaded experts negative adjustments, with magnitudes proportional to the load deviation. This gives the bias an explicit load-correction role, rather than merely increasing gating flexibility. The feedback applies even to experts receiving no assignments.

Supp. Fig. 6 shows per-expert utilization at the same MoE block position over the first 40,000 updates. Without either balancing mechanism, routing gradually concentrates on experts 0/1, each approaching 50% of assignment slots, while experts 2/3 become nearly unused. This is consistent with Top-2 expert collapse, where only two of four experts remain active. Both the auxiliary-loss baseline and the full model maintain participation from all four experts. Utilization curves use a 50-update moving average for display.

Supp. Fig. 7 quantifies the load distribution using normalized EMA-load entropy:

H(t)=−∑i=1Nf^i(t)ln⁡f^i(t)ln⁡N.H^{(t)}=-\frac{\sum_{i=1}^{N}\hat{f}_i^{(t)}\ln\hat{f}_i^{(t)}}{\ln N}.

Mean entropy over the first 40,000 updates is 0.5247 without balancing, 0.7576 for the auxiliary-loss baseline, and 0.8293 for the full model. Without balancing, entropy approaches 0.5, consistent with approximately equal use of only two experts. Entropy is computed at each logged update before display smoothing.

Together, the load-dependent bias update, utilization curves, and entropy statistics support the conclusion that bias feedback counteracts persistent expert collapse in the reported run. The auxiliary-loss baseline also avoids sustained collapse, while the full model achieves the highest mean normalized EMA-load entropy.

Q2. Comparison with a matched dense backbone.

We trained a dense non-MoE baseline matched to Spk2ImgMoE at 0.364M parameters and 0.126T FLOPs. The SREDS results are:

MethodPSNR ↑SSIM ↑LPIPS ↓
Dense non-MoE37.240.95770.0243
Spk2ImgMoE (Ours)37.760.96000.0201

At the same parameter and FLOP budget, Spk2ImgMoE achieves 0.52 dB higher PSNR, higher SSIM, and lower LPIPS.

Table 3 and Supp. Fig. 8 provide complementary quantitative and visual comparisons. All Complex Experts uses more parameters and FLOPs than Ours (0.410M vs. 0.364M; 0.137T vs. 0.126T), yet achieves lower PSNR (37.50 vs. 37.76 dB), higher LPIPS (0.0218 vs. 0.0201), and worse NIQE/BRISQUE scores on real data (6.819/18.923 vs. 6.693/15.183).

The close-up in Supp. Fig. 8 shows the complementary reconstruction behavior of the expert architectures. The separately trained complex-only model recovers sharper edges but leaves artifacts in smooth backgrounds. The simple-only model produces smoother backgrounds but preserves edge details less effectively. Ours combines cleaner backgrounds with clearer edges.

Together, these results show that the heterogeneous model outperforms both a dense model with the same parameter and FLOP budget and a larger all-complex model. They support the benefit of allocating different expert capacities to different features.

Reviewer_t3yLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Spk2ImgMoE, a heterogeneous Mixture-of-Experts framework for spike-to-image reconstruction. The method introduces simple and complex experts with different computational capacities. Experiments on synthetic and real-world datasets demonstrate improved reconstruction performance compared with existing methods.

Critical strengths

This paper introduces a heterogeneous MoE architecture for spike-to-image reconstruction and achieves better reconstruction performance over existing methods.

Critical weaknesses
  1. Adding a learnable bias to the router is a standard operation rather than a novel idea. The author claims that adding bias can achieve load balancing, but does not provide relevant proof.
  2. The authors propose a heterogeneous MoE motivated by the different computational needs of simple and complex regions. However, they do not provide evidence that the router assigns features to the appropriate experts based on these needs.
Questions for authors
  1. How does the proposed learnable bias achieve load balancing? Could the authors provide evidence to support this claim?
  2. Could the authors provide evidence that the router assigns features to different experts according to their computational needs?
1 comment

Thank you for identifying the need to substantiate both load balancing and content-dependent routing. Supporting figures are included in the Appendix (Supp. Figs. 6–12); we cite them below.

Q1. Load balancing.

The routing bias provides direct feedback to correct expert-load imbalance. We maintain an exponential moving average (EMA) of actual Top-2 assignment shares with a decay of 0.95. After each optimizer update, we compare each expert's normalized EMA load with the uniform target and update its bias:

bi(t+1)=bi(t)+ηb(1N−f^i(t)).b_i^{(t+1)}=b_i^{(t)}+\eta_b\left(\frac{1}{N}-\hat{f}_i^{(t)}\right).

Here, f^i(t)\hat{f}_i^{(t)} is expert ii's normalized EMA load, NN is the number of experts, and the bias update rate ηb=0.001\eta_b=0.001. Underused experts receive positive corrections to their routing logits, while overloaded experts receive negative corrections. The correction magnitude increases with the load deviation. This feedback applies even to experts receiving no assignments and requires no auxiliary balancing loss.

Supp. Fig. 6 compares three variants at the same MoE block position over the first 40,000 updates: no balancing, standard auxiliary balancing (Baseline + MoE), and load-feedback balancing (full model). Without routing bias or an auxiliary balancing loss, assignments concentrate on experts 0/1, each approaching 50%, while experts 2/3 become nearly unused. Both balancing methods maintain participation from all four experts. Utilization curves use a 50-update moving average for display.

Supp. Fig. 7 quantifies load balance using normalized EMA-load entropy:

H(t)=−∑i=1Nf^i(t)ln⁡f^i(t)ln⁡N.H^{(t)}=-\frac{\sum_{i=1}^{N}\hat{f}_i^{(t)}\ln\hat{f}_i^{(t)}}{\ln N}.

Entropy is computed at each logged update before display smoothing. Mean entropy over the first 40,000 updates is 0.5247 without balancing, 0.7576 with standard auxiliary balancing, and 0.8293 for the full model.

These results support the effectiveness of bias feedback in counteracting persistent expert collapse. In the reported runs, our method also achieves more balanced EMA loads than the auxiliary-loss baseline.

Q2. Evidence for content-dependent expert allocation.

Thank you for this question. We examine whether heterogeneous expert capacity is allocated according to local content through two complementary analyses: reconstruction ablations with different expert architectures and visualization of actual Top-2 assignments.

Supp. Fig. 8 illustrates why different regions may benefit from different expert architectures. The separately trained complex-only model preserves a sharper boundary but produces visible artifacts in the bright background. The simple-only model produces a smoother background but a less distinct boundary. The heterogeneous model combines a cleaner background with better-preserved boundary details. These observations support the complementary reconstruction behavior of the two expert types.

Supp. Figs. 9–12 provide spatial evidence through additional routing analysis. Experts 0/2 are complex, while experts 1/3 are simple. Actual Top-2 assignment maps show different routing patterns between foreground structures and surrounding backgrounds in Football_1 and Keyboard_1. Reconstruction-gradient maps identify strong spatial intensity variations, including object boundaries and fine textures. Reconstruction-change maps show differences between consecutive outputs, providing qualitative temporal context, including changing object boundaries.

Supp. Figs. 10 and 12 quantify complex-expert selection within the displayed high-gradient regions. Complex experts occupy 71.50% of Top-2 assignment slots in Football_1 and 92.61% in Keyboard_1. In the selected Keyboard_1 region, 88.87% of tokens select both complex experts. These statistics indicate frequent use of higher-capacity experts within the selected structurally detailed regions.

Together, the architecture ablations and actual routing maps provide complementary evidence of spatially selective capacity allocation associated with local structural content.

Reviewer_JiA9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper inserts three MoE blocks into a Mamba-based spike-to-image network. Each block has four experts (two simple, two complex) with top-2 routing. The routing logits are W_g x + b, and the authors say the learnable bias b balances expert load without an auxiliary loss. On SREDS, PSNR rises from 37.46 dB to 37.76 dB over the baseline, while parameters go from 0.282M to 0.364M and FLOPs from 0.099T to 0.126T. On real captures, NIQE and BRISQUE improve slightly.

Critical strengths

•Evaluation covers one synthetic dataset and three real ones, against physics-based, SNN, CNN, and Mamba methods, with PSNR, SSIM, and LPIPS plus no-reference metrics (Table 1).

•LPIPS improves most clearly, from 0.0239 to 0.0201 over the Mamba baseline, and BRISQUE on real data goes from 17.97 to 15.18.

•Ablations cover MoE integration, expert type, expert count, and placement (Tables 2 to 4), with visual comparisons in Fig. 4.

•The network is small and the setup is described clearly enough to follow.

Critical weaknesses

•As written, the bias cannot do what the paper says. Eq. 9 renormalizes the selected experts' probabilities as p_i / Σ_{j∈T} p_j. The softmax denominator from Eq. 8 cancels, so the output depends only on the logits of the selected experts, and the top-K choice itself carries no gradient. An expert that is never selected receives exactly zero gradient on its logit and its bias, so b_i cannot rise "naturally" for under-utilized experts (Sec. 3.3), and nothing in the loss rewards balance. This is also unlike the Loss-Free Balancing strategy in DeepSeek-V3, which the paper cites as precedent: there the bias is updated by a load-based rule and used only for selection. Here it is an ordinary gate bias trained by the reconstruction loss.

•There is no evidence of imbalance or of its cure. The paper shows no per-expert load, no routing entropy, and no comparison with an MoE trained with an auxiliary loss, although Sec. 2.2 states that such a loss conflicts with pixel fidelity. The claim that plain MoE is imbalanced rests on its lower PSNR, which is circular.

•The size of the bias effect is hard to believe. The bias adds 12 scalars (four per block, three blocks), yet Table 2 credits it with +0.13 dB PSNR and a BRISQUE drop from 18.27 to 15.18 on real data. Only two training-loss curves (Fig. 5) are given, with no variance on final metrics. I would treat the 0.1 to 0.2 dB differences between rows in Tables 2 to 4 as unresolved. That covers placement (37.54, 37.58, and 37.76), N and k (37.59 to 37.76), and expert type (37.50 and 37.53 against 37.76).

•A capacity-matched dense control is missing. The full model uses 29% more parameters and 27% more FLOPs than the baseline (Table 1), so "negligible overhead" in Sec. 4.2 overstates it, and the MoE does not reduce compute relative to the baseline. Plain Baseline + MoE already provides +0.17 dB of the +0.30 dB total, and on real data it is worse than the baseline (NIQE 7.032 against 6.733, BRISQUE 18.27 against 17.97). A dense network widened to 0.364M parameters would show whether sparse routing matters.

•"Adaptive capacity allocation" is never demonstrated. The router is not told which experts are complex, and no analysis shows which tokens or regions go where. The idea of experts with different complexity comes from cited image-restoration work (Zamfir et al., 2024), so the new parts are the application and the bias term.

•Smaller issues. Table 4 lists 0.354M parameters for N=4, k=1 and 0.364M for N=4, k=2, and the same 0.364M for N=4, 6, and 8, so it is unclear whether activated or total parameters are counted. Real-data results use only NIQE and BRISQUE, given as one number per method with no per-dataset breakdown, and these metrics can reward sharpening. Efficiency is reported as FLOPs only, without latency, and code is promised only after acceptance.

Questions for authors

1.Given Eq. 9, how do non-selected experts, and their biases, ever receive a gradient? If they do not, what prevents the routing from settling on the same experts?

2.Can you show per-expert load over training, and routing entropy, for Baseline + MoE and the full model?

3.How do an MoE with a standard auxiliary loss and a Loss-Free-Balancing-style bias update perform under the same budget?

4.What does a dense baseline matched to 0.364M parameters and 0.126T FLOPs achieve?

5.Over several seeds, what are the mean and standard deviation of PSNR, SSIM, and LPIPS for the rows of Tables 2 to 4?

6.Does Table 4 report activated or total parameters, and why are they constant across N?

7.Which pixels or tokens are routed to complex experts? Maps for noisy, textured, and motion-heavy regions would help.

8.Can you give per-dataset NIQE and BRISQUE, plus latency and memory?

2 comments

Thank you for the detailed assessment. Supporting figures are included in the Appendix (Supp. Figs. 6–12); we cite them below.

Q1. Load-feedback bias update. After each optimizer update, we adjust routing biases using the EMA of actual expert-selection shares fi(t)f_i^{(t)}:

mi(t)=ρmi(t−1)+(1−ρ)fi(t),m_i^{(t)}=\rho m_i^{(t-1)}+(1-\rho)f_i^{(t)},

f^i(t)=mi(t)∑j=1Nmj(t),\hat{f}i^{(t)}=\frac{m_i^{(t)}}{\sum_{j=1}^{N}m_j^{(t)}},

bi(t+1)=bi(t)+ηb(1N−f^i(t)).b_i^{(t+1)}=b_i^{(t)}+\eta_b\left(\frac{1}{N}-\hat{f}_i^{(t)}\right).

Here, fi(t)f_i^{(t)} is expert ii's share of actual Top-2 assignment slots, ρ=0.95\rho=0.95 is the EMA decay, and ηb=0.001\eta_b=0.001 is the bias update rate. Underused experts receive positive corrections and overused experts negative corrections, making underused experts more likely to be selected. Feedback remains active even for unassigned experts and can correct persistent routing concentration without an auxiliary balancing loss.

Q2. Per-expert utilization and routing entropy. Supp. Figs. 6–7 compare no balancing, standard auxiliary balancing (Baseline + MoE), and load-feedback balancing (full model) at the same MoE block position over the first 40,000 updates. Without balancing, experts 0/1 each approach 50% of assignment slots, while 2/3 become nearly unused. Both balancing strategies maintain participation from all four experts. Normalized EMA-load entropy is: H(t)=−∑i=14f^i(t)ln⁡f^i(t)ln⁡4.H^{(t)}=-\frac{\sum_{i=1}^{4}\hat{f}_i^{(t)}\ln\hat{f}_i^{(t)}}{\ln 4}.

Mean entropy is 0.5247, 0.7576, and 0.8293, respectively. Entropy is computed before display smoothing. These diagnostics support reduced persistent expert collapse in the reported runs.

Q3. Performance under the same budget. Table 2 compares the same four-expert, Top-2 architecture. Both Baseline + MoE and Ours have the same reported complexity (0.364M parameters and 0.126T FLOPs). Baseline + MoE uses an auxiliary-loss coefficient of 0.01; Ours uses load feedback without that loss. SREDS PSNR/SSIM/LPIPS are 37.63/0.960/0.0206 versus 37.76/0.960/0.0201; aggregate real-data NIQE/BRISQUE are 7.032/18.269 versus 6.693/15.183. These results show that load-feedback balancing achieves better reconstruction quality than standard auxiliary-loss balancing.

Q4. Dense control. We trained a dense non-MoE control matched to Spk2ImgMoE at 0.364M parameters and 0.126T FLOPs.

MethodPSNR ↑SSIM ↑LPIPS ↓
Dense non-MoE37.240.95770.0243
Spk2ImgMoE (Ours)37.760.96000.0201

The 0.52 dB improvement supports the MoE design under this profiling protocol。

Q5. Multi-seed uncertainty. Following the evaluation protocol of prior work, we report single-run results. We observed stable training and similar performance across runs but did not systematically record multi-seed statistics for Tables 2–4. We therefore cannot provide reliable means and standard deviations and will clarify this limitation.

Q6. Parameter accounting. Thank you for pointing out this ambiguity. The parameter values currently reported in Table 4 are obtained by profiling a single forward pass with THOP; they do not represent the total number of parameters stored in the model. For the configurations with N = 4, 6, and 8, k is fixed at 2. Therefore, with the expert widths held constant, each token activates two expert subnetworks, keeping the per-token expert-parameter budget approximately constant as N varies.

Q7. Routing maps. Our router operates on intermediate feature tokens rather than raw pixels. Supp. Figs. 9–12 show actual Top-2 assignments for Football_1 and Keyboard_1, together with reconstruction-gradient maps highlighting spatial detail and consecutive-output-change maps visualizing temporal variation. Experts 0 and 2 are complex, while experts 1 and 3 are simple. The maps show spatially varying assignments around object boundaries and textures. The selected regions are locations with strong spatial gradients in the reconstructed images, highlighting pronounced edges and fine structures. Within these regions, complex experts account for 71.50% of assignment slots in Football_1 and 92.61% in Keyboard_1; 88.87% of tokens in the Keyboard_1 region select both complex experts.

Supp. Fig. 8 compares separately trained all-complex, all-simple, and heterogeneous models. The all-complex model preserves sharper boundaries but produces background artifacts; the all-simple model gives smoother backgrounds but less distinct boundaries. Ours combines cleaner backgrounds with preserved boundary details.

Q8. Quality and practical efficiency. We report reference-based metrics (PSNR/SSIM/LPIPS) for SREDS (720 × 1280, with GT) and no-reference metrics (NIQE/BRISQUE) for real-captured data (250 × 400, without GT). The tables below also report inference latency (ms/frame) and peak GPU memory (MiB) for all compared methods, measured on an RTX 4090.

SREDS (720 × 1280)

ModelPSNR ↑SSIM ↑LPIPS ↓Mean inference latency (ms/frame)Peak memory (MiB)
TFP25.350.6900.25470.279166
TFI18.500.6380.25904.167186
TFSTP20.680.6180.27613.185164
Spk2ImgNet36.130.9500.0294828.1159,748
BSF37.320.9560.0263734.31320,298
WGSE37.440.9580.0241451.3193,844
Spk2ImgMamba37.460.9580.023982.3093,088
Spk2ImgMoE (Ours)37.760.9600.0201103.2573,308

Real-captured data (250 × 400)

ModelNIQE ↓BRISQUE ↓Mean inference latency (ms/frame)Peak memory (MiB)
TFP21.38982.0650.06418
TFI15.96764.2522.67220
TFSTP13.62954.1100.67418
Spk2ImgNet6.95232.38667.2117,392
BSF9.27532.011103.5456,202
WGSE8.34930.22943.1551,190
Spk2ImgMamba6.73317.97310.361370
Spk2ImgMoE (Ours)6.69315.18317.503372

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.