- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GFTC converts pretrained multimodal language models to finite-precision time-to-first-spike (TTFS) representations while freezing their weights. It allocates nonuniform codewords using activation density and squared task-gradient sensitivity, then calibrates codebooks layer by layer on executed SNN trajectories using a teacher-output accept/reject objective. Token ranges and modality-specific codebooks address distribution variation. The paper gives a high-resolution weighted-quantization result, a conditional layer-error bound, and convergence of accepted scalar objective values. Evaluation covers two LLaVA models and three Qwen3-VL scales, with cumulative ablations and an analytical energy estimate for a specified prefill workload. The nominal 6/8-bit settings use 64/256 temporal slots, respectively, plus an implicit silent zero code.
Critical strengths
-
The reported conversion results are strong across substantially different MLLMs without weight retraining. At 8-bit temporal precision, LLaVA-1.5 averages 61.18 versus 61.36 for the ANN, and LLaVA-NeXT averages 63.44 versus 63.42. Qwen3-VL at 4B/8B/32B also improves substantially over the adapted TTFSFormer baseline, although it remains 1.67/1.32/1.62 points below the corresponding ANN averages. This is meaningful evidence that learned finite codebooks can mitigate severe conversion loss.
-
Calibrating against the upstream SNN trajectory is a coherent response to accumulated conversion error. The cumulative TextVQA ablation increases from 47.63 to 52.48 with gradient-guided nonuniform coding, then to 56.31 with iterative calibration, and finally to 58.06 with the remaining components (Table 5). The paper also carefully limits Theorem 2 to assumed local perturbation bounds and Theorem 3 to scalar objective-value convergence; it does not claim that the accept/reject test certifies all local errors.
Critical weaknesses
-
The finite interface is insufficiently specified for the claimed complete conversion. Equations 2-3 and 26-27 require positive thresholds and positive codewords; values below the smallest positive threshold reconstruct to zero. Yet the converted boundaries include signed Q/K and RoPE outputs, centered LayerNorm, SiLU/GELU, residuals, and vocabulary logits (Appendix B). The continuous TTFS reference allows a general [a,b] range, but no sign/offset mapping and inverse correction are given for the learned finite codebooks. TTFSFormer already supports signed values via a zero-reference temporal mapping, so signed TTFS is feasible; what is missing is its composition with GFTC's stated finite maps. Calibration units also need clarification: Equation 61 (p19) enforces a minimum gap and minimum final codeword of 0.006, implying c_1 >= 1.536 at T=256 and >=24.576 at T=4096 for projected candidate codebooks in the stated units. Rejected candidates can leave initialization unchanged. How this projection applies to codebooks initialized from bounded softmax/sigmoid outputs is unexplained. These omissions prevent checking whether the displayed algorithm describes the evaluated complete finite model, rather than merely a positive-domain component of it.
-
The execution specification does not justify the energy ledger for learned nonuniform codewords. Appendix B supplies value-domain decode/operator/re-encode correspondences. For a linear operator, an event represents a contribution w_ij*c_t; attention products combine two dynamic decoded values. Appendix E nevertheless counts event fan-out as ACs via N_AC=sum_l S_l F_l (Eq66, p22), leaving only 0.13G MACs. Inherited temporal kernels could implement these operations, but their adaptation to token/operator-specific learned codewords, extra neuronal operations, codebook/lookup storage, and access costs are not specified. The reported 64.68% reduction is an analytical estimate for 604-position prefill plus one vocabulary readout, explicitly excluding autoregressive generation. Its arithmetic is correct, but 18.70 of 21.74 J comes from system costs whose underlying per-operator read/write/comparison counts are unavailable. The current evidence therefore does not establish the practical energy benefit of the claimed fully temporal implementation.
Questions for authors
-
Please describe how arbitrary learned codewords are processed by the temporal linear, QK, AV, and nonlinear kernels, including any preweighted matrices, lookup tables, added neurons, or state. Can you provide the per-operator event/MAC/AC/read/write/comparison ledger producing Table 3, including these costs, and its sensitivity to the assumed weight reuse? A matched nonspiking activation-quantization control using the same codebooks would help distinguish temporal execution savings from quantization savings. This requests support for the current prefill claim, not mandatory deployment on a new hardware platform.
-
Can the same finite budget and trajectory calibration be compared with unweighted distribution-aware codebooks, such as density proportional to p(u)^(1/3) or Lloyd-Max initialization? Table 5 compares uniform coding with the combined gradient-guided nonuniform design, so its 4.85-point first increment does not isolate gradient sensitivity.