acceptodds
Under review as a conference paper at ICLR 2027

SpinSCI: Mask-Conditioned Spatial-Temporal Spin Transformer for Video Snapshot Compressive Imaging

Abstract

Video snapshot compressive imaging (SCI) recovers multiple frames from one spatiotemporally coded measurement. This paper proposes *SpinSCI*, a mask-conditioned Spatial-Temporal Spin Transformer whose *Mask-Conditioned Ising Refinement Block (MC-IRB)* combines attention logits and sensing information to form signed spin couplings. Finite mean-field updates gate residual feature refinement. Physics-guided initialization and final data consistency incorporate the measurement model. Compared with the state-of-the-art method HiSViT, SpinSCI achieves average PSNR gains of **0.50 dB**, **0.23 dB**, and **0.04 dB** on *small-scale gray, mid-scale color, and large-scale color benchmarks*, respectively, with **16.9–17.0% fewer parameters, 21.0–21.6% fewer FLOPs, 77.0–84.8% lower inference latency, and 77.0–78.0% lower peak memory**. Qualitative comparisons on the *real benchmark* further show recovery of fine details and textures. If the paper is accepted, the code will be released by the camera-ready deadline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video snapshot compressive imaging, computational imaging, spatiotemporal transformers, Ising model, mean-field inference, mask-conditioned reconstruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tiPnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a spatial-temporal transformer for video snapshot compressive imaging (SCI), built on a slimmed STFormer-style backbone with parallel windowed spatial and temporal attention, physics-guided initialisation, and a final data-consistency step. The main contribution is a Mask-Conditioned Ising Refinement Block, integrated into each attention branch. It generates token biases and signed pairwise couplings from pre-softmax logits and conditioning signals, then uses mean-field iterations to gate a residual MLP correction to the attention output. Experiments show modest PSNR gains over baselines across three benchmarks, with approximately 20% fewer parameters and FLOPs, substantially lower latency and peak memory, and qualitative validation on real data.

Critical strengths
  1. The model achieves competitive reconstruction quality across three benchmarks while substantially reducing latency and peak memory, supporting practical large-scale reconstruction.

  2. The evaluation is rigorous and includes well-designed and controlled ablations, and the main claims are generally supported by empirical evidence.

  3. The proposed conditioning mechanism appears to promote generalization, with comparable or better performance on an unseen mask than on the training mask.
Critical weaknesses
  1. With regards to novelty, MC-IRB is closely related to established techniques, namely sigmoid gating of attention outputs, signed attention, Hopfield-style inference, and unrolled mean-field methods. The Ising formulation does not deliver additional insight. In particular, the paper provides no explicit energy function, no analysis of the three-step mean-field iteration, and no interpretation of the learned couplings.
  2. Moreover, the reconstruction quality gains are modest, and it seems that the efficiency improvements stem largely from the slimmed backbone rather than from MC-IRB (Table 1). Without a fully trained backbone-only baseline, the incremental contribution of MC-IRB cannot be assessed.
  3. Regarding prior work, the paper omits relevant ML prior art, as well as major SCI approaches (e.g., deep unfolding, mask-conditioned reconstruction, and efficient video SCI methods).
  4. Lastly, the exposition is heavy in physics terminology (spins, couplings, magnetisations) that is neither introduced nor properly motivated for the SCI readership. A self-contained introduction would clarify the method.
Questions for authors
  1. Could the authors please report the fully trained backbone without MC-IRB in Tables 4 - 6, under the same training schedule and evaluation protocol?
  2. Do the ablation gains persist at full training, given that they decrease between 40 and 80 epochs?
  3. What does the mean-field iteration provide beyond a sigmoid gate on the attention output (e.g., Gated Attention, NeurIPS 2025), or beyond signed attention (e.g., Differential Transformer, ICLR 2025)?
  4. Is there an underlying energy function derivation? Do the iterations contract to a meaningful fix point?
  5. How does SpinSCI perform across mask distributions, compression ratios, and calibration errors?
3 comments
AuthorsOpenReview ↗

full-training backbone, energy and fixed point, and prior work

W1. Novelty and the Ising formulation. Appendix E compares MC-IRB with gating, signed attention, Hopfield-style and mean-field methods. None of them ties the gates of different tokens through signed couplings built from attention logits and the sensing mask\color{blue}{\textbf{None of them ties the gates of different tokens through signed couplings built from attention logits and the sensing mask}}, which is what MC-IRB adds. The paper provides all three items listed:

  • Explicit energy function. Eq. 2 in Section 3.2.
  • Analysis of the three-step iteration. See Q4; accuracy saturates at three steps (Table 2).
  • Interpretation of the learned couplings. In the first coupled temporal block, the couplings follow exposure state (+0.35 between frames sharing a token's mask value, −0.53 otherwise), the magnetizations label unmeasured tokens, which receive 1.93× larger gated updates (Appendix B.5, Figure B.2).

W2 / Q1 / Q2. Gains and the fully trained backbone. Read together with cost, the gains are not modest: SpinSCI-B moves beyond the previous accuracy–cost frontier on all three benchmarks\color{blue}{\textbf{SpinSCI-B moves beyond the previous accuracy–cost frontier on all three benchmarks}} (see the general response).

  • Backbone at full training (Q1). With only the MC-IRB switch changed in the Tables 4–6 configurations, the backbone reaches 36.99, 37.83 and 37.29 dB; SpinSCI-B leads it by 0.52, 0.41 and 0.39 dB\color{blue}{\textbf{SpinSCI-B leads it by 0.52, 0.41 and 0.39 dB}} (Table B.1).
  • Persistence (Q2). The gain persists: 0.45, 0.39 and 0.37 dB at 40, 80 and 200 epochs under the shared protocol (Table B.1), and 0.41 dB over five seeds (95% CI [0.226, 0.591] dB; Table B.2). At full training, permuting the temporal couplings costs 1.04 dB (Table B.3).
  • Efficiency. MC-IRB adds 7.2% FLOPs and outperforms a larger, deeper backbone by 0.20 dB (Table 2).

W3. Prior work. New Appendix E covers unrolled mean-field inference (CRF-as-RNN, DPN), energy-based attention (modern Hopfield, Energy Transformer), signed and gated attention, deep equilibrium models, deep unfolding for SCI, and recurrent, reversible and graph-based video SCI networks. It also defines mask-conditioned reconstruction and adds MST, DAUHST and Yin et al. (2025), which modulate individual features or stages. To our knowledge, SpinSCI is the first video SCI method in which the masks condition the pairwise interactions of a refinement module\color{blue}{\textbf{To our knowledge, SpinSCI is the first video SCI method in which the masks condition the pairwise interactions of a refinement module}}; removing the mask channels costs 0.30–0.38 dB on an unseen mask (Table 3). We already compare the efficient EfficientSCI-B and MambaSCI-B and will add any method the reviewer names.


W4. Physics terminology. Section 3.2 now defines each term: a spin is a latent binary state of a token, a coupling favors agreement or disagreement between two tokens, a field is a token's own evidence, and the magnetization is the mean spin that sets the token's gate.


Q3. Beyond a sigmoid gate and signed attention. MC-IRB couples the gates of different tokens; neither alternative does\color{blue}{\textbf{MC-IRB couples the gates of different tokens; neither alternative does}}.

  • Sigmoid gate (Gated Attention). It gates each token from that token alone; inside our block it trails MC-IRB by 0.23 dB (0.26 dB over five seeds, 95% CI [0.061, 0.452] dB; Tables 2 and B.2).
  • Signed attention (Differential Transformer). It subtracts two softmax maps to sign attention weights; MC-IRB's signs instead couple gates, and making them unsigned costs 0.14 dB (0.19 dB over five seeds).
  • What mean-field inference adds. Each gate depends on its partners' states, which lets the temporal block find unmeasured tokens.

Q4. Energy function and fixed point. Yes to both. Eq. 2 is the energy; Appendix A.5 derives the update as the mean-field stationarity condition (Proposition A.7). Theorem A.9 shows that if β∥J∥2<1\beta\|\mathbf{J}\|_2<1, the iteration has a unique fixed point and converges geometrically; on the released checkpoints it holds for 9 of 12 coupled temporal blocks in every refinement group, and the other three are locally contractive in over 99.999% of groups. The reached state is meaningful: it nearly minimizes the pairwise energy and separates measured from unmeasured tokens (Appendix B.5).


Q5. Mask distributions, compression ratios and calibration.

  • Mask distributions and calibration. See KEV3 (W5/Q5): SpinSCI keeps 37.07 dB on an unseen binary mask, and Appendix B.6 tests every method under mask calibration error.
  • Compression ratios. Like STFormer, MetaSCI and RevSCI, SpinSCI uses one checkpoint per CR on the real benchmark (CR 10 and 50), both released in our \href\href{https&#58;//anonymous.4open.science/r/video-sci-benchmark-7C3C/}{\color{blue}{\textbf{anonymous repository}}}. Beyond those on our real benchmark (STFormer-B, CR 10 only; PnP-FastDVDNet, CR-agnostic), no baseline has released official real-data checkpoints. New Appendix D.6 simulates seven UVG videos with the real mask: the CR-50 checkpoint reaches 34.73 dB at CR 50 and keeps 34.70 and 34.20 dB at CR 40 and 30 without retraining\color{blue}{\textbf{keeps 34.70 and 34.20 dB at CR 40 and 30 without retraining}}, whereas PnP-FastDVDNet stays 4.91–12.17 dB below SpinSCI's better checkpoint at every CR (22.56 dB at CR 50).

I thank the authors for their detailed response and for clarifying all the issues that I raised in my review. I am therefore raising my recommendation to weak accept.

I would recommend revising the abstract and introduction to state explicitly that the efficiency gains come predominantly from the backbone, and the accuracy gains from MC-IRB.

AuthorsOpenReview ↗

Thank you for the updated recommendation

Thank you for raising your recommendation. We have addressed your suggestion in our revised manuscript as follows:

  1. Abstract. We kept the abstract unchanged, since inserting the breakdown there would break the flow between its sentences.
  2. Introduction. We appended the following sentence to the second contribution: "This efficiency comes mainly from the STSFormer backbone design, while the quality gains come from MC-IRB."

Thank you again, and please let us know if anything else should be added.

Reviewer_W3CxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SpinSCI for video snapshot compressive imaging (SCI). The backbone follows STFormer, with parallel spatial and temporal window attention. It adds HiSViT-style pooled key/value heads, a downsampled token grid, a full-resolution stem skip, an exposure-normalized back-projection initialization, and a final data-consistency (DC) projection.

Critical strengths
  1. Careful control experiments for the proposed block. Table 2 is the strongest part of the paper. The controls cover the obvious alternative explanations ("it is just extra gating" or "just extra capacity"):
    • a fixed g=1/2g=1/2 gate;
    • a conditioned gate without mean-field;
    • a field-only gate;
    • Kmf=0K_{mf}=0;
    • J=0J=0 and ∣J∣|J|;
    • a capacity-matched 10-block backbone without MC-IRB. The repeated-seed protocol (37.07 ± 0.10 dB, paired intervals) supports that the 0.14–0.29 dB effects are not seed noise. The mask-transfer experiment (Table 3) is a practically relevant check that explicit mask conditioning helps on an unseen mask. This matters to my assessment because it shows MC-IRB has a real, if modest, effect within a fixed backbone.
  2. Good quality at modest model size on the small- and mid-scale benchmarks. The model has 7.5M parameters. The +0.50 dB average on small-scale gray is meaningful in this area, with +1.8 dB on Aerial and +1.2 dB on Crash.
Critical weaknesses
  1. The headline efficiency claims are not attributable to the proposed method and may be an artifact of execution settings.
    • SpinSCI is timed with BF16 autocast and torch.compile on a GH200. Baseline execution settings are deferred to Appendix C.
    • FLOPs drop by only 21% vs HiSViT9 (1214 vs 1536 G on small-gray), yet latency drops about 4.3× (41 vs 178 ms) and peak memory about 4.3× (1130 vs 4904 MiB). A precision or compilation mismatch (e.g., BF16+compile vs FP32 eager) would plausibly explain a gap of this size.
    • Table 1 shows that MC-IRB itself increases latency by 28.6% (117 → 150 ms) and FLOPs by 7.2%. Any efficiency advantage therefore comes from backbone choices (e.g., token-grid resolution and width, parallel branches, half-width temporal heads) and/or execution settings, not from the paper's contribution.
    • The abstract nevertheless presents the 77–85% latency and memory reductions as a co-equal headline result. Without baselines timed under identical precision and compile settings, the efficiency comparison cannot be interpreted. [If Appendix C in the supplementary shows that all baselines were run with BF16+compile, soften or remove this point.]
  2. The comparison with HiSViT is confounded, the margins are small, and the Ising framing is largely descriptive.
    • Confounds. Several ingredients besides MC-IRB differ from HiSViT: the Muon optimizer, a final DC projection, auxiliary intermediate supervision, the stem skip, and the training recipe. For binary masks the DC projection makes the output measurement-consistent up to ϵ\epsilon. Small- and mid-scale baseline numbers are copied from publications, while "mask differences" are documented in an appendix.
    • Missing ablation. The full-protocol (200-epoch) backbone without MC-IRB is never reported. The reader therefore cannot tell how much of the +0.23 / +0.50 dB over HiSViT is due to MC-IRB.
    • Large-scale result. On large-scale color the gain is +0.04 dB, with lower SSIM (0.979 vs 0.980) and losses on Football and Swinger. Given the 0.10 dB seed SD, this is a tie, not "state of the art".
    • Ising/mean-field framing. No energy is minimized and convergence is not enforced (acknowledged in Sec. 3.2). With Kmf=3K_{mf}=3, MC-IRB is effectively a three-step message-passing gate over the attention graph.
    • Unsupported mechanism claim. The claim that signed couplings "support the recovery of fine details and edges" has no direct evidence, such as magnetization or gate maps compared against edges.
    • "Formal analysis". The stated properties — symmetric JJ, bounded tanh/sigmoid, and DC residual reduction for binary masks — hold by construction. They do not strengthen the empirical case.
    • Related work. Prior work on unrolled mean-field inference as differentiable recurrent layers (e.g., CRF-as-RNN, Zheng et al., ICCV 2015) and on energy-based views of attention is not discussed.
Questions for authors
  1. Were all baselines' latency and memory measured with the same BF16 autocast + torch.compile on the same GPU? If not, please report baselines under matched settings, or SpinSCI in FP32 eager. This determines whether the efficiency claims stand.
  2. Are SpinSCI's small- and mid-scale results computed with exactly the same masks as the published baseline numbers? What are the "mask differences" in Appendix C? What does HiSViT9 (official checkpoint) obtain on your masks?
  3. What is the full-protocol (200 epochs, same recipe) PSNR of the backbone without MC-IRB on the three benchmarks? How much does the final DC step contribute (SpinSCI with vs. without DC)? If HiSViT does not apply a final DC projection, does adding one to HiSViT close the gap? Answers to Q2–Q3 determine how much of the SOTA gain is due to the paper's contribution.
  4. Can you visualize magnetizations and gates for representative frames, and quantify whether they correlate with edges or mask-transition regions? This would substantiate the mechanism claim.
  5. (Clarification) Does accuracy improve monotonically with Kmf∈{1,2,3,5}K_{mf}\in\{1,2,3,5\}, as the mean-field interpretation would suggest, or does it saturate or degrade?
5 comments
AuthorsOpenReview ↗

matched timings, isolated MC-IRB gains, and Ising evidence

W1 / Q1. Execution settings and attribution. All cross-method timings in the main paper now use FP32 eager inference for every method\color{blue}{\textbf{All cross-method timings in the main paper now use FP32 eager inference for every method}} (Appendix C). SpinSCI-B has 16.1–49.0% lower latency and 59.9–62.7% lower peak memory than HiSViT₉; the abstract now reports these instead of the submitted 77.0–84.8%, and Appendix C.1 adds a BF16-compiled comparison. Efficiency comes from the whole design; MC-IRB buys accuracy within it: +7.2% FLOPs for +0.39 dB, whereas a deeper, larger backbone without MC-IRB is 0.20 dB lower (Table 2).


W2. Confounds. $\color{blue}{\textbf{The comparison with HiSViT may seem confounded because SpinSCI is a new design rather than an incremental one}}$$\color{blue}{\textbf{, but the paper provides controlled ablations that isolate MC-IRB from every other ingredient}}$:

  • Muon optimizer, training recipe and stem skip. Every ablation in Tables 1–3 holds all of these fixed (Section 4.2), so the MC-IRB gains cannot come from them. Under the full recipe of Tables 4–6, which also holds them fixed, SpinSCI-B leads the backbone without MC-IRB by 0.52, 0.41, and 0.39 dB\color{blue}{\textbf{SpinSCI-B leads the backbone without MC-IRB by 0.52, 0.41, and 0.39 dB}} on the three benchmarks (Q3).
  • Final DC projection. It contributes +0.03 dB to SpinSCI-B (38.211 → 38.243 dB on mid-scale color; Appendix B.5). Applying the same final DC to the official HiSViT₉ checkpoints raises their PSNR by only 0.04 dB on small-scale gray (37.00 → 37.03) and 0.03 dB on mid-scale color (38.01 → 38.04). With DC on both methods, SpinSCI-B still leads by 0.48 and 0.20 dB, so the margins are not due to the final correction.

W2. Missing ablation / Q3. With only the MC-IRB switch changed, the backbone reaches 36.99, 37.83 and 37.29 dB (Table B.1); over five seeds the gain is 0.41 dB (95% CI [0.226, 0.591] dB). Final DC: see Confounds.


W2. Large-scale result. Equal quality at half the cost is not a tie.\color{blue}{\textbf{Equal quality at half the cost is not a tie.}} SpinSCI-B has the highest average PSNR (SSIM within 0.001 of HiSViT₉). Each wins two of four sequences (SpinSCI-B Hummingbird and Messi, HiSViT₉ Football and Swinger), and SpinSCI-B matches HiSViT₉ at about half the latency and 2.5× less peak memory\color{blue}{\textbf{at about half the latency and 2.5× less peak memory}}, a cost gap the 0.10 dB seed SD cannot close.


W2. Ising and mean-field framing. The framing is not descriptive; the trained model behaves as an Ising system\color{blue}{\textbf{The framing is not descriptive; the trained model behaves as an Ising system}}. Eq. 2 states the energy; the update is its mean-field stationarity condition (Appendix A.5).

  • Minimization. In the first coupled temporal block, magnetization signs agree with 99.9% of the coupling weight, so the reached state nearly minimizes the pairwise energy\color{blue}{\textbf{the reached state nearly minimizes the pairwise energy}}.
  • Convergence. 9 of the 12 coupled temporal blocks are contractions with a unique fixed point\color{blue}{\textbf{9 of the 12 coupled temporal blocks are contractions with a unique fixed point}} (Theorem A.9); the other three are locally contractive in over 99.999% of groups.

The unrolling follows CRF-as-RNN; what is new is the energy, with signed couplings from attention logits and sensing penalties.


W2. Formal analysis and related work. These guarantees hold for every input; final DC cuts the residual from 3.5×10⁻³ to 5.3×10⁻⁸, and Lean machine-checks their core identities (Appendix A.8). Appendix E adds CRF-as-RNN, the Deep Parsing Network, and Hopfield, energy-based, gated and signed attention.


Q2 / W2 Copied baselines. Yes: the masks and test videos are byte-identical\color{blue}{\textbf{Yes: the masks and test videos are byte-identical}} (SHA-256) to the official HiSViT release, on which its official checkpoints reproduce the published scores:

BenchmarkHiSViT₉ reportedHiSViT₉ measuredSpinSCI-B
Small-scale gray37.00 / 0.97837.00 / 0.97737.51 / 0.980
Mid-scale color38.01 / 0.97238.01 / 0.97238.24 / 0.973

The "mask differences" concern PnP-FastDVDNet and EfficientSCI-B, whose official grayscale configurations use other masks; on ours they obtain 32.53 and 36.35 dB (published 32.35 and 36.48 dB). Table C.1 re-runs every small- and mid-scale baseline on our masks; SpinSCI-B stays first.


Q4 / W2 Mechanism claim. The revised manuscript’s new maps show that MC-IRB is organized by the sensing mask\color{blue}{\textbf{The revised manuscript's new maps show that MC-IRB is organized by the sensing mask}} (Appendix B.5, Figure B.2):

  • Exposure state. In temporal block 3 of the color model, couplings are +0.35 between frames sharing a token's mask value and −0.53 otherwise; magnetizations split measured (mi≈−1m_i\approx-1) from unmeasured (mi≈+1m_i\approx+1) tokens, which get 1.93× larger gated updates; the gray model learns the same.
  • Mask transitions. The gates ignore them (AUC 0.49) but separate measured from unmeasured tokens perfectly (AUC 1.00).
  • Edges. In spatial block 8, gated updates follow ground-truth image gradients (Spearman 0.41 color, 0.73 gray).

Q5. KmfK_{\mathrm{mf}}. Table 2 trains Kmf=0,1,3,5K_{\mathrm{mf}}=0, 1, 3, 5 (36.81, 36.98, 37.07, 37.07 dB): accuracy saturates at three steps, so we consider a separate Kmf=2K_{\mathrm{mf}}=2 run unnecessary.

Thank you for the response. The efficiency comparisons address my main concern, and I also appreciate the ablation studies.

I still have some reservations about the novelty of the Ising formulation. I also view the largescale PSNR results as comparable to HiSViT, rather than a significant improvement.

Overall, I'm now leaning toward boederline. I'd like to hear the other reviewers' perspectives before making a decision.

AuthorsOpenReview ↗

Real Gains Measured on Real Hardware

Thank you for the follow-up and for reconsidering the efficiency results.

We understand that ICLR is not a hardware venue, but, as stated in our general response, video SCI is a case where hardware cost must be considered\color{green}{\textbf{video SCI is a case where hardware cost must be considered}}.

On the large-scale benchmark, we agree that PSNR is comparable. However, the gain there is hardware cost at equal quality (Table 6):\color{green}{\text{However, the gain there is hardware cost at equal quality (Table 6):}}

  • \color{green}{\text{21.6}\\%\text{ fewer FLOPs}}
  • \color{green}{\text{49.0}\\%\text{ lower FP32 eager latency}}
  • \color{green}{\text{59.9}\\%\text{ lower peak memory}}

Appendix F measures the hardware cost of every method on a 22nm ASIC implementation, not only on GPUs,\color{green}{\textbf{Appendix F measures the hardware cost of every method on a 22nm ASIC implementation, not only on GPUs,}} given the real-world applications of video SCI\color{green}{\textbf{given the real-world applications of video SCI}}. At large scale, compared with HiSViT$_9$, SpinSCI-B:

  • needs 1.58× less energy (Table F.3)\color{green}{\textbf{needs 1.58× less energy (Table F.3)}}
  • needs 2.1× less on-chip memory (Table F.3)\color{green}{\textbf{needs 2.1× less on-chip memory (Table F.3)}}
  • uses 1.73× less measured GPU energy per frame on the GH200 (Table F.4)\color{green}{\textbf{uses 1.73× less measured GPU energy per frame on the GH200 (Table F.4)}}

We note that costing every method under a common hardware model is not trivial, and we hope this helps your assessment. Thank you again for taking the time to review our work!

Reviewer_KEV3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SpinSCI for video snapshot compressive imaging (SCI). Its core component, the Mask-Conditioned Ising Refinement Block (MC-IRB), constructs signed couplings from attention logits and sensing information, and uses mean-field iterations to generate gates for spatial-temporal feature refinement. The framework also incorporates physics-guided initialization and data consistency. Experiments report competitive reconstruction quality and computational efficiency on simulated and real SCI benchmarks.

Critical strengths

1) The proposed mask-conditioned Ising refinement provides an interesting mechanism for integrating sensing information with attention-based feature learning. 2) The experiments cover grayscale, color, large-scale, and real SCI datasets. The supplementary material includes extensive ablations and repeated-run evaluations that support the effectiveness of the proposed refinement.

Critical weaknesses

-1) The backbone largely follows STFormer's parallel spatial-temporal attention and incorporates efficient attention designs related to HiSViT. The main novelty lies in MC-IRB, whose advantages over conventional message-passing or adaptive gating mechanisms remain insufficiently established.

-2)Unfair efficiency comparisons. Appendix C reveals that SpinSCI uses BF16 compiled inference, whereas the baselines use FP32 eager inference. Therefore, the reported 77–85% latency reduction and substantial memory savings may partly result from different execution settings.

-3) Appendix Table B.6 shows that spatial coupling is disabled in seven of nine blocks, and modifying spatial couplings has negligible effects on PSNR. This raises questions about the necessity of spatial Ising refinement.

-4) SpinSCI improves average PSNR over HiSViT by only 0.04 dB on the large-scale color benchmark, while performing worse on two of four sequences. The claimed superiority is therefore not consistently supported across different scenes.

-5) The method is primarily evaluated using standard binary masks. Appendix B reports substantial performance degradation under changes in the mask distribution and on calibrated masks. The generalization to different sensing operators and real measurement conditions remains unclear.

Questions for authors

Can the authors provide efficiency comparisons under identical precision and compilation settings? What is the benefit of spatial Ising interactions when their removal has negligible impact on PSNR? How does MC-IRB compare with parameter-matched message-passing or adaptive gating modules? How stable are the improvements across different training seeds and large-scale video sequences? Can the authors explain the substantial performance degradation under different mask distributions and calibrated masks?

1 comment
AuthorsOpenReview ↗

matched FP32 efficiency, seed-stable gains, and operator mismatch

W1 / Q3. MC-IRB versus message passing and adaptive gating. Parameter-matched controls under one protocol establish MC-IRB’s advantage\color{blue}{\textbf{Parameter-matched controls under one protocol establish MC-IRB's advantage}} (Table 2):

  • Adaptive gating. A sigmoid gate on the local field (36.84 dB), a conditioned gate without mean-field inference (36.87 dB) and a fixed gate g=1/2g=1/2 (36.78 dB), all at 7.46–7.47M parameters, trail MC-IRB (37.07 dB) by 0.20–0.29 dB.
  • Message passing. Removing the interactions (J=0\mathbf{J}=0), their signs (∣J∣|\mathbf{J}|) or the mean-field steps (Kmf=0K_{\mathrm{mf}}=0) costs 0.14–0.26 dB.
  • Capacity. A 10-block backbone without MC-IRB, with more parameters and FLOPs, is 0.20 dB lower.
  • Seeds. Over five seeds, the gains over the field-only gate, J=0\mathbf{J}=0 and ∣J∣|\mathbf{J}| are 0.26, 0.24 and 0.19 dB, and all paired 95% intervals lie above zero (Table B.2).

The couplings are also interpretable: they separate measured from unmeasured tokens, which receive 1.93× larger gated updates (Appendix B.5).


W2 / Q1. Efficiency under identical settings. We agree that matched settings are essential, and every cross-method comparison in the main paper now uses FP32 eager inference\color{blue}{\textbf{every cross-method comparison in the main paper now uses FP32 eager inference}} on the same GH200, batch size 1, T=8T=8, identical TF32/cuDNN settings (Appendix C). Under these fully identical FP32 eager settings, SpinSCI-B remains the best method\color{blue}{\textbf{Under these fully identical FP32 eager settings, SpinSCI-B remains the best method}} (see the general response): the most accurate on all three benchmarks, with 16–49% lower latency and 60–63% lower peak memory than HiSViT₉ (revised Figure 1, Tables 4–6). Every value is reproducible via the Reproducibility section of the\color{blue}{\textbf{(revised Figure 1, Tables 4–6). Every value is reproducible via the Reproducibility section of the}} \href\href{https&#58;//anonymous.4open.science/r/video-sci-benchmark-7C3C/}{\color{blue}{\textbf{anonymous repository}}}.

SpinSCI-B / HiSViT₉Latency (ms)Memory (MiB)
Small-scale gray131 / 1761,850 / 4,840
Mid-scale color635 / 7577,151 / 19,156
Large-scale color4,169 / 8,1732,695 / 6,715

With BF16 compilation for all six methods (Table C.2), SpinSCI-B also keeps the highest PSNR and is 30% faster and 33% lighter than HiSViT₉; only STFormer-B and EfficientSCI-B are faster, at 0.78 and 1.07 dB lower PSNR.


W3 / Q2. Spatial Ising refinement. Spatial MC-IRB is needed\color{blue}{\textbf{Spatial MC-IRB is needed}}: it adds 0.16 dB with or without temporal MC-IRB, out of the 0.39 dB that full MC-IRB adds (Table 1). In the spatial branch it works through its sensing-conditioned gate and residual MLP rather than pairwise coupling, which is why editing the couplings leaves PSNR unchanged; its gated corrections are nearly as large as the temporal ones (0.219 vs. 0.242). Pairwise coupling matters in the temporal branch, where permuting it costs 1.04 dB\color{blue}{\textbf{Pairwise coupling matters in the temporal branch, where permuting it costs 1.04 dB}}.


W4 / Q4. Large-scale sequences and seed stability. We firmly disagree that this result is marginal (see the general response). SpinSCI-B has the highest average PSNR; it wins Hummingbird and Messi (+0.51, +0.68 dB), HiSViT₉ wins Football and Swinger (+0.57, +0.45 dB), so both reach top quality, but SpinSCI-B is 2.0× faster and uses 2.5× less peak memory\color{blue}{\textbf{SpinSCI-B is 2.0× faster and uses 2.5× less peak memory}}. Five training seeds give 37.07 ± 0.10 dB\color{blue}{\textbf{Five training seeds give 37.07 ± 0.10 dB}}, and the mean gain over the backbone without MC-IRB is 0.41 dB, four times the seed SD, with a paired 95% interval of [0.226, 0.591] dB (Table B.2).


W5 / Q5. Mask distributions and calibrated masks. The degradation is an operator mismatch, which affects every learned method\color{blue}{\textbf{The degradation is an operator mismatch, which affects every learned method}}. Revised Table 3 now separates two cases:

  • New mask realization. On an unseen binary mask, SpinSCI keeps its benchmark-mask 37.07 dB, and removing its mask conditioning costs 0.30–0.38 dB there.
  • New operator family. Each checkpoint performs best on the family it was trained with, as expected of any learned network.

New Appendix B.6 runs every Table 5 method with a miscalibrated mask. Under sub-pixel shifts, SpinSCI-B keeps the highest PSNR among the three strongest methods\color{blue}{\textbf{SpinSCI-B keeps the highest PSNR among the three strongest methods}}; under noise on the mask values, it loses more (34.13 vs. 34.24–34.77 dB at σ = 0.05); STFormer-B and MambaSCI-B fall below 23 dB. No compared paper reports such tests\color{blue}{\textbf{No compared paper reports such tests}}: only STFormer tests new binary realizations of the same mask type. For real data, SpinSCI is trained with the calibrated mask (Appendix C); the 19.04 dB in Appendix B.2 is a deliberate stress test of a binary-trained checkpoint on an unseen operator.

Comments on the paper

1 comment
AuthorsOpenReview ↗

SpinSCI advances the accuracy–cost frontier under one FP32 protocol

Acknowledgment. We thank the reviewers and the AC for their valuable time and feedback. We answer every point, state exactly what was revised, and mark key points in blue. Under one protocol, reproducible from our repository, SpinSCI stays on the accuracy–cost Pareto frontier: in video SCI, a hardware application, it is the most accurate at a far lower hardware cost.

Revised manuscript. The main paper and supplementary are now one PDF. Figure 1 and Tables 4–6 use FP32 eager for every method; Tables 2–3 add the supplementary's design and mask-operator comparisons. Requested results are in the appendix; new Appendix F shows why hardware cost matters and where SpinSCI stands. Changed figures, tables and new sections are in blue.\color{blue}{\textbf{Changed figures, tables and new sections are in blue.}}


Video SCI is a hardware application, so its cost is part of the result.\color{blue}{\textbf{Video SCI is a hardware application, so its cost is part of the result.}} Video SCI is a computational camera: a coded-aperture encoder such as CACTI and a reconstruction network together produce the video, so the compute, latency and memory of the reconstruction decide whether the camera can be built and deployed. Every top method in Tables 4–6 operates at 36–38 dB, where the human eye mostly sees no large difference between them, so what separates them in practice is their hardware cost. On this criterion SpinSCI-B is the best method\color{blue}{\textbf{what separates them in practice is their hardware cost. On this criterion SpinSCI-B is the best method}}: every cheaper method is at least 0.75 dB less accurate.


Accuracy–cost frontier: why SpinSCI’s gains are not marginal.\color{blue}{\textbf{Accuracy–cost frontier: why SpinSCI's gains are not marginal.}} We do not accept that SpinSCI's gains are marginal, and the numbers in Tables 4–6 do not permit that conclusion. A reconstruction method is judged on quality and cost together. Every latency and memory figure below is FP32 eager inference under identical settings for every method (Appendix C). The revised manuscript uses it throughout (Figure 1, Tables 4–6), and every value can be reproduced via the Reproducibility section of the \href\href{https&#58;//anonymous.4open.science/r/video-sci-benchmark-7C3C/}{\color{blue}{\textbf{anonymous repository}}}.

Method (year); gray / mid / largePSNR (dB)Latency (ms)Peak memory (MiB)
PnP-FastDVDNet (2022)32.35 / 34.60 / 34.431,265 / 40,317 / 297,970618 / 2,648 / 1,029
EfficientSCI-B (2023)36.48 / 37.49 / 36.2649 / 211 / 1,5971,355 / 5,207 / 2,046
STFormer-B (2023)36.34 / 37.32 / 36.3882 / 369 / 2,5781,745 / 6,368 / 2,478
MambaSCI-B (2024)— / 35.70 / 30.92— / 974 / 7,645— / 3,809 / 1,587
HiSViT₉ (2025)37.00 / 38.01 / 37.64176 / 757 / 8,1734,840 / 19,156 / 6,715
SpinSCI-B37.51 / 38.24 / 37.68131 / 635 / 4,1691,850 / 7,151 / 2,695

Before SpinSCI, accuracy was bought with compute.\color{blue}{\textbf{Before SpinSCI, accuracy was bought with compute.}} From EfficientSCI-B (2023) to HiSViT₉ (2025), average PSNR rose by 0.52, 0.52 and 1.38 dB on small-scale gray, mid-scale color and large-scale color, and the price was 3.6×, 3.6× and 5.1× the latency and 3.3–3.7× the peak memory.

SpinSCI-B breaks this trend.\color{blue}{\textbf{SpinSCI-B breaks this trend.}} It is more accurate than HiSViT₉ on all three benchmarks (+0.51, +0.23 and +0.04 dB) and, at the same time, cheaper on every cost axis: 2.5–2.7× less peak memory and 1.2–2.0× lower latency, with 21% fewer FLOPs and 17% fewer parameters.

  • Every method that is cheaper than SpinSCI-B is at least 0.75 dB less accurate, so SpinSCI-B lies beyond the previous Pareto front, not on it (Figure 1).
  • Large-scale color is the clearest case. SpinSCI-B reaches HiSViT₉'s top quality at about half the latency and 2.5× less peak memory. Seed variation (SD 0.10 dB) cannot close this gap, because these costs are fixed properties of the architecture.

New Appendix F estimates this hardware cost directly.\color{blue}{\textbf{New Appendix F estimates this hardware cost directly.}} It applies two cost models identically to every method: one built only from published 45 nm per-operation energies (Horowitz, 2014), and one calibrated on a place-and-route implementation of an integer transformer accelerator at 22 nm, with published DRAM constants. SpinSCI-B needs 1.46–2.10× (45 nm) and 1.50–1.58× (22 nm) less energy per reconstructed frame\color{blue}{\textbf{1.46–2.10× (45 nm) and 1.50–1.58× (22 nm) less energy per reconstructed frame}} than HiSViT₉, and 1.41–2.16× less time. Measured on the GH200 used for Tables 4–6, SpinSCI-B uses 1.16–1.73× less GPU energy per frame than HiSViT₉ at FP32, and 1.25–2.08× less at FP16 (Table F.4), where neither network changes PSNR by more than 0.01 dB. Without DRAM, under the 22 nm model, SpinSCI-B fits on a 164 mm² die at small and large scale, and with 256-pixel crops at mid scale, while HiSViT₉ fits at no scale.

Across all compared methods, SpinSCI-B lies on the PSNR–energy Pareto front of every benchmark under both models\color{blue}{\textbf{Across all compared methods, SpinSCI-B lies on the PSNR–energy Pareto front of every benchmark under both models}}, and we ask the reviewers and the AC to assess SpinSCI on this accuracy–cost frontier, which it clearly advances with top accuracy at a substantially smaller compute budget, measured under one protocol for every method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.