acceptodds
Under review as a conference paper at ICLR 2027

Long-Range Latent Modeling for Quanta Video Reconstruction and Transmission

Abstract

Quanta image sensors (QIS) capture dynamic scenes through high-rate binary photon measurements, offering fine temporal resolution but extremely sparse photon evidence in each exposure. High-quality video reconstruction therefore requires jointly exploiting photon evidence over extended observation intervals. Existing QIS reconstruction methods typically use bounded temporal contexts to generate frames at predefined timestamps, fixing the reconstruction as a discrete image sequence before compression and transmission. We propose a temporal latent representation framework for long QIS sequences that models scene evolution throughout the observation interval as queryable latent trajectories. A hierarchical long-range encoder jointly aggregates sparse photon evidence and produces multi-scale temporal latent anchors, providing discrete support for latent trajectories at complementary resolutions. Given an arbitrary temporal coordinate within the encoded interval, a temporal query module constructs the corresponding latent state through interpolation between neighboring anchors and learned residual refinement, and a shared decoder maps this state to a video frame. Scene evolution is thus retained in a reusable latent representation, allowing receivers to select output timestamps after encoding and transmission without revisiting the raw measurements or rerunning the encoder. The same multi-scale latent state supports compact transmission through cross-scale predictive compression, unifying long-range reconstruction, state compression, and on-demand temporal querying within a shared representation. Experiments across five photon levels show that our method outperforms the evaluated QIS reconstruction baselines and effectively exploits photon evidence from longer observation intervals to improve reconstruction quality. For a 4096-frame binary QIS sequence, the compressed latent state occupies only 4.628 MiB, compared with 64.000 MiB for the bit-packed raw measurements, while retaining arbitrary temporal query capability after transmission.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Quanta image sensors, video reconstruction, long-range temporal modeling, time-queryable latent representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4NHEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a temporal latent representation for long binary quanta image sensor (QIS) sequences. Instead of reconstructing frames at fixed timestamps prior to compression, a hierarchical encoder aggregates binary photon streams into multi-scale temporal latent anchors. At the receiver side, a temporal query module interpolates neighboring anchors with learned residual refinement to decode frames at arbitrary timestamps, while cross-scale predictive coding compresses the latent state for transmission. The method is evaluated across five simulated photon levels and on real SPAD sensor captures, alongside analyses of payload size and encoding/query runtime.

Critical strengths
  1. Encoding the binary measurement stream once into a queryable latent state is a very sensible design for QIS transmission. Decoupling sender-side photon aggregation from receiver-side timestamp selection avoids committing to a fixed frame rate upfront and fits well with the high-speed nature of binary photon streams.

  2. The evaluation covers five simulated photon-rate settings, α ∈ {0.05, 0.1, 0.2, 0.3, 0.5}, as well as real QIS recordings. I appreciate the reported serialized latent payload of 4.628 MiB for a 4096-frame sequence and the separation of sender encoding, receiver initialization, and per-query costs, including the explicit qualification of Bit2Bit’s sample-specific optimization.

Critical weaknesses
  1. Key details regarding how the baselines were adapted and evaluated are missing, which makes the ~3 dB PSNR gain over QUIVEr hard to verify. In the original QUIVEr paper, simulated experiments primarily use 3-bit inputs under a different noise setup, whereas here the input is 1-bit Bernoulli data with hot-pixel and dark-count defects. It is unclear whether QUIVEr and other baselines were retrained from scratch on the new 200-video training set or tested using pretrained weights, and how their temporal windows, intensity scaling, and defect handling were adjusted. Also, since the 200/13 dataset split differs from QUIVEr’s original 249/31 split, we need to be sure that none of the 13 test videos were seen during baseline pretraining. Finally, it is not fully clear whether "averaging over all reconstructed frames" evaluates all methods at the exact same target timestamps and effective temporal windows.

  2. The advantage of querying in latent space is not cleanly separated from simply having a stronger base reconstructor with a longer temporal receptive field. A very natural baseline is missing: running a strong QIS reconstruction method at sparse keyframes, compressing them, and applying an off-the-shelf video frame interpolator at the receiver for arbitrary timestamps. Without comparing against "sparse reconstruction + frame interpolation" at matched bitrates and query times, it is hard to tell how much gain actually comes from the continuous latent query mechanism itself.

  3. Appendix F explicitly states that the timing examples compare workflows with different native output counts and include sample-specific optimization for Bit2Bit; these results should therefore be interpreted within that scope. The submitted source package also provides substantial architecture and training configurations. The remaining reproducibility issue is their mapping to the reported experiments: for example, the reconstruction configuration uses batch size 4, a warmup/plateau schedule, and loss weights that differ from Section 4.1, while the README describes independently trained reconstruction and compressed variants initialized through a linear-anchor run. Please identify the exact configuration and checkpoint lineage for each reported model and reconcile these differences with the manuscript.

Questions for authors
  1. Were the baseline methods (especially QUIVEr) retrained on your 1-bit 200-video split, or evaluated using released checkpoints? How were bit-depth differences, photon levels, intensity scaling, and sensor defects handled for each baseline?

  2. If any pretrained weights were used for the baselines, can you confirm there is no overlap between their original training sets and your 13 test videos (e.g., by listing the test video IDs)?

  3. In Table 1, are all methods evaluated at the exact same timestamps and ground-truth frames, and how does this align with the different native output frame counts noted in the runtime comparison?

  4. How would your method compare against sparse reconstruction followed by video frame interpolation under matched payloads and query timestamps? Please also identify which submitted configuration produced each reported model, including its initialization lineage, interpolation variant, and final 13-video evaluation procedure.

Reviewer_ibbLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a framework for reconstructing and transmitting videos from sparse binary measurements captured by quanta image sensors. A hierarchical encoder aggregates long-range photon evidence into multi-scale latent anchors, while temporal interpolation and learned refinement enable frame reconstruction at arbitrary timestamps within the encoded interval. Cross-scale predictive compression preserves this querying capability after transmission. Experiments show better reconstruction quality than the evaluated baselines across five photon levels.

Critical strengths
  1. The formulation is practically useful. It encodes raw photon measurements into a compressible latent representation, allowing receivers to choose reconstruction timestamps after transmission.
  2. The reported results are strong. Under the authors’ evaluation protocol, the method outperforms the compared baselines across all five photon levels, with PSNR gains of approximately 2.2–3.1 dB over the strongest external baseline.
  3. The payload reduction is substantial. In the reported experiment, the transmitted data decreases from 64 MiB to 4.628 MiB, while retaining repeated queries at arbitrary timestamps.
Critical weaknesses
  1. The data split needs clarification. Results are reported on 13 validation videos, but their relationship to QUIVEr’s original 31-video test set is not explained. Results on the full original test split are also missing.
  2. Insufficient validation of the benefits of post-transmission temporal querying. The paper highlights receiver-selected timestamps after transmission as a key advantage and demonstrates its feasibility in Figure 2. However, Table 3 compares reconstruction followed by H.265 compression and access to existing frames, without including receiver-side frame interpolation. Consequently, the practical benefits over a complete reconstruction–compression–interpolation pipeline remain unclear. A direct comparison using the same QIS input and receiver-selected query timestamps, particularly timestamps not explicitly represented by stored frames in the transmitted video, would help establish these benefits in terms of final reconstruction quality, transmitted payload, and total processing time.
Questions for authors
  1. How are the 13 validation videos related to QUIVEr’s original 31-video test set, and can the authors report results on the full test split?
  2. How does the proposed method compare with a complete reconstruction–compression–interpolation pipeline in terms of reconstruction quality, transmitted payload, and total processing time when querying arbitrary timestamps?
1 comment
Reviewer_ibbLOpenReview ↗

Response to Author

The authors have satisfactorily addressed all of my concerns. Therefore, I will maintain my original score.

Reviewer_iV7VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a long-range latent representation framework for Quanta Image Sensor (QIS) video reconstruction and transmission. Unlike conventional frame-centric approaches, the proposed method encodes long sequences of sparse binary photon measurements into a compact, reusable latent state, enabling video reconstruction at arbitrary timestamps without reprocessing the original measurements. The framework integrates three main components: (1) a hierarchical long-range encoder that aggregates temporal information into multi-scale latent anchors; (2) a temporal query module combining linear interpolation and learned residual refinement to reconstruct frames at requested timestamps; and (3) a cross-scale predictive compression scheme for efficient latent-state transmission. The method is evaluated on simulated QIS data across five photon levels and qualitative real-world QIS recordings. Experimental results demonstrate improvements in PSNR, SSIM, and LPIPS over the evaluated reconstruction baselines, including QBP, QUIVEr, gQIR, and Bit2Bit. Ablation studies examine the effects of temporal residual refinement, compression, and input sequence length.

Critical strengths

Unified, flexible latent representation: The paper moves QIS reconstruction away from the conventional frame-centric approach toward a reusable, time-queryable latent representation. It brings long-range photon aggregation, temporal querying, and latent-state compression together in a single framework, so the receiver can reconstruct frames at any timestamp it chooses without reprocessing the original measurements. This is a meaningful contribution because it improves reconstruction quality and transmission flexibility at the same time, rather than treating them as separate problems.

Consistent empirical gains & effective latent compression: The method consistently outperforms several established QIS baselines across five photon levels. Quantitative results in PSNR, SSIM, and LPIPS support this, as do qualitative results on real QIS recordings.These results are encouraging evidence that the framework works in practice, especially for photon-limited video reconstruction.

Critical weaknesses

insufficient ablation studies: The ablation studies do not adequately isolate the individual contributions of the hierarchical encoder, the separate latent scales, and the cross-scale predictive compression. As a result, it is hard to tell which components drive the reported gains, and therefore hard to judge how significant the work is methodologically for the broader machine learning community.

Limitations: The experimental comparisons do not always use matched output conditions. In addition, arbitrary-time querying is a central contribution, yet the evaluation includes little dedicated analysis of temporal reconstruction accuracy under complex motion, occlusions, or unseen query timestamps.

Questions for authors

Could you please clarify Methodological Novelty: clarify the key methodological innovations beyond integrating existing hierarchical spatiotemporal encoding, latent interpolation, residual refinement, and predictive compression techniques? In particular, which components introduce fundamentally new learning mechanisms, and how does the proposed representation differ from existing continuous-time or implicit neural video representations?

1 comment
AuthorsOpenReview ↗

Gentle Follow-up on Our Responses

Dear Reviewer,

As the reviewer–author discussion period is nearing its end, we would like to briefly follow up on our responses submitted on November 14 and the revised manuscript.

We have provided additional clarifications and experiments to address your main concerns, and we would greatly appreciate your feedback on whether these additions have adequately resolved them. If any questions remain, we would be happy to provide further clarification before the discussion period closes.

If you find that our responses and additional evidence have addressed your main concerns, we would be grateful if you could reconsider your overall rating in light of the revised manuscript.

Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.