acceptodds
Withdrawn from ICLR 2027

MCRRL: Recovering Hidden Screen Content from Diffuse Wall Reflections

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We ask: when a camera cannot directly capture a screen, can the screen content be recovered using only an image of the light cast by the screen onto a wall? With the display outside the camera view, diffuse reflection spreads and mixes screen light, weakening local correspondence and attenuating the directional evidence needed for small text, thin lines, and interface structure. Direct residual regression can also average foreground with background or introduce texture where the measurement is ambiguous. We therefore propose Measurement-Conditioned Raster Representation Learning (MCRRL) to learn structural correspondence between weak wall-light evidence and the hidden screen. Measurement-Conditioned Proximal Raster Fusion (MPRF) organizes directional cues through an orientation-masked dictionary and finite proximal updates, learning structural correspondence across scales. Evidence-Calibrated Digital Raster Composition Reconstruction (DRCR) uses the resulting decoder features to form latent foreground, background, and opacity fields, then uses an evidence-conditioned gate and clipping to refine their composition. Together, the two stages organize the surviving screen cues and control how they contribute to the reconstruction. Experiments on four real paired datasets cover interfaces, charts, dense glyph patterns, and browser layouts. MCRRL outperforms the compared reconstruction methods in PSNR, SSIM, and LPIPS across all four datasets. The recovered images reveal brightness distributions, regional layouts, principal contours, and portions of the displayed content. These results establish the preliminary feasibility of recovering hidden screen information from diffuse wall light with an ordinary RGB camera.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hidden-screen reconstruction, Passive non-line-of-sight imaging, Optical side channel, Diffuse reflection, Computational imaging

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3Rd5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies hidden-screen reconstruction from indirect wall observations captured by an ordinary camera. The authors propose Measurement-Conditioned Raster Representation Learning (MCRRL), a two-stage approach consisting of 1) MPRF, which organizes observation evidence using measurement-conditioned directional primitives and unrolled sparse inference and 2) DRCR, which reconstructs screen content through a foreground-background-opacity decomposition followed by a confidence-gated and clipped residual refinement.

The proposed method is evaluated on four real paired datasets (Screen, Charts, Password2, and Browser). Quantitative comparisons against a range of image restoration baselines show improvements in PSNR, SSIM, and LPIPS. Qualitative examples show successful recovery of screen layouts, chart structures, glyph organization, and browser-page structures from indirect wall observations.

Critical strengths
  1. Coherent architecture tailored to screen structure

The paper presents a clear architectural design. MPRF is specifically designed to organize weak directional evidence through horizontal, vertical, diagonal, and free-form raster primitives. DRCR introduces a structured reconstruction mechanism based on foreground-background composition. The proposed architecture is directly aligned with the goal, which is to recover structured screen content.

  1. Strong empirical results and targeted ablations

The paper reports consistent improvements across multiple datasets and metrics (PSNR, SSIM, LPIPS). The ablation studies are generally well designed and evaluate key components of the proposed method. These experiments provide reasonably strong evidence that both MPRF and DRCR modules contribute meaningfully to the reported performance gains.

Critical weaknesses
  1. Related work and novelty positioning

A closely related recent work, Bounded Hidden-Screen Reconstruction from Diffuse Wall Reflections via Survivability-Guided Representation Learning, appears to be missing from the related-work discussion. This work considers essentially the same task of reconstructing an out-of-view screen from a single RGB observation of diffuse wall reflections. Given the substantial similarity in problem formulation and sensing setting, the authors should discuss this work and clarify how the proposed MCRRL differs from it, both methodologically and in terms of its contribution.

  1. Generalization beyond the training distribution is not demonstrated

The authors train a separate model for each dataset and evaluates only on fixed train/test splits from the same dataset. Thus, the paper demonstrates performance within individual content distributions, but does not establish whether the learned reconstruction mechanism generalizes beyond the training data. It remains unclear whether MCRRL learns a general hidden-screen reconstruction mechanism or a prior specialized to the data-collection setup and content distribution. This substantially limits confidence in the broader applicability of the reported results.

  1. Some aspects of the sensing configuration remain insufficiently explained

The paper presents multiple acquisition configurations and evaluates on four real paired datasets, but some elements of the sensing setup are not clearly connected to the rest of the paper. For example, Figure 1 includes an explicit occlusion-modulator/shadow configuration, yet the subsequent method description, experiments, and discussion consistently describe the measurements as wall observations or diffuse wall reflections, with little discussion of the role of the occlusion element itself. Similarly, the relationship between the configurations presented in the main paper and supplementary material is not always clear.

Questions for authors
  1. Dataset provenance and acquisition

Please clarify the provenance of all four datasets. Were all datasets collected by the authors? Are the datasets publicly available? Were any datasets, acquisition protocols, or preprocessing pipelines adopted from or related to prior hidden-screen reconstruction work?

  1. Positioning relative to recent hidden-screen reconstruction literature

A closely related published work, Bounded Hidden-Screen Reconstruction from Diffuse Wall Reflections via Survivability-Guided Representation Learning, studies hidden-screen reconstruction from diffuse wall reflections using an RGB camera. It would be helpful to clarify how the sensing setup, datasets, and reconstruction framework differ from that work. In particular, the second capture configuration shown in the supplementary material appears visually similar to the diffuse-wall-reflection setup used there.

Reviewer_vZY8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates passive non-line-of-sight (NLOS) screen content recovery: reconstructing what is displayed on a hidden digital screen using only a single ordinary RGB camera image of its diffuse reflection on a nearby wall (with an occlusion modulator/shadow edge in the scene). To address the severe spatial mixing and directional attenuation of diffuse reflections, the authors propose Measurement-Conditioned Raster Representation Learning (MCRRL), a two-stage restoration architecture built on a multiscale U-Net backbone:

  1. Measurement-Conditioned Proximal Raster Fusion (MPRF): Injects resized wall observations across encoder scales and uses an orientation-masked dictionary (horizontal, vertical, diagonal, and free-form supports) with a 3-step unrolled proximal-gradient solver, coarse-to-fine consensus, and an energy-difference gate to extract stroke and boundary primitives.
  2. Evidence-Calibrated Digital Raster Composition Reconstruction (DRCR): Models the output via latent foreground, background, and opacity fields (alpha-matting style convex composition) combined with a confidence-gated, clipped residual update and a parallel gradient-conditioned RGB correction head.

The method is validated on four real-world paired wall-spot/screen-target datasets (Screen, Charts, Password2, and Browser) collected across two physical scene setups, comparing against 20 baselines (including NLOS reconstruction and general image restoration networks) and evaluating robustness under synthetic image-space perturbations.

Critical strengths
  • Well-Motivated Domain Priors for Digital Screen Reconstruction: Unlike natural scenes, digital displays consist of structured horizontal/vertical layouts, sharp high-contrast edges, and distinct foreground-background layers (e.g., text glyphs over uniform backgrounds). Embedding these inductive biases via orientation-masked sparse coding (MPRF) and bounded alpha-compositing (DRCR) is intuitive and well-aligned with the target domain, yielding consistent quantitative and qualitative gains (Table 2, Table 3) over 20 general-purpose restoration and NLOS baselines.
  • Comprehensive Empirical Benchmarking and Ablations: The paper evaluates across four diverse screen content categories (interfaces, charts, 4-character password strings, and web browsers) and provides thorough module deletions, same-interface replacements, and internal component ablations (Table 3, Tables S5–S8) that clearly justify the contribution of both MPRF and DRCR.
Critical weaknesses
  • Fundamental Disconnect Between Non-Local Diffuse Light Transport and Pixel-Aligned 2D Restoration Heuristics: Although Eq. (1) formulates the problem via an inverse light-transport operator x=Γ(y)+nx = \Gamma(y) + n, the proposed architecture does not actually model or invert non-local light transport. Because diffuse Lambertian reflections globally integrate light from the entire display onto every wall point (modulated only by the occluder's shadow), pixel (i,j)(i, j) in the camera's wall observation xx has no direct spatial correspondence to pixel (i,j)(i, j) on the hidden screen yy. Yet MCRRL repeatedly treats xx and yy as spatially aligned images—computing pixel-wise differences ∣F−R∣|F - R| against resized wall observations in the encoder (Eq. 2), applying local horizontal/vertical raster stroke kernels in the encoder before the bottleneck globally unmixes the scene (where local wall edges correspond to the occluder's shadow rather than screen text strokes), injecting raw wall gradients grad(x)\text{grad}(x) into the final decoder readout (Figure 2, Appendix B.4), and formulating the final screen prediction as an additive residual over the raw wall image itself ( Eq. 14). Moreover, the constituent blocks ( Gregor & LeCun, 2010; Porter & Duff, 1984; Xu et al., 2017) are combinations of standard image-space restoration primitives, and the pipeline is heavily over-engineered without clear theoretical justification for why all components are simultaneously necessary.
  • Lack of Clarity in Key Methodological Formulations and Experimental Validation:
    1. Ambiguous Operator & Dimension Definitions in MPRF: In Eqs. (3)–(5), the exact spatial kernel size, atom count aa, channel dimensions of DrawD^{\text{raw}}, and how the adjoint DTD^T is implemented (weight-tied transposed convolution vs. separate learned projection) are omitted from both the main text and Appendix B.2.
    2. Unconstrained Latent Layer Identifiability in DRCR: DRCR claims to decompose foreground ff, background bb, and opacity α\alpha (Eq. 10) alongside a "calibrated confidence" qq (Eq. 12) using only final RGB supervision—without trimaps, layer supervision, or sparsity/entropy priors on α\alpha—and then adds an unconstrained parallel residual rfr_f (Eq. 14). It is unclear what prevents degenerate solutions where rfr_f or r0r_0 absorbs the reconstruction while f,b,αf, b, \alpha collapse to arbitrary smooth fields, and no visualizations of the learned α,f,b,q\alpha, f, b, q maps are provided to verify meaningful layer decomposition.
    3. Generalization, Dataset Split Leakage Concerns, and Fixed-Geometry Overfitting: A separate 13.7M-parameter model is trained independently for each dataset across only two fixed physical capture setups (Figure 1, Figure S1) where a specific T-shaped occluder/modulator casts a fixed shadow pattern on the wall. The paper does not specify how the train/test splits were partitioned (e.g., whether frames from the same desktop wallpaper, UI session, or chart template appear in both train and test sets). Visual inspection of Screen test outputs (e.g., the Windows Vista wallpaper and dialogs in Figures 3, 5, S21–S30) suggests strong dataset prior memorization rather than true inverse light-transport generalization to unseen physical rooms or arbitrary screen content.
    4. Simulated Image-Space Perturbations Instead of Real Physical Robustness Tests: In Section 4.4 (Figure 5) and Appendix A.4/C, robustness to viewing distance (3.5m–5m), display brightness (500–590 nit), view angle, and wall material (matte, plaster, eggshell) is evaluated by applying deterministic 2D image-space operators to the captured wall images rather than physically moving the screen/camera or changing the wall surface. Calling these "Controlled Wall-Response/Distance/View-Angle Variations" in the main text is somewhat misleading, as 2D image warps and intensity scalings do not capture the true 3D radiometric and geometric light-transport changes of an NLOS scene.
Questions for authors
  1. Clarification on Dataset Collection & Train/Test Splitting: How were the source images in Screen, Charts, Password2, and Browser generated and split between training and testing? Specifically, in Screen and Browser, are the test targets drawn from completely held-out websites/applications/wallpapers, or are they random splits of captured frames that share templates/backgrounds with the training set?
  2. Clarification on Evaluation Target Resizing Discrepancies (Appendix A.3): Appendix A.3 notes that "The Screen architectural comparison retains this clean reference [Lanczos-resized targets], whereas the Browser architectural comparison uses bicubic-resized targets." Why were different interpolation schemes used for the ground-truth targets across different tables (leading to slight metric discrepancies for Full MCRRL on Browser between Table 2 [18.258 dB] and Table 3 [18.353 dB])?
  3. Clarification on Physical Generalization: Can a single MCRRL model trained jointly on all four datasets (or trained on Scene 1 and evaluated zero-shot or fine-tuned on Scene 2) still recover meaningful structure, or does the network strictly require per-domain, per-geometry training?
Reviewer_tUjxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the recovery of screen content from diffuse wall reflections captured by an ordinary RGB camera. The authors propose Measurement-Conditioned Raster Representation Learning (MCRRL), consisting primarily of Measurement-Conditioned Proximal Raster Fusion (MPRF) and Evidence-Calibrated Digital Raster Composition Reconstruction (DRCR). MPRF aims to organize weak directional evidence using an observation-conditioned sparse dictionary and unrolled proximal updates, while DRCR models the reconstructed image as a foreground-background composition controlled by a learned spatial gate. Experiments are conducted on four paired datasets covering interfaces, charts, dense glyph patterns, and browser layouts. The proposed method outperforms a range of NLOS reconstruction and general image-restoration baselines in PSNR, SSIM, and LPIPS.

Critical strengths
  1. Interesting problem definition. Recovering screen structure from diffuse wall illumination is a challenging inverse problem with potential relevance to non-line-of-sight imaging and optical side-channel analysis.
  2. Structured architectural design. The decomposition into structural evidence aggregation and foreground-background composition is well motivated. MPRF and DRCR provide a more problem-specific inductive bias than a generic image-restoration network.
Critical weaknesses
  1. The manuscript explicitly presents x=Γ(y)+nx=\Gamma(y)+n as a phenomenological observation model and does not claim to estimate the transport operator Γ\Gamma. High PSNR, SSIM, and LPIPS establish predictive fidelity under the tested distribution, but they do not by themselves establish that the hidden screen content is physically identifiable from a single diffuse observation. Experiments involving content-held-out splits, ambiguous observation-target pairs, target permutation controls, or independent forward-consistency checks would help clarify this issue. 2.The remaining issue concerns the physical independence of the train–test partitions. The paper does not clearly specify whether the partitions are disjoint with respect to physical scene, wall material, display device, camera geometry, or acquisition session. Consequently, whether the current description does not rule out scene-specific memorization or other forms of physical overlap between training and test data. The reported results therefore establish performance on the tested configurations, but do not yet demonstrate transfer to a new room, wall, display, camera, or geometric setup. More real world generalization experiment is important.
  2. The term “evidence-calibrated” may be stronger than what is demonstrated by the current analysis. The manuscript defines qq as a learned scalar spatial confidence used for adaptive gating, but it does not provide a reliability diagram, expected calibration error, confidence–error correlation, or another quantitative calibration analysis. If qq is intended only as an internal gating weight, the terminology should be clarified. If it is intended to represent a calibrated confidence estimate, additional calibration experiments are needed.
Questions for authors

The central unresolved issue is whether the method performs genuine measurement-driven recovery or distribution-dependent prediction. Although the manuscript reports substantial training and test sets and evaluates independently trained models, the experiments do not establish that the train–test partitions are physically disjoint or that the predictions remain reliable when screen content, scene configuration, wall material, display device, and imaging geometry are substantially changed. Consequently, the current results demonstrate in-distribution predictive performance, but provide limited evidence for physical identifiability and cross-scene generalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.