- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks into a passive optical side‑channel issue with clear information‑security implications. The goal is to reconstruct what’s shown on a blocked‑off screen, just by taking photos of diffuse reflections on a wall using an ordinary camera, when you can’t see the screen face‑on. The authors put forward Coalesced Evidential Representation Learning (CERL). It breaks the reconstruction pipeline down into three complementary modules. OERL pulls together optical clues from wall reflections, covering things like brightness, relative variation, boundaries, cross‑scale differences and local contrast. ISTL uses anchor‑relative and bounded state updates to stop states drifting during step‑by‑step reconstruction. CCM gets more consistent final outputs by soft‑voting and checking agreement among multiple local correction candidates. The paper makes it clear that only the observation y is used during inference; the target x and valid mask m are only for training.
Experiments run on five dataset groups: Screen, Figure, Game, OShome and Photo. Evaluations use LPIPS, SSIM and PSNR. In Table 1, CERL scores the best reported point estimates across all 15 dataset‑metric combinations. The paper also includes module‑wise ablation tests, plus extra analyses covering distance, brightness, noise, wall texture, occlusion, cropping, camera position and camera shake.
Critical strengths
This study targets a real‑world passive optical side‑channel scenario with clear privacy and security risks — you use an off‑the‑shelf camera to recover hidden screen content from diffuse wall reflections. This task is different from standard image restoration work. The authors also break down task difficulties reasonably. Design choices fit closely with problem motivations. OERL doesn’t just blindly concatenate features; it explicitly builds multiple types of evidence pulled from observations. ISTL runs updates anchored to reference points. CCM uses candidate voting and reduces disagreement to fix inconsistent local predictions. Plenty of existing restoration methods are compared in main tables, covering five different content types. CERL achieves top point estimates across all 15 dataset‑metric pairs in Table 1. Fine‑grained ablation experiments are done for OERL, ISTL and CCM. Tests over distance, noise level, brightness and partial observations show authors understand how heavily performance depends on real‑world physical conditions.
Critical weaknesses
- Current evaluation metrics don’t really reflect what “Reading Screens” is supposed to measure. Main experiments rely only on LPIPS/SSIM/PSNR. Meanwhile, dataset descriptions mention labels contain web pages, tables, UI elements and even password‑style characters. But the conclusion cautiously admits the model mostly reconstructs rough brightness patterns, region layouts and main outlines. It would help a lot to add metrics like OCR accuracy, character recognition rates or UI/text region detection, or at least some downstream content‑recognition metric.
- Some robustness test claims overstate what the perturbation experiments actually show. Most notably the camera‑jitter test: simulated pixel shifts get framed as robustness against motion blur and camera shake. Similarly, for brightness changes, noise injection and wall surfaces like marble or textured fiber walls, authors need to make it crystal clear which test cases use real‑world captured footage and which are digitally altered.
- The main text only lists Adam optimizer, learning rate and batch size 16, stating other hyperparameters take default values. But readers still need to know training epochs / total iterations, learning‑rate scheduling, augmentation strategies, sample counts per dataset, model size and FLOPs. Also, the valid mask affects multiple loss terms and evaluation calculations, yet how this mask gets built is never explained clearly.
Questions for authors
- How many training / validation / test samples are there for each of the five datasets?
- For the distance‑robustness experiment: is CERL trained only on data captured at 1.0 m and directly tested on footage from 0.5–4.0 m? Or does training data already include multiple different distances? This distinction really changes how we interpret distance robustness.
- For the camera‑position movement experiment in C.11: did authors physically move the camera and record new footage, or just shift pixels digitally? In C.12, given they used synthetic image shifts, why do they interpret those results as motion blur resistance? Could you fix either the experiments or the wording?
- Could you share concrete architectures for modules such as , , , and ? Also please explain how valid masks are constructed and give the exact implementation for inside loss .
- Since your work focuses on recovering screen content and datasets include UI, text and password‑like material, can you add OCR / character‑level or downstream recognition metrics? These would demonstrate real‑world security impacts far better than PSNR or SSIM.
5 comments
Response to TfSN(2/3)
- The revision states the procedure of every test. The distance evaluation follows the protocol of Section 4.4, with the setup shown in Figures 4 and S18. On your second question, the training data contain a single camera position. CERL is trained on the dataset captures, and Figure 5 is computed with the checkpoint that produces Table 1, with fixed parameters, so the model is tested directly on 0.5 to 4.0 m. The tests of C.4 to C.8, C.11 and C.12 are applied in image space to the Screen test captures. On your third question, C.11 translates the image horizontally by up to ±12% of its width and C.12 applies a synthesized jitter of up to 8 pixels, and the revision titles both as image-space tests and describes them as translation and jitter. The table lists these tests, and each appendix section opens with its procedure.
| Test | Procedure |
|---|---|
| Brightness (C.4) | Image space. Input brightness scaled from 24% below to 24% above nominal |
| Noise (C.5, Table 6) | Image space. Additive Gaussian noise at 20 to 40 dB SNR |
| Wall appearance (C.6) | Image space. Marble-like and fiber-like surface patterns of graded strength applied to the test captures |
| Local occlusion (C.7) | Image space. One cell of a 3×3 grid masked |
| Cropping (C.8) | Image space. Progressive right-side crop |
| Horizontal translation (C.11) | Image space. Horizontal translation up to ±12% of the image width, bicubic interpolation |
| Jitter (C.12) | Image space. Synthesized displacement with a radius of 0 to 8 pixels |
- Training details and sample counts are collected here and in Appendix A.3.
| Item | Value |
|---|---|
| Captures, training/validation/test | Screen 2008/251/251, Figure 2000/250/250, Game 2000/250/250, OShome 6400/800/800, Photo 4800/600/600 |
| Split | all captures of the same content are in the same subset |
| Input and output size | 384 × 512, aspect ratio preserved with padding, with deterministic preprocessing only |
| Optimizer | Adam, learning rate fixed at 1e-4 throughout training, batch size 16, seed 42 |
| Training length | up to 60 epochs with early stopping |
| Model size | 2.45 M parameters |
| Latency and memory | 108 ms per image and 0.92 GB peak GPU memory on an RTX 3080 at 1 × 3 × 384 × 512, FP32, median of 10 timed runs after warm-up |
| FLOPs at 384 × 512 (torch.profiler) | CERL 129.3 G, OKNet 211.6 G, IRNeXt 661.2 G, SFNet 721.4 G, ConvIR 748.3 G |
- The valid mask is the rectangle occupied by the target on the 384 × 512 canvas after aspect-preserving resizing, 1 inside and 0 on the padding. Screen targets fill the canvas, the other datasets average 81 to 82% coverage, and every method is evaluated inside the same rectangle. L5 uses LPIPS with a fixed AlexNet backbone on the valid crop resized to 224 × 224.
Response to TfSN(3/3)
- The modules ψ, ϕ, η, χ and o have the following structure, with hidden width 48, and this table is added to Appendix B.
| Symbol | Role | Layers |
|---|---|---|
| ψ (OERL) | cue projection and feature correction | 1×1 conv C→48 on the feature, 3×3 conv 5→48 on the five cues, 7×7 depthwise and 3×3 dilated depthwise mixing, a two-layer 1×1 confidence branch 48→24→1 with sigmoid, 1×1 conv 48→C followed by tanh |
| ϕ (ISTL) | state transport | 1×1 conv 3C→48 on the current state, the reference state and their difference, 3×3 conv 5→48 on the cues, 5×5, 1×5 and 5×1 depthwise mixing, 1×1 conv 48→C, a two-layer 1×1 spatial gate |
| η (CCM) | candidate map | 1×1 projection 24→48, parallel 5×5 depthwise and 3×3 dilated depthwise mixing, 1×1 projection to 12 channels forming four RGB candidates, 1×1 vote head with softmax over candidates |
| χ | observation descriptor | two fully connected layers 10→64→64 on the mean and standard deviation of RGB, gray level and Sobel response |
| o | RGB readout | 3×3 conv 48→3, residual to the input |
OERL acts at the bottleneck and the first decoder level (C = 192 and 96) and ISTL at three levels (C = 192, 96 and 48). Mixing layers use SiLU with global response normalization, and the three CCM steps share one set of parameters.
- The abstract and the introduction now state what is recovered at the level these answers measure, which is layout, screen identity and large text. The abstract ends with the character error rates of point 1, and Appendix A.2 gives the character heights.
Upon acceptance we will release the datasets, the random-string test set with its wall photographs and outputs, the split files, the trained weights and the code. We hope the character recognition, the text detection, the identification among candidate screens in two rooms and the stated procedure of every robustness test meet the requests at the center of your review, and we would be grateful if you would consider raising your score.
Response to TfSN(1/3)
Thank you for the close reading of both the main text and the appendix, and for the precise questions on what each robustness test measures.
We took your first weakness as a question about the title and measured what can be read. We used text of known size and ran character recognition on what each method recovers from the wall photographs, and we located and read the text of the existing test screens in the same way. On robustness, the revision states the procedure of every test and titles the two tests of C.11 and C.12 as image-space tests. We have also revised the presentation, and Section 3 now opens with a one-table overview of the three stages. The answers follow the order of your review.
The 95% bootstrap intervals over the 80 screens of each cell are within ±0.04 of each entry. CERL reads large text, with an error rate of 0.10 at 96 px and 0.24 at 48 px, where OKNet is at 0.43 and 0.68. This is the reading that the title refers to, and the revised abstract and Section 4.2 state these character heights.
CERL finds most large text boxes, with an F1 of 0.879 against 0.702 for OKNet, and reads more of their characters than any other method at every height.
The identification holds in another room. The Screen model of Table 1 is applied unchanged to the wall observations of 250 screens in a room other than the one of the datasets, and each output is matched by LPIPS distance against candidate sets of 100, 250 and 500 screens that contain its own target. Each cell gives top-1 / top-5 in percent, and chance for top-1 is 1.0%, 0.4% and 0.2%.
CERL stays ahead of OKNet at every candidate count in the other room, and the 95% intervals of the two methods are separated. The table is added to Appendix D.