acceptodds
Under review as a conference paper at ICLR 2027

Reading Screens from Wall Reflections: Coalesced Evidential Representation Learning for Hidden Screen Content Recovery

Abstract

Recovering electronic screen content from wall reflections exposes a practical optical side channel for hidden-screen observation. We develop a passive hidden-screen recovery framework that reconstructs screen content using only wall light spots captured by an ordinary camera and confronts three core challenges: diffuse reflection entangles screen structure with wall appearance and imaging noise; incomplete optical evidence causes reconstruction states to drift from the underlying screen layout; and locally plausible inversions produce globally inconsistent content. To address these challenges, we propose Coalesced Evidential Representation Learning (CERL), which coalesces complementary optical cues throughout reconstruction to recover screen structure from mixed wall reflections. CERL integrates Optical Evidence Representation Learning (OERL) to organize broad luminance, relative variation, boundaries, cross-scale changes, and local contrast; Inverse State Transport Learning (ISTL) to anchor progressive reconstruction to the observed evidence; and a Collective Consistency Module (CCM) to reconcile competing local corrections into a coherent screen estimate. Evaluated across five content categories, CERL surpasses a broad range of mainstream reconstruction methods across perceptual, structural, and pixel-level metrics. This consistent performance advantage validates that CERL effectively addresses the core challenges of this passive optical side-channel problem and enables recovery of hidden screen content from ordinary wall reflections.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Screen Reconstruction, Deep Learing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TfSNOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into a passive optical side‑channel issue with clear information‑security implications. The goal is to reconstruct what’s shown on a blocked‑off screen, just by taking photos of diffuse reflections on a wall using an ordinary camera, when you can’t see the screen face‑on. The authors put forward Coalesced Evidential Representation Learning (CERL). It breaks the reconstruction pipeline down into three complementary modules. OERL pulls together optical clues from wall reflections, covering things like brightness, relative variation, boundaries, cross‑scale differences and local contrast. ISTL uses anchor‑relative and bounded state updates to stop states drifting during step‑by‑step reconstruction. CCM gets more consistent final outputs by soft‑voting and checking agreement among multiple local correction candidates. The paper makes it clear that only the observation y is used during inference; the target x and valid mask m are only for training.

Experiments run on five dataset groups: Screen, Figure, Game, OShome and Photo. Evaluations use LPIPS, SSIM and PSNR. In Table 1, CERL scores the best reported point estimates across all 15 dataset‑metric combinations. The paper also includes module‑wise ablation tests, plus extra analyses covering distance, brightness, noise, wall texture, occlusion, cropping, camera position and camera shake.

Critical strengths

This study targets a real‑world passive optical side‑channel scenario with clear privacy and security risks — you use an off‑the‑shelf camera to recover hidden screen content from diffuse wall reflections. This task is different from standard image restoration work. The authors also break down task difficulties reasonably. Design choices fit closely with problem motivations. OERL doesn’t just blindly concatenate features; it explicitly builds multiple types of evidence pulled from observations. ISTL runs updates anchored to reference points. CCM uses candidate voting and reduces disagreement to fix inconsistent local predictions. Plenty of existing restoration methods are compared in main tables, covering five different content types. CERL achieves top point estimates across all 15 dataset‑metric pairs in Table 1. Fine‑grained ablation experiments are done for OERL, ISTL and CCM. Tests over distance, noise level, brightness and partial observations show authors understand how heavily performance depends on real‑world physical conditions.

Critical weaknesses
  1. Current evaluation metrics don’t really reflect what “Reading Screens” is supposed to measure. Main experiments rely only on LPIPS/SSIM/PSNR. Meanwhile, dataset descriptions mention labels contain web pages, tables, UI elements and even password‑style characters. But the conclusion cautiously admits the model mostly reconstructs rough brightness patterns, region layouts and main outlines. It would help a lot to add metrics like OCR accuracy, character recognition rates or UI/text region detection, or at least some downstream content‑recognition metric.
  2. Some robustness test claims overstate what the perturbation experiments actually show. Most notably the camera‑jitter test: simulated pixel shifts get framed as robustness against motion blur and camera shake. Similarly, for brightness changes, noise injection and wall surfaces like marble or textured fiber walls, authors need to make it crystal clear which test cases use real‑world captured footage and which are digitally altered.
  3. The main text only lists Adam optimizer, learning rate 10−410^{-4} and batch size 16, stating other hyperparameters take default values. But readers still need to know training epochs / total iterations, learning‑rate scheduling, augmentation strategies, sample counts per dataset, model size and FLOPs. Also, the valid mask affects multiple loss terms and evaluation calculations, yet how this mask gets built is never explained clearly.
Questions for authors
  1. How many training / validation / test samples are there for each of the five datasets?
  2. For the distance‑robustness experiment: is CERL trained only on data captured at 1.0 m and directly tested on footage from 0.5–4.0 m? Or does training data already include multiple different distances? This distinction really changes how we interpret distance robustness.
  3. For the camera‑position movement experiment in C.11: did authors physically move the camera and record new footage, or just shift pixels digitally? In C.12, given they used synthetic image shifts, why do they interpret those results as motion blur resistance? Could you fix either the experiments or the wording?
  4. Could you share concrete architectures for modules such as g2g_2, g3(j)g_3^{(j)}, g4(j)g_4^{(j)}, χθ0\chi_{\theta_0} and ηk\eta_k? Also please explain how valid masks are constructed and give the exact implementation for ϕ\phi inside loss L5L_5.
  5. Since your work focuses on recovering screen content and datasets include UI, text and password‑like material, can you add OCR / character‑level or downstream recognition metrics? These would demonstrate real‑world security impacts far better than PSNR or SSIM.
5 comments
AuthorsOpenReview ↗

Response to TfSN(1/3)

Thank you for the close reading of both the main text and the appendix, and for the precise questions on what each robustness test measures.

We took your first weakness as a question about the title and measured what can be read. We used text of known size and ran character recognition on what each method recovers from the wall photographs, and we located and read the text of the existing test screens in the same way. On robustness, the revision states the procedure of every test and titles the two tests of C.11 and C.12 as image-space tests. We have also revised the presentation, and Section 3 now opens with a one-table overview of the three stages. The answers follow the order of your review.

  1. Large text is read from the outputs, and this is the character-level metric you ask for. We tested on 480 screens, 80 for each of six character heights from 16 to 96 px in the 384 × 512 target, each filled with random words and digit strings. PaddleOCR is run with identical settings on the targets, on the wall photographs and on the outputs, and the character error rate is computed against the displayed strings.
Character height in the target96 px72 px48 px32 px24 px16 px
Target image0.0100.0120.0100.0110.0140.084
Wall photograph0.9900.9910.9920.9910.9930.991
OKNet0.4340.5420.6830.8390.8980.951
CERL0.1020.1530.2350.5520.7590.904

The 95% bootstrap intervals over the 80 screens of each cell are within ±0.04 of each entry. CERL reads large text, with an error rate of 0.10 at 96 px and 0.24 at 48 px, where OKNet is at 0.43 and 0.68. This is the reading that the title refers to, and the revised abstract and Section 4.2 state these character heights.

  1. Text is located and read on the existing test screens as well, which is the text-region detection you suggest. We ran PaddleOCR on the 251 Screen test targets to obtain reference text boxes with their strings, ran it with identical settings on each output, counted a box as found at IoU 0.5, and split the boxes by height. Each cell gives the F1 of box detection, the character error rate inside the matched boxes and the IoU of the text mask.
Image≥ 32 px, 494 boxes16 to 31 px, 1160 boxesbelow 16 px, 2752 boxes
Wall photograph0.063 / 0.975 / 0.1120.032 / 0.976 / 0.0660.007 / 0.985 / 0.031
ConvIR0.609 / 0.750 / 0.5430.342 / 0.891 / 0.3020.106 / 0.956 / 0.116
OKNet0.702 / 0.695 / 0.5790.431 / 0.837 / 0.3580.155 / 0.949 / 0.160
CERL0.879 / 0.441 / 0.7540.659 / 0.697 / 0.5500.328 / 0.905 / 0.278

CERL finds most large text boxes, with an F1 of 0.879 against 0.702 for OKNet, and reads more of their characters than any other method at every height.

  1. As a downstream recognition metric, an output identifies its own screen. Each output is matched against the 250 distinct Screen test targets by LPIPS distance, and we count how often the true target is ranked first or within the first five. Chance is 0.4% and 2.0%.
Query imageTop-1 (%)Top-5 (%)
Wall photograph0.84.0
DGUNet17.135.9
IRNeXt20.742.2
ConvIR22.746.2
OKNet25.550.6
CERL39.468.1

The identification holds in another room. The Screen model of Table 1 is applied unchanged to the wall observations of 250 screens in a room other than the one of the datasets, and each output is matched by LPIPS distance against candidate sets of 100, 250 and 500 screens that contain its own target. Each cell gives top-1 / top-5 in percent, and chance for top-1 is 1.0%, 0.4% and 0.2%.

Candidates100250500
Wall photograph0.8 / 5.20.4 / 2.40.4 / 0.8
OKNet22.0 / 46.414.8 / 34.49.6 / 26.4
CERL40.0 / 69.228.8 / 51.620.4 / 44.4

CERL stays ahead of OKNet at every candidate count in the other room, and the 95% intervals of the two methods are separated. The table is added to Appendix D.

AuthorsOpenReview ↗

Response to TfSN(2/3)

  1. The revision states the procedure of every test. The distance evaluation follows the protocol of Section 4.4, with the setup shown in Figures 4 and S18. On your second question, the training data contain a single camera position. CERL is trained on the dataset captures, and Figure 5 is computed with the checkpoint that produces Table 1, with fixed parameters, so the model is tested directly on 0.5 to 4.0 m. The tests of C.4 to C.8, C.11 and C.12 are applied in image space to the Screen test captures. On your third question, C.11 translates the image horizontally by up to ±12% of its width and C.12 applies a synthesized jitter of up to 8 pixels, and the revision titles both as image-space tests and describes them as translation and jitter. The table lists these tests, and each appendix section opens with its procedure.
TestProcedure
Brightness (C.4)Image space. Input brightness scaled from 24% below to 24% above nominal
Noise (C.5, Table 6)Image space. Additive Gaussian noise at 20 to 40 dB SNR
Wall appearance (C.6)Image space. Marble-like and fiber-like surface patterns of graded strength applied to the test captures
Local occlusion (C.7)Image space. One cell of a 3×3 grid masked
Cropping (C.8)Image space. Progressive right-side crop
Horizontal translation (C.11)Image space. Horizontal translation up to ±12% of the image width, bicubic interpolation
Jitter (C.12)Image space. Synthesized displacement with a radius of 0 to 8 pixels
  1. Training details and sample counts are collected here and in Appendix A.3.
ItemValue
Captures, training/validation/testScreen 2008/251/251, Figure 2000/250/250, Game 2000/250/250, OShome 6400/800/800, Photo 4800/600/600
Splitall captures of the same content are in the same subset
Input and output size384 × 512, aspect ratio preserved with padding, with deterministic preprocessing only
OptimizerAdam, learning rate fixed at 1e-4 throughout training, batch size 16, seed 42
Training lengthup to 60 epochs with early stopping
Model size2.45 M parameters
Latency and memory108 ms per image and 0.92 GB peak GPU memory on an RTX 3080 at 1 × 3 × 384 × 512, FP32, median of 10 timed runs after warm-up
FLOPs at 384 × 512 (torch.profiler)CERL 129.3 G, OKNet 211.6 G, IRNeXt 661.2 G, SFNet 721.4 G, ConvIR 748.3 G
  1. The valid mask is the rectangle occupied by the target on the 384 × 512 canvas after aspect-preserving resizing, 1 inside and 0 on the padding. Screen targets fill the canvas, the other datasets average 81 to 82% coverage, and every method is evaluated inside the same rectangle. L5 uses LPIPS with a fixed AlexNet backbone on the valid crop resized to 224 × 224.
AuthorsOpenReview ↗

Response to TfSN(3/3)

  1. The modules ψ, ϕ, η, χ and o have the following structure, with hidden width 48, and this table is added to Appendix B.
SymbolRoleLayers
ψ (OERL)cue projection and feature correction1×1 conv C→48 on the feature, 3×3 conv 5→48 on the five cues, 7×7 depthwise and 3×3 dilated depthwise mixing, a two-layer 1×1 confidence branch 48→24→1 with sigmoid, 1×1 conv 48→C followed by tanh
ϕ (ISTL)state transport1×1 conv 3C→48 on the current state, the reference state and their difference, 3×3 conv 5→48 on the cues, 5×5, 1×5 and 5×1 depthwise mixing, 1×1 conv 48→C, a two-layer 1×1 spatial gate
η (CCM)candidate map1×1 projection 24→48, parallel 5×5 depthwise and 3×3 dilated depthwise mixing, 1×1 projection to 12 channels forming four RGB candidates, 1×1 vote head with softmax over candidates
χobservation descriptortwo fully connected layers 10→64→64 on the mean and standard deviation of RGB, gray level and Sobel response
oRGB readout3×3 conv 48→3, residual to the input

OERL acts at the bottleneck and the first decoder level (C = 192 and 96) and ISTL at three levels (C = 192, 96 and 48). Mixing layers use SiLU with global response normalization, and the three CCM steps share one set of parameters.

  1. The abstract and the introduction now state what is recovered at the level these answers measure, which is layout, screen identity and large text. The abstract ends with the character error rates of point 1, and Appendix A.2 gives the character heights.

Upon acceptance we will release the datasets, the random-string test set with its wall photographs and outputs, the split files, the trained weights and the code. We hope the character recognition, the text detection, the identification among candidate screens in two rooms and the stated procedure of every robustness test meet the requests at the center of your review, and we would be grateful if you would consider raising your score.

Reviewer_4rEfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CERL to estimate hidden screen images from photographs of light reflected on a nearby wall. It combines optical image cues, anchor-based feature refinement, and a module that combines several learned RGB corrections. Experiments on five content categories report improvements in PSNR, SSIM, and LPIPS. The demonstrated recovery mainly concerns coarse screen brightness, region layout, and major contours.

Critical strengths
  1. A relevant passive-imaging problem. Screen information leaking through nearby wall reflections is worth studying. Using ordinary-camera observations makes the setting practically interesting, and the paired real captures are a useful basis for investigating what information survives the indirect optical path.

  2. Several types of experimental evaluation. The paper considers five content categories and compares with a broad set of reconstruction methods. The real camera-distance experiment is also useful for showing how performance changes with acquisition conditions, rather than reporting results only at one fixed distance.

Critical weaknesses
  1. Recovery of unseen screen content is not established. Section 4.1 gives a random 8:1:1 split, but the numbers of distinct screen images and repeated captures are not reported. It is also unclear whether all captures of the same source image or interface template stay in one split. A network may learn familiar wallpapers or layouts while matching their overall colors. The noise and brightness tests do not answer this question. I understand that the main text limits recovery to coarse structures, but the security examples involving specific screen content go beyond what these results currently establish.

  2. The mechanism studies do not clearly isolate the proposed effects. Under Appendix B’s settings, the final CCM updates can change each RGB value by at most 0.0111, while its agreement factor remains above about 0.96. This does not mean that CCM is useless, but its actual contribution needs to be measured within the same trained model. Removing CCM and retraining can also change the earlier features and base prediction. Tables 7–8 raise another concern: they use a frozen U-Net, but do not clearly explain how different representations are adapted to it or which parameters are trained. Sharing a decoder alone does not make these comparisons equally controlled.

Questions for authors
  1. Content separation. How many distinct screen images and captures are in each split? Are repeated captures of the same source content kept together? Do any existing results evaluate entirely unseen images or interface layouts?

  2. Controlled comparisons. What are the scores before and after CCM within the same trained model, and what range does the agreement gate take? For Tables 7–8, please clarify the feature adapters, decoder training, and trainable parameters.

4 comments
AuthorsOpenReview ↗

Response to 4rEf (1/3)

Thank you for the review. Both weaknesses shaped what we measured for this response.

Your first weakness asks whether the network reads the wall or recalls a familiar screen. All captures of the same content are in the same subset, and point 1 reports the counts you ask for. We then tested content made of random strings. We used 480 screens of random words and digit strings and read the text from what CERL recovers. We compared every output with the nearest training screen. We also ordered the Screen test captures by their distance from the nearest training screen and scored each group separately. For your second weakness we measured CCM inside one trained model, before and after its three steps.

  1. All captures of the same content, including repeated captures, are assigned to the same subset. The table gives the numbers of captures and of distinct screen images in each split, with distinct images counted by pixel identity.
DatasetTraining, captures / distinct imagesValidationTest
Screen2008 / 1800251 / 249251 / 250
Figure2000 / 1780250 / 242250 / 250
Game2000 / 1760250 / 245250 / 250
OShome6400 / 5800800 / 780800 / 800
Photo4800 / 4550600 / 594600 / 600

Every result of Table 1 is therefore measured on test content that is separate from the training content, on all five datasets. Appendix A.3 reports the table. Points 2 to 4 extend this to layouts.

  1. Random strings are read from the wall. We tested on 480 screens prepared for this test, 80 for each of six character heights from 16 to 96 px in the 384 × 512 target, each filled with random words and digit strings. Their content is generated at random and is independent of the source pages, applications and templates of the training set. PaddleOCR is run with identical settings on the targets, on the wall photographs and on the outputs, and the character error rate is computed against the displayed strings.
Character height in the target96 px72 px48 px32 px24 px16 px
Target image0.0100.0120.0100.0110.0140.084
Wall photograph0.9900.9910.9920.9910.9930.991
Nearest training neighbour0.9570.9750.9790.9630.9620.978
OKNet0.4340.5420.6830.8390.8980.951
CERL0.1020.1530.2350.5520.7590.904

The 95% bootstrap intervals over the 80 screens of each cell are within ±0.04 of each entry. The nearest training neighbour is the training target retrieved for the most similar wall photograph, which is what recalling a familiar screen returns. Its error rate stays above 0.95 at every height, and CERL reads the same strings with an error rate of 0.10 at 96 px and 0.24 at 48 px, so the characters that CERL gets right come from the wall photograph. This is Table S10 of the revised PDF, with a summary in Table 1b of Section 4.2, and the revised abstract and Appendix A.2 state the security examples in terms of the text heights at which characters are read.

  1. On the Screen test set an output identifies its own screen far beyond what retrieval of a training screen gives. Each output is matched against the 250 distinct Screen test targets by LPIPS distance, and we count how often the true target is ranked first or within the first five. Chance is 0.4% and 2.0%.
Query imageTop-1 (%)Top-5 (%)
Wall photograph0.84.0
Nearest training neighbour8.025.1
DGUNet17.135.9
IRNeXt20.742.2
ConvIR22.746.2
OKNet25.550.6
CERL39.468.1
CERL, the third of the test captures farthest from training (87)33.360.9

The retrieval row is what recalling a familiar screen achieves. CERL ranks the true screen first in 39.4% of the cases against 8.0%, and in 33.3% on the third of the test captures farthest from every training target.

AuthorsOpenReview ↗

Response to 4rEf(2/3)

  1. The lead of CERL holds at every distance from the training screens, and its margin over the nearest training screen widens with that distance. We computed the perceptual-hash distance between each Screen test target and its nearest training target, split the 251 captures into quartiles and scored each quartile. The last column is the layout IoU, on Otsu-binarized bright and dark regions, that is reached by returning the nearest training screen itself.
Quartile of distance from trainingCapturesLPIPS, CERL / OKNetPSNR, CERL / OKNetLayout IoU, CERL / OKNetLayout IoU of the nearest training screen
1, nearest630.373 / 0.39921.46 / 21.160.791 / 0.6880.674
2630.402 / 0.43820.56 / 20.390.750 / 0.6490.578
3630.444 / 0.46120.20 / 19.970.717 / 0.6070.517
4, farthest620.450 / 0.48120.11 / 19.440.686 / 0.5800.464

The margin of CERL over the nearest training screen grows from 0.117 in the nearest quartile to 0.222 in the farthest. In the farthest quartile the nearest training screen reaches a layout IoU of 0.464 and CERL reaches 0.686, and CERL stays ahead of OKNet in every quartile.

  1. Inside one trained model the three CCM steps improve all three metrics on all four datasets. The scores before and after the steps and the measured range of the gate are the two quantities of your second question, and we measured both on the checkpoints of Table 1.
DatasetBefore CCM, LPIPS / SSIM / PSNRAfter CCM, LPIPS / SSIM / PSNR
Screen0.4190 / 0.7378 / 20.5430.4172 / 0.7384 / 20.584
Figure0.5678 / 0.7254 / 19.4450.5657 / 0.7259 / 19.478
Game0.5429 / 0.6678 / 18.1230.5414 / 0.6682 / 18.151
OShome0.6247 / 0.6234 / 16.5110.6235 / 0.6237 / 16.547

LPIPS falls and SSIM and PSNR rise in every row, which is the contribution of CCM measured within the same model. The measured agreement factor lies between 0.982 and 1.000 with a median of 0.997. It attenuates a step where the candidates disagree, the selection is made by the per-pixel votes, and the placement by the spatial and image-level gates. The ablations that remove the module retrain the network, so they measure its effect on training together with its effect at inference. The revision reports both (Table S4 in C.3), and Section 3.4 states these roles.

  1. In Tables 7 and 8 the U-Net is fixed and one projection layer is trained, and the revision states the three items you ask about. The feature adapter is one projection layer in front of the U-Net, which maps each representation to the input of the U-Net and has identical size in every row. The decoder is frozen, and the U-Net is fixed in every row of both tables. The trainable parameters are those of the projection layer, trained with identical schedule and data in every row. In Table 7 the rows differ only in the representation that enters the projection, and in Table 8 only in the objective under which the same projection is trained. The two tables therefore compare what a fixed reconstruction network receives, and the revised C.9 and C.10 open with this protocol.
AuthorsOpenReview ↗

Response to 4rEf(3/3)

  1. A safeguards paragraph is added to the ethics statement of the main text and to Appendix A. It lists the conditions that the tests of Appendix C indicate to weaken the channel, which are reduced display brightness and strong ambient light (C.4), textured wall finishes (C.6), and shielding of the central wall region behind the monitor, the area that C.7 identifies as carrying most of the recoverable signal. It pairs them with low-cost protections, a matte backing panel or hood behind the display and a display orientation that avoids a nearby light-colored wall. The security examples are stated at the level that the character test establishes.
  2. The presentation is revised for readability. Section 3 now opens with a one-table overview of the three stages and gives each module by its purpose, its inputs, one update rule and one bound.

Upon acceptance we will release the datasets, the random-string test set with its wall photographs and outputs, the split files, the trained weights and the code. We hope the counts of each split, the tests on random strings, the retrieval reference, the analysis by distance from training and the measurement of CCM inside one model address the concerns behind your rating, and we would be grateful if you would consider raising your score.

Reviewer_4rEfOpenReview ↗

Post-rebuttal comment

Thanks to the authors for the response. I read it, but my main concerns are still there.

  1. The random string test shows CERL can read big text (96 px). At 32 px the error is 0.55, and at 16 px it is 0.90. Normal screen text is small, so the paper still can not support the security examples about real screen content. Also, the nearest training screen already gets layout IoU 0.67 to 0.46, so a lot of the layout can come from training data.

  2. CCM only improves PSNR by about 0.04 dB, and the gate is almost always close to 1. So CCM does very little. It should not be a main contribution.

  3. Tables 7 and 8 only train one layer before a frozen U-Net. This shows which feature fits this U-Net, not which feature is better.

  4. Most new results are only in the rebuttal. The paper needs a big rewrite and a new review.

So I keep my score.

Reviewer_ys1TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how an ordinary camera can recover hidden screen content from diffuse wall reflections without directly viewing the display. It proposes Coalesced Evidential Representation Learning (CERL) to address weak optical signals, reconstruction drift, and inconsistencies between locally plausible predictions. The proposed method introduces Optical Evidence Representation Learning (OERL) module, which combines luminance, relative variation, boundaries, cross-scale differences, and local contrast through bounded feature updates. Its Inverse "State Transport Learning" module anchors progressive reconstruction to observation-derived evidence and reference states to limit unsupported changes. Extensive experiments are conducted on a series of benchmarking datasets to report LPIPS, SSIM, and PSNR results. The proposed method outperforms the baseline state of the art methods.

Critical strengths
  1. This paper has conducted comprehensive experiments and evaluations. E.g. Table 1 includes interfaces, figures, games and operating-system scenes, etc. And it is evaluated under multiple quantitative metrics including LPIPS, SSIM and PSNRs.
  2. CERL has made considerable improvements on LPIPS metric of 6.1% relative to baseline methods such as OKNet.
  3. The ablations study is sufficiently convincing. For instance, Table 3, 4, and 5 examine evidence channels, residual attention alternatives, etc. to convey the effectiveness support of the proposed method. Also, appendix C has included additional tests and evaluations to scrutinize the pipeline components.
Critical weaknesses
  1. The main concern of the experiment setup and evaluation metrics is that the evaluation measures LPIPS, SSIM and PSNR, which are pixel-level similarity including luminance, colors, contours, etc. However, it is not clear whether the text, numerical values, or other specific information from the screen is correctly recovered rather than replaced with plausible-looking structures. In appendix A.2, the paper explicitly mentions password characters regarding content level evaluation in terms of accuracy and precision.
  2. The disagreement gate raised in Eq. B.19 to Eq. B21 is mathematically restricted to being nearly constant.
  3. The other main concerns in the experiments is that some of the improvements are too marginal or small. For example, in Table 1, CERL only improves 0.0001 SSIM and 0.015dB PSNR over IRNeXt method, and also it exceeds DGUNet on OSHome data by 0.0001 SSIM.
  4. (reproducibility) What is the dataset sample counts and training duration in Sect 4.1 ?
  5. There are some noticeable contradictions: In section 4.4, it says OKNet deteriorates sharply beyond about 2.0 m, however, the blue OKNet curve in Figure 5 is comparatively stable and the obvious cyan deterioration belongs to HVI-CIDnet method.
Questions for authors

I would like to raise a few more technical detail questions:

  1. What is the relationship to IR4Net's passive wall-to-screen task, and can both be evaluated on the same source-disjoint data?
  2. What is trained in the frozen-U-Net representation/objective experiments, and how are feature dimensions and objective changes accommodated?
  3. Which robustness tests use physical recapture, what checkpoint is used, and how are camera pose, digital translation, and motion blur distinguished?
3 comments
AuthorsOpenReview ↗

Response to ys1T(1/3)

Thank you for the careful reading.

Your first weakness asks whether the output is the screen that was displayed or a plausible structure in its place. We took that question to the saved outputs of every method. We used screens of random words and digits and read the text of the outputs with a character recognizer, we matched each output to its own screen among all test screens, and we measured where its bright and dark regions lie. The answers follow the order of your review.

  1. Large text is read from the outputs. You ask whether text and numerical values are recovered, and we measured this with character recognition. We tested on 480 screens, 80 for each of six character heights from 16 to 96 px in the 384 × 512 target, each filled with random words and digit strings. PaddleOCR is run with identical settings on the targets, on the wall photographs and on the outputs, and the character error rate is computed against the displayed strings.
Character height in the target96 px72 px48 px32 px24 px16 px
Target image0.0100.0120.0100.0110.0140.084
Wall photograph0.9900.9910.9920.9910.9930.991
Nearest training neighbour0.9570.9750.9790.9630.9620.978
OKNet0.4340.5420.6830.8390.8980.951
CERL0.1020.1530.2350.5520.7590.904

The 95% bootstrap intervals over the 80 screens of each cell are within ±0.04 of each entry. The nearest training neighbour is the training target retrieved for the most similar wall photograph. The strings are random, so the characters that CERL gets right come from the wall photograph. CERL reads them with an error rate of 0.10 at 96 px and 0.24 at 48 px, where OKNet is at 0.43 and 0.68. The revised abstract, Section 4.2 and Appendix A.2 state the text heights at which characters are read.

  1. Text is located and read on the existing test screens as well. We ran PaddleOCR on the 251 Screen test targets to obtain reference text boxes with their strings, ran it with identical settings on each output, counted a box as found at IoU 0.5, and split the boxes by height. Each cell gives the F1 of box detection, the character error rate inside the matched boxes and the IoU of the text mask.
Image≥ 32 px, 494 boxes16 to 31 px, 1160 boxesbelow 16 px, 2752 boxes
Wall photograph0.063 / 0.975 / 0.1120.032 / 0.976 / 0.0660.007 / 0.985 / 0.031
ConvIR0.609 / 0.750 / 0.5430.342 / 0.891 / 0.3020.106 / 0.956 / 0.116
OKNet0.702 / 0.695 / 0.5790.431 / 0.837 / 0.3580.155 / 0.949 / 0.160
CERL0.879 / 0.441 / 0.7540.659 / 0.697 / 0.5500.328 / 0.905 / 0.278

CERL finds most large text boxes, with an F1 of 0.879 against 0.702 for OKNet, and reads more of their characters than any other method at every height.

  1. An output identifies its own screen. Each output is matched against the 250 distinct Screen test targets by LPIPS distance, and we count how often the true target is ranked first or within the first five. Chance is 0.4% and 2.0%.
Query imageTop-1 (%)Top-5 (%)
Wall photograph0.84.0
Nearest training neighbour8.025.1
DGUNet17.135.9
IRNeXt20.742.2
ConvIR22.746.2
OKNet25.550.6
CERL39.468.1

CERL ranks the true screen first in 39.4% of the cases against 25.5% for the best other method.

The identification holds in another room. The Screen model of Table 1 is applied unchanged to the wall observations of 250 screens in a room other than the one of the datasets, and each output is matched by LPIPS distance against candidate sets of 100, 250 and 500 screens that contain its own target. Each cell gives top-1 / top-5 in percent, and chance for top-1 is 1.0%, 0.4% and 0.2%.

Candidates100250500
Wall photograph0.8 / 5.20.4 / 2.40.4 / 0.8
OKNet22.0 / 46.414.8 / 34.49.6 / 26.4
CERL40.0 / 69.228.8 / 51.620.4 / 44.4

CERL stays ahead of OKNet at every candidate count in the other room, and the 95% intervals of the two methods are separated. The table is added to Appendix D.

AuthorsOpenReview ↗

Response to ys1T (2/3)

  1. The recovered layout is the layout of the displayed screen. Target and output are binarized into bright and dark regions with Otsu thresholding and compared by IoU on the 251 Screen test captures. Two references are included, the mean training screen and the nearest training neighbour.
Image compared with targetBright IoUDark IoUMean IoU
Wall photograph0.3380.3910.364
Mean training screen0.4800.5080.494
Nearest training neighbour0.5630.5390.551
DGUNet0.5500.5900.570
IRNeXt0.6240.6500.637
ConvIR0.6030.6580.630
OKNet0.6110.6520.631
CERL0.7180.7540.736

Every learned method is above both references in mean IoU. CERL reaches 0.736 against 0.637 for the strongest of the others. The 95% intervals are within ±0.012 of each mean.

  1. The agreement factor of Eq. B.21 attenuates a step where the candidates disagree, and it lies between 0.982 and 1.000 on the checkpoints of Table 1. The selection among the four candidates is made by the per-pixel votes of Eq. B.20, and the update is placed by the spatial and image-level gates of Eq. B.22, which vary over the image. Section 3.4 now states these roles, and C.3 reports the scores before and after the three steps inside one trained model, where the steps improve all three metrics on all four datasets (Table S4).
  2. In the two cases you cite, CERL is ahead in LPIPS by 0.0321 over IRNeXt on Game and by 0.0279 over DGUNet on OShome.
DatasetMethodSSIMPSNRLPIPS
GameIRNeXt0.668118.1360.5735
GameCERL0.668218.1510.5414
OShomeDGUNet0.623616.5210.6514
OShomeCERL0.623716.5470.6235

On Screen, the content measures of points 3 and 4 separate the methods in the same way.

Method on ScreenSSIMPSNRLayout IoUTop-1 identification (%)
DGUNet0.696018.9620.57017.1
IRNeXt0.723419.9560.63720.7
OKNet0.731620.2430.63125.5
CERL0.738420.5840.73639.4

CERL is ahead of OKNet by 0.105 in layout IoU and by 13.9 points in identification. In LPIPS, CERL is ahead of the best other method on every dataset, and the 95% interval of every difference lies above zero. We resampled the test captures 2,000 times and recomputed the paired LPIPS difference, and we added a paired Wilcoxon test on the per-image differences.

DatasetBest other methodLPIPS reductionRelative reduction95% intervalWilcoxon p
ScreenOKNet0.02736.1%0.0175 to 0.03685.5e-08
FigureSFNet0.02834.8%0.0183 to 0.03813.6e-07
GameSFNet0.02855.0%0.0184 to 0.03861.4e-07
OShomeSFNet0.01392.2%0.0084 to 0.01927.4e-07
PhotoRetinexformer0.05296.7%0.0475 to 0.05823.8e-54

Section 4.2 states the comparison in these terms.

AuthorsOpenReview ↗

Response to ys1T(3/3)

  1. Section 4.1 now gives the size of each dataset, and Appendix A.3 gives the splits and the training settings.
ItemValue
Captures, training/validation/testScreen 2008/251/251, Figure 2000/250/250, Game 2000/250/250, OShome 6400/800/800, Photo 4800/600/600
Splitall captures of the same content are in the same subset
Training duration on one RTX 3080about 9 hours for Screen, Figure and Game, 24 hours for OShome and 20 hours for Photo
Training lengthup to 60 epochs with early stopping
OptimizerAdam, fixed learning rate 1e-4, batch size 16, seed 42
Input and output size384 × 512, aspect ratio preserved with padding
Model size and compute2.45 M parameters, 129.3 G FLOPs, 108 ms per image and 0.92 GB peak GPU memory on an RTX 3080
  1. Section 4.4 now names each curve of Figure 5 by its legend entry. HVI-CIDNet (cyan) deteriorates sharply beyond 2.0 m, Retinexformer fluctuates at the longer distances, and OKNet (blue) stays comparatively stable. CERL has the lowest LPIPS and the highest SSIM and PSNR at every distance.
  2. IR4Net addresses the same task, the recovery of a screen from a photograph of the wall, and it is now evaluated on the same data and the same split as CERL. We trained IR4Net on the five datasets with the split of Table 1 using its official implementation and settings and scored it on the test sets of Table 1, in the two metrics that the IR4Net paper and Table 1 share.
DatasetPSNR, IR4Net / CERLSSIM, IR4Net / CERL
Screen12.094 / 20.5840.3830 / 0.7384
Figure17.377 / 19.4780.5678 / 0.7259
Game16.355 / 18.1510.5347 / 0.6682
OShome12.822 / 16.5470.3811 / 0.6237
Photo11.735 / 13.8090.2682 / 0.4228

CERL is ahead on all five datasets in both metrics, by 1.8 to 8.5 dB. The two works differ in the method and in the evidence. In the method, IR4Net embeds a radiative-transfer model in a learnable optimizer (PRIrr-Approximation) and restores global structure through semantic reprojection (ICSR), while CERL ties every update to cues computed from the current observation alone, under explicit bounds. In the evidence, IR4Net reports image similarity, and this work adds layout, screen identification and character recognition. Related Work now positions the two, and the comparison is added to Appendix D. On source-disjoint evaluation, the split of Table 1 keeps all captures of the same content in the same subset, so the comparison above is on source-disjoint data. The 480 screens of point 1 contain random strings prepared for this test. They will be released upon acceptance with their wall photographs, targets and scoring scripts, together with the split files of the five datasets, so that IR4Net and later methods can be scored on identical unseen content.

  1. In Tables 7 and 8 the U-Net is fixed and the only trained part is one projection layer in front of it, with identical size, schedule and data in every row. In Table 7 the projection maps each representation to the input of the U-Net, which is how the different feature dimensions are accommodated. In Table 8 the same projection is trained under each objective. The two tables therefore compare what a fixed reconstruction network receives, and the revised C.9 and C.10 open with this protocol.
  2. The camera pose varies in the distance evaluation, which follows the protocol of Section 4.4 with the setup shown in Figures 4 and S18. The tests of C.4 to C.8, C.11 and C.12 are applied in image space to the Screen test captures, and the revision says so in the description of each. C.11 is a digital translation of the image by up to ±12% of its width, and C.12 is a synthesized jitter of up to 8 pixels, and the revision titles both as image-space tests. Every test uses the checkpoint that produces Table 1, with fixed parameters. Appendix C.13 lists the procedure of each test in one table.

We have also revised the presentation. Section 3 now opens with a one-table overview of the three stages and gives each module by its purpose, its inputs, one update rule and one bound.

On acceptance we'll release data, splits, weights, code, and the random-string test set with wall photos/outputs. We would be grateful if you would consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.