acceptodds
Under review as a conference paper at ICLR 2027

Learning Where Light Goes: Transport Routing and Coun- terfactual Recovery for Hidden Screen Reconstruction

Abstract

Can an ordinary RGB camera recover the content of a hidden screen from the light it casts onto a wall? Diffuse propagation spreads each screen region across multiple measurement locations and mixes distant contributions, obscuring where weak evidence originates. The resulting ambiguity also allows plausible character strokes, boundaries and textures to enter a reconstruction without sufficient measurement support. We propose Transport Coupling and Counterfactual Recovery Learning (TCCRL) to connect evidence attribution with local structure verification. Screen-Wall Transport Coupling Learning learns a nonnegative sparse coupling shared across contents of each fixed capture geometry. The same coupling predicts measurement effects and routes wall evidence back to an initial screen state. Counterfactual Transport Verification and Selective Write-back then tests bounded local additions and deletions against complementary measurements and sequentially incorporates accepted changes. Across five screen-content benchmarks, TCCRL improves reconstruction fidelity and perceptual quality over 19 baselines. The recovered images preserve principal colours, regional layouts, page structure, chart outlines, character shapes and interface boundaries. These results establish the initial feasibility of hidden screen reconstruction from wall-mediated observations in fixed physical scenes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Hidden Screen Reconstruction、Learned Light Transport、Computational Imaging、 Measurement Consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6rWjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies hidden-screen reconstruction from wall-mediated RGB observations in fixed physical scenes. The proposed TCCRL framework contains two parts: a scene-sahred transport coupling network that learns the correspondence between screen and wall features, and a verification module that tests local reconstruction changes against complementary measurements before write-backs. The method is evaluated on five screen-content benchmarks, compared to a broad set of prior NLOS and generic image restoration baselines.

Critical strengths
  1. The proposed route-then-verify formulation is well-motivated and coherent.

  2. The evaluation is broad and informative.

Critical weaknesses
  1. The practical value and cost of the task settings is unclear.

The papers states that a new physical geometry requires a new coupling and needs to be separately trained. It is not clear how much such adaption costs in practice (e.g, time/computation power), which will affects the practical value of the method.

Moreover, while the paper clearly frames its contribution as establishing initial feasibility in fixed physical scenes, there is still limited discussion of why this particular setting is practically meaningful. A clearer discussion of the significance and intended use cases of this settings would help establish its broader relevance.

2.The baseline comparison is broad, but coverage of the closest passive transport-aware methods is surprisingly limited.

The evaluation includes prior hidden-scene methods, but several of them originate from largely different sensing settings. Generic restoration methods, on the other hand, are a reasonable category to compare with, but add little to isolating the value of TCCRL's particular formulation compared to closest passive-NLOS prior methods.

In particular, a closer method Zhang et al. (2025) is already mentioned in the related work section, but is surprisingly not included in the evaluation.

Questions for authors
  1. Can author include discussion of why such task settings is meaningful, and the practical cost of doing per-scene adaption?

  2. Can author include more closer passive transport-aware baselines?

3 comments

We thank Reviewer 6rWj. Both of your requests admit an experimental answer, and we address them in that form, beginning with the comparison against the closest passive transport-aware methods and then turning to the cost of per-scene adaptation.


W2 + Q2. The closest passive transport-aware methods

We added the two closest passive transport-aware methods, and TCCRL outperforms both on all five datasets, under both splits and on all three metrics. The revision includes Zhang et al. (2025, TIP, light-transport modulation for passive NLOS) and Chen et al. (2025, lightweight multi-scale fusion for passive NLOS), both cited in Section 2.2. Your point motivated this choice directly, since our related-work section describes building on Zhang et al., and NLOS-OT, already in Table 2, shares four authors with that work.

Both methods take passive projection images as input, the same modality as ours, so adaptation was minimal. We ran the official code with the epoch count, crop size and learning-rate search budget matched to the other baselines. We report both splits, using the original split to check that the comparison holds under the submitted protocol and the content-disjoint split to check whether content correlation could account for the margin. The main text of the revision now states that the submitted split is defined at the sample level (8:1:1, fixed seed) and reports the content-disjoint results alongside it.

DatasetSplitTCCRLZhang 2025Chen 2025Margin over Zhang
CtMoriginal17.9816.7416.21+1.24
CtMcontent-disjoint17.6515.3914.88+2.26
SmMoriginal26.1922.3521.88+3.84
SmMcontent-disjoint25.4021.1620.64+4.24
browseroriginal25.4019.9519.46+5.45
browsercontent-disjoint24.8519.0718.57+5.78
charoriginal20.0414.0813.76+5.96
charcontent-disjoint19.5513.5213.24+6.03
softwareoriginal27.8420.8020.38+7.04
softwarecontent-disjoint27.1020.0119.51+7.09

The table reports PSNR. The revision also reports SSIM and LPIPS for both methods, using the same LPIPS implementation and evaluation region as Table 2, and TCCRL leads on all three metrics in every one of the 20 comparisons (2 methods × 5 datasets × 2 splits).

Zhang et al. derive their advantage from modulation across differing transport conditions, whereas each of our datasets is captured under one fixed geometry, so the condition on which that modulation depends barely varies. Section 2.1 of our paper describes the method as inferring and quantizing a transport-condition code from the projection image and using that code to modulate jointly trained reconstruction and reprojection networks. Because the geometry is held fixed throughout each dataset (a single fixed geometry per dataset), the condition code is nearly constant within a dataset, and reconstruction rests mainly on the backbone.

The new rows are compared on equal footing, and the same protocol applies to the entire baseline table. The revision retrains the full submitted baseline set on the content-disjoint split under the matched protocol, so every row above is evaluated on the same split as TCCRL, and the strongest baseline for each dataset is the best result any method reaches within its own tuning budget.

This fixed-geometry regime is precisely the setting our formulation is designed for. TCCRL learns an explicit sparse correspondence within one fixed geometry and then verifies local edits against held-out parts of the same measurement. We therefore draw a deliberately narrow conclusion from the comparison, namely that for passive single-frame recovery under one fixed geometry, routing followed by verification is the better-matched design, and we do not extend it to a general statement about that line of work. Under multi-geometry joint training the comparison could come out differently, and the revision identifies that setting as the one in which Zhang et al. should be evaluated again.

Neither method becomes the strongest baseline on any dataset, so every reference to the strongest baseline elsewhere in the paper remains unchanged. We read your request as concerning both whether the comparison exists and whether its protocol is fair, and the revision therefore states the protocol explicitly, covering official code, a matched budget for epochs, crop size and learning-rate search, and results on both splits.


W1 + Q1 (first part). What per-scene adaptation actually costs

Per-scene adaptation recovers roughly half of the loss with two minutes of registration and comes within half a dB of a full retrain with half an hour of capture plus about ten minutes of GPU time, whereas the full retrain takes 6.5 to 9 hours. We measured this on two non-reference geometric conditions, both lying on the Appendix C.6 sweeps, namely char in Scene A with a camera tilt of +5° on the angle axis, and software in Scene B with a camera-to-wall distance of 0.925·d0 on the distance axis. All values are PSNR on each condition's own test set. The reference for each condition is its own full-data retrain, since the Table 2 numbers belong to a different geometry.

Adaptationpairscharsoftware
Zero-shot014.2019.64
Homography registration only017.3523.90
Refit coupling only20019.2826.85
Refit coupling only50019.6627.38

The target level is each condition's own full-data retrain, measured on that condition's test set so that it shares the split with every adaptation row: 19.80 on char and 27.60 on software. The curves of Appendix C.6 are on the archived validation split of that study, where software stays within 27.77–27.84 dB across the whole distance sweep, so they show the same stability on a different split rather than supplying these two cells. The revision states the split of each. Against the test-set references, coupling-only refitting with 500 pairs falls short by 0.14 and 0.22 dB.

Cost itemcharsoftware
Homography registration (no pairs needed)~2 min~2 min
Capture 500 pairs~25 min~25 min
Coupling refit, GPU~8 min~11 min
Full retrain, GPU (reference)~6.5 h~9 h
Trainable parameters / full network~4k / ~3.2M~4k / ~3.2M

The mechanism follows from the planarity of the wall, under which a small camera translation or tilt acts approximately as a homography on the wall image. This homography can be estimated from the static scene without any paired captures, and this step alone recovers about half of the loss in two minutes. Most of the remainder is closed by refitting only the coupling, about 4k parameters, at a cost of roughly half an hour of capture and ten minutes of GPU time, compared with 6.5–9 hours for a full retrain.

A new geometry therefore does require a new coupling, and the revision states this while showing that obtaining one is a calibration step measured in minutes of GPU time. The scope of the result is limited in two respects. It is per-geometry adaptation, in which each condition keeps its own coupling and the planar-homography argument holds for small changes. It also covers only camera tilt and camera-to-wall distance. Occluder repositioning alters the visibility structure itself, which no warp of the wall image can represent, so it belongs to a different geometric family that this result does not cover.


W1 + Q1 (second part). Why this setting is practically meaningful

The intended setting is defensive, and in that setting the fixed geometry follows from the application itself and imposes no additional restriction.

The use case is optical leakage auditing, that is, assessing how much a given room or display installation leaks through indirect reflection. The auditor has physical access to the room, can display known content on the screen being audited, and can calibrate once for a fixed installation. These are exactly the conditions our method requires, namely cooperative capture, a fixed geometry and one-time calibration, so the same requirements that limit adversarial use are met by construction in an audit.

The same requirements exclude covert use, and the revision now states the threat model on this basis. Applying the method requires thousands of known-content captures on the target screen, a fixed geometry with re-adaptation whenever it changes, capture and adaptation under the target illumination, and a screen-to-occluder distance of about 0.5 m. Together these conditions demand physical access and cooperative data collection, which places covert reading of arbitrary private screens beyond the reach of the method. An audit on these terms therefore gives a defender two kinds of actionable information. The primary one is the set of capture requirements above, since each is a condition the defender controls and denying any one of them defeats the method. The second is a measured quantity, namely the decline in recoverability as ambient illuminance rises, which the revision quantifies at three illumination levels.


On isolating the value of the formulation

The margin is attributable to the two proposed modules. Your remark that generic restoration baselines add little to isolating the value of our particular formulation is addressed most directly by the passive-NLOS comparison above. The remark also admits a second reading, namely whether the gains come from the formulation or from the backbone. Table 3 contained no row with both modules disabled, so we ran that check directly.

DatasetFullweakest single-module ablationbackbone-onlystrongest baseline
SmM26.1924.6523.52LEAP 24.06
software27.8426.0022.96Flatland 23.25

The backbone and training recipe alone remain below the strongest baseline, so they cannot be the source of the margin. The backbone-only variant keeps the encoder, decoder and local-contrast branch, replaces the routed skips with ordinary same-scale connections, removes the verification stage together with its losses, and is trained from scratch under the protocol of the Full row. It scores below every single-module ablation, so performance decreases monotonically across the table as modules are removed, and it falls below the strongest baseline on both datasets, by 0.54 dB on SmM and 0.29 dB on software.

Grafting the two modules onto Restormer, an independently strong backbone of different design, improves on Restormer's own results by 1.57 dB on SmM and 3.39 dB on software. The gain therefore transfers with the modules to a different architecture, and this transferability is the only claim we draw from the experiment. We make no claim about relative backbone quality, and the grafted model still trails the full TCCRL.

Reviewer_wfd8OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes reconstructing hidden screen content from indirect wall illumination using an ordinary RGB camera. The method learns a scene designed transport mapping between wall and screen features, followed by a refinement stage that accepts local changes based on measurement consistency. The approach is evaluated on five datasets against 19 baselines, reporting substantial improvements.

I would lean towards reject for this paper, there are multiple inconsistencies. However this paper is not particular my research field, so I'm open to hear from other reviewers what they suggest.

Critical strengths

The problem of reconstructing screen content from indirect illumination is interesting and has practical relevance.

The authors evaluate two physical capture setups and provide many comparisons and ablations.

Critical weaknesses

The authors reduced the line height for the entire paper, which is very bad practice in order to add more content.

A pipeline figure should compress the content of the work, such that the highlevel idea is easy to follow. The pipeline figure used in this paper depicts everything in detail, so that one is unable to follow. The paper is very hard to read, and hides behind many fancy terms. E.g. something like would have been simpler and helped much more. Wall photograph > CNN encoder + learned transport matrix > Initial screen reconstruction > Generate and verify four local edits > Final screen image

Questionable experimental validation: The reconstruction quality, especially for fine details such as characters, is surprisingly high, how can such diffuse light allow to reconstruct this. With random sample train/test splits, it remains unclear to me whether the model genuinely recovers unseen information or exploits correlations in the dataset. Did your experiments run on complete unseen content?

Reproducibility and reporting: The dataset counts are inconsistent between Tables 1 and 5 (e.g., 305 character test samples versus 453 held-out samples).

Questions for authors

How are the datasets constructed, and how do you ensure that similar screenshots or content do not appear across training and test splits? Can you demonstrate reconstruction of completely unseen character strings or layouts?

Why do the held out sample counts in Table 5 differ from the test-set counts in Table 1?

Can you provide a cleaner statement of what your method achieves and how?

4 comments

We thank Reviewer wfd8. Your three requests concern a plain statement of the method, the discrepancy between two tables, and whether the reconstructions are genuine. We address them in that order, and for the third we report experiments designed to be verifiable without domain expertise.


Q3. A cleaner statement of what the method does

We adopt the form you suggested, which describes the method exactly.

Wall photograph → CNN encoder → learned transport matrix → initial screen reconstruction → generate and verify four local edits against a held-out half of the wall measurement → final screen image.

Stated in plain terms, a camera photographs a wall that is lit indirectly by a screen outside its field of view. We learn a sparse matrix specifying which wall locations carry evidence about which screen locations, and this matrix yields a first reconstruction. We then propose small local edits to that reconstruction, compute for each edit the wall observation it implies, and compare that prediction with a part of the wall measurement not used to produce the reconstruction, keeping the edit only if the two agree. This second stage prevents the network from inventing plausible detail, because an invented edit implies a change in the held-out half that does not match what was measured.

On the presentation points we give the schedule plainly instead of leaving it implied. The camera-ready makes the five-step flow above the overview figure, moves the current detailed version to the appendix, restores the template's standard spacing, and rewrites the abstract and introduction so that each mechanism is stated in these plain terms before any acronym for it appears. That is scheduling, not refusal. The existing figure is a full architecture specification, and redrawing it belongs with the final typesetting pass rather than a mid-discussion upload.


W4 + Q2. Why Table 1 and Table 5 disagree (305 vs. 453)

Table 5 was computed on an outdated evaluation index, and rerunning it on the current split changes no conclusion. The table below gives the full accounting.

DatasetTable 1 testTable 5 NWhere Table 5's N came fromTraining samples foundAfter rerun
CtM700350first 350 of val, a fixed subsetnone700
SmM250124124 usable val samplesnone250
browser450453fixed 453-image eval set from an earlier data versionnone450
char305453same earlier 453-image indexnone305
software450453same earlier 453-image indexnone450

Two features of the data identify this index and show that it predates the current split. First, three datasets whose test sets contain 450, 305 and 450 images all report N = 453, and a single number shared across different totals indicates a fixed index unrelated to the current split. Second, the 453 reported for char exceeds both its val and test sets (305 each), and a held-out set cannot be larger than the split it belongs to.

Whether this affects any result depends on whether those images include training images, and none of them do. We compared all 453 indices against the current split image by image. For char, 298 fall in the current test set, 155 in the current val set and none in the training set. The same check across all five datasets found no training sample in any of them. The revision lists the per-dataset counts and the comparison method, so the check can be verified independently.

We reran the analysis on the Table 1 test sets, with inference only and no retraining, so that both tables use identical N. The revision gives both sets of N with the provenance of the stale one, states the per-dataset overlap counts, and records that the set of cells separated from the control is the same under both, so the rerun changes the figures and leaves the conclusions of that subsection standing. We did not relabel 350 as 700 or 453 as 305, since doing so would report samples that were never evaluated.

Your review refers to multiple inconsistencies and names this one, so we re-checked every quantity that appears in more than one table, and all of them agree. The check compared anchor values against the original Tables 2 and 3 digit by digit, recomputed every summary number from its source table, and verified every confidence interval against its degrees of freedom. If a specific second inconsistency prompted that remark, we would be grateful if you could point to it, and we will account for it in the same way.

W3 + Q1. Content recovery or correlation exploitation, tested on content-disjoint splits

The model recovers content, as a direct test with a rebuilt split shows. When no content can be shared between train and test, the margin widens on all five datasets, whereas exploitation of content correlation would narrow it.

The unit of the new split is the template group. Each template group is assigned as a whole to train, val or test (8:1:1), so all sources and all captures of one template fall in the same set, with an embedded photograph counted as part of the same template. The assignment uses a fixed seed, and the split files are released with the code. TCCRL and every baseline were retrained on this split with unchanged hyperparameters.

DatasetOriginal split (TCCRL / strongest baseline / margin)Content-disjoint split (TCCRL / strongest baseline / margin)
CtM17.98 / 17.72 / +0.2617.65 / 15.92 / +1.73
SmM26.19 / 24.06 / +2.1325.40 / 22.60 / +2.80
browser25.40 / 23.06 / +2.3424.85 / 21.95 / +2.90
char20.04 / 17.87 / +2.1719.55 / 16.90 / +2.65
software27.84 / 23.25 / +4.5927.10 / 22.50 / +4.60

On fully unfamiliar content TCCRL loses 0.33–0.79 dB, while the strongest baselines lose 0.75–1.80 dB, so the margin widens on all five datasets. The clearest case is CtM, where the strongest baseline drops 1.80 dB on unseen content and TCCRL only 0.33 dB, so a margin of 0.26 dB grows to 1.73 dB.

Neither memorisation nor a content prior accounts for the results, and two training-free controls bound both. Pixel nearest-neighbour retrieval reaches 9.12–12.63 dB across the five datasets, below every deep baseline on every dataset and 5.45–14.47 dB below TCCRL. A constant mean-screen prediction reaches 8.31–12.84 dB.


W3 + Q1 (second part). Completely unseen character strings

Yes, strings never seen in training are recovered at a character accuracy of 0.884, which is 91.9% of the recogniser's own ceiling. We measured this on char with a fixed recogniser (Tesseract 4.1.1, with the exact version shipped alongside the code). The relevant ceiling is the recogniser's accuracy on ground-truth images, 0.962, which falls short of 100%, and every number below should be read relative to it.

Groupcharacter accuracyfull-string accuracyCER
Ground truth (recogniser ceiling)0.9620.8400.041
TCCRL, seen strings0.9080.6480.098
TCCRL, unseen strings0.8840.5750.123
Strongest baseline, unseen strings0.3780.0140.638
Retrieval0.1900.0010.824
Mean-screen0.0210.0000.980
Chance0.0160.0000.984

Unseen strings reach 0.884 character accuracy against 0.908 for seen strings, a cost of 0.024. The strongest baseline reaches 0.378 on the same unseen strings, and the gap widens at the whole-string level, 0.575 against 0.014, because a single character error invalidates the entire string. Retrieval reaches only 0.190, which rules out matching against memorised training images as an explanation.

This experiment measures recovery of unseen character strings, which is the case you asked about. Because char uses a fixed font, the claim concerns strings the model has not seen and does not extend to glyph shapes it has never encountered.

For unseen layouts, the current data do not permit a measurement. Your question also covers layouts, but char keeps character positions fixed with 4–5 characters per image, so layout offers no variation to generalise over and the string result carries no information about that dimension. Testing it would require a capture set in which character position, count and spacing vary independently of string identity, evaluated under the same protocol. The capability-limits section of the revision accordingly marks unseen layouts as outside the demonstrated range.


W3 (first part). How can diffuse light support this much detail?

The wall carries linearly recoverable information down to a measurable cutoff scale, and TCCRL recovers detail finer than that scale. Equation (14) is linear in screen radiance, so a ridge regression fitted on paired data is the data-driven form of classical linear passive inversion. We fitted this regression and measured how much linear information the wall carries. Every column below is on the content-disjoint split, the ridge fit and the rank included, so the TCCRL and baseline numbers are that split's rather than the submitted table's.

Datasetmean-screenridge regressionstrongest baselineTCCRLeffective rank (99% energy)cutoff scale
CtM12.8413.1015.9217.65710 / 40962.8 cm
SmM8.7212.3522.6025.40355 / 40964.0 cm
browser8.6110.9521.9524.85281 / 21605.9 cm
char12.7012.9516.9019.55167 / 40965.8 cm
software8.3113.0522.5027.10203 / 21607.0 cm

Linear inversion recovers the coarse structure, and the fine detail requires learning. Ridge regression exceeds mean-screen on every dataset, so the wall does carry linearly recoverable low-frequency information, yet it also falls well short of the strongest baseline on every dataset. This finding concerns the limit of linear recovery and is a separate matter from whether classical methods fail. The effective rank gives the number of independent spatial modes the wall can support linearly, and the cutoff scale follows directly as √(content area / effective rank). The content area is the region actually captured and evaluated, which measures 74.7 × 74.7 cm on the three 256×256 datasets and covers the full 132.8 × 74.7 cm display on the two 512×288 datasets, both taken from the 60-inch 16:9 screen described in Appendix A. With these areas and the per-dataset ranks in the table, each centimetre figure can be recomputed from this single formula. The submitted paper stated no resolution figure, so this analysis adds new evidence and corrects nothing previously reported.

Your doubt can be tested directly against an upper bound on what invented detail can achieve, and this bound is deliberately biased in favour of your hypothesis. We trained the same backbone to map low-pass ground truth to full-resolution ground truth, without passing through the wall. The result is the ceiling for perfect coarse information combined with a learned prior, which is the most a model could reach by inventing detail from a prior alone. The ceiling model holds the advantage on its input, since it receives ground-truth low-pass content while TCCRL must recover that coarse structure through the wall, and it is therefore given exactly the information whose absence would otherwise explain a shortfall.

This test is on the original split, where the ceiling was trained, so its three rows are that split's values.

charsoftware
TCCRL, full resolution20.0427.84
Low-pass ceiling (prior-only), full resolution18.3626.08
Strongest baseline, full resolution17.8723.25

TCCRL recovers wall information finer than the cutoff scale, which none of the baselines achieves. TCCRL exceeds the prior-only ceiling by 1.68 and 1.76 dB, and no baseline exceeds it on either dataset. The detail produced by the baselines is therefore consistent with prior-driven invention from coarse structure, whereas the detail produced by TCCRL is not, and this distinction lies at the centre of your question. All three rows report full-resolution PSNR. We do not compare band-limited values against this ceiling, since band-limited PSNR is mechanically higher and such a comparison would be uninformative.


Ethics flag

The capability requires cooperative access to the target device, which excludes covert use, and the revision states this as an explicit threat model. Applying the method to a target screen requires displaying and capturing thousands of known images on that screen, a fixed geometry with re-adaptation whenever it changes, capture and adaptation under the target illumination, and a screen-to-occluder distance of roughly 0.5 m. Together these conditions demand physical access and cooperative data collection on the target device, which places passive covert reading of arbitrary private screens beyond the reach of the method, and the wording of the revision reflects this.

The intended use is defensive, namely auditing the optical leakage risk of a room or display installation, where cooperative capture is available by construction. An audit on these terms gives a defender two kinds of actionable information. The primary one is the set of capture requirements above, since each is a condition the defender controls and denying any one of them defeats the method. The second is a measured quantity, namely the decline in recoverability as ambient illuminance rises, which the revision quantifies at three illumination levels. The same section states the limits of the demonstrated capability, which covers a fixed font at fixed positions with 4–5 characters per image, and it places normal-size continuous text, small UI elements, and illumination or geometry beyond the captured conditions outside the demonstrated range.

Reviewer_nBtFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles hidden screen reconstruction, aiming to recover 2D display content solely from diffuse light cast onto a nearby wall using an ordinary RGB camera. The authors propose Transport Coupling and Counterfactual Recovery Learning (TCCRL), which combines a scene-shared sparse transport coupling (SWTCL) for dual forward/reverse light routing with counterfactual verification (CTVSW) that tests bounded local edits against complementary observation partitions before selective write-back. The framework is validated across five screen-content benchmarks against 19 restoration and non-line-of-sight baselines, supported by policy ablations, spatial attribution, and physical robustness analyses.

Critical strengths
  1. Dual-Orientation Sparse Transport Coupling: Learning a scene-shared sparse matrix CsC_s for dual forward prediction and reverse routing provides an effective, physics-inspired inductive bias. This cleanly bridges diffuse screen-wall propagation without requiring explicit 3D geometry calibration.
  2. Counterfactual Verification Against Unseen Partitions: Partitioning wall measurements into complementary checkerboard views provides an independent physical sanity check for local screen hypotheses. Testing signed additions and deletions against unseen observations effectively suppresses ungrounded generative hallucinations.
Critical weaknesses

Disclaimer: Non-line-of-sight imaging and computational periscopy are outside my primary research area, and the following observations reflect my best understanding based on general inverse-problem and computer vision principles. If any concern arises from domain-specific misunderstandings, the authors are very welcome to clarify or disregard it.

  1. Severe Geometric Overfitting and High Retraining Burden: The learned coupling C_s is strictly tied to a static capture geometry and cannot generalize across camera shifts, occluder movements. Because it is learned via overfitting rather than parameterized by optical transforms, adapting to any minor alteration requires recapturing thousands of paired samples and fully retraining the network.
  2. Missing Implementation Details for Retrained Baselines: Although Section 4.1 claims that Appendix B specifies task adaptations for competing methods, the appendix contains zero implementation details for how the 19 baselines were adapted or trained. Crucial specifications are missing, such as how the baselines are adapted to this task.
  3. (Minor) Unrealistic Laboratory Darkroom Assumptions: The physical capture setups rely on an idealized, completely dark laboratory environment with high-contrast screen contents. In practical indoor settings, ambient daylight and lamp illumination would overpower diffuse signals by multiple orders of magnitude, causing subtle wall variations to drown in sensor shot noise.
  4. (Policy) Missing Mandatory AI Use Statement: This submission appears to omit the mandatory AI / LLM Use Statement required by the ICLR 2027 Author Guidelines.
Questions for authors

Since retraining C_s from scratch on thousands of captures is impractical for real-world deployment, could the learned coupling be analytically updated or warm-started given coarse measurements of physical geometry shifts (e.g., small camera translations or wall angle rotations)?

3 comments

We thank Reviewer nBtF for a careful review. W1, on geometric overfitting and the retraining burden, is also your single Question for Authors, so we ran the experiment you proposed and answer it first on the basis of direct measurements.


W1 + Q1. Can the coupling be analytically updated or warm-started from coarse geometry measurements?

Yes, the geometric component of the coupling can be updated analytically. Two minutes of registration recovers about half of the loss, and half an hour of capture followed by eight to eleven minutes of GPU refitting closes most of the remainder. Adaptation therefore requires neither recapturing thousands of paired samples nor retraining the network.

The mechanism follows from the planarity of the wall, under which a small camera translation or tilt acts approximately as a homography on the wall observation. We estimate this homography from the static scene alone, without screen content or paired captures, and apply the reference-geometry model unchanged after warping.

We evaluated this on two non-reference geometric conditions already present in our capture set, both lying on the Appendix C.6 sweeps, namely char in Scene A with a camera tilt of +5° on the angle axis, and software in Scene B with a camera-to-wall distance of 0.925·d0 on the distance axis. All values below are PSNR on each condition's own test set.

Adaptationpairscharsoftware
Zero-shot (reference model, unmodified)014.2019.64
Homography registration only017.3523.90
Refit C_s only5018.6025.70
Refit C_s only20019.2826.85
Refit C_s only50019.6627.38
Refit C_s only100019.7127.51
Full fine-tune, same pairs50019.5227.20
Strongest baseline, same pairs50016.9522.70

The target level is each condition's own full-data retrain, measured on that condition's test set so that it shares the split with every adaptation row: 19.80 on char and 27.60 on software. The curves of Appendix C.6 are on the archived validation split of that study, where software stays within 27.77–27.84 dB across the whole distance sweep, so they show the same stability on a different split rather than supplying these two cells. The revision states the split of each. Every adaptation number is new.

Three observations follow from the table.

  1. The coupling is bound to its geometry, which is why registration comes first. Applied without modification, it yields 14.20 on char and 19.64 on software, against 19.80 and 27.60 respectively. Each comparison uses the condition's own full-data retrain as the reference, since the Table 2 numbers belong to the reference geometry.
  2. Registration alone, with zero paired samples, recovers about half of that gap. It raises char from 14.20 to 17.35 and software from 19.64 to 23.90 without any screen content or paired captures, which is the analytic warm start you asked about.
  3. Refitting only C_s on 500 pairs closes most of the remainder, reaching 19.66 on char against 19.80 and 27.38 on software against 27.60, so the remaining gap is 0.14 and 0.22 dB. C_s is parameterized by U and V (256×8 each) and 8 modality weights, about 4k trainable parameters compared with roughly 3.2M for the full network. Coupling-only refitting also outperforms full fine-tuning with the same pairs (19.52 and 27.20), which we attribute to the far smaller parameter count being less prone to overfitting at this sample size.

The measured cost of each step is listed below.

Cost itemcharsoftware
Homography registration (no pairs needed)~2 min~2 min
Capture 500 pairs~25 min~25 min
Coupling refit, GPU~8 min~11 min
Full-data retrain, GPU (for reference)~6.5 h~9 h
Trainable parameters~4k~4k

A small geometry change therefore costs two minutes for half of the recovery, and about half an hour of capture plus ten minutes of training for most of the remainder. At this cost adaptation amounts to a calibration step, far removed from the thousands of pairs and hours of retraining that a full rebuild would require.

This result is confined to camera tilt and camera-to-wall distance under per-geometry adaptation. Each condition uses its own coupling, the homography argument applies to small changes within which the planar approximation holds, and the corresponding section of the revision is titled accordingly. Moving the occluder alters the visibility structure itself, which no warp of the wall image can represent, so we make no claim that the homography result covers occluder repositioning, and we have no measurement of it at present. Extending the result would require a capture series at several occluder positions, with the same registration-then-refit protocol applied at each. The revision states this gap in the limitations section, and it is the first experiment we would run next.

C_s is a correspondence learned in feature space, and its geometric component is the part that admits an analytic update. The revision states explicitly that C_s should not be read as a calibrated optical kernel, and that the analytic updatability of its geometric component is what makes two-minute registration possible.


W2. Implementation and training details for the 19 baselines

Appendix B now documents the protocol for each method. Every method in the table, TCCRL included, runs its authors' official implementation with no pretrained weights loaded, and shares one training configuration, checkpoint rule and tuning budget, all stated once above the table. The table then gives per method its original input modality and the exact adaptation made; which components were kept and which were replaced; the loss, number of epochs, learning-rate schedule, crop size and batch size; the tuning budget relative to that of TCCRL; the checkpoint rule, with confirmation that the test set was never used for selection; and the supervision signals each method receives, including the transport-consistency and verification losses that TCCRL uses in addition.

Six methods (Flatland, LEAP, NLOSNeuS, NLOST, Under-scanning and Virtual scanning) were designed for time-resolved measurements and receive a single RGB wall image here. The adaptation for each is described in this table and flagged in the main comparison table, so the meaning of adaptation is explicit for every entry.

Documenting the protocol also allowed us to confirm that every baseline operates at a usable point. We checked the cross-sample output variance of each baseline, since a near-constant output indicates a run that failed to converge, and for the cells identified by this check we applied a uniform additional tuning budget (learning rate scaled by 0.1, 1 or 10, selected on validation, with nothing else changed). The revision reports pre- and post-tuning numbers side by side, so no original figure is overwritten and the effect of the extra budget is visible for each cell. Every baseline is thus reported at the better of its two configurations, which is the standard we would want applied to our own method.

TCCRL outperforms every baseline after retuning. On the content-disjoint split, which is the harder split for all methods, the smallest post-tuning margin over the strongest baseline is 1.73 dB, on CtM: TCCRL reaches 17.65 dB there against 15.92 dB for the strongest baseline, so the margin can be checked directly rather than taken as a summary. Retuning raised baseline scores and lowered none, so no margin relies on a baseline that fell short of its own budget. On the thinnest column this question matters most for, no CtM cell was flagged by the variance check at all, so every CtM figure on either split stands at its originally reported configuration and carries no retuning adjustment. The split is defined in the revision itself. Its unit is the template group, so all sources and captures of one template fall in a single set, and the assignment uses a fixed seed. The split files are released with the code, which makes the 1.73 dB figure directly verifiable.

W3 (Minor). Darkroom assumption and ambient illumination

The method remains effective under room lighting and under 2000 lux daylight, with margins over a same-condition mean-screen baseline of 15.79 to 18.54 dB on software and 3.93 to 6.41 dB on char. We recaptured and retrained under ambient illumination, and because you named daylight specifically, we added a 2000 lux window-side condition beyond room lighting. Wall illuminance was metered at each level, and each row is compared against a mean-screen baseline captured under the same condition, since comparison across lighting conditions would inflate the margin.

Conditionsoftwarecharvs. same-condition mean-screen
Darkroom (reference)27.8420.04+19.48 / +7.17
200 lux, retrained26.9019.28+18.54 / +6.41
500 lux, retrained26.1018.61+17.74 / +5.74
2000 lux daylight, retrained24.1516.80+15.79 / +3.93

Noise was calibrated separately at each illumination level, because shot noise is the component that grows with illuminance and reusing the darkroom σ would make the argument circular. The sensor σ rises from 0.0018 to 0.0043 and then 0.0081 across the three levels, remaining below the content signal in every case. Over 5 repeated captures of the same screen, the standard deviation of reconstruction PSNR is 0.11, 0.19 and 0.34 dB, and the recognised characters are identical across repeats in the darkroom and at 500 lux.

Shot noise does rise by orders of magnitude with illuminance, and spatial aggregation is what absorbs it. The method operates on an aggregated wall response, in which per-capture noise structure averages out. On char, a per-cell content variance of 2.4–4.0×10⁻⁵ corresponds to roughly 1.2–1.6 grey levels at 8 bits, above the calibrated sensor σ at every level we tested, which explains why reconstructions remain consistent across repeated captures.

Equation (14) includes a static illumination term, so stable ambient light can in principle be absorbed, and the table above quantifies its practical effect. The revision accordingly makes the conditional claim supported by these data, namely that the method applies when the target illumination condition is captured and adapted to.


W4 (Policy). AI / LLM Use Statement

The revision includes the AI / LLM Use Statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.