acceptodds
Withdrawn from ICLR 2027

ReCAST: Beyond Reconstruction—Learning Compositional Visual Responses

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Shadow removal is usually determined by alignment with a reference shadow-free image. Under a fixed observation, however, distinct latent responses can yield similar appearance yet diverge once illumination or occlusion changes, so endpoint fidelity alone does not certify a reusable response. We introduce Missing-Transport Inversion (MTI), which recovers the radiance residual suppressed by occlusion so that it remains predictive under withheld illumination–occlusion recombinations, without assuming a uniquely identifiable physical decomposition. Based on this, we propose ReCAST, which grounds the occluded receiver with geometry-compatible non-shadow evidence, learns a shared compositional operator over receiver, illumination, and occlusion factors, and renders appearance with a response-conditioned diffusion decoder that also supports forward re-shadowing. On a controlled same-scene Transport Lattice, it predicts a fixed withheld pairing from complementary observations in a target-blind protocol, reducing normalized missing-transport error (NMTE) from to 0.589 against matched Cross-attention. On ISTD+, SRD, INS, and WSRD+, the same representation remains competitive in PSNR, showing that compositional response modeling does not trade away standard restoration quality. Code is available in the supplementary materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Shadow removal, Compositional response recovery, Missing-response inversion, Factor recombination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RVAoOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a novel evaluation protocol (a transport lattice of illumination-occlusion combinations) and a learned operator that supports inverse restoration and forward re-shadowing. The approach is evaluated on the shadow removal task (Table 8) against related work on four different benchmarks and shows strong results.

It would be great to see a controlled transport lattice protocol evaluation also for real scenes, not just synthetic. Apart from the two contributions above, I struggle to identify which parts of the architecture are commonly used in previous work and which are novel contributions. This could be stated more clearly.

Critical strengths
  1. A novel evaluation protocol (a transport lattice of illumination-occlusion combinations). Formulating the problem as learning one "held-out" combination makes sense.
  2. A learned operator that supports inverse restoration and forward re-shadowing. Strong results in the shadow removal task
Critical weaknesses

I found the current draft terse and difficult to parse. The illustrations are hard to follow, notation is dense, with many terms not defined at first use (or left undefined). For that reason, I find it challenging to assess the technical soundness and novelty aspects of this paper. I feel that this submission would benefit from a thorough editing pass to clarify the method description, so unfortunately, despite the strong results, I will not recommend the current draft for acceptance at ICLR.

I also struggle to identify which parts of the architecture are commonly used in previous work and which are novel contributions. This could be stated more clearly.

It would be great to see a controlled transport lattice protocol evaluation also for real scenes, not just synthetic.

Several figures and illustrations are duplicated in the supplemental material. I would have liked to see more visual results.

Questions for authors

There are visible color shifts compared to GT in Fig 5. It may be worth a discussion. Can this be addressed?

How large is the network, how many diffusion steps are used and what is the inference cost of the proposed approach?

For visual results, I would recommend including additional visual examples in the supplemental material. Currently, Figure 5 and Figure 8 show the same examples, which are also reused in Figure 1. I would also recommend including visual examples of limitations to understand where there are remaining challenges.

How are the geometric condition features provided? Are they extracted from the input image, and if so, with what tools, or are they readily available (as gbuffers) in the synthetic dataset?

Fig 2 is confusing to me. "Transport residual" is included in many places, and it is unclear to me if that is a constant tensor that is being reused, or if it is updated in each pipeline stage. The "Shared Factorized Transport Renderer" are included twice, with arrows from the input. I assume the lower row is a "blow-up" of the upper red box, but having the split arrow from the input makes this confusing, it looks like two parallel blocks. What is "FOTA adapter" referring to? If this is a commonly known architecture from previous work, please add citations or additional details.

Fig 3 is very dense, and hard to parse. Make it bigger (fonts are hard to read). I would also recommend adding more text to explain how it fits into the overall pipeline.

Mixed up notation in Fig 1: It is unclear to me what O_A and O_B represent. In Fig 1, it looks like (L1,VA) and (L1, VB) differs only in the occluder, so I assuime O_A and O_B represents different occluders. This could be stated explicitly early in the paper.

What is "support m" and "context F" referring to (l:160)?

Define non-shadow anchor.

Section 3.2 is hard to parse. I would recommend introducing what projected appearance and geometry features are, and where in the pipeline they are introduced.

Section 3.3 starts abruptly, and I don't know where in Fig 2 or Fig 3 Eq. 4 fits in. Some additional text would help the reader navigate here.

It is unclear to me how large the network is, how many diffusion steps are applied, what diffusion model is used, and what the runtime cost of the proposed method is.

Reviewer_PjKiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReCAST, which predicts missing radiance responses using non-shadow anchors and a shared compositional operator over surface, illumination, and occlusion factors. The response features guide diffusion-based shadow removal, while the framework also supports re-shadowing. The authors evaluate response reuse and image restoration quality through controlled unseen-combination prediction experiments and four standard shadow removal benchmarks.

Critical strengths

The paper investigates the compositional generalization of responses beyond reconstruction quality, which provides a worthwhile research motivation. The ablation studies are broad in scope, and the paper reports favorable results on both controlled compositional prediction tasks and standard shadow removal benchmarks.

Critical weaknesses
  1. The paper is poorly written. The task and contributions are not clearly positioned, and the title, abstract, and core experiments are insufficiently connected. First, the title is poorly chosen: “Beyond Reconstruction” and “Compositional Visual Responses” are overly broad, neither identifying shadow processing as the specific research topic nor clearly conveying illumination–occlusion combination prediction as the central task. Second, the paper starts from single-image shadow removal, whereas the Transport Lattice experiment supporting its core contribution requires three illumination–occlusion observations of the same scene to predict a fourth combination. The paper does not clearly distinguish these two settings at the outset or explain the role of re-shadowing within this framework. Furthermore, the opening introduces numerous broad statements that are not directly related to the task, making it difficult to understand the actual problem being addressed.

  2. The method incorporates numerous priors and modules, but its core technical contribution has not been sufficiently distilled into a coherent and logically developed insight. ReCAST combines depth and normals, DINOv2 features, non-shadow anchors, low-rank multiplicative composition, diffusion decoding, shared operators, closure, and reliability prediction. Although the paper provides ablations supporting the effectiveness of some components, the justification for the overall design remains fragmented. It does not clearly explain how these components jointly address the central challenges of compositional prediction or what new technical insights they offer beyond existing prior-fusion and conditional-generation methods.

  3. The method description and figures are not clearly aligned, making the key inference pipeline difficult to reconstruct. Figure 2 mixes the backbone, response prediction, diffusion injection, and re-shadowing, while the correspondence between FOTA, routing, and the various gates and the shared operator and response decoder in the equations remains unclear. For the target-condition occlusion features Ω⋆\Omega^\star, context F⋆F^\star, soft mask m⋆m^\star, and clean-radiance estimate I^c⋆\hat I_c^\star, the paper mainly emphasizes that they must be derived from permitted observations, without explaining their actual computation with comparable clarity. This hinders both understanding of the model’s mechanism and reproducibility.

  4. The discussion of related work is insufficient. For example, SG-ShadowNet already uses non-shadow regions to guide restoration, yet the paper does not cite this work despite its relevance to the motivation behind receiver grounding. Several other relevant works are also missing from the literature review and comparisons:

    • RASM: Regional Attention for Shadow Removal, ACM MM 2024
    • Shadow Removal Refinement via Material-Consistent Shadow Edges, WACV 2025
    • SoftShadow: Leveraging Soft Masks for Penumbra-Aware Shadow Removal, CVPR 2025
    • Semantic and Geometric Guidance for Shadow Removal via Cascaded Refinement, 2026 preprint
Questions for authors

refer to Critical Weaknesses

Reviewer_pVpfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper reframes shadow removal as Missing-Transport Inversion: instead of regressing a clean image, it predicts the radiance residual R_miss = I_c − I_s as a receiver response and factorizes it into a low-rank multiplicative operator T_R = LN(Σ_r α_r φ^G_r(G) ⊙ φ^ℓ_r(ℓ) ⊙ φ^Ω_r(Ω)) over geometry, illumination code, and occluder descriptor (R=4). Receiver evidence comes from geometry-compatible non-shadow anchors filtered by frozen DINOv2 semantics with reliability-gated fusion; the response conditions a latent-diffusion decoder via FiLM injection, and a forward re-shadowing path plus a multi-path closure loss constrains the operator to be reusable rather than memorized.

Evaluation has two halves. A synthetic Transport Lattice (path-traced 3D-FRONT) factorially crosses illumination {L1, L2, L1+L2} with occluders {O0, OA, OB} and withholds the joint pair (L2, OB), scoring normalized missing-transport error (NMTE) under a target-blind contract; ReCAST reaches 0.589 vs 0.667 for matched cross-attention (Table 2), with 12-/16-family confirmation sets. Standard benchmarks (ISTD+, SRD, INS, WSRD+) report PSNR/SSIM against ten baselines (Table 8), plus an anchor receiver-consistency metric ARC on three of them. A ~35-page appendix documents protocol registers, clean-room target-isolation audits, and robustness reruns.

Critical strengths

The counterfactual protocol design itself. Withholding the joint cell (L2, OB) from a factorial illumination×occluder lattice, with a target-blind input contract and a documented clean-room audit (§A.10, 240 checked cases), is a well-posed test of response reusability that the shadow-removal literature currently lacks. The supporting rigor is real: paired per-scene statistics (165/200 wins, −0.078 paired difference with CI), five scene-disjoint splits, two independent family-count confirmation sets, and an equal-input control showing the gain does not come merely from ReCAST's extra geometry/semantic inputs (Table A.58: −0.064 under image-only input, CI [−0.091, −0.037]). Whatever one concludes about the model, the protocol and its bookkeeping are a contribution others can reuse.

Critical weaknesses
  1. The paper's own "decisive evidence" is synthetic-only and is never compared to any prior work, and the absolute numbers are weak. §1 states plainly: "The lattice recombination protocol is the decisive evidence for our claim; real benchmarks serve as compatibility checks" (line 85). But the entire lattice comparison (Table 2) is against four internal constructions — zero response (1.000), nearest-condition copy (0.852), concatenation (0.731), cross-attention (0.667) — with no published method of any kind evaluated on it. No intrinsic-decomposition, relighting, or inverse-rendering baseline appears, so the central novelty is tested only against the authors' own fusion ablations. Meanwhile ReCAST's 0.589 means ~59% of the zero-response residual error survives, and the margin over the one real competitor construction is 0.078 — small relative to the error that remains. The complementary direction gives no relief: §A.17.5 concedes "the ISTD+, SRD, WSRD+, and INS benchmark comparison remains system-level because the public baseline interfaces are not identical" (line 3943). So the controlled claim has no external comparison, the external comparison is admittedly uncontrolled, and the compositional-reuse claim has zero real-image validation. That gap is architectural to the paper's evaluation design, not a missing experiment.

  2. The headline benchmark result is both confounded and internally contradicted. Table A.29 records that all ten baselines are scored from "official release" checkpoints while the ReCAST row is an "author-trained checkpoint". The symptoms are visible in the numbers: HomoFormer scores 20.78 dB and DeS3 18.45 dB on WSRD+, i.e. 4.7–7 dB below ShadowFormer/DenseSR on the same split — the signature of non-retrained weights evaluated out of domain. The claim of "highest PSNR on all four benchmarks" (+0.46/+0.55/+0.70/+0.63 dB) is therefore not like-for-like, and the WSRD+ margin in particular is argued against a field whose weakest entries are unadapted. Worse, the same declared run is reported with two different values: Table 8 gives SRD SSIM 0.971, while Table A.28 — which states "the same declared ReCAST evaluation run supplies the full-image fidelity scores" — gives SRD SSIM 0.968. The difference is decision-relevant: at 0.968 ReCAST falls below ShadowDiffusion (0.970), PhaSR (0.970) and OmniSR (0.969) instead of ranking second-best. The response metric has the same problem at larger scale: "full ReCAST" NMTE is 0.589 in Tables 2/3/4 but 0.270 in Table 7, and Table A.30 registers six distinct full-model values (0.61, 0.589, 0.54688, 0.270, 0.55, 0.0936) across different populations with the caption conceding that "values across rows are not a leaderboard." Table 7's caption does say "a separate controlled protocol," but no mapping between the scales is ever given, so there is no single number a reader can hold the method to.

Questions for authors
  1. Table 8 versus Table A.28: is ReCAST's SRD SSIM 0.971 or 0.968? Since A.28 states that both come from the same declared evaluation run, please explain the discrepancy and state which value belongs in Table 8 - it changes ReCAST's SRD SSIM ranking from second-best to fourth.

  2. Checkpoint provenance: were any baselines retrained or fine-tuned on the common split, or are all ten scored from released weights as Table A.29 indicates? Given HomoFormer at 20.78 dB and DeS3 at 18.45 dB on WSRD+, can you provide at least one baseline retrained under ReCAST's budget on WSRD+, so that the +0.63 dB claim is comparable?

  3. Prior art on the lattice: can any existing method with an explicit illumination or reflectance factorization, or a relighting model, be run on the Transport Lattice? Without one, what rules out the conclusion that the lattice is simply hard and that 0.589 reflects task difficulty rather than a reusable response operator?

  4. Table 6 internal consistency: ReCAST's shadow-core RMSE is 0.002568 (approximately 52 dB) while its full-image RMSE is 0.025052 (32.0 dB) and its shadow-support RMSE is 0.041691 - 16x worse than the core it contains. Taking the support fraction from the No-write row (approximately (0.077783 / 0.241589)^2 = 10.4%), even zero non-shadow error gives a full-image RMSE near 0.013, not 0.025. How is a near-zero core error obtained from a diffusion-rendered output, and where does the remaining full-image error live if not in the support the method claims to restore?

  5. Target isolation: Section A.10 states that the sentinel check "does not exclude precomputed features containing target information or establish training/test disjointness". What positively establishes that the cached geometry and semantic features for the withheld (L2, OB) cell carry no target information?

  6. Benchmark training: the paper never describes how the G, l, and Omega factors are obtained, or how the model is trained, on the four real benchmarks, which carry no illumination or occluder annotations. Please specify this - it determines whether Table 8 evaluates the compositional operator at all.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.