acceptodds
Withdrawn from ICLR 2027

IntrinSync: Joint Intrinsic Decomposition and Reciprocal Rendering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Inverse rendering decomposes an image into intrinsic properties such as appearance, illumination, geometry, and material, yet these properties are inherently interdependent. A reliable decomposition should produce intrinsic maps that are not only individually plausible, but also mutually compatible in explaining the image. However, existing methods either model intrinsic channels in isolation or treat inverse and forward rendering as separate processes, leaving the interdependence underexploited. In this paper, we introduce IntrinSync, a unified framework that captures this interdependence through joint-channel modeling and reciprocal inverse–forward rendering. At the channel level, we jointly decompose an input RGB into albedo, shading, surface normal, roughness, and metallic maps through a 1-to-N mapping, enabling information exchange across channels throughout generation. At the process level, we establish inverse–forward reciprocity through a dual cycle-consistent objective that aligns corresponding predictions across a closed loop. Experiments on three datasets demonstrate that our method achieves competitive intrinsic estimation and forward rendering performance, improving coherence and physical consistency. Beyond decomposition, IntrinSync provides a physically grounded interface for image editing, allowing intrinsic properties to be explicitly manipulated and rendered back into RGB images.

Primary area:
generative models
Keywords:
Intrinsic Decomposition, Diffusion Transformer, Cycle Consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P8gCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses inverse rendering, i.e., decomposing an image into intrinsic properties such as appearance, illumination, geometry, and material. Both the inverse and the forward rendering directions are obtained by fine-tuning Qwen-Image-Layered. In the inverse direction, the five intrinsic channels are denoised jointly within a single token sequence, so that the channels exchange information through attention; in the forward direction the roles are swapped and the RGB image is generated conditioned on the five intrinsic latents, with missing channels set to zero so that datasets with incomplete annotations can be used. Each direction uses its own rank-32 LoRA on a shared frozen backbone.

The core novelty claimed is the dual cycle: along the two reciprocal paths I→X’→I’ and X→I→X, the velocity predictions obtained from cycle-generated inputs are aligned with those obtained from real inputs. In practice, the output of the inverse pass is used as the condition of the forward pass, and the output of the forward pass as the condition of the inverse pass.

For validation, all quantitative evaluation is carried out on test subsets of three synthetic datasets. The metrics are PSNR/LPIPS for each channel (SSIM and RMSE additionally for albedo; mean angular error and the percentage of pixels within 11.25° for normals), together with PSNR/LPIPS of forward rendering from ground-truth intrinsics and of RGB reconstruction from predicted intrinsics (Table 1). The paper reports the best albedo results on all three datasets, “competitive” results on the other channels, and a lead across the board on MatrixCity.

Critical strengths
  1. A single base model carries both decomposition and rendering and can be trained on multiple datasets with heterogeneous annotations; the design is simple. The ablation shows that joint denoising improves consistently over per-channel prediction on all three attribute types: albedo +1.27 dB, irradiance +0.91 dB, normals −2.32° (Table 2, a→c).
Critical weaknesses
  1. The model itself is a rather limited improvement over prior work, and the central claim has no direct evidence. The contribution of the dual cycle is marginal: the dual cycle designed in this paper improves albedo by only +0.07 dB over joint-only, and it is worse than the vanilla cycle inherited from Ouroboros in LPIPS and in RGB reconstruction. Once the dual cycle is removed, the only gain with evidence behind it comes from the multi-layer joint generation mechanism that the backbone already provides, not from a design specific to this paper.

  2. The main comparison is unfair, and the results themselves are mixed. The model is trained on the training splits of the same three datasets used for testing (including 20K MatrixCity images), whereas the baselines are run from their official checkpoints; the real images are supervised with pseudo-labels produced by Ouroboros, which is then used as a baseline. Public baselines trained on exactly these two datasets, Marigold-IID and PRISM, are not included in the comparison. The large lead on MatrixCity (where all baselines have normal errors of 22 to 43°) looks more like because of in-domain training; on the two indoor datasets that the baselines may have seen, the method falls behind on normals (Hypersim 18.76° vs. 13.29°), on InteriorVerse roughness/metallic (14.95 vs. 16.20 and 17.18 vs. 19.74 dB), and on Hypersim RGB reconstruction (17.15 vs. 17.64 for RGB↔X). Table 1 therefore cannot serve as evidence for the claims of “competitive” performance and a “mutually compatible” representation, which directly determines my soundness score.

Questions for authors
  1. On which datasets was each baseline’s official checkpoint trained? Please add baselines trained on the same data (Marigold-IID, PRISM, or Ouroboros/RGB↔X retrained on your 55K synthetic images); otherwise Table 1 cannot support the “competitive” conclusion.

  2. What are the results without the real images pseudo-labeled by Ouroboros? How much of the improvement over Ouroboros on real images comes from distillation?

  3. Why are the normals on Hypersim still 5.5° worse than Ouroboros despite training on Hypersim? Is the 22 to 43° normal error of all baselines on MatrixCity a convention mismatch?

  4. Please measure cross-channel consistency directly, for example by reconstructing the input from the predicted intrinsics with a fixed physically based renderer, or with a leakage metric, rather than with each method’s own learned renderer.

Reviewer_dt42OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focus on intrinsic decomposition and forward rendering in a unified architecture. For intrinsic decomposition, it jointly predicts albedo, shading, normals, roughness, and metallic maps conditioned on an RGB image. For forward rendering, it reverses the roles of conditions and targets to synthesize RGB images from intrinsic maps. In addition, a Dual Cycle-Consistent Tuning loss further couples the two processes by aligning predictions across reciprocal paths. Evaluation on comprehensive datasets (Hypersim, InteriorVerse, and MatrixCity) shows consistent improvements. The paper also demonstrates real-image decomposition and intrinsic-guided editing.

Critical strengths
  • The paper is well written and easy to follow. The figures clearly illustrate the overall framework and the proposed training objective, making the method accessible to readers.
  • The problem is clearly stated, the model design is reasonable, and the evaluation is comprehensive. Compared with 1-to-1 decomposition, the 1-to-N approach is well motivated, as it allows information exchange among interdependent intrinsic channels. The evaluation covers both indoor and outdoor scenes across three datasets, with ablation studies supporting the benefits of joint decomposition.
  • The paper proposes a dual cycle-consistent training loss that couples intrinsic decomposition and forward rendering. By aligning predictions across reciprocal paths, the objective encourages consistency between the two processes beyond independent direct supervision.
Critical weaknesses

Based on my own experiments, both forward rendering and intrinsic decomposition can benefit substantially from the pretrained backbone (Qwen-Image-Layered in this paper). Therefore, controlled ablations are important to distinguish the contributions of the proposed designs from those of the backbone.

Main question:

A concern about dual cycle-consistent tuning.

Let II denote an image, cc its intrinsic maps, R\text{R} the forward renderer, and InvR\text{InvR} the inverse renderer. The two predicted conditions are: c′=InvR(I),I∗=R(c).c'=\text{InvR}(I), I^*=\text{R}(c).

I agree that the same image can admit multiple valid intrinsic explanations. Therefore, training the renderer to reconstruct II from the predicted intrinsics c′c' is reasonable, provided that c′c' remains a valid explanation of II (which means the InvR\text{InvR} is good enough to generate reasonable results). Similarly, using the rendered image I∗I^* to train inverse rendering is reasonable if it preserves the information needed to recover the corresponding intrinsics.

However, Dual cycle-consistent tuning could improve both models through mutual correction, as intended, but it is also possible to accumulate each other's errors, particularly early in training when the predicted conditions are unreliable. For example, the renderer might learn to disregard inaccurate intrinsic conditions rather than encourage the inverse renderer to produce more accurate maps. The resulting pair could become more self-consistent without becoming more physically correct.

Furthermore, Eq. (2) changes both the conditioning input and the noisy target state when aligning velocity predictions. It is therefore unclear which part of the improvement comes from condition consistency and which comes from aligning the generated targets. Please clarify how the training prevents mutual error accommodation and why this objective should specifically reduce intrinsic leakage.

Suggested experiment.

Starting from the same directly supervised checkpoint, freeze the forward renderer and fine-tune only the inverse renderer, retaining its direct supervision. Compare the rendering velocities under ground-truth and predicted intrinsic conditions while keeping the noisy RGB target, timestep, and noise identical: Lcond=Et,ϵ[∥vR(zIt,t;c′)−sg[vR(zIt,t;c)]∥22],c′=InvR(I).\mathcal{L}_{\text{cond}} = \mathbb{E}_{t,\epsilon}\left[\left\|v_{\text{R}}(z_I^t,t;c') - \text{sg}\left[v_{\text{R}}(z_I^t,t;c)\right]\right\|_2^2\right], c'=\text{InvR}(I).

Compare this variant with dual-cycle training under matched settings, reporting intrinsic accuracy through the same frozen renderer. This would help distinguish better decomposition from the renderer learning to compensate for decomposition errors.

If my reasoning is flawed, please point this out directly, in which case the experiment would be unnecessary. Otherwise, convincing results addressing this concern would validate the design's value and I will raise my rating.

Minor issues:

  • Technical details about conditioning: How are the conditions injected into the model? Does the model follow Qwen-Image-Layered in using 3D RoPE (l,y,x)(l, y, x), with the ll-axis distinguishing different conditioning maps?

  • Unified multimodal learning is intuitive. Previous work, such as PRISM (arxiv 2504.14219), has already explored intrinsic decomposition through a shared representation. Therefore, I suggest that the authors focus on validating the Dual Cycle-Consistent Tuning component.

Questions for authors

See weaknesses.

Reviewer_u6mrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IntrinSync jointly estimates albedo, shading, surface normals, roughness, and metallic maps from a single RGB image, and supports RGB synthesis conditioned on these maps and text. It adapts Qwen-Image-Layered with attention across intrinsic channels, using a shared pretrained backbone with separate adapters for decomposition and rendering. Its main training addition is a dual-cycle objective that aligns denoising predictions across reciprocal paths, alongside direct supervision. Training combines synthetic annotations with real images pseudo-labeled by Ouroboros. Evaluation covers three synthetic benchmarks, a controlled Hypersim ablation, and qualitative real-image decomposition and editing.

Critical strengths
  1. The controlled ablation supports joint prediction over independent per-channel prediction. In Table 2, joint prediction improves all six decomposition metrics: albedo PSNR increases from 17.915 to 19.184, shading/irradiance PSNR from 17.402 to 18.307, and normal error decreases from 25.712 to 23.390 degrees. This provides concrete evidence that interaction across intrinsic channels is useful in this implementation.

  2. The complete system achieves substantial gains on MatrixCity. Compared with Ouroboros, albedo PSNR increases from 22.034 to 27.867, normal error decreases from 23.803 to 13.254 degrees, and roughness, metallicity, and rendering results also improve. These are meaningful gains, even though the indoor results are less consistent. The qualitative editing examples further demonstrate the utility of the recovered maps as controls for image synthesis.

Critical weaknesses
  1. The methodological advance over the closest work appears incremental. PRISM already jointly generates intrinsic maps with cross-modal transformer attention and supports both decomposition and conditional RGB generation. Ouroboros already couples inverse and forward rendering through bidirectional cycle consistency. IntrinSync's prediction-level dual-cycle objective is a distinct addition, but the broader joint/unified formulation is substantially anticipated by these works. The discussion should distinguish shared architecture and joint prediction from the specific new training constraint more precisely. PRISM is cited but absent from the quantitative comparison, leaving the benefit over this particularly relevant framework unclear.

  2. The evidence for the specific dual-cycle contribution is limited and mixed. In Table 2, relative to the vanilla cycle, dual-cycle training improves albedo PSNR from 18.781 to 19.250 and shading/irradiance PSNR from 18.211 to 18.633. Normal mean error improves from 19.365 to 19.039 degrees, with a larger improvement in the fraction below 11.25 degrees. Forward-rendering PSNR changes only from 18.025 to 18.092. Meanwhile, albedo and shading LPIPS worsen, and RGB reconstruction PSNR drops from 19.767 to 18.132. The authors acknowledge this tradeoff and provide qualitative leakage examples in Figure 7; however, the controlled ablation is limited to a Hypersim subset, and the reduced-leakage explanation is not directly quantified. Agreement between a learned decomposer and a caption-conditioned learned renderer does not by itself establish physical correctness. Strong full-system results therefore do not yet establish a substantial, general benefit from the principal new objective. This uncertainty, together with the incremental methodological contribution, leads me to weak rejection.

Questions for authors
  1. Can you provide stronger evidence that the dual-cycle objective improves decomposition rather than primarily changing the tradeoff among metrics? The most useful additions would be the same controlled vanilla-versus-dual comparison on another benchmark, with backbone, data, and training budget held fixed, and a direct test of reduced leakage. For example, a small synthetic set with fixed camera and materials under varying illumination could measure both albedo accuracy and its stability across lighting changes. Please keep the caption protocol consistent across variants and report variation across runs for the small gains. These results could materially change my assessment. Existing qualitative examples should be retained; some additional image grids could move to the supplement to make room for this evidence.

  2. Please clarify the implementation of Equation 2: how are the intermediate predictions generated, what noise is shared between the two terms in each comparison, and which branches receive gradients or are detached? A short training algorithm would help establish precisely what constraint is optimized and make the main contribution reproducible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.