acceptodds
Under review as a conference paper at ICLR 2027

From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models

Abstract

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Representative generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets. This target-side interface is inherited from RGB synthesis rather than derived from the task. Unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered. Our starting point is that a pretrained DiT already organizes RGB inputs through a patchtokenpatch lattice on the image plane. Each token therefore indexes a fixed output patch whose channels can carry task-native quantities instead of RGB appearance. We instantiate this idea as ReChannel. We retain the VAE encoder to preserve the DiT's input distribution, remove the target-side decoder, adapt the frozen DiT with per-task LoRA, and map each token to its pixel-space patch using a shared token-local linear head. The head has about 33K parameters, performs no spatial mixing, and is trained directly with pixel-space, task-native supervision. Using FLUX-Klein, we evaluate ReChannel on six dense prediction tasks and more than a dozen benchmarks. This minimal interface achieves state-of-the-art results on trimap-free matting, KITTI depth, and referring segmentation, while remaining competitive on normals, saliency, and pose. Under a matched 4B backbone, it is more accurate and up to 2.48 faster than latent, VAE-decoded, and edit output interfaces, while larger heads provide no further gains. These results show that dense prediction can inherit the prior of generative pretraining without inheriting its output interface.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dense prediction, text-to-image model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Qt1hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores ReChannel, a more efficient approach for adapting the existing text-to-image (T2I) model FLUX to dense prediction tasks without using a VAE decoder or training an additional decoder. ReChannel retains the VAE encoder on the input side, adapts the FLUX-Klein backbone using task-specific LoRA, and replaces the VAE decoding process on the output side with a shared, token-local linear projection layer. By mapping each token to a pixel patch corresponding to the target task and assembling these patches into the final prediction, the method enables loss computation directly in pixel space. The proposed approach achieves competitive performance and faster inference on tasks such as depth estimation, normal estimation, and image matting.

Critical strengths

The proposed ReChannel method demonstrates good generalizability while improving the efficiency of dense prediction tasks. Since adaptation for each task is performed independently, the method effectively extends the applicability of a single pretrained generative model to a broader range of tasks. The results demonstrate that competitive performance can be achieved without relying on a task-specific spatial decoder.

For the 4B-parameter model, the proposed method significantly outperforms approaches based on latent targets and pixel-supervised frozen VAE decoding in both accuracy and inference speed on the image matting task. These results demonstrate that ReChannel is better suited for tasks requiring fine-grained predictions.

Critical weaknesses

The 33K parameters emphasized in the abstract apply only to single-channel tasks, rather than to the prediction heads of all tasks. Furthermore, this number does not include the parameters introduced by LoRA. The paper does not provide a complete breakdown of the total number of trainable parameters.

The proposed method essentially follows the paradigm of combining a pretrained backbone, task-specific adaptation, and a linear prediction head, which has already been widely explored in computer vision. The main contribution is therefore the application of this established design paradigm to FLUX-Klein. However, the paper does not sufficiently demonstrate whether the observed performance improvements primarily come from the stronger backbone rather than the ReChannel design itself. In particular, compared with Edit2Perc in Table 1, which uses FLUX.1, the FLUX-Klein backbone employed in this work may already provide inherent performance advantages.

The ablation results in Table 5 show that the Frozen + Linear Head configuration performs poorly, indicating that the pretrained generative model cannot directly solve the target tasks through a simple linear projection without task-specific adaptation. The use of LoRA therefore appears to be essential. However, this observation also weakens the authors' central intuition that the pretrained token field already contains task-relevant information that can be directly read out.

Questions for authors

Could the authors provide a complete comparison of the actual number of trainable parameters? If experiments with different LoRA configurations have been conducted, could the authors also report the parameter counts and corresponding performance for these configurations?

Given that the final predictions are obtained by adapting spatial features through LoRA and mapping them to pixel values using a linear head, why is a generative DiT backbone necessary? Could a DINOv3 + LoRA + Linear Readout configuration achieve comparable performance?

Reviewer_nnpZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose directly predicting the dense prediction output in image space rather than latent space as in previous work. This can help avoid the reconstruction error of VAE especially for fine details. To achieve that goal, they train a linear head to map the spatial latent to pixel space. Experiments show that direct mapping outperform latent space diffusion and achieve strong results on several benchmarks.

Critical strengths
  • The proposed approach is simple and effective, it achieves strong performance across numbers of tasks.
  • The experiment is well designed and verify the effectiveness of predicting in pixel-space compared to VAE
Critical weaknesses
  • While the author demonstrated that predicting on pixel-space is better than VAE, these kind of results has been shown by various prior work e.g., [1,2]. The main different is that they used SD1.5/SD2 so the design is somewhat different ( e.g., linear layer + upsampling). The pixelshuffle approach used here is standard and straighforward for DiT backbone.
  • The baseline in Table 1/2 doesn't seem fair to me. Baseline such as have substantial smaller number of parameters (around 1B) while the reported baseline of the author has 4/9B.
  • Some writing is confusing and hard to interpret:
    • "Our view separates three roles that are often coupled in generative adaptation: the input interface, which maps RGB observations into the representation expected by the pretrained model; the backbone representation, which organizes the observation into a spatial token field; and the output interface, which converts that field into the task target.". Is it VAE -> DiT backbone -> per-task output head?
    • "The backbone itself already organizes the image into this spatially indexed field, so the remaining output interface need not reconstruct spatial structure; the minimal readout admitted by the lattice is a reinterpretation of each carrier’s channels." Do you mean pixelshuffle after pretrained DiT is sufficient?

[1] Three Things We Need to Know About Transferring Stable Diffusion to Visual Dense Prediction Tasks

[2] Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

Questions for authors
  • It would strengthen the approach if the author can extend the ablation in Table 5 to more tasks e.g., segmentation, depth estimation etc
Reviewer_JuaSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReChannel, a dense-prediction interface built on FLUX-Klein. It retains the VAE encoder as the input pathway, freezes the pretrained backbone, and trains a task-specific LoRA together with a token-local linear readout. Each adapted DiT token is mapped directly to a pixel-space output patch, without a target-side VAE decoder or an output-side spatial decoder.

The method is evaluated on six tasks: monocular depth, surface-normal estimation, trimap-free matting, referring segmentation, saliency detection, and human pose estimation. The paper reports strong results across more than a dozen benchmarks and presents a matched-backbone diagnostic comparing ReChannel with alternatives such as latent prediction followed by VAE decoding, a frozen VAE decoder with pixel-space supervision, an edit-style formulation, a larger convolutional head, and full fine-tuning.

Critical strengths
  1. The paper distinguishes the input representation from the output interface and questions whether a dense target should pass through an RGB-trained target decoder. This is a meaningful design question for adapting generative models to perception.

  2. The proposed readout has no cross-token mixing or spatial decoding. Under the reported setup, it adds little inference overhead relative to the backbone. This simplicity makes the method easy to understand and potentially useful as a baseline for future work.

  3. The experiments span geometry, matting, language-conditioned masks, saliency, and pose. The paper also compares against strong task-specific systems; for example, DAv2 is included in the depth comparison.

Critical weaknesses
  1. The scope of “One RGB prior, all dense fields” is overstated. The experiments support a more limited claim: a shared FLUX-Klein backbone can serve as the initialization for six evaluated dense-prediction tasks, with a separately trained LoRA adapter and readout head for each task. They do not show that a single task-agnostic model handles all dense fields, or that the approach generalizes to arbitrary output types or other backbone families. In particular, the evaluated referring-segmentation task predicts a single text-conditioned mask; it does not establish performance on standard multi-class semantic or panoptic segmentation. The authors should either narrow the claim—for example, to “One FLUX-Klein backbone across six evaluated dense-prediction tasks”—and clearly state that task-specific adapters are used, or provide additional evidence for the broader scope implied by the current wording.

  2. The role of LoRA is insufficiently analyzed. The paper shows that task-specific LoRA is necessary for the proposed pixel-space prediction pipeline, but does not investigate how LoRA enables the adapted DiT to support dense prediction. It therefore remains unclear how much adaptation capacity is required and where in the transformer the task-relevant prediction capability emerges. I encourage the authors to conduct the following analyses: (1) evaluate LoRA ranks �0��0� to measure the effect of rank and identify the minimum effective rank; (2) after training, freeze the adapted model and train the same readout on features extracted from different transformer blocks, to determine at which depth task outputs become decodable; and (3) train LoRA only in early, middle, or late blocks, using comparable trainable parameter budgets, to assess which parts of the DiT require task-specific adaptation. These experiments would provide a clearer account of LoRA’s role in enabling pixel-space dense prediction.

  3. The generality of the proposed approach is not established. All experiments use FLUX-Klein (4B and 9B), so the results do not show that the proposed readout generalizes beyond the FLUX family. The authors should evaluate the same approach on other pretrained text-to-image backbones, such as Stable Diffusion 3.5; including SDXL would further test whether the approach extends beyond DiT-based architectures. Otherwise, the claims should be explicitly limited to FLUX-Klein.

  4. The semantic-segmentation evidence is also insufficient for the claim of “all dense fields.” The paper evaluates referring segmentation as a single-channel mask prediction, but does not test standard multi-class semantic segmentation, where every pixel must be assigned to one of many classes. The authors should evaluate the method on a high-class-count benchmark, such as ADE20K-Full (847 classes in the commonly used train/validation setting), and report standard segmentation metrics. This would test whether the proposed readout remains effective when the output requires substantially more channels than the tasks currently evaluated.

Questions for authors

Despite above concerns, I find the authors’ idea compelling and am positively inclined toward the paper. My current recommendation is weak rejection because several experiments are still needed to establish the scope and generality of the conclusions. I hope the authors will address these concerns in their response; if they can provide convincing evidence that resolves them, I would be willing to raise my final score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.