acceptodds
Withdrawn from ICLR 2027

ReVision: Reprogramming Image Generators for Dense Visual Perception

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent image generators exhibit rich visual priors that can support dense visual perception. However, existing approaches typically encode multiple perception tasks into a shared parameterization, making joint training sensitive to the mixture and weighting of heterogeneous data and prone to cross-task interference. To address this issue, we introduce ReVision, an instruction-conditioned parameter generation framework built on a frozen multimodal understanding model and a frozen image generator. Given an image and a free-form instruction, the understanding model jointly encodes them into a multimodal image–instruction representation, which a trainable parameter generator maps to transient low-rank updates for the generator. Each image–instruction pair induces a distinct set of transient prediction parameters, enabling instance-specific adaptation while both pretrained backbones remain unchanged. ReVision requires no task-specific heads, explicit task identifiers, or stored adapters. A shared RGB output interface unifies segmentation, depth estimation, and surface-normal prediction. Across diverse dense prediction benchmarks, ReVision achieves competitive performance with state-of-the-art methods while supporting flexible task specification through natural-language instructions. Our analysis further reveals substantial gradient disagreement across tasks, highlighting the conflict introduced when heterogeneous objectives share a single parameterization. Meanwhile, the generated updates exhibit structured task- and input-dependent organization in weight space, further validating instance-level parameter generation for multi-task perception.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Generation, Dense Visual Perception, Parameter Generation, Instance-Level Prediction, Multimodal Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EjKVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReVision repurposes a frozen image generator for dense visual perception through input-conditioned parameter generation. A frozen Qwen3.5-9B model encodes the reference image and instruction, and a query-based hypernetwork generates rank-16 LoRA factors for 112 injection sites in a frozen FLUX.2-klein-base-9B generator. These updates are generated once per sample and remain fixed throughout denoising. Only the hypernetwork is trained, using a flow-matching objective and RGB-encoded targets for semantic, instance, and referring segmentation, depth, and surface normals.

Critical strengths
  1. A coherent approach to input-conditioned adaptation of generative perception models. The design combines multimodal understanding with low-rank parameter generation while preserving both pretrained backbones. Generating updates once per input provides a concrete mechanism for conditioning the predictor beyond a fixed task adapter, without requiring per-example optimization at inference.

  2. Broad empirical evaluation with useful within-backbone comparisons. The paper evaluates five perception tasks and includes both task-specific and shared multi-task LoRA baselines using the same image generator. Improvements are particularly substantial in several geometry settings: for example, NYUv2 depth AbsRel improves from 5.7/5.9 for the two LoRA baselines to 4.0. The condition-routing ablations also provide evidence that the learned parameter generator uses sample-specific information rather than merely producing an unconditional update.

Critical weaknesses
  1. Unclear semantic-segmentation protocol. Appendix D states that Cityscapes prompts list the classes present in each image, but does not explain how these lists are obtained. If derived from ground truth, they provide privileged information that may make the 72.75 mIoU result incomparable with standard evaluations. Results using a fixed list of all 19 classes are needed.

  2. Insufficient isolation of the proposed mechanism. ReVision adds a 918.3M-parameter hypernetwork and multimodal conditioning, without clearly matching baseline capacity or access to the same features. Thus, the gains cannot be confidently attributed to instance-specific weight generation. The gradient analysis also does not directly measure conflict in the shared hypernetwork, and latency/memory costs remain unquantified.

Questions for authors
  1. Cityscapes protocol: How are the per-image class lists constructed? Are any ground-truth annotations used? Please report results with a fixed prompt containing all 19 classes and clarify whether the external baselines receive equivalent category information.

  2. Can you measure cross-task gradient interactions in the shared hypernetwork parameters and compare them with a static multi-task model under matched samples and noise? How should the current gradients with respect to sample-dependent LoRA factors be interpreted as evidence about shared-parameter optimization?

  3. Evaluation and computational cost: Please specify the subsets and settings behind Table 6, whose reference results differ from the main tables, and explain Qwen’s instance-existence role and the instance-palette decoding procedure on SA-Co/Gold. What are the end-to-end latency, peak memory, and training cost relative to the static-LoRA alternatives?

Reviewer_sJNeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReVision trains a hypernetwork that conditions on a multimodal model’s (Qwen3.5-9B) representation and produces LoRA weights for an image generator. The paper demonstrates that this allows the image generator to achieve competitive performance on several vision tasks, including segmentation, depth and surface normal estimation. The authors postulate that this is because the hypernetwork predicts different weights for different tasks/prompts, which allows it to mitigate the gradient conflict that shared parameters would have been subject to.

Critical strengths
  • The performance of the method is respectable on several tasks; it beats vision banana on semantic segmentation, and matches or beats several zero-shot models on the other tasks as well. It outperforms both LoRA baselines.
Critical weaknesses
  • My main problem is with the analysis. The main point of this paper is that using a hypernetwork allows for generating different weights for different tasks, mitigating the gradient conflict if we were to use shared parameters (e.g., LoRA) for all tasks. This ignores that the gradient conflict could move to the hypernetwork. While the conflict analysis in section 4 shows that there is gradient conflict between gradients of generated LoRA parameters, nothing ablates or analyzes the gradient conflict in the shared parameters of the hypernetwork that exist during optimization. The paper also doesn’t directly measure gradient conflict in the multi-task LoRA baseline directly.
  • The improvement of ReVision over both LoRA baselines is confounded, because the parameter generator has significantly more parameters than just using LoRA weights for all tasks (for multi-task LoRA). It isn’t a fair comparison with task-specific LoRA either, because the compute is more.
Questions for authors
  1. To establish that conflict due to heterogeneous objectives is the causal factor for performance degradation, it should explicitly be shown how the hypernetwork’s design helps mitigate the gradient conflict (experienced by multi-task LoRA). The clustering of the updates based on task category alone doesn’t show why this method works; for example, this doesn’t disprove the existence of shared weights that obtain a greater performance on all tasks.
  2. I would also recommend removing/ablating the confounders to address the second weakness.
  3. A third confounder with the baselines is the image generative model being implicitly conditioned on the features of the multimodal understanding model via the parameter generator.

I will increase my score if the weaknesses are addressed.

Reviewer_Q1NCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ReVision generates image- and instruction-conditioned LoRA updates for a frozen image generator. A frozen multimodal model supplies the conditioning features, while a trainable hypernetwork produces the updates. Five dense-prediction tasks share an RGB output interface. Experiments compare ReVision with fixed LoRA baselines and existing perception methods.

Critical strengths

1 This paper is easy to follow and understand.

2 The framework supports five perception tasks through one RGB interface. Both pretrained backbones remain frozen, and task behavior is specified through language.

3 Comparisons using the same image generator show clear improvements over multi-task LoRA. These include Cityscapes mIoU increasing from 64.68 to 72.75 and SA-Co/Gold cgF1 increasing from 30.71 to 40.17.

Critical weaknesses

1 The method combines conditional LoRA generation with an RGB prediction interface, both discussed in the related work. The main distinction is joint image–instruction conditioning. Comparisons with task-conditioned hypernetworks or adapter routing are missing, leaving the value of instance-specific weight generation insufficiently established.

2 The task-conflict motivation is not adequately validated. Figure 3 compares gradients at different generated LoRA parameters, rather than at a common shared parameter state. The hypernetwork itself is still shared across tasks. These measurements do not show that ReVision reduces conflict in the parameters actually optimized during training.

3 The Cityscapes evaluation may provide privileged information. Appendix D states that each instruction lists the classes present in that image. The source of this list is not explained. If it comes from ground-truth annotations, the reported score is not directly comparable with methods that receive no image-specific class list.

4 The gains do not clearly justify the added capacity. The parameter generator contains 918.3M trainable parameters, but no capacity-matched baseline is provided. On RefCOCOg, the gain over task-specific LoRA is only 0.7 cIoU (53.1 to 53.8). Runtime and memory comparisons are also absent. The contribution of instance-specific adaptation remains difficult to separate from additional model capacity.

Questions for authors

1 How does ReVision compare with a task- or instruction-conditioned hypernetwork of similar capacity? Please also report inference time and memory.

2 Does ReVision reduce cross-task gradient conflict in the shared hypernetwork parameters compared with the shared LoRA baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.