acceptodds
Under review as a conference paper at ICLR 2027

RefGC-SR²: Reference-guided Super-Resolution and Refinement of AI Generated Content

11 views

Abstract

Reference-guided generation (e.g., object compositing, customization) has progressed rapidly, yet current pipelines share a fundamental limitation: the object-centric high-resolution reference image (HRRI) provided by users is downsampled to a fixed low-resolution (LR) before being fed into the model, so the fine-grained details are discarded before the output is even produced. In addition, the generation step then introduces its own artifacts (e.g., identity distortion) on top of this loss. Existing reference-guided generated content refinement (RefGCR) methods can correct some of these artifacts but still operate in the LR domain; reference-guided super-resolution (RefSR) methods recover resolution but assume natural-image degradations and ignore the artifact distribution of generative pipelines. To address both gaps in a single formulation, we introduce a new task: reference-guided generated content super-resolution-refinement (RefGC-SR²), where the original HRRI is reused at the post-processing stage to recover lost details, refine generative artifacts, and upscale the output simultaneously. We construct the first real-world triplet data generation pipeline for this RefGC-SR² task, training a diptych-conditioned generator to synthesize paired low-quality anchors that public pretrained models cannot provide. We further present a frequency-aware diffusion transformer model for RefGC-SR² Model that selectively injects fine details from the HRRI while removing generative artifacts. Extensive experiments demonstrate that our RefGC-SR² Model successfully (i) refines the object identity faithfully with respect to the reference, and (ii) recovers high-resolution details, so that the final result is significantly higher quality and practically more usable compared to existing RefGCR and RefSR baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Super-Resolution, Image Refinement, Generative Model, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Yqy7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RefGC-SR², which aims to jointly perform super-resolution and generative artifact refinement on low-resolution generated images with the guidance of high-resolution reference images. To address the lack of training data, the authors propose DipRefGC, which decouples object appearance and pose to construct LRGI-HRRI-HRGT triplets. Furthermore, the authors introduce a frequency-adaptive mixture of LoRA experts and a frequency-adaptive loss to model global structure and fine-grained details. Experimental results demonstrate that the proposed method outperforms existing baselines in terms of multiple quantitative metrics and visual quality.

Critical strengths

1.The data construction strategy of DipRefGC is well motivated. By decoupling object appearance and pose, it generates training samples that contain generative artifacts while maintaining reasonable pose consistency with the target images, providing effective supervision for the proposed task.

2.The designs of FreqMoLE and the frequency-adaptive loss are reasonable and well motivated. They leverage the frequency characteristics of different network layers to encourage the modeling of global structure and the recovery of fine-grained textures.

3.The experimental evaluation is relatively comprehensive. The authors compare their method with various super-resolution and image refinement approaches and provide ablation studies, generalization experiments on outputs from real-world generative models, and a user study to demonstrate the effectiveness of the proposed method.

Critical weaknesses

1.Although the ablation studies demonstrate the effectiveness of FreqMoLE, there is still limited evidence that the LF and HF experts learn their intended frequency specializations. An analysis of expert responses and learned gating weights would help better validate the proposed frequency decomposition mechanism.

2.The high-frequency loss matches channel-wise statistics between the output and HRRI to avoid cross-view pixel alignment. However, statistical consistency does not necessarily ensure spatially accurate texture recovery, especially for structured details such as text, logos, and complex patterns.

3.While the authors evaluate common reference degradations such as blur, JPEG compression, and downsampling, the model’s robustness to more challenging cases remains unclear. Further analysis of occlusions, incorrect reference details, or irrelevant information would help assess whether the model can reliably control detail transfer.

Questions for authors

Please refer to the weaknesses above.

7 comments

To reviewer Yqy7

Thank you very much for your valuable comments that helped us significantly improve this submission. Below are detailed point-to-point responses about how we address your questions.

[W1] Frequency role of FreqMoLE

We thank the reviewer for suggesting a more direct analysis of the LF and HF experts. To investigate their utilization across network depth, we analyzed their gate-weighted Q/K/V LoRA responses across 57 layers using RefGC-SR² Benchmark and three denoising steps.

For each expert, we computed the squared magnitude of the 2D Fourier transform over spatial image tokens after applying its gate and excluded the DC component. Using a normalized frequency-radius cutoff of 0.25, we calculated each expert's share of the sum of the two experts' individual energies in each frequency band. We then divided the layers into four approximately equal groups and averaged the energy shares across layers, frequency bands, and denoising steps.

As shown in Table A, the LF expert dominates the early layers with an energy share of 84.18%, while the HF expert's share increases progressively with depth (15.82% →\rightarrow 60.74% →\rightarrow 88.56% →\rightarrow 98.25%). This pattern shows a shift in gate-weighted expert utilization from LF to HF across network depth, consistent with the intended depth-dependent design.

To examine whether the experts also have different frequency roles at the output level, we generate three outputs using the same input and seed: one with both experts enabled and two with either expert disabled. We apply a 2D Fourier transform to the difference between the full-model output and each ablated output within the object region, and measure the fraction of the change energy above a normalized frequency radius of 0.5. Across the same RefGC-SR² Benchmark, disabling the HF expert produces changes with a higher mean high-frequency share than disabling the LF expert (0.044 vs. 0.025), with a 95% confidence interval of [0.014, 0.023] for the difference, estimated using 10,000 image-level bootstrap resamples. The same direction holds in 150 of 200 images. These results support different relative frequency roles in the trained model, with changes induced by disabling the HF expert containing a larger proportion of high-frequency energy.

We further compare our proposed depth-dependent gate initialization with uniform and reversed initializations. As shown in Table B, our initialization achieves higher PSNR and SSIM and lower LPIPS than both alternatives. This ablation supports the effectiveness of the initialization strategy. The response analysis and output-level analysis provide complementary evidence: the former characterizes expert utilization across depth, while the latter reveals differences in the frequency composition of the output changes induced by disabling each expert.

Table A:

Layer groupLayersLF expert energy shareHF expert energy share
Early0–1384.18%15.82%
Early-middle14–2739.26%60.74%
Late-middle28–4111.44%88.56%
Late42–561.75%98.25%

Table B:

SettingCLIP-I ↑DINO ↑PSNR ↑SSIM ↑LPIPS ↓
Uniform0.86960.748217.27750.61870.2835
Reverse0.86790.746417.31860.62160.2845
Ours0.86960.747417.51480.63350.2746

[W2] The high-frequency loss matches channel-wise statistics

We thank the reviewer for raising this point. We agree that matching channel-wise statistics alone does not determine where details appear. Our HF loss serves as an auxiliary constraint to help exploit useful HRRI details that may not be fully utilized through flow-matching supervision alone. It is not designed to directly supervise the exact spatial arrangement of text, logos, or complex patterns, or to independently guarantee their accurate reconstruction.

Spatial conditioning is provided by the FLUX-Kontext backbone: HRRI and LRGI are encoded as context tokens alongside the target tokens, and every DiT block attends to these context tokens (Sec. 3, L289-299). The flow-matching loss supervises HRGT reconstruction conditioned on LRGI and HRRI, while the LF loss further anchors the output to the HRGT structure. However, these HRGT-based objectives do not explicitly encourage the output to incorporate HRRI's high-frequency characteristics. The HF loss complements this reconstruction and structural supervision by providing such guidance.

Because HRRI can differ from LRGI and the target output in viewpoint, pose, and object placement, direct pixel-wise supervision against HRRI can force noncorresponding spatial locations to match, potentially conflicting with reconstruction in the target viewpoint. We therefore use channel-wise statistical matching to incorporate HRRI's high-frequency characteristics without imposing incorrect spatial correspondences (Appendix G.1, L1303-1309).

Table 11 (Appendix G.1, L1320-1327) supports this design. Adding the frequency loss (Lf\mathcal{L}_f) to flow matching alone, (a)→\rightarrow(d), improves all five metrics (CLIP-I 0.8673 → 0.8696, DINO 0.7317 → 0.7474, PSNR 17.3893 → 17.5148, SSIM 0.6221 → 0.6335, LPIPS 0.2896 → 0.2746). Under the same FreqMoLE backbone and training schedule, replacing pixel-wise HRRI supervision with channel-wise statistical supervision, (b)→\rightarrow(d), improves PSNR from 16.9123 to 17.5148, SSIM from 0.6202 to 0.6335, and LPIPS from 0.2960 to 0.2746, while also improving CLIP-I and DINO. These results support using HRRI's high-frequency statistics as an auxiliary constraint.

[W3] Reference mismatch (1/2): Occlusion and incorrect details

We thank the reviewer for raising these challenging reference conditions. We evaluated occlusion and reference mismatch without retraining.

1. Occlusion

We occluded 25% of the foreground object region in the HRRI and evaluated the resulting outputs on 100 samples. RefGC-SR² achieved a DINO score of 0.734, PSNR of 16.97 dB, and LPIPS of 0.298, obtaining the best values among the three configurations reported in the table. These results indicate that the model can produce competitive outputs even when part of the reference object is unavailable. This evaluation was conducted using the same 100 samples as in Table 13. We have incorporated the corresponding qualitative and quantitative results into Fig. 10-(g), (h) and Table 13, respectively, and added the relevant discussion in L1384–L1401 of the revised manuscript.

HRRI conditionModelDINO ↑PSNR ↑LPIPS ↓
CleanAdaRefSR*0.69217.340.289
CleanImageCritic*0.74217.620.285
CleanRefGC-SR² (Ours)0.75118.540.256
OcclusionAdaRefSR*0.694 (+0.002)16.10 (−1.24)0.308 (+0.019)
OcclusionImageCritic*0.713 (−0.029)16.83 (−0.79)0.313 (+0.028)
OcclusionRefGC-SR² (Ours)0.734 (−0.017)16.97 (−1.57)0.298 (+0.042)

2. Incorrect details

In the RefGC-SR² setting, LRGI may contain generative artifacts introduced by the upstream reference-guided generation process, including substantial distortions of object identity. For example, Fig. 11 shows a DreamFuse compositing result in which the frog-shaped dispenser loses its protruding eyes and facial features and is rendered as a cylindrical bottle, although these characteristics are present in both HRRI and HRGT, as discussed in L1437-L1440. This example illustrates why preserving the appearance of LRGI alone may be insufficient for faithful refinement of outputs from reference-guided generation models. We therefore assume that a corresponding HRRI provides more reliable information about the target object’s identity and appearance than LRGI and use it to guide refinement. However, this reliance also makes the model susceptible to transferring incorrect instance-specific details when HRRI is unreliable.

To examine this limitation, we constructed 100 reference-substitution pairs by replacing the correct HRRI with an image of a different object from the same category. This setting evaluates references that are semantically similar but contain instance-specific details inconsistent with the target object. We measure DINO similarity against the substituted HRRI, whereas PSNR and LPIPS are measured against the original HRGT.

As shown in Table 14 and discussed in L1454-L1457, under same-category reference substitution, RefGC-SR² exhibits higher DINO similarity to the substituted HRRI than ImageCritic and AdaRefSR, while its fidelity to the original HRGT decreases, as reflected by lower PSNR and higher LPIPS relative to the clean-reference condition. However, DINO similarity alone does not identify which specific details are transferred.

The qualitative examples also illustrate how incorrect references can influence instance-specific appearance. In the bowl example in Fig. 12(a), RefGC-SR² produces a white interior and a denser exterior pattern resembling the substituted reference, rather than retaining the original green-toned appearance, as described in L1554-L1556. In the wine-glass example in Fig. 12(b), the originally yellow glass similarly changes to a turquoise appearance resembling the substituted reference, as described in L1556-L1559. These observations are consistent with the expected behavior of our model, which relies on HRRI as a reliable source of instance-specific appearance information to guide refinement.

A future research direction is to make this balance controllable, for example through a user-adjustable slider that regulates the influence of HRRI on the output. Such a mechanism could allow users to adjust reference guidance according to their preference in the reference.

HRRI conditionModelDINO ↑PSNR ↑LPIPS ↓
CleanAdaRefSR*0.651317.27060.2934
CleanImageCritic*0.673017.07650.4148
CleanOurs0.723118.67310.2679
Incorrect detailsAdaRefSR*0.381817.28250.2990
Incorrect detailsImageCritic*0.479214.59110.5015
Incorrect detailsOurs0.486813.41830.4174
Reviewer_G8CxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RefGC-SR², which jointly performs super-resolution and artifact refinement for reference-guided generated images, addressing detail loss and generative inconsistencies. The paper makes two main contributions. First, DipRefGC synthesizes 40K training triplets of low-resolution generated images (LRGI), high-resolution reference images (HRRI), and ground-truth images (HRGT), with pose alignment and realistic artifacts. Second, a frequency-aware diffusion transformer combines a Frequency-adaptive Mixture of LoRA Experts (FreqMoLE) and a frequency-adaptive loss to preserve structures and recover reference-specific details. Experiments on synthetic and in-the-wild benchmarks demonstrate improvements over existing SR, RefSR, and RefGCR methods. Ablation studies and a user study further validate the approach.

Critical strengths
  1. The proposed RefGC-SR² task provides a unified formulation for these two objectives. More importantly, the authors develop DipRefGC to construct pose-aligned LRGI-HRRI-HRGT triplets, addressing a key challenge in obtaining supervised training data.

  2. The proposed method consistently outperforms representative SR, RefSR, and RefGCR baselines across multiple evaluation metrics, including identity preservation, reconstruction fidelity, and perceptual quality.

Critical weaknesses
  1. The proposed dataset relies on DipRefGC to synthesize low-resolution generated images (LRGIs) containing generative artifacts. However, it remains unclear how closely these synthetic artifacts match the diverse artifact distributions produced by existing reference-guided generation models. Although the paper includes in-the-wild evaluations, these experiments cover a relatively limited set of generation models and samples.

  2. The proposed FreqMoLE is motivated by the observation that low-frequency information stabilizes in early FLUX-Kontext layers, whereas high-frequency information emerges in later layers. However, this observation does not necessarily establish that assigning separate low- and high-frequency LoRA experts according to network depth is the most effective design.

Questions for authors
  1. The proposed RefGC-SR² Dataset relies on DipRefGC to synthesize training images containing generative artifacts. How closely do these synthetic artifacts resemble those produced by existing reference-guided generation models? Although the paper includes in-the-wild evaluations, could the authors provide quantitative evidence supporting the similarity between the synthetic and real artifact distributions?

  2. How are existing fixed-resolution refinement methods, particularly ImageCritic, adapted to the 4× super-resolution setting? Are the output resolutions, training budgets, and inference configurations comparable across methods?

  3. The proposed FreqMoLE is motivated by the observation that low-frequency information stabilizes in early layers while high-frequency information emerges in later layers. However, does this observation necessarily justify the proposed depth-dependent allocation of LF and HF experts? Could the authors clarify whether alternative designs, such as uniform expert mixing or reversed frequency allocation, have been evaluated?

4 comments

To reviewer G8Cx

Thank you very much for your valuable comments that helped us significantly improve this submission. Below are detailed point-to-point responses about how we address your questions.

[W1 & Q1] Similarity between synthetic and real generative artifacts

More generation models and samples. To address the relatively limited set of generation models and samples, we extend Table 8 (Table 7 in the submission; four generators × 50 pairs, L1227–1236) with two recent reference-guided generation models, SHINE [1] for compositing (Comp.) and IC-Custom [2] for customization (Cust.). All six generators and DipRefGC process the same 200 benchmark pairs, giving 1,200 real and 400 DipRefGC outputs.

Artifact measures. To quantify how closely the synthetic artifacts match the real ones, we score each output on the measures of Table 8 (L1212–1225), with one change for detail: SSIM requires pixel-level alignment with the HRGT, which these generators do not provide (L1279–1281), so we follow FlowFixer [3] and count the keypoints matched by OmniGlue [4] between the object crops of the output and the HRRI. To also compare low-level degradation profiles, we add LIQE [5], which predicts for each image the probabilities of 11 distortion types (e.g., blur, noise, and compression).

Comparison protocol. To compare artifact distributions free of task differences, each 1-Wasserstein distance is computed between two generators of the same task on the same 200 pairs (for LIQE, averaged over the 11 types): DipRefGC vs. each external generator (6 distances), compared with two external generators (6 pairs). The latter serves as a reference scale for how much existing generators differ from one another. For uncertainty, we resample the 200 pairs 10,000 times, applying the same resampled pairs to both tasks and all generators, and report 95% confidence intervals (CIs). Differences are computed before rounding.

MeasureDipRefGC vs. external generator ↓Between external generatorsDifference (column 3 - column 2)
Identity similarity0.137 [0.124, 0.152]0.196 [0.179, 0.216]0.059 [0.047, 0.071]
Detail similarity (matched keypoints)25.8 [21.5, 31.0]45.6 [37.9, 54.1]19.8 [16.2, 23.4]
Texture preservation0.290 [0.230, 0.390]0.361 [0.287, 0.494]0.071 [0.021, 0.139]
Overall quality0.176 [0.152, 0.240]0.252 [0.203, 0.313]0.076 [0.012, 0.108]
Distortion type (LIQE)0.0365 [0.0346, 0.0420]0.0447 [0.0421, 0.0506]0.0083 [0.0048, 0.0112]

For all five measures, DipRefGC is closer to the external generators on average than they are to one another and the CI of the difference lies above zero. To also compare the measures jointly, we compute MMD² [6] with an RBF kernel (median-heuristic bandwidth) on the four score measures (identity, detail, texture, and overall), standardized with statistics shared by all generators: 0.0616 vs. 0.1183 (difference 0.0566 [0.0462, 0.0680]). To further compare images beyond these selected measures, we compute CMMD [7] on CLIP image features: 0.1315 vs. 0.1973 (difference 0.0658 [0.0521, 0.0836]). As in Table 8 (L1233–1236), these results show relative closeness on the measured dimensions rather than a formal test that the two distributions are identical: on these measures, the synthetic artifacts resemble those produced by existing reference-guided generation models.

Transfer to real generators. To verify that what our model learns from DipRefGC data transfers to real artifacts, we further evaluate it in a harder in-the-wild setting than those in the paper: 100 new HRRI–HRGT pairs from sources unseen in training, harder than Table 17 in viewpoint, scale, and color, processed by the six external generators (600 real LRGIs). Trained only on DipRefGC data, our model ranks first among all compared methods in CLIP-I, DINO, and the fine-detail metrics of FlowFixer [3] (AKI and K_Gain). For details, please refer to our response to Reviewer 27Ve (W2&Q3).

We have added these analyses to Appendix F.2 of the revised paper (L1269–1295, Tables 9–10).

[1] Lu et al., “Does FLUX Already Know How to Perform Physically Plausible Image Composition?”, ICLR’26.

[2] Li et al., “IC-Custom: Diverse Image Customization via In-Context Learning”, ICLR’26.

[3] Jun et al., “FlowFixer: Towards Detail-Preserving Subject-Driven Generation”, CVPR’26.

[4] Jiang et al., “OmniGlue: Generalizable Feature Matching with Foundation Model Guidance”, CVPR’24.

[5] Zhang et al., “Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective”, CVPR’23.

[6] Gretton et al., “A Kernel Two-Sample Test”, JMLR’12.

[7] Jayasumana et al., “Rethinking FID: Towards a Better Evaluation Metric for Image Generation”, CVPR’24.

[Q2] Training, inference configuration

Thank you for requesting clarification. We adapt fixed-resolution refinement methods, including ImageCritic, to the 4× setting by first upsampling the 256×256 LRGI to the target resolution of 1024×1024 and then performing refinement at that resolution. The final outputs are evaluated at 1024×1024. This preprocessing only resizes LRGI; it does not provide additional high-resolution information.

The baselines marked with an asterisk (*) are DiT4SR, TSD-SR, AdaRefSR, and ImageCritic. These four methods are initialized from their released checkpoints and fine-tuned on our dataset. Both our model and all four fine-tuned baselines are trained for two epochs with a batch size of 16. LRGI is provided at 256×256 and HRGT at 1024×1024. For AdaRefSR, ImageCritic, and our model, HRRI is also provided at 1024×1024; DiT4SR and TSD-SR do not use HRRI.

Regarding inference configurations, ImageCritic and our model share the same default settings: 28 sampling steps, FlowMatchEulerDiscreteScheduler and guidance_scale = 3.5. Other methods use their respective sampling procedures: TSD-SR and AdaRefSR perform one-step inference, while DiT4SR and ReFIR use their default 40 and 50 steps, respectively. OmniPaint uses 28 steps. All methods generate one output per input without best-of-N selection or reranking, and use a seed of 42. We have added a table summarizing these details as Table 5 and included the corresponding discussion in L1005–1019 of the revised paper.

ConfigurationDiT4SR*TSD-SR*AdaRefSR*ReFIROmniPaintImageCritic*Ours
Training
Fine-tuning on our datasetYesYesYesNoNoYesYes
Training epochs¹222——22
Batch size161616——1616
LRGI resolution256×256256×256256×256——256×256256×256
HRGT resolution1024×10241024×10241024×1024——1024×10241024×1024
HRRI resolutionNot usedNot used1024×1024——1024×10241024×1024
Inference
Input LRGI resolution (downsampled)256×256256×256256×256256×256256×256256×256256×256
Upsampled LRGI resolution1024×10241024×10241024×10241024×10241024×10241024×10241024×1024
Output / evaluation resolution1024×10241024×10241024×10241024×10241024×10241024×10241024×1024
Sampling steps²401150282828
Sampler / scheduler²FlowMatch EulerDirect latent updateDDPM, single stepRestoreEDMFlowMatch EulerFlowMatch EulerFlowMatch Euler
Guidance scale²8.0Not appliedNot applied4.0 → 7.53.53.53.5
Seed42424242424242
Automatic best-of-N / rerankingNoneNoneNoneNoneNoneNoneNone

[W2 & Q3] Depth-dependent expert allocation and alternative initializations

We thank the reviewer for this suggestion. To further validate our FreqMoLE design, we conducted additional ablation experiments comparing the LF/HF gating directions specified at the start of training. We compared our proposed depth-dependent gating direction with two alternatives: Uniform, which assigns the same expert mixing weights across layers, and Reverse, which reverses the proposed LF/HF weighting direction across network depth.

As shown in Table A, our full model achieves the highest PSNR (17.5148) and SSIM (0.6335), and the lowest LPIPS (0.2746) among the three variants. Compared with Uniform and Reverse, it improves PSNR by 0.2373 dB and 0.1962 dB, respectively, while reducing LPIPS by 0.0089 and 0.0099. Our model also matches Uniform in CLIP-I, although Uniform achieves a slightly higher DINO score (0.7482 vs. 0.7474).

The higher PSNR and SSIM scores indicate improved reconstruction fidelity, while the lower LPIPS indicates improved perceptual similarity to HRGT. These results support the effectiveness of our proposed depth-dependent gating direction in improving these aspects of restoration quality compared with uniform weighting and the reversed gating direction.

Table A:

SettingCLIP-I ↑DINO ↑PSNR ↑SSIM ↑LPIPS ↓
Uniform0.86960.748217.27750.61870.2835
Reverse0.86790.746417.31860.62160.2845
Ours0.86960.747417.51480.63350.2746

Thank you for the detailed rebuttal and additional experiments. After carefully reading all the reviews and the authors’ rebuttal, I have decided to maintain my original score.

Reviewer_27VeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper jointly refines and upscales generated images using a high-resolution reference. It builds 40K training triplets with a pose-conditioned generator and adapts FLUX-Kontext using two LoRA experts and frequency-based losses.

Critical strengths

The data pipeline tackles a useful problem: creating generative errors while keeping the input aligned with a clean target.

The evaluation includes relevant baselines, including fine-tuned ImageCritic and AdaRefSR, and tests beyond the training image sources.

Critical weaknesses
  • The claimed frequency specialization is not directly verified. Table 4 shows that FreqMoLE helps, but the parameter budget of the ablation is unclear. It is therefore hard to tell whether the gain comes from the frequency-based design or additional capacity.

  • The gains over ImageCritic are small when both image sources and input generators change (Table 12), and no uncertainty estimates are reported. In this setting, reference similarity alone provides limited evidence of improved fine details and preservation of the input layout.

Questions for authors
  • Does FreqMoLE still outperform a single LoRA with a matched parameter budget? Is there direct evidence that the experts learn different frequency roles?

  • How does the method compare with a staged refinement–upscaling pipeline?

  • Are the Table 12 gains consistent across reference pairs, and how much does refinement change the input structure?

4 comments

To reviewer 27Ve

Thank you very much for your valuable comments that helped us significantly improve this submission. Below are detailed point-to-point responses about how we address your questions.

[W1 & Q1] Parameter budget and frequency roles of the experts

To clarify the parameter budget, the baseline uses a single LoRA with rank 128, whereas FreqMoLE uses two LoRA experts with rank 64 each. Both configurations contain 149,422,080 trainable LoRA parameters. Including the learnable router, FreqMoLE has 149,422,194 trainable parameters, only 114 more than the baseline, or approximately 0.000076% of its parameter count. Thus, the improvements in Table 4 are obtained under practically identical trainable parameter budgets.

To examine the experts’ frequency roles at the output level, we generate three outputs using the same input and seed: one with both experts enabled and two with either expert disabled. We apply a 2D Fourier transform to the difference between the full-model output and each ablated output within the object region, and measure the fraction of the change energy above a normalized frequency radius of 0.5. Across RefGC-SR² Benchmark, disabling the HF expert produces changes with a higher mean high-frequency share than disabling the LF expert (0.044 vs. 0.025), with a 95% confidence interval of [0.014, 0.023] for the difference, estimated using 10,000 image-level bootstrap resamples. The same direction holds in 150 of 200 images. These results support different relative frequency roles in the trained model, with changes induced by disabling the HF expert containing a larger proportion of high-frequency energy.

Disabling either expert also degrades all evaluation metrics. For example, PSNR drops from 17.51 with both experts enabled to 14.90 and 13.74 in the two ablated settings. This indicates that both experts contribute to the performance of the trained model. Together with the spectral analysis, these results support their performance contributions and distinct relative frequency roles.

To complement this output-level analysis, we examine the gate-weighted Q/K/V LoRA responses across 57 layers using the same RefGC-SR² Benchmark and three denoising steps. We compute the squared magnitude of the 2D Fourier transform over spatial image tokens and exclude the DC component. Using a normalized frequency-radius cutoff of 0.25, we calculate each expert’s share of the sum of the two experts’ individual energies in each frequency band.

As shown in the table below, the LF expert dominates the early layers with an energy share of 84.18%, while the HF expert’s share increases progressively with depth (15.82% → 60.74% → 88.56% → 98.25%). This shift in gate-weighted expert utilization from LF to HF is consistent with the intended depth-dependent design. The layer-wise analysis characterizes how strongly each expert is utilized within the model, while the output-level analysis shows that disabling the two experts produces changes with different frequency compositions.

Layer groupLayersLF expert energy shareHF expert energy share
Early0–1384.18%15.82%
Early-middle14–2739.26%60.74%
Late-middle28–4111.44%88.56%
Late42–561.75%98.25%

[W2 & Q3] Gain size and consistency, fine details, and input-layout preservation

Thank you for your constructive comments, which helped us strengthen the evaluation.

Consistency on Table 17. We agree that the margins in Table 17 (Table 12 in the submission) are small. Across its 80 reference pairs, the mean gains over ImageCritic are positive in both tasks but inconclusive. We therefore built Hard-100 to test more challenging pairs and additional generators.

Hard-100. Hard-100 is an additional, independent pool of 100 HRRI–HRGT pairs collected from seven sources [1–7]. None of these sources is used in our training, and none of the 100 pairs appears in Table 17. The pairs were selected by the three difficulty measures of Appendix E.2 (L1136–1142, Table 6) and are more difficult than Table 17 on all three (mean viewpoint difference 60.7° vs. 28.0°, |log scale ratio| 0.48 vs. 0.22, foreground-color distance 34.0 vs. 25.7). To cover more diverse real artifacts, we add SHINE [8] for compositing (Comp.) and IC-Custom [9] for customization (Cust.) to the four generators of Table 17, giving 3 generators and 300 inputs per task.

Fine details. Since the external generators do not align the output pose with the HRGT, pixel-wise reconstruction metrics do not apply (L1716–1718). To further assess fine details, we use AKI [10], the gain in OmniGlue [11] keypoint matches with the HRRI over the input LRGI (background-removed object crops), and K_Gain, the fraction of inputs with positive AKI. Ours is the best among all compared methods on all four metrics, whereas three baselines even lose matches (negative AKI). On this harder set, the margin over ImageCritic also grows: in CLIP-I and DINO, from 0.0007 and 0.0032 to 0.0087 and 0.0160 in Comp., and from 0.0005 and 0.0028 to 0.0100 and 0.0152 in Cust.

MethodComp CLIP-I ↑DINO ↑AKI ↑K_Gain ↑Cust CLIP-I ↑DINO ↑AKI ↑K_Gain ↑
OmniPaint0.79860.5560−48.533.0%0.81360.6232−29.438.7%
DiT4SR*0.79540.5409−38.123.0%0.80600.6237−37.030.0%
TSD-SR*0.80570.5961−26.728.3%0.82140.6581−30.721.0%
ReFIR0.82500.6173+0.453.0%0.83640.6632+5.652.7%
AdaRefSR*0.82190.6087+2.654.0%0.83460.6633+5.956.3%
ImageCritic*0.83500.6407+11.559.7%0.84260.6790+22.463.0%
Ours0.84370.6567+20.064.7%0.85260.6942+25.669.7%

As AKI counts only sparse keypoints, we also report DreamSim [12], a human-aligned perceptual distance: Ours reduces the distance to the HRRI from 0.357/0.275 (input; Comp./Cust.) to 0.315/0.251, the lowest of all methods.

Input-layout preservation. Using SAM3 object masks at 512×512, the native LRGI resolution (outputs are downsampled), we check whether the output keeps the object layout of the input LRGI rather than that of the HRRI. The median mask IoU with the input LRGI is 0.93 in Comp. and 0.92 in Cust. (IQRs 0.88–0.95 and 0.90–0.94), whereas that with the HRRI (0.32 and 0.62) stays almost the same as the input’s (0.31 and 0.61).

Uncertainty of mean improvements. For each reference pair, we average the improvement of Ours over a baseline across its three generator outputs, then bootstrap the 100 pair averages 10,000 times per task, so outputs of the same pair are not treated as independent samples. Against ImageCritic*, the 95% confidence intervals (CIs) of the mean improvement are:

TaskCLIP-IDINODreamSim (distance reduction)
Comp.[0.0025, 0.0158][0.0052, 0.0287][0.013, 0.032]
Cust.[0.0050, 0.0153][0.0072, 0.0244][0.011, 0.023]

All lie above zero, and the mean improvements are also positive for each of the six generators. Together, these results show that Ours improves fine details from the HRRI while largely preserving the input layout.

We have added Hard-100 and these analyses to Appendix G.7 of the revised paper (L1748–1817, Tables 19–20).

[1] Wikimedia Commons, commons.wikimedia.org.

[2] Collins et al., ABO, CVPR’22.

[3] Openverse, openverse.org.

[4] Peng et al., DreamBench++, ICLR’25.

[5] Ruiz et al., DreamBooth, CVPR’23.

[6] Unsplash, unsplash.com.

[7] Jampani et al., NAVI, NeurIPS’23.

[8] Lu et al., “Does FLUX Already Know How to Perform Physically Plausible Image Composition?”, ICLR’26.

[9] Li et al., “IC-Custom: Diverse Image Customization via In-Context Learning”, ICLR’26.

[10] Jun et al., “FlowFixer: Towards Detail-Preserving Subject-Driven Generation”, CVPR’26.

[11] Jiang et al., “OmniGlue: Generalizable Feature Matching with Foundation Model Guidance”, CVPR’24.

[12] Fu et al., “DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data”, NeurIPS’23.

[Q2] Staged refinement–upscaling pipeline

We thank the reviewer for this suggestion. We evaluated four cascaded pipelines on our RefGC-SR² Benchmark, using TSD-SR or AdaRefSR for upscaling and ImageCritic for refinement. Each pairing was evaluated in both orders: upscaling followed by refinement, and refinement followed by upscaling. We used the fine-tuned checkpoints of all three methods.

PipelineCLIP-I ↑DINO ↑PSNR ↑SSIM ↑LPIPS ↓
TSD-SR → ImageCritic0.82910.679716.08730.57300.3738
ImageCritic → TSD-SR0.81750.665316.14370.52880.4015
AdaRefSR → ImageCritic0.86070.735816.09920.59150.3173
ImageCritic → AdaRefSR0.85220.705516.01480.58440.3187
Ours (Table 2)0.86960.747417.51480.63350.2746

Our method outperforms all four cascaded pipelines across all five metrics. Cascaded pipelines may accumulate errors across independently trained stages, as artifacts or inaccurate details introduced by the first model may persist or be amplified by the subsequent model. Similar concerns have been discussed for sequential video super-resolution and deblurring in FMA-Net++ [1] and for step-by-step image restoration in Hybrid Agents for Image Restoration [2]. We have added these results to Table 18 and discussed them in the revised paper (L1739–1746).

[1] Youk et al., “FMA-Net++: Motion- and Exposure-Aware Joint Video Super-Resolution and Deblurring.” ECCV’26.

[2] Li et al., “Hybrid Agents for Image Restoration.” CVPR’26.

Reviewer_27VeOpenReview ↗

feedback

Thanks for the detailed response and the extra experiments. The parameter-matched ablation, the Hard-100 results, and the comparisons with two-stage pipelines address my main concerns. I’m changing my rating from Weak Reject to Weak Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.