acceptodds
Under review as a conference paper at ICLR 2027

SegDem: Segmentation helps Demosaicing

Abstract

Image demosaicing reconstructs a full-color image from incomplete color measurements produced by a sensor covered with a color filter array (CFA). Most existing methods formulate demosaicing as pixel-level reconstruction and mainly rely on local textures, cross-channel correlations, and low-level image statistics. Our core insight is that reconstruction and visual understanding can be viewed as complementary views of shared scene structure: both are grounded in the same underlying physical world, and therefore the structural and physical information inferred from an image should remain consistent across the two tasks. We instantiate this idea with instance segmentation and propose SegDem, a cross-task decoder representation transfer framework for demosaicing. SegDem first learns region- and boundary-aware representations through instance-aware structural pretraining and then transfers the decoder to RAW-conditioned reconstruction. Segmentation- and demosaicing-conditioned features are further anchored to a shared frozen DINOv2 representation space to preserve structural organization across tasks. We instantiate SegDem with convolutional, Transformer-based, and state-space backbones for unified Single- and Quad-Bayer demosaicing. Extensive experiments on synthetic, external, and challenging datasets demonstrate consistent improvements across different architectures and CFA layouts.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
image demosaicing, segmentation, representative learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6G1qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SegDem, a demosaicing method which tries to incorporate structural information in order to improve image demosaicing performance. The proposed method consists of 2 training stages. The first stage uses MSCOCO instance segmentation dataset to pretrain the model for instance segmentation. Then, the next stage trains the demosaicing network (initialized from the 1st stage weights). Both stages use representation alignment with DINOv2 features to provide further structural supervision. Ablation experiments on different backbones show consistent gains in the evaluation metrics. In the Single- and Quad-Bayer demosaicing settings SegDem demonstrates strong results on synthetic benchmarks and competitive performance on real RAW data.

Critical strengths
  • Using structural information to improve demosaicing performance is well-motivated, as it could help to better fill the missing color information around the edges by avoiding the averaging artifacts around boundaries.

  • The proposed method is evaluated in both Single-Bayer and Quad-Bayer settings using multiple backbones and it demonstrates consistent gains across all evaluated configurations.

Critical weaknesses
  • The novelty of the paper appears limited, as the proposed approach primarily combines 2 widely used approaches in Computer Vision literature (transfer learning and representation alignment) for demosaicing. While this may still offer a value, the experiments do not show whether the gains come from segmentation pretraining or such gains could be achieved through other pre-training, such as image classification, image inpainting or denoising.

  • The writing quality is poor. For example the Method section has some unimportant details, while the details of instance segmentation loss are left to the Appendix without a cross-reference. Also Figure 2 is hard to follow and does not help with understanding of the method. For example, it is not clear how and why DINOv2 features are concatenated to the Linear RGB image given their different spatial and channel dimensions (should be a mistake). The input to the rightmost DINOv2 Encoder and the purpose of different arrowed lines are unspecified. This figure needs a major revision in order to clearly communicate the proposed method.

  • An ablation experiment is missing for the inclusion of segmentation and demosaicing embeddings in the method. As Stage 1 and Stage 2 are trained sequentially, it is unclear why such embeddings are necessary, so an ablation experiment is necessary for justification.

Questions for authors
  • Why did the authors choose to use cached Qwen2.5-VL embeddings instead of other simpler alternatives?

  • Why did authors not report PSNR, SSIM and LPIPS metrics in the real-world RAW evaluation table (Table 2)?

  • Given that the instance segmentation head predicts an RGB representation, how do authors compute the dice, focal and boundary losses in equation 10 during the pre-training and how does this help to the underlying instance segmentation task?

2 comments

Thank you for the rebuttal. It addresses my main questions. However, I still think that in the current form the paper is hard to follow and would like the authors to make a major revision to improve the readability. Assuming that authors will improve the writing, I am willing to increase the score to weak accept.

Thanks for recognizing our rebuttal and score improvement. We’re working on revising the core framework diagram and method description to make them more readable. Specifically, we’re adding detailed inputs for stage 1 in the core diagram, giving specific meanings to the line colors, and including some necessary structural information and loss functions. In the methods section, we’re adding the exact loss functions to the stage 1 description and including the analysis from the discussion to further illustrate the value of our framework. Also, we’re adding the necessary ablation experiments and analyses to the experimental section, and finally, we are polishing the appendix.

Reviewer_xKfFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SegDem, which explores how instance segmentation pretraining can improve image demosaicing. The method first learns region and boundary representations from instance annotations on COCO, then transfers these representations to image reconstruction conditioned on RAW observations. During reconstruction, an initial RGB image generated by a frozen demosaicing model is refined using RAW measurements to recover colors and details. Both training stages use features from a frozen DINOv2 encoder for auxiliary alignment to help preserve structural information. The authors evaluate its reconstruction performance on Single Bayer and Quad Bayer patterns across multiple test datasets.

Critical strengths

The paper explores using instance segmentation pretraining to improve demosaicing, which is a worthwhile research direction. The experiments cover CNN, Transformer, and Mamba backbones, as well as Single Bayer and Quad Bayer patterns, and report strong reconstruction results on synthetic data, external datasets, and challenging texture scenes.

Critical weaknesses
  1. The ablations do not rule out gains from additional data and training. In Table 4, the segmentation-pretrained models receive additional COCO images and an extra pretraining stage, whereas the baselines are trained directly on demosaicing. The improvements may therefore partly come from the additional data and training, and cannot be directly attributed to the segmentation task. A comparison with non-segmentation pretraining on the same COCO images is missing.

  2. The effects of segmentation supervision and DINOv2 guidance are not sufficiently disentangled. Table 6 examines the effect of removing DINOv2, but does not evaluate retaining DINOv2 guidance while removing segmentation supervision. Without DINOv2 in the segmentation stage, EDMUNet achieves a linear-RGB PSNR of 43.6770 dB on Single Bayer, essentially matching the 43.6831 dB obtained without segmentation pretraining. On Quad Bayer, the improvement is only about 0.12 dB, although gains remain in the sRGB metrics. Further evidence is needed to establish how much segmentation supervision contributes beyond DINOv2 guidance.

  3. The comparisons do not sufficiently cover representative methods to establish the method’s advantage over advanced approaches. The paper compares against Jd3Net, BMTNet, FFNet, and ms-demosaic, and includes SANet in the appendix. However, several relevant methods are missing: GRL (CVPR 2023) explicitly evaluates standard Bayer demosaicing; DRNet (AAAI 2024) specifically addresses joint Quad Bayer demosaicing and denoising; KLAP (ICCV 2023) is a representative unified demosaicing method across CFA patterns and is cited but not compared; and DFENet (2025 preprint) directly addresses high-frequency textures, false colors, and moiré artifacts, making it particularly relevant to the structural reconstruction benefits emphasized in this paper.

  4. The lack of inference-speed comparisons makes it difficult to assess the computational cost of the improvements. Table 1 reports only parameter counts, without comparing end-to-end inference time.

  5. The necessity of introducing a language model is unclear. The paper uses Qwen2.5-VL to encode two fixed instructions, “perform segmentation” and “perform demosaicing,” to help the network distinguish the tasks. However, these tasks are already predetermined. The authors do not explain why two simple task identifiers would not suffice or what additional benefit the language model provides.

Questions for authors

refer to Critical Weaknesses

7 comments
AuthorsOpenReview ↗

R3: Additional baseline comparisons

Response.

We thank the reviewer for identifying these relevant methods. We have added comparisons with GRL (CVPR 2023), KLAP (ICCV 2023), and DFENet (2025 preprint) on both Single-Bayer and Quad-Bayer demosaicing, reporting PSNR and SSIM in linear RGB and PSNR, SSIM, and LPIPS in sRGB.

The additional results support SegDem’s reconstruction performance across both CFA layouts. Among the newly added baselines, DFENet achieves the highest linear-RGB PSNR, with 44.6444 dB on Single Bayer and 42.4651 dB on Quad Bayer. SegDem(M) achieves 45.5766 dB and 43.9385 dB, respectively. SegDem(R) achieves the highest sRGB PSNR in the expanded comparison, with 35.1883 dB on Single Bayer and 34.0871 dB on Quad Bayer, while SegDem(E) achieves the lowest LPIPS on both layouts. These results demonstrate complementary strengths across the three backbone configurations.

For DRNet (AAAI 2024), we were unable to locate an available official implementation and therefore have not included a reproduced result. The expanded table below also reports inference latency to make the computational trade-offs visible.

Table 1. Expanded comparison on Single- and Quad-Bayer demosaicing. Existing reconstruction results are retained from the submitted paper; GRL, KLAP, DFENet, and latency measurements are added during rebuttal. The best result in each column is shown in bold.

MethodSingle Linear PSNR ↑SSIM ↑Single sRGB PSNR ↑SSIM ↑LPIPS ↓Quad Linear PSNR ↑SSIM ↑Quad sRGB PSNR ↑SSIM ↑LPIPS ↓Latency (ms) ↓
JD3Net41.10880.979830.74960.88300.157339.03790.970729.52060.86560.2036293.53
BMTNet42.23420.982332.72750.90220.133438.68480.975930.53400.88800.174618.91
FFNet44.35370.986933.69950.91780.112334.44640.921128.86070.82530.2267400.45
ms-demosaic44.31980.986133.98640.91970.101242.56510.976933.03680.91590.1211224.07
GRL42.62140.966430.08470.89530.151341.00310.960829.60460.88440.1855597.93
KLAP43.52340.981232.73210.91340.122142.01420.980131.24150.89950.1701394.68
DFENet44.64440.986133.62640.91000.137642.46510.982332.19630.89920.1651490.63
SegDem(E)44.57530.988134.41570.91760.083243.12150.985833.85430.92240.0954426.35
SegDem(R)45.56540.990835.18830.93120.089143.92750.988434.08710.92490.1061926.85
SegDem(M)45.57660.990735.13040.93270.095143.93850.988334.03550.92610.1129584.88
AuthorsOpenReview ↗

R5: Inference-speed comparisons

Reviewer’s concern.

Table 1 reports parameter counts but does not compare end-to-end inference time.

Response.

We thank the reviewer for requesting an explicit comparison of computational cost. We have added inference latency to the expanded comparison table in our response “R3: Additional baseline comparisons.” The reported latencies are 426.35 ms for SegDem(E), 926.85 ms for SegDem(R), and 584.88 ms for SegDem(M).

These measurements show the reconstruction–runtime trade-off across the three backbones. SegDem(R) achieves the highest sRGB PSNR but also has the largest latency, while SegDem(E) has the lowest latency among our variants and achieves the lowest LPIPS on both CFA layouts. SegDem(M) achieves the highest linear-RGB PSNR with an intermediate latency among our variants.

The instance segmentation head, DINOv2 encoder, and instruction encoder are used only during training and are excluded from demosaicing inference. The reconstruction proxy and restoration backbone remain part of the inference pipeline and contribute to its computational cost. Our claim is therefore that the proposed auxiliary supervision introduces no additional inference modules, not that SegDem is faster than all compared methods.

Thank you for the rebuttal. It addressed most of my concerns. My remaining question is how instance-level supervision helps recover missing color channels within an object. The paper explains why instance boundaries can help reduce color bleeding across object contours. Within an instance, however, material boundaries, shadows, and periodic textures can all affect which pixels provide reliable color evidence for reconstruction. Instance membership alone offers limited guidance here. Could the authors clarify what cues the instance-code pretraining objective is expected to teach the decoder beyond object contours, and how those cues could support reconstruction within an instance after transfer?

Reviewer_Q3M8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the relationship between visual understanding and the low-level task of image restoration. The authors argue that, rather than treating them as isolated objectives, both essentially focus on recovering and representing scene structure, and thus can be complementary: high-level image understanding can provide structural cues and help recover underlying image structures more efficiently. Based on this idea, the authors propose SegDem, a two-stage training recipe for single- and quad-Bayer demosaicing. The estimation backbone is first trained on COCO to predict RGB-coded instance maps from clean linear RGB and the mosaic RAW, with an auxiliary loss that aligns decoder tokens to DINOv2 tokens. The decoder is subsequently fine-tuned for demosaicing, which refines the output of a frozen JD3Net.

Critical strengths
  1. The idea of connecting visual understanding with low-level tasks like image registration through shared structural representations is interesting.

  2. SegDem is used across convolutional, transformer, and state-space restoration backbones, although the gains are not really clear in all settings.

  3. The additional overheads, like the instance head, DinoV2, and text encoder, are used only in training, which means they incur no additional cost during inference.

Critical weaknesses
  1. The ablations do not support the claim that segmentation helps. With EDMUNet segmentation, training without dynamic alignment reaches 43.6 decibels in Table 6, matching the result with no pretraining in Table 4. Removing DinoV2 from both stages gives 43.16 dB. The dB gain appears only when DinoV2 alignment is added in stage 1, and a segmentation-pretrained decoder fine-tuned without the DinoV2 anchor is about 1.4 dB below the no-pretraining baseline. The important factor here seems to be dynamic feature distillation on COCO, not instance segmentation.

  2. Most of the improvement over external baselines comes from cascading a large restoration network on top of that baseline. Every variant of this method refines the output of a frozen Jd3Net, which is one of the baselines. Without this, SegDem falls to 40.53 dB, below every baseline, including Jd3Net, in Table 5. Meanwhile, Restormer and MambaIR, without any segmentation pretraining, beat all baselines by about 1 dB. Table 1 essentially compares Jd3Net with a strong restorer and raw conditioning with single-stage methods.

  3. The hypothesis that high-level supervision helps low-level restoration is already established in the literature, and the closest works are not cited. For example, Liu et al. (IJCAI 2018) trained the encoder with a segmentation loss and dropped the segmentation network during inference. Zhang et al. (CVPR 2024) distilled a semantic prior into a restoration network during training with no inference cost, which is basically SegDem's design goal.

  4. Where segmentation pretraining seems to help, it's confounded with extra training, and the gain is also relatively small. For example, it adds 0.25 dB to Restormer and 0.09 dB to MambaIR. The "no baselines" get only the 10 LSDIR fine-tuning epochs, while SegDem gets the masks on COCO for some unknown number of steps. So, additional data and computation are not separated from the segmentation objective.

Questions for authors

N/A

2 comments

We really appreciate and understand that the reviewers took the time to review our manuscript. We are wondering if we have addressed your questions. If there are any unresolved issues or questions, feel free to let us know, and we will do our best to answer them. Thank you again for the constructive reviews. If those clarification addresses your remaining concern, we would be grateful if you would update your score accordingly.

AuthorsOpenReview ↗

Follow-up to Reviewer Q3M8

Dear Reviewer Q3M8,

Thank you again for your review and for raising questions about the contribution of instance supervision, the Jd3Net proxy, novelty, and the pretraining protocol. We have revised the manuscript and added experiments that address these points, including controls requested during the broader review discussion.

1. Contribution of instance supervision beyond DINOv2. You asked whether the gains could mainly come from DINOv2 feature alignment rather than instance supervision. To examine this, we added a Stage 1 control that retains DINOv2 token alignment but removes the instance head and its supervision, followed by the same Stage 2 demosaicing protocol. Compared with this control, adding instance supervision improves linear-RGB PSNR by 0.98 dB on Single Bayer and 1.44 dB on Quad Bayer, while also improving the sRGB metrics and LPIPS. This comparison supports the contribution of instance-aware pretraining beyond DINOv2 alignment alone. See revised Table 6.

We also clarified the original Stage 1 alignment ablation: disabling the alignment loss lowers foreground Dice from 0.7643 to 0.7114, indicating that DINOv2 guidance helps the instance-pretraining stage learn region structure. The two controls therefore examine complementary aspects of our design.

2. Role of the Jd3Net proxy. The proxy provides an initial reconstruction, while the trainable network also receives RAW measurements. To test dependence on Jd3Net, we replaced it with bilinear interpolation: EDMUNet achieves 43.54/42.35 dB on Single/Quad Bayer. With the same bilinear proxy, Restormer and MambaIR achieve 45.2146 dB and 45.3013 dB on Single Bayer, respectively. These results show that SegDem can also use a simpler reconstruction proxy. See revised Table 5 and the appendix.

3. Contribution relative to prior work. We clarified that our contribution is specifically the transfer of an instance-pretrained decoder representation to RAW-conditioned demosaicing, with DINOv2 providing a shared reference across the two stages. Our claim is about this transfer framework, rather than the general idea that high-level supervision can benefit restoration.

4. Additional pretraining data and computation. We clarified that the pretrained models use five epochs of COCO pretraining, whereas the no-pretraining models do not. Both subsequently use the same backbone, LSDIR data, reconstruction objective, ten-epoch demosaicing schedule, and test set. Table 4 evaluates the benefit of the proposed initialization under that matched downstream protocol; the DINOv2-only control provides an additional comparison within the COCO pretraining stage. See revised Sections 4.1 and 4.5.

5. How instance-aware representations help beyond boundaries. Following a related question from another reviewer, we added a missing-channel analysis on held-out COCO instances. Compared with no pretraining, SegDem(R) reduces interior MAE by 4.99%/4.29% on Single/Quad Bayer, with gains also present in high-gradient and high-texture interiors. We added an analysis of decoder region separation before and after transfer. See revised Figure 4 and Section 3.4.

We would be grateful to hear whether these clarifications and experiments address your concerns. If you find that the revised evidence resolves your concerns, we would also appreciate your consideration of an updated rating.

Thank you again for your time.

Authors of Submission 9580

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair and Program Chairs

Dear Area Chair and Program Chairs,

We thank the reviewers for their careful evaluation. As we have not yet received follow-up comments on our rebuttal, we summarize the main concerns and additional evidence for your consideration.

R1, R2, and R3 denote Reviewers Q3M8, xKfF, and 6G1q, respectively.

Key strengths recognized by the reviewers

  1. A well-motivated connection between instance-level structural understanding and RAW demosaicing (R1, R2, R3).
  2. Evaluation across convolutional, Transformer, and state-space backbones, covering Single- and Quad-Bayer inputs (R1, R2, R3).
  3. Strong reconstruction results across synthetic benchmarks and challenging evaluation settings (R2, R3).
  4. Training-only auxiliary supervision, with the instance head and foundation-model encoders removed at inference (R1).

Main concerns and responses

  1. Instance supervision versus DINOv2 guidance (R1, R2). We added the requested DINOv2-only pretraining control, retaining Stage-1 token alignment while removing the segmentation head and supervision, followed by the same demosaicing protocol. Adding instance supervision improves all five reconstruction metrics on both layouts: linear-RGB PSNR increases by 0.98/1.44 dB, sRGB PSNR by 0.85/1.26 dB, and LPIPS decreases from 0.1143/0.1312 to 0.0832/0.0954 on Single/Quad Bayer. This directly demonstrates benefits beyond DINOv2 alignment alone. See xKfF, “R4: Regarding the effects of segmentation supervision and DINOv2 guidance.”

  2. Dependence on the JD3Net proxy (R1). The proxy supplies an initial reconstruction, while the trainable network also receives RAW measurements. Replacing JD3Net with bilinear interpolation yields 43.54/42.35 dB on Single/Quad Bayer with EDMUNet. Additional Single-Bayer experiments achieve 45.2146 dB with Restormer and 45.3013 dB with MambaIR, showing that strong performance does not require JD3Net specifically. See Q3M8, “R2. Is the gain mainly due to cascading JD3Net with a large restoration network?”

  3. Pretraining protocol and novelty (R1, R2, R3). We clarified that Table 4 evaluates representation transfer with the same backbone, LSDIR data, reconstruction objective, ten-epoch demosaicing schedule, and test set, with or without five-epoch COCO pretraining. The new DINOv2-only control further examines instance supervision within pretraining. Our contribution is transferring an instance-aware decoder to RAW-conditioned reconstruction, using DINOv2 as a shared cross-stage reference, rather than claiming that semantic guidance or transfer learning alone is new.

  4. Baseline coverage and computational cost (R2). We added GRL, KLAP, and DFENet comparisons on both layouts and reported inference latency. SegDem(M) achieves the highest linear-RGB PSNR, SegDem(R) the highest sRGB PSNR, and SegDem(E) the lowest LPIPS in the expanded comparison. We also stated that an available official DRNet implementation could not be located. See xKfF, “R3: Additional baseline comparisons” and “R5: Inference-speed comparisons.”

  5. Task embeddings (R2, R3). Cached embeddings modulate auxiliary features through channel-wise scale and shift. With conditioning, Stage-2 L1 loss reaches approximately 0.04 in the first epoch, compared with the third epoch without it; final ten-epoch metrics are nearly identical. We therefore identify this component as a convergence aid, not the central contribution. No instruction encoder is used at inference.

  6. Presentation and evaluation (R3). We clarified spatial matching before image–DINO feature concatenation, the alignment encoder’s input, and the RGB-coded instance losses described in Appendix F.2. We also explained that native real-RAW captures lack clean, spatially aligned references, motivating no-reference evaluation there and full-reference metrics on paired data. The responses specify the figure annotations and cross-references needed to clarify these details.

The new DINOv2-only control directly addresses the central concern about whether instance supervision contributes beyond alignment. Together with the bilinear-proxy experiments and expanded comparisons, it strengthens the evidence for transferring instance-aware structure to demosaicing.

We respectfully ask that these additional results be considered alongside the original submission. We remain available for further clarification and thank the reviewers, Area Chair, and Program Chairs for their time.

Authors of Submission 9580

AuthorsOpenReview ↗

Revision Summary

We thank the reviewers for their valuable comments and suggestions. We have made the following changes in the revised paper:

  1. Improved the method presentation and revised the framework overview to clarify the two-stage workflow, decoder weight transfer, DINOv2 supervision, and the roles of the inputs and connections. See Section 3.1 and Figure 2.

  2. Added the instance-code pretraining objective to the main text, explaining how its loss terms supervise instance regions and boundaries. Detailed descriptions and weights are provided in the appendix. See Section 3.2 and Appendix, “Segmentation Head Details.”

  3. Clarified DINOv2 feature fusion and representation alignment, including spatial resizing before channel concatenation, the reference image used by DINOv2, and the distinction between Stage 1 input conditioning and alignment supervision. See Section 3.4 and Appendix, “DINOv2 Tokens.”

  4. Added a DINOv2-only Stage 1 control that retains token alignment while removing instance supervision, evaluating the additional contribution of instance-aware pretraining. See Section 4.5 and Table 6.

  5. Expanded the analysis of structural representations and regional reconstruction. We analyze how pretraining changes intra-region similarity, inter-region similarity, region separation, and boundary contrast, and report missing-channel reconstruction gains within instances and near boundaries. See Section 3.4 and Figure 4.

  6. Added GRL, KLAP, and DFENet to the synthetic demosaicing comparison, together with parameter counts and inference latency. See Sections 4.1–4.2 and Table 1.

  7. Added bilinear-proxy results for Restormer and MambaIR and clarified the effect of cached task embeddings on convergence and final performance. See Appendix, “Additional Evidence for Instance-Aware Transfer.”

  8. Clarified the training and evaluation protocols, including the pretraining and fine-tuning schedules, the no-pretraining control, and the use of no-reference metrics for real RAW evaluation. See Section 4.1 and Appendix, “Dataset Construction, Training, and Evaluation Details.”

  9. Added a Discussion section outlining future directions for transferring representations from other visual understanding tasks and improving inference efficiency. See Section 5.

We thank the reviewers again for their feedback which encourage us go further on exploring unified demsaicing and high-level tasks.

Authors of Submission 9580

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.