Decompose to Restore: Factor-Specific Adaptation for Image Restoration under Composite Degradations

Image restoration under composite degradations requires a unified model to handle multiple degradation factors that often co-occur in real-world images. Existing all-in-one methods typically treat each degradation type as an independent restoration condition or use degradation cues only as implicit conditioning, which limits factor-specific restoration behavior under composite degradations. We observe that restoration models already encode degradation-factor information, but such information is not well organized for factor-wise adaptation. We propose DFLoRA, a unified restoration network that decomposes composite degradations into reusable factors with dedicated low-rank adaptations. DFLoRA explicitly organizes latent degradation information into reusable factor representations and composition coefficients, decomposes a composite degradation into its fundamental factors, and equips each factor with a dedicated low-rank restoration executor. The resulting factor-specific adaptations are then composed according to the inferred degradation composition, directly translating degradation composition into adaptation composition. On the composite degradation restoration benchmark, DFLoRA achieves a PSNR gain of 1.23 dB over the strongest competing method, and it also achieves state-of-the-art performance across multiple single-degradation tasks in the standard all-in-one restoration setting. Our code will be publicly available.