acceptodds
Under review as a conference paper at ICLR 2027

Cross-Expert Learning for All-in-One Image Restoration

Abstract

Existing mixture-of-experts (MoE) methods typically assign the learner and supervisor roles to the same experts. In this paper, we introduce Cross-Expert Learning (CEL), in which each learner receives utility-weighted joint supervision from all experts. To construct this supervision, Cross-Expert Utility Estimation (CUE) evaluates all experts on shared input features to obtain candidate restoration responses. Each candidate response is scored by its local alignment with the restoration descent direction, which defines its utility. The utilities are input-dependent, varying with the degradation characteristics and image content of the current input. CUE estimates these utilities and uses them to combine the expert responses into supervision for the learner. Throughout this learning process, CEL requires neither degradation labels nor an external teacher. The additional expert responses are used only during training and do not enter the native prediction path. Experiments on standard and composite all-in-one restoration benchmarks show consistent performance improvements of CEL across different expert-based backbones, without additional model parameters or inference FLOPs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
All-in-One Image Restoration, Mixture-of-Experts, Cross-Expert Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_U6fdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Cross-Expert Learning (CEL) for all-in-one image restoration. During training, responses from all experts are weighted by their cosine alignment with the negative restoration gradient to form a detached supervision target. Only the natively selected expert receives the auxiliary update, preserving the original inference path. Experiments on AIO-3, AIO-5, and CDD11 demonstrate improvements across two expert-based backbones without additional inference parameters or FLOPs.

Critical strengths
  1. CEL provides a simple training-only mechanism for transferring useful information from non-selected experts while preserving sparse inference. The supervision construction and gradient boundaries are clearly specified.
  2. The mechanism analysis includes meaningful controls: uniform and router-weighted supervision, direct-gradient targets, and a gradient-norm-matched coupled-update baseline. Together with improvements on standard and composite degradations, these experiments support the proposed learning design.
Critical weaknesses
  1. The distinction from existing cross-expert distillation is discussed but not directly established experimentally. The controlled baselines are useful, yet comparisons with closely related non-activated-expert or mutual-expert distillation methods would better establish the contribution beyond gradient-based target weighting.
  2. Training efficiency and result variability remain insufficiently quantified. CEL evaluates the full expert bank and obtains intermediate restoration gradients, but reports memory overhead without training throughput or compute-matched comparisons. Some gains are modest, particularly the 0.12 dB average improvement on AIO-3, and repeated-training uncertainty is not reported. The M2Restore comparison also uses an externally retrained baseline, making its larger gain less tightly controlled.
Questions for authors
  1. How does CEL compare with an existing cross-expert distillation method adapted to the same backbone and training protocol?
  2. What is the training-time overhead, and do the improvements persist when the baseline receives an equivalent wall-clock budget? Please also report variability across training seeds and clarify whether the M2Restore baseline and CEL runs use identical settings.
  3. CUE scores normalized response directions but aggregates unnormalized responses. How sensitive are the results to differences in expert-response magnitude, and would normalizing or scale-matching the target responses change performance?
2 comments
AuthorsOpenReview ↗

Response to Reviewer U6fd

Thank you for suggesting these comparisons and controls.

W1/Q1: Contribution and MoDE comparison. We compare MoDE [1] and CEL on MoCE-IR Top-2 with identical data, initialization, optimizer, and 100K updates. Both methods apply auxiliary updates to the selected experts. MoDE minimizes the mean squared difference between their responses, with gradients to both. CEL independently supervises each selected expert with a detached, CUE-weighted target from all experts.

We calibrate λD\lambda_D on a common 10K Baseline checkpoint and 32 fixed training batches. The median per-batch ratio of CEL auxiliary-gradient norm to unweighted MoDE gradient norm, measured on expert parameters, is 0.040. We test 0.3/1/3 times this value:

MethodλD\lambda_DPSNRSSIM
Baseline—28.38250.8949
MoDE0.01228.46820.8979
MoDE0.04028.38450.8978
MoDE0.12028.41760.8974
CEL—28.55400.8988

CEL gains 0.1715 dB over Baseline versus 0.0857 for the best tested MoDE, exceeding it by 0.0858 dB/0.0009 SSIM. In separate Top-1 controls, with the same supervision target and matched total auxiliary-gradient norm, learner-only updates yield 28.7057 dB versus 28.5482 when peers also receive the loss (+0.1575). Uniform target weights yield 28.4820 dB. These controls examine the update recipients and target weighting separately.

W2/Q2: Training efficiency and variability. Top-1 profiling uses four RTX 4090 GPUs, FP32 with TF32 disabled, global batch 32, and 128×128128\times128 patches. Both models retain 25,353,796 inference parameters and identical inference operations.

Methods/stepImages/sPeak GiB/GPU
Baseline0.4208376.0417.211
CEL0.5461258.6018.427

Equal time. Full CEL training takes 1,065,840 optimizer updates and T=151.24T=151.24 h. Baseline receives the same time budget, more updates, and learning-rate decay spread across them; hardware, data, initialization, and validation are matched. Logged elapsed times include data loading, validation, and saves, excluding interruptions; the per-step figures above are separate steady-training measurements.

Full AIO-5 trainingBudget (h)PSNRSSIM
Extended Baseline151.2430.780.922
CEL151.2430.960.924

The original full-schedule Baseline reaches 30.58 dB: CEL gains 0.38 dB at the original schedule and 0.18 dB after extending Baseline to equal time. At the shorter budget, Baseline receives 11,301 additional seconds:

MethodUpdatesSecondsPSNR
Baseline100,00042,71028.4887
CEL100,00054,01128.7057
Extended Baseline126,49154,01128.7104

CEL gains 0.2170 dB at 100K updates; at the shorter equal-time budget, extended Baseline is 0.0047 dB higher.

Seeds. We repeat CEL under the same MoCE-IR Top-1, AIO-5 protocol for 100,000 updates:

CEL seedPSNRSSIM
12328.70570.9020
4228.71420.9022
202728.69830.9018

Across these runs, mean ± sample standard deviation is 28.7061 ± 0.0080 dB in PSNR and 0.9020 ± 0.0002 in SSIM. Gains relative to the reported Baseline (28.4887 dB) are 0.2096–0.2255 dB.

M2Restore. Both runs share initial weights, code, and expert-response combination in training/evaluation; CEL is the only method change. Settings are CDD11, seed 42, Adam with initial LR 0.0002, global batch 8, 224×224224\times224 patches, Top-2 over six experts, and 200 epochs (325,400 updates). CEL supervises the selected experts' combined response. Final evaluation on the same 2,200 test images gives Baseline 27.5126/0.8640 and CEL 29.4799/0.8776 PSNR/SSIM, gains of 1.9673 dB/0.0136 SSIM.

Q3: Response magnitude. For each input and supervision location, we preserve response directions and equalize their L2L_2 norms to the mean expert norm or selected learner's norm before target aggregation. Across 3,072 response sets, the largest/smallest norm ratio has median 2.229 and 90th percentile 4.480. We retain the original CUE weights and detach both targets and scale factors during backpropagation. Initialization, loss weights, and the 100K schedule are matched.

TargetPSNRSSIMGain over BaselineRetained gain
Original CEL28.70570.9020+0.2170100.0%
Mean norm28.69180.9018+0.203193.6%
Learner norm28.69860.9019+0.209996.7%

The PSNR drops are 0.0139 and 0.0071 dB; both controls remain over 0.20 dB above Baseline (28.4887 dB).

The added experiments and analyses are in the revised manuscript. We will incorporate the remaining reporting details and clarifications into the manuscript and supplement.

[1] Xie et al. MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (2024). arXiv:2402.00893

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer U6fd,

As the discussion period is nearing its end, we would greatly appreciate your feedback on whether our responses address your concerns, so that we can promptly address any remaining questions before the deadline.

Thank you for your time and consideration.

Reviewer_vTYUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cross-Expert Learning, a training strategy for MoE-based all-in-one image restoration. The proposed cross-expert utility estimation evaluates all experts’ local responses on a shared feature input and weights them by their cosine alignment with the local negative gradient of the restoration loss, and the utility-weighted response serves as a detached target that supervises only the selected learner. Experiments on AIO3, AIO5, and CDD11 with two MoE backbones show the effectiveness.

Critical strengths
  • CEL is applied only during training and introduces no extra parameters or FLOPs at inference.

  • The stop-gradient design is interesting, which ensures peer experts act as supervisors without being updated, preventing collapse into identical experts and maintaining native routing.

  • The authors provide careful ablations to disentangle where gains come from.

  • The paper is well written and easy to follow.

Critical weaknesses
  • The method relies on supervised restoration gradients, applicability to settings without paired ground truth or with weak/noisy supervision is unclear.

  • Performance gains on 3-degradation and 5-degradation is not significant. For instance, the propsoed method only achieves the best PNSR/SSIM on Dehaze for 3-degradation, which may weaken the contribution of the proposed method

  • The compared baseline is not up-to-date. e.g. What is the performance comparison on 3-degradation and 5-degradation task with MIRAGE model?

Questions for authors

How sensitive are results to \tau and \lambda_{CEL}? it is suggested to include a small hyperparameter sweep to show robustness and the existence of a broad sweet spot.

2 comments
AuthorsOpenReview ↗

Response to Reviewer vTYU

Thank you for the suggestions. We have added MIRAGE comparisons and a hyperparameter sweep, and clarify the supervision scope and task-wise gains below.

W1 Supervision requirements

CEL uses the backbone's paired training data and restoration objective. CUE evaluates expert responses with the restoration-loss gradient and combines them to supervise the selected learners. We have not evaluated settings without paired ground truth or with weak/noisy supervision. Effectiveness in these settings would depend on whether the available objective provides reliable gradients for estimating expert utility.

W2 Magnitude and consistency of the gains

Average PSNR gains are 0.12 dB on AIO-3 and 0.38 dB on AIO-5 over MoCE-IR, and 0.58 dB on CDD11 over MoCE-IR-S, with unchanged inference cost. The benchmarks contain three tasks, five tasks, and eleven single, double, and triple degradation conditions, respectively.

AIO-3 gains are concentrated in dehazing (+0.53 dB) and deraining (+0.10 dB); denoising changes remain within 0.01 dB. CEL improves all five AIO-5 tasks, with the largest gain in deblurring (+0.74 dB), and average SSIM rises from 0.919 to 0.924.

Table V1. AIO-5 PSNR and gains (dB).

TaskBaselineCELGain
Dehazing30.4830.83+0.35
Deraining38.0438.49+0.45
Denoising31.3431.45+0.11
Deblurring30.0530.79+0.74
Low light23.0023.28+0.28
Average30.5830.96+0.38

All eleven CDD11 conditions improve, including +1.45 dB for haze and +0.86 dB for haze–rain.

W3 Comparison with MIRAGE

For AIO-3 and AIO-5, we evaluate official MIRAGE weights using the same image loading, noise generation, and full-RGB metric code as CEL. CDD11 MIRAGE results come from Table 3 of the original MIRAGE paper and are compared with our MoCE-IR-S + CEL result.

Table V2. PSNR (dB) / SSIM.

MethodAIO-3AIO-5CDD11
MIRAGE-T32.81 / 0.91630.50 / 0.91128.86 / 0.883
MIRAGE-S32.97 / 0.91730.74 / 0.91529.33 / 0.887
CEL32.85 / 0.91730.96 / 0.92429.63 / 0.886

CEL's PSNR differences from MIRAGE-S are −0.12, +0.22, and +0.30 dB on AIO-3, AIO-5, and CDD11; differences from MIRAGE-T are +0.04, +0.46, and +0.77 dB, respectively. CEL matches MIRAGE-S in AIO-3 SSIM and exceeds both variants in AIO-5 SSIM. Its CDD11 SSIM is above MIRAGE-T and 0.001 below MIRAGE-S.

Q1 Sensitivity to τ\tau and λCEL\lambda_{\mathrm{CEL}}

We vary one parameter at a time, keeping the other at its original value, with the same random seed and 100,000 updates. Smaller τ\tau concentrates CUE weights on higher-scoring experts; λCEL\lambda_{\mathrm{CEL}} weights the CEL loss.

Table V3. Hyperparameter sensitivity.

τ\tauλCEL\lambda_{\mathrm{CEL}}PSNRSSIMGain over Baseline
0.250.0528.70570.9020+0.2170
0.100.0528.65280.9017+0.1641
0.500.0528.58650.8988+0.0978
0.250.0128.82040.9016+0.3317
0.250.1028.81880.9011+0.3301

All five configurations exceed Baseline at 28.4887 dB by 0.0978–0.3317 dB. These separate sweeps cover τ\tau from 0.10 to 0.50 and λCEL\lambda_{\mathrm{CEL}} from 0.01 to 0.10. We retain the original τ=0.25\tau=0.25 and λCEL=0.05\lambda_{\mathrm{CEL}}=0.05, which gives the highest SSIM among the tested configurations.

The additional experiments and analyses are included in the revised manuscript. We will incorporate the remaining reporting details and clarifications into the manuscript and supplementary material.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer vTYU,

As the discussion period is nearing its end, we would greatly appreciate your feedback on whether our responses address your concerns, so that we can promptly address any remaining questions before the deadline.

Thank you for your time and consideration.

Reviewer_TxF8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigated how to organize learning signals among experts in Mixture-of-Experts (MoE) models designed for all-in-one image restoration tasks. Observing that traditional MoE training typically couples the "learner" and the "supervisor", and finding through experiments that valuable responses often originate from unselected experts, the authors were motivated to decouple these two roles. To this end, the authors propose Cross-Expert Learning (CEL). This method allows the selected learner to receive supervisory information from all experts and employs Cross-Expert Utility Estimation (CUE) to measure the alignment between individual expert responses and the local descent direction of the restoration objective. Experiments conducted across multiple datasets demonstrate that this method outperforms the corresponding baseline models.

Critical strengths

1.The paper identifies a meaningful distinction between the expert that receives an update and the experts that can provide useful supervision. This provides a concrete empirical basis for learner–supervisor decoupling rather than introducing CEL as an ad hoc auxiliary objective.

2.The experiments show that CEL consistently improves different expert-based restoration backbones and degradation settings while preserving the native inference path and introducing no additional inference parameters or FLOPs.

Critical weaknesses
  1. CEL is motivated as a general learner–supervisor decoupling principle for MoE learning, but the experiments are restricted to image-restoration models using Top-1 sparse expert selection. The paper states that CEL extends to multiple selected learners by independently applying it to each learner, but this extension is not empirically evaluated. It is therefore unclear whether the same benefits hold under Top-k or dense routing, where multiple experts already participate in the forward path and the distinction between selected and non-selected experts changes substantially.

  2. Weighted aggregation empirically outperforms uniform averaging and using only the highest-utility expert, but the paper does not analyze whether multiple high-utility responses are mutually compatible or whether aggregation can cancel or suppress complementary expert-specific information. Since this target construction is central to CEL, the lack of such analysis leaves an important part of the proposed learning mechanism insufficiently understood.

  3. Figure 4 validates CUE through equal-scale feature perturbations, but the actual CEL procedure updates learner parameters through an L1 response-matching objective, whose effect depends on the learner’s Jacobian and the displacement between its current response and the supervision target. Consequently, favorable alignment of an expert response does not directly establish that learning from it produces a beneficial parameter update.

  4. The paper lacks quantitative verification of its unchanged parameter count and inference FLOPs, and does not report the additional training computation or runtime introduced by CEL.

Questions for authors
  1. The method is formulated to support multiple selected learners, but all experiments use native Top-1 routing. The paper does not evaluate how CEL behaves under Top-k routing, nor does it quantify the resulting training/inference cost, memory, FLOPs, or performance trade-off as k increases.

  2. I suggest that the authors add two controls: 1) randomly permute CUE weights across experts for each input and 2) replace input-dependent weights with fixed per-expert weights estimated by averaging CUE weights over the training samples.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TxF8

Thank you for the detailed comments.

W1/Q1 Multiple selected learners

The original submission includes M2Restore Top-2. We additionally test MoCE-IR Top-2. Each Baseline–CEL pair shares initialization, data, optimizer and schedule. MoCE-IR's two selected experts independently match the same detached target with averaged L1 losses; M2Restore selects two of six experts per token, with CEL supervising their combined response. Both construct supervision from the full expert bank.

SettingBaseline PSNRCEL PSNRGain (dB)
MoCE-IR Top-1, AIO-5, 100K28.488728.7057+0.2170
MoCE-IR Top-2, AIO-5, 100K28.382528.5540+0.1715
M2Restore Top-2, CDD11, 200 epochs27.512629.4799+1.9673

Top-2 SSIM gains are 0.0039 for MoCE-IR and 0.0136 for M2Restore. Top-1 also accesses the full bank for supervision while retaining one selected expert at inference. W4 reports the Top-1/Top-2 costs.

W2 Combining expert responses

Across 3,072 response sets, the two highest-CUE responses have mean cosine similarity 0.3331; 18.78% have negative cosine. For learner-relative differences dk=rk−rseld_k=r_k-r_{\mathrm{sel}} and weights qkq_k, the cancellation ratio ∥∑kqkdk∥2/∑kqk∥dk∥2\|\sum_kq_kd_k\|_2/\sum_kq_k\|d_k\|_2 is 0.7875 for CUE and 0.7367 for uniform weights; higher values indicate less cancellation.

At 100K, we split 64 training batches equally by the proportion of supervision locations with negative cosine between the top-two responses. Additional restoration-loss reductions (×10−7\times10^{-7}), with 95% paired-bootstrap confidence intervals over batches, are:

ConflictCEL over highest-utility expertCEL over uniform
Low0.37 [0.12,0.62]1.04 [0.70,1.38]
High0.49 [0.20,0.78]1.20 [0.82,1.58]

Positive values favor CEL. Both groups support useful aggregation under the observed disagreement.

W3 Utility estimates and actual parameter updates

CUE scores response directions by alignment with the negative restoration gradient. We test actual L1-based updates from Baseline checkpoints at 50K and 100K on the same 64 batches. Before each comparison, we restore identical parameters, optimizer state, learning rate, input and random state. Auxiliary variants differ only in their targets; Baseline retains its original objective without CEL. Post-update restoration loss is evaluated on the same batch, excluding the auxiliary term.

Each entry is the second method's post-update loss minus the first method's (×10−7\times10^{-7}; mean and 95% paired-bootstrap confidence interval):

CheckpointCEL over BaselineBest over worstCEL over Shuffle
50K2.94 [2.36,3.52]2.46 [1.82,3.10]2.11 [1.50,2.72]
100K1.27 [0.99,1.55]1.03 [0.76,1.30]0.98 [0.70,1.26]

Best/worst denote the highest/lowest-scoring non-selected experts. All intervals are positive, supporting a mean advantage through updates that account for the learner's Jacobian, target displacement and optimizer state.

We also continue Baseline, Shuffle and CEL for 1,000 updates from the same 100K model using identical training batches. CEL has lower validation loss throughout the final 500 updates. On 50 fixed validation images, final losses are 0.14496 (CEL), 0.14544 (Shuffle) and 0.14563 (Baseline).

W4 Parameters and computational cost

MoCE-IR Baseline and CEL each have 25,353,796 parameters and identical inference operations. Their outputs are identical with the same weights, inputs and routing. Both have 23.6475 billion supported profiled operations, excluding FFT and other unsupported operations.

Training uses four RTX 4090 GPUs, FP32 with TF32 disabled, global batch size 32 and 128×128128\times128 patches.

Routing / methods/stepImages/sPeak GiB/GPU
Top-1 Baseline0.4208376.0417.211
Top-1 CEL0.5461258.6018.427
Top-2 Baseline0.4549970.3318.764
Top-2 CEL0.5771855.4420.649

Step-time overhead is 29.77% (Top-1) and 26.85% (Top-2); extra peak memory is 1.216 and 1.885 GiB/GPU, respectively. Single-GPU, batch-1 inference takes 40.944/41.081 ms (Baseline/CEL, Top-1) and 44.238/44.351 ms (Top-2).

Q2 Shuffled and fixed utility weights

Under the original 100K protocol, Shuffle randomly permutes CUE weights across experts for each input; Fixed uses each expert's mean weight over the training set. All other settings are unchanged.

MethodPSNRGain over Baseline
Baseline28.4887—
Shuffle28.6396+0.1509
Fixed28.5573+0.0686
CEL28.7057+0.2170

CEL exceeds Shuffle by 0.0661 dB and Fixed by 0.1484 dB, supporting expert-specific assignment and input-dependent weighting. W3 also favors CEL over Shuffle after actual updates.

The added experiments and analyses are in the revised manuscript. We will incorporate the remaining reporting details and clarifications into the manuscript and supplementary material.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer TxF8,

As the discussion period is nearing its end, we would greatly appreciate your feedback on whether our responses address your concerns, so that we can promptly address any remaining questions before the deadline.

Thank you for your time and consideration.

Thanks for the authors' detailed response, which has largely addressed my concerns. After reviewing the questions raised by other reviewers and the authors' replies, I have decided to increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.