- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Cross-Expert Learning (CEL) for all-in-one image restoration. During training, responses from all experts are weighted by their cosine alignment with the negative restoration gradient to form a detached supervision target. Only the natively selected expert receives the auxiliary update, preserving the original inference path. Experiments on AIO-3, AIO-5, and CDD11 demonstrate improvements across two expert-based backbones without additional inference parameters or FLOPs.
Critical strengths
- CEL provides a simple training-only mechanism for transferring useful information from non-selected experts while preserving sparse inference. The supervision construction and gradient boundaries are clearly specified.
- The mechanism analysis includes meaningful controls: uniform and router-weighted supervision, direct-gradient targets, and a gradient-norm-matched coupled-update baseline. Together with improvements on standard and composite degradations, these experiments support the proposed learning design.
Critical weaknesses
- The distinction from existing cross-expert distillation is discussed but not directly established experimentally. The controlled baselines are useful, yet comparisons with closely related non-activated-expert or mutual-expert distillation methods would better establish the contribution beyond gradient-based target weighting.
- Training efficiency and result variability remain insufficiently quantified. CEL evaluates the full expert bank and obtains intermediate restoration gradients, but reports memory overhead without training throughput or compute-matched comparisons. Some gains are modest, particularly the 0.12 dB average improvement on AIO-3, and repeated-training uncertainty is not reported. The M2Restore comparison also uses an externally retrained baseline, making its larger gain less tightly controlled.
Questions for authors
- How does CEL compare with an existing cross-expert distillation method adapted to the same backbone and training protocol?
- What is the training-time overhead, and do the improvements persist when the baseline receives an equivalent wall-clock budget? Please also report variability across training seeds and clarify whether the M2Restore baseline and CEL runs use identical settings.
- CUE scores normalized response directions but aggregates unnormalized responses. How sensitive are the results to differences in expert-response magnitude, and would normalizing or scale-matching the target responses change performance?
2 comments
Official Comment by Authors
Dear Reviewer U6fd,
As the discussion period is nearing its end, we would greatly appreciate your feedback on whether our responses address your concerns, so that we can promptly address any remaining questions before the deadline.
Thank you for your time and consideration.
Response to Reviewer U6fd
Thank you for suggesting these comparisons and controls.
W1/Q1: Contribution and MoDE comparison. We compare MoDE [1] and CEL on MoCE-IR Top-2 with identical data, initialization, optimizer, and 100K updates. Both methods apply auxiliary updates to the selected experts. MoDE minimizes the mean squared difference between their responses, with gradients to both. CEL independently supervises each selected expert with a detached, CUE-weighted target from all experts.
We calibrate λD on a common 10K Baseline checkpoint and 32 fixed training batches. The median per-batch ratio of CEL auxiliary-gradient norm to unweighted MoDE gradient norm, measured on expert parameters, is 0.040. We test 0.3/1/3 times this value:
CEL gains 0.1715 dB over Baseline versus 0.0857 for the best tested MoDE, exceeding it by 0.0858 dB/0.0009 SSIM. In separate Top-1 controls, with the same supervision target and matched total auxiliary-gradient norm, learner-only updates yield 28.7057 dB versus 28.5482 when peers also receive the loss (+0.1575). Uniform target weights yield 28.4820 dB. These controls examine the update recipients and target weighting separately.
W2/Q2: Training efficiency and variability. Top-1 profiling uses four RTX 4090 GPUs, FP32 with TF32 disabled, global batch 32, and 128×128 patches. Both models retain 25,353,796 inference parameters and identical inference operations.
Equal time. Full CEL training takes 1,065,840 optimizer updates and T=151.24 h. Baseline receives the same time budget, more updates, and learning-rate decay spread across them; hardware, data, initialization, and validation are matched. Logged elapsed times include data loading, validation, and saves, excluding interruptions; the per-step figures above are separate steady-training measurements.
The original full-schedule Baseline reaches 30.58 dB: CEL gains 0.38 dB at the original schedule and 0.18 dB after extending Baseline to equal time. At the shorter budget, Baseline receives 11,301 additional seconds:
CEL gains 0.2170 dB at 100K updates; at the shorter equal-time budget, extended Baseline is 0.0047 dB higher.
Seeds. We repeat CEL under the same MoCE-IR Top-1, AIO-5 protocol for 100,000 updates:
Across these runs, mean ± sample standard deviation is 28.7061 ± 0.0080 dB in PSNR and 0.9020 ± 0.0002 in SSIM. Gains relative to the reported Baseline (28.4887 dB) are 0.2096–0.2255 dB.
M2Restore. Both runs share initial weights, code, and expert-response combination in training/evaluation; CEL is the only method change. Settings are CDD11, seed 42, Adam with initial LR 0.0002, global batch 8, 224×224 patches, Top-2 over six experts, and 200 epochs (325,400 updates). CEL supervises the selected experts' combined response. Final evaluation on the same 2,200 test images gives Baseline 27.5126/0.8640 and CEL 29.4799/0.8776 PSNR/SSIM, gains of 1.9673 dB/0.0136 SSIM.
Q3: Response magnitude. For each input and supervision location, we preserve response directions and equalize their L2 norms to the mean expert norm or selected learner's norm before target aggregation. Across 3,072 response sets, the largest/smallest norm ratio has median 2.229 and 90th percentile 4.480. We retain the original CUE weights and detach both targets and scale factors during backpropagation. Initialization, loss weights, and the 100K schedule are matched.
The PSNR drops are 0.0139 and 0.0071 dB; both controls remain over 0.20 dB above Baseline (28.4887 dB).
The added experiments and analyses are in the revised manuscript. We will incorporate the remaining reporting details and clarifications into the manuscript and supplement.
[1] Xie et al. MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (2024). arXiv:2402.00893