RefRouteFR: Demand-Driven Sparse Reference-Token Routing for One-Step Face Restoration

Reference-based face restoration uses high-quality images of the same identity to recover personalized details from degraded observations. However, reference images exhibit substantial input-dependent redundancy: only a subset of reference regions is useful for a given low-quality (LQ) input, whereas dense processing wastes computation and may introduce mismatched textures or attributes. We propose RefRouteFR, a one-step framework centered on demand-driven sparse reference-token routing. RefRouteFR exploits the native multimodal interface of a pretrained Diffusion Transformer to incorporate reference guidance and text control without requiring auxiliary reference-injection networks. A lightweight router estimates the restoration-dependent usefulness of reference tokens from the current multimodal context and forwards only the most informative subset through each Transformer block, while redundant tokens bypass expensive computation. To enable stable one-step restoration, a learned latent calibrator mitigates the mismatch between degraded LQ latents and the pretrained flow prior. Experiments on multiple datasets demonstrate state-of-the-art restoration performance, with RefRouteFR matching the quality of its dense counterpart while reducing inference time by approximately 40%.