Source or Prefix? Retaining What Helps the Target in Autoregressive 3D Medical Image Translation

Translating between 3D medical imaging modalities requires preserving patient anatomy while recovering target-specific properties. A source-conditioned blockwise autoregressive model uses the source as a fixed anatomical anchor and earlier blocks as an evolving target prefix, and the difference between predictions with and without the prefix defines the prefix correction. The balance between source and prefix evidence varies across regions, while the magnitude of a correction's prediction change (effect) does not necessarily reveal whether it helps the true target (utility). In this work, we propose Source-Prefix Influence-aware Residual Autoregression (SPIRA), which learns how much of each prefix correction to retain across regions. During training, SPIRA finds the largest removable portion of each correction without increasing target loss at the current prediction states and trains a lightweight gate to reproduce the resulting predictive distribution, thus learning how to balance source and prefix during generation. Extensive experiments across multiple modalities and diverse 3D medical translation tasks demonstrate consistent gains in translation quality. Mechanism analyses show that similar prefix effects can have different utilities depending on target alignment, while SPIRA better matches oracle decisions and achieves larger gains when more prefix correction can be safely retracted.