Beyond Alignment: Test-Time Neural Consistency for Cross-Subject Visual Decoding

Cross-subject visual decoding aims to reuse pretrained brain decoders across individuals with limited subject-specific data. Existing approaches mainly address functional alignment, enabling neural representations to transfer across subjects, but they do not explicitly constrain the subsequent generative trajectory to remain faithful to the neural evidence observed at test time. We propose $\text{Dual-Space Data Consistency (DS-DC)}$, a training-free inference framework that reintroduces complementary neural and structural evidence into two stages of cross-subject generation. During diffusion-prior sampling, $\text{Measurement-Space Consistency (MSC)}$ feeds the observed source-subject fMRI back into the latent trajectory through a frozen learned neural forward operator. During image enhancement, $\text{Query-Guided Structural Consistency (QSC)}$ uses the fMRI-derived low-level estimate as a reliability-weighted multi-scale structural reference. A key challenge arises in MSC: unlike physical inverse problems with known forward operators, the visual-to-fMRI mapping is only available through an imperfect learned surrogate. We therefore use weak step-wise guidance together with $finite$ latent refinement rather than exact measurement fitting. Our analysis shows that such refinement simultaneously reduces latent estimation error and propagates surrogate-model residuals, inducing an implicit spectral regularization that preferentially corrects well-observed directions while limiting error amplification in weakly constrained modes. On the Natural Scenes Dataset, DS-DC consistently improves a pretrained MindAligner decoder across $12$ directed cross-subject transfers without updating its parameters. MSC reduces source-fMRI measurement MSE by $24.45$%, while the full framework improves all reported low- and high-level decoding metrics. These results show that test-time consistency provides a principled complement to cross-subject alignment, while learned neural constraints are most effective when enforced finitely rather than aggressively.