Learning What and When to Transfer for Cross-Scene Hyperspectral Clustering

Cross-scene clustering of hyperspectral imagery (HSI) asks an unlabeled source scene to help organize an unlabeled target scene into consistent land-cover groups despite scene-dependent spectral shifts. Existing methods typically align scene representations as a whole, although transferable land-cover semantics and scene-specific spectral variation may remain entangled. This makes it difficult to determine both what structure should be transferred and when target evidence should enter the transfer process. We propose disentangled transfer with recurrent admission (DTRA), which treats these two decisions as coupled parts of one recurrent transfer process. To learn what to transfer, it factorizes each representation into semantic and scene-specific branches, tests whether semantic assignments are preserved under cross-scene variation exchange, and summarizes the preserved semantic structure by balanced prototypes. These prototypes are then matched across scenes to initialize the target-oriented transfer state. To learn when to transfer, assignment confidence is combined with the exchange-induced assignment discrepancy to rank target regions and admit them progressively. The terminal target-oriented prototype state is carried into the next training round, so an admission decision also changes the prototype geometry used to judge subsequent transfer content. We further provide an assignment-preservation analysis that connects semantic deviation under cross-scene variation exchange to the assignment discrepancy reused by recurrent admission, and formulate training as recurrent prototype-state refinement under a unified clustering objective. Experiments on six directional tasks from Houston, Pavia, and HyRANK have conducted to validate the effectiveness and superiority of our DTRA method. The code is released at https://github.com/Sky-byte-box/DTRA_code.