ReRank-TTA: Learning to Rerank at Test Time for Robust Text Recognition

Text recognition faces deployment-time distribution shifts arising from changes in visual appearance, writing styles, image acquisition, and language. When target labels and source training data are unavailable, source-free test-time adaptation (TTA) offers a way to adapt deployed recognizers. However, common confidence-, entropy-, and consistency-based objectives do not explicitly optimize the relative ordering of competing transcriptions, while decoding-time search and LM reranking leave the recognizer unchanged. We propose ReRank-TTA, a source-free TTA framework that turns hypothesis preferences into relative supervision for autoregressive text recognition. Given unlabeled target images, ReRank-TTA generates candidate transcriptions, combines recognizer scores with a frozen language prior to select preferred and competing hypotheses, and updates an episodic recognizer with a pairwise sequence-ranking loss. This objective learns from relative preferences without treating any candidate as ground truth. Across IAM, RIMES, George Washington, and Bentham, the full pipeline consistently improves four TrOCR checkpoints over frozen greedy decoding, reducing CER/WER by an average of 3.74/5.57 absolute points, with mean relative improvements of 23.2%/17.9%. Matched-decoding ablations demonstrate additional gains from adaptation beyond beam search and LM reranking. Experiments with PARSeq and PARSeq-tiny on SVTP and IC15 further demonstrate applicability to scene text recognition.