acceptodds
Withdrawn from ICLR 2027

ReRank-TTA: Learning to Rerank at Test Time for Robust Text Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

18 views

Abstract

Text recognition faces deployment-time distribution shifts arising from changes in visual appearance, writing styles, image acquisition, and language. When target labels and source training data are unavailable, source-free test-time adaptation (TTA) offers a way to adapt deployed recognizers. However, common confidence-, entropy-, and consistency-based objectives do not explicitly optimize the relative ordering of competing transcriptions, while decoding-time search and LM reranking leave the recognizer unchanged. We propose ReRank-TTA, a source-free TTA framework that turns hypothesis preferences into relative supervision for autoregressive text recognition. Given unlabeled target images, ReRank-TTA generates candidate transcriptions, combines recognizer scores with a frozen language prior to select preferred and competing hypotheses, and updates an episodic recognizer with a pairwise sequence-ranking loss. This objective learns from relative preferences without treating any candidate as ground truth. Across IAM, RIMES, George Washington, and Bentham, the full pipeline consistently improves four TrOCR checkpoints over frozen greedy decoding, reducing CER/WER by an average of 3.74/5.57 absolute points, with mean relative improvements of 23.2%/17.9%. Matched-decoding ablations demonstrate additional gains from adaptation beyond beam search and LM reranking. Experiments with PARSeq and PARSeq-tiny on SVTP and IC15 further demonstrate applicability to scene text recognition.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, learning under distribution shift, unsupervised adaptation, hypothesis reranking, handwritten text recognition

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ChXrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReRank-TTA, which combines recognizer scores with a frozen language model to select preferred transcriptions from beam-search candidates, then updates the recognizer through a pairwise ranking loss against competing hypotheses. The recognizer is reset for each test batch. Experiments cover four handwriting datasets with four TrOCR checkpoints and two scene-text datasets with PARSeq variants.

Critical strengths

(1) HRA provides supervision for a central recognition decision, which complete transcription to prefer. It turns language-model-guided preferences into recognizer updates without treating any candidate as ground truth. I see this as a contribution for adaptation without source data or target labels.

(2) With beam search and LM reranking held fixed, HRA reduces TrOCR-base CER and improves all eight CER/WER measurements across the four handwriting datasets. Results with PARSeq variants extend the evidence beyond handwriting. These comparisons mean that recognizer updates contribute to the reported gains.

Critical weaknesses

(1) The value of pairwise ranking over simpler self-training is not established. There is no comparison against cross-entropy training on the same LM-selected transcriptions with matched update and decoding settings. This control matters because it would separate the contribution of the ranking objective from that of the language-model supervision, directly testing the paper’s central methodological claim.

(2) The paper does not report numerical values for the adaptation candidate count KK, ranking margin mm, or LM weight βLM, nor explain how hyperparameters are selected without target labels. Their omission prevents faithful reproduction and leaves unclear how the reported performance can be obtained under the stated deployment constraints.

Questions for authors

(1) How does HRA compare with cross-entropy self-training on the same LM-selected transcriptions, holding trainable parameters, adaptation steps, and final decoding fixed? Does this comparison show an advantage attributable to pairwise ranking?

(2) What numerical values are used for the adaptation candidate count KK, ranking margin mm, and LM weight βLM? Which data and labels, if any, informed hyperparameter selection, and were settings shared across target datasets or tuned separately?

(3) When the LM changes the recognizer’s top choice, how often does that change reduce transcription error, assessed using evaluation labels only for analysis? How does HRA affect recognition when the induced preference is incorrect?

Reviewer_zDPLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReRank-TTA studies source-free test-time adaptation of autoregressive text recognizers under changes in handwriting, language and image acquisition. For each unlabeled batch, it generates beam-search transcriptions, combines recognizer scores with a frozen language prior to select a preferred hypothesis, and identifies a high-scoring competing transcription. A temporary recognizer is updated with a pairwise sequence-ranking loss, decoded and reset before the next batch. Evaluation covers four TrOCR checkpoints on four handwriting datasets, plus PARSeq variants on two scene-text benchmarks. The complete pipeline reports average absolute CER/WER reductions of 3.74/5.57 percentage points; matched-decoding ablations separately assess the smaller incremental contribution of adaptation beyond beam search and LM reranking.

Critical strengths
  1. The method is simple and understandable.
  2. Evaluation covers multiple checkpoints and handwriting/scene-text datasets.
  3. Matched-decoding ablations help separate adaptation from search and reranking.
Critical weaknesses
  1. The closest LM-guided self-training alternative is not evaluated under matched conditions.
  2. Incremental adaptation gains are smaller than full-pipeline gains and include a stage1/GW CER degradation.
  3. Comparisons with entropy-based methods mix objective, supervision and parameter-budget differences.
  4. Essential hyperparameters, statistical uncertainty and complete test-time costs are missing.
Questions for authors
  1. Does pairwise adaptation outperform LM-selected pseudo-label self-training with identical candidates, parameters and compute?
  2. Can the authors provide the candidate count, margin, LM weight, tuning protocol, paired confidence intervals and full latency?
  3. Can they report preference-error and candidate-coverage analyses?
Reviewer_kkk4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies source free test time adaptation for text recognition under distribution shift and proposes ReRank TTA. Its core module, HRA, first generates top K transcription candidates with the original recognizer. It then combines the recognizer score with a frozen language model score to select a preferred hypothesis y+, and selects the highest recognizer scored candidate with different text as the competitor y-. A temporary recognizer is updated with a pairwise margin ranking loss so that y+ receives higher sequence likelihood than y-. The model is reset after each test batch to limit cross batch error accumulation, and final decoding can additionally use beam search and LM reranking.

Critical strengths
  1. The method is well aligned with sequence recognition. HRA turns competing transcriptions generated by the recognizer itself into sequence level adaptation signals, using a frozen LM as an external language prior. It requires no source data or target labels at deployment, and episodic reset limits error accumulation across batches.

  2. The empirical coverage is broad, and the matched decoding ablation is useful. The paper covers TrOCR and PARSeq, HTR and STR, and several model sizes. Table 4 shows that, under the same final beam plus LM decoding condition, HRA still adds gains in most settings. This suggests that the results are not entirely explained by final LM reranking.

Critical weaknesses
  1. The independent merit of the core innovation is not substantiated by controlled experiments. The paper does not include a matched control baseline that preserves the exact same candidate set, language model, preferred hypothesis selection rule, trainable parameter scope, and update budget, while only replacing the pairwise ranking loss with standard sequence NLL self-training loss on the preferred hypothesis y+. The existing TENT and EATA baselines neither leverage equivalent language prior supervision nor have comparable trainable parameter scales, making it impossible to attribute the performance gain to relative ranking learning rather than generic LM-guided self-training. This deficiency weakens the persuasiveness of the core technical contribution.

  2. Test-time deployment efficiency lacks quantitative analysis, limiting practical reference value. ReRank-TTA incurs non-negligible computation and latency overhead by performing multi-step inner-loop updates, multi-candidate beam search, and LM scoring per test batch. The paper fails to quantify critical deployment metrics including inference latency and GPU memory footprint, and does not discuss the performance-efficiency trade-off with respect to update steps and candidate set size. For test-time adaptation scenarios highly sensitive to inference efficiency, this gap reduces the practical applicability of the work.

Questions for authors
  1. Could you add a controlled pseudo-label self-training baseline that exactly matches HRA in candidate generation, language model, preferred hypothesis selection rule, trainable parameter scope, and number of update steps, with only the pairwise ranking loss replaced by sequence NLL loss on y+ , to verify the independent value of the relative ranking objective?

  2. Could you provide comparison results with domain-specific text recognition TTA methods (e.g., Tula et al., 2023) under a unified backbone and experimental protocol, to more comprehensively validate the performance advantage of the proposed method?

  3. Could you quantify the inference latency and peak GPU memory usage of backbones with different sizes (e.g., TrOCR-base, TrOCR-large) under the default 5-step update configuration? Please also explain the performance-efficiency trade-off regarding the number of update steps and candidate set size, as well as any corresponding efficiency optimization schemes.

Reviewer_adfnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReRank-TTA, a source-free test-time adaptation framework for autoregressive text recognition. Its key component, hypothesis ranking adaptation (HRA), updates the recognizer by increasing the relative likelihood of preferred hypotheses over competing ones. Experiments across six benchmarks demonstrate improvements in text recognition performance.

Critical strengths
  1. Well-motivated problem. The paper addresses a practical problem in text recognition, where labeled target data and source training data are unavailable during deployment. This provides a clear motivation for source-free test-time adaptation.

  2. Simple and clear methodology. The proposed method is straightforward and easy to understand, making it relatively easy to follow the overall framework.

  3. Useful ablation studies. The paper includes ablation studies that help examine the contributions of different components of the proposed framework.

Critical weaknesses
  1. Limited contribution of HRA. From Table 4, the additional contribution of the key component, HRA, appears to be relatively small. For example, the reported IAM CER values are 3.49 and 3.70. This raises questions about how much improvement comes from the proposed ranking adaptation itself.

  2. Missing baseline comparisons. The paper lacks comparisons with some relevant baselines, such as LM-guided test-time adaptation. This makes it difficult to assess the advantages of the proposed method over alternative adaptation strategies.

  3. Insufficient hyperparameter details. Some important hyperparameter settings are not clearly specified, such as the LM weight, ranking margin, and number of adaptation candidates. These details are important for reproducibility and for understanding the sensitivity of the proposed method.

Questions for authors
  1. Pseudo-label self-training baseline. Could the authors provide a baseline that directly uses the preferred hypothesis (y+y^+) as a pseudo-label and applies cross-entropy-based self-training? This comparison would help clarify whether the proposed pairwise ranking objective offers advantages over a simpler pseudo-labeling approach.

  2. Computational overhead. Could the authors report the inference latency and adaptation overhead of the full pipeline? Since the method involves candidate generation, LM-based scoring, and test-time model updates, understanding its computational cost would help assess its practicality for real-world deployment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.