- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ReRank-TTA, which combines recognizer scores with a frozen language model to select preferred transcriptions from beam-search candidates, then updates the recognizer through a pairwise ranking loss against competing hypotheses. The recognizer is reset for each test batch. Experiments cover four handwriting datasets with four TrOCR checkpoints and two scene-text datasets with PARSeq variants.
Critical strengths
(1) HRA provides supervision for a central recognition decision, which complete transcription to prefer. It turns language-model-guided preferences into recognizer updates without treating any candidate as ground truth. I see this as a contribution for adaptation without source data or target labels.
(2) With beam search and LM reranking held fixed, HRA reduces TrOCR-base CER and improves all eight CER/WER measurements across the four handwriting datasets. Results with PARSeq variants extend the evidence beyond handwriting. These comparisons mean that recognizer updates contribute to the reported gains.
Critical weaknesses
(1) The value of pairwise ranking over simpler self-training is not established. There is no comparison against cross-entropy training on the same LM-selected transcriptions with matched update and decoding settings. This control matters because it would separate the contribution of the ranking objective from that of the language-model supervision, directly testing the paper’s central methodological claim.
(2) The paper does not report numerical values for the adaptation candidate count , ranking margin , or LM weight βLM, nor explain how hyperparameters are selected without target labels. Their omission prevents faithful reproduction and leaves unclear how the reported performance can be obtained under the stated deployment constraints.
Questions for authors
(1) How does HRA compare with cross-entropy self-training on the same LM-selected transcriptions, holding trainable parameters, adaptation steps, and final decoding fixed? Does this comparison show an advantage attributable to pairwise ranking?
(2) What numerical values are used for the adaptation candidate count , ranking margin , and LM weight βLM? Which data and labels, if any, informed hyperparameter selection, and were settings shared across target datasets or tuned separately?
(3) When the LM changes the recognizer’s top choice, how often does that change reduce transcription error, assessed using evaluation labels only for analysis? How does HRA affect recognition when the induced preference is incorrect?