- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies serving-context sensitivity in text classifiers: whether a fixed checkpoint produces different labels or class scores for the same text when padding, batch size, batch composition, numerical precision, quantization, device, or backend changes. The authors train 180 models across four classifier formulations, three model sizes, five datasets, and three seeds, with additional pretrained-model, hardware, and backend controls. The paper separates a structural padding-position effect, which changes the represented input, from numerical effects, which change the execution of a fixed represented input. The main findings are that identical serving contexts are repeatable, left padding with learned absolute position embeddings can produce substantial label changes, numerical effects mainly change low-margin predictions, and fully generative classifiers are more sensitive than discriminative classifiers. The paper also evaluates mechanism-specific mitigations and gives formal conditions for padding and label invariance.
Critical strengths
- Careful and large-scale experiments. The paper uses paired comparisons that hold the checkpoint and text fixed, establishes an identical-context repeatability baseline, and evaluates a large collection of classifier formulations, datasets, model sizes, seeds, serving settings, and output metrics. This provides substantially stronger evidence than isolated examples of serving-context sensitivity.
- Clear mechanism separation and validation. The paper distinguishes the structural padding-position effect from numerical effects and uses targeted interventions to verify the proposed causes. In particular, removing and restoring width-dependent position IDs provides convincing evidence for the padding mechanism, while the quantization controls identify the source of batch-composition dependence.
- Useful practical evaluation. The paper shows that stable labels do not imply stable probabilities or downstream decisions. It also evaluates mechanism-specific mitigations and reports their accuracy and computational costs. These results provide useful guidance for testing and deploying text classifiers.
Critical weaknesses
-
The novelty and practical prevalence of the dominant effect are unclear. The largest label changes arise from manually left-padded GPT-2 models with learned absolute position embeddings and width-dependent position IDs. This mechanism and its correction are already known, and it can be avoided through right padding or real-token-order position IDs. The paper demonstrates that the effect can be large, but it does not establish how common this vulnerable configuration is in correctly implemented classifier deployments. This limits the practical significance of the headline result.
-
The evidence for modern production classifiers is limited. The primary grid consists of models trained from random initialization, using a relatively weak task-quality threshold. The pretrained and modern-model controls are much smaller, and the paper does not test common production engines or distributed serving configurations. It is therefore unclear whether the reported magnitudes generalize to current pretrained classifiers and production systems.
-
The theoretical contribution is not fully supported by the experiments. The padding and margin results mainly formalize direct properties of position embeddings and decision margins. The speed–accuracy–invariance theorem depends on an information-loss assumption that is not shown to hold for the evaluated systems, and the experiments do not demonstrate the proposed three-way trade-off. Further analysis would make the theory more of a major contribution than supporting material.
Questions for authors
The first two questions address the main issues affecting my assessment.
-
Why were the GPT-2 classifiers manually left padded using width-dependent position IDs? How common is this exact configuration in real classifier deployments? Please compare against standard classification or generation pipelines that correctly construct position IDs, and clarify whether the main padding result represents a common deployment practice or an avoidable implementation error.
-
Can the authors provide complete task-accuracy and decision-margin results for all retained models, and repeat the main summaries on classifiers with competitive task performance? How strongly are label flip rates associated with task accuracy after controlling for classifier formulation and dataset?
-
Can the authors expand or better justify the generalization to modern production systems? In particular, it would be beneficial to test additional pretrained classifiers, current decoder models, or one production serving engine. If this is not possible, the claims should be narrowed to the evaluated configurations.
-
Can the authors instantiate the premise of the speed–accuracy–invariance theorem for any evaluated implementation? If not, what empirical conclusion does this theorem support beyond stating a conditional possibility?
8 comments
Response to Reviewer mL8Z part 2
(continued from the previous comment)
Numbering the real tokens from zero removes all label changes caused by padding length. It also gives an existing checkpoint different position embeddings from those used during training. We therefore tested a third rule. Each real token receives the same absolute position it occupied when the input was padded to 512 tokens during training. Total variation distance measures the change in the complete vector of class probabilities, and the last column reports the largest change in any class score before softmax. The fixed-width reference is the prediction obtained with tensor-column positions when every input is padded to 512 tokens. Each row pools all five datasets and contains 9,821 matched examples for the named classifier formulation.
| Classifier formulation | Position assignment | Padded length | Matched examples | Labels changed versus the fixed-width tensor-column reference | Mean probability total variation versus that reference | Largest absolute class-logit change versus that reference |
|---|---|---|---|---|---|---|
| Pseudo-generative GPT-2 | Tensor-column positions | Determined by longest input in batch | 9,821 | 16.247% | 0.1256 | 17.40 |
| Pseudo-generative GPT-2 | Real tokens numbered from zero | Fixed or determined by batch | 9,821 | 16.775% | 0.1425 | 13.67 |
| Pseudo-generative GPT-2 | Positions preserved from training width 512 | Determined by longest input in batch | 9,821 | 0.000% (0 of 9,821) | 2.55e-7 | 1.81e-5 |
| Fully generative GPT-2 | Tensor-column positions | Determined by longest input in batch | 9,821 | 17.373% | 0.1012 | 76.49 |
| Fully generative GPT-2 | Real tokens numbered from zero | Fixed or determined by batch | 9,821 | 17.597% | 0.1060 | 76.49 |
| Fully generative GPT-2 | Positions preserved from training width 512 | Determined by longest input in batch | 9,821 | 0.000% (0 of 9,821) | 1.89e-6 | 2.56e-3 |
Preserving the positions used during training produces the same label as the training-width computation for every evaluated input. Each batch can still use a shorter padded length. The revised draft will report all five datasets for the evaluated GPT-2 checkpoints and all three position rules.
Right padding also prevents padded length from changing the position indices of real tokens. Our result identifies why the public left-padding implementations can change predictions and gives a correction that preserves the predictions of an existing checkpoint.
Response to Reviewer mL8Z part 3
2. Can the authors report complete task accuracy and show how the results vary with the separation between the two highest class probabilities? Do the results hold for accurate classifiers, and are label changes related to task accuracy?
Yes, we do observe prediction instabilities on classifiers with high test accuracy.
We computed complete accuracy and decision-margin results for every checkpoint in the original and pretrained sets. The decision margin is the difference between the two largest class probabilities before the serving conditions are changed. The revised draft will provide the complete accuracy, F1, calibration, predicted-class, and decision-margin results by dataset, classifier formulation, model size, and checkpoint.
Here we present the representative case of the GPT-2-medium classifiers, which achieve test accuracy from 0.8446 to 0.8660 on SST-2, 0.9343 to 0.9387 on AG News, and 0.9403 to 0.9444 on IMDb. We evaluated the same checkpoints and texts while changing one serving condition at a time.
The following terms define the columns in the table:
| Term | Meaning |
|---|---|
| Run pair | Two inference runs using the same checkpoint and texts and differing only in the serving condition named in the row |
| Prediction comparison | One text whose prediction is compared across one run pair; the same text can appear in several run pairs |
| Overall label-change rate | Total number of changed labels divided by all prediction comparisons |
| Average label-change rate per run pair | Label-change rate computed separately for each run pair and then averaged so every run pair receives equal weight |
| Largest label-change rate in one run pair | Highest label-change rate observed in any individual run pair |
| Run pairs with label changes | Number of run pairs in which at least one text receives a different label |
| Average probability movement | Average total variation distance between the two class-probability vectors across all prediction comparisons |
| Largest probability movement | Largest total variation distance observed for any individual text |
| Average largest class-score change | For each text, the largest absolute change among its raw class scores, averaged over all prediction comparisons |
| Change tested | Run pairs | Prediction comparisons | Overall label-change rate | Average label-change rate per run pair | Largest label-change rate in one run pair | Run pairs with label changes | Average probability movement | Largest probability movement | Average largest class-score change |
|---|---|---|---|---|---|---|---|---|---|
| Length to which the batch is padded | 9 | 17,463 | 9.9811% | 10.3359% | 23.5585% | 9 | 0.1121 | 1.0000 | 26.4 |
| Numerical precision | 45 | 516,315 | 0.2849% | 0.4557% | 1.3180% | 45 | 0.003995 | 0.6205 | 0.797 |
| Batch size | 216 | 2,478,312 | 0.0189% | 0.0328% | 0.9885% | 18 | 0.0003754 | 0.5486 | 0.074 |
| Other examples in the batch | 54 | 104,778 | 0.0000% | 0.0000% | 0.0000% | 0 | 0.0000 | 0.0000 | 0.0000 |
| Order of the other examples | 54 | 619,578 | 0.0000% | 0.0000% | 0.0000% | 0 | 5.043e-10 | 5.245e-6 | 2.63e-8 |
| Row occupied by the measured example | 27 | 52,389 | 0.0000% | 0.0000% | 0.0000% | 0 | 0.0000 | 0.0000 | 0.0000 |
We also measured how the decision margin relates to label changes. The number in parentheses is the number of individual prediction comparisons in that interval.
| Decision margin | Original classifiers: label-change rate and prediction comparisons | Pretrained ModernBERT and Qwen3: label-change rate and prediction comparisons |
|---|---|---|
| 0.00 to 0.01 | 10.43% (2,016,637) | 8.20% (5,208) |
| 0.01 to 0.05 | 5.81% (1,788,668) | 1.94% (23,214) |
| 0.05 to 0.10 | 3.00% (760,578) | 0.23% (29,157) |
| 0.10 to 0.25 | 1.45% (1,699,943) | 0.032% (85,100) |
| 0.25 to 1.01 | 0.26% (15,679,090) | 0.0001% (3,283,021) |
Across the GPT-2-medium classifiers with high test accuracy, changing the padded length has the largest effect on predicted labels, changing 9.9811% overall and up to 23.5585% in one run pair, while changing numerical precision or batch size changes far fewer labels but can still move class probabilities by as much as 0.6205 and 0.5486, respectively; across both the original classifiers and the pretrained ModernBERT and Qwen3 classifiers, the label-change rate decreases steadily as the difference between the two highest class probabilities increases, from 10.43% and 8.20% when that difference is below 0.01 to 0.26% and 0.0001% when it exceeds 0.25.
The revised draft will provide the complete results by dataset, classifier formulation, model size, checkpoint, and serving condition.
Response to Reviewer mL8Z part 1
We thank the reviewer for the careful assessment and for recognizing the scale of the experiments, the separation of mechanisms, and the evaluation of downstream decisions. We address the four questions below, beginning with the two that the reviewer identifies as most important.
We used left padding to match public text classification implementations that use this convention for generative classification EMNLP outstanding paper official implementation. The Hugging Face PEFT sequence-classification guide recommends left padding for GPT and OPT classifiers and pads each batch to the length of its longest input. The public Promptzl classification library also left-pads batches containing inputs of different lengths. It calls a causal language model without supplying position indices and reads the class-token scores at the final sequence position. Its tests include GPT-2. The DaMinCi paper also uses the the same padding convention for GPT-2 classification. These sources document left padding in several independent classifier implementations.
Hugging Face assigns GPT-2 position indices differently depending on how GPT-2 is called. A direct model call without supplied position indices numbers tokens by their tensor columns. The generation code can instead number the tokens from the attention mask. Generation and direct classification can therefore assign different positions to the same left-padded tokens. The OpenCompass evaluation framework uses one tokenizer for text generation and for selecting among candidate labels by sequence likelihood. The MS-SWIFT library uses one padding implementation for causal language models and sequence classifiers during inference. These examples show that generation and classification can use the same padding convention. Our response to Reviewer 6N8S captures the full details of all the previous literature which use this convention and expound the difference between generative classifiers and using a decoder-based language model for classification via the last token embedding.
After receiving the reviews, we ran a comparison that holds the checkpoint, text, numerical precision, and batch size fixed while changing the padded length and the position indices. The table reports all five datasets for one stored checkpoint from each GPT-2 classifier formulation. The accuracy columns report test accuracy at the fixed reference length under the named position rule.
Main takeaway: Changing padded length alters GPT-2 predictions only when it also changes the learned absolute positions of real tokens; numbering tokens independently of padding removes all such label changes, but applying that a new position rule only at inference can reduce accuracy because it differs from the rule used during training.
(continued in the next comment)