acceptodds
Under review as a conference paper at ICLR 2027

Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers

Abstract

Deterministic inference is essential for reliable and trustworthy machine learning. Prior studies of text generation have shown that changing factors such as batch size, batch composition, hardware, or inference engine can substantially alter the generated text, even when the prompt, model parameters, and sampling randomness are fixed. These differences have been attributed in part to floating-point non-associativity, shape-dependent kernel selection, and other implementation-level differences in numerical execution. However, it remains unclear whether, when, and to what extent the same factors affect text classification. To the best of our knowledge, this is the first systematic study of serving-context non-invariance in text classifiers. We train 180 models (spanning discriminative, pseudo-generative, and fully generative classifier formulations), and evaluate them across four different categories of serving contexts. We show that label stability can conceal substantial score instability e.g. changing only the batch shape changes no labels across fp32 comparisons, yet under bf16 it redistributes up to 56.7 percentage points of predicted probability mass, with label flipping concentrated around small margins. Fully generative classifiers are particularly susceptible, compared to their discriminative counterpart. We derive sufficient conditions for label stability under various serving contexts and give a separate mitigation for each non-invariance mechanism as well. Taken together, our results identify and quantify the previously understudied but important serving conditions that must be factored in for reproducible text classification.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
nondeterminism, text classification, invariance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mL8ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies serving-context sensitivity in text classifiers: whether a fixed checkpoint produces different labels or class scores for the same text when padding, batch size, batch composition, numerical precision, quantization, device, or backend changes. The authors train 180 models across four classifier formulations, three model sizes, five datasets, and three seeds, with additional pretrained-model, hardware, and backend controls. The paper separates a structural padding-position effect, which changes the represented input, from numerical effects, which change the execution of a fixed represented input. The main findings are that identical serving contexts are repeatable, left padding with learned absolute position embeddings can produce substantial label changes, numerical effects mainly change low-margin predictions, and fully generative classifiers are more sensitive than discriminative classifiers. The paper also evaluates mechanism-specific mitigations and gives formal conditions for padding and label invariance.

Critical strengths
  1. Careful and large-scale experiments. The paper uses paired comparisons that hold the checkpoint and text fixed, establishes an identical-context repeatability baseline, and evaluates a large collection of classifier formulations, datasets, model sizes, seeds, serving settings, and output metrics. This provides substantially stronger evidence than isolated examples of serving-context sensitivity.
  2. Clear mechanism separation and validation. The paper distinguishes the structural padding-position effect from numerical effects and uses targeted interventions to verify the proposed causes. In particular, removing and restoring width-dependent position IDs provides convincing evidence for the padding mechanism, while the quantization controls identify the source of batch-composition dependence.
  3. Useful practical evaluation. The paper shows that stable labels do not imply stable probabilities or downstream decisions. It also evaluates mechanism-specific mitigations and reports their accuracy and computational costs. These results provide useful guidance for testing and deploying text classifiers.
Critical weaknesses
  1. The novelty and practical prevalence of the dominant effect are unclear. The largest label changes arise from manually left-padded GPT-2 models with learned absolute position embeddings and width-dependent position IDs. This mechanism and its correction are already known, and it can be avoided through right padding or real-token-order position IDs. The paper demonstrates that the effect can be large, but it does not establish how common this vulnerable configuration is in correctly implemented classifier deployments. This limits the practical significance of the headline result.

  2. The evidence for modern production classifiers is limited. The primary grid consists of models trained from random initialization, using a relatively weak task-quality threshold. The pretrained and modern-model controls are much smaller, and the paper does not test common production engines or distributed serving configurations. It is therefore unclear whether the reported magnitudes generalize to current pretrained classifiers and production systems.

  3. The theoretical contribution is not fully supported by the experiments. The padding and margin results mainly formalize direct properties of position embeddings and decision margins. The speed–accuracy–invariance theorem depends on an information-loss assumption that is not shown to hold for the evaluated systems, and the experiments do not demonstrate the proposed three-way trade-off. Further analysis would make the theory more of a major contribution than supporting material.

Questions for authors

The first two questions address the main issues affecting my assessment.

  1. Why were the GPT-2 classifiers manually left padded using width-dependent position IDs? How common is this exact configuration in real classifier deployments? Please compare against standard classification or generation pipelines that correctly construct position IDs, and clarify whether the main padding result represents a common deployment practice or an avoidable implementation error.

  2. Can the authors provide complete task-accuracy and decision-margin results for all retained models, and repeat the main summaries on classifiers with competitive task performance? How strongly are label flip rates associated with task accuracy after controlling for classifier formulation and dataset?

  3. Can the authors expand or better justify the generalization to modern production systems? In particular, it would be beneficial to test additional pretrained classifiers, current decoder models, or one production serving engine. If this is not possible, the claims should be narrowed to the evaluated configurations.

  4. Can the authors instantiate the premise of the speed–accuracy–invariance theorem for any evaluated implementation? If not, what empirical conclusion does this theorem support beyond stating a conditional possibility?

8 comments
AuthorsOpenReview ↗

Response to Reviewer mL8Z part 1

We thank the reviewer for the careful assessment and for recognizing the scale of the experiments, the separation of mechanisms, and the evaluation of downstream decisions. We address the four questions below, beginning with the two that the reviewer identifies as most important.

  1. Why were the GPT-2 classifiers left padded, how common is this configuration, and how does it compare with correct position handling?

We used left padding to match public text classification implementations that use this convention for generative classification EMNLP outstanding paper official implementation. The Hugging Face PEFT sequence-classification guide recommends left padding for GPT and OPT classifiers and pads each batch to the length of its longest input. The public Promptzl classification library also left-pads batches containing inputs of different lengths. It calls a causal language model without supplying position indices and reads the class-token scores at the final sequence position. Its tests include GPT-2. The DaMinCi paper also uses the the same padding convention for GPT-2 classification. These sources document left padding in several independent classifier implementations.

Hugging Face assigns GPT-2 position indices differently depending on how GPT-2 is called. A direct model call without supplied position indices numbers tokens by their tensor columns. The generation code can instead number the tokens from the attention mask. Generation and direct classification can therefore assign different positions to the same left-padded tokens. The OpenCompass evaluation framework uses one tokenizer for text generation and for selecting among candidate labels by sequence likelihood. The MS-SWIFT library uses one padding implementation for causal language models and sequence classifiers during inference. These examples show that generation and classification can use the same padding convention. Our response to Reviewer 6N8S captures the full details of all the previous literature which use this convention and expound the difference between generative classifiers and using a decoder-based language model for classification via the last token embedding.

After receiving the reviews, we ran a comparison that holds the checkpoint, text, numerical precision, and batch size fixed while changing the padded length and the position indices. The table reports all five datasets for one stored checkpoint from each GPT-2 classifier formulation. The accuracy columns report test accuracy at the fixed reference length under the named position rule.

Classifier formulationDatasetTest examplesLabels changed when padded length changes, tensor-column positionsLabels changed when padded length changes, real tokens numbered from zeroAccuracy with tensor-column positionsAccuracy with real tokens numbered from zero
Pseudo-generative GPT-2AG News2,0009.70%0.00%92.50%87.65%
Pseudo-generative GPT-2Emotion2,00014.05%0.00%92.35%86.00%
Pseudo-generative GPT-2IMDb2,0002.30%0.00%86.15%84.85%
Pseudo-generative GPT-2SST-21,82119.38%0.00%74.41%64.52%
Pseudo-generative GPT-2SST-52,00035.80%0.00%35.55%33.60%
Fully generative GPT-2AG News2,0006.95%0.00%89.65%89.40%
Fully generative GPT-2Emotion2,00012.10%0.00%85.15%82.30%
Fully generative GPT-2IMDb2,0001.00%0.00%85.45%86.10%
Fully generative GPT-2SST-21,82124.16%0.00%68.42%68.20%
Fully generative GPT-2SST-52,00042.65%0.00%33.55%32.45%

Main takeaway: Changing padded length alters GPT-2 predictions only when it also changes the learned absolute positions of real tokens; numbering tokens independently of padding removes all such label changes, but applying that a new position rule only at inference can reduce accuracy because it differs from the rule used during training.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer mL8Z part 2

(continued from the previous comment)

Numbering the real tokens from zero removes all label changes caused by padding length. It also gives an existing checkpoint different position embeddings from those used during training. We therefore tested a third rule. Each real token receives the same absolute position it occupied when the input was padded to 512 tokens during training. Total variation distance measures the change in the complete vector of class probabilities, and the last column reports the largest change in any class score before softmax. The fixed-width reference is the prediction obtained with tensor-column positions when every input is padded to 512 tokens. Each row pools all five datasets and contains 9,821 matched examples for the named classifier formulation.

Classifier formulationPosition assignmentPadded lengthMatched examplesLabels changed versus the fixed-width tensor-column referenceMean probability total variation versus that referenceLargest absolute class-logit change versus that reference
Pseudo-generative GPT-2Tensor-column positionsDetermined by longest input in batch9,82116.247%0.125617.40
Pseudo-generative GPT-2Real tokens numbered from zeroFixed or determined by batch9,82116.775%0.142513.67
Pseudo-generative GPT-2Positions preserved from training width 512Determined by longest input in batch9,8210.000% (0 of 9,821)2.55e-71.81e-5
Fully generative GPT-2Tensor-column positionsDetermined by longest input in batch9,82117.373%0.101276.49
Fully generative GPT-2Real tokens numbered from zeroFixed or determined by batch9,82117.597%0.106076.49
Fully generative GPT-2Positions preserved from training width 512Determined by longest input in batch9,8210.000% (0 of 9,821)1.89e-62.56e-3

Preserving the positions used during training produces the same label as the training-width computation for every evaluated input. Each batch can still use a shorter padded length. The revised draft will report all five datasets for the evaluated GPT-2 checkpoints and all three position rules.

Right padding also prevents padded length from changing the position indices of real tokens. Our result identifies why the public left-padding implementations can change predictions and gives a correction that preserves the predictions of an existing checkpoint.

AuthorsOpenReview ↗

Response to Reviewer mL8Z part 3

2. Can the authors report complete task accuracy and show how the results vary with the separation between the two highest class probabilities? Do the results hold for accurate classifiers, and are label changes related to task accuracy?

Yes, we do observe prediction instabilities on classifiers with high test accuracy.

We computed complete accuracy and decision-margin results for every checkpoint in the original and pretrained sets. The decision margin is the difference between the two largest class probabilities before the serving conditions are changed. The revised draft will provide the complete accuracy, F1, calibration, predicted-class, and decision-margin results by dataset, classifier formulation, model size, and checkpoint.

Here we present the representative case of the GPT-2-medium classifiers, which achieve test accuracy from 0.8446 to 0.8660 on SST-2, 0.9343 to 0.9387 on AG News, and 0.9403 to 0.9444 on IMDb. We evaluated the same checkpoints and texts while changing one serving condition at a time.

The following terms define the columns in the table:

TermMeaning
Run pairTwo inference runs using the same checkpoint and texts and differing only in the serving condition named in the row
Prediction comparisonOne text whose prediction is compared across one run pair; the same text can appear in several run pairs
Overall label-change rateTotal number of changed labels divided by all prediction comparisons
Average label-change rate per run pairLabel-change rate computed separately for each run pair and then averaged so every run pair receives equal weight
Largest label-change rate in one run pairHighest label-change rate observed in any individual run pair
Run pairs with label changesNumber of run pairs in which at least one text receives a different label
Average probability movementAverage total variation distance between the two class-probability vectors across all prediction comparisons
Largest probability movementLargest total variation distance observed for any individual text
Average largest class-score changeFor each text, the largest absolute change among its raw class scores, averaged over all prediction comparisons
Change testedRun pairsPrediction comparisonsOverall label-change rateAverage label-change rate per run pairLargest label-change rate in one run pairRun pairs with label changesAverage probability movementLargest probability movementAverage largest class-score change
Length to which the batch is padded917,4639.9811%10.3359%23.5585%90.11211.000026.4
Numerical precision45516,3150.2849%0.4557%1.3180%450.0039950.62050.797
Batch size2162,478,3120.0189%0.0328%0.9885%180.00037540.54860.074
Other examples in the batch54104,7780.0000%0.0000%0.0000%00.00000.00000.0000
Order of the other examples54619,5780.0000%0.0000%0.0000%05.043e-105.245e-62.63e-8
Row occupied by the measured example2752,3890.0000%0.0000%0.0000%00.00000.00000.0000

We also measured how the decision margin relates to label changes. The number in parentheses is the number of individual prediction comparisons in that interval.

Decision marginOriginal classifiers: label-change rate and prediction comparisonsPretrained ModernBERT and Qwen3: label-change rate and prediction comparisons
0.00 to 0.0110.43% (2,016,637)8.20% (5,208)
0.01 to 0.055.81% (1,788,668)1.94% (23,214)
0.05 to 0.103.00% (760,578)0.23% (29,157)
0.10 to 0.251.45% (1,699,943)0.032% (85,100)
0.25 to 1.010.26% (15,679,090)0.0001% (3,283,021)

Across the GPT-2-medium classifiers with high test accuracy, changing the padded length has the largest effect on predicted labels, changing 9.9811% overall and up to 23.5585% in one run pair, while changing numerical precision or batch size changes far fewer labels but can still move class probabilities by as much as 0.6205 and 0.5486, respectively; across both the original classifiers and the pretrained ModernBERT and Qwen3 classifiers, the label-change rate decreases steadily as the difference between the two highest class probabilities increases, from 10.43% and 8.20% when that difference is below 0.01 to 0.26% and 0.0001% when it exceeds 0.25.

The revised draft will provide the complete results by dataset, classifier formulation, model size, checkpoint, and serving condition.

Reviewer_oCLxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the non determinism in text classifiers, where given a fixed text and fixed classifier, the prediction scores can change leading to potentially altered labels. This change can happen when the serving setup changes, such as padding width, batch size, precision, int8 compression, hardware and runtime. The authors train 180 classifiers (generative and discriminative) from scratch, then change one setting at a time. They find that the padding has the largest effect, and that the probabilities change often without flipping the labels. Finally, they propose some mitigations to improve determinism.

Critical strengths

The study is comprehensive and its claims are supported by several results. The authors provide a control experiment where they showed the reruns are bit identical, then isolated each change in different setups. This strengthens the main findings of the paper.

Critical weaknesses
  • The major weakness of the paper is that it offers little new insights. Many central findings in the paper were already published or well-known in other settings and are therefore expected to hold in text classification as well. Also claims like "this is the first systematic study of serving-context non-invariance in text classifiers" are not justified, as the same or similar effect is already measured in text classification. The paper also fails to cite important relevant work. Concrete examples are:

    • "Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression" (Xu et al., 2021, EMNLP) measured per-example label and probability changes of a fine-tuned BERT classifier under quantization.
    • "Accuracy is Not All You Need" (Dutta et al., 2024, NeurIPS) and "When Quantization Affects Confidence of Large Language Models?" (Proskurina et al., 2024, NAACL Findings) showed that quantization flips or shifts the decisions of LLMs used as classifiers mostly on low-confidence examples.

Examples from other settings: - "The Curious Case of Absolute Position Embeddings" (Sinha et al., 2022, EMNLP Findings) showed that models with absolute position embeddings degrade when the input starts at a shifted position. Hugging Face's GPT-2 documentation also states: "Pad inputs on the right because GPT-2 uses absolute position embeddings." - "Architectural Backdoors for Within-Batch Data Stealing and Model Inference Manipulation" (Küchler et al., 2026, SaTML) found that quantization affects output in a batch.

  • Another weakness is that the findings are not impactful in practice. For instance, the biggest concern for determinism is padding, which occurs only in left-padded models with absolute position embeddings. It is avoided by right padding (as Hugging Face recommends for GPT-2) or by counting positions from the first real token, and the paper itself measures 0 flips for ModernBERT and Qwen3. Other effects are negligible.

  • Finally the formal results in section 5.5 add very little insights and do not meaningfully contribute to the paper's findings. Theorem 5.1 states a known fact, Proposition 5.1 is a standard margin bound, and theorem 5.2 relies on assumptions not relevant for the classifiers considered.

Questions for authors
  • Given all the prior work mentioned in weaknesses, can the authors clarify which of their findings are new or not already expected by the prior work?
  • What is a practical use case where the findings of this paper would matter or fix a problem that was otherwise not solvable?
  • Are the authors aware of this work: "Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?" Khatri 2026. This is concurrent work that the authors might consider as related work as it is highly relevant.
10 comments
AuthorsOpenReview ↗

Response to Reviewer oCLx - Part I

We thank the reviewer for recognizing the comprehensiveness of the study and the controls used to isolate changes in inference. We address the questions about prior work, practical significance, and the formal results below.

1. “Given all the prior work mentioned in weaknesses, can the authors clarify which of their findings are new or not already expected by the prior work?”

The cited papers are relevant, and we will include them in the revised discussion. We respectfully don't fully agree that these studies already establish the findings of our submission. Our study compares how serving changes affect discriminative, pseudo-generative, and fully generative text classifiers, whose predictions are computed differently. A discriminative classifier applies a classification head to a representation; a pseudo-generative classifier predicts the label through a language-model objective; and a fully generative classifier compares the likelihood of the input under each candidate label. These differences matter when determining how numerical changes affect the final class scores.

Xu et al. study label and probability agreement after BERT compression. Dutta et al. examine answer changes and probability differences under LLM compression, while Proskurina et al. study quantization-induced changes in confidence and calibration. These establish important precedents for evaluating individual predictions rather than relying on accuracy alone. Our study extends this evaluation to comparisons across classifier formulations and to changes in batching, padding, precision, and execution within a fixed deployment. Our study compares these effects across classifier formulations and distinguishes changes made when configuring a deployment from changes that occur as requests are batched within that deployment. Even with the checkpoint, numerical precision, and quantization method fixed, the prediction for an input can depend on which other requests are processed alongside it.

The following table identifies the overlap and the additional experiments in our submission. “Yes” indicates that the aspect is explicitly studied; “—” indicates that it is not separately evaluated.

WorkCompares all three classifier formulationsPer-example label or probability changesPadding or position shiftsBatch size or other examples in the batchPrecision or quantizationDownstream decisions beyond the predicted labelLatency of batch-invariant execution
Xu et al., EMNLP 2021—Yes——Yes——
Dutta et al., NeurIPS 2024—Yes——Yes——
Proskurina et al., NAACL Findings 2024—Yes——Yes——
Sinha et al., EMNLP Findings 2022—Task accuracy and perplexityYes————
Küchler et al., SaTML 2026—Attack outcomes—YesYes——
Khatri, 2026—Yes—YesYes——
Our submissionYesYesYesYesYesYesYes

The final column specifically concerns the runtime of inference that preserves outputs across batch shapes. Xu et al. also measure compression speed and methods for improving agreement, while Küchler et al. measure the runtime of their offline batch-isolation checker. These are different measurements and should receive appropriate credit.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer oCLx - Part II

(continued from the previous comment)

Our experiments show that the same change in serving conditions can affect classifier formulations differently. Under reduced precision, changing batch shape produced a largest observed class-score change of approximately 0.447 for the fully generative classifier and 0.03125 for the discriminative classifier in the compared configurations. These are changes in the scores before conversion to class probabilities. Fully generative classification sums token log probabilities across the text, so changes in individual token contributions can accumulate in the final class score. Appendix E.4 bounds this accumulation in terms of text length, and the experiments show a consistent trend: score changes increase as the number of text tokens increases, even when the padded width is held fixed.

The experiments also establish that batch shape has a differential effect depending on numerical precision. Under full precision, changing batch shape produces no label changes across 226,104 comparisons, with probability changes below 10−510^{-5}, whereas under reduced precision the same type of change can produce substantially larger probability changes. This comparison shows why the effect of batching must be evaluated together with the precision used for inference.

Our experiments also show that label agreement can substantially underestimate changes in downstream decisions. Appendix B.21 evaluates probability thresholds, abstention, prediction sets, and ranking under the same serving changes. The decision rules are fitted using the reference configuration and then held fixed. Across 799,620 full-precision-to-bf16 comparisons, 0.991% of predicted labels change, whereas 3.077% of threshold-based decisions change. The additional finding is that numerical changes affect decisions even when they are insufficient to change the highest-probability class. For an application that uses these probabilities to filter an input or send it for review, checking label agreement would therefore miss part of the effect.

Sinha et al. study sensitivity to shifted absolute positions, including on classification tasks, while Küchler et al. identify dependence between batch entries through dynamic quantization. Our experiments measure how these effects change classifier predictions and show what happens when their causes are removed. Assigning positions by real-token order eliminates the padding-induced label changes; deliberately restoring column-based positions brings them back on the same models. In the quantization experiments, replacing the other examples in a batch changes the target prediction even though batch size and padded width remain fixed. For the tested GPT-2 classifier, changing only the other examples produces label changes in 8 of 384 comparisons at batch size 8 and 14 of 384 comparisons at batch size 256, corresponding to 2.08% and 3.65%, respectively. The batch dimensions remain fixed within each comparison, so these changes cannot be attributed to additional padding or a different tensor shape. Computing the activation scale independently of those examples removes this dependence.

We also measure how much additional computation is required to negate the instabilities in prediction arising from kernel non-invariances. Section 5.4 and Appendix B.14 evaluate batch-invariant execution at matched numerical precision, obtaining bitwise-identical outputs across the tested batch shapes with approximately 1.3–1.5× latency at large batch sizes. Appendix B.22 examines selective recomputation: predictions with small margins are reevaluated in a fixed reference configuration. On 1,274 held-out examples, this policy detects all 44 observed label disagreements while recomputing 22.45% of requests. These measurements quantify the cost of preserving outputs across batch shapes and the extent to which selective recomputation can recover reference labels.

We will replace the broad “first systematic study” wording with a description of the specific contribution: a controlled comparison of serving-context sensitivity across classifier formulations, including downstream decision changes and the accuracy and execution costs of remedies.

AuthorsOpenReview ↗

Response to Reviewer oCLx - Part III

2. “Another weakness is that the findings are not impactful in practice.” / “What is a practical use case where the findings of this paper would matter or fix a problem that was otherwise not solvable?”

Consider a message-filtering service that is evaluated in full precision and then deployed with reduced precision and batching to improve throughput. The model checkpoint, input text, and filtering threshold remain unchanged. Before adopting this configuration, the practitioner needs to establish whether it preserves the filtering decisions and whether those decisions depend on the other requests processed alongside an input. Our experiments examine both questions and evaluate remedies together with their accuracy and execution costs.

For a binary classifier, a probability changing from 0.81 to 0.79 leaves the predicted class unchanged but changes the action under a threshold of 0.8. Appendix B.21 measures this distinction with decision rules fitted on the reference configuration and then held fixed. Across the evaluated full-precision-to-bf16 comparisons, 0.991% of labels change, compared with 3.077% of threshold-based decisions. In this example, the classifier remains confident in the same class, but the filtering decision changes because the probability crosses the application’s threshold. Concentration of label changes among low-confidence examples therefore does not establish that other decisions are unaffected. The relevant comparison is whether the probability change crosses the boundary used by the application.

The dependence on other requests also appears in our physical-device experiments with a BERT classifier. Appendix B.20 evaluates the same BERT SST-2 classifier on a Galaxy S25 and Pixel 8a, holding batch size and padded width fixed. With dynamic int8 quantization, replacing the other seven texts in the batch changes all 64 target probability vectors on the phone CPUs. Under partial NNAPI execution on the Galaxy, the experiment also changes one threshold-based action despite changing no predicted labels. The static-int8 exports produce bitwise-identical target outputs under the same change of neighboring texts on both phone CPUs. For a practitioner investigating inconsistent decisions, rerunning the affected input alone does not reproduce the other requests that determined its quantization scale. Our experiment tests this dependence directly by keeping the target input fixed and replacing its batch neighbors, then shows that fixing the scale removes the observed dependence on those neighbors.

We agree that right padding or real-token-order positions remove the structural position-shift mechanism, and our controls explicitly test this. As documented in our response to Reviewer 6N8S (Parts 3–4), left padding appears in public classifier implementations (essentially we did not conveniently choose this for inflating the headline numbers), while changing the position convention only at inference can reduce accuracy by up to 9.88 percentage points (Appendix B.7). Removing padding dependence and preserving an existing classifier’s accuracy therefore require separate checks.

However, the zero padding-induced flips for ModernBERT and Qwen3 do not establish that the other effects are negligible. The 3.077% threshold-decision change under reduced precision and the dependence on neighboring requests observed with a BERT classifier on physical phones arise from separate numerical mechanisms. Right padding does not address those mechanisms, and label agreement does not establish that filtering or review decisions remain unchanged. Their practical significance must be assessed against the decisions the application actually makes.

The paper also measures what preserving predictions costs. Batch-invariant execution incurs approximately 1.3–1.5× latency at large batch sizes in our measurements. Appendix B.22 examines an alternative that recomputes selected predictions in a reference configuration: on 1,274 held-out examples, the label-margin policy detects all 44 observed label disagreements while recomputing 22.45% of requests. The separate threshold policy detects 18 of 21 action disagreements at a 4% recomputation rate. These results let practitioners assess the observed coverage and cost of a remedy for the decision their application actually uses.

The practical problem is preserving a classifier’s validated decisions while changing how efficiently it is served. Our results show why the appropriate remedy depends on the cause: changing position assignment can remove padding dependence while affecting a previously trained model’s accuracy, fixed quantization scales remove the observed dependence on neighboring requests, and batch-invariant execution has a measurable latency cost. These experiments provide the evidence needed to choose a remedy and check that it preserves the behavior required by the application.

Reviewer_6N8SOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper evaluates serving-context sensitivity in text classifiers across padding, batching, precision, quantization, and execution backends. Experiments include 180 models spanning four formulations, three sizes, five datasets, and three seeds. The dominant padding effect arises from left padding with width-dependent absolute position indices (20.1% label disagreement across the tested GPT-2 comparisons), rather than numerical execution of an unchanged represented input. Consistent, padding-independent position assignment removes the observed padding-induced label flips in the controls. The paper also measures numerical score drift, demonstrates that stable labels can conceal probability changes, and evaluates stability conditions and mechanism-specific mitigations.

Critical strengths
  1. The evaluation is detailed and beyond label agreement. The paper measures probability drift and its effects on downstream decisions, rather than relying solely on classification accuracy. For example, switching from fp32 to bf16 changes approximately 0.99% of labels but 3.08% of threshold-based decisions. This provides concrete evidence that label-level stability can underestimate operational sensitivity, making the evaluation relevant to deployed classifiers.
  2. The paper presents controlled diagnosis and cost-aware mitigation. Fixed-shape quantization experiments isolate batch-global activation scales as a source of cross-example dependence, and fixed-scale controls remove that dependence. The paper also evaluates mitigation costs, including a 1.3–1.5× large-batch latency overhead for batch-invariant execution. These controlled comparisons and cost measurements provide actionable evidence for choosing deployment configurations.
Critical weaknesses

The dominant padding result is caused by incorrect position-ID handling. For models with learned absolute position embeddings, correctly implemented left padding should preserve the logical position IDs of real tokens. The evaluated baseline instead assigns positions by tensor column, so changing padding width changes position_ids and therefore the actual input embeddings. Different predictions are an expected consequence of feeding the model different inputs, not a novel inference instability. The reported 20.1% label disagreement is thus driven by an improperly configured baseline. Indeed, the paper’s own correctly configured control produces zero flips in 25,986 comparisons. Presenting this basic implementation error and its standard correction as a central finding substantially undermines the paper’s novelty and the interpretation of its headline result.

Questions for authors

Why does the main GPT-2 baseline use width-dependent position ids rather than preserving real-token positions under left padding? I think this is an incorrectly configured baseline rather than a realistic deployment practice. Besides, even without inputing the correct input_ids, the problem can be easy to address using right padding and get the last token representation using torch.gather.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 6N8S - Part 1

We thank the reviewer for their positive comments on all the other aspects as mentioned in the strengths and would like to clarify the main weakness & questions raised around the padding experiment.

1. Why does the main GPT-2 baseline use width-dependent position ids rather than preserving real-token positions under left padding? I think this is an incorrectly configured baseline rather than a realistic deployment practice.

We respectfully would like to clarify a possible misunderstanding here between a generative classifier and a decoder language model being used for discriminative classification. As mentioned in the paper, we are following the convention defined in Li et al. (ICLR 2025, Sec. 4.3) and Kasa et al. (2025, Sec. 3.2) , a generative classifier is where we evaluate the joint likelihood of the label text along with the input text, and take an argmax over all the label texts to get the final predicted label. This is materially different from using a generative model such as GPT-2 for sequence classification where the final token representation is mapped to the class logits. The widely used GPT2ForSequenceClassification for training and inference but falls under the latter category; in fact, the recommendation that the reviewer is pointing to "Besides, even without inputing the correct input_ids, the problem can be easy to address using right padding and get the last token representation using torch.gather." is specifically for the latter discriminative approach, not the former generative one. Below we give a brief overview of these different classifier paradigms

ClassifierFormulationExample
Discriminative classification head (GPT2ForSequenceClassification)pθ(y∣x)=softmax⁡(Whread(x)+b)yp_\theta(y\mid x)=\operatorname{softmax}(Wh_{\mathrm{read}}(x)+b)_yInput: A wonderful movie. A classification head maps the final real-token representation to class logits.
Pseudo-generative label-token scorerpθ(vy∣x,m)=softmax⁡(zθ(x,m))vyp_\theta(v_y\mid x,m)=\operatorname{softmax}(z_\theta(x,m))_{v_y}Input: A wonderful movie. [label_gen] Compare the next-token probabilities of [positive] and [negative].
Fully generative class-conditional scorerpθ(y∣x)=softmax⁡(s(x)+log⁡π)yp_\theta(y\mid x)=\operatorname{softmax}(s(x)+\log\pi)_yCompare the text likelihood under [label_gen] [positive] A wonderful movie. with that under [label_gen] [negative] A wonderful movie.
Here, hreadh_{\mathrm{read}} is the representation used by the classification head, mm is the generation marker, vyv_y is a class token, zz denotes vocabulary logits, and π\pi contains the class priors.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer 6N8S - Part 2

This is not an incorrectly configured baseline nor something we conveniently made up for the sake of reporting huge number in this paper: The GPT-2 baseline follows the evaluation convention in the official implementation accompanying Kasa et al. (2025), an EMNLP 2025 Outstanding Paper. That implementation explicitly left-pads evaluation inputs and calls GPT2LMHeadModel without supplying explicit position IDs. GPT-2’s default forward path consequently assigns positions by tensor column. The combination being evaluated therefore has a concrete precedent in a published peer-reviewed papers at reputed venues.

Left padding also appears in discriminative GPT-2 classification implementations accompanying RoAST (Findings of EMNLP 2023), Marjanović et al. (Findings of ACL 2024), and Zhao et al. (ICML 2025). Hugging Face’s PEFT sequence-classification guide additionally specifies left padding for GPT/OPT/BLOOM and uses longest-in-batch padding. These examples motivate evaluating the stability of this convention and comparing it against position-preserving alternatives.

We compile a short list of some famous peer reviewed works published at reputable venues which use this convention below: (continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer 6N8S - Part 3

Public precedents for left padding in text classification. Left padding appears in released implementations of both generative causal-language-model label scoring and discriminative sequence classification. The table below identifies the classifier formulation and the specific supporting evidence.

SourceClassifier formulationEvidence of left paddingPublic implementation or documentation
EvoPrompt, ICLR 2024, Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt OptimizersCausal language-model label-token scoringThe causal-LM tokenizer is configured with padding_side="left". The classification path directly calls the model and compares verbalizer logits at the final sequence position.tokenizer configuration; label-token scoring
Promptzl, public classification libraryCausal language-model label-token scoringIts causal classifier explicitly loads a left-padding tokenizer and selects final-position verbalizer logits from a direct model forward pass.Classifier implementation
RoAST, Findings of EMNLP 2023, Kim et al.Discriminative GPT2ForSequenceClassificationThe GPT-2 branch explicitly sets tokenizer.padding_side = "left", uses EOS as PAD, and loads a sequence-classification head.Paper; model and tokenizer configuration
Marjanović et al., Findings of ACL 2024, Investigating the Impact of Model Instability on Explanations and UncertaintyDiscriminative GPT2ForSequenceClassificationThe GPT-2 training branch explicitly sets left padding and EOS as PAD, and instantiates GPT2ForSequenceClassification.padding configuration; classifier construction
Zhao et al., ICML 2025, Unlocking Post-hoc Dataset Inference with Synthetic DataDiscriminative GPT-2 sequence classifierThe released GPT-2 classifier wrapper explicitly sets left padding and EOS as PAD. The implementation includes GPT2ForSequenceClassification construction and evaluation.classifier implementation
ElasticFlow, ASPLOS 2023, released Pollux testbed workloadDiscriminative GPT2ForSequenceClassificationThe included GPT-2 workload explicitly sets left padding. Its collator uses padding=True, padding each batch to its longest sequence.Research artifact; padding configuration; batch collator
DaMinCi, IberLEF 2022, Cabestany et al.Discriminative GPT-2 sequence classifierSection 4.1 explicitly describes using left padding with the final-token representation for GPT-2 classification. The reported experiment uses batch size 1.Paper, Section 4.1
Hugging Face PEFT v0.7.1, official sequence-classification guideDiscriminative sequence classification with prompt tuningThe guide explicitly instructs users to select left padding for GPT, OPT, and BLOOM models and uses DataCollatorWithPadding(..., padding="longest"). Its worked example uses RoBERTa; the left-padding instruction applies to the decoder-model alternatives.Official guide

These examples establish that left padding is a documented choice in public classification implementations, including implementations accompanying published research and official library guidance. The causal-LM examples above concern label-token scoring.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised manuscript and summary of changes

We thank the reviewers and Area Chair for their time and feedback.

We have uploaded a revised manuscript incorporating the additional experiments and clarifications presented during the rebuttal. The main changes are:

Padding and position assignment. We clarify when changing padded length changes the position embeddings supplied to a classifier. We also demonstrate a correction for existing GPT-2 checkpoints: retaining the position IDs assigned by the fixed-width reference preserves all 9,821 held-out predictions for each GPT-2 formulation while allowing variable padded lengths, without retraining (§§5.2–5.4; Appendix B.8).

Classifier quality and larger pretrained models. We provide accuracy, predicted-class diversity, and decision-margin summaries for all 180 original classifiers, including those excluded from the main comparisons, and for the pretrained ModernBERT and Qwen3 classifiers. Additional experiments with accurate GPT-2-medium classifiers reproduce the padding sensitivity: changing padded length changes 9.98% of 17,463 compared predictions (Appendices B.2 and B.10).

Serving systems and practical consequences. We add Qwen3 inference experiments using vLLM, evaluations on two physical phones, and results for threshold decisions, abstention, prediction sets, and rankings. The vLLM comparison explicitly accounts for differences between repeated single-request executions. We also report mitigation costs and evaluate selective recomputation on held-out examples (Appendices B.16 and B.22–B.25).

Comparison with prior work. We expand the discussion of related studies and add a comparison table identifying the classifier formulations, serving conditions, and outputs examined by each work. We also document public classifier implementations that use left padding (§2, Table 1, and Appendix D).

Formal results. We clarify the assumptions and interpretation of the formal statements and provide an explicit quantization example illustrating the conditional infeasibility result, with its information-access restrictions and computation costs defined (§5.5; Appendix E.6).

These additions connect the controlled comparisons to accurate pretrained classifiers, serving implementations, and decisions that depend on classifier probabilities.

AuthorsOpenReview ↗

Summarizing the Author-Reviewer Discussion Phase

We thank the reviewers and the Area Chair for the constructive discussion. The reviews consistently recognized the paper’s careful experimental design, controlled mechanism analysis, evaluation beyond label agreement, and practical assessment of mitigation costs.

Reviewer mL8Z engaged with our responses and confirmed that the public implementation evidence, stronger classifiers, and production-serving experiment addressed their concerns. The reviewer stated that they had no further questions, described the experiments as valuable, and maintained their weak-accept score.

Reviewer oCLx also engaged throughout the discussion. The reviewer acknowledged that the findings may be useful to the text-classification community, agreed that prior work does not study our exact problem, and described the paper as very well executed. Their remaining reservation concerns the perceived novelty and impact. We addressed this by reviewing the cited literature and clarifying the complete contribution: prior work studies individual ingredients in different settings, whereas our paper systematically measures how changing serving context affects one fixed text classifier, identifies the mechanisms responsible, connects probability changes to downstream decisions, and evaluates targeted remedies and their costs.

Reviewer 6N8S focused on the left-padding configuration. We showed that this configuration appears in public classifier implementations and arises naturally when decoder-only models are adapted for batched classification. More importantly, our controlled interventions separate physical padding from position assignment and establish the exact cause of the observed changes. The reviewer’s concern therefore does not invalidate the experiment. Reviewer 6N8S did not respond to these results during the discussion, despite the Area Chair’s reminder, and did not contest the paper’s numerical, quantization, downstream-decision, or mitigation findings.

The discussion therefore resolved the practical concerns raised by the supporting reviewer, clarified the paper’s distinction from prior work, and directly answered the criticism of the padding experiment with implementation evidence and controlled causal tests. We respectfully ask the Area Chair to weigh the submission as a whole, including the reviewers’ stated strengths, Reviewer mL8Z’s confirmation that their concerns were addressed, Reviewer oCLx’s acknowledgment of usefulness and exact distinction from prior work, and our unanswered response to Reviewer 6N8S.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.