acceptodds
Withdrawn from ICLR 2027

Prediction Instability and Task Utility in Low-Precision Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selective precision and quantization repair are evaluated through changes in predictions, reconstruction error, and task accuracy. We examine how these measurements relate using a controlled audit of low-precision language models. A prespecified experiment evaluates two instruction-tuned models on 1,024 HellaSwag training examples across four precision paths and three execution layouts. Changing option batch size while preserving padding changes per-item precision-upgrade utility on 42 examples for Qwen and 14 for SmolLM2; corresponding full-model FP32 arithmetic controls show zero changes. Within fixed 256-example margin selections, opposing changes cancel, leaving aggregate shifts of -6 and +1 net-correct units. Subsequent SmolLM2 repair diagnostics compare four teacher-target objectives with the same rank-16 parameterization. Centered-score fitting reduces training error by 46.17%, while error on two evaluation panels increases by 15.24% and 8.16%. On the previously observed 1,024-question transfer panel, it yields three additional correct answers, compared with five for option-level distillation. An ARC replay further distinguishes offline fallback utility from executed predictions and cost. The measurements identify where prediction stability and reconstruction fidelity diverge from task utility, and provide a protocol for evaluating these quantities together.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization, selective precision, numerical reproducibility, knowledge distillation, language model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_hMuFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes prediction instability and downstream task utility in low-precision language models. The authors examine how execution layouts affect precision upgrade utility per sample and aggregate fallback policy performance. The paper also compares offline estimates with actual fallback execution and investigates whether improving reconstruction fidelity through low-rank correction leads to better task accuracy.

Critical strengths
  • Separating per sample rescue and harm cases, aggregate policy gains and total accuracy provides more informative evaluation the reporting prediction instability or accuracy alone.
  • The experimental setup includes useful controls, such as fixed sample selection and FP32 arithmetic controls to isolate the effects of different execution configurations.
Critical weaknesses
  • Insufficient evidence for the uncertainty based motivation. The paper starts from the premise that uncertainty score can identify error prone inputs in low-precision language models, but does not directly examine this relationship. The reported fallback results do not clearly establish how well uncertainty predicts baseline correctness.
  • Limited experimental scope. The main experiments use only two small language models (mainly around 1B scale), a single dataset and NF4 quantization. It is unclear whether the findings generalize to larger models or other quantization settings such as FP4, FP8 or Int8, which are particularly relevant to practical low-precision inference.
  • Limited analysis of the main findings. The paper shows that batch size changes affect per sample utility, while positive and negative changes largely cancel out at the aggregate level. However, such cancellation is not specific to batch size changes and may naturally arise when utility changes have mixed signs. The paper does not sufficiently establish whether the observed result is systematic.
Questions for authors
  • How well do the uncertainty scores predict the correctness of low-precision models? Could the authors report baseline error rates and rescue/harm rates across different uncertainty levels to better support motivation?
  • Do the findings hold for larger language models (at least > 7B) or other quantization settings? Could the authors extend the experiments to more representative low-precision inference scenarios?
  • Could the authors provide further analysis of why observed utility changes largely cancel out? Does this pattern hold across different models and quantization settings?
Reviewer_6MGqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the divergence between prediction stability, reconstruction fidelity, and actual task utility in low-precision language models, offering a rigorously controlled audit of precision-upgrade strategies. By examining how varying execution layouts and precision controls alter individual item utility versus aggregate policy drift, the authors present a compelling case that item-level instability does not necessarily translate to systemic failure.

Critical strengths

The paper reports its negative results and experimental protocol transparently. However, I do not find a substantive methodological or empirical contribution that offsets the concerns, see below.

Critical weaknesses
  • While without clear evidence, the paper appears to be AI-generated. Just by looking at the abstract and intro, we can see it pretty clearly that the paper flow is broken. For example, the paper moves between prediction instability, fallback utility, and reconstruction error without clearly explaining how these questions fit together.
  • The contribution and main hypothesis (item-level instability produces substantial drift in fallback-policy utility) are limited and not supported at the same time.
  • The experiments are limited in scope. The main study uses two small models, one quantization method, and a single HellaSwag dataset.
  • Policy utility and cost are never established together on the main experiment. Its principal HellaSwag comparisons are composites of saved Q and H predictions, rather than executed fallbacks. The actual execution and cost study instead uses an earlier ARC panel, answer-letter scoring, a different gate, and Qwen alone.
Questions for authors
  • Can you execute the same HellaSwag margin policy used in Table 1 and report realized predictions, latency, and peak memory against Q-only and H-only at each layout?
  • Which Q and H operations account for the native layout changes? Can kernel or arithmetic ablations distinguish BF16 effects from NF4-specific behavior?
Reviewer_QgX2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates prediction instability and task utility in low-precision language models. The authors examine how execution configurations affect prediction outcomes and selective precision-upgrade utility, and whether improvements in quantization reconstruction translate into task accuracy. Experiments are conducted on two small language models, with additional analyses of fallback execution and low-rank repair.

Critical strengths
  • The paper investigates a relevant problem concerning the reliability of low-precision LLM inference.

  • The experiments include controlled comparisons of precision configurations and detailed reproducibility information.

  • The paper is well-written and easy to follow.

Critical weaknesses
  • The main findings, including the cancellation of individual prediction changes and the mismatch between reconstruction error and task accuracy, are relatively intuitive. The paper provides limited new methodological or theoretical insights beyond these empirical observations.

  • The paper combines prediction instability, selective precision fallback, and quantization repair, but the connections between these analyses are not sufficiently established. It is unclear what unified research contribution or actionable insight emerges from the proposed evaluation framework.

  • The paper does not demonstrate meaningful improvements in model accuracy or inference efficiency. In particular, the selective fallback implementation is slower and less accurate than direct BF16 inference, while the low-rank repair experiments yield only marginal or inconsistent accuracy improvements.

Questions for authors
  • What are the main novel insights beyond the observation that prediction instability and task utility are not necessarily correlated?

  • How do the fallback and quantization repair experiments contribute to a unified research objective?

  • What practical guidance or improvements can be derived from the proposed evaluation framework?

  • Can the authors validate their findings on larger models and more diverse tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.