- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes prediction instability and downstream task utility in low-precision language models. The authors examine how execution layouts affect precision upgrade utility per sample and aggregate fallback policy performance. The paper also compares offline estimates with actual fallback execution and investigates whether improving reconstruction fidelity through low-rank correction leads to better task accuracy.
Critical strengths
- Separating per sample rescue and harm cases, aggregate policy gains and total accuracy provides more informative evaluation the reporting prediction instability or accuracy alone.
- The experimental setup includes useful controls, such as fixed sample selection and FP32 arithmetic controls to isolate the effects of different execution configurations.
Critical weaknesses
- Insufficient evidence for the uncertainty based motivation. The paper starts from the premise that uncertainty score can identify error prone inputs in low-precision language models, but does not directly examine this relationship. The reported fallback results do not clearly establish how well uncertainty predicts baseline correctness.
- Limited experimental scope. The main experiments use only two small language models (mainly around 1B scale), a single dataset and NF4 quantization. It is unclear whether the findings generalize to larger models or other quantization settings such as FP4, FP8 or Int8, which are particularly relevant to practical low-precision inference.
- Limited analysis of the main findings. The paper shows that batch size changes affect per sample utility, while positive and negative changes largely cancel out at the aggregate level. However, such cancellation is not specific to batch size changes and may naturally arise when utility changes have mixed signs. The paper does not sufficiently establish whether the observed result is systematic.
Questions for authors
- How well do the uncertainty scores predict the correctness of low-precision models? Could the authors report baseline error rates and rescue/harm rates across different uncertainty levels to better support motivation?
- Do the findings hold for larger language models (at least > 7B) or other quantization settings? Could the authors extend the experiments to more representative low-precision inference scenarios?
- Could the authors provide further analysis of why observed utility changes largely cancel out? Does this pattern hold across different models and quantization settings?