- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on a widely adopted implicit assumption in the field of large language models: lower cross‑entropy loss implies better downstream‑task performance. It takes multiple‑choice question answering (MCQA) as its primary test scenario and conducts experiments across six model families: OPT, Cerebras‑GPT, DeepSeek, Qwen, Llama, and OpenELM. The experiments are divided into a supervised setting (where cross‑entropy loss is computed using ground‑truth correct answers) and an unsupervised setting (where loss is calculated solely from question context without ground‑truth labels).
Critical strengths
It evaluates both supervised cross‑entropy loss computed from ground‑truth answers and unsupervised loss derived solely from question context under label‑free conditions, which closely approximates real‑world scenarios for unannotated evaluation.
The related‑work section offers comprehensive literature coverage. It systematically surveys three categories of works: scaling laws, unsupervised model‑performance estimation, and LLM evaluation. It clearly positions the paper’s contribution: identifying the failure of loss as a proxy metric in cross‑model settings.
Critical weaknesses
Real‑world LLMs extensively employ open‑ended generation tasks. However, the paper does not validate its conclusions on free‑form generation or reasoning tasks, leaving it unclear whether the findings generalize to more general‑purpose scenarios.
It adopts the average negative log‑likelihood (NLL) over the full sequence of each option and selects the option with minimal loss as the prediction. This approach is not fully equivalent to the standard few‑shot inference paradigm of LLMs that directly output answers, creating a gap with real‑world inference behavior.
Although an upper‑bound property is proven at the single‑sample level, fine‑grained sample‑level analysis is missing: it does not investigate which samples exhibit “high dispersion yet incorrect predictions”, and qualitative analysis of failure cases is absent.
Questions for authors
The paper observes a strong loss‑performance correlation within each model family, whereas this correlation breaks down across different families. If we perform loss normalization for models from diverse families (e.g., normalization based on training‑set loss), would it restore cross‑model predictive power?
This work is entirely built upon MCQA tasks. Would a similar cross‑model failure pattern also arise between autoregressive cross‑entropy loss and generation quality for open‑ended generation tasks? What kinds of tasks are most susceptible to such failure?
The paper states that loss dispersion serves as an error upper bound at the single‑sample level, yet this property fails at the dataset level. Can sample‑level dispersion therefore be used for estimating sample difficulty, even if it cannot be employed for ranking different models?