- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether LLM judges can evaluate one aspect of text quality without being distracted by others. The authors propose CorrGap to measure this dependence and DimCheck to reduce it by removing irrelevant evidence from evaluation rationales before rescoring. Their experiments suggest that the problem is common and that DimCheck helps, with little change in scoring accuracy.
Critical strengths
- The paper highlights an important limitation of LLM judges: good agreement with human scores does not necessarily mean that each dimension is evaluated independently
- CorrGap controls for the target-score distribution when forming groups, and the authors validate it through synthetic perturbations and human annotation analyses
- DimCheck reduces CorrGap while largely preserving scoring correlation. Ablations and prompt variations help support the approach
- The study examines multiple judging frameworks across ten LLMs and four tasks, providing substantial evidence that the issue is widespread
Critical weaknesses
- The human evaluation checks whether DimCheck identifies rationales containing irrelevant evidence, but not whether it edits them correctly. It remains unclear how often useful target-dimension evidence is removed, particularly when dimensions overlap.
- The computational overhead is substantial relative to the gains. The smaller editor is a useful mitigation, but an end-to-end cost comparison is still needed to judge the trade-off.
- Applicability beyond fixed evaluation rubrics is not yet clear. The experiments use four to six predefined dimensions, whereas more complex tasks may require many task-specific criteria with overlapping evidence.
Questions for authors
- Could you directly evaluate the deleted passages, checking both whether they are irrelevant and whether useful evidence is preserved?
- How does the cost performance trade-off change as the number of dimensions increases?
- Can the smaller editor trained on SummEval transfer to another task or an unseen rubric without retraining?
3 comments
- Can the smaller editor trained on SummEval transfer to another task or an unseen rubric without retraining?
In our current setting, the smaller editor trained on SummEval cannot be directly transferred to other tasks with different evaluation dimensions, since its training is specific to the dimensions used in SummEval. However, we would like to highlight that adapting the smaller editor to new tasks is relatively straightforward, as its training does not require human-annotated ground-truth scores. Instead, training examples can be generated using a stronger LLM, making it possible to adapt the smaller editor to new evaluation rubrics without additional human scoring annotations.
Thank you for the detailed response. The additional human evaluation and cost analysis address most of my concerns. The discussion of broader applicability and scalability is also helpful. I will maintain my current score.
We conduct a human evaluation to check whether the evidence removed by DimCheck is more related to the non-target dimension than the target dimension. Specifically, we randomly sample pairs of COTs before and after each evidence removal step of DimCheck. For these randomly sampled pairs, we extract evidence that is removed from the original COT. We then instruct human annotators to select the dimension that the removed evidence is more related to.
Following the original human evaluation, we randomly sample 10 pairs of COTs from each of the SummEval and Topical dataset and each of three LLMs used by DimCheck, resulting in 60 samples in total. Each pair is annotated by three annotators recruited from Prolific whose first language is English and is in either US, UK, or Canada. We report the result of pairwise comparison in the following table.
These results suggest that DimCheck generally removes evidence more relevant to non-target dimensions, supporting the effectiveness of its evidence-editing strategy.
We analyze the computation cost as the number of tokens on the SummEval dataset with Llama3.3-70b. The results of different methods are shown in the following table. For Analyze-Rate+DimCheck w/trained, we additionally report the number of tokens of fine-tuned Llama3.1-8b for editing COTs.
From the table, we can find that Self-Consistency and DimCheck have almost the same number of tokens. Self-reflection generally requires much more tokens due to the need for multi-turn debate between agents. The inference cost of DimCheck can be substantially reduced by using a smaller, fine-tuned model for CoT editing. When employing a fine-tuned Llama3.1-8B editor, total token usage decreases from approximately 3.2E+07 to 1.4E+07. More importantly, 7.2E+06 of these tokens are processed by the 8B model, further reducing the computational burden.
We believe the underlying idea of CorrGap can be extended to tasks with task-specific evaluation criteria under two scenarios depending on the number of responses available for each task.
(1) Sufficient responses per task. When enough responses are available to estimate correlations, CorrGap can be directly extended by computing correlation differences at the task level. This allows CorrGap to accommodate different evaluation criteria across tasks.
(2) Limited responses per task. When task-level correlations cannot be reliably estimated, an alternative is to examine differences in LLM judge accuracy across groups of responses. Specifically, we can divide responses into two groups based on the variance of ground-truth scores across evaluation criteria, following the intuition behind CorrGap. We then compare the judge's evaluation accuracy between these groups to investigate potential inter-dimension dependence. We acknowledge that the second approach requires further investigation, particularly to control for confounding factors such as task difficulty and differences in evaluation criteria. We leave this extension for future work.
DimCheck removes evidence related to one non-target dimension at a time. Since the evaluated benchmarks contain only 4–6 dimensions, the computational cost remains manageable. However, as the number of dimensions increases, the inference cost grows approximately linearly. To improve efficiency for tasks with more dimensions, a potential solution is to group multiple non-target dimensions (e.g., 2–3) into a single editing step, reducing the number of editing calls. We will add this as a discussion in the paper.