acceptodds
Under review as a conference paper at ICLR 2027

Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Text

Abstract

LLM-as-a-judge methods are widely used for evaluating the quality of generated open-ended text. Such evaluations are generally multi-dimensional, since the error patterns in texts can be different for different dimensions. Therefore, reliable LLM judges should evaluate each target dimension independently. To quantify the extent to which LLM judges depend on non-target dimensions when evaluating a target dimension, i.e., inter-dimension dependence, we propose CorrGap. To measure this, CorrGap uses the difference in correlations between LLM-predicted scores and ground truth scores across different groups of texts. Using CorrGap, we show that inter-dimension dependence is pervasive across LLM judges in open-ended text evaluation tasks. To mitigate inter-dimension dependence, we propose DimCheck, a method that iteratively removes unrelated evidence from COTs generated by LLM judges in a step-wise way. We show that DimCheck mitigates inter-dimension dependence and outperforms strong baselines across three LLMs and four tasks. We also show that smaller trained LLMs can approximate larger LLMs in DimCheck, with much lower inference costs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Dimensional Evaluation, LLM judge, Open-Ended Text Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9hc2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether LLM judges can evaluate one aspect of text quality without being distracted by others. The authors propose CorrGap to measure this dependence and DimCheck to reduce it by removing irrelevant evidence from evaluation rationales before rescoring. Their experiments suggest that the problem is common and that DimCheck helps, with little change in scoring accuracy.

Critical strengths
  1. The paper highlights an important limitation of LLM judges: good agreement with human scores does not necessarily mean that each dimension is evaluated independently
  2. CorrGap controls for the target-score distribution when forming groups, and the authors validate it through synthetic perturbations and human annotation analyses
  3. DimCheck reduces CorrGap while largely preserving scoring correlation. Ablations and prompt variations help support the approach
  4. The study examines multiple judging frameworks across ten LLMs and four tasks, providing substantial evidence that the issue is widespread
Critical weaknesses
  1. The human evaluation checks whether DimCheck identifies rationales containing irrelevant evidence, but not whether it edits them correctly. It remains unclear how often useful target-dimension evidence is removed, particularly when dimensions overlap.
  2. The computational overhead is substantial relative to the gains. The smaller editor is a useful mitigation, but an end-to-end cost comparison is still needed to judge the trade-off.
  3. Applicability beyond fixed evaluation rubrics is not yet clear. The experiments use four to six predefined dimensions, whereas more complex tasks may require many task-specific criteria with overlapping evidence.
Questions for authors
  1. Could you directly evaluate the deleted passages, checking both whether they are irrelevant and whether useful evidence is preserved?
  2. How does the cost performance trade-off change as the number of dimensions increases?
  3. Can the smaller editor trained on SummEval transfer to another task or an unseen rubric without retraining?
3 comments
  1. The human evaluation checks whether DimCheck identifies rationales containing irrelevant evidence, but not whether it edits them correctly. It remains unclear how often useful target-dimension evidence is removed, particularly when dimensions overlap.

We conduct a human evaluation to check whether the evidence removed by DimCheck is more related to the non-target dimension than the target dimension. Specifically, we randomly sample pairs of COTs before and after each evidence removal step of DimCheck. For these randomly sampled pairs, we extract evidence that is removed from the original COT. We then instruct human annotators to select the dimension that the removed evidence is more related to.

Following the original human evaluation, we randomly sample 10 pairs of COTs from each of the SummEval and Topical dataset and each of three LLMs used by DimCheck, resulting in 60 samples in total. Each pair is annotated by three annotators recruited from Prolific whose first language is English and is in either US, UK, or Canada. We report the result of pairwise comparison in the following table.

DatasetModelNon-target DimensionTarget DimensionTie
SummEvalLlama3.3721
SummEvalGemma3811
SummEvalQwen3532
TopicalChatLlama3.3721
TopicalChatGemma3523
TopicalChatQwen3910

These results suggest that DimCheck generally removes evidence more relevant to non-target dimensions, supporting the effectiveness of its evidence-editing strategy.

  1. An end-to-end cost comparison is still needed to judge the trade-off.

We analyze the computation cost as the number of tokens on the SummEval dataset with Llama3.3-70b. The results of different methods are shown in the following table. For Analyze-Rate+DimCheck w/trained, we additionally report the number of tokens of fine-tuned Llama3.1-8b for editing COTs.

Method#Input#Output#Total
Analyze-Rate5.6E+066.9E+056.2E+06
Self-Consistency2.8E+073.4E+063.1E+07
CheckEval2.3E+077.2E+052.4E+07
Self-Reflection5.4E+074.3E+065.8E+07
Analyze-Rate + DimCheck2.6E+076.0E+063.2E+07
Analyze-Rate + DimCheck (trained)1.3E+077.6E+051.4E+07
↳ Edit COT (Llama-8B)7.0E+061.8E+057.2E+06

From the table, we can find that Self-Consistency and DimCheck have almost the same number of tokens. Self-reflection generally requires much more tokens due to the need for multi-turn debate between agents. The inference cost of DimCheck can be substantially reduced by using a smaller, fine-tuned model for CoT editing. When employing a fine-tuned Llama3.1-8B editor, total token usage decreases from approximately 3.2E+07 to 1.4E+07. More importantly, 7.2E+06 of these tokens are processed by the 8B model, further reducing the computational burden.

  1. Applicability beyond fixed evaluation rubrics is not yet clear. The experiments use four to six predefined dimensions, whereas more complex tasks may require many task-specific criteria with overlapping evidence.

We believe the underlying idea of CorrGap can be extended to tasks with task-specific evaluation criteria under two scenarios depending on the number of responses available for each task.

(1) Sufficient responses per task. When enough responses are available to estimate correlations, CorrGap can be directly extended by computing correlation differences at the task level. This allows CorrGap to accommodate different evaluation criteria across tasks.

(2) Limited responses per task. When task-level correlations cannot be reliably estimated, an alternative is to examine differences in LLM judge accuracy across groups of responses. Specifically, we can divide responses into two groups based on the variance of ground-truth scores across evaluation criteria, following the intuition behind CorrGap. We then compare the judge's evaluation accuracy between these groups to investigate potential inter-dimension dependence. We acknowledge that the second approach requires further investigation, particularly to control for confounding factors such as task difficulty and differences in evaluation criteria. We leave this extension for future work.

  1. How does the cost performance trade-off change as the number of dimensions increases?

DimCheck removes evidence related to one non-target dimension at a time. Since the evaluated benchmarks contain only 4–6 dimensions, the computational cost remains manageable. However, as the number of dimensions increases, the inference cost grows approximately linearly. To improve efficiency for tasks with more dimensions, a potential solution is to group multiple non-target dimensions (e.g., 2–3) into a single editing step, reducing the number of editing calls. We will add this as a discussion in the paper.

  1. Can the smaller editor trained on SummEval transfer to another task or an unseen rubric without retraining?

In our current setting, the smaller editor trained on SummEval cannot be directly transferred to other tasks with different evaluation dimensions, since its training is specific to the dimensions used in SummEval. However, we would like to highlight that adapting the smaller editor to new tasks is relatively straightforward, as its training does not require human-annotated ground-truth scores. Instead, training examples can be generated using a stronger LLM, making it possible to adapt the smaller editor to new evaluation rubrics without additional human scoring annotations.

Thank you for the detailed response. The additional human evaluation and cost analysis address most of my concerns. The discussion of broader applicability and scalability is also helpful. I will maintain my current score.

Reviewer_aSnuOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether LLM judges score each dimension independently in multi-dimensional evaluation of open-ended text. It proposes CorrGap, a metric comparing judge–human correlation between low- and high-variance groups (stratified by ground-truth score), showing inter-dimension dependence is pervasive across 10 LLMs, 3 frameworks, and 4 datasets. It proposes DimCheck, which iteratively removes non-target-dimension evidence from judge CoTs before re-scoring, plus a distilled small-model variant for efficiency.

Critical strengths

CorrGap's design is thoughtful: percentile-based grouping with ground-truth distribution control (Sec. 3) addresses an obvious confound, and the "w/o control" ablation (Tab. 3, 4) shows this control matters.

Metric validation is unusually thorough: FActScore as a low-dependence control (Tab. 3), synthetic γ-graded perturbations (Tab. 4), and a leave-one-annotator-out human comparison (Tab. 5) together show CorrGap captures genuine dependence, not noise.

Critical weaknesses

Effect sizes for DimCheck are modest: in Tab. 7, τ is essentially unchanged vs. Analyze-Rate and average CG drops only ~1.5–2 points (e.g., SummEval 30.0→27.9); the "overall" gain (61.6→62.8) rests on an ad-hoc, equally weighted average of τ and 1−CG. The abstract's "outperforms strong baselines" should be tempered or the overall score justified.

Circularity: DimCheck asks the same (possibly biased) LLM to remove its own non-target evidence, and Tab. 8b shows self-editing works best — also consistent with self-preference rather than debiasing. A cross-family editor analysis would help.

Questions for authors

How sensitive is the "overall" ranking in Tab. 6/7 to the equal weighting of τ and 1−CG? Would conclusions change under other weightings?

In Tab. 8b, why does self-editing outperform cross-model editing? Did you test editors from a different model family to rule out self-preference?

Given Hanna's low inter-annotator agreement (Tab. 5), how reliable are CorrGap and DimCheck results on Hanna specifically?

Does DimCheck ever remove genuinely relevant evidence (false-positive edits), and how are score changes distributed among edited COTs?

CorrGap requires multi-dimensional human annotations; how should practitioners apply it when such annotations are scarce?

3 comments
  1. Effect sizes for DimCheck are modest and rests on an ad-hoc, equally weighted average of τ and 1−CG. How sensitive is the "overall" ranking in Tab. 6/7 to the equal weighting of τ and 1−CG? Would conclusions change under other weightings?

Paired bootstrap resampling confirms that the improvements are statistically significant. Importantly, DimCheck consistently reduces inter-dimension dependence while largely preserving the correlation with human judgments, demonstrating its effectiveness in improving the independence of multi-dimensional evaluation.

We also test the sensitivity of the overall score with respect to the weight (α\alpha) assigned to the correlation for tables 6 and 7. For table 6, when considering the average of overlapping 7 models across three frameworks, the analyze-rate still shows the best performance when α\alpha is greater than 0.26. For table 7, DimCheck shows the best overall performance on average for all three LLMs when α\alpha is smaller than 0.73. Thus, its ranking remains unchanged across a wide range of weighting choices.

Furthermore, beyond the composite overall score, we report the correlations on the high-variance group (the group exhibiting lower correlations) in Table 10. DimCheck also achieves the best performance under this complementary evaluation.

  1. Circularity: DimCheck asks the same (possibly biased) LLM to remove its own non-target evidence, and Tab. 8b shows self-editing works best — also consistent with self-preference rather than debiasing. A cross-family editor analysis would help.

Self-editing is not the best for all LLMs. As in table 8, when the judge LLM is Gemma3-27b, Llama3.3-70b is the best edit LLM. This suggests that editing effectiveness depends on the choice of editor rather than whether the judge and editor are the same LLM.

To check the potential self-biasing of editing LLM, we compare the average proportions of COTs edited by edit LLM across four benchmarks for different combinations of edit LLM and judge LLM.

Judge LLM \ Edit LLMLlama3.3-70BGemma3-27BQwen3-32B
Llama3.3-70B56.183.369.1
Gemma3-27B60.984.074.6
Qwen3-32B49.979.764.6

Although different editors exhibit different editing rates, the ranking of judge models in terms of the proportion of edited CoTs is identical across all three editors (Qwen3-32B < Llama3.3-70B < Gemma3-27B). This consistent ranking across model families suggests that the editing decisions are not driven solely by the relationship between the judge and editor models.

  1. Given Hanna's low inter-annotator agreement (Tab. 5), how reliable are CorrGap and DimCheck results on Hanna specifically?

We believe our findings on Hanna remain informative for the two reasons although the inter-annotator agreement is low. First, CorrGap measures differences in inter-dimensional dependence rather than absolute agreement with human annotations. As in Table 5, the agreement levels are comparable between these groups. This reduces the concern that the observed dependence is driven primarily by systematic differences in annotation reliability between the two groups. Second, human evaluations exhibit substantially lower CorrGap than LLM judges on Hanna (6.9 vs. 10.4 on average), providing evidence that the dependence observed in LLM judges is not merely inherited from the human annotations.

  1. CorrGap requires multi-dimensional human annotations; how should practitioners apply it when such annotations are scarce?

CorrGap is primarily designed to quantify inter-dimension dependence on benchmarks with multi-dimensional human annotations, avoiding the need for carefully designed dimension-specific perturbations. We acknowledge that its direct application is limited when such annotations are unavailable. However, DimCheck does not require human annotations and can still be applied to identify and remove potentially irrelevant evidence from an LLM judge's reasoning traces. The evidence removal process can indirectly identify the existence of inter-dimension dependence.

  1. Does DimCheck ever remove genuinely relevant evidence (false-positive edits)?

We conduct a human evaluation to check whether the evidence removed by DimCheck is more related to the non-target dimension than the target dimension. Specifically, we randomly sample pairs of COTs before and after each evidence removal step of DimCheck. For these randomly sampled pairs, we extract evidence that is removed from the original COT. We then instruct human annotators to select the dimension that the removed evidence is more related to.

Following the original human evaluation, we randomly sample 10 pairs of COTs from each of the SummEval and Topical dataset and each of three LLMs used by DimCheck, resulting in 60 samples in total. Each pair is annotated by three annotators recruited from Prolific whose first language is English and is in either US, UK, or Canada. We report the result of pairwise comparison in the following table.

DatasetModelNon-target DimensionTarget DimensionTie
SummEvalLlama3.3721
SummEvalGemma3811
SummEvalQwen3532
TopicalChatLlama3.3721
TopicalChatGemma3523
TopicalChatQwen3910

These results suggest that DimCheck generally removes evidence more relevant to non-target dimensions, supporting the effectiveness of its evidence-editing strategy.

  1. How are score changes distributed among edited COTs?

We already report the proportion of samples whose COTs are edited and the proportion of samples whose scores therefore change in Table 13.

Thank you again for your thoughtful and constructive feedback. We have posted our rebuttal addressing your concerns and have included additional analyses and experiments. We would greatly appreciate it if you could take a moment to review our responses when you have a chance.

We would also be happy to provide further clarification or address any remaining concerns as we are approaching the end of the rebuttal stage.

Reviewer_2E1uOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a relatively comprehensive study of inter-dimension dependence in LLM-as-a-Judge for multi-dimensional evaluation of open-ended text. The authors propose CorrGap, a metric that measures the difference in evaluation correlations between groups of texts with different levels of disagreement across quality dimensions. They further introduce DimCheck, a method that iteratively removes evidence related to non-target dimensions from the judge's generated reasoning traces through dimension-specific prompting, followed by re-evaluation.

The paper includes extensive experiments across four benchmarks, multiple LLM judges, and different evaluation frameworks. It also provides synthetic validation, human evaluation, baseline comparisons, ablation studies, and experiments with smaller models to reduce inference costs. Overall, the authors have made a considerable effort to investigate the problem from both measurement and mitigation perspectives, providing a relatively comprehensive empirical study.

However, the paper has several weaknesses. The novelty of CorrGap is not sufficiently distinguished from prior work, its validity as a measure of inter-dimension dependence requires further justification, and DimCheck appears to rely mainly on repeated prompting, raising concerns about its methodological novelty. In addition, the improvements brought by DimCheck seem relatively modest compared with its additional inference cost.

Critical strengths
  1. An important problem in multi-dimensional evaluation.

The paper addresses a relevant limitation of LLM-based evaluation: a judge may achieve high agreement with human ratings while still being influenced by irrelevant evaluation dimensions. Distinguishing evaluation accuracy from dimension independence is an interesting and potentially useful research direction.

  1. Evaluation across multiple models and tasks.

The authors evaluate CorrGap across different LLM families, evaluation frameworks, and text generation tasks. The paper also includes synthetic experiments, human annotation analysis, comparisons with existing baselines, and ablation studies. These experiments provide some empirical support for the proposed approaches.

Critical weaknesses
  1. Limited novelty. The problem of evaluation-criterion confusion has been studied in prior work, particularly Hu et al. (ACL 2024). The paper does not sufficiently establish the novelty and advantages of CorrGap over existing approaches.

  2. Unclear validity of CorrGap. A difference in correlations between two sample groups does not necessarily indicate inter-dimension dependence. A more direct controlled evaluation is needed to validate the metric.

  3. Limited methodological novelty of DimCheck. DimCheck mainly relies on repeatedly prompting an LLM to remove irrelevant evidence from its reasoning traces. The necessity and advantages of this iterative prompting strategy over simpler single-step editing are not sufficiently demonstrated.

  4. Limited improvements versus additional cost. DimCheck provides relatively modest improvements over the original judge while introducing substantial additional inference costs. Its practical benefits therefore remain unclear.

  5. Reproducibility and presentation. Some implementation details appear inconsistent with the paper, particularly the evaluation criteria used in the OpinSummEval editing scripts. The paper also suffers from unclear explanations and repetitive writing.

  6. Confusing AI-use statement. The statement repeatedly denies AI use for disclosure-required tasks, yet acknowledges AI-assisted manuscript editing. This apparent contradiction makes the disclosure difficult to assess.

Questions for authors
  1. How does CorrGap differ fundamentally from existing approaches to measuring evaluation-criterion confusion, particularly Hu et al. (ACL 2024)?

  2. How can the authors demonstrate that CorrGap specifically measures inter-dimension dependence rather than differences in sample difficulty?

  3. Why is iterative dimension-specific prompting necessary? What advantages does it offer over a single-step editing prompt?

  4. Do the improvements of DimCheck remain meaningful under matched inference budgets?

  5. Can the authors clarify the implementation inconsistencies and provide evidence that the reported results are reproducible?

5 comments
  1. The problem of evaluation-criterion confusion has been studied in prior work, particularly Hu et al. (ACL 2024). The paper does not sufficiently establish the novelty and advantages of CorrGap over existing approaches.

Thank you for pointing this paper out and we will cite this paper. However, compared with CorrGap, the metric proposed by Hu et al. measures a kind of related phenomenon but from a completely different angle. It measures whether LLM judge evaluation on a certain dimension can remain stable/change facing the perturbation that does not change/change the quality. Instead, CorrGap measures whether LLM judge evaluation on a certain dimension is affected by quality on other dimensions.

Besides, for such metrics based on perturbation, it is difficult to design perturbations that significantly affect the text’s quality on the target dimension while keeping other dimensions unchanged as we discussed in Lines 97-101. This issue is more severe for dimensions that are more subjective yet not explored by Hu et al., such as empathy in the Hanna dataset.

Compared to perturbation methods, CorrGap does not require human effort to design specific perturbation for each of the evaluated dimensions. Besides, CorrGap also proposes to control the distribution of ground-truth scores when comparing the correlation between two groups. This control is important because differences in distributions can themselves affect measured correlations.

  1. A difference in correlations between two sample groups does not necessarily indicate inter-dimension dependence. A more direct controlled evaluation is needed to validate the metric. How can the authors demonstrate that CorrGap specifically measures inter-dimension dependence rather than differences in sample difficulty?

To address this concern, we already provide several complementary analyses that validate CorrGap and examine potential confounding factors in our paper.

First, CorrGap controls for differences in the target-score distribution. Specifically, CorrGap constructs the two groups while controlling the distribution of human annotations on the target dimension (Lines 132–140). This reduces the possibility that the observed correlation gap is driven simply by differences in the target-score distribution.

Second, we examine whether the two groups differ in annotation difficulty. We use human inter-annotator agreement as a proxy for annotation difficulty, based on the intuition that more ambiguous or difficult samples tend to produce lower agreement. As shown in Table 5 (left), the two groups exhibit comparable inter-annotator agreement. This suggests that substantial differences in human annotation difficulty are unlikely to be the primary explanation for CorrGap.

Third, human annotators exhibit substantially smaller CorrGap than LLM judges under the same grouping procedure (Table 5, right), suggesting that the observed dependence is not solely attributable to inherent differences between the groups.

Finally, we directly examine the validity of CorrGap through the controlled experiments in Sec. 5. Our synthetic perturbation experiments demonstrate that CorrGap is sensitive to deliberately introduced inter-dimension dependence. In addition, our FactScore experiments provide complementary evidence by showing relatively low CorrGap for an evaluation method designed to assess a specific dimension. These results further support CorrGap's ability to capture inter-dimension dependence.

  1. Limited methodological novelty of DimCheck. DimCheck mainly relies on repeatedly prompting an LLM to remove irrelevant evidence from its reasoning traces. The necessity and advantages of this iterative prompting strategy over simpler single-step editing are not sufficiently demonstrated.

The main contribution of DimCheck is not iterative editing itself, but a dimension-aware reasoning refinement framework specifically designed to mitigate inter-dimension dependence in multi-dimensional LLM evaluation. Unlike generic reasoning editing, DimCheck explicitly identifies evidence associated with individual non-target evaluation dimensions and removes such evidence while preserving reasoning relevant to the target dimension.

In ablation study (Sec. 6.5), we show DimCheck outperforms its ablated version that removes evidence in a single step in Table 8 (a). We believe the main reason is that putting all non-target dimensions in a single step can put too much burden on the LLM.

  1. Limited improvements versus additional cost. DimCheck provides relatively modest improvements over the original judge while introducing substantial additional inference costs. Its practical benefits therefore remain unclear. Do the improvements of DimCheck remain meaningful under matched inference budgets?

Regarding the performance improvement. We perform paired bootstrap resampling to test whether the improvement over baselines are statistically significant (Lines 411-412) using paired bootstrap resampling. The results show that the improvement of DimCheck is statistically significant.

Additionally, we analyze the inference budgets as the number of tokens on the SummEval dataset with Llama3.3-70b. The results of different methods are shown in the following table. For Analyze-Rate+DimCheck w/trained, we additionally report the number of tokens of fine-tuned Llama3.1-8b for editing COTs.

Method#Input#Output#Total
Analyze-Rate5.6E+066.9E+056.2E+06
Self-Consistency2.8E+073.4E+063.1E+07
CheckEval2.3E+077.2E+052.4E+07
Self-Reflection5.4E+074.3E+065.8E+07
Analyze-Rate + DimCheck2.6E+076.0E+063.2E+07
Analyze-Rate + DimCheck (trained)1.3E+077.6E+051.4E+07
↳ Edit COT (Llama-8B)7.0E+061.8E+057.2E+06

From the table, we can find that Self-Consistency and DimCheck have almost the same number of tokens. Self-reflection generally requires much more tokens due to the need for multi-turn debate between agents. The inference cost of DimCheck can be substantially reduced by using a smaller, fine-tuned model for CoT editing. When employing a fine-tuned Llama3.1-8B editor, total token usage decreases from approximately 3.2E+07 to 1.4E+07. More importantly, 7.2E+06 of these tokens are processed by 8B model, further reducing the computational burden.

  1. Some implementation details appear inconsistent with the paper, particularly the evaluation criteria used in the OpinSummEval editing scripts.

The evaluation criteria used in the uploaded script of OpinSummEval is a mistake. The script we used to experiment already fixed that issue. We will also update the paper and code with correct details.

  1. Confusing AI-use statement.

In the AI-use statement, we follow the definition of required AI disclosure at https://iclr.cc/Conferences/2027/AIPolicyForAuthors. According to the policy, our usage of AI does not belong to the required AI disclosure but belongs to the recommended usage of AI.

Thank you again for your thoughtful and constructive feedback. We have posted our rebuttal addressing your concerns and have included additional analyses and clarification regarding novelty. We would greatly appreciate it if you could take a moment to review our responses when you have a chance.

We would also be happy to provide further clarification or address any remaining concerns as we are approaching the end of the rebuttal stage.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Summary

We thank the reviewers for their thoughtful feedback. In response to the discussion, we revised the manuscript to clarify what CorrGap measures and to provide further evidence on performance and computation costs of DimCheck. All additional experiments reported during the discussion are included in the revised manuscript.

(1) Difference between CorrGap and perturbation-based methods: We clarified that the two methodes answer different questions about inter-dimension dependence. Perturbation-based methods ask whether a judge’s score for a target dimension changes when the text is deliberately modified along a non-target dimension. CorrGap asks whether the judge’s agreement with human ratings on the target dimension differs between naturally occurring examples where human ratings across dimensions are more similar or more different.

(2) Human Evaluation on Evidence Removed by DimCheck: In a human evaluation on SummEval and Topical-Chat, annotators more often judged the removed evidence to be more relevant to the non-target dimension being edited for than to the target dimension (Sec. 6.4).

(3) Whether iterative editing helps at comparable inference budgets: We added a baseline that independently repeats single-step editing as many times as DimCheck performs editing steps and aggregates the resulting scores by majority vote in Table 9 and 24.

(4) Computational Cost Analysis of DimCheck: We added a detailed inference cost analysis in Table 26. Reviewer 9hc2 noted that the additional human evaluation and cost analysis addressed most of their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.