acceptodds
Under review as a conference paper at ICLR 2027

Can LLM Examiners Explain Their Own Judgments?

Abstract

LLMs may replace or complement human decision makers in domains requiring professional discretion. In such settings, providing justifications for decisions may be ethically and legally necessary. LLMs can provide convincing justifications and explanations for their decisions, but it remains unclear whether these accounts correspond to decision policies expressed in their behavior. This issue is particularly important in high-stakes discretionary evaluation, where several considerations must be integrated without a fixed rule determining their relative importance. This study explores a pragmatic method to compare justifications and potential decision policies in the context of LLM-based exam grading using 120 synthetic answers to an authentic law exam. Each answer is scored independently by five LLM sub-judges on ten dimensions of answer quality derived from the exam grading rubric. The averaged sub-judge scores give an emulated objective quality assessment for each of the 120 answers. The examiner LLM, the main protagonist, assigns holistic grades from A to F and provides a justification for the same 120 answers without knowledge of the dimensions or the sub-judges' dimension scores. The examiner grades are then regressed on the sub-judge dimension assessments to examine which dimensions explain the variance in the examiner's grading most strongly. The resulting behavioral policy is then compared with a post-grading self-report and justifications given for individual grades. The study is replicated four times with different models as examiners (Gemma 3 (27B), Ministral 14B, DeepSeek V4 Pro, and GPT-5.6 Sol). The ten quality dimensions accounted for variation in individual grades (). The four examiner models largely agreed on which answers were stronger or weaker (), despite differences in grading severity. Behavioral cue-utilization profiles showed varying correspondence with both post-grading self-reports (–) and local justification profiles (–). Self-reports and local justifications were more consistently aligned with each other (–). The local justifications largely fit the predefined answer dimensions but also identified additional evaluative considerations. Overall, the findings suggest that LLM explanations can contain meaningful information about judgments, even without fully revealing how information is systematically weighted across decisions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, Self-explanation Faithfulness, AI Decision Support, Automated Grading, Interpretability, Professional Discretion, Language Model Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_GXZAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks whether LLM examiners’ explanations of their grades reflect what actually drives their grading. It adapts a method from human judgment research: infer a judge’s implicit priorities from its decisions, then compare them with the judge’s own account. Using synthetic answers to a real law exam, several LLM examiners grade the answers holistically and justify each grade. Their stated priorities are then compared with weights inferred statistically from their grading behavior, using rubric-based quality scores. The authors find that the examiners grade consistently and agree with one another, but their explanations match their behavior only partially. The models tend, for instance, to understate how much the sheer number of cited cases influences their grades.

Critical strengths

A principled, transferable framework for testing LLM explanations: Importing lens-model and policy-capturing methodology gives the field a structured, black-box way to compare what a judge says with what its decisions reveal.

Critical weaknesses
  1. Unvalidated, non-independent measurements. The cue scores come entirely from LLM sub-judges, four of which are the examiner models themselves. No human expert validates these scores, the synthetic answers, or the justification coding. The self-report stage also shows examiners their own justifications before eliciting weights, so the agreement between the two is largely built into the design. These measurements underpin every comparison in the paper.

  2. No baseline or statistical support for the self-insight claims. Each correspondence is a Spearman correlation over only ten dimensions, reported without confidence intervals, yet the paper ranks models by self-insight. More importantly, the self-reports may simply echo the rubric’s stated emphasis (which foregrounds ECHR jurisprudence) rather than reflect each model’s own grading. A simple test with existing data is missing: does each model’s self-report match its own behavioral profile better than other models’ profiles? Without such baselines, the headline claim that explanations are informative cannot be separated from a rubric-prior artifact.

Questions for authors
  1. Does each examiner’s self-report correspond more closely to its own behavioral profile than to the other examiners’ profiles? And how do self-reports from a model that never graded, given only the rubric, compare? If own-model correspondence is no better than these baselines, the results reflect the rubric’s emphasis rather than self-insight.
  2. How do the results change when each examiner’s own sub-judge scores are excluded from the cue measurements? Substantial shifts in R² or in the behavioral profiles would indicate that shared-model artifacts drive the findings.
  3. Were the sub-judge scores, the synthetic answers, or the justification coding validated by human experts, even on a subset? Did the sub-judge scores track the intended quality targets? Without such evidence, the behavioral policies may rest on mismeasured cues.
  4. What are the confidence intervals for the ten-point Spearman correlations? If the intervals overlap substantially, the model-to-model comparisons of self-insight are not supported.
  5. Were self-reports elicited without showing the examiner its own justifications? If not, the agreement between self-reports and justifications may be an artifact of the design rather than evidence of consistent introspection.
2 comments
AuthorsOpenReview ↗

Response to Reviewer GXZA: Critical Weakness

Critical Weakness 1: Unvalidated, non-independent measurements. The cue scores come entirely from LLM sub-judges, four of which are the examiner models themselves. No human expert validates these scores, the synthetic answers, or the justification coding. The self-report stage also shows examiners their own justifications before eliciting weights, so the agreement between the two is largely built into the design. These measurements underpin every comparison in the paper.

Response:

The sub-judge assessments serve as operational measurements of rubric-derived answer qualities, rather than ground-truth evaluations. Importantly, for each examiner, we already excluded its own sub-judge scores when constructing the averaged quality measurements. Thus, the behavioral profiles were estimated using assessments from four other models.

The justification coding follows a predefined codebook with explicit criteria for identifying each dimension. Independent legal-expert validation is not essential to our study, as our primary objective is to investigate correspondence between observed judgment patterns and stated evaluative priorities, rather than establish the correctness of individual examination assessments.

The self-reports were intentionally elicited after the examiners received their previous grades and justifications, allowing them to reflect on their completed evaluations. We therefore do not interpret agreement between self-reports and justifications as independent evidence of self-insight.

Critical Weakness 2: No baseline or statistical support for the self-insight claims. Each correspondence is a Spearman correlation over only ten dimensions, reported without confidence intervals, yet the paper ranks models by self-insight. More importantly, the self-reports may simply echo the rubric’s stated emphasis (which foregrounds ECHR jurisprudence) rather than reflect each model’s own grading. A simple test with existing data is missing: does each model’s self-report match its own behavioral profile better than other models’ profiles? Without such baselines, the headline claim that explanations are informative cannot be separated from a rubric-prior artifact.

Response:

We address these concerns in our responses to Questions 1 and 4, where we present the cross-examiner baseline and confidence interval analysis, respectively. Both analyses have been added to Appendix D.
AuthorsOpenReview ↗

Response to Reviewer GXZA: Questions for Authors

Question 1: Does each examiner’s self-report correspond more closely to its own behavioral profile than to the other examiners’ profiles? And how do self-reports from a model that never graded, given only the rubric, compare? If own-model correspondence is no better than these baselines, the results reflect the rubric’s emphasis rather than self-insight.

Response:

We conducted a cross-examiner comparison of each examiner's self-reported importance with the behaviorally recovered cue utilization policies of all four examiners (Appendix D, Table 16).

Only two examiners showed their highest correspondence with their own behavioral profiles. Mean within-examiner correspondence was 0.534, compared with 0.484 across examiners.

These findings highlight the value of our framework: it allows us to identify when an examiner's stated priorities correspond to its observed grading behavior and when they do not. Our objective is to assess this correspondence, rather than assume that examiners accurately describe their own judgments. We have clarified this distinction in Section 3.3 and included the full comparison in Appendix D, Table 16.

Question 2: How do the results change when each examiner’s own sub-judge scores are excluded from the cue measurements? Substantial shifts in R² or in the behavioral profiles would indicate that shared-model artifacts drive the findings.

Response:

Each examiner's own sub-judge scores were already excluded when calculating its averaged quality measurements. Consequently, the reported behavioral profiles are based on assessments from the four remaining models, avoiding direct overlap between the examiner and its sub-judges.

As a sensitivity check, we repeated the analysis using all five sub-judges. The recovered behavioral profiles remained highly similar ($\rho$ = 0.952–-0.988), with changes in $R^2$ no greater than 0.0072. These results indicate that our behavioral importance estimates are robust to the sub-judge aggregation choice. We have clarified the procedure in Section 2.3 and included the sensitivity results in the Appendix D Table 16.

Question 3: Were the sub-judge scores, the synthetic answers, or the justification coding validated by human experts, even on a subset? Did the sub-judge scores track the intended quality targets? Without such evidence, the behavioral policies may rest on mismeasured cues.

Response:

The ten quality dimensions were derived from the official grading guidelines and assessed using predefined scoring criteria. The synthetic answers were generated with intended quality targets, while justification coding followed a detailed codebook with explicit coding rules.

Our objective was to examine LLM judgment patterns rather than establish objectively correct grades. Although independent human-expert validation was not conducted, the standardized assessment procedure provides a consistent basis for comparing the examiners. For our purpose, the key consideration is whether the measurements capture the intended rubric dimensions, rather than whether they reproduce expert grading judgments.

Question 4: What are the confidence intervals for the ten-point Spearman correlations? If the intervals overlap substantially, the model-to-model comparisons of self-insight are not supported.

Response:

We calculated $95\%$ confidence intervals for all twelve Spearman correlations. The intervals are relatively wide, reflecting the limited number of dimensions ($n=10$). We therefore interpret the correlations descriptively and avoid drawing definitive conclusions about differences in self-insight between examiners. The confidence intervals have been added to the appendix D Table 17.

Question 5: Were self-reports elicited without showing the examiner its own justifications? If not, the agreement between self-reports and justifications may be an artifact of the design rather than evidence of consistent introspection.

Response:

The self-reports were intentionally elicited after the examiners received their previous grades and justifications, allowing them to reflect retrospectively on their completed evaluations. However, our primary focus is the correspondence between reported priorities and observed grading behavior, rather than agreement between the two verbal accounts. We therefore do not interpret self-report–justification correspondence as independent evidence of self-insight and have clarified this distinction in the revised manuscript in Appendix F.
Reviewer_fc8aOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to examine correlation between llm's explanation on its own overall judgment and its judgement on grading rubrics. The authors used synthetic legal examination data for the experiment. Based on human judgement and decision making literature, they attempted to offer a method for mining relation between llm's explanation and rubric scores, and found some correlation between overall judgment and grading rubrics, and between local explanation and self-reported policy on rubrics.

Critical strengths

This paper has the following strengths:

  • Offering a new method for examining the behavior of llm judges, by comparing overall judgment, local explanation, and self-reported policy on rubrics
Critical weaknesses

This paper has the following weaknesses:

  • As the local explanation and self-reported policy could be an outcome of statistical prediction of LLMs, we need some care on treating them as evidence of LLM's reasoning process. The paper does not provide any causal relationship between local explanation and the overall judgment, and that between self-reported policy and the rubric scores. It should be clearly checked whether the explanation and the policy is faithful, by checking they actually affected the computation of overall/rubric scores.
  • This paper discusses their findings only on synthetic answers. As LLM's writing might differ from human and LLM might prefer its own writing, evaluating only on the synthetic data might not provide entire view about the paper's claim. Also, the domain of the data is limited to be generalized to other situations.
Questions for authors

See the weakness points above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fc8a

Critical Weakness 1: As the local explanation and self-reported policy could be an outcome of statistical prediction of LLMs, we need some care on treating them as evidence of LLM's reasoning process. The paper does not provide any causal relationship between local explanation and the overall judgment, and that between self-reported policy and the rubric scores. It should be clearly checked whether the explanation and the policy is faithful, by checking they actually affected the computation of overall/rubric scores.

Response:

Our analysis is not intended to establish a causal relationship between the examiners' explanations and their grading decisions. However, our objective is not to reconstruct the models' internal reasoning processes, but to examine whether their stated evaluative priorities correspond to patterns observed in their grading behavior.

Thus, the recovered behavioral policies represent statistical associations rather than causal mechanisms. Establishing causal faithfulness would require additional interventions on the factors influencing judgments, which is beyond the scope of the present study.

Critical Weakness 2: This paper discusses their findings only on synthetic answers. As LLM's writing might differ from human and LLM might prefer its own writing, evaluating only on the synthetic data might not provide entire view about the paper's claim. Also, the domain of the data is limited to be generalized to other situations.

Response:

We acknowledge that synthetic examination answers may differ from authentic student responses and that our findings are limited to the studied setting.

However, our primary objective was not to evaluate the quality of the examination answers themselves, but rather to investigate how LLM examiners evaluate them and whether their explanations correspond to their observed judgment patterns. The examination served as a controlled setting for studying discretionary judgment, where multiple criteria must be considered without a fixed decision rule. Our framework examines general aspects of judgment behavior and explanatory correspondence and is therefore intended to be applicable across domains.

Furthermore, the answers were generated using Claude Opus 4.8, whereas the four examiner models were Gemma, Ministral, DeepSeek, and GPT-5.6 Sol. Thus, none of the examiners evaluated answers generated by itself, reducing the risk of direct self-preference.
Reviewer_PMWqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether LLM examiners' explanations correspond to their observable grading behavior. Using 120 synthetic law-exam answers, the authors compare regression-based importance profiles of ten answer-quality dimensions with self-reported importance weights and local justifications across four LLM examiners.

Critical strengths
  1. The paper addresses an important question about the reliability of explanations provided by LLM evaluators.

  2. The study includes four examiner models and 50 randomized grading runs per model, providing useful evidence about grading consistency.

Critical weaknesses
  1. The answer-quality dimensions are assessed by LLM sub-judges without independent expert validation. Four sub-judge models also serve as examiners, and the robustness of the inferred importance profiles to alternative assessments remains unclear.

  2. The three compared profiles measure different quantities: regression-based importance, retrospective self-reported weights, and justification frequencies. Their correlations across ten dimensions offer limited evidence about explanation faithfulness, and the uncertainty of these comparisons is not quantified.

  3. The evaluation uses synthetic answers from a single law examination. It remains unclear whether the findings generalize to real student answers or other evaluative tasks.

Questions for authors
  1. How stable are the inferred importance profiles when overlapping sub-judge models are excluded or independent expert assessments are used?

  2. Can the authors quantify uncertainty in the reported profile correlations and clarify what conclusions about explanation reliability these correlations support?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PMWq: Critical Weakness

Critical Weakness 1: The answer-quality dimensions are assessed by LLM sub-judges without independent expert validation. Four sub-judge models also serve as examiners, and the robustness of the inferred importance profiles to alternative assessments remains unclear.

Response:

For each examiner, we have already excluded its own sub-judge assessments when calculating the averaged quality scores. Thus, the behavioral policies were estimated using assessments from four other models, rather than the examiner's own evaluations.
Our objective is to investigate LLM judgment patterns, not the correctness of examination grading. The law examination serves as a controlled setting for studying discretionary judgments, so legal-expert ground truth is not essential to our research question. We have clarified the exclusion procedure in the revised manuscript.

Critical Weakness 2: The three compared profiles measure different quantities: regression-based importance, retrospective self-reported weights, and justification frequencies. Their correlations across ten dimensions offer limited evidence about explanation faithfulness, and the uncertainty of these comparisons is not quantified.

Response:

It is true that the three profiles represent different quantities. As discussed in our response to GyCN, We compare their relative rankings using Spearman correlations rather than treating their numerical values as equivalent. These correlations describe correspondence between predictive importance and reported or mentioned priorities. We have also added $95\%$ confidence intervals to quantify the uncertainty of these comparisons Table 17 Appendix D.

Critical Weakness 3: The evaluation uses synthetic answers from a single law examination. It remains unclear whether the findings generalize to real student answers or other evaluative tasks.

Response:

The primary objective of our study is not to investigate examination grading itself, but to examine whether LLMs' explanations of their judgments correspond to their observed decision patterns.

We selected examination grading as a controlled setting for discretionary judgment, where multiple criteria must be weighed without a fixed decision rule. This provides a suitable setting for evaluating our proposed framework, which is intended to be applicable to other discretion-based decisions, including those in the public sector.

Our framework investigates general constructs of judgment behavior and explanatory correspondence, rather than domain-specific grading accuracy. While the empirical evaluation is limited to one setting, the framework itself is designed to be domain-independent.
AuthorsOpenReview ↗

Response to Reviewer PMWq: Questions

Question 1: How stable are the inferred importance profiles when overlapping sub-judge models are excluded or independent expert assessments are used?

Response:

For each examiner, we already excluded its own sub-judge assessments when calculating the averaged quality scores. Therefore, the recovered behavioral profiles are based entirely on assessments from other models. Furthermore, comparing the profiles with and without the examiner's own sub-judge scores showed only minor differences, suggesting that the results are not substantially affected by this overlap.

Our objective is to investigate LLM judgment patterns rather than establish correct examination grades. Independent legal-expert validation was therefore not central to our research question, although it could provide an additional robustness check.

Question 2: Can the authors quantify uncertainty in the reported profile correlations and clarify what conclusions about explanation reliability these correlations support?

Response:

The reported Spearman correlations measure the correspondence between the relative rankings of the ten dimensions across the three profiles. They do not establish that the explanations faithfully represent the models' internal decision processes. We have calculated $95\%$ confidence intervals (Appendix D Table 17). The intervals are relatively wide, reflecting the limited number of dimensions ($n=10$). We therefore interpret these correlations descriptively rather than drawing definitive conclusions about differences between examiners.
Reviewer_GYcNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLM examiners’ local justifications and retrospective importance reports correspond to the factors associated with their grading behavior. Using 120 synthetic responses to an authentic law examination, the authors obtain ten rubric-derived quality measurements from five LLM sub-judges and collect holistic grades and justifications from four examiner models across 50 randomized grading runs. They use regression and dominance analysis to estimate behavioral importance profiles, which are compared with self-reported dimension weights and dimension mention frequencies in local justifications. The results show substantial explained variance in grades and partial correspondence between the three profiles, leading the authors to suggest that LLM explanations can provide useful information about judgments without fully revealing the underlying decision process.

Critical strengths
  1. The paper addresses a meaningful question with a useful conceptual framing. Whether an evaluator’s explanations reflect its grading behavior matters for understanding and scrutinizing automated assessment. Connecting this question to the judgment-and-decision-making literature on policy self-insight provides a productive perspective, and the distinction between local justifications and retrospective self-reports helps separate different forms of explanation.
Critical weaknesses
  1. The three importance profiles measure different quantities, and the validity of comparing them is not sufficiently established. Dominance shares measure contributions to explained variance, self-reports estimate relative influence, and justification profiles measure mention frequency (Sections 3.2–3.3; Listing 6). Normalizing these quantities to sum to 100 does not make them equivalent. In particular, dominance shares depend on predictor variance and covariance, so discrepancies can arise even when an examiner accurately describes its decision rule. Although the authors acknowledge the associational nature of their analysis, a known-policy validation or a clearer operational account of what agreement should look like is needed before interpreting profile differences as evidence about explanatory accuracy.
  2. The aggregate justification analysis does not establish that explanations provide information beyond the grade itself. Randomly shuffling justifications across answers within an examiner would leave the aggregate mention frequencies, and therefore the corresponding correlations in Table 3, unchanged. The analysis consequently cannot distinguish explanations that are informative about particular judgments from explanations assigned to the wrong answers. Section 4 acknowledges the absence of an individual-answer informativeness test, but the conclusion still suggests that justifications provide information beyond grades. An answer-level evaluation against a grade-only or generic-rubric baseline would directly address this gap.

The retrospective self-reports are conditioned on the same justifications with which they are subsequently compared. The second-stage prompt explicitly supplies the examiner’s previous grades and local justifications before eliciting dimension weights (Section 2.4; Listing 6). The relatively high self-report–justification correspondence may therefore reflect summarization of the supplied explanations. Reporting this consistency is reasonable, but it provides limited additional evidence that either account reflects the grading behavior. A matched condition that withholds the earlier justifications would help determine how much of the observed agreement follows from this conditioning.

Questions for authors

See weakness above

3 comments
AuthorsOpenReview ↗

Response to Reviewer GyCN: Critical Weakness 1

Critical Weakness 1: The three importance profiles measure different quantities, and the validity of comparing them is not sufficiently established. Dominance shares measure contributions to explained variance, self-reports estimate relative influence, and justification profiles measure mention frequency (Sections 3.2–3.3; Listing 6). Normalizing these quantities to sum to 100 does not make them equivalent. In particular, dominance shares depend on predictor variance and covariance, so discrepancies can arise even when an examiner accurately describes its decision rule. Although the authors acknowledge the associational nature of their analysis, a known-policy validation or a clearer operational account of what agreement should look like is needed before interpreting profile differences as evidence about explanatory accuracy.

Response:

The three profiles represent different aspects of judgment: which dimensions predict the assigned grades, which dimensions the examiner retrospectively reports as important, and which dimensions it emphasizes in local justifications.

Our aim is not to establish equality between these quantities, but to investigate their correspondences. Specifically, we consider the profiles to agree when dimensions that contribute more strongly to predicting grades are also relatively more prominent in the examiner's reported priorities or justifications. We therefore compare their relative rankings using Spearman correlations.

The recovered behavioral profile provides a statistical description of cue utilization within the observed dataset, not an estimate of the examiner's true internal decision weights. Accordingly, discrepancies between profiles indicate differences between behavioral predictive importance and stated evaluative priorities, rather than proving that an explanation is unfaithful to the model's internal reasoning.
We have clarified this operational interpretation in the revised manuscript and moved the retrospective self-report analysis to the Appendix F.
AuthorsOpenReview ↗

Response to Reviewer GyCN: Critical Weakness 2

Critical Weakness 2: The aggregate justification analysis does not establish that explanations provide information beyond the grade itself. Randomly shuffling justifications across answers within an examiner would leave the aggregate mention frequencies, and therefore the corresponding correlations in Table 3, unchanged. The analysis consequently cannot distinguish explanations that are informative about particular judgments from explanations assigned to the wrong answers. Section 4 acknowledges the absence of an individual-answer informativeness test, but the conclusion still suggests that justifications provide information beyond grades. An answer-level evaluation against a grade-only or generic-rubric baseline would directly address this gap.

Response:

We thank the reviewer for this insightful comment, which prompted us to examine the informativeness of local justifications more closely and extend our analytical framework beyond mention frequencies. Specifically, we conducted an additional answer-level directional analysis, coding whether evaluations of each quality dimension were positive, negative, or neutral and comparing them with the corresponding sub-judge scores.

The results provide further evidence that local justifications are informative about individual answers. Directional evaluations generally tracked independently assessed answer quality, substantially outperforming both shuffled-justification and mention-only baselines. This directly addresses the reviewer's concern about interchangeability: although randomly reassigning justifications would preserve aggregate mention frequencies, it would disrupt the observed correspondence between the evaluations and the quality of the specific answers. Our findings therefore indicate that local justifications are not interchangeable across answers, but contain meaningful answer-specific information beyond merely identifying relevant grading dimensions.

We believe this additional analysis strengthens our central contribution by providing a more fine-grained assessment of justification informativeness. We have incorporated these findings into the revised Results and Discussion sections.
AuthorsOpenReview ↗

Response to Reviewer GyCN: Critical Weakness 3

Critical Weakness 3: The retrospective self-reports are conditioned on the same justifications with which they are subsequently compared. The second-stage prompt explicitly supplies the examiner’s previous grades and local justifications before eliciting dimension weights (Section 2.4; Listing 6). The relatively high self-report–justification correspondence may therefore reflect summarization of the supplied explanations. Reporting this consistency is reasonable, but it provides limited additional evidence that either account reflects the grading behavior. A matched condition that withholds the earlier justifications would help determine how much of the observed agreement follows from this conditioning

Response:

 The retrospective self-reports were deliberately elicited after the examiners received their previous grades and justifications, allowing them to reflect on their completed evaluations. Our purpose was not to obtain an independent assessment of the answers, but to examine how the examiners retrospectively described their evaluative priorities and how these accounts corresponded to their observed grading behavior. We therefore do not treat the correspondence between self-reports and local justifications as independent evidence of self-insight. The primary comparisons concern the relationship between each verbal account and the behaviorally recovered cue-utilization profile. We have made this distinction more explicit in the manuscript and moved the self report analysis to Appendix F

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response and Summary of Revisions

We thank all reviewers for their careful evaluation of our manuscript and their constructive feedback. We appreciate the opportunity to clarify our methodological choices, address the concerns raised, and strengthen the presentation and interpretation of our findings.

In response to the reviewers' comments, we have made several revisions to the manuscript. In particular, we have:

  • Clarified the interpretation and comparability of the different measures of examiner judgment.
  • Strengthened the validation and interpretation of the sub-judge assessments and recovered behavioral policies.
  • Extended our analysis of local justifications with an answer-level directional analysis, providing further evidence that justifications contain meaningful information about individual grading decisions beyond merely mentioning relevant dimensions.
  • Refined the discussion of our findings and limitations, and moved the retrospective self-report analysis to the Supplementary Materials.

Below, we respond to each comment individually and outline the corresponding clarifications, revisions, and additional analyses where applicable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.