acceptodds
Under review as a conference paper at ICLR 2027

Aligned Textual Scoring Rules

Abstract

Reference-based text scoring systems evaluate a reported text by comparing it with a ground truth text. Existing metrics and language-model judges can be vulnerable to strategic manipulations that increase scores without improving the information conveyed. Conversely, scoring systems designed for strategic robustness may align poorly with scores assigned by human or language-model judges. We develop a framework for learning judge-aligned textual scoring rules within proper-scoring-rule families. Our approach encodes reported and ground truth texts in a shared numerical representation, learns a common linear projection, and fits Brier or Log-Cosh scoring rules to target judge scores. We assume that text representations encode linear properties of beliefs expressed over an underlying semantic space. Under this assumption, truthful reporting maximizes the expected score. When the assumption holds only approximately, we bound the gain from strategic reporting by a multiple of the squared representation error. Experiments on peer grading and question answering evaluate both alignment with target scores and robustness to strategic manipulation. The learned scoring rules improve alignment with judge scores while exhibiting robustness to the evaluated manipulation strategies. These findings support learning within proper-scoring-rule families as an approach to aligning textual evaluation with judge preferences while preserving incentive guarantees under explicit representation assumptions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Proper scoring rules, information elicitation, mechanism design

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_Y5DhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for learning reference-based textual scoring rules that align with human or LLM judge scores while encouraging truthful reporting. It uses frozen language-model representations, learned linear projections, and two families of proper scoring rules. The authors establish approximate-properness bounds under representation errors and evaluate statistical alignment and robustness to predefined textual manipulations

Critical strengths
  1. The paper addresses an interesting problem by connecting proper scoring rules from information elicitation with learned textual evaluation metrics.

  2. The experiments include multiple datasets, manipulation tests, and useful ablations of scoring functions and representation regularization.

Critical weaknesses
  1. Assumption 3.1 requires text embeddings to encode linear properties of the beliefs expressed in natural language. However, pretrained language-model representations may also capture linguistic style, wording, and other nonlinear factors, and it is unclear how closely they satisfy this assumption for real-world reports.

  2. The main Brier and Log-Cosh implementations use different backbone models and representation processing. Their results therefore cannot directly establish the superiority of one scoring family over the other. Moreover, removing the role penalty increases alignment while producing 30/40 significant positive manipulation effects, suggesting that practical robustness depends substantially on representation regularization rather than properness alone.

Questions for authors
  1. Can the authors provide more direct evidence that the learned projected representations satisfy the linear-property assumption on real-world peer-grading or QA examples? In particular, how large are the projected representation errors that determine the approximate-properness guarantees in these settings?
2 comments

We thank the reviewer for the encouraging feedback!

W2 the same experiment pipeline for both scoring rules

We repeat the same experiment pipeline as the current LogCosh for Brier on the peer grading dataset. Brier is manipulable by 0/40 attacks. The alignment results are shown below:

MetricHuman targetHaiku 4.5 targetGPT-5-mini target
Test Pearson0.602 [0.454, 0.723]0.656 [0.577, 0.711]0.640 [0.544, 0.707]
Test Spearman0.551 [0.407, 0.666]0.687 [0.579, 0.755]0.662 [0.544, 0.739]

We will add the full evaluation in a revised version of the paper.

W2 regularization and robustness

Our regularization selects meaningful dimensions in the representation such that the dimensions represent uncertainty for a scoring rule to be proper. Without regularization, scoring rules cannot meaningfully score beliefs and are not guaranteed to be robust. With regularization, log-cosh is manipulable by 0/40 attacks; without regularization, log-cosh is manipulable by 30/40. We will add this comparison to the paper.

Q2 evaluation of the assumption for the full learning pipeline

It will be challenging to design an evaluation over a real review space. Our current test evaluates on a synthetic review space, using the trained log-cosh scoring rule. A real review induces a very high-dimensional semantic space and potentially correlated semantic meanings across dimensions. For such a space, it would be hard to design tests that have linear interpolation representations. We believe the current synthetic test and the manipulation results reveal promising evidence for our approach.

Thanks to the response. I assume that the first "W2" and "Q2" in the response refer to "W1" and "Q1", respectively. Most of my concerns have been addressed, although the validity of Assumption 3.1 in real-world settings remains uncertain. Overall, I will maintain my original score.

Reviewer_1txVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies text scoring rules that can align with target scores from human or LLM judges while resisting strategic manipulation. The authors map reported texts and ground-truth texts into a shared representation space, produce scores through linear projections and proper scoring rules, and jointly learn the projection and scoring parameters under supervision from target scores. The theoretical analysis examines truthful-reporting incentives under a linear representation assumption and derives an approximate-properness bound under representation error. Experiments evaluate both score alignment and robustness under six manipulation strategies. The results show that the proposed method achieves relatively good alignment and avoids significant positive score gains under the tested attack conditions.

Critical strengths

The paper addresses a clearly important problem. Existing text scores based on lexical overlap, semantic similarity, or LLMs are easily affected by factors such as length, phrasing, and expressed confidence, while pursuing game-theoretic robustness in isolation may become disconnected from the target score. The paper formulates this tension as a clear research question.

Combining linear projections with proper scoring rules such as Brier and Log-Cosh is a concise and principled solution. The theoretical analysis goes beyond empirical regularization by examining truthful-reporting incentives and how representation error affects robustness.

Critical weaknesses

The paper's theoretical guarantee relies on the assumption that text representations linearly encode expectations of semantic properties. However, the current approximation test covers only a specific Log-Cosh configuration and constructed texts, and it does not sufficiently establish how well the assumption holds for practical encoders and the full model.  Some key comparisons are not fully matched. Different methods use different encoders, evaluation subsets, preprocessing, or effect-size normalization, and several correlation intervals overlap. I recommend comparing the main robustness baselines under the same encoder, data split, and optimization conditions and reporting paired significance tests.

Questions for authors

Could the authors repeat experiments and report significance?

Could the authors directly quantify the approximation error of Assumption 3.1 for the full model, covering both Brier and the six-layer Log-Cosh ensemble?

1 comment

We thank the reviewer for the encouraging feedback!

Q1 Could the authors repeat experiments and report significance?

We repeat the same experiment pipeline as the current LogCosh for Brier on the peer grading dataset. Brier is manipulable by 0/40 attacks. The alignment results are shown below:

MetricHuman targetHaiku 4.5 targetGPT-5-mini target
Test Pearson0.602 [0.454, 0.723]0.656 [0.577, 0.711]0.640 [0.544, 0.707]
Test Spearman0.551 [0.407, 0.666]0.687 [0.579, 0.755]0.662 [0.544, 0.739]

We will add the full evaluation in a revised version of the paper.

Q2 evaluation of the assumption for the full learning pipeline

It will be challenging to design an evaluation over a real review space. Our current test evaluates on a synthetic review space, using the trained log-cosh scoring rule. A real review induces a very high-dimensional semantic space and potentially correlated semantic meanings across dimensions. For such a space, it would be hard to design tests that have linear interpolation representations. We believe the current synthetic test and the manipulation results reveal promising evidence for our approach.

Reviewer_97MEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the core challenge of rule-based scoring by proposing a framework that learns the scoring rule from numerical representations of reported and ground-truth data. The approach assumes that text representations encode linear uncertainty in a semantic space. The paper claims that the experiments consider both alignment and the strategic robustness of the learned scores. The approach's potential implication is the proper use of scoring rules for retraining.

Critical strengths

A timely problem to address is devising a representative proper scoring rule to overcome the challenges of using LLMs as judges. The paper is well documented, with theory and attempts to establish a mathematical foundation. The reported results on Brier and Log-Cosh metrics provide evidence for the validity of the proposed framework. A key strength is the use of proper scoring rules, particularly the Brier and Log-Cosh families, together with a theoretical analysis that bounds potential gains from strategic reporting under imperfect representations. The experimental evaluation covers multiple datasets and manipulation strategies, and the ablation studies provide useful insights into the contributions of scoring-rule structure, representation regularization, and model selection. The strong performance on simulated peer grading and the lack of statistically significant positive manipulation effects for the proposed Log-Cosh method are promising.

Critical weaknesses

The abstract shall not have citations. Sec. 3.1, on the main assumption, should also report the empirical evaluation results because the current state of theory is insufficient to establish the validity of the assumption on which the entire framework relies. Figure 1 would be clearer with explicit descriptions of each component. The paper is more theory than experimental evaluation; many equations use symbols repeatedly, making them confusing and hard to follow. The paper's main mathematical derivations are largely correct, but the theoretical guarantees are conditional on a strong representation assumption that the cited literature does not establish. The experiments provide promising evidence, but they are not sufficient to establish general strategic robustness. Assumption 3.1 is theoretically coherent but empirically unverified. The referenced literature motivates exploring it; it does not justify treating it as an established property of frozen Llama or Qwen representations. The authors should characterize it as a modeling hypothesis, not an implication of the existing linear representation literature. The authors correctly prove a property of numerical scoring rules. Still, they have not demonstrated that the real text representations satisfy the conditions needed to transfer that property into truthful-reporting guarantees.

Questions for authors
  1. It would be beneficial to provide stronger theoretical or empirical evidence that LLM representations preserve linearity over mixtures of expressed beliefs, as required by Assumption 3.1.
  2. The practical significance of the approximation-error bounds in Theorem 3.2, particularly regarding the larger incentive gaps observed for verbal uncertainty in Table 8, should be clarified.
  3. Further theoretical justification for the use of Wasserstein regularization in Equation (30) is needed, considering that truthful reports and realized outcomes may not have identical marginal distributions.
  4. It is recommended to better isolate the contribution of properness from the effects of role regularization, scoring-function choice, and model capacity.
  5. Evaluation of adaptive score-maximizing attacks and semantic sensitivity is encouraged to determine whether the reported 0/40 significant manipulation effects reflect genuine strategic robustness.
1 comment

We thank the reviewer for their thoughtful feedback.

Q1 Linearity assumption

We agree that the assumption may not be perfect. In fact, our framework does not require the assumption to hold perfectly (Theorem 3.2). Our empirical simulation shows that an imperfect assumption need not cause a large strategic gain to our approach.

Q2 Practical meaning of the gap in Table 8 and theorem 3.2

Theorem 3.2 shows that representation errors do not necessarily translate into equally large incentives for strategic misreporting. Table 8 provides an empirical assessment of the guarantee. The result of verbal probability 0.5 is larger than other uncertainty representations. This suggests that the learned representations may encode verbal expressions of uncertainty less accurately than numerical probabilities. We will add this discussion to the paper.

Q3 theoretical justification for regularizer.

We view the regularizer as selecting dimensions where the beliefs from the texts are correctly specified. The regularizer penalizes the difference in distribution of reported text and ground truth text, which corresponds to report having a misspecified belief over the state space. Heuristically, states with a misspecified report space may not correspond to the semantic spaces that a score should depend on. We will add this justification to the revised version of the paper.

Q4 isolate the contribution of properness from the effects of role regularization, scoring-function choice, and model capacity

Regularization selects meaningful dimensions in the representation such that the dimensions represent uncertainty for a scoring rule to be proper. Without regularization, scoring rules cannot meaningfully score beliefs and are not guaranteed to be robust. With regularization, log-cosh is manipulable by 0/40 attacks; without regularization, log-cosh is manipulable by 30/40. We will add this comparison to the paper.

Scoring function choice The scoring function choice mostly specifies the richness of the hypothesis space and affects alignment results, as shown in our main results in Table 2. The Brier score is more restrictive than log-cosh, which uniformly depends on the distance between the report and the truth, ignoring the value of the report. This could be the reason why the Brier score aligns worse than log-cosh.

Model capacity We will add experiments to compare a weaker and a stronger model in a future revised version.

Q5 Adaptive attack

We thank the reviewer for this suggestion. We list adaptive strategic manipulations as an important future question. Our evaluation follows from the evaluation framework in Xu et al., which covers a representative set of observed strategic attacks/shortcuts in the literature of reward hacking and reward overoptimization, including verbose rewrites, changes in style, positive and negative opinion shifts, hedging, and added confidence. (references can be found around lines 45-49)

Reviewer_tj9aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns reference-based text scores that agree with a target judge while keeping an incentive for truthful reporting. A frozen LLM encodes the reported text and the ground-truth text, a shared learned linear projection maps both to KK coordinates, and a proper scoring rule (a sum of Brier or Log-Cosh scores) compares them. The projection and the rule are fit by squared error to judge scores; Log-Cosh training also penalizes scoring directions aligned with the average difference between reference and report representations. If a text's representation is linear in the belief it expresses (Assumption 3.1), truthful reporting maximizes the expected score; if this holds only approximately, the gain from misreporting is bounded by the squared projected representation error (Theorem 3.2, Corollaries 3.3–3.4). Experiments on real and simulated peer grading, MedAESQA and LFQA-E, with human, synthetic and LLM-judge targets, measure agreement (Pearson/Spearman) and robustness to six fixed rewrites from Xu et al. (2026). Log-Cosh reaches target-macro .629/.606, and both Log-Cosh and Brier show 0/40 significant score increases, compared with 16/28 for raw LLM judges and 23/40 for linear regression on the Log-Cosh representations.

Critical strengths

A promising, practical way to train a grader that agrees with human or LLM ratings and is harder to game. The method fits a proper scoring rule on frozen LLM representations from judge-scored pairs, so it can be trained to imitate a given human or LLM judge. Under the six fixed manipulations it shows 0/40 significant score increases, against 16/28 for the raw LLM judges it imitates and 23/40 for linear regression on the same representations (Table 2b, Table 15), while agreeing better than ROUGE-L, BLEU and BERTScore on every target (target-macro Pearson/Spearman .629/.606). Pairing agreement with resistance to common edits addresses a real weakness of LLM judges used for grading and training.

Critical weaknesses
  1. The robustness claim rests on an assumption checked only in a small synthetic test, and on preset edits. The guarantee (Theorem 3.2, Corollaries 3.3–3.4) depends on the frozen language model correctly translating an answer's meaning and uncertainty into numbers (Assumption 3.1). If that translation is wrong, even a well-designed proper scoring rule can reward a manipulated answer over the truthful one; the bound on that gain scales with the squared projected representation error. The paper checks this requirement only in Section E.5: one four-coordinate Log-Cosh rule trained on Simulated Peer Grading, tested on 100 constructed sentence families per uncertainty type, which by the authors' account covers neither Brier nor the six-layer ensemble actually evaluated (lines 1396–1398). So it remains unclear whether the guarantee applies to the full system on real tasks. The robustness tests also use six preset edits, each generated once (lines 376–380, 1197), such as adding filler or making an answer sound more confident. They do not test someone who repeatedly changes an answer, checks its score, and keeps whichever version scores highest. The authors list this as future work (lines 471–476), yet the abstract states that the learned scores "are robust to strategic manipulations". Partly fixable during the discussion: a small search-based attack (Q2) and the E.5 check for the deployed ensemble on real texts (Q3) are feasible; otherwise the claims should be limited to the tested edits.

  2. The added value of the embeddings over a calibrated judge is untested. A useful missing baseline is a calibrated judge: take an existing judge's scores or sampled votes and learn how to adjust them to better match independent ratings, without hidden-state embeddings. Raw judges appear only in the manipulation tests (Table 2b), and judge-to-human agreement is not reported (line 438). The comparison also needs independent targets: six of the eight targets are the same judges' temperature-0 ratings (lines 976–977), so simply copying the judge would count as perfect agreement. Only the Peer Grading human score (150 test items in 24 reference groups) and the synthetic Simulated score are independent of the judges. Until the original judge, a calibrated judge and the proposed method are compared on the same training and test data, for both agreement with independent ratings and resistance to manipulation, the added value of the embedding-based approach remains unclear. Fixable during the discussion: the judge scores, human scores and rewrites already exist, so this needs little new compute (Q1).

Questions for authors
  1. Calibrated-judge comparison on independent targets. [Requested analysis] On the Peer Grading human target, and on Simulated Peer Grading if you run the judges there, please compare the raw judge, a judge calibrated on the training split (from its scores or several sampled votes, without hidden states), and your method, using the same splits. Please report Pearson/Spearman and the six manipulation tests for all three. If your method matches the calibrated judge's agreement while keeping 0 significant increases where the calibrated judge has some, I would raise my score; if the calibrated judge matches your method on both, I would lower it.

  2. A search-based attack. [Requested analysis] For a subset of test reports (for example, Peer Grading), let an LLM propose several rewrites per round with instructions not to add information, keep the highest-scoring one, and repeat for a few rounds. Please report the resulting score gain for your method, the raw judge and linear regression, and check with an independent judge that the information content is unchanged. Small gains for your method relative to these baselines would raise my score; large gains would mean the robustness claim should be limited to the preset edits.

  3. The key assumption for the deployed system on real texts. [Requested analysis] Please report the Section E.5 error for the six-layer Log-Cosh ensemble (and for Brier) on real texts, for example hedged combinations of two real ground-truth reviews from Peer Grading, and express it relative to the spread of clean scores. Small errors would support applying the guarantee to the full system and raise my score; large errors would mean the robustness is empirical rather than guaranteed, and the claims should say so.

Score criteria: I would raise my rating to weak accept if Q1 shows a clear advantage for the embedding-based method on independent targets (at least in manipulation resistance, without a large loss in agreement), and if Q2 and Q3 either support the robustness claims for the full system or the claims are limited to what is tested. I would lower my rating if a calibrated judge matches the method in both agreement and manipulation resistance.

2 comments

We thank the reviewer for the suggestions.

Q1. Calibrated judge comparison.

We calibrate the LLM judges on both peer grading dataset and simulated peer grading dataset. The judges remain manipulable. We will include the comparison in a revised version of the paper.

On both datasets, we compare Log-Cosh with raw and calibrated Haiku 4.5 and GPT-5-mini on the same held-out reports and edits. We fit group-balanced isotonic maps on the original training splits (600 human-rated and 2,400 simulated reports).

Human-rated Peer Grading (150 test reports). Log-Cosh had 0/6 significant score increases, compared with 4/6 for calibrated Haiku and 6/6 for calibrated GPT-5-mini. Calibrated GPT-5-mini had higher agreement, with a Pearson advantage of .152 over Log-Cosh (paired 95% CI [.014, .268]).

MetricLog-CoshHaiku rawHaiku cal.GPT rawGPT cal.
Pearson.618.599.669.726.770
Spearman.548.663.663.751.751
Significant gains0/65/64/66/66/6
Hedged opinion shift−.10.20.15.14.08
Meaningless elongation−.22.02.01.15.17
Negative opinion shift−.03.41.49.40.48
Positive opinion shift−.20.13.10.14.10
Rephrasing−.10.16.13.16.12
Strong opinion shift−.07.17.16.12.10

Simulated Peer Grading (300 test reports). Log-Cosh achieved the highest agreement with the independent rubric scores (.933 Pearson, .929 Spearman) and had 0/6 significant score increases. It exceeded the strongest calibrated judge, GPT-5-mini, by .174 Pearson (paired 95% CI [.132, .221]).

MetricLog-CoshHaiku rawHaiku cal.GPT rawGPT cal.
Pearson.933.577.583.757.759
Spearman.929.562.562.749.749
Significant gains0/60/60/62/62/6
Hedged opinion shift−.07.06.05.01−.01
Meaningless elongation.07.03.04.10.09
Negative opinion shift−.17.05.03.32.30
Positive opinion shift−.04.06.05.08.07
Rephrasing−.05−.02−.03−.05−.06
Strong opinion shift−.09−.09−.10−.16−.17

Q2: adaptive attack

We thank the reviewer for this suggestion. We list adaptive strategic manipulations as an important future question. Our evaluation follows from the evaluation framework in Xu et al., which covers a representative set of observed strategic attacks/shortcuts in the literature of reward hacking and reward overoptimization, including verbose rewrites, changes in style, positive and negative opinion shifts, hedging, and added confidence. (references can be found around lines 45-49)

Q3: evaluation of assumption on the full pipeline.

It will be challenging to design an evaluation over a real review space. Our current test evaluates on a synthetic review space, using the trained log-cosh scoring rule. A real review induces a very high-dimensional semantic space and potentially correlated semantic meanings across dimensions. For such a space, it would be hard to design tests that have linear interpolation representations. We believe the current synthetic test and the manipulation results reveal promising evidence for our approach.

Thank you for the calibrated-judge comparison. It answers part of Q1: calibration alone does not fix manipulability. On the human-rated Peer Grading data the calibrated judges still show 4/6 and 6/6 significant score increases, against 0/6 for Log-Cosh, although the judges agree better with the human score (.770/.751 for calibrated GPT-5-mini against .618/.548).

One question remains open. The comparison swaps a calibrated judge for your whole method, which changes three things at once: the hidden-state embeddings, the proper scoring rule, and the role penalty. Your own ablations show that the embeddings alone are not enough (23/40 with linear regression, 30/40 without the role penalty). What has not been tested is the other direction: the same Log-Cosh rule and an equivalent penalty on a representation that is not a hidden-state embedding, for example an LLM's answers to a fixed set of questions about each text. Without that, we know the combination works, but not whether the embedding itself is needed. Could you run this on Peer Grading and Simulated Peer Grading, or, if that is not feasible, state this limit clearly in the paper/response?

Q2 and Q3 were not addressed with new evidence. I do not see either as decisive, but the paper/response should address that the robustness evidence covers the six preset edits, and that the Section E.5 check covers one four-coordinate rule rather than the evaluated ensemble.

Reviewer_FhPPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for learning reference-based textual scoring rules that aim to achieve both statistical alignment with human or LLM judgments and robustness to strategic manipulation. The method applies learned linear projections to frozen language model representations and optimizes scoring functions within proper-scoring-rule families, including quadratic (Brier) and Log-Cosh scores. Under a linear-property representation assumption, the resulting scoring rules incentivize truthful reporting. The paper further derives approximate-properness guarantees under representation error and evaluates the proposed methods on peer grading and question-answering datasets.

The problem is well motivated, and the combination of learned text representations with proper scoring rules is potentially useful. The empirical results are encouraging, particularly the reported robustness of the Log-Cosh scoring rule. However, I have several concerns regarding the novelty of the approach, the validity of its central representation assumption, and the strength of the experimental evidence.

Critical strengths

1.Well-motivated problem. The paper addresses an important limitation of existing text evaluation metrics: statistical agreement with human judgments does not necessarily imply robustness to strategic manipulation. Learning scoring rules that balance these two objectives is a meaningful direction.

2.Principled formulation. The use of proper scoring rules provides a clear theoretical foundation. The approximate-properness analysis is useful for understanding how representation errors affect reporting incentives.

3.Encouraging empirical results. The experiments cover multiple evaluation tasks and include comparisons with traditional metrics, LLM judges, and strategically robust baselines. The additional ablations and scoring-family controls are valuable.

Critical weaknesses

1. The novelty relative to existing work is not sufficiently established.

The central idea of optimizing proper scoring rules to align with human or LLM judgments has already been explored in Aligned Textual Scoring Rules (Lu et al., 2025, arXiv:2507.06221). More generally, eliciting linear properties through proper scoring rules and constructing such rules using convex potentials are well-established ideas.

The main technical differences in the present paper appear to be the use of learned projections of LLM hidden representations, the particular Log-Cosh parameterization, and the analysis of approximate properness under representation error. These are potentially useful extensions, but the paper does not sufficiently establish their incremental contributions relative to the earlier approaches.

A more explicit comparison with the 2025 work, ideally including a direct empirical baseline under the same evaluation protocol, would help clarify the novelty and practical benefits of the proposed framework.

2. The theoretical guarantees rely on a strong and insufficiently validated representation assumption.

Assumption 3.1 requires text representations to encode linear properties of expressed beliefs over semantic outcomes. In particular, a text expressing a probabilistic mixture of semantic outcomes is assumed to have a representation corresponding to the same linear mixture of their representations.

This is a strong assumption for frozen LLM hidden states, which may also encode stylistic, syntactic, or other nonsemantic information. While the paper provides approximate-properness bounds when this assumption is violated, these bounds depend on the representation error itself and therefore do not independently establish robustness for realistic text distributions.

The empirical validation in Appendix E.5 is relatively limited and concerns a specific four-coordinate Log-Cosh scoring rule rather than the full ensemble. Moreover, verbal uncertainty at mixture probability 0.5 exhibits a noticeably larger incentive gap than the corresponding numerical uncertainty condition.

I would like to see a more systematic evaluation of the linear-property assumption and the resulting incentive gaps across different text distributions, representations, and the final scoring model.

3. The experimental comparisons do not fully isolate the proposed method's contributions.

The comparison with TVD-FVar-Statement relies on previously reported results obtained on different evaluation subsets. Furthermore, the Brier and Log-Cosh pipelines differ in backbone representations, preprocessing, regularization, model selection, and some QA evaluation rubrics. These differences complicate the interpretation of performance gains.

The ablations also reveal that role regularization plays a crucial role in robustness. In Table 9, removing the role penalty increases the number of significantly positive manipulation effects from 0/40 to 30/40, despite retaining the proper-scoring-rule structure. In addition, the matched quadratic scoring rule exhibits 6/40 positive effects, suggesting that the specific scoring parameterization matters substantially.

These observations are interesting, but they also indicate that the empirical robustness cannot be attributed to properness alone. A more controlled experimental comparison would help disentangle the effects of the scoring-rule family, representation learning, and regularization.

4. The strategic robustness evaluation is limited to predefined manipulations.

The main robustness claim is based on the absence of statistically significant positive average score changes under a fixed set of manipulation strategies. While encouraging, 0/40 significant positive effects does not establish robustness against adaptive or optimization-based attacks.

In particular, a strategic reporter with access to the scoring function could potentially search for profitable deviations beyond the predefined transformations. This setting seems especially relevant given the paper's game-theoretic motivation.

I would encourage the authors to evaluate adaptive attacks that directly optimize the proposed score, while checking whether the manipulated reports preserve or degrade the underlying information quality.

Additionally, most alignment targets are LLM-generated judgments rather than independent human annotations. This limits the evidence that the learned scores align with human evaluation criteria more broadly.

Questions for authors

1. What are the main conceptual and technical differences between this work and the earlier Aligned Textual Scoring Rules (2025)? Can the authors provide a direct comparison under a common experimental setup?

2. How well does Assumption 3.1 hold for the learned representations in the full scoring pipeline? Can the authors provide empirical distributions of projected representation errors and corresponding incentive gaps?

3. How robust are the proposed scoring rules to adaptive attacks that explicitly optimize the score? To what extent do the robustness improvements come from the Log-Cosh scoring family versus role regularization?

4. Since the Log-Cosh model contains a ground-truth-only term, could the authors report within-reference-group alignment and an ablation of this term to rule out improvements driven by reference-specific score differences?

1 comment

We thank the reviewer for their thoughtful feedback. We respond to the questions below.

Q1: difference from prior work.

First, the prior work is restricted to the structure of the particular peer grading dataset, while our method applies to other datasets in the experiments section. They fit assignment-specific proper scores on discrete rubrics. They convert text into numerical reports by treating an uncertain answer as the empirical average over the ground truth of a particular homework assignment. This use of the empirical average exploits the structure of the peer grading dataset, where the same homework assignment has multiple iid submissions. Their approach does not apply to general datasets such as in our paper.

Second, the method in prior work does not generalize to the test split in a dataset. As mentioned above, their discrete rubrics are task-specific. On a new task, the method requires obtaining a new set of rubrics and fitting the scoring rule on the new task. Thus, prior work did not test their method using train/test splits.

For the comparison on the same peer grading dataset, prior work achieves MSE / Pearson / Spearman were .0400 / .402 / .421 (note that these are performances treating the full dataset as training set), which is worse than .0252 / .618 / .548 for our Log-Cosh (tested on actual test split).

Q2: Linearity assumption for the full pipeline.

It will be challenging to design an evaluation over a real review space. Our current test evaluates on a synthetic review space, using the trained log-cosh scoring rule. A real review induces a very high-dimensional semantic space and potentially correlated semantic meanings across dimensions. For such a space, it would be hard to design tests that have linear interpolation representations. We believe the current synthetic test and the manipulation results reveal promising evidence for our approach.

Q3: robustness to adaptive attacks

We thank the reviewer for this suggestion. We list adaptive strategic manipulations as an important future question. Our evaluation follows from the evaluation framework in Xu et al., which covers a representative set of observed strategic attacks/shortcuts in the literature of reward hacking and reward overoptimization, including verbose rewrites, changes in style, positive and negative opinion shifts, hedging, and added confidence. (references can be found around lines 45-49)

Q4: ground truth only term in log cosh.

We note that the ground-truth-only term is part of the proper scoring rule, which does not affect the properness. This term is not critical to our proposed method or the evaluation pipeline.

We tested our method without the term. Jointly fitting the term provides a modest improvement in predictive performance. Pearson / Spearman are .618 / .548 for the original hypothesis space, .537 / .481 after subtracting the term at inference, and .583 / .486 after refitting without the term. Jointly fitting the term improves within-reference Pearson by .066 over that refit (.664 vs. .598, paired 95% CI [.017, .118]).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.