acceptodds
Under review as a conference paper at ICLR 2027

ChiEngMixBench: When “Model Preference” Depends on How You Score It

Abstract

Technical concepts can often be expressed either with English terms or their Chinese equivalents. A common way to ask which form a language model favors is to compare the likelihoods the model assigns to the two forms. Yet such “preference” is not directly observed from the model; it is induced by the scoring rule used to measure it. Whether we score the whole sentence or the tokens overlapping the changed text, and whether we sum or average token-level scores, can change the resulting judgment. We introduce ChiEngMixBench, a controlled testbed for studying this issue through paired Chinese–English terminology substitutions, and compare several natural likelihood-based definitions of preference. We find an easily overlooked pattern: scoring definitions can agree on the majority direction while disagreeing substantially about the winners for individual pairs. Thus, agreement on the majority direction alone does not establish which individual items or subgroups behave consistently across scoring definitions; robustness must be checked at the level of the claim.

Primary area:
datasets and benchmarks
Keywords:
likelihood-based evaluation, model preference, measurement robustness, evaluation methodology, multilingual evaluation, Chinese-English terminology, code-mixing, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bcYiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ChiEngMixBench, a collection of 1,167 closely matched sentence pairs containing Chinese or English realizations of technical terminology. It compares four likelihood-based scoring definitions, combining whole-sentence versus target-span scoring with summed versus mean token log-probabilities. Across four models, aggregate preference directions remain relatively stable, while individual preference decisions can differ substantially. For Qwen2.5-7B, the largest disagreement reaches 19.83%.

Critical strengths

The controlled comparison isolates two concrete scoring choices while using the same underlying token probabilities. This makes the measurement issue clear and the results straightforward to interpret. The item-level analysis adds useful information beyond aggregate preference rates. For Qwen2.5-7B, the English-preferred sets under whole-sentence sum and target-span mean have a Jaccard similarity of only 0.083, despite agreeing on the majority preference direction.

Critical weaknesses

The empirical analysis mainly establishes that scoring definitions disagree, with limited explanation of the observed flips. A clearer account of the roles of token counts, target-span scores, and surrounding-context scores would strengthen the paper’s substantive contribution. The described release omits the source sentences and term-normalization records. Although metadata and numerical outputs support inspection of the reported comparisons, the procedure for obtaining or reconstructing the sentence pairs is unclear. This limits the benchmark’s usefulness for subsequent model evaluation.

Questions for authors

Using the existing outputs, could you explain representative preference flips by separating the effects of token count, target-span likelihood, and scores outside the target span? How can researchers obtain or reconstruct the underlying sentence pairs? Please specify the available access procedure or reconstruction materials.

1 comment

Thank you for the detailed response. The token-level examples clarify how token counts, aggregation, and contextual contributions lead to preference flips, while the boundary sensitivity analysis shows that item-level disagreements persist under the revised span definition. The clarified access to the frozen sentence pairs also addresses my concern about the benchmark’s availability for further evaluation. These additions substantially address my original concerns and strengthen the paper’s empirical support. I am therefore raising my rating to Weak Accept.

Reviewer_s4p2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ChiEngMixBench a controlled benchmark of 1167 Chinese technical sentences paired with minimal terminology substitutions English term vs. Chinese counterpart to study how likelihood-based “preference” depends on the scoring rule. Crossing two natural choices scope whole sentence vs. changed span and aggregation (cumulative sum vs. mean per token) the authors show that all four definitions often agree on the aggregate direction (Chinese-majority), yet differ substantially on which specific items are exceptions, with pairwise disagreement up to 20% and very low Jaccard overlap of English-favored sets.

Critical strengths
  • The paper crisply isolates two ubiquitous yet under-specified choices in likelihood-based evaluation scope and aggregation and studies their interactions in a controlled, practically salient environment (Chinese-English terminology choice).
  • The difference between agreement on the direction of the majority versus consistency on item-level winners is interesting and has implications for how we interpret paired likelihood studies.
  • The formalization (Δ scores, disagreement rates, Jaccard overlaps) provides simple but informative diagnostics that are easy to adopt elsewhere.
Critical weaknesses
  • The analysis largely remains descriptive it does not probe causal drivers of disagreement (e.g., token-length effects, tokenizer segmentations, suffix interactions) beyond reporting rates.
  • The span definition uses a mask of the full-sentence tokenization, which might combine local and contextual effects; other span constructions or marginalization over tokenizations are mentioned but not evaluated in the main results.
  • Larger models are evaluated using quantified GGUF assets. Quantification can affect the probabilities in ways that could impact the close call pairwise judgments.
  • No statistical uncertainty is reported (e.g., CIs via bootstrapping) for aggregate rates or disagreement making it hard to assess the stability of the measured differences.
  • There is no relationship exists between likelihood-based preferences and actual generation behavior or human judgments.
  • The study does not test debiasing or other scoring baselines that reduce length/tokenization artifacts, which would sharpen the diagnosis of what aspects drive disagreement.
Questions for authors
  • Can you report bootstrap confidence intervals for aggregate preference rates and disagreement/Jaccard metrics, and assess stability under random subsampling of items?
  • The larger-model results rely on quantized GGUF assets; did you check whether full-precision runs even on a subset materially change disagreement rates or item-level overlaps?
  • Did you examine correspondence with generation behavior (e.g., prompting the model to continue the sentence and measuring the realized terminology form)? It could then be checked to see which scoring rules could be more likely to predict actual usage.
Reviewer_pCBUOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a measurement question: when researchers use likelihood scores to infer whether a model prefers Chinese or English terminology, does the scoring method affect the conclusion? The authors construct 1,167 sentence pairs and compare four methods: scoring either the full sentence or only the replaced term span, and either summing or averaging token scores. All four methods favor Chinese terms in aggregate, but they disagree substantially about which individual sentences favor English. The authors argue that robustness should be assessed at the same level of granularity as the claim.

Critical strengths
  1. A controlled paired design isolates the effect of scoring choices. The same terminology pairs are evaluated under all four scoring definitions, enabling direct comparison between aggregate trends and item-level agreement.
  2. The paper distinguishes aggregate conclusions from item-level conclusions. Across several models, the overall direction is consistent while individual judgments differ. This supports the authors’ recommendation to test robustness at the same granularity as the claim.
Critical weaknesses
  1. There is no external criterion for determining whether item-level disagreements matter in practice. The paper measures how likelihood-based preference judgments vary across scoring definitions, but does not validate them against human preferences, model choices during generation, or other downstream behavior. The experiments show that scoring choices change the results, but not which result better reflects actual language choices.
  2. Independent verification of the dataset is limited. The authors do not provide the source sentences or original texts in the review materials, and note that a full rerun requires separately available sentence and model files. In addition, the semantic equivalence and naturalness of the terminology substitutions are not independently validated by human judgments. This limits reproducibility and auditing of the paired examples.
Questions for authors

see weaknesses

1 comment
Reviewer_pCBUOpenReview ↗

Response to the authors

Thanks for the authors’ response. I’ll keep my rating but lower my confidence in it, good luck

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.