acceptodds
Withdrawn from ICLR 2027

DECAF: Disentangling Capability and Translation Artifacts for Multilingual LLM Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While most large language models (LLMs) support a variety of languages beyond English, accurately evaluating their language-specific capabilities remains a challenge. Multilingual benchmarks are central to this evaluation; however, because they are typically translated from an English baseline using machine translation, the translation process often introduces item-level artifacts, such as translation errors and culturally sensitive content. These translation artifacts obscure the accurate estimation of a model's language-specific capabilities. To address this challenge, this paper introduces the Multilingual Differential Item Functioning Item Response Theory (MDIF-IRT) model. The MDIF-IRT framework advances multilingual LLM evaluation by disentangling language effects into distinct model capability shifts and item difficulty shifts. An application to the MMLU-ProX dataset, evaluating 25 models across 29 languages, demonstrates the utility of the proposed method. The MDIF-IRT model estimates language-specific latent capabilities while accounting for item-specific cross-lingual difficulty variation to improve estimation of capability and item parameters. Furthermore, the framework provides a tool for identifying benchmark items with significant cross-lingual translation artifacts and improves prediction of unobserved item responses compared to empirical accuracy and conventional IRT models.

Primary area:
datasets and benchmarks
Keywords:
AI evaluation, multilingual benchmark, item response theory, differential item functioning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_B72xOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DECAF (Disentangling Capability and Translation Artifacts), centered around a Multilingual Differential Item Functioning Item Response Theory (MDIF-IRT) model. The authors aim to separate true model capability variations across languages from artificial difficulty shifts introduced by machine translation artifacts in benchmarks like MMLU-ProX.

Critical strengths

The motivation is interesting and practically significant.

Critical weaknesses
  1. Validity Dilemma of the Diagnostic Pipeline: The authors position their framework as an essential diagnostic tool for benchmark refinement, using GPT-5.5 to validate that 71.8% of the 748 flagged items suffer from translation errors. This reveals a fatal methodological dilemma:

    • If LLMs are already reliable judges of translation errors in low-resource languages: Then the entire complex pipeline (benchmarking 25 LLMs across 29 languages → collecting response matrices → running GVEM → flagging DIF items) is completely redundant. Benchmark creators could simply deploy an LLM-as-a-judge quality-estimation pipeline directly on the translated text before running evaluations, saving enormous compute.

    • If LLMs are NOT reliable judges of these languages (which is widely established for low-resource languages like Wolof): Then using GPT-5.5 as the sole ground truth to claim that "71.8% of flagged items are translation errors" is ungrounded, circular, and scientifically invalid.

  2. The paper’s titular claim is "Disentangling Capability and Translation Artifacts." However, to ensure identifiability, the authors impose a zero-centered prior on item difficulty shifts: (lines 208–212). This introduces a critical theoretical breakdown: If a machine-translation pipeline produces systematically poor or degraded translations across an entire low-resource language (e.g., Wolof or Yoruba), the mean translation error is non-zero. Under the zero-mean constraint, the model mathematically forces the entirety of this average translation degradation into the model capability parameter. Consequently, the framework cannot disentangle true model capability from benchmark-wide translation failure; it only isolates item-level variance around the language's average degradation. Calling this "disentanglement of translation artifacts" is an over-claim.

  3. The proposed model is only compared exclusively against strawman baselines: Empirical Accuracy, Pooled Rasch, and Independent Rasch

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

We thank the reviewer for raising these critical questions that helped us significantly improve our manuscript.

We thank the reviewer for raising these critical questions that helped us significantly improve our manuscript. Please find our point-by-point responses below.

  1. Validity Dilemma of the Diagnostic Process:

We would like to clarify that the automated qualitative review process was intended as a post-hoc illustration of the types of translation artifacts present in the flagged items, rather than as a validation of our method. In addition, as we discuss in the updated manuscript, our model-based approach provides benefits that an LLM-based judge cannot capture, because the LLM judge focuses only on the linguistic aspects of the items. To make this clear, we have added the following discussion to the revised manuscript in the Experiment section.

“However, note that this automated qualitative review serves as a post-hoc illustration of the types of translation artifacts that may be present in the flagged items, rather than a validation of the MDIF-IRT framework. The results demonstrate that relying on an LLM judge to review translated benchmark items may fail in two ways. First, among poorly translated items, it cannot distinguish those that cause a substantial difficulty shift and those with a negligible shift that does not affect LLM evaluation. Second, it fails to detect items that exhibit significant difficulty shifts unexplained by literal translation artifacts. In contrast, the MDIF-IRT framework relies on empirical response patterns to disentangle true language-specific capabilities from artificial difficulty shifts.”

  1. Limited Baseline Comparison:

While a comparison to a more advanced method (e.g., the multilingual IRT model) was considered, the original estimation code for that method is not currently publicly available. We contacted the authors multiple times to request access, but were informed that the code cannot be shared due to restricted disclosure.

Although a direct empirical comparison is therefore infeasible, our MDIF-IRT model offers advantages over this existing approach, as discussed in our manuscript:

  • Bypasses Normality Assumption: First, the latent capabilities are assumed to follow a normal distribution in the multilingual IRT model. However, the empirical distributions of LLM capabilities may deviate from this assumption (Zhou et al., 2026; Jiang et al., 2026), potentially affecting the reliability of capability estimation. On the other hand, our method treats the item parameters as random, and capability parameters as fixed, being more robust to the violation of normality assumptions.

  • Interpretation of Difficulty Shift: Second, although these multilingual benchmarks are initially developed in English and subsequently translated, the existing approach models item difficulty as a composite of a baseline difficulty shared across all languages and a language-specific deviation. Such a formulation prevents the language-specific deviations from being directly interpreted as artifacts of the translation process.

  • Psychometric Framework DIF/Alignment: Third, our approach implements an alignment (Asparouhov & Muthén, 2014) and DIF framework from psychometrics (Holland et al., 1993) to capture effects of translation on both item characteristics and model capabilities. While the item-by-language difficulty shift in the multilingual IRT is restricted by a strict sum-to-zero constraint for identifiability, our method uses a Gaussian prior and a post-hoc alignment. This allows for a non-zero average difficulty shift across a language. Furthermore, by adopting the psychometric notion of DIF, we interpret shifts in item difficulties and model capabilities across languages as a function of the translation process, rather than unexplained residual variance in model performance.

We now describe these advantages in the Introduction of the updated manuscript. To ensure reproducibility and accessibility of our method, we will make our complete code publicly available.

AuthorsOpenReview ↗

Identifiability and Disentangling Capability and Translation Artifacts:

We sincerely thank the reviewer for this insightful point. We agree that the zero-centered prior used for identifiability of the model may fail to capture systematic translation degradation in low-resource languages. To mitigate this issue, we implemented a multigroup alignment method (Asparouhov & Muthen, 2014). We have updated the MDIF-IRT Model section to detail this alignment implementation:

The GVEM estimation sets a zero-mean Gaussian prior for the item difficulty shift parameters γ1j\gamma_{1j} for identifiability, which acts as a shrinkage penalty. However, translation artifacts can systematically inflate the difficulty of a subset of items rather than affecting them symmetrically. For low-resource languages (e.g., Wolof) or those with high typological distance from the English reference (e.g., Japanese), these artifacts pull the true mean of the difficulty shifts away from zero. To address this issue and allow for a non-zero average difficulty shift without altering the overall model fit, we implement the multigroup alignment method \citep{asparouhov2014multiple}. The alignment method rescales the latent parameters under the assumption that the majority of item difficulties remain invariant relative to the English reference. Specifically, for each language ll, we estimate an alignment constant clc_l that minimizes the objective function:

F=∑j=1J(γ^1jl−cl)2+ϵ,F = \sum_{j = 1}^J \sqrt{\sqrt{(\hat{\gamma}_{1jl} - c_l)^2 + \epsilon}},

where γ^1jl\hat{\gamma}_{1jl} is the initial difficulty shift estimate (i.e., the posterior mean) and ϵ=10−4\epsilon = 10^{-4} is a small constant. The item difficulty and model capability shift parameters are updated as follows:

γ^1jl∗=γ^1jl−cl\hat\gamma_{1jl}^* = \hat\gamma_{1jl} - c_l

β^1il∗=β^1il−cl.\hat\beta_{1il}^* = \hat\beta_{1il} - c_l. As a result, this alignment minimizes the deviation of language-specific item difficulty from the English reference. In doing so, it extracts the systematic benchmark-wide translation degradation (clc_l) and ensures language-specific capability estimates are disentangled from translation artifacts, while preserving the predictive probabilities pijlp_{ijl}.

  • Empirical Verification via Simulation: To verify that alignment mitigates the identifiability shrinkage without distorting downstream DIF screening, we conducted a simulation study under a sparsity condition (Appendix C). The simulation result confirms that the zero-mean prior constraint forces the pre-aligned model to absorb the translation degradation into the LLM capability scores (Bias = -0.293). The alignment mitigates this bias (Bias = -0.149) and substantially reduces RMSE (0.335 vs 0.220). In addition, when comparing the downstream cross-lingual DIF screening, the pre- and post-aligned DIF statistic (D_j) show strong rank correlation (Spearman 0.859) and a 97.4% agreement rate in item flags. This confirms that the translation artifact detection is consistent, while the alignment mitigates the bias in capability estimates induced by the zero-mean identifiability constraint.

Table A3: Parameter recovery and DIF consistency pre- and post-alignment

MetricPre-AlignmentPost-Alignment
Parameter Recovery
Bias (γ1\gamma_1)-0.301-0.157
RMSE (γ1\gamma_1)0.4690.392
Bias (β1\beta_1)-0.293-0.149
RMSE (β1\beta_1)0.3350.220
DIF Screening Consistency
DIF Stat Rank Correlation (Spearman ρ\rho)---0.859
DIF Flag Agreement Rate---97.4%
  • Empirical Benchmark Application: Consistent with these simulation findings, our analysis of the benchmark data (Figure A1, Appendix E) showed that before alignment, the zero-mean prior compressed the estimated benchmark-wide difficulty shifts to a maximum of 0.029. After applying the alignment method, the language-specific difficulty shift expands up to 0.069 for low-resource (e.g., Wolof) or typologically distant languages (e.g., Japanese).

Table: Maximum language-specific difficulty shift before and after alignment

LanguagePre-alignmentPost-alignment
AF0.0100.021
AR0.0130.041
BN0.0160.032
CS0.0070.013
DE0.0040.001
ES0.002-0.005
FR0.003-0.003
HI0.0160.024
HU0.0090.015
ID0.0040.000
IT0.003-0.002
JA0.0120.041
KO0.0130.036
MR0.0150.020
NE0.0170.013
PT0.003-0.003
RU0.0060.014
SR0.0090.019
SW0.0200.024
TE0.0160.017
TH0.0170.044
UK0.0090.018
UR0.0150.019
VI0.0080.016
WO0.0290.069
YO0.0260.015
ZH0.0080.032
ZU0.0220.021
Reviewer_obNPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focus on multilingual LLM evaluation, where benchmark scores can reflect both model capability differences and translation-related changes in item difficulty. It proposes DECAF, a framework based on an MDIF-IRT model that represents language-specific shifts in model capability and item difficulty separately. The framework is evaluated on MMLU-ProX using responses from 25 models across 29 languages. It improves held-out response prediction over empirical accuracy and Rasch baselines.

Critical strengths
  1. The paper addresses a real and important problem where multilingual evaluation can conflate model capability differences with translation quality
  2. Sharing information across languages improves prediction of unobserved responses and provides a practical tool for screening potentially problematic benchmark items
Critical weaknesses
  1. The interpretation of the decomposition is insufficiently validated. Item-language difficulty shifts may reflect genuine domain-specific knowledge gaps rather than translation artifacts, and improved response prediction alone does not establish that these sources have been disentangled
  2. The translation-artifact diagnoses rely on LLM judgments without independent validation. Although the paper provides examples, it lacks a bilingual human audit and a comparison with unflagged items to assess the reliability of the diagnoses and the precision of the screening method
  3. The paper cites Multilingual-IRT, its most directly related prior work, but does not compare against it experimentally, leaving the incremental value of the proposed method unclear
Questions for authors
  1. Could you provide evidence that the estimated item-language difficulty shifts distinguish translation artifacts from genuine language-specific domain knowledge gaps? A small bilingual audit of flagged and matched unflagged items, or reevaluation after correcting identified translations, would help validate this interpretation.
  2. Could you compare MDIF-IRT with Multilingual-IRT on the same held-out response prediction and artifact-screening tasks? This would help clarify the incremental benefit of the proposed formulation.
3 comments
AuthorsOpenReview ↗

We thank the reviewer for their constructive comments and questions. Below, we provide our response to their questions.

We thank the reviewer for their constructive comments and questions. Below, we provide our response to their questions. All substantive changes are highlighted in blue in the updated PDF.

  1. Interpretation of the Decomposition:

In MDIF-IRT, the shift in model performance (accuracy) is decomposed into language-specific capability shift and item difficulty shift. Therefore, domain-specific knowledge gaps of a model are explained by this language-specific capability shift. This is better explained in the MDIF-IRT Model section in the updated manuscript.

“Language-specific model capability (θil\theta_{il}) represents the intrinsic reasoning, knowledge retrieval, and generative proficiency of model ii when operating in language ll. Therefore, a shift in language-specific capability can be attributed to model-side characteristics, such as language-dependent knowledge gaps stemming from imbalances in the model's pretraining corpus. Conversely, language-specific item difficulty (bjlb_{jl}) reflects the inherent cognitive demand of the original item jj combined with any language-specific characteristics (e.g., lack of technical terms) and artifacts (e.g., translation errors or culturally specific content) introduced by translating it into language ll.”

  1. LLM Judgement without Human Validation:

We agree with the reviewer that validating the separation between translation artifacts and genuine knowledge gaps is critical. To assess the reliability and precision of the diagnoses, the authors conducted a small bilingual audit. We randomly selected 27 items, 15 from the flagged subset and 12 from the unflagged subset, whose translated language with the largest difficulty shift was Chinese or Korean. We manually reviewed the original English item and the translated item, and compared our human diagnoses against the automated LLM judge. The results in Table A7 in Appendix G show strong alignment between the human bilingual judge and the LLM judge. The agreement rate for translation error detection was 26/27 (96.3%) and 23/27 (85.2%) for culturally specific content detection.

Table A7: Comparison of LLM-judge and human-judge diagnostics for the 27 audited items

ItemLang.GroupDIF Stat.Trans. (LLM)Trans. (Human)Trans. AgreeCult. (LLM)Cult. (Human)Cult. Agree
10306zhFlagged146.6No errorNo error✓UniversalUniversal✓
5962zhFlagged144.2No errorNo error✓UniversalUniversal✓
4726zhFlagged144.1No errorNo error✓Region/Country (SE Asia)Region/Country (Asia)✓
8970zhFlagged131.0No errorNo error✓UniversalUniversal✓
4958zhFlagged122.3No errorNo error✓Region/Country (US)Region/Country✓
4657zhFlagged121.8No errorNo error✓Region/Country (UK)Region/Country✓
6023zhFlagged93.5No errorNo error✓UniversalUniversal✓
710zhFlagged92.4No errorNo error✓UniversalUniversal✓
6052zhFlagged87.3No errorNo error✓UniversalUniversal✓
5032zhFlagged81.7No errorNo error✓Region/Country (US)Universal✗
3911zhControl5.1No errorNo error✓UniversalUniversal✓
7456zhControl6.0No errorNo error✓UniversalUniversal✓
8641zhControl6.4No errorNo error✓UniversalUniversal✓
11543zhControl7.1No errorNo error✓UniversalUniversal✓
874zhControl7.2No errorNo error✓Region/Country (US)Universal✗
7929zhControl12.1No errorNo error✓UniversalUniversal✓
7118zhControl15.1No errorNo error✓UniversalUniversal✓
359zhControl15.9No errorNo error✓UniversalUniversal✓
6459zhControl17.4No errorNo error✓Region/Country (US)Universal✗
3198zhControl19.7No errorNo error✓UniversalUniversal✓
4574koFlagged86.3No errorNo error✓UniversalUniversal✓
1014koFlagged72.7No errorNo error✓Region/Country (US)Region/Country✓
8413koFlagged72.5No errorNo error✓UniversalUniversal✓
1913koFlagged72.0No errorFormatting Error✗UniversalUniversal✓
2546koFlagged67.1No errorNo error✓UniversalReligion/Philosophical✗
9778koControl1.4No errorNo error✓UniversalUniversal✓
9511koControl5.3No errorNo error✓UniversalUniversal✓
AuthorsOpenReview ↗

continued...

  1. Comparison to Multilingual-IRT:

While a comparison to the multilingual IRT model was considered, the original estimation code is not currently publicly available. We contacted the authors multiple times to request access, but were informed that the code cannot be shared due to restricted disclosure.

Although a direct empirical comparison is therefore infeasible, our MDIF-IRT model offers two advantages over this existing approach, as discussed in our manuscript:

  • Bypasses Normality Assumption: First, the latent capabilities are assumed to follow a normal distribution in the multilingual IRT model. However, the empirical distributions of LLM capabilities may deviate from this assumption (Zhou et al., 2026; Jiang et al., 2026), potentially affecting the reliability of capability estimation. On the other hand, our method treats the item parameters as random, and capability parameters as fixed, being more robust to the violation of normality assumptions.

  • Interpretation of Difficulty Shift: Second, although these multilingual benchmarks are initially developed in English and subsequently translated, the existing approach models item difficulty as a composite of a baseline difficulty shared across all languages and a language-specific deviation. Such a formulation prevents the language-specific deviations from being directly interpreted as artifacts of the translation process.

  • Psychometric Framework DIF/Alignment: Third, our approach implements an alignment (Asparouhov & Muthén, 2014) and DIF framework from psychometrics (Holland et al., 1993) to capture effects of translation on both item characteristics and model capabilities. While the item-by-language difficulty shift in the multilingual IRT is restricted by a strict sum-to-zero constraint for identifiability, our method uses a Gaussian prior and a post-hoc alignment. This allows for a non-zero average difficulty shift across a language. Furthermore, by adopting the psychometric notion of DIF, we interpret shifts in item difficulties and model capabilities across languages as a function of the translation process, rather than unexplained residual variance in model performance.

To ensure reproducibility and accessibility of our method, we will make our complete code publicly available.

Thank you for the additional experiments and clarifications. These additions are helpful and give me a clearer understanding of the method and its assumptions. I remain somewhat uncertain about the incremental contribution relative to the broader line of multilingual IRT research. This remains my main reservation regarding the overall contribution.

Reviewer_gtACOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multilingual problems and argues that the translation process often introduces item-level artifacts, such as translation errors and culturally sensitive content. And the translation artifacts obscure the accurate estimation of a model’s language-specific capabilities. The paper proposes a new method named Multilingual Differential Item Functioning Item Response Theory (MDIF-IRT) model to disentangle language effects into distinct model capability shifts and item difficulty shifts. The experiments are based on MMLU-ProX dataset, evaluating 25 models across 29 languages and demonstrate the utility of the proposed method.

Critical strengths

The paper introduces IRT theory to multilingual problems.

The experiment results are interesting.

Critical weaknesses

The motivation of the paper is not clear. It would be better to provide more explanation. It seems like that the paper is to apply IRT theory to multilingual setting.

It would be better to have clear definition about language-specific capability and characteristics of benchmark items.

The current experimental results are limited. For example, the paper just uses one dataset.

Questions for authors

What are language-specific latent capabilities?

What does the language effect mean?

4 comments
AuthorsOpenReview ↗

We thank the reviewer for their suggestions to improve the clarity of our manuscript.

We thank the reviewer for their suggestions to improve the clarity of our manuscript. Below, we address your comments in a point-by-point manner. All substantive changes are highlighted in blue in the updated PDF.

  1. Clarifying Motivation: We provided more explanation about the limitation of standard metrics like raw accuracy in multilingual LLM evaluation and how the novel method addresses it in the Introduction section.

“Despite the worldwide adoption of large language models (LLMs), their evaluation practices have primarily focused on the English language (Xuan et al., 2025; Singh et al., 2025; Xu et al., 2025). Recently, multilingual benchmarks, such as MMLU-ProX (Xuan et al., 2025), Global-MMLU (Singh et al., 2025), MGSM (Shi et al., 2022), and MGSM-Pro (Xu et al., 2026) have been developed to evaluate model capabilities across diverse languages. However, evaluating multilingual AI poses unique challenges due to artifacts introduced during benchmark item translation, such as direct translation errors and culturally sensitive item content (Peter et al., 2026; Lior et al., 2026; Singh et al., 2025; Xu et al., 2025). These item-level artifacts obscure the evaluation of the models’ true language-specific capabilities (Singh et al., 2025). Consequently, a systematic approach is needed to disentangle a model’s language-specific capability from the language-specific characteristics of the benchmark items. When a model’s performance drops in another language, relying on standard metrics like raw accuracy cannot distinguish between a model’s poor multilingual reasoning and a benchmark’s poor translation quality. To address this, this study proposes a novel item response theory (IRT) framework based on differential item functioning (DIF; Holland et al. (1993); Osterlind & Everson (2009)) to disentangle a model’s true language-specific capability from benchmark item difficulty shifts induced by translation artifacts.”

  1. Defining Language-Specific Capabilities and Item Characteristics: We have added conceptual definitions for both language-specific capability and item difficulty in the updated manuscript in the MDIF-IRT Model section.

“Language-specific model capability (θil\theta_{il}) represents the intrinsic reasoning, knowledge retrieval, and generative proficiency of model i when operating in language l. Therefore, a shift in language-specific capability can be attributed to model-side characteristics, such as language-dependent knowledge gaps stemming from imbalances in the model's pretraining corpus. Conversely, language-specific item difficulty (bjlb_{jl}) reflects the inherent cognitive demand of the original item j combined with any language-specific characteristics (e.g., lack of technical terms) and artifacts (e.g., translation errors or culturally specific content) introduced by translating it into language l.”

  1. Defining Language Effect: We appreciate the reviewer pointing out this ambiguity. In our study, the language effect refers to the observed performance discrepancy (e.g., a drop in raw accuracy) when evaluating a model in a non-English language compared to the English baseline. To improve clarity, we have replaced the term “language effect” with the more precise term “observed cross-lingual performance drops” in the Abstract of the updated manuscript.

  2. Scope of Experimental Dataset: While we acknolwedge that our experiment is limited to one dataset, it is important to emphasize its diversity. The empirical response evaluated in our study encompasses 25 LLMs across 29 languages and 11759 items. Furthermore, we conducted a comprehensive series of analysis on this data to thoroughly demonstrate the utility of the proposed method. Nevertheless, we agree that verifying the framework on different datasets is an important next step. We dicuss this as a limitation of the current work in the Limitations section.

Thank you for the response. Experiments would be improved.

AuthorsOpenReview ↗

We further improved our method with additional experiments

Thank you for your comment. During this rebuttal period, we have further improved our method by implementing an alignment method (Asparouhov & Muthen, 2014) and evaluating its impact through simulations and empirical experiments.

The GVEM estimation sets a zero-mean Gaussian prior for the item difficulty shift parameters γ1j\gamma_{1j} for identifiability, which acts as a shrinkage penalty. However, translation artifacts can systematically inflate the difficulty of a subset of items rather than affecting them symmetrically. For low-resource languages (e.g., Wolof) or those with high typological distance from the English reference (e.g., Japanese), these artifacts pull the true mean of the difficulty shifts away from zero. To address this issue and allow for a non-zero average difficulty shift without altering the overall model fit, we implement the multigroup alignment method \citep{asparouhov2014multiple}. The alignment method rescales the latent parameters under the assumption that the majority of item difficulties remain invariant relative to the English reference. Specifically, for each language ll, we estimate an alignment constant clc_l that minimizes the objective function:

F=∑j=1J(γ^1jl−cl)2+ϵ,F = \sum_{j = 1}^J \sqrt{\sqrt{(\hat{\gamma}_{1jl} - c_l)^2 + \epsilon}},

where γ^1jl\hat{\gamma}_{1jl} is the initial difficulty shift estimate (i.e., the posterior mean) and ϵ=10−4\epsilon = 10^{-4} is a small constant. The item difficulty and model capability shift parameters are updated as follows:

γ^1jl∗=γ^1jl−cl\hat\gamma_{1jl}^* = \hat\gamma_{1jl} - c_l

β^1il∗=β^1il−cl.\hat\beta_{1il}^* = \hat\beta_{1il} - c_l. As a result, this alignment minimizes the deviation of language-specific item difficulty from the English reference. In doing so, it extracts the systematic benchmark-wide translation degradation (clc_l) and ensures language-specific capability estimates are disentangled from translation artifacts, while preserving the predictive probabilities pijlp_{ijl}.

  • Empirical Verification via Simulation: To verify that alignment mitigates the identifiability shrinkage without distorting downstream DIF screening, we conducted a simulation study under a sparsity condition (Appendix C). The simulation result confirms that the zero-mean prior constraint forces the pre-aligned model to absorb the translation degradation into the LLM capability scores (Bias = -0.293). The alignment mitigates this bias (Bias = -0.149) and substantially reduces RMSE (0.335 vs 0.220). In addition, when comparing the downstream cross-lingual DIF screening, the pre- and post-aligned DIF statistic (D_j) show strong rank correlation (Spearman 0.859) and a 97.4% agreement rate in item flags. This confirms that the translation artifact detection is consistent, while the alignment mitigates the bias in capability estimates induced by the zero-mean identifiability constraint.

Table A3: Parameter recovery and DIF consistency pre- and post-alignment

MetricPre-AlignmentPost-Alignment
Parameter Recovery
Bias (γ1\gamma_1)-0.301-0.157
RMSE (γ1\gamma_1)0.4690.392
Bias (β1\beta_1)-0.293-0.149
RMSE (β1\beta_1)0.3350.220
DIF Screening Consistency
DIF Stat Rank Correlation (Spearman ρ\rho)---0.859
DIF Flag Agreement Rate---97.4%
  • Empirical Benchmark Application: Consistent with these simulation findings, our analysis of the benchmark data (Figure A1, Appendix E) showed that before alignment, the zero-mean prior compressed the estimated benchmark-wide difficulty shifts to a maximum of 0.029. After applying the alignment method, the language-specific difficulty shift expands up to 0.069 for low-resource (e.g., Wolof) or typologically distant languages (e.g., Japanese).

Table: Maximum language-specific difficulty shift before and after alignment

LanguagePre-alignmentPost-alignment
AF0.0100.021
AR0.0130.041
BN0.0160.032
CS0.0070.013
DE0.0040.001
ES0.002-0.005
FR0.003-0.003
HI0.0160.024
HU0.0090.015
ID0.0040.000
IT0.003-0.002
JA0.0120.041
KO0.0130.036
MR0.0150.020
NE0.0170.013
PT0.003-0.003
RU0.0060.014
SR0.0090.019
SW0.0200.024
TE0.0160.017
TH0.0170.044
UK0.0090.018
UR0.0150.019
VI0.0080.016
WO0.0290.069
YO0.0260.015
ZH0.0080.032
ZU0.0220.021
AuthorsOpenReview ↗

In addition, we conducted bilingual human audit to partially validate the LLM judge's diagnostic on both flagged/control items

To assess the reliability and precision of the LLM-judge's diagnoses, the authors conducted a small bilingual audit. We randomly selected 27 items, 15 from the flagged subset and 12 from the unflagged subset, whose translated language with the largest difficulty shift was Chinese or Korean. We manually reviewed the original English item and the translated item, and compared our human diagnoses against the automated LLM judge. The results in Table A7 in Appendix G show strong alignment between the human bilingual judge and the LLM judge. The agreement rate for translation error detection was 26/27 (96.3%) and 23/27 (85.2%) for culturally specific content detection.

Table A7: Comparison of LLM-judge and human-judge diagnostics for the 27 audited items

ItemLang.GroupDIF Stat.Trans. (LLM)Trans. (Human)Trans. AgreeCult. (LLM)Cult. (Human)Cult. Agree
10306zhFlagged146.6No errorNo error✓UniversalUniversal✓
5962zhFlagged144.2No errorNo error✓UniversalUniversal✓
4726zhFlagged144.1No errorNo error✓Region/Country (SE Asia)Region/Country (Asia)✓
8970zhFlagged131.0No errorNo error✓UniversalUniversal✓
4958zhFlagged122.3No errorNo error✓Region/Country (US)Region/Country✓
4657zhFlagged121.8No errorNo error✓Region/Country (UK)Region/Country✓
6023zhFlagged93.5No errorNo error✓UniversalUniversal✓
710zhFlagged92.4No errorNo error✓UniversalUniversal✓
6052zhFlagged87.3No errorNo error✓UniversalUniversal✓
5032zhFlagged81.7No errorNo error✓Region/Country (US)Universal✗
3911zhControl5.1No errorNo error✓UniversalUniversal✓
7456zhControl6.0No errorNo error✓UniversalUniversal✓
8641zhControl6.4No errorNo error✓UniversalUniversal✓
11543zhControl7.1No errorNo error✓UniversalUniversal✓
874zhControl7.2No errorNo error✓Region/Country (US)Universal✗
7929zhControl12.1No errorNo error✓UniversalUniversal✓
7118zhControl15.1No errorNo error✓UniversalUniversal✓
359zhControl15.9No errorNo error✓UniversalUniversal✓
6459zhControl17.4No errorNo error✓Region/Country (US)Universal✗
3198zhControl19.7No errorNo error✓UniversalUniversal✓
4574koFlagged86.3No errorNo error✓UniversalUniversal✓
1014koFlagged72.7No errorNo error✓Region/Country (US)Region/Country✓
8413koFlagged72.5No errorNo error✓UniversalUniversal✓
1913koFlagged72.0No errorFormatting Error✗UniversalUniversal✓
2546koFlagged67.1No errorNo error✓UniversalReligion/Philosophical✗
9778koControl1.4No errorNo error✓UniversalUniversal✓
9511koControl5.3No errorNo error✓UniversalUniversal✓

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.