- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces DECAF (Disentangling Capability and Translation Artifacts), centered around a Multilingual Differential Item Functioning Item Response Theory (MDIF-IRT) model. The authors aim to separate true model capability variations across languages from artificial difficulty shifts introduced by machine translation artifacts in benchmarks like MMLU-ProX.
Critical strengths
The motivation is interesting and practically significant.
Critical weaknesses
-
Validity Dilemma of the Diagnostic Pipeline: The authors position their framework as an essential diagnostic tool for benchmark refinement, using GPT-5.5 to validate that 71.8% of the 748 flagged items suffer from translation errors. This reveals a fatal methodological dilemma:
-
If LLMs are already reliable judges of translation errors in low-resource languages: Then the entire complex pipeline (benchmarking 25 LLMs across 29 languages → collecting response matrices → running GVEM → flagging DIF items) is completely redundant. Benchmark creators could simply deploy an LLM-as-a-judge quality-estimation pipeline directly on the translated text before running evaluations, saving enormous compute.
-
If LLMs are NOT reliable judges of these languages (which is widely established for low-resource languages like Wolof): Then using GPT-5.5 as the sole ground truth to claim that "71.8% of flagged items are translation errors" is ungrounded, circular, and scientifically invalid.
-
-
The paper’s titular claim is "Disentangling Capability and Translation Artifacts." However, to ensure identifiability, the authors impose a zero-centered prior on item difficulty shifts: (lines 208–212). This introduces a critical theoretical breakdown: If a machine-translation pipeline produces systematically poor or degraded translations across an entire low-resource language (e.g., Wolof or Yoruba), the mean translation error is non-zero. Under the zero-mean constraint, the model mathematically forces the entirety of this average translation degradation into the model capability parameter. Consequently, the framework cannot disentangle true model capability from benchmark-wide translation failure; it only isolates item-level variance around the language's average degradation. Calling this "disentanglement of translation artifacts" is an over-claim.
-
The proposed model is only compared exclusively against strawman baselines: Empirical Accuracy, Pooled Rasch, and Independent Rasch
Questions for authors
See Weaknesses
2 comments
Identifiability and Disentangling Capability and Translation Artifacts:
We sincerely thank the reviewer for this insightful point. We agree that the zero-centered prior used for identifiability of the model may fail to capture systematic translation degradation in low-resource languages. To mitigate this issue, we implemented a multigroup alignment method (Asparouhov & Muthen, 2014). We have updated the MDIF-IRT Model section to detail this alignment implementation:
The GVEM estimation sets a zero-mean Gaussian prior for the item difficulty shift parameters for identifiability, which acts as a shrinkage penalty. However, translation artifacts can systematically inflate the difficulty of a subset of items rather than affecting them symmetrically. For low-resource languages (e.g., Wolof) or those with high typological distance from the English reference (e.g., Japanese), these artifacts pull the true mean of the difficulty shifts away from zero. To address this issue and allow for a non-zero average difficulty shift without altering the overall model fit, we implement the multigroup alignment method \citep{asparouhov2014multiple}. The alignment method rescales the latent parameters under the assumption that the majority of item difficulties remain invariant relative to the English reference. Specifically, for each language , we estimate an alignment constant that minimizes the objective function:
where is the initial difficulty shift estimate (i.e., the posterior mean) and is a small constant. The item difficulty and model capability shift parameters are updated as follows:
As a result, this alignment minimizes the deviation of language-specific item difficulty from the English reference. In doing so, it extracts the systematic benchmark-wide translation degradation () and ensures language-specific capability estimates are disentangled from translation artifacts, while preserving the predictive probabilities .
- Empirical Verification via Simulation: To verify that alignment mitigates the identifiability shrinkage without distorting downstream DIF screening, we conducted a simulation study under a sparsity condition (Appendix C). The simulation result confirms that the zero-mean prior constraint forces the pre-aligned model to absorb the translation degradation into the LLM capability scores (Bias = -0.293). The alignment mitigates this bias (Bias = -0.149) and substantially reduces RMSE (0.335 vs 0.220). In addition, when comparing the downstream cross-lingual DIF screening, the pre- and post-aligned DIF statistic (D_j) show strong rank correlation (Spearman 0.859) and a 97.4% agreement rate in item flags. This confirms that the translation artifact detection is consistent, while the alignment mitigates the bias in capability estimates induced by the zero-mean identifiability constraint.
Table A3: Parameter recovery and DIF consistency pre- and post-alignment
| Metric | Pre-Alignment | Post-Alignment |
|---|---|---|
| Parameter Recovery | ||
| Bias () | -0.301 | -0.157 |
| RMSE () | 0.469 | 0.392 |
| Bias () | -0.293 | -0.149 |
| RMSE () | 0.335 | 0.220 |
| DIF Screening Consistency | ||
| DIF Stat Rank Correlation (Spearman ) | --- | 0.859 |
| DIF Flag Agreement Rate | --- | 97.4% |
- Empirical Benchmark Application: Consistent with these simulation findings, our analysis of the benchmark data (Figure A1, Appendix E) showed that before alignment, the zero-mean prior compressed the estimated benchmark-wide difficulty shifts to a maximum of 0.029. After applying the alignment method, the language-specific difficulty shift expands up to 0.069 for low-resource (e.g., Wolof) or typologically distant languages (e.g., Japanese).
Table: Maximum language-specific difficulty shift before and after alignment
| Language | Pre-alignment | Post-alignment |
|---|---|---|
| AF | 0.010 | 0.021 |
| AR | 0.013 | 0.041 |
| BN | 0.016 | 0.032 |
| CS | 0.007 | 0.013 |
| DE | 0.004 | 0.001 |
| ES | 0.002 | -0.005 |
| FR | 0.003 | -0.003 |
| HI | 0.016 | 0.024 |
| HU | 0.009 | 0.015 |
| ID | 0.004 | 0.000 |
| IT | 0.003 | -0.002 |
| JA | 0.012 | 0.041 |
| KO | 0.013 | 0.036 |
| MR | 0.015 | 0.020 |
| NE | 0.017 | 0.013 |
| PT | 0.003 | -0.003 |
| RU | 0.006 | 0.014 |
| SR | 0.009 | 0.019 |
| SW | 0.020 | 0.024 |
| TE | 0.016 | 0.017 |
| TH | 0.017 | 0.044 |
| UK | 0.009 | 0.018 |
| UR | 0.015 | 0.019 |
| VI | 0.008 | 0.016 |
| WO | 0.029 | 0.069 |
| YO | 0.026 | 0.015 |
| ZH | 0.008 | 0.032 |
| ZU | 0.022 | 0.021 |
We thank the reviewer for raising these critical questions that helped us significantly improve our manuscript.
We thank the reviewer for raising these critical questions that helped us significantly improve our manuscript. Please find our point-by-point responses below.
We would like to clarify that the automated qualitative review process was intended as a post-hoc illustration of the types of translation artifacts present in the flagged items, rather than as a validation of our method. In addition, as we discuss in the updated manuscript, our model-based approach provides benefits that an LLM-based judge cannot capture, because the LLM judge focuses only on the linguistic aspects of the items. To make this clear, we have added the following discussion to the revised manuscript in the Experiment section.
“However, note that this automated qualitative review serves as a post-hoc illustration of the types of translation artifacts that may be present in the flagged items, rather than a validation of the MDIF-IRT framework. The results demonstrate that relying on an LLM judge to review translated benchmark items may fail in two ways. First, among poorly translated items, it cannot distinguish those that cause a substantial difficulty shift and those with a negligible shift that does not affect LLM evaluation. Second, it fails to detect items that exhibit significant difficulty shifts unexplained by literal translation artifacts. In contrast, the MDIF-IRT framework relies on empirical response patterns to disentangle true language-specific capabilities from artificial difficulty shifts.”
While a comparison to a more advanced method (e.g., the multilingual IRT model) was considered, the original estimation code for that method is not currently publicly available. We contacted the authors multiple times to request access, but were informed that the code cannot be shared due to restricted disclosure.
Although a direct empirical comparison is therefore infeasible, our MDIF-IRT model offers advantages over this existing approach, as discussed in our manuscript:
Bypasses Normality Assumption: First, the latent capabilities are assumed to follow a normal distribution in the multilingual IRT model. However, the empirical distributions of LLM capabilities may deviate from this assumption (Zhou et al., 2026; Jiang et al., 2026), potentially affecting the reliability of capability estimation. On the other hand, our method treats the item parameters as random, and capability parameters as fixed, being more robust to the violation of normality assumptions.
Interpretation of Difficulty Shift: Second, although these multilingual benchmarks are initially developed in English and subsequently translated, the existing approach models item difficulty as a composite of a baseline difficulty shared across all languages and a language-specific deviation. Such a formulation prevents the language-specific deviations from being directly interpreted as artifacts of the translation process.
Psychometric Framework DIF/Alignment: Third, our approach implements an alignment (Asparouhov & Muthén, 2014) and DIF framework from psychometrics (Holland et al., 1993) to capture effects of translation on both item characteristics and model capabilities. While the item-by-language difficulty shift in the multilingual IRT is restricted by a strict sum-to-zero constraint for identifiability, our method uses a Gaussian prior and a post-hoc alignment. This allows for a non-zero average difficulty shift across a language. Furthermore, by adopting the psychometric notion of DIF, we interpret shifts in item difficulties and model capabilities across languages as a function of the translation process, rather than unexplained residual variance in model performance.
We now describe these advantages in the Introduction of the updated manuscript. To ensure reproducibility and accessibility of our method, we will make our complete code publicly available.