- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates how AI-generated content homogenization affects the behavior and internal representations of LLM evaluators. The authors analyze pairwise hidden-state differences using PCA and ridge regression to identify components associated with evaluator preferences before and after LLM rewriting. They find that evaluator accuracy remains largely stable despite substantial changes in the predictive importance of representation components. Building on this analysis, the paper proposes DACO, a representation-guided content optimization method that converts selected components into actionable rewriting guidance. Experiments across human-preference, assistant-response ranking, and hiring tasks evaluate these representation shifts, their causal relevance, and DACO's effectiveness against rewriting baselines, with additional cross-model and human evaluations.
Critical strengths
I think the paper addresses a timely and relevant problem at the intersection of AI-assisted writing and LLM-based evaluation. The experimental setup is reasonably comprehensive, covering multiple evaluation domains and model families. The authors also incorporate thoughtful experimental controls, including position-bias mitigation, held-out data splits, and explicit protected-literal preservation checks. I also appreciate the author provides detailed implementation and statistical reporting, supporting reproducibility.
Critical weaknesses
1. Insufficient validation of the content homogenization setting.
The paper operationalizes content homogenization through standardized LLM rewriting, but provides limited evidence that the resulting texts actually become less distinguishable. Although TF-IDF similarity is mentioned as a diagnostic, systematic measurements of homogenization are not reported, and there is no controlled comparison against general rewriting without homogenization. Consequently, the observed representation shifts could reflect rewriting-induced distribution changes rather than homogenization specifically. Furthermore, the original preference labels in SHP and OASST1 are reused after rewriting, even though changes in writing quality or presentation could alter human preferences without changing factual content. Without validating label consistency, the claim that evaluator accuracy remains stable under homogenization is also difficult to interpret.
2. Predictive importance shifts do not establish changes in evaluator decision mechanisms.
The paper interprets changes in PCA component importance as evidence that homogenization alters the features associated with evaluator decisions. However, permutation importance is distribution-dependent: changes in feature variance or correlations can reorder importance rankings even when the evaluator's underlying decision rule remains unchanged. Using a shared PCA basis does not eliminate this ambiguity. Although the authors attempt causal validation, none of the 12 component interventions achieves significance after multiple-testing correction. Thus, the experiments establish shifts in predictive associations but provide insufficient evidence that the evaluator actually relies on different decision-relevant features. This limits the interpretation of the paper's central representation-level finding.
3. Limited evidence of the effectiveness of DACO.
My primary concern is that DACO does not demonstrate a consistent advantage over simpler rewriting baselines. Across the three primary evaluators, none of its paired comparisons shows a statistically significant positive improvement. Its accuracy is only marginally higher than B0 on Gemma, lower on Ministral, and identical on Phi-4. The external evaluations also provide little additional support. Moreover, the functional validation primarily demonstrates movement along selected representation directions rather than improved ranking outcomes. Thus, it remains unclear to me whether the proposed representation-guided approach provides meaningful benefits beyond conventional content rewriting.
Questions for authors
See weakness above.
Thank you for the detailed response and the additional analyses. I appreciate the transparency, in particular the explicit acknowledgment that TF-IDF does not show lexical convergence, and that no causal claim is made. The clarifications on who generates the guidance, which candidates are rewritten, and the new decision-flip counts are helpful. Most of my concerns have been addressed, and I have raised my score to 2.
By the way, the layout of the experimental results table in the paper could be made clearer.