- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CARE is a judge for Office files. It scores how well each file satisfies task rubrics. It is trained on controlled file edits, so that scores respond to relevant changes and stay stable when rubric satisfaction does not change.
Critical strengths
- This paper tackles a meaningful evaluation problem. Files that appear identical can still differ in required formulas, styles, or speaker notes.
Critical weaknesses
- The practical advantage over SFT is still not clear. On all three test splits, full CARE achieves lower macro-F1 than Artifact-SFT. Invariance improves consistency, but this gain comes at the cost of weaker classification performance. Greater score consistency alone does not make a better judge.
- The data and preprocessing pipeline is hard to reproduce. The paper mentions "registered, redistributable Office sources" but does not identify them. It also provides limited detail on how task-specific edits are built and how relevant file information is chosen for each rubric.
- The OfficeVal results contain a consequential inconsistency. The introduction reports 0.06705 for full CARE and 0.06999 without invariance, whereas Table 2 and §6.5 report the opposite. This flips the conclusion about whether invariance helps or hurts candidate selection.
- The OfficeVal candidate pool offers limited evidence about stronger outputs. Even choosing the best candidate with the official verifier reaches only 0.09225. It remains unclear whether CARE can reliably tell apart smaller quality differences among higher-quality files.
Questions for authors
See Weaknesses 1–4.