- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RubricDuo, which organizes historical query–rubric pairs into a knowledge graph connecting evaluation concepts with their source criteria and queries. It generates task-specific rubrics through concept-guided drafting followed by criterion-level refinement, improving rubric alignment and downstream judging on RubricBench and HealthBench.
Critical strengths
The method is clearly presented, with an understandable knowledge organization scheme and a well-defined draft–refine pipeline.
Critical weaknesses
-
Insufficient component-level validation. Table 3 compares graph-guided and flat retrieval, but does not isolate the contributions of concept prompting, concept-based filtering, and source-query context. Consequently, the benefit of the proposed multi-level graph organization remains insufficiently established. Table 4 only evaluates refinement as a whole.
-
Sec. 3 requires offline graph construction and online concept matching, retrieval, and criterion-level refinement, yet their costs are not reported. Given the modest refinement gains in Table 4, the quality–cost trade-off is difficult to assess.
Questions for authors
-
Table 5 shows that with Qwen3.8-max, RubricDuo improves rubric F1 but does not outperform RubricRAG in downstream accuracy. What explains this discrepancy, and do the gains persist with stronger judges or judges from another model family?
-
Appendix A.1 uses an LLM to match generated and human-authored criteria. Which model performs this matching, and has its reliability been checked against human judgments?
-
How sensitive is shared-concept induction to the order of historical cases and the initialization batch? Can the authors provide examples of successful cross-topic transfer and failures caused by inappropriate concept matching?