- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how pairwise AI judgments can be aggregated into scalar rewards for group-based policy optimization. It proposes Topological Consensus Rewards (TCR), which uses the Eades feedback-arc-set heuristic to construct an ordering, retains forward edges, and computes retained-graph net-degree rewards followed by group-relative standardization. The paper also introduces Cycle Incidence Rate (CIR) as a diagnostic and provides a conditional triangle-count analysis under independent comparison errors. Validation includes five-seed experiments using Qwen3-14B with GRPO and Qwen3-8B with GSPO on 1,000 WildChat queries, evaluated on Arena-Hard, MT-Bench, and WritingBench.
Critical strengths
Converting pairwise judgments into scalar rewards for GRPO or GSPO is a concrete problem in RLAIF. TCR reuses the comparisons collected by an all-pairs PREF baseline, so graph filtering requires no additional judge calls and leaves the optimizer unchanged.
Critical weaknesses
-
The main experimental evidence is not sufficiently reliable. Algorithm 10 reverses the direction of preference-based transitions and can assign higher rewards to worse responses. The differing P2 results between Tables 1 and 4, the conflicting P3 CIR values between Figure 1(b) and Table 4. Essential training and evaluation configurations are also missing, and no experimental code or accessible code link is provided to resolve the implementation ambiguities.
-
The experiments do not adequately separate and validate the contributions of Eades-based greedy edge selection and retained-graph net-degree scoring, leaving unclea. The cost analysis further limits the practical case for TCR because it does not establish how different feedback acquisition methods compare under comparable resource budgets.
Questions for authors
-
Does the Rank Centrality implementation used in Table 3 follow Algorithm 10 as written? What rewards does the implementation assign to a strictly transitive comparison graph such as (A>B>C)?
-
What makes HodgeRank and PREF produce different rewards in the experiments reported in Table 3? Please specify any differences in comparison edges, weights, tie handling, or reward preprocessing. Do their standardized advantages differ when computed from the same comparison matrix?
-
Which prompt and configuration were used for Table 1? Please explain why the P2 results differ between Tables 1 and 4 and why the P3 CIR differs between Figure 1(b) and Table 4.
-
Provide the complete training and evaluation configurations for the main comparisons, including group sizes, training schedules, decoding parameters, reward preprocessing, checkpoint selection, and benchmark scoring? Please also clarify whether Accuracy excludes judge-generated ties or counts them as unsuccessful comparisons, and provide the relevant implementation details or code.
-
Were the TCR and PREF runs paired across seeds? What do the seed-level differences show in each training setting, and what uncertainty estimates do they support?
-
Do you have controlled results that distinguish the contribution of edge selection from that of retained-graph net-degree scoring? What measured token or computational costs support the practical comparison with other feedback acquisition methods?
Response to Reviewer vjB2
Thank you for the careful review. We clarify the baseline definitions and the evidence for TCR's edge selection below.
Q1. Rank Centrality implementation
The printed transition direction is incorrect. With Wii=0 before smoothing, Algorithm 10 approaches (2,3,6)/11 for A>B>C as smoothing tends to zero, rewarding C most.
For Mij=1 when i wins, set qij=(1−Mij)/2 and Pij=(qij+α)/[(1+2α)(G−1)] for i=j, α>0; set Pii=1−∑j=iPij. This sends mass to winners with common normalization and self-loops (Rank Centrality). For α=1, it gives (0.5,0.3,0.2). This corrects the specification, without establishing which implementation generated Table 3. We set aside the RC row as supporting evidence; the PREF and prompt comparisons do not depend on it.
Q2. Why HodgeRank and PREF differ
Under the stated complete, unweighted, antisymmetric comparisons, with ties encoded as zero, your equivalence concern is correct. Let δi=∑jMij. Centered Hodge gives si=δi/G; half-win-tie PREF gives wi=1/2+δi/[2(G−1)].
Their standardized advantages agree at ϵ=0 (constant groups receive zero). With wi=asi+1/2, a=G/[2(G−1)], positive stabilizers require ϵPREF=aϵHodge for exact agreement. Table 3's differing means do not establish distinct reward mechanisms under these conventions. TCR's distinction is selecting comparisons before net-degree scoring.
Q3. P2 results and P3 CIR
Section 4.3 and Appendix C label the main feedback Qwen3-32B/P2. The reported TCR/PREF means are 52.9/51.6 in Table 1 and 51.8/50.6 in Table 4's P2 column. These entries are not reconciled in the manuscript. Matching TCR's 52.9 under P5 is not a basis for relabeling the main run. We therefore use Table 4 for prompt-specific conclusions.
Figure 1(b) and Table 4 give P3 CIR as 3.7% and 5.7%. We acknowledge this reporting inconsistency and exclude this entry and the associated CIR correlation from our argument. Table 4's within-column TCR-PREF mean gains remain +1.2,+1.2,+0.8,+1.7 Arena-Hard points across P2–P5. This supports higher reported means across the four tested prompts, without relying on either disputed CIR value.
Q4. Protocol and tie handling
Section 4.1 specifies Qwen3-14B/GRPO and Qwen3-8B/GSPO in verl, 1,000 shared WildChat training queries, 200 validation queries for checkpoint selection, and five runs per method. Feedback uses Qwen3-32B; evaluation uses GPT-4.1 for Arena-Hard/MT-Bench and Claude-3.7-Sonnet for WritingBench.
The reward specification uses one judgment per unordered pair, Mji=−Mij, Mii=0; ties give PREF half a win and TCR no edge. TCR standardizes retained net degrees within each group; constant groups receive zero advantage. This specifies aggregation, not a complete executable training configuration.
For accuracy, let C be correct, D non-tie, and N all comparisons. For D>0, the caption describes C/D, while Algorithm 4 computes C/N=(D/N)(C/D): conditional accuracy versus unconditional success. We exclude the disputed accuracy percentages from the error-removal argument. TCR's training rewards require no human labels.
Q5. Seeds and uncertainty
Section 4.1 reports five runs per method; we make no claim of matched seed pairs or a paired significance test. Table 1 gives TCR/PREF Arena means of 52.9±1.1 / 51.6±1.0 for GRPO/14B and 52.5±1.0 / 51.9±0.8 for GSPO/8B. These are mean±SD, describing between-run variation.
The reported improvements extend beyond Arena: TCR-PREF MT-Bench one-/two-turn mean differences are +0.48/+0.44 for GRPO/14B and +0.57/+0.13 for GSPO/8B. WritingBench differences are +0.12 and -0.02, respectively. Both settings thus show higher conversational means; the writing results are mixed. We present this as descriptive evidence across tasks, not statistical significance or uniform superiority.
Q6. Edge selection, scoring, and cost
At fixed comparisons and matched normalization, PREF and TCR apply net-degree scoring before and after selection, respectively. RandomResolve and ReverseResolve also specify net-degree scoring. Table 3 reports Arena means of 51.6/51.6/51.0/52.9 for PREF/RandomResolve/ReverseResolve/TCR. These are reported graph-treatment comparisons, not a scorer ablation on a fixed retained graph.
Appendix B adds evidence under independent comparison flips. At p=0.8, Table 5 reports 85.7% removed-edge precision versus a 20.1% random-edge baseline across sizes 8–12. This supports selective removal under the stated model, not error identification in every real cycle or optimality of the scorer.
TCR can reuse PREF's G(G−1)/2 judgments, adds zero judge calls, and leaves the optimizer unchanged. This supports using existing pairwise feedback, not a measured wall-time or token-cost advantage over other feedback formats.