acceptodds
Under review as a conference paper at ICLR 2027

Voting with the Graph: Consensus Rewards for RLAIF via Topological Consistency

Abstract

Pairwise LLM judgments provide local preferences, whereas group-based policy optimization requires scalar rewards. We study how to aggregate these comparisons for Reinforcement Learning from AI Feedback (RLAIF). We introduce Topological Consensus Rewards (TCR), which uses a greedy feedback-arc-set heuristic to retain an acyclic subset of comparisons and computes net-degree rewards on the retained graph. Its distinguishing operation is evidence selection: raw win-tie rates and original-graph net degrees are equivalent under group standardization, up to the numerical stabilizer. We use Cycle Incidence Rate (CIR) as a structural diagnostic and analyze triangle participation under an independent-error model to motivate the aggregation choice. Experiments on Arena-Hard, MT-Bench, and WritingBench evaluate TCR in two Qwen3 training settings, with higher mean Arena-Hard overall scores than the compared pairwise baselines. The results support transitivity as a useful reward-aggregation bias when a single preference ordering is an appropriate approximation, rather than a guarantee of preference-error recovery.

Primary area:
reinforcement learning
Keywords:
reinforcement learning from AI feedback, preference aggregation, pairwise comparisons, reward design, graph-based alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vjB2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how pairwise AI judgments can be aggregated into scalar rewards for group-based policy optimization. It proposes Topological Consensus Rewards (TCR), which uses the Eades feedback-arc-set heuristic to construct an ordering, retains forward edges, and computes retained-graph net-degree rewards followed by group-relative standardization. The paper also introduces Cycle Incidence Rate (CIR) as a diagnostic and provides a conditional triangle-count analysis under independent comparison errors. Validation includes five-seed experiments using Qwen3-14B with GRPO and Qwen3-8B with GSPO on 1,000 WildChat queries, evaluated on Arena-Hard, MT-Bench, and WritingBench.

Critical strengths

Converting pairwise judgments into scalar rewards for GRPO or GSPO is a concrete problem in RLAIF. TCR reuses the comparisons collected by an all-pairs PREF baseline, so graph filtering requires no additional judge calls and leaves the optimizer unchanged.

Critical weaknesses
  1. The main experimental evidence is not sufficiently reliable. Algorithm 10 reverses the direction of preference-based transitions and can assign higher rewards to worse responses. The differing P2 results between Tables 1 and 4, the conflicting P3 CIR values between Figure 1(b) and Table 4. Essential training and evaluation configurations are also missing, and no experimental code or accessible code link is provided to resolve the implementation ambiguities.

  2. The experiments do not adequately separate and validate the contributions of Eades-based greedy edge selection and retained-graph net-degree scoring, leaving unclea. The cost analysis further limits the practical case for TCR because it does not establish how different feedback acquisition methods compare under comparable resource budgets.

Questions for authors
  1. Does the Rank Centrality implementation used in Table 3 follow Algorithm 10 as written? What rewards does the implementation assign to a strictly transitive comparison graph such as (A>B>C)?

  2. What makes HodgeRank and PREF produce different rewards in the experiments reported in Table 3? Please specify any differences in comparison edges, weights, tie handling, or reward preprocessing. Do their standardized advantages differ when computed from the same comparison matrix?

  3. Which prompt and configuration were used for Table 1? Please explain why the P2 results differ between Tables 1 and 4 and why the P3 CIR differs between Figure 1(b) and Table 4.

  4. Provide the complete training and evaluation configurations for the main comparisons, including group sizes, training schedules, decoding parameters, reward preprocessing, checkpoint selection, and benchmark scoring? Please also clarify whether Accuracy excludes judge-generated ties or counts them as unsuccessful comparisons, and provide the relevant implementation details or code.

  5. Were the TCR and PREF runs paired across seeds? What do the seed-level differences show in each training setting, and what uncertainty estimates do they support?

  6. Do you have controlled results that distinguish the contribution of edge selection from that of retained-graph net-degree scoring? What measured token or computational costs support the practical comparison with other feedback acquisition methods?

1 comment
AuthorsOpenReview ↗

Response to Reviewer vjB2

Thank you for the careful review. We clarify the baseline definitions and the evidence for TCR's edge selection below.

Q1. Rank Centrality implementation

The printed transition direction is incorrect. With Wii=0W_{ii}=0 before smoothing, Algorithm 10 approaches (2,3,6)/11(2,3,6)/11 for A>B>CA>B>C as smoothing tends to zero, rewarding C most.

For Mij=1M_{ij}=1 when ii wins, set qij=(1−Mij)/2q_{ij}=(1-M_{ij})/2 and Pij=(qij+α)/[(1+2α)(G−1)]P_{ij}=(q_{ij}+\alpha)/[(1+2\alpha)(G-1)] for i≠ji\ne j, α>0\alpha>0; set Pii=1−∑j≠iPijP_{ii}=1-\sum_{j\ne i}P_{ij}. This sends mass to winners with common normalization and self-loops (Rank Centrality). For α=1\alpha=1, it gives (0.5,0.3,0.2)(0.5,0.3,0.2). This corrects the specification, without establishing which implementation generated Table 3. We set aside the RC row as supporting evidence; the PREF and prompt comparisons do not depend on it.

Q2. Why HodgeRank and PREF differ

Under the stated complete, unweighted, antisymmetric comparisons, with ties encoded as zero, your equivalence concern is correct. Let δi=∑jMij\delta_i=\sum_jM_{ij}. Centered Hodge gives si=δi/Gs_i=\delta_i/G; half-win-tie PREF gives wi=1/2+δi/[2(G−1)]w_i=1/2+\delta_i/[2(G-1)].

Their standardized advantages agree at ϵ=0\epsilon=0 (constant groups receive zero). With wi=asi+1/2w_i=as_i+1/2, a=G/[2(G−1)]a=G/[2(G-1)], positive stabilizers require ϵPREF=aϵHodge\epsilon_{\mathrm{PREF}}=a\epsilon_{\mathrm{Hodge}} for exact agreement. Table 3's differing means do not establish distinct reward mechanisms under these conventions. TCR's distinction is selecting comparisons before net-degree scoring.

Q3. P2 results and P3 CIR

Section 4.3 and Appendix C label the main feedback Qwen3-32B/P2. The reported TCR/PREF means are 52.9/51.6 in Table 1 and 51.8/50.6 in Table 4's P2 column. These entries are not reconciled in the manuscript. Matching TCR's 52.9 under P5 is not a basis for relabeling the main run. We therefore use Table 4 for prompt-specific conclusions.

Figure 1(b) and Table 4 give P3 CIR as 3.7% and 5.7%. We acknowledge this reporting inconsistency and exclude this entry and the associated CIR correlation from our argument. Table 4's within-column TCR-PREF mean gains remain +1.2,+1.2,+0.8,+1.7 Arena-Hard points across P2–P5. This supports higher reported means across the four tested prompts, without relying on either disputed CIR value.

Q4. Protocol and tie handling

Section 4.1 specifies Qwen3-14B/GRPO and Qwen3-8B/GSPO in verl, 1,000 shared WildChat training queries, 200 validation queries for checkpoint selection, and five runs per method. Feedback uses Qwen3-32B; evaluation uses GPT-4.1 for Arena-Hard/MT-Bench and Claude-3.7-Sonnet for WritingBench.

The reward specification uses one judgment per unordered pair, Mji=−MijM_{ji}=-M_{ij}, Mii=0M_{ii}=0; ties give PREF half a win and TCR no edge. TCR standardizes retained net degrees within each group; constant groups receive zero advantage. This specifies aggregation, not a complete executable training configuration.

For accuracy, let C be correct, D non-tie, and N all comparisons. For D>0D>0, the caption describes C/DC/D, while Algorithm 4 computes C/N=(D/N)(C/D)C/N=(D/N)(C/D): conditional accuracy versus unconditional success. We exclude the disputed accuracy percentages from the error-removal argument. TCR's training rewards require no human labels.

Q5. Seeds and uncertainty

Section 4.1 reports five runs per method; we make no claim of matched seed pairs or a paired significance test. Table 1 gives TCR/PREF Arena means of 52.9±1.1 / 51.6±1.0 for GRPO/14B and 52.5±1.0 / 51.9±0.8 for GSPO/8B. These are mean±SD, describing between-run variation.

The reported improvements extend beyond Arena: TCR-PREF MT-Bench one-/two-turn mean differences are +0.48/+0.44 for GRPO/14B and +0.57/+0.13 for GSPO/8B. WritingBench differences are +0.12 and -0.02, respectively. Both settings thus show higher conversational means; the writing results are mixed. We present this as descriptive evidence across tasks, not statistical significance or uniform superiority.

Q6. Edge selection, scoring, and cost

At fixed comparisons and matched normalization, PREF and TCR apply net-degree scoring before and after selection, respectively. RandomResolve and ReverseResolve also specify net-degree scoring. Table 3 reports Arena means of 51.6/51.6/51.0/52.9 for PREF/RandomResolve/ReverseResolve/TCR. These are reported graph-treatment comparisons, not a scorer ablation on a fixed retained graph.

Appendix B adds evidence under independent comparison flips. At p=0.8p=0.8, Table 5 reports 85.7% removed-edge precision versus a 20.1% random-edge baseline across sizes 8–12. This supports selective removal under the stated model, not error identification in every real cycle or optimality of the scorer.

TCR can reuse PREF's G(G−1)/2G(G-1)/2 judgments, adds zero judge calls, and leaves the optimizer unchanged. This supports using existing pairwise feedback, not a measured wall-time or token-cost advantage over other feedback formats.

Reviewer_k8hkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper is studying how AI-generated pairwise relative comparisons should be converted into single-response scalar rewards used for policy optimization. A graph-based apporach TCR is proposed, which is motivated by the fact that there is a consistent mapping relationship between raw win-tie rates and graph net degrees. After collecting every pairwise AI comparisons given a finite response group, TCR first construct the preference graph. And the scalar reward used for group relative advantage calculation is the residual of the response in the retained subgraph. The main experiments are conducted using Qwen3-14B/8B based on 1k queries from WildChat-1M, where the benchmarks used are Arena-Hard, MT-Bench, and Writing Bench. The proposed TCR demonstrates a consistent improvement over the base model.

Critical strengths
  1. The paper is clearly written, including the motivating observation of non-zero CIR% in LLM judges of high accuracy, anbd the proposed methodology pipeline of TCR.
  2. Comprehensive benchmarks and baselines covered.
Critical weaknesses
  1. The motivation is not strong enough. AI feedback is a particular compromise we need to make when there is no access to human feedback. These AI feedback can easily be elicited via modifying the pairwise comparison prompt to single-response scoring prompt. This undermines the motivation of this paper.
  2. The underlying mechanism making TCR more preferable among the baselines remains unclear. Why removing preference cycles particularly via the way proposed by TCR is more effective in preference modelling?
  3. The experimental configuration needs further justification. The authors sub-sampled WildChat-1M to a dataset size of 1K for single-turn dialog alignment. Beside, part of the empirical analysis is built upon medium to minor performance gains over the baseline methods.
  4. Efficiency analysis is incomplete. A throughput comparison table or a performance vs. wall-time graph is required in the mainbody.
Questions for authors
  1. Why increasing group size lfrom 4 to 7 leads to a performance reduction in Figure 3(b). This is because TRC requiring every one-on-one comparisons restrains this ablation study from further increasing to 16, 32?
  2. Please provide further information on the exact LLM judge configuration used for pairwise feedback. And, can we reduce CIR via greedy decoding?
2 comments
AuthorsOpenReview ↗

Response to Reviewer k8hk

Thank you for the review.

W1. Why pairwise aggregation when direct scoring is available?

Direct scoring is included as the PW baseline: the same Qwen3-32B judge scores each response independently on a 1–10 scale (Section 4.2). Table 1 reports TCR/PW Arena-Hard means of 52.9/48.4 for GRPO/14B and 52.5/50.2 for GSPO/8B, differences of +4.5 and +2.3 points. These are performance comparisons under the reported setups, not equal-cost comparisons.

Our question is how to use relative judgments as group rewards. A scalar prompt supplies an alternative signal; it does not specify how to aggregate existing comparisons. TCR supplies that aggregation step and leaves the policy optimizer unchanged.

W2. Why this cycle-removal rule?

The selection rule seeks an ordering that preserves as many observed comparisons as possible. Under Appendix B's latent total order, complete comparisons without ties, and independent comparison flips with correctness probability 1/2<p<11/2 < p < 1, an order agreeing with A of m comparisons has likelihood pA(1−p)m−Ap^A(1-p)^{m-A}. Maximizing agreement therefore maximizes likelihood. For latent rank i (1 is best), expected net degree is (n−2i+1)(2p−1)(n-2i+1)(2p-1), which motivates degree-based ordering. Eades' heuristic targets this objective; these identities do not guarantee recovery by its adaptive greedy steps.

At p=0.8p=0.8, Table 5 reports that 85.7% of removed edges are simulated errors, versus 20.1% for the random-edge baseline, across graph sizes 8–12. Table 3 gives Arena means of 51.6/51.6/51.0/52.9 for PREF/RandomResolve/ReverseResolve/TCR. These means support the specific selection rule over arbitrary cycle breaking in the tested setting.

With identical comparisons and matched tie/normalization conventions, PREF is affine-equivalent to original-graph net degree. TCR instead scores the retained graph. Figure 2 makes the change concrete: removing E→BE\to B changes scores (4,0,0,-2,-2) to (4,1,0,-2,-3), distinguishing B from C. This explains the reward intervention. The synthetic evidence assumes noisy comparisons of a total order; genuine intransitivity can make filtering discard useful preferences.

W3. The 1K-query setting and modest gains

Section 4.1 uses the same 1,000 WildChat queries for all methods after filtering non-substantive inputs, with 200 validation queries for checkpoint selection and five runs per method. This supports comparison within a shared training-data setting. It does not establish that 1,000 queries are sufficient at larger scales.

Table 1 reports TCR/PREF Arena means of 52.9±1.1/51.6±1.0 for GRPO/14B and 52.5±1.0/51.9±0.8 for GSPO/8B (mean±SD). The MT-Bench one-/two-turn gains are +0.48/+0.44 and +0.57/+0.13, respectively. WritingBench gains are +0.12/-0.02, so the writing results are mixed. Table 4 also reports positive TCR-PREF Arena mean differences under each of P2–P5: +1.2,+1.2,+0.8,+1.7. These results support modest conversational gains across the tested settings; we do not claim statistical significance or a dataset-size scaling result.

W4. Throughput and wall time

For G candidates, TCR and all-pairs PREF can use the same G(G−1)/2G(G-1)/2 judgments. TCR adds graph processing and zero judge calls. Section 3.2.2 gives O((∣V∣+∣E∣)log⁡∣V∣)O((|V|+|E|)\log |V|) for a heap-based graph implementation. The supported efficiency benefit is reuse of pairwise feedback. The paper contains no measured throughput or wall-time comparison, so this complexity bound cannot establish an end-to-end advantage over pointwise or listwise feedback.

Q1. Group-size decline and G=16/32

Section 4.4/Table 2 already evaluates TCR with 16 candidates, reporting 52.5±1.0 on Arena-Hard. This establishes operation beyond G=7G=7, although it is a separate experiment rather than an extension of the same sweep.

In Figure 3(b), both TCR and PREF peak at G=6G=6 and reach 51.5 at G=7G=7 at the displayed precision. The decline also occurs without TCR's filtering; the plot alone does not identify its cause. All-pairs costs are 6,21,120,496 comparisons for G=4,7,16,32G=4,7,16,32. The quadratic acquisition cost remains; the existing results do not establish performance at G=32G=32.

Q2. Judge configuration and greedy decoding

Appendix C specifies qwen3-32b with Prompt P2 for the main feedback; P2–P5 are printed for the prompt study. Section 3.2.1 maps A/B/tie to +1/-1/0, enforces Mji=−MijM_{ji}=-M_{ij}, and omits ties from the strict graph. The manuscript specifies this model/prompt/aggregation protocol but omits temperature, top-p, and output-length settings.

Greedy decoding may change CIR, but it does not impose transitivity across comparison inputs: a deterministic judge can return A>BA>B, B>CB>C, and C>AC>A on three different prompts. TCR's selection step addresses that cross-comparison inconsistency and can also operate on greedily decoded judgments. The paper contains no decoding ablation supporting a numerical CIR reduction.

Thank you for the detailed response and clarifications. The rebuttal addressed my concern about the motivation and clarifies the rationale for the proposed cycle-removal rule. I also acknowledge the existing evaluation with 16 candidates, which addresses part of my question about larger group sizes.

Meanwhile, my concerns about the strength and scope of the empirical contribution remain. The improvements over PREF, the most directly comparable baseline, are modest, and the evidence remains limited to the 1K-query training setting. Although TCR requires no additional judge calls over PREF, the requested throughput or wall-time measurements are still absent, leaving the practical performance–cost tradeoff unclear.

The remaining empirical and efficiency concerns leave me unconvinced that the practical benefit is sufficiently established. I therefore maintain my rating of 2.

Reviewer_WTNgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a genuine gap between pairwise LLM judging and group-based policy optimization.

Critical strengths

A key conceptual contribution is the equivalence observation: with matched tie conventions and group standardization, raw win-tie rates and original-graph net degrees yield identical standardized rewards (up to the stabilizer).

Critical weaknesses

The CIR/accuracy analysis is on RewardBench2 with human labels, and the training feedback (WildChat) has no ground truth.

Questions for authors

Can you include a cost-matched baseline?

What is the CIR on the actual WildChat training-distribution groups?

1 comment
AuthorsOpenReview ↗

Response to Reviewer WTNg

Thank you for recognizing the pairwise-to-group reward problem and the equivalence between raw win-tie rates and original-graph net degree. We address the training-label distinction and your two requested checks.

W1. Human labels in the diagnostic versus WildChat training

The RewardBench2 diagnostic uses human preferences to calculate judge accuracy. CIR itself counts cycles in the judge's pairwise graph and requires no human labels. WildChat training also needs no reference ranking: for each generated response group, TCR constructs the graph from AI judgments, retains edges consistent with the greedy order, and standardizes retained net-degree rewards for GRPO/GSPO. We do not use the RewardBench2 labels to train the policy.

The downstream comparison evaluates this reward choice separately from the labeled diagnostic. In Table 4, TCR/PREF Arena-Hard means under P2–P5 are 51.8/50.6, 51.4/50.2, 52.4/51.6, and 52.9/51.2, respectively. The within-prompt differences are +1.2, +1.2, +0.8, and +1.7 points. These reported policy results do not require any removed edge to be identified as a ground-truth error.

Q1. Cost-matched baseline

All-pairs PREF is the matched feedback-acquisition baseline for TCR. Given the same G candidates and comparison matrix, both use G(G−1)/2 judgments from the same pairwise judge; TCR selects edges before computing rewards and adds no judge calls or optimizer changes. At the aggregation level, the PREF/TCR comparison evaluates two ways to use the same number of pairwise judgments per group.

This matching does not extend to PW, LW, or repeated voting. They request feedback in different formats or quantities, and the paper reports no matched token-budget or wall-time study for those alternatives. Our efficiency claim is limited to reusing PREF's comparisons, rather than a measured end-to-end speedup.

Q2. CIR on actual WildChat training groups

The relevant quantity is the fraction of WildChat rollout groups whose pre-filter comparison graph contains a directed cycle. The paper does not report its numerator or denominator, so it cannot give a WildChat training CIR. Figure 1's RewardBench2 diagnostic uses G=4 and cannot stand in for this training-distribution measurement. The post-filter CIR is zero by construction and does not answer the question.

This distinction also bounds the mechanism claim. On an acyclic comparison graph, TCR retains every edge, and its group-standardized reward agrees with PREF under matched tie and stabilizer conventions. Thus the fraction of training groups where edge selection can change rewards is governed by the unreported pre-filter WildChat CIR. The existing downstream results show higher reported means in the tested settings; they do not establish how often training rewards changed or whether WildChat cycles represent judge mistakes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.