acceptodds
Under review as a conference paper at ICLR 2027

The Catfish Effect: Heterogeneous Competition in Group-Relative Learning

Abstract

Learning systems do not evolve in isolation: their progress is shaped by the populations against which they compete. Yet most reinforcement learning methods focus on improving the learner itself, while largely treating the composition of its competitive population as fixed. This raises a broader question: can introducing a heterogeneous competitor reshape learning without providing demonstrations or direct supervision? We investigate this question through Catfish GRPO, a group-relative reinforcement learning framework that introduces a frozen, moderately stronger model as a competitor rather than a teacher. The Catfish contributes only a scalar task outcome to reshape relative learning signals and guide rollout allocation; its generations, tokens, log probabilities, and parameters never enter the student loss. In our main comparison, Catfish GRPO exceeds the strongest evaluated baseline by up to 2.5 percentage points in-domain and 3.2 percentage points out-of-domain. Beyond performance, we study how heterogeneous competitors alter learning dynamics. Catfish encourages greater behavioral diversity early in training, increasing the chance that the student discovers correct on-policy trajectories on difficult problems and expanding the learning opportunities available in later stages. The quality of the competitor also plays a critical role: weaker or poorly matched competitors can substantially diminish the benefit, while stronger competitors are not necessarily better. Together, these findings position Catfish GRPO not only as an RL method, but also as a controlled testbed for studying how population composition and heterogeneous competition shape learning and co-evolution.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning, group-relative policy optimization, competitive learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MvXROpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper introduces one new axis of improving reasoning ability of LLMs based on GRPO methods, which the author refers to "catfish effect" by introducing one real or virtual reward signal conditionally involving in resampling process and advantage calculation. The author conducts different experiments to show the phenomenon that under this design, specific conditioning method by shifting curriculum can have better average score than that of vanilla GRPO and other baselines.

Critical strengths
  1. Involve the idea of Heterogeneous Competition is interesting and novel.
  2. The author analyzes the experiment under different models/datasets and make many ablations to make statement strong.
Critical weaknesses
  1. The paper's writing is not clear from my perspective, and I cannot find all contents are coherent. Although the paper begins with "catfish effect", the method is basically one mechanism to decide when to resample the all wrong completions prompt and how to calculate the advantages if the competitor signal comes in. The author compares both real competitor and virtual competitor with different (p1,p2) configs. However, the most effective one is the one setting in the virtual case, and the downgrade performance for the real competitor is only given with hypothesis and no more clear analysis. That makes the algorithm is basically becoming one Bernoulli conditioning shifting algorithm which I don't think relevant to "catfish effect".

  2. The central idea for why the algorithm works relates to papers such as [1] and [2], which states that negative samples always help the diversity, generalization and pass k metric , and the positive samples always help sharpening for more deterministic result which boost pass 1 metric. The curriculum 10 01 introduced in this paper is by using the virtual signal to control the weighting of negative and positive samples, so for my perspective it is doing the thing that can be explained by my cited works. The author does mention in the related work but the whole paper emphasizes its contribution to the comparison population - the algorithm with its analysis is not centering around how one different competitor can influence the training (to be specific, this step is not using random p1,p2, but should be using the true strong teacher's evaluation).

  3. Typo of "..." in line 431.

[1]: NGRPO: Negative-enhanced Group Relative Policy Optimization [2]: Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards

Questions for authors
  1. Can you provide the curriculum result for 10 -> 00 which should be one important baseline?
  2. Can you provide pass k metrics for the summary table?
  3. Can you do further evidences and explanations to explain why real competitor has downgrade performances? I think that should be the key of this work to show its unique contribution.
5 comments
AuthorsOpenReview ↗

Response 1 to Reviewer MvXR

W1: The revised paper now follows one coherent Catfish mechanism

“I cannot find all contents are coherent.”
“The method is basically one mechanism to decide when to resample [...] and how to calculate the advantages [...] I don't think [this is] relevant to the ‘catfish effect’.”

We appreciate this concern and have substantially reorganized the manuscript. We have uploaded the revised version, with substantive additions and changes marked in blue, and hope that it is now easier to follow. The revision centers on one scientific question: how does introducing a heterogeneous competitive reference into an otherwise homogeneous student group change exploration and group-relative learning?

The Catfish effect is the heterogeneous-competitor intervention itself, rather than any particular retry rule or Bernoulli setting. Catfish GRPO operationalizes this intervention through two channels: the competitor's scalar outcome determines which unsuccessful prompts receive further exploration, and it changes how successful and unsuccessful student responses are compared through group-relative advantage normalization. The competitor therefore perturbs the group's learning environment and stimulates successful on-policy exploration without providing an answer for imitation.

Real and virtual Catfish use the same training operator but generate the reference differently: the real variant preserves a frozen model's prompt-specific outcome, whereas the virtual variant samples from a specified conditional success profile. The placement, λ\lambda, stage-wise schedule, and (p1,p2)(p_1,p_2) settings are complementary designs of this same intervention. They determine where the signal acts, how strongly it affects normalization, when its placement changes, and what reference capability it represents.

The experimental narrative follows the same progression:

  1. Effectiveness: Table 1 evaluates real Catfish across four students and three seeds, while Appendix Table 11 reports the corresponding virtual-Catfish comparison.
  2. Early exploration: Table 2 and Figure 3 test successful on-policy discovery using the initial student group, before additional retries.
  3. Stage dependence: Table 3, Figure 3, and Appendix Table 13 examine how effective signal placement changes during training.
  4. Reference quality: Tables 4–5 study how the student–competitor capability relationship governs continued learning.
  5. Controls and robustness: Tables 6–7 provide matched-sampling and model-FLOPs comparisons, while Appendix C.3 varies the virtual reference over broad probability neighborhoods.

The aggregate results support this mechanism beyond any single setting. In Table 1, real Catfish obtains the highest GSM8K mean for every student and the highest displayed mean in seven of eight model–evaluation combinations. In Appendix Table 11, every student has a virtual configuration that improves both evaluations over paired Vanilla.

The revised paper therefore proceeds from the heterogeneous-reference intervention, to its effect on early exploration, to its stage-dependent placement, and finally to reference quality, compute controls, and robustness.

W1 & Q3: Real-competitor degradation depends on relative capability

“Can you do further evidences and explanations to explain why real competitor has downgrade performances?”

A competitor remains useful only while it maintains an informative capability difference relative to the evolving student. A calibrated virtual competitor preserves fixed conditional success probabilities by design. A frozen real competitor does not: as the student improves, its relative advantage decreases and its direct-shaping signal becomes less informative.

Table 4 compares the calibrated virtual reference, continued use of the 14B real reference, and replacement with a stronger 32B real reference in Epoch 2:

PlacementVirtual E2Real 14B E2Real 32B E2
1083.5/58.078.9/49.283.9/57.1
1184.3/59.378.7/50.783.4/56.7

Under the calibrated virtual reference, 10 and 11 improve from 82.0/55.5 and 82.2/56.6 in Epoch 1 to 83.5/58.0 and 84.3/59.3 in Epoch 2. Under the fixed 14B reference, they instead decline to 78.9/49.2 and 78.7/50.7.

Replacing only the Epoch-2 reference with 32B restores 5.0/7.9 and 4.7/6.0 points, while direct-off 00/01 change by at most 0.2. A very weak virtual reference similarly harms direct-on configurations while affecting direct-off configurations much less (Table 5). The relevant factor is not whether the competitor is real or virtual, but whether it preserves an informative relative-capability gap as the student evolves. The full analysis appears in Section 4.2.4, Competitor Utility Depends on Relative Capability.

AuthorsOpenReview ↗

Response 2 to Reviewer MvXR

W2: Catfish is not reducible to sample polarity

“For my perspective it is doing the thing that can be explained by my cited works.”

The cited works explain one within-group consequence, while Catfish additionally governs which prompts are explored and how newly discovered responses are reinforced. NGRPO adds a fixed maximum-reward virtual sample so that an all-wrong group receives negative advantages, while sample-polarity analysis describes how positive and negative trajectories affect optimization once they already exist. Catfish instead uses an external 0/1 competitor outcome to determine whether an unsuccessful prompt receives further exploration. It does not optimize the original all-wrong group; a successful retry reinforces the student's newly discovered on-policy response.

The signal can also be placed separately on successful direct and retry groups and changed across training stages. Table 2 shows higher initial-group pass@4 before retries; Table 6 shows that retry-matched controls do not reproduce the gains; Table 13 isolates retry-branch shaping; and Tables 4–5 show that the same placement behaves differently as competitor capability changes. Sample polarity therefore explains some local update directions, but not competitor-conditioned exploration, branch- and stage-specific placement, or relative-capability dependence.

“The algorithm [...] is not centering around how one different competitor can influence the training [...] this step [...] should be using the true strong teacher's evaluation.”

Real and virtual Catfish share exactly the same student-side training mechanism. The competitor's answer never enters the student loss; only its binary outcome affects retry routing and advantage computation. Once this outcome is given, the downstream training operation is identical regardless of whether it was produced by a real model or sampled virtually.

A virtual Catfish outcome can therefore be viewed as a sample from the conditional outcome distribution of a real competitor with matching (p1,p2)(p_1,p_2). This enables us to study how different competitor capability profiles affect student learning without constructing a separate real-model cache for every setting. A real model additionally retains prompt-specific competence, whereas the virtual construction matches the conditional outcome distribution.

We provide extensive evidence for both forms. Appendix Table 11 reports the four-student virtual comparison, with at least one configuration improving both evaluations for every student. Appendix C.3 evaluates 14 Qwen-3B and five Qwen-7B settings, with GSM8K improving at all of them; Table 15 reports gains at eight of nine settings for both placements on 2Wiki. These results complement the real evidence: every Catfish entry in Table 1 uses a real frozen competitor across four students and three seeds, while Tables 4–5 directly study competitor capability.

W3: Typographical correction

The typographical error identified in W3 has been corrected in the revised manuscript.

Q1: The requested 10→00 comparison

Table 13 reports 10→00, 11→01, and 10→01 across all four students and three seeds:

Student10→0010→01
Qwen2.5-3B73.98/44.5974.12/46.03
Qwen2.5-7B83.55/57.6284.10/57.40
Llama-3B73.83/37.4775.62/37.93
Llama-8B81.65/41.1683.45/42.57

10→01 outperforms 10→00 in seven of eight evaluations. Both schedules remove direct shaping after Epoch 1, but only 10→01 shapes newly successful retry groups. This supports relocating the competitive signal to successful retries rather than merely turning direct shaping off.

Q2: Initial-group pass@4

Table 2 reports the requested three-seed Llama-8B comparison:

MethodFirst-half pass@4Epoch-1 pass@4Epoch-1 rollout reward
Vanilla GRPO93.1593.6680.63
DAPO92.5492.8576.69
RL-ZVP92.3293.7579.60
LENS92.2293.3579.39
AR3PO91.7693.4772.35
Catfish1094.0194.8182.93

All pass@4 values use only the initial four student responses, before retries, rollout expansion, filtering, or response reuse. Catfish therefore improves successful on-policy discovery rather than merely benefiting from additional attempts. The corresponding analysis is provided in Section 4.2.2, Early Successful Exploration.

Thank you for the response and the additional experiments. The paper claims that the method works because a stronger competitor is introduced. However, the virtual version replaces the competitor with a random Bernoulli outcome and still works, in some settings even better, and the response agrees that the relevant factor is not whether the competitor is real or virtual. So the competition story does not explain why the method works. What remains is extra retries and a reweighting of positive and negative samples, which needs more interpretation. Therefore the method still looks like an empirical trick to me. Although the idea is interesting, I will keep my score.

Reviewer_fkDaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Catfish GRPO, which is with a very novel method name using a real or virtual competitor’s binary outcome to alter advantage normalization and allocate extra student rollouts. The student never trains on the competitor’s answer. The paper studies where to apply this signal and whether to change its placement during training. Results on Qwen and Llama models show gains in some settings, but also substantial sensitivity to the reference and update schedule. The four placement settings distinguish shaping on the initial successful group from shaping on a successful retry group. Even the 00 setting still uses competitor-guided retries, so it is not ordinary GRPO. The virtual reference removes competitor generation but also loses prompt-specific information about which failures are recoverable. These choices create several distinct mechanisms that need to be separated when interpreting the gains.

Critical strengths
  • It seems like that using only a scalar competitor outcome is a useful way to separate this idea from answer imitation. The four placement settings also make retry allocation and advantage shaping easier to study independently.
  • The paper reports meaningful failures as well as gains. Fixed direct shaping can deteriorate late in training, and the best schedule does not win on every task. An independent three-seed control improves GSM8K from 72.38±0.43 to 74.45±0.20, which supports taking the effect seriously. The real-versus-virtual comparison is especially interesting because it tests how much of the effect requires a stronger model’s actual per-question competence. It also shows why “use a stronger competitor” is not a complete prescription: the same shaping placement can help early and hurt later. This is a worthwhile observation even if the current causal account is incomplete. This makes the paper's findings much more signifigant.
Critical weaknesses
  • The distinction from existing rollout and advantage-shaping methods is narrower than the competition framing suggests. Exisiting works already retries unsuccessful prompts and includes an option that recomputes group advantages using a reused correct response while stopping its gradient and updating only current-policy responses. Thus excluding the reference response from the loss is not by itself new. Catfish differs by using a frozen external or virtual outcome, appending it to the normalization group, and shaping successful direct/retry groups rather than training the initial all-wrong group. For nondegenerate groups, its shaped advantage is an affine rescaling and shift of the original GRPO advantage. This can be useful, but the paper needs to establish what the competitor interpretation adds beyond a particular reference-conditioned advantage rule and retry schedule. A frozen model or conditional Bernoulli variable also does not demonstrate reciprocal co-evolution.
  • The comparisons somehow do not isolate that remaining contribution. It is unclear whether the gains come from the competitor signal or from giving the student more chances to find a correct answer. Catfish changes both the training signal and the sampling process: the reference outcome affects advantage normalization and determines whether failed prompts receive additional rollouts. Matching the number of base prompts therefore does not match the amount of generated training data or compute. The reported gains over ordinary GRPO could partly reflect these extra attempts; the experiments do not establish how much benefit remains when GRPO receives the same total generation budget. The virtual competitor makes this question especially important, since it selects retries using conditional probabilities rather than knowing which particular questions are recoverable.
Questions for authors
  • How much of the improvement actually requires a competitor? The virtual version only uses whether the student group contains a correct answer and two probabilities. What evidence distinguishes the proposed competition mechanism from adjusting advantage normalization and spending more samples on failed prompts?
  • What does knowing that a real competitor solved a particular question add? If its outcomes were shuffled within the successful and all-wrong student groups, preserving the conditional success rates, would the benefit remain?
  • Does Catfish still improve over ordinary GRPO when both receive the same total generation budget, including retries and real-competitor inference or cache construction? Matching base prompts does not settle this question. It would be useful to know whether the advantage comes from using extra samples more effectively, rather than simply generating more of them.
  • Just for discussion, why does direct shaping become harmful later in training, including with an uncorrupted fixed real reference? The corrupted-cache experiment shows sensitivity to unreliable signals, but does not by itself explain this separate failure. What evidence supports a changing student–competitor relationship as the cause, rather than an overly aggressive advantage shift or a change in the mix of training examples? This distinction matters for whether the proposed schedule addresses the cause of the instability.
6 comments
AuthorsOpenReview ↗

Response 1 to Reviewer fkDa

Thank you for recognizing the value of the real/virtual comparison and stage-dependent findings. We first clarify what distinguishes Catfish and then address the requested conditional randomization.

W1: Catfish is mechanistically distinct and empirically stronger

Catfish differs from existing rollout and shaping methods in how it uses an external competitive outcome. The response-reuse mechanism highlighted in the review is AR3PO's optional branch, which we implement directly. AR3PO reuses a historical correct response as a fixed anchor, NGRPO adds a maximum-reward virtual sample, and RL-ZVP/LENS use entropy or confidence. Catfish instead uses a real or virtual competitor outcome both to route exploration and to shape subsequent student-only updates. Thus, the competitor changes not only within-group advantages but also which prompts are explored.

The four placements, stage-wise schedules, and real/virtual references jointly determine where, when, and with what conditional strength the competitive signal acts.

These distinctions yield substantial gains over the most closely related baseline. Qwen-7B Catfish reaches 84.1/57.4 versus AR3PO's 81.8/50.7 on GSM8K/MATH-500 (Table 1). On Llama-8B, initial-group pass@4 reaches 94.81% versus AR3PO's 93.47% before retries or response reuse are counted (Table 2). Catfish consequently supports both more successful on-policy discovery and stronger final task performance. We accordingly replace “co-evolution” with “group-relative learning dynamics.”

Q2: Virtual Catfish implements the requested conditional randomization

The anchor virtual Catfish is the distributional counterpart of shuffling real outcomes within the two student-group states. Conditioned on whether the student group contains a correct answer, it samples from p1p_1 or p2p_2 independently of prompt identity. It therefore removes prompt-specific correspondence while preserving the two conditional success rates requested in the review.

The benefits remain stable as these rates vary:

SettingProfilesp1p_1 spanp2p_2 spanResult
Qwen2.5-3B140.60–0.950.10–0.6514/14 GSM8K gains; 11/14 MATH gains + 1 tie
Qwen2.5-7B50.85–0.950.25–0.705/5 gains on both evaluations
2Wiki, Catfish1090.70–0.900.20–0.408/9 above Vanilla
2Wiki, Catfish1190.70–0.900.20–0.408/9 above Vanilla

Appendix Table 11 also reports a virtual configuration that improves both GSM8K and MATH-500 for every student. Thus, Virtual Catfish does not require per-question competitor knowledge or one precise (p1,p2)(p_1,p_2) anchor; its effectiveness persists across broad conditional-success profiles.

AuthorsOpenReview ↗

Response 2 to Reviewer fkDa

Following the mechanism and conditional-randomization results above, we report matched-sampling controls, exact model-compute accounting, and interventions probing late-stage degradation.

W2 & Q1: Catfish outperforms retry-matched student-only controls

Table 6 compares Catfish with random and failure-targeted adaptive retries. Each paired run matches the retry count, depth, temperatures, and total additional student completions; both controls retain student-only GRPO advantages.

StudentRandom retryAdaptive retryCatfish
Qwen2.5-3B71.1/41.370.2/41.774.1/46.0
Qwen2.5-7B80.9/56.281.7/55.484.1/57.4
Llama-3B72.2/37.273.4/37.575.6/37.9
Llama-8B80.4/38.781.2/40.483.5/42.6

Catfish improves both evaluations over both controls for every student, gaining 2.2–3.0 GSM8K points over the stronger retry control. Thus, matched additional sampling alone does not reproduce the gains from competitor-conditioned routing and shaping.

Table 13 further distinguishes turning direct shaping off from relocating the signal to successful retries:

Student10→0010\rightarrow0010→0110\rightarrow01
Qwen2.5-3B73.98/44.5974.12/46.03
Qwen2.5-7B83.55/57.6284.10/57.40
Llama-3B73.83/37.4775.62/37.93
Llama-8B81.65/41.1683.45/42.57

10→0110\rightarrow01 wins seven of eight evaluations. Both schedules remove direct shaping after Epoch 1, but only 10→0110\rightarrow01 shapes newly successful retry groups.

Q3: Gains hold at comparable model compute

Table 7 accounts for generation, actor forward/backward and activation recomputation, reference-policy forward compute, and the reusable competitor cache. Actual generated lengths are included in generation FLOPs.

StudentOnline FLOPs vs. VanillaTotal FLOPs, cache amortized over 3 seeds
Qwen2.5-3B0.982×1.016×
Qwen2.5-7B0.943×0.958×
Llama-3B0.967×0.999×
Llama-8B0.957×0.971×

Shorter student generations and lower actor-update and reference-forward costs offset the retry overhead. Three of four total-compute ratios remain at or below Vanilla; Qwen-3B is 1.6% higher. Appendix A.7 reports approximately 438k competitor-generated tokens and a one-time cache cost of 25.53 PFLOP.

Q4: Direct shaping depends on the evolving student–competitor relationship

Effective Catfish learning requires an informative relative-capability relationship between the competitor and the student. With a fixed real competitor, this relationship changes as the student improves: the competitor's relative advantage narrows, and its original competitive signal may become less informative. A calibrated virtual Catfish, in contrast, maintains its specified conditional success probabilities (p1,p2)(p_1,p_2) throughout training.

Table 4 provides direct evidence through both virtual and real competitor interventions:

PlacementVirtual E2Real 14B E2Real 32B E2
1083.5/58.078.9/49.283.9/57.1
1184.3/59.378.7/50.783.4/56.7

Under the calibrated virtual reference, direct-on placements continue improving from Epoch 1 to Epoch 2 (10: 82.0/55.5 → 83.5/58.0; 11: 82.2/56.6 → 84.3/59.3). Under the fixed real 14B reference, both deteriorate instead.

Crucially, replacing only the Epoch-2 real competitor with 32B reverses this degradation without changing the shaping rule. Placements 10 and 11 recover 5.0/7.9 and 4.7/6.0 points, respectively, while direct-off 00/01 change by at most 0.2 points.

Table 5 further supports this explanation: a very weak virtual reference substantially harms direct-on placements, whereas direct-off placements are much less affected.

Together, these findings show that the effectiveness of the same advantage-shaping mechanism depends on the competitor's relative capability as the student evolves. Maintaining an informative competitive reference, rather than simply applying stronger shaping, is therefore central to continued learning (§4.2.4).

Substantive revisions are marked in blue in the uploaded manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer fkDa,

Thank you again for your time and effort in reviewing our submission.

As the discussion period will end in 24 hours, we hope you can review our rebuttal and share your feedback on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions or issues.

Thank you for your time and consideration.

Reviewer_AwHsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Catfish GRPO, which uses a frozen stronger model’s binary task outcome to guide additional student sampling and optionally modify group-relative advantage normalization. A virtual variant replaces the external model’s outcome with a conditional Bernoulli variable. Experiments cover mathematical reasoning and question answering, with ablations on signal placement, competitor quality, and stage-wise scheduling.

Critical strengths
  • The method is simple and separates rollout allocation from advantage shaping. The placement ablations provide useful evidence that these interventions have different effects and can become harmful when applied throughout training.
  • The virtual competitor is an interesting observation: improvements can be obtained without using external model responses as training targets or performing competitor inference during training.
Critical weaknesses
  • The comparison does not isolate the proposed mechanism from additional computation. The experiments match base-prompt exposure rather than total rollout tokens or compute. Catfish can generate up to three additional student rollout groups, while the real variant also requires competitor generation and verification to construct its cache. A compute-matched resampling baseline with the same temperatures and update rules is missing. In particular, virtual Catfish with placement 00 amounts to randomly selecting all-wrong groups for additional sampling. Consequently, the results do not establish whether heterogeneous competition provides benefits beyond additional sampling and advantage shaping.
  • The performance advantage over strong baselines is inconsistent and insufficiently validated. Several improvements in Table 1 are marginal: Qwen-7B gains only 0.1 percentage points on GSM8K and ties on MATH, while Llama-3B remains below DAPO on GSM8K. The broad comparison uses a single seed, and several baselines are modified or incomplete implementations. The separate three-seed experiment supports improvement over Vanilla GRPO in one setting, but does not establish reliable superiority over stronger baselines. Moreover, the exploration explanation lacks a Vanilla pass@4 comparison and direct behavioral-diversity measurements.
  • The contribution of stronger-model information is not adequately isolated. The real Catfish uses prompt-specific outcomes from a stronger model, while the virtual variant uses conditional success probabilities calibrated from student–teacher interactions, explicitly so in the Qwen-3B experiments. Thus, removing teacher inference during training does not eliminate dependence on teacher-derived information. Without compute-matched resampling controls and comparisons against teacher-independent probability settings, the experiments cannot disentangle the benefits of teacher-derived information, additional sampling, and advantage shaping.
Questions for authors
  • Can the authors report total student generation tokens, competitor cache construction costs, and performance against direct resampling under a matched total compute budget?
  • Under identical retry temperatures, sample selection, and weighting, how much improvement comes from real competitor-based selection versus random selection, and how much comes from advantage shaping?
  • Are the gains over the strongest baselines reproducible across seeds, particularly for comparisons differing by less than one percentage point?
4 comments
AuthorsOpenReview ↗

Response 1 to Reviewer AwHs

Thank you for recognizing the separation between rollout allocation and advantage shaping. We first report the matched-budget and compute analyses; the complete baseline and robustness results follow in a second comment.

W1 & Q1–Q2: Stronger learning at matched sampling and comparable compute

1) Matched student-rollout budgets

For each student and seed, the random- and adaptive-retry controls match Catfish's per-step retry count, depth, temperatures, and total additional completions. Random retry allocates this budget uniformly, whereas adaptive retry targets initial all-wrong groups. Both use student-only GRPO advantages (Appendix A.6).

StudentRandom retryAdaptive retryCatfish
Qwen2.5-3B71.1/41.370.2/41.774.1/46.0
Qwen2.5-7B80.9/56.281.7/55.484.1/57.4
Llama-3B72.2/37.273.4/37.575.6/37.9
Llama-8B80.4/38.781.2/40.483.5/42.6

Catfish improves both evaluations over both controls for every student, gaining 2.2–3.0 GSM8K points over the stronger retry control. Competitor-conditioned routing and shaping therefore use the matched additional-sampling budget more effectively.

2) Retry-branch shaping

Table 13 compares turning direct shaping off with relocating the signal to successful retries:

Student10→0010→01
Qwen2.5-3B73.98/44.5974.12/46.03
Qwen2.5-7B83.55/57.6284.10/57.40
Llama-3B73.83/37.4775.62/37.93
Llama-8B81.65/41.1683.45/42.57

10→01 outperforms 10→00 in seven of eight evaluations. Both schedules remove direct shaping after Epoch 1, while 10→01 additionally shapes newly successful retry groups. This isolates the benefit of retaining the competitive signal on the retry branch.

3) Exact model-FLOPs accounting

Because response lengths differ, we account for actual generated lengths in the generation-FLOPs component rather than treating every completion as equally costly. Table 7 also includes actor forward/backward and activation recomputation, and reference-policy forward compute. Component columns report three-seed mean absolute PFLOP differences relative to Vanilla.

StudentRetry overheadGeneration savingUpdate/forward savingOnline FLOPs+ cache/3 seeds
Qwen2.5-3B+1.14 PF−1.25 PF−6.23 PF0.982×1.016×
Qwen2.5-7B+1.73 PF−5.36 PF−26.29 PF0.943×0.958×
Llama-3B+0.92 PF−1.36 PF−6.53 PF0.967×0.999×
Llama-8B+0.79 PF−4.45 PF−21.91 PF0.957×0.971×

Shorter generations and lower actor-update and reference-forward costs offset the retry overhead. Online compute is 0.943–0.982× Vanilla. After amortizing the reusable cache across three seeds, three of four settings remain at or below Vanilla; Qwen-3B is 1.6% higher. Appendix A.7 reports approximately 438k competitor-generated tokens and a one-time cache cost of 25.53 PFLOP.

AuthorsOpenReview ↗

Response 2 to Reviewer AwHs

Following the matched-budget results above, we report the complete three-seed comparison, early-exploration statistics, and virtual-reference sensitivity analysis.

W2 & Q3: Complete baseline implementations and three-seed comparisons

1) Broad three-seed comparison

We completed the baseline implementations and reran every method with three seeds under a shared training and evaluation protocol. Appendix A.5 details the implemented mechanisms, including AR3PO response reuse and RL-ZVP entropy-guided advantages.

The complete non-Catfish comparison is:

StudentVanillaDAPORL-ZVPLENSAR3PO
Qwen2.5-3B71.9/42.172.6/41.971.3/42.171.0/41.171.1/41.5
Qwen2.5-7B80.6/57.280.3/52.575.1/43.778.4/48.281.8/50.7
Llama-3B73.5/36.373.6/36.172.4/37.769.3/36.370.5/38.3
Llama-8B82.1/39.681.9/40.980.5/38.280.4/38.780.6/39.2

The corresponding real-Catfish results are:

StudentCatfish0*Catfish1*Catfishsw
Qwen2.5-3B73.4/45.273.5/44.874.1/46.0
Qwen2.5-7B83.2/57.178.8/50.084.1/57.4
Llama-3B74.9/38.174.4/38.275.6/37.9
Llama-8B82.3/41.082.8/39.583.5/42.6

Catfish achieves the highest displayed mean in seven of eight model–evaluation combinations. Its GSM8K gains over the strongest non-Catfish baseline are +1.5, +2.3, +2.0, and +1.4 points for Qwen-3B, Qwen-7B, Llama-3B, and Llama-8B, respectively. The largest MATH-500 gain is +3.9 points on Qwen-3B.

2) Early successful on-policy exploration

Table 2 reports the three-seed Llama-8B initial-group comparison:

MethodFirst-half pass@4Epoch-1 pass@4Epoch-1 rollout reward
Vanilla GRPO93.1593.6680.63
DAPO92.5492.8576.69
RL-ZVP92.3293.7579.60
LENS92.2293.3579.39
AR3PO91.7693.4772.35
Catfish1094.0194.8182.93

All pass@4 values use only the initial four student responses, before retries, rollout expansion, filtering, or response reuse. Catfish therefore improves successful on-policy discovery at the same initial group size.

W3: Robustness across wide (p1,p2)(p_1,p_2) ranges

Virtual Catfish remains effective across substantially different conditional-success profiles. For Qwen-3B, the 14 tested profiles span p1=0.60p_1=0.60–0.950.95 and p2=0.10p_2=0.10–0.650.65. For Qwen-7B, the five tested profiles span p1=0.85p_1=0.85–0.950.95 and p2=0.25p_2=0.25–0.700.70.

StudentTested profilesp1p_1 spanp2p_2 spanGSM8K above VanillaMATH-500 above Vanilla
Qwen2.5-3B140.60–0.950.10–0.6514/1411/14, plus one tie
Qwen2.5-7B50.85–0.950.25–0.705/55/5

For example, Qwen-7B reaches 83.47/60.38 at (0.95,0.25)(0.95,0.25), compared with 81.05/54.45 for paired Vanilla (Table 14). This profile has the same p1p_1 but a substantially lower p2p_2 than the near-anchor (0.95,0.70)(0.95,0.70) setting, yet improves both evaluations.

The effect also transfers to a full 3×33\times3 grid on 2Wiki, with p1∈{0.70,0.80,0.90}p_1\in\{0.70,0.80,0.90\} and p2∈{0.20,0.30,0.40}p_2\in\{0.20,0.30,0.40\}:

PlacementTested gridp1p_1 spanp2p_2 spanAccuracy rangeMean gain vs. VanillaAbove Vanilla
Catfish103×33\times30.70–0.900.20–0.4063.87–66.21+0.768/9
Catfish113×33\times30.70–0.900.20–0.4064.26–65.92+0.568/9

Both placements outperform Vanilla at eight of nine settings; the only exceptions occur in the weak p1=0.70p_1=0.70 region. Together, the GSM8K/MATH-500 and 2Wiki sweeps show that useful competitive feedback persists across substantial changes in both conditional probabilities rather than requiring one exact calibrated anchor.

We have uploaded the revised manuscript, with substantive additions and revisions marked in blue.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer AwHs,

Thank you again for your time and effort in reviewing our submission.

As the discussion period will end in 24 hours, we hope you can review our rebuttal and share your feedback on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions or issues.

Thank you for your time and consideration.

Dear Reviewer AwHs,

Thank you again for your time and effort in reviewing our submission.

As the discussion period will end in 3 hours, we hope you can review our rebuttal and share your feedback on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions or issues.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

Dear Area Chair and Reviewers,

Thank you for your feedback and the discussion so far. We have uploaded a revised manuscript incorporating the additional experiments and clarifications presented in our responses. The main updates are summarized below, with substantive additions and changes marked in blue.

1. Matched-sampling controls and model-compute accounting

We added random- and adaptive-retry controls matched to Catfish in retry count, depth, temperatures, and total additional student completions. Catfish outperforms both controls on both evaluations for all four students. On GSM8K, it exceeds the better-performing retry control for each student by 2.2–3.0 percentage points (Table 6).

We also added a detailed computational-cost analysis. Although retries introduce additional computation, shorter student responses and lower actor-update and reference-forward costs offset this overhead. Real Catfish uses 0.943–0.982× Vanilla’s online model FLOPs and 0.958–1.016× total model FLOPs after amortizing the reusable competitor cache across three seeds (Table 7).

We additionally completed the baseline implementations and reran the full comparison with three seeds under a shared training and evaluation protocol. The updated results show Catfish’s overall performance advantage across all four student models (Table 1).

2. New evidence for the competitive mechanism

We expanded the experiments and analyses to connect Catfish’s benefits to successful exploration, stage-dependent shaping, and competitor strength:

  • Exploration and placement. New initial-group comparisons show that Catfish raises Llama-8B Epoch-1 pass@4 from 93.66% to 94.81%, before additional retries are counted (Table 2). The requested 10→0010\rightarrow00 comparison further shows that retaining shaping on successful retries improves seven of eight evaluations, supporting its continued contribution after direct shaping is removed (Appendix Table 13).

  • Competitor strength over training. The new 14B-to-32B intervention restores late-stage performance from the same Epoch-1 checkpoints without changing the shaping placement or rule (Table 4). Together with calibrated virtual references and weak-reference controls (Table 5), these experiments show why maintaining an informative competitive reference matters as the student improves.

  • Robustness across competitor profiles. Added virtual-reference sweeps show GSM8K gains across all 14 Qwen-3B and five Qwen-7B settings; both tested placements also improve on 2Wiki at eight of nine settings (Appendix C.3). The benefits therefore extend beyond one calibrated probability pair.

3. Manuscript-wide reorganization around this mechanism

We reorganized the manuscript around how a heterogeneous competitor guides exploration and shapes student updates. Signal placement, training schedules, and real/virtual references are presented as complementary design dimensions of this mechanism, with the experimental narrative following their roles in exploration, continued learning, and robustness.

We hope these updates make the experimental evidence and the role of heterogeneous competition clearer. Thank you again for your feedback and suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.