- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces one new axis of improving reasoning ability of LLMs based on GRPO methods, which the author refers to "catfish effect" by introducing one real or virtual reward signal conditionally involving in resampling process and advantage calculation. The author conducts different experiments to show the phenomenon that under this design, specific conditioning method by shifting curriculum can have better average score than that of vanilla GRPO and other baselines.
Critical strengths
- Involve the idea of Heterogeneous Competition is interesting and novel.
- The author analyzes the experiment under different models/datasets and make many ablations to make statement strong.
Critical weaknesses
-
The paper's writing is not clear from my perspective, and I cannot find all contents are coherent. Although the paper begins with "catfish effect", the method is basically one mechanism to decide when to resample the all wrong completions prompt and how to calculate the advantages if the competitor signal comes in. The author compares both real competitor and virtual competitor with different (p1,p2) configs. However, the most effective one is the one setting in the virtual case, and the downgrade performance for the real competitor is only given with hypothesis and no more clear analysis. That makes the algorithm is basically becoming one Bernoulli conditioning shifting algorithm which I don't think relevant to "catfish effect".
-
The central idea for why the algorithm works relates to papers such as [1] and [2], which states that negative samples always help the diversity, generalization and pass k metric , and the positive samples always help sharpening for more deterministic result which boost pass 1 metric. The curriculum 10 01 introduced in this paper is by using the virtual signal to control the weighting of negative and positive samples, so for my perspective it is doing the thing that can be explained by my cited works. The author does mention in the related work but the whole paper emphasizes its contribution to the comparison population - the algorithm with its analysis is not centering around how one different competitor can influence the training (to be specific, this step is not using random p1,p2, but should be using the true strong teacher's evaluation).
-
Typo of "..." in line 431.
[1]: NGRPO: Negative-enhanced Group Relative Policy Optimization [2]: Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards
Questions for authors
- Can you provide the curriculum result for 10 -> 00 which should be one important baseline?
- Can you provide pass k metrics for the summary table?
- Can you do further evidences and explanations to explain why real competitor has downgrade performances? I think that should be the key of this work to show its unique contribution.
5 comments
Response 2 to Reviewer MvXR
W2: Catfish is not reducible to sample polarity
“For my perspective it is doing the thing that can be explained by my cited works.”
The cited works explain one within-group consequence, while Catfish additionally governs which prompts are explored and how newly discovered responses are reinforced. NGRPO adds a fixed maximum-reward virtual sample so that an all-wrong group receives negative advantages, while sample-polarity analysis describes how positive and negative trajectories affect optimization once they already exist. Catfish instead uses an external 0/1 competitor outcome to determine whether an unsuccessful prompt receives further exploration. It does not optimize the original all-wrong group; a successful retry reinforces the student's newly discovered on-policy response.
The signal can also be placed separately on successful direct and retry groups and changed across training stages. Table 2 shows higher initial-group pass@4 before retries; Table 6 shows that retry-matched controls do not reproduce the gains; Table 13 isolates retry-branch shaping; and Tables 4–5 show that the same placement behaves differently as competitor capability changes. Sample polarity therefore explains some local update directions, but not competitor-conditioned exploration, branch- and stage-specific placement, or relative-capability dependence.
“The algorithm [...] is not centering around how one different competitor can influence the training [...] this step [...] should be using the true strong teacher's evaluation.”
Real and virtual Catfish share exactly the same student-side training mechanism. The competitor's answer never enters the student loss; only its binary outcome affects retry routing and advantage computation. Once this outcome is given, the downstream training operation is identical regardless of whether it was produced by a real model or sampled virtually.
A virtual Catfish outcome can therefore be viewed as a sample from the conditional outcome distribution of a real competitor with matching . This enables us to study how different competitor capability profiles affect student learning without constructing a separate real-model cache for every setting. A real model additionally retains prompt-specific competence, whereas the virtual construction matches the conditional outcome distribution.
We provide extensive evidence for both forms. Appendix Table 11 reports the four-student virtual comparison, with at least one configuration improving both evaluations for every student. Appendix C.3 evaluates 14 Qwen-3B and five Qwen-7B settings, with GSM8K improving at all of them; Table 15 reports gains at eight of nine settings for both placements on 2Wiki. These results complement the real evidence: every Catfish entry in Table 1 uses a real frozen competitor across four students and three seeds, while Tables 4–5 directly study competitor capability.
W3: Typographical correction
The typographical error identified in W3 has been corrected in the revised manuscript.
Q1: The requested 10→00 comparison
Table 13 reports 10→00, 11→01, and 10→01 across all four students and three seeds:
| Student | 10→00 | 10→01 |
|---|---|---|
| Qwen2.5-3B | 73.98/44.59 | 74.12/46.03 |
| Qwen2.5-7B | 83.55/57.62 | 84.10/57.40 |
| Llama-3B | 73.83/37.47 | 75.62/37.93 |
| Llama-8B | 81.65/41.16 | 83.45/42.57 |
10→01 outperforms 10→00 in seven of eight evaluations. Both schedules remove direct shaping after Epoch 1, but only 10→01 shapes newly successful retry groups. This supports relocating the competitive signal to successful retries rather than merely turning direct shaping off.
Q2: Initial-group pass@4
Table 2 reports the requested three-seed Llama-8B comparison:
| Method | First-half pass@4 | Epoch-1 pass@4 | Epoch-1 rollout reward |
|---|---|---|---|
| Vanilla GRPO | 93.15 | 93.66 | 80.63 |
| DAPO | 92.54 | 92.85 | 76.69 |
| RL-ZVP | 92.32 | 93.75 | 79.60 |
| LENS | 92.22 | 93.35 | 79.39 |
| AR3PO | 91.76 | 93.47 | 72.35 |
| Catfish10 | 94.01 | 94.81 | 82.93 |
All pass@4 values use only the initial four student responses, before retries, rollout expansion, filtering, or response reuse. Catfish therefore improves successful on-policy discovery rather than merely benefiting from additional attempts. The corresponding analysis is provided in Section 4.2.2, Early Successful Exploration.
Thank you for the response and the additional experiments. The paper claims that the method works because a stronger competitor is introduced. However, the virtual version replaces the competitor with a random Bernoulli outcome and still works, in some settings even better, and the response agrees that the relevant factor is not whether the competitor is real or virtual. So the competition story does not explain why the method works. What remains is extra retries and a reweighting of positive and negative samples, which needs more interpretation. Therefore the method still looks like an empirical trick to me. Although the idea is interesting, I will keep my score.
Response 1 to Reviewer MvXR
W1: The revised paper now follows one coherent Catfish mechanism
We appreciate this concern and have substantially reorganized the manuscript. We have uploaded the revised version, with substantive additions and changes marked in blue, and hope that it is now easier to follow. The revision centers on one scientific question: how does introducing a heterogeneous competitive reference into an otherwise homogeneous student group change exploration and group-relative learning?
The Catfish effect is the heterogeneous-competitor intervention itself, rather than any particular retry rule or Bernoulli setting. Catfish GRPO operationalizes this intervention through two channels: the competitor's scalar outcome determines which unsuccessful prompts receive further exploration, and it changes how successful and unsuccessful student responses are compared through group-relative advantage normalization. The competitor therefore perturbs the group's learning environment and stimulates successful on-policy exploration without providing an answer for imitation.
Real and virtual Catfish use the same training operator but generate the reference differently: the real variant preserves a frozen model's prompt-specific outcome, whereas the virtual variant samples from a specified conditional success profile. The placement, λ, stage-wise schedule, and (p1,p2) settings are complementary designs of this same intervention. They determine where the signal acts, how strongly it affects normalization, when its placement changes, and what reference capability it represents.
The experimental narrative follows the same progression:
The aggregate results support this mechanism beyond any single setting. In Table 1, real Catfish obtains the highest GSM8K mean for every student and the highest displayed mean in seven of eight model–evaluation combinations. In Appendix Table 11, every student has a virtual configuration that improves both evaluations over paired Vanilla.
The revised paper therefore proceeds from the heterogeneous-reference intervention, to its effect on early exploration, to its stage-dependent placement, and finally to reference quality, compute controls, and robustness.
W1 & Q3: Real-competitor degradation depends on relative capability
A competitor remains useful only while it maintains an informative capability difference relative to the evolving student. A calibrated virtual competitor preserves fixed conditional success probabilities by design. A frozen real competitor does not: as the student improves, its relative advantage decreases and its direct-shaping signal becomes less informative.
Table 4 compares the calibrated virtual reference, continued use of the 14B real reference, and replacement with a stronger 32B real reference in Epoch 2:
Under the calibrated virtual reference, 10 and 11 improve from 82.0/55.5 and 82.2/56.6 in Epoch 1 to 83.5/58.0 and 84.3/59.3 in Epoch 2. Under the fixed 14B reference, they instead decline to 78.9/49.2 and 78.7/50.7.
Replacing only the Epoch-2 reference with 32B restores 5.0/7.9 and 4.7/6.0 points, while direct-off 00/01 change by at most 0.2. A very weak virtual reference similarly harms direct-on configurations while affecting direct-off configurations much less (Table 5). The relevant factor is not whether the competitor is real or virtual, but whether it preserves an informative relative-capability gap as the student evolves. The full analysis appears in Section 4.2.4, Competitor Utility Depends on Relative Capability.