- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoEvoke, a framework for collecting multi-agent trajectories with more checking, feedback and repair behaviors. It introduces controlled perturbations, such as corrupting correct candidates or using weaker models for initial generation, together with reference-informed guidance to help agents identify and fix errors. The collected trajectories are converted into supervision for training judges, critics and verifiers, while keeping solving agents frozen. Experiments show much more corrective interactions during collection, and downstream improvements on ACPBench-Hard and SWE-bench Verified.
Critical strengths
- The motivation makes sense. When strong agents already solve most tasks correctly, natural rollouts contain very few useful examples about when and how to request repair. Simply collecting more trajectories may not help much. Creating recoverable errors is a reasonable solution.
- I also like the matched-data training experiment. Same tasks, same number of decision records and training updates, but quite different downstream results. This provides some real evidence that collection strategy matters. Keeping solvers frozen and removing oracle guidance during evaluation also makes the result more meaningful.
Critical weaknesses
My main concern is what actually contributes to the downstream improvement. Figure 3 shows huge increase in feedback, but this is somewhat expected. The method deliberately changes correct candidates into incorrect ones, and the checker has access to reference answers during collection. More feedback under this setting is not very surprising.
The more interesting evidence is the Natural-SFT vs Perturb-SFT comparison. However, Figure S8 shows that Natural-SFT contains almost only acceptance decisions, while Perturb-SFT has roughly half repair-routing decisions. So the training label distribution is completely different. Maybe the improvement mainly comes from exposing the judge to more negative candidates and teaching it to reject, rather than learning useful multi-agent collaboration from feedback-revision trajectories. A much simpler corruption-based classification or routing dataset could possibly achieve similar gains. Current experiments cannot distinguish these explanations.
Another concern is the contribution of the full collaboration framework. The SWE data uses Guide without Perturb, and verifier training is compared with untrained models. Interestingly, the patch-only verifier performs almost the same as the trajectory-based verifier. This makes me wonder how much additional value comes from the recorded collaboration process itself. Also, Appendix F.5 shows that the single-agent baseline clears previous reasoning context between invocations. This seems a relatively weak way to use the same computation budget. The multi-agent advantages in Figure 4 may partly come from better context retention and structured iteration. Hard to attribute these gains specifically to multi-agent collaboration.
Questions for authors
-
For the matched ACPBench-Hard experiment, can you compare with a judge trained on simple corrupted candidates and corresponding accept/repair labels, without collecting full feedback-revision trajectories? Ideally matching the label distribution.
-
How do Natural-SFT and Perturb-SFT differ in actual repair frequency, unnecessary repairs and model calls during evaluation? Does Perturb-SFT really make better decisions, or mainly requests more repair attempts?
-
For SWE verifier training, what happens if the training records are collected from ordinary unguided trajectories with the same task and supervision budget? This would help clarify the contribution of Guide.
Thanks for the detailed responses and additional experiments. They address my main concerns and strengthen the paper. I appreciate the authors' efforts and update my rating to 3.