- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TrapArena, a benchmark of 113 function- and repository-level tasks evaluating code-repair agents' robustness against plausible, runtime adversarial feedback. Across 12 models, trajectory-conditioned distractions significantly degraded repair success (up to 10.4 pp on SWE-bench and 37.2 pp on HumanEvalFix), while showing that behavioral susceptibility does not always correlate with actual performance loss.
Critical strengths
- The authors focus on a relatively important research question in the current coding agent setting: when distracting guidance is provided to coding agents, how will the coding agent's capability be affected?
- This paper presents a good observation: being more easily influenced in behavior does not equal greater damage to task performance. In the ablation of the 72B model, after adding curator memory, the repair loss on HumanEvalFix actually decreased.
Critical weaknesses
- The TRAPARENA benchmark constructed in the paper has a small number of samples, only 113 tasks, and they are basically constructed based on previously well-known coding agent benchmarks (e.g., SWE-bench Verified, HumanEvalFix) with only simple processing; therefore, the benchmark's contribution is relatively small.
- Many models used in the paper's experiments (e.g., deepseek-coder-v2, Codestral-22B) have relatively outdated capabilities, and results from more capable models should be added.
Questions for authors
see weeknesses
Response to Reviewer dwPY
Thank you for recognizing the importance of studying distracting guidance during code repair. We provide our response to the questions below.
W1. Benchmark size and contribution.
TrapArena’s main contribution is a dynamic evaluation environment for testing whether a coding agent can still repair a defect when a collaborator gives it misleading advice. In everyday debugging, a teammate may see a failing test and point to the wrong file. TrapArena can introduce such advice in response to the agent’s actions. We compare the agent’s repair success on the same defect with and without that advice, using the same hidden tests.
The 113 tasks from three benchmarks provide a common test set. We selected the SWE-bench Verified subset for baseline solvability so that a decline under feedback would be observable. Across ten unaided trials per task, MiniMax-M2’s pass rate was 23.5% on the 48 selected tasks and 0.9% on the other 147 candidates. We propose clarifying in §2 how users can bring their own tasks.
W2. Results for more capable models.
Table 1 includes 12 repair targets, with open-weight models up to 480B total parameters as well as GPT-4o. On the 48 SWE-bench Verified tasks, trajectory-conditioned feedback significantly reduced repair success for both Qwen3-235B-A22B and GPT-4o.
Separate existing runs with GLM-4.6 (FP8) on HumanEvalFix yielded a 96.7% pass rate without advice and 94.7% with prewritten feedback. To clarify how newer models can be tested, we propose documenting in §3.1 the interface provided in our anonymous code release.