acceptodds
Desk rejected at ICLR 2027

TrapArena: Evaluating Code Repair Agents Under Misleading Collaborative Feedback

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents rely on feedback from users and other agents to guide software repair, yet this guidance can be mistaken or deliberately misleading. Existing repair benchmarks primarily assess whether agents can resolve defects, leaving their ability to maintain repair success under misleading guidance insufficiently understood. We introduce TrapArena, a benchmark for evaluating code repair agents under misleading collaborative feedback. TrapArena augments repair tasks with plausible but incorrect advice delivered during execution, without allowing the attacker to modify the repository or alter tool outputs. It separates the information available to the agent from hidden evaluation criteria and measures changes in repair success against a baseline without misleading feedback. The benchmark supports both scripted and trajectory-conditioned attacks, with an optional curator that distills target reactions into persistent notes to inform subsequent attacks. Across 113 function-level and repository-level repair tasks, our main evaluation of 12 target models finds that trajectory-conditioned attacks significantly reduce repair success for eight targets on at least one task source, with drops of up to 10.4 percentage points on SWE-bench and 37.2 points on HumanEvalFix. However, stronger behavioral influence does not necessarily produce greater repair degradation. In a controlled ablation on a 72B target, curator memory increases measured susceptibility while leaving repair loss unchanged or smaller. These findings demonstrate that misleading collaborative feedback can undermine software repair and highlight the importance of evaluating its impact on task outcomes rather than inferring harm from apparent compliance. The code, tutorial, and dataset are available at http://traparena.pages.dev.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Coding agents, adversarial evaluation, agent robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dwPYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TrapArena, a benchmark of 113 function- and repository-level tasks evaluating code-repair agents' robustness against plausible, runtime adversarial feedback. Across 12 models, trajectory-conditioned distractions significantly degraded repair success (up to 10.4 pp on SWE-bench and 37.2 pp on HumanEvalFix), while showing that behavioral susceptibility does not always correlate with actual performance loss.

Critical strengths
  1. The authors focus on a relatively important research question in the current coding agent setting: when distracting guidance is provided to coding agents, how will the coding agent's capability be affected?
  2. This paper presents a good observation: being more easily influenced in behavior does not equal greater damage to task performance. In the ablation of the 72B model, after adding curator memory, the repair loss on HumanEvalFix actually decreased.
Critical weaknesses
  1. The TRAPARENA benchmark constructed in the paper has a small number of samples, only 113 tasks, and they are basically constructed based on previously well-known coding agent benchmarks (e.g., SWE-bench Verified, HumanEvalFix) with only simple processing; therefore, the benchmark's contribution is relatively small.
  2. Many models used in the paper's experiments (e.g., deepseek-coder-v2, Codestral-22B) have relatively outdated capabilities, and results from more capable models should be added.
Questions for authors

see weeknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer dwPY

Thank you for recognizing the importance of studying distracting guidance during code repair. We provide our response to the questions below.

W1. Benchmark size and contribution.

TrapArena’s main contribution is a dynamic evaluation environment for testing whether a coding agent can still repair a defect when a collaborator gives it misleading advice. In everyday debugging, a teammate may see a failing test and point to the wrong file. TrapArena can introduce such advice in response to the agent’s actions. We compare the agent’s repair success on the same defect with and without that advice, using the same hidden tests.

The 113 tasks from three benchmarks provide a common test set. We selected the SWE-bench Verified subset for baseline solvability so that a decline under feedback would be observable. Across ten unaided trials per task, MiniMax-M2’s pass rate was 23.5% on the 48 selected tasks and 0.9% on the other 147 candidates. We propose clarifying in §2 how users can bring their own tasks.

Repair outcomes are compared against a baseline without injected feedback. Users can evaluate repair tasks from their own projects by providing a code snapshot, an issue description, and public and hidden tests in the documented task format.

W2. Results for more capable models.

Table 1 includes 12 repair targets, with open-weight models up to 480B total parameters as well as GPT-4o. On the 48 SWE-bench Verified tasks, trajectory-conditioned feedback significantly reduced repair success for both Qwen3-235B-A22B and GPT-4o.

Repair targetUnaided repair rateChange with feedback (percentage points, 95% CI)
Qwen3-Coder-480B-A35B25.7%−7.3 [−15.7, +0.2]
Qwen3-235B-A22B26.2%−3.9 [−7.7, −0.9]
GPT-4o25.0%−5.6 [−10.0, −1.7]

Separate existing runs with GLM-4.6 (FP8) on HumanEvalFix yielded a 96.7% pass rate without advice and 94.7% with prewritten feedback. To clarify how newer models can be tested, we propose documenting in §3.1 the interface provided in our anonymous code release.

Users can evaluate the latest coding model by specifying its model name and an OpenAI-compatible endpoint. This lets them compare models using the same tasks, feedback policy, and hidden tests.

Reviewer_io6sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TrapArena, a benchmark for evaluating code repair agents under misleading collaborative feedback. TrapArena augments repair tasks with plausible yet incorrect advice delivered during execution, while restricting the distractor from directly modifying the repository or altering tool outputs. An evolving curator summarizes the target agent's reactions into persistent memory to adaptively refine distractions across subsequent rounds. This work is well-motivated. In real-world software engineering, coding agents frequently encounter noisy, imprecise, or misleading guidance from users or external context. This requires coding agents to reason robustly and localize faults to remain reliable.

Critical strengths

Timely and Practical Problem Formulation: The paper addresses a realistic and important challenge in human-AI collaborative coding, evaluating agent robustness against plausible but flawed intermediate guidance.

Realistic Threat: Evaluating both single-turn and adaptive multi-turn misleading feedback without corrupting the underlying environment or tool outputs.

Interesting Empirical Findings: The authors uncover an interesting phenomenon where misleading feedback does not uniformly degrade performance. Notably, Qwen3-Coder-30B-A3B improves by 13.7 points on HumanEvalFix, demonstrating that adversarially generated advice can occasionally act as a beneficial perturbation rather than purely harmful interference.

Critical weaknesses

Outdated Model Selection: The evaluated models are relatively dated. Including more recent frontier models (e.g., the latest GPT and Claude series) is necessary to assess whether current state-of-the-art coding agents remain vulnerable to these distractions.

Limited Diversity of Misleading Feedback: It is unclear whether the generated misleading advice is sufficiently diverse. The benchmark may fail to capture the breadth and complexity of misleading feedback encountered in real-world software development.

Questions for authors

Evaluation on Frontier Models: Could you provide supplementary experiments on newer frontier models (such as the latest GPT and Claude models) to demonstrate how state-of-the-art coding agents perform?

Measurement of Susceptibility Rate: How is the susceptibility rate measured? Could you clarify the exact evaluation protocol used to determine whether an agent followed the misleading advice?

Diversity and Coverage of Feedback: How diverse is the generated misleading feedback? What gaps remain between the generated distractions and the breadth and complexity of misleading feedback in real-world software development?

2 comments
AuthorsOpenReview ↗

Response to Reviewer io6s

Thank you for highlighting the practical relevance of imperfect collaborator advice. We provide our response to the questions below.

W1. Capability of the evaluated targets.

Qwen2.5-Coder-32B had a 96.1% no-advice pass rate on HumanEvalFix and lost 20.0 percentage points under feedback (95% CI for the change [−24.7, −15.2]). GPT-4o lost 5.6 points on SWE-bench and 4.3 on HumanEvalFix (Table 1). High baseline performance can coexist with sensitivity to advice.

W2. Feedback diversity.

TrapArena varies both the content of the advice and how it is produced. Table 4 compares five scripted profiles, and §2.2 describes task-specific advice and messages generated from the current interaction. To quantify the coverage of the prewritten collection, we propose adding one sentence to §2.2.

Task-specific message pools are generated offline from the issue and reference patch, then screened before use. The offline collection contains 382 distinct message texts across 108 of the 113 evaluation tasks, with two to four messages per covered task.


Q1. What evidence is available for newer repair targets?

We reanalyzed existing runs with GLM-4.6 (FP8) as the repair target under prewritten feedback. Each task receives equal weight. Intervals use 5,000 paired bootstrap resamples over tasks.

Task sourceTasksNo adviceWith adviceChange in hidden-test pass rate, pp [95% CI]
SWE-bench Verified subset4822.1%20.4%−1.7 [−5.8, +2.3]
HumanEvalFix6096.7%94.7%−2.0 [−6.2, +1.3]
Terminal-Bench subset532.0%6.0%−26.0 [−62.0, 0.0]

The two larger sources show small estimated changes with intervals spanning zero. For the proprietary models, GPT-4o was evaluated as a repair target. GPT-5.4 and GPT-5.4-mini generated feedback.

Q2. How is susceptibility measured?

Thank you for asking for clarification. We use an automatic diagnostic based on the recorded message, test outcomes, and patch location. The paired repair comparison uses the hidden-test outcome. To make both measures reproducible, we have drafted the following clarification for §2.3 and Equation (3).

The evaluator assesses the submitted patch without relying on the collaborator’s claims or the agent’s declaration of completion. For a patch PP, the repair outcome is Y(P)=H(P)Y(P)=H(P), where H(P)=1H(P)=1 if the patch passes all hidden tests for the task and 00 otherwise. We separately record overlap with the reference patch and static flags for superficial fixes as diagnostics.

Our proposed wording for §2.4 specifies the flagging rule and the denominator in Figure 4.

We also report the recorded susceptibility rate, an automatic diagnostic. In the main cross-round analysis, we flag an episode when it receives a message with a designated misleading-feedback tag and the patch passes the public tests but fails the hidden tests. When reference coverage is available, the patch must also leave all reference-edited lines untouched. For each round in Figure 4, we divide the number of flagged episodes by the number of error-free episodes in that round.

For the paired outcome in Table 1, the feedback condition includes only episodes in which at least one message was delivered.

Q3. What does this cover in real development?

Table 4 varies advice about the diagnosis, the proposed edit, edge cases, core logic, and when to submit. On the same 48 tasks, MiniMax-M2’s repair rate ranged from 24.2% to 26.3% across these profiles. This comparison measures sensitivity to specific kinds of guidance. The frequency of each in real projects, disagreements among several reviewers, and comments that become stale as code changes lie outside this controlled setting.

Thank you for the rebuttal. My original concerns remain largely unaddressed, so I maintain my original rating.

Reviewer_w9qfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TrapArena, a benchmark for evaluating code-repair agents under misleading collaborative feedback. It supports trajectory-conditioned attacks and optional curator memory to adapt attacks across rounds. Experiments on 113 tasks and 12 models show that misleading advice can significantly degrade repair success.

Critical strengths
  • Studies an important problem: robustness of coding agents to misleading collaborative feedback.

  • Evaluates actual repair outcomes against a no-attack baseline rather than relying solely on behavioral compliance.

  • Includes multiple models and ablations of attack configurations.

Critical weaknesses
  • Limited evaluation of adaptive adversarial robustness: Robustness is fundamentally concerned with strong, potentially worst-case attacks. TrapArena instead measures average performance against a particular LLM-based attacker. Conditioning on defender trajectories and maintaining natural-language memory constitute contextual adaptation, but do not demonstrate effective adversarial optimization. Without stronger attack baselines or attack-budget scaling, it is unclear whether the reported robustness reflects defender resilience or attacker limitations.

  • Fixed tasks limit conclusions about attacker adaptation: The attacker repeatedly interacts with the same 113 repair tasks, potentially learning task-specific patterns rather than generalizable attack strategies. There is no held-out evaluation demonstrating that strategies developed across rounds transfer to unseen tasks.

  • Unclear effectiveness of curator-based adaptation: Curator memory increases measured susceptibility but does not increase repair degradation, and sometimes reduces it. This raises questions about whether the proposed adaptation mechanism actually improves the attacker's objective of inducing repair failures.

  • Limited generalizability: The benchmark uses a small, selected task collection and a limited set of attack configurations. It remains unclear whether the findings generalize to more diverse repair tasks, stronger attackers, or different interaction settings.

Questions for authors
  • How does TrapArena distinguish genuine defender robustness from limitations of its attack generator?

  • How does attack success scale with attacker optimization or interaction budget?

  • Do curator-learned strategies transfer to held-out repair tasks?

2 comments
AuthorsOpenReview ↗

Response to Reviewer w9qf

Thank you for recognizing the importance of misleading feedback in code repair, the value of a no-advice baseline, and the breadth of our model comparisons. We provide our response to the questions below.

W1. Scope of adaptive robustness.

Equation (4) compares each target on the same tasks with and without misleading feedback. A decline establishes vulnerability to the evaluated generator. A near-zero change at that budget leaves worst-case robustness open. To make the scope of the estimate explicit, we propose the following wording for the Table 1 caption.

Base (%) denotes the task-averaged hidden-test pass rate without feedback. For each target, Δ (pp) gives the paired change in this rate, counting only episodes with delivered advice in the feedback condition.

W2. Repeated exposure to fixed tasks.

Each round restarts from the original code. Only the distractor carries curator notes forward. Reusing the defects keeps task difficulty fixed while we measure what changes across rounds.

W3. Does curator memory increase repair harm?

For Qwen2.5-72B, mean susceptibility rose from 6.5% in the first three rounds to 17.7% in the last three with memory. Without memory it was 6.2% in both windows (Figure 4). Yet on HumanEvalFix, the target’s repair rate was 7.2 percentage points higher with memory than without it (95% CI [+2.2, +12.5]). On SWE-bench the difference was +1.3 points [−1.8, +5.0] (Table 2). The proposed revision to §4.1 explains why the susceptibility score alone would rank these conditions incorrectly on HumanEvalFix.

A flagged episode in the curator ablation fails the hidden tests by construction. The target can also fail that task without advice. The paired estimate captures the net change in success rate over retained tasks, without assigning a counterfactual outcome to each individual episode.

W4. Generality.

The 48 SWE-bench Verified tasks span 10 repositories. The suite also includes 60 HumanEvalFix tasks and five Terminal-Bench tasks. Eight of 12 targets showed a significant decline on at least one of the two larger sources under the main feedback setting (Table 1, individual 95% paired intervals). We report the five-task source separately.

Q1. How do you distinguish defender resilience from generator limitations?

Table 9 varies both sides of the interaction. With Qwen2.5-72B and the feedback filter fixed, the HumanEvalFix change was −7.2 points [−12.7, −2.1] with GPT-5.4-mini feedback and −23.7 [−31.6, −16.6] with GPT-5.4. Holding GPT-5.4 fixed, adding verification instructions to the target reduced the observed loss to 8.2 points against its instruction-matched baseline. Both the generator and the target’s handling of advice affect the outcome.

Q2. How does success scale with budget?

At the fixed cap of 60 target actions, HumanEvalFix repair loss was 4.4 points smaller in late rounds for Qwen2.5-72B with GPT-4o feedback, but 11.7 points larger for Gemma-3-27B with GPT-5.4 feedback (Tables 5 and 6). Additional rounds showed no uniform increase in harm. The experiment did not independently vary the per-episode interaction or optimization budget.

Q3. Do curator notes transfer to unseen tasks?

All 113 tasks appeared during the repeated rounds. The memory ablation establishes what carrying notes forward changes on previously encountered repairs. Transfer to held-out repairs remains open.

Reviewer_w9qfOpenReview ↗

Rebuttal comments

Thank you for the clarification. The rebuttal largely confirms my original concerns. While the findings are useful, the evidence does not yet support broader claims about adaptive adversarial robustness. Therefore, I maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.