acceptodds
Under review as a conference paper at ICLR 2027

Opera: A Verbal Critic Framework for Long-horizon Coding Agents

Abstract

Long-horizon coding agents need timely corrections, yet feedback can be ineffective or even harmful when it misjudges ongoing work or fails to address the underlying problem. Existing critics focus on evaluating trajectories and generating feedback, but rarely track what happens after feedback is delivered. We present Opera, a verbal critic framework that treats each correction as a persistent note, followed until the diagnosed problem is resolved. Opera decides when to review through periodic and event-driven triggers, diagnoses issues with typed operators, audits feedback against visible evidence before delivery, and tracks the agent's subsequent actions to distinguish mere compliance from actual resolution. As a test-time critic, Opera improves the resolve rate of non-critic agents by up to 12.4, 15.0, and 8.9 percentage points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1, respectively, across four policy models, and achieves the highest mean resolve rate among competitive critic baselines on all three benchmarks, and also improves policy models when the policy critiques itself. Beyond inference, Opera-guided rollouts provide approximately on-policy training data: fine-tuning Qwen3.5-9B on them improves its resolve rate on held-out SWE-Bench Pro repositories by 10.2 percentage points without a critic at inference time, matching fine-tuning on rollouts from a stronger model, while preserving its performance when switching harness, i.e., from Openhands to Terminus-2, which the latter substantially degrades.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Coding agent, Critic model, Self-reflection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zoL2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on how to make corrective feedback reliably help a coding agent, where tasks typically span long horizons. The authors argue that existing feedback-based approaches rarely track what happens “after” feedback is delivered, highlighting three challenges: when to intervene, whether the feedback is justified, and whether it resolves the issue. To tackle this, they propose Opera, a verbal critic framework that manages each correction as a “persistent note” to address these challenges. It diagnoses errors, audits feedback before delivery, and tracks each correction as a persistent note until the underlying issue is resolved. Experiments show that it effectively improves task resolve rates at test time on coding tasks by providing relevant feedback, and further introduce it to a critic-guided feedback loop as a training recipe in which a student model is fine-tuned on successful Opera-guided trajectories.

Critical strengths
  1. The idea of maintaining corrective feedback as a persistent note is interesting and intuitively clear. While existing work on long-horizon agents often focuses on generating feedback and leaves its outcomes largely untracked, Opera follows each correction through subsequent actions, which helps distinguish following a suggestion from resolving the underlying issue. I believe this provides a promising design contribution as long-horizon correction may require continued follow-up and revised guidance as execution unfolds.

  2. The training recipe is effective and strikes a good balance between on-policyness and corrective supervision from a stronger model. While many recent works introduce OPD and step-wise RL, Opera offers a practical recipe for turning critic-guided recovery at inference time into (near on-policy) SFT data, which is quite efficient. While I am curious why SFT on teacher trajectories has significantly degraded performance and how Opera compares with OPD, I think this is a practical recipe for teaching students a recovery behavior.

Critical weaknesses
  1. The contributions of the individual components (that were introduced to address each three challenges) are not sufficiently isolated with these analyses not covered in the main pages and only few analyses deferred to the Appendix with limited scope, which makes this paper lean toward rejection. For the first challenge, hybrid scheduling is motivated qualitatively but not compared against periodic-only or event-only review. For the second and third challenges, Table 6 (in Appendix) removes admission and release audits together, making it difficult to separate the effects of screening unjustified feedback and verifying issue resolution. In addition, Figures 9 and 12 only provide statistics on note management, but do not support whether persistent note tracking improves correction outcomes or final task success. I would reassess my recommendation only if these contributions are clearly isolated through ablations.

  2. The gain over baselines is marginal and does not stem from the persistent note design. Opera beats the strongest baseline by only 0.6 to 1.9 pp, which is within one standard deviation. Moreover, without the audits (in Table 6), Opera falls to or below most baselines. This suggests the advantage comes from an extra verification step that could be attached to any baseline, rather than from the persistent note design.

Questions for authors
  1. The analysis in Figure 6(b) is interesting, but could Opera’s persistent note design itself contribute to the observed regressions? If a mistaken or unnecessary correction passes the admission audit, the fixed resolution criterion and one-open-note constraint may keep subsequent reviews focused on that correction. How does the rescue–regression tradeoff change when feedback is still audited but subsequent reviews are not tied to the unresolved note?

  2. Can you explain the detailed reason why the OOD performance after teacher SFT drops sharply? Some possible candidates are forgetting of existing capabilities, specialization to the (in-domain) training harness, or differences in the training data. Given the substantial difference in sample counts between the two sources, I wonder how their trajectory counts and length distributions differ, and whether these differences may have resulted to the gap.

7 comments
AuthorsOpenReview ↗

Responses to Reviewer zoL2 (part 1/3)

We sincerely thank the reviewer for the valuable comments and insightful suggestions, and for recognizing the significance of our framework design and our training recipe. We hope the following clarifications and additional experiments can help address the concerns.

W1: Isolating the contributions of hybrid scheduling, admission/release audits, and persistent note tracking

Q1: Whether persistent notes and the one-open-note constraint amplify regressions from incorrect interventions

To better isolate the contributions of Opera's components and examine the potential risks introduced by persistent corrections, we conduct two additional ablations focusing on (1) when to intervene through review scheduling and (2) how to maintain and manage unresolved corrections through persistent notes. These experiments complement our existing admission/release audit ablation, which evaluates the safeguards applied to proposed corrections.

1. Ablation of Review Scheduling.

We compare hybrid scheduling against periodic-only and event-only variants while keeping the operator critic, audits, and note management unchanged.

MethodTB-2.1 (%)SWE-Pro (%)DeepSWE (%)
Periodic-only71.9 ± 4.178.0 ± 3.040.7 ± 3.9
Event-only72.7 ± 2.878.3 ± 2.540.7 ± 3.2
Opera Full (Hybrid)73.8 ± 2.379.3 ± 3.541.3 ± 3.1

Hybrid scheduling achieves the highest mean resolve rate across all three benchmarks, although both restricted schedules remain competitive. This suggests that periodic and event-driven reviews may offer complementary intervention opportunities, while neither trigger type alone accounts for the full framework's performance.

2. Ablation of Note Persistence.

To examine whether persistent corrections introduce additional risks, we compare Opera with a stateless variant that delivers independently audited feedback without persistent tracking, and a variant that retains persistent notes but removes the single-open-note constraint. Both retain the original hybrid review schedule.

MethodTB-2.1 (%)SWE-Pro (%)DeepSWE (%)
Stateless + Audit73.4 ± 3.478.3 ± 3.540.1 ± 3.1
Persistent w/o Single-Open-Note Constraint70.0 ± 2.377.0 ± 2.638.3 ± 3.6
Opera Full73.8 ± 2.379.3 ± 3.541.3 ± 3.1

The stateless variant retains performance close to Opera Full, suggesting that audited operator feedback provides substantial benefits. However, removing the single-open-note constraint consistently reduces mean resolve rates across all benchmarks. This is consistent with the concern that too many persistent corrections may become counterproductive when multiple unresolved objectives compete for the agent's attention. The results are also added in Section 5.4 of our revised draft.

AuthorsOpenReview ↗

Responses to Reviewer zoL2 (part 2/3)

W2: Whether Opera's marginal gains are driven by additional audits rather than persistent note management

We would like to clarify that auditing and persistent note management are closely coupled in Opera's design. Admission auditing determines whether a proposed correction should become an active persistent note, while release auditing checks whether the original issue has been resolved before the note is closed. To further examine this, we augment runtime baselines with admission auditing and compare an outcome-only variant of Opera against audited outcome critics.

1. Comparison with audited runtime critics

BenchmarkMethodAvg ± Std (%)
TB-2.1SWE-PRM + Audit67.4 ± 3.0
TB-2.1SWE-Search + Audit67.8 ± 1.7
TB-2.1Opera73.8 ± 2.3
SWE-ProSWE-PRM + Audit78.0 ± 1.7
SWE-ProSWE-Search + Audit76.0 ± 3.6
SWE-ProOpera79.3 ± 3.5
DeepSWESWE-PRM + Audit36.9 ± 1.8
DeepSWESWE-Search + Audit38.1 ± 4.1
DeepSWEOpera41.3 ± 3.1

2. Comparison with audited outcome critics

BenchmarkMethodAvg ± Std (%)
TB-2.1LLM-as-verifier + Audit71.9 ± 3.0
TB-2.1Agentic Rubrics + Audit71.5 ± 4.7
TB-2.1Opera Outcome-only73.4 ± 4.5
SWE-ProLLM-as-verifier + Audit78.7 ± 1.5
SWE-ProAgentic Rubrics + Audit78.3 ± 2.9
SWE-ProOpera Outcome-only79.7 ± 3.5
DeepSWELLM-as-verifier + Audit38.3 ± 3.1
DeepSWEAgentic Rubrics + Audit39.8 ± 3.5
DeepSWEOpera Outcome-only40.4 ± 2.7

These results show taht Opera achieves higher than the corresponding baselines with audition augmention across both runtime and outcome-only settings. Importantly, adding admission auditing to existing critics does not consistently improve their performance. Sometimes, additional verification calls can contribute to performance degration in baselines.

To better understand why auditing has a different role in Opera, we examine the proportion of proposed corrections rejected by the admission audit.

MethodTB-2.1 (%)SWE-Pro (%)DeepSWE (%)
SWE-PRM + Audit8.84.75.8
SWE-Search + Audit7.79.24.8
LLM-as-verifier + Audit2.53.26.1
Agentic Rubrics + Audit4.55.95.2
Opera Outcome-only21.920.925.8
Opera Full25.621.224.5

The audit rejects a substantially larger proportion of Opera's proposed corrections than those of existing baselines, indicating that it plays a more active filtering role. One explanation is that Opera's operator-typed feedback is more specific and intervention-oriented, proposing concrete diagnoses and corrective actions that may redirect execution when unsupported. These corrections can be maintained as persistent notes and influence subsequent reviews, making their admission and eventual closure particularly consequential, thus making audition to play a more crucial role here. The results are also added in Appendix B.1 of our revised draft.

AuthorsOpenReview ↗

Responses to Reviewer zoL2 (part 3/3)

Q2: More analysis and explanations of teacher-SFT OOD degradation

To better understand the cross-harness OOD degradation of Teacher-SFT, we analyze the differences in supervision characteristics and downstream agent behavior between Teacher-SFT and Opera-guided SFT. Both settings use the same selected training tasks but differ in their trajectory sources and supervised output lengths.

Training Data# SamplesSupervised TokensAvg. Supervised Output Tokens / Sample
Opera-guided SFT8,3782.69M321.1
Teacher-SFT3,4762.25M647.3

Teacher-SFT contains approximately 2.02× as many supervised output tokens per sample as Opera-guided SFT, with much of the difference attributable to longer reasoning outputs.

To examine whether this pattern persists during execution, we compare the average output length and valid tool-call generation rate of finetuned models across evaluation harnesses.

Evaluation HarnessModelAvg. Output Tokens / TurnOutputs Containing Valid Tool Calls (%)
SWE-Pro (OpenHands)Base301.299.4
SWE-Pro (OpenHands)Opera-guided SFT325.899.5
SWE-Pro (OpenHands)Teacher-SFT644.194.9
TB-2.1 (Terminus 2)Base378.898.9
TB-2.1 (Terminus 2)Opera-guided SFT405.698.8
TB-2.1 (Terminus 2)Teacher-SFT890.980.4

We observe that Teacher-SFT produces substantially longer outputs and fewer valid tool calls on both benchmarks, with a more pronounced reduction in valid tool-call generation on TB-2.1. As the teacher demonstrations are collected using OpenHands, their reasoning and action patterns are more closely aligned with SWE-Pro evaluation. In contrast, transferring these patterns to Terminus 2 can introduce additional difficulties in producing valid actions. Although harness-level retry mechanisms can sometimes recover from invalid outputs, the substantially lower valid tool-call rate on TB-2.1 can disrupt task execution more frequently, consistent with its larger performance degradation. Above results and analysis are also added in Appendix C.4 of our revised draft.

These observations suggest a potential reasoning–action mismatch: directly imitating longer reasoning trajectories from a stronger teacher may encourage a weaker student to overemphasize reasoning at the expense of timely and correctly formatted tool calls, particularly when transferring across harnesses. This is consistent with prior findings on the difficulty of distilling complex reasoning into smaller models like [1,2].

[1] Li et al. Small Models Struggle to Learn from Strong Reasoners. ACL 2025

[2] Chen et al. Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning. ACL 2025

Reviewer_Z1dFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Opera, a verbal critic framework for coding agents. Opera manages corrective feedback over agent trajectories through persistent notes: each note records one diagnosed issue, its current guidance, and a fixed resolution criterion. The critic thereby follows up on its own feedback across long-horizon runs until the issue is resolved or the note is refined.

Critical strengths

The paper’s most consequential strength is that Opera is not merely another prompting-based critic. It formulates verbal criticism as the management of persistent corrective notes, and couples this with hybrid review scheduling, operator-typed diagnoses, and audited note admission/closure. This design directly targets the three hard problems of long-horizon agent supervision: when to review, how to deliver actionable feedback, and how to verify that the original issue is actually resolved rather than merely acknowledged.

Critical weaknesses

The central baseline comparison in Table 1 fixes the policy model to Qwen3.8-27B and the critic model to GPT-5.6-Sol, and compares Opera against four critic baselines under that single configuration. Although Figure 4 varies the policy model and Section 5.3/Appendix D.2 discuss critic-model choice, the paper does not provide a systematic cross-product evaluation in which the four competitive baselines are also run under weaker critics (e.g., self-critique or a smaller critic). This matters because Opera’s core mechanism—persistent notes, admission audit, and release audit—depends heavily on the quality of the critic’s diagnoses. If the critic is substantially weaker, it may generate noisier notes, misjudge resolution criteria, or fail the admission audit in ways that erode the framework’s advantage.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Responses to Reviewer Z1dF

We sincerely thank the reviewer for recognizing the significance of our problem formulation, our design and we hope the following additional experiments can help address the concern.

W1: Comparison with critic baselines using weaker critic models

We agree that the effectiveness of critic-based methods depend on the capability of the critic model. To further investigate the effects of using weaker critics on other baselines, we evaluate all critic baselines using Qwen3.8-27B as both the policy and the critic to compare their performance under the same self-critic setting.

MethodTB-2.1 (%)SWE-Pro (%)DeepSWE (%)
Non-critic65.9 ± 1.775.3 ± 1.232.4 ± 1.0
SWE-PRM (Self)65.5 ± 3.974.7 ± 1.532.7 ± 2.7
SWE-Search (Self)65.2 ± 5.175.3 ± 2.533.6 ± 2.3
LLM-as-verifier (Self)66.3 ± 3.076.3 ± 3.135.1 ± 2.7
Agentic Rubrics (Self)65.5 ± 3.976.0 ± 2.035.7 ± 4.5
Opera Full (Self)66.7 ± 3.277.0 ± 2.638.6 ± 3.4

The results suggest that with a weaker self-critic, several baselines perform similarly to or below the non-critic policy, suggesting that additional feedback is not necessarily beneficial when its quality is limited. Nevertheless, Opera still maintains the highest, consistently improving over the non-critic policy. Overall, the results suggest that Opera's relative performance advantage is retained under self-criticism, while also highlighting the importance of critic capability for the effectiveness of feedback-based methods. The results are also added in Appendix B.2 of our revised draft.

Dear Reviewer,

As the rebuttal period is approaching its end, we just wanted to gently follow up on our responses and the additional experiments we provided to address your concerns. We would also be happy to clarify or discuss any remaining concerns before the discussion period closes.

Thank you again for your time, thoughtful feedback, and consideration.

Best regards,
The Opera authors

Reviewer_v2yBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

OPERA is a stateful verbal critic for long-horizon coding agents. It uses periodic/event-driven reviews to discover problems, a typed operator to generate a diagnosis and resolution criterion, an admission audit before delivery, and subsequent trajectories to distinguish adherence from actual resolution; a release audit finally closes the note. The authors evaluate the test-time critic on three coding benchmarks, four policy models, and multiple harnesses, and use OPERA-guided student trajectories for SFT.

Critical strengths
  • The persistent note’s resolution criterion is a clear and practical abstraction. It turns one piece of natural-language feedback into a correction target that can be tracked over time, avoiding the mistake of treating “the model attempted to follow the suggestion” as “the original bug was solved.”
  • The coverage across policies, critics, benchmarks, and harnesses is broad. Audit ablations, self versus external critics, and rescue–regression analyses suggest that the framework can help in multiple settings rather than only on one model.
Critical weaknesses
  • The inference budget is not matched, so the performance improvement cannot be attributed. A single OPERA review may involve multiple calls from the operator critic, admission audit, and release audit, in addition to both periodic and event-driven triggers. The baselines are not matched by tokens, calls, latency, or GPU/API cost. The current claim of being “better than other critics” may simply mean “using more model evaluations.”
  • The key components and training gains are subject to selection bias. Jointly removing only the two audit types does not establish the independent contribution of persistent notes, fixed criteria, event triggers, or a single-open-note policy. The SFT experiment also selects 67 tasks that are recoverable and have successful trajectories, which may overestimate transfer.
Questions for authors
  1. Does OPERA still outperform each baseline under matched critic tokens, number of calls, wall-clock time, and timeout budget? If not, how should the paper’s main contribution be repositioned?
  2. Can the authors separately ablate note persistence, fixed criteria, event/periodic scheduling, the single-open-note policy, and ad On the complete, unfiltered training split and without retaining only successful trajectories, does OPERA-guided SFT retain its gains on held-out tasks and across harnesses?
  3. Are the gains stable when the last-30-message truncation, tool-output format, and critic timeout are changed?
6 comments
AuthorsOpenReview ↗

Responses to Reviewer v2yB (Part 1/3)

We sincerely thank the reviewer for the valuable comments and insightful suggestions, and for recognizing the significance of our problem formulation, our framework and our ablation studies. We hope the following clarifications and additional experiments can help address the concerns.

W1 & Q1: Comparison with critic baselines under matched inference budgets

To better understand whether additional critic LLM calls account for Opera's improvements, we conduct two complementary comparisons: (1) augmenting existing runtime critics (SWE-PRM and SWE-Search) with an admission audit, and (2) introducing an outcome-only variant of Opera to enable a more comparable evaluation against outcome critics (LLM-as-verifier and Agentic Rubrics). We also report the average number of critic LLM calls per task to characterize the inference budget of each method.

1. Comparison with audited runtime critics

BenchmarkMethodAvg. Critic Calls / TaskAvg ± Std (%)
TB-2.1SWE-PRM + Audit20.867.4 ± 3.0
TB-2.1SWE-Search + Audit23.167.8 ± 1.7
TB-2.1Opera21.273.8 ± 2.3
SWE-ProSWE-PRM + Audit8.278.0 ± 1.7
SWE-ProSWE-Search + Audit9.176.0 ± 3.6
SWE-ProOpera6.879.3 ± 3.5
DeepSWESWE-PRM + Audit35.636.9 ± 1.8
DeepSWESWE-Search + Audit38.038.1 ± 4.1
DeepSWEOpera38.441.3 ± 3.1

2. Comparison with audited outcome critics

BenchmarkMethodAvg. Critic Calls / TaskAvg ± Std (%)
TB-2.1LLM-as-verifier + Audit1.5871.9 ± 3.0
TB-2.1Agentic Rubrics + Audit1.6371.5 ± 4.7
TB-2.1Opera Outcome-only1.7073.4 ± 4.5
SWE-ProLLM-as-verifier + Audit1.8278.7 ± 1.5
SWE-ProAgentic Rubrics + Audit1.7978.3 ± 2.9
SWE-ProOpera Outcome-only1.8479.7 ± 3.5
DeepSWELLM-as-verifier + Audit1.3638.3 ± 3.1
DeepSWEAgentic Rubrics + Audit1.2839.8 ± 3.5
DeepSWEOpera Outcome-only1.4040.4 ± 2.7

Overall, Opera achieves higher mean resolve rates than the audited baselines under comparable critic LLM call budgets, while adding an admission audit to existing baselines does not consistently improve their performance. To better understand this difference, we analyze the proportion of critic feedback proposals rejected by the audit.

MethodTB-2.1 (%)SWE-Pro (%)DeepSWE (%)
SWE-PRM + Audit8.84.75.8
SWE-Search + Audit7.79.24.8
LLM-as-verifier + Audit2.53.26.1
Agentic Rubrics + Audit4.55.95.2
Opera Outcome-only21.920.925.8
Opera Full25.621.224.5

The audit rejects substantially more feedback proposals in Opera than in the existing baselines, indicating a more active filtering role. We look through the audition cases and conjecture that Opera's operator-style feedback, which offers concrete diagnoses and corrective actions, can be more consequential when unsupported, especially when retained as a persistent note that influences subsequent reviews. Therefore, auditing is particularly important for deciding which corrections should be delivered and maintained. Together with the performance degradation observed when removing Opera's audits, these results suggest that the value of auditing depends on the nature of the proposed interventions and how they are managed, rather than simply adding more LLM verification calls. Above results are also added into Appendx B.1 of our revised draft.

AuthorsOpenReview ↗

Responses to Reviewer v2yB (part 2/3)

W2: Finer-grained ablations of Opera's components

We thank the reviewer for pointing out the need to better isolate the contributions of Opera's individual components. We conducted two additional groups of ablations focusing on (1) review scheduling and (2) note persistence, which corresponds to our research questions: when to intervene and how to track the intervention.

1. Ablation of Review Scheduling.

We compare the full hybrid scheduling strategy with periodic-only and event-only variants, while keeping the operator critic, audits, and persistent note management unchanged.

MethodTB-2.1 (%)SWE-Pro (%)DeepSWE (%)
Periodic-only71.9 ± 4.178.0 ± 3.040.7 ± 3.9
Event-only72.7 ± 2.878.3 ± 2.540.7 ± 3.2
Opera Full (Hybrid)73.8 ± 2.379.3 ± 3.541.3 ± 3.1

The full hybrid schedule achieves the highest, although the periodic-only and event-only scheduling can have comparative performance.

2. Ablation of Note Persistence.

To examine the contribution of persistent note management, we consider the following variants: (1) Stateless + Audit: removing note persistence while retaining operator-typed feedback and admission auditing, and (2) Persistent w/o Single-Open-Note Constraint: removing the single-open-note constraint to allow the critic to consider new issues without remaining committed to the current unresolved note. Both variants retain the original hybrid review schedule.

MethodTB-2.1 (%)SWE-Pro (%)DeepSWE (%)
Stateless + Audit73.4 ± 3.478.3 ± 3.540.1 ± 3.1
Persistent w/o Single-Open-Note Constraint70.0 ± 2.377.0 ± 2.638.3 ± 3.6
Opera Full73.8 ± 2.379.3 ± 3.541.3 ± 3.1

The stateless variant retains substantial performance while removing the single-open-note constraint reduces the mean resolve rate relatively significant. This result is consistent with our motivation for maintaining a focused correction objective: allowing the critic to shift attention away from an unresolved issue may weaken the continuity of corrective guidance. Results are also added into Section 5.4 of our revised draft.

AuthorsOpenReview ↗

Responses to Reviewer v2yB (part 3/3)

Q2: The effects of SFT training-data selection

Training on successful trajectories is a common practice in coding-agent SFT and rejection-sampling fine-tuning, as demonstrated by SWE-Gym [1], SWE-Prime [2]. Similarly, we select successful Opera-guided trajectories to provide outcome-verified demonstrations of task execution and error recovery. To further examine the impact of this selection and its data efficiency, we additionally train an unfiltered variant using both successful and unsuccessful trajectories from the 270 task pool.

Training Data# Data SamplesSupervised TokensSWE-Pro OOD (%)TB-2.1 OOD (%)
Base——28.7 ± 1.923.6 ± 1.9
Opera-guided SFT8,3782.69M38.9 ± 2.425.8 ± 2.2
Opera-guided SFT (unfiltered)29,6828.13M39.8 ± 4.027.3 ± 3.6

The unfiltered variant achieves slightly higher performance on both OOD benchmarks, suggesting that some unsuccessful trajectories may contain locally correct intermediate steps. However, successful-only SFT achieves competitive performance using only 28.2% of the training samples and 33.1% of the supervised tokens compared with the unfiltered variant. In particular, on SWE-Pro OOD, it achieves a comparable resolve rate with substantially less supervision. These results support the data efficiency of our selective training recipe in terms of supervised data volume, while also indicating that unsuccessful trajectories are not inherently unhelpful. Results are also added into Appendix C.3 of our revised draft.

Q3: Robustness to context truncation, tool-output formatting, and critic timeout

Our evaluation has already covered the resistence to these factors that are inherited from the harness differences and critic configuration differences.

Context management. Opera is evaluated with multiple coding-agent harnesses, including Terminus 2, mini-swe-agent, and OpenHands. In particular, Terminus 2 and OpenHands already support history condensation. And Opera operates on the observations made available by the underlying harness rather than requiring access to the complete execution history. This design allows it to work with different context-management mechanisms.

Tool-output formatting. The evaluated harnesses also already differ in their action and observation interfaces. Terminus 2 and Mini-swe-agent uses text-based command parsing, while OpenHands supports structured tool calls and action-observation records; Opera processes these interactions through harness-specific adapters that provide a consistent representation to the critic.

Critic timeout. Opera adopts a fail-open mechanism to handle critic failures. As described in Appendix A, when a critic request times out or returns an unparseable response, the proxy forwards the original request to the policy without applying the intervention, allowing agent execution to continue. This prevents critic failures from blocking the underlying agent, although missed reviews may reduce correction opportunities. We have also evaluated Opera with different critic models, demonstrating compatibility with different inference configurations.

Overall, Opera has been evaluated across heterogeneous agent harnesses and critic configurations, providing evidence of practical compatibility with different execution interfaces and operating conditions.

[1] Pan et al. Training Software Engineering Agents and Verifiers with SWE-Gym. ICML 2025

[2] Zheng et al. SWE-Prime: Fewer Trajectories, Better Performance. 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.