- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on how to make corrective feedback reliably help a coding agent, where tasks typically span long horizons. The authors argue that existing feedback-based approaches rarely track what happens “after” feedback is delivered, highlighting three challenges: when to intervene, whether the feedback is justified, and whether it resolves the issue. To tackle this, they propose Opera, a verbal critic framework that manages each correction as a “persistent note” to address these challenges. It diagnoses errors, audits feedback before delivery, and tracks each correction as a persistent note until the underlying issue is resolved. Experiments show that it effectively improves task resolve rates at test time on coding tasks by providing relevant feedback, and further introduce it to a critic-guided feedback loop as a training recipe in which a student model is fine-tuned on successful Opera-guided trajectories.
Critical strengths
-
The idea of maintaining corrective feedback as a persistent note is interesting and intuitively clear. While existing work on long-horizon agents often focuses on generating feedback and leaves its outcomes largely untracked, Opera follows each correction through subsequent actions, which helps distinguish following a suggestion from resolving the underlying issue. I believe this provides a promising design contribution as long-horizon correction may require continued follow-up and revised guidance as execution unfolds.
-
The training recipe is effective and strikes a good balance between on-policyness and corrective supervision from a stronger model. While many recent works introduce OPD and step-wise RL, Opera offers a practical recipe for turning critic-guided recovery at inference time into (near on-policy) SFT data, which is quite efficient. While I am curious why SFT on teacher trajectories has significantly degraded performance and how Opera compares with OPD, I think this is a practical recipe for teaching students a recovery behavior.
Critical weaknesses
-
The contributions of the individual components (that were introduced to address each three challenges) are not sufficiently isolated with these analyses not covered in the main pages and only few analyses deferred to the Appendix with limited scope, which makes this paper lean toward rejection. For the first challenge, hybrid scheduling is motivated qualitatively but not compared against periodic-only or event-only review. For the second and third challenges, Table 6 (in Appendix) removes admission and release audits together, making it difficult to separate the effects of screening unjustified feedback and verifying issue resolution. In addition, Figures 9 and 12 only provide statistics on note management, but do not support whether persistent note tracking improves correction outcomes or final task success. I would reassess my recommendation only if these contributions are clearly isolated through ablations.
-
The gain over baselines is marginal and does not stem from the persistent note design. Opera beats the strongest baseline by only 0.6 to 1.9 pp, which is within one standard deviation. Moreover, without the audits (in Table 6), Opera falls to or below most baselines. This suggests the advantage comes from an extra verification step that could be attached to any baseline, rather than from the persistent note design.
Questions for authors
-
The analysis in Figure 6(b) is interesting, but could Opera’s persistent note design itself contribute to the observed regressions? If a mistaken or unnecessary correction passes the admission audit, the fixed resolution criterion and one-open-note constraint may keep subsequent reviews focused on that correction. How does the rescue–regression tradeoff change when feedback is still audited but subsequent reviews are not tied to the unresolved note?
-
Can you explain the detailed reason why the OOD performance after teacher SFT drops sharply? Some possible candidates are forgetting of existing capabilities, specialization to the (in-domain) training harness, or differences in the training data. Given the substantial difference in sample counts between the two sources, I wonder how their trajectory counts and length distributions differ, and whether these differences may have resulted to the gap.
7 comments
Responses to Reviewer zoL2 (part 2/3)
W2: Whether Opera's marginal gains are driven by additional audits rather than persistent note management
We would like to clarify that auditing and persistent note management are closely coupled in Opera's design. Admission auditing determines whether a proposed correction should become an active persistent note, while release auditing checks whether the original issue has been resolved before the note is closed. To further examine this, we augment runtime baselines with admission auditing and compare an outcome-only variant of Opera against audited outcome critics.
1. Comparison with audited runtime critics
| Benchmark | Method | Avg ± Std (%) |
|---|---|---|
| TB-2.1 | SWE-PRM + Audit | 67.4 ± 3.0 |
| TB-2.1 | SWE-Search + Audit | 67.8 ± 1.7 |
| TB-2.1 | Opera | 73.8 ± 2.3 |
| SWE-Pro | SWE-PRM + Audit | 78.0 ± 1.7 |
| SWE-Pro | SWE-Search + Audit | 76.0 ± 3.6 |
| SWE-Pro | Opera | 79.3 ± 3.5 |
| DeepSWE | SWE-PRM + Audit | 36.9 ± 1.8 |
| DeepSWE | SWE-Search + Audit | 38.1 ± 4.1 |
| DeepSWE | Opera | 41.3 ± 3.1 |
2. Comparison with audited outcome critics
| Benchmark | Method | Avg ± Std (%) |
|---|---|---|
| TB-2.1 | LLM-as-verifier + Audit | 71.9 ± 3.0 |
| TB-2.1 | Agentic Rubrics + Audit | 71.5 ± 4.7 |
| TB-2.1 | Opera Outcome-only | 73.4 ± 4.5 |
| SWE-Pro | LLM-as-verifier + Audit | 78.7 ± 1.5 |
| SWE-Pro | Agentic Rubrics + Audit | 78.3 ± 2.9 |
| SWE-Pro | Opera Outcome-only | 79.7 ± 3.5 |
| DeepSWE | LLM-as-verifier + Audit | 38.3 ± 3.1 |
| DeepSWE | Agentic Rubrics + Audit | 39.8 ± 3.5 |
| DeepSWE | Opera Outcome-only | 40.4 ± 2.7 |
These results show taht Opera achieves higher than the corresponding baselines with audition augmention across both runtime and outcome-only settings. Importantly, adding admission auditing to existing critics does not consistently improve their performance. Sometimes, additional verification calls can contribute to performance degration in baselines.
To better understand why auditing has a different role in Opera, we examine the proportion of proposed corrections rejected by the admission audit.
| Method | TB-2.1 (%) | SWE-Pro (%) | DeepSWE (%) |
|---|---|---|---|
| SWE-PRM + Audit | 8.8 | 4.7 | 5.8 |
| SWE-Search + Audit | 7.7 | 9.2 | 4.8 |
| LLM-as-verifier + Audit | 2.5 | 3.2 | 6.1 |
| Agentic Rubrics + Audit | 4.5 | 5.9 | 5.2 |
| Opera Outcome-only | 21.9 | 20.9 | 25.8 |
| Opera Full | 25.6 | 21.2 | 24.5 |
The audit rejects a substantially larger proportion of Opera's proposed corrections than those of existing baselines, indicating that it plays a more active filtering role. One explanation is that Opera's operator-typed feedback is more specific and intervention-oriented, proposing concrete diagnoses and corrective actions that may redirect execution when unsupported. These corrections can be maintained as persistent notes and influence subsequent reviews, making their admission and eventual closure particularly consequential, thus making audition to play a more crucial role here. The results are also added in Appendix B.1 of our revised draft.
Responses to Reviewer zoL2 (part 3/3)
Q2: More analysis and explanations of teacher-SFT OOD degradation
To better understand the cross-harness OOD degradation of Teacher-SFT, we analyze the differences in supervision characteristics and downstream agent behavior between Teacher-SFT and Opera-guided SFT. Both settings use the same selected training tasks but differ in their trajectory sources and supervised output lengths.
| Training Data | # Samples | Supervised Tokens | Avg. Supervised Output Tokens / Sample |
|---|---|---|---|
| Opera-guided SFT | 8,378 | 2.69M | 321.1 |
| Teacher-SFT | 3,476 | 2.25M | 647.3 |
Teacher-SFT contains approximately 2.02× as many supervised output tokens per sample as Opera-guided SFT, with much of the difference attributable to longer reasoning outputs.
To examine whether this pattern persists during execution, we compare the average output length and valid tool-call generation rate of finetuned models across evaluation harnesses.
| Evaluation Harness | Model | Avg. Output Tokens / Turn | Outputs Containing Valid Tool Calls (%) |
|---|---|---|---|
| SWE-Pro (OpenHands) | Base | 301.2 | 99.4 |
| SWE-Pro (OpenHands) | Opera-guided SFT | 325.8 | 99.5 |
| SWE-Pro (OpenHands) | Teacher-SFT | 644.1 | 94.9 |
| TB-2.1 (Terminus 2) | Base | 378.8 | 98.9 |
| TB-2.1 (Terminus 2) | Opera-guided SFT | 405.6 | 98.8 |
| TB-2.1 (Terminus 2) | Teacher-SFT | 890.9 | 80.4 |
We observe that Teacher-SFT produces substantially longer outputs and fewer valid tool calls on both benchmarks, with a more pronounced reduction in valid tool-call generation on TB-2.1. As the teacher demonstrations are collected using OpenHands, their reasoning and action patterns are more closely aligned with SWE-Pro evaluation. In contrast, transferring these patterns to Terminus 2 can introduce additional difficulties in producing valid actions. Although harness-level retry mechanisms can sometimes recover from invalid outputs, the substantially lower valid tool-call rate on TB-2.1 can disrupt task execution more frequently, consistent with its larger performance degradation. Above results and analysis are also added in Appendix C.4 of our revised draft.
These observations suggest a potential reasoning–action mismatch: directly imitating longer reasoning trajectories from a stronger teacher may encourage a weaker student to overemphasize reasoning at the expense of timely and correctly formatted tool calls, particularly when transferring across harnesses. This is consistent with prior findings on the difficulty of distilling complex reasoning into smaller models like [1,2].
[1] Li et al. Small Models Struggle to Learn from Strong Reasoners. ACL 2025
[2] Chen et al. Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning. ACL 2025
Responses to Reviewer zoL2 (part 1/3)
We sincerely thank the reviewer for the valuable comments and insightful suggestions, and for recognizing the significance of our framework design and our training recipe. We hope the following clarifications and additional experiments can help address the concerns.
To better isolate the contributions of Opera's components and examine the potential risks introduced by persistent corrections, we conduct two additional ablations focusing on (1) when to intervene through review scheduling and (2) how to maintain and manage unresolved corrections through persistent notes. These experiments complement our existing admission/release audit ablation, which evaluates the safeguards applied to proposed corrections.
1. Ablation of Review Scheduling.
We compare hybrid scheduling against periodic-only and event-only variants while keeping the operator critic, audits, and note management unchanged.
Hybrid scheduling achieves the highest mean resolve rate across all three benchmarks, although both restricted schedules remain competitive. This suggests that periodic and event-driven reviews may offer complementary intervention opportunities, while neither trigger type alone accounts for the full framework's performance.
2. Ablation of Note Persistence.
To examine whether persistent corrections introduce additional risks, we compare Opera with a stateless variant that delivers independently audited feedback without persistent tracking, and a variant that retains persistent notes but removes the single-open-note constraint. Both retain the original hybrid review schedule.
The stateless variant retains performance close to Opera Full, suggesting that audited operator feedback provides substantial benefits. However, removing the single-open-note constraint consistently reduces mean resolve rates across all benchmarks. This is consistent with the concern that too many persistent corrections may become counterproductive when multiple unresolved objectives compete for the agent's attention. The results are also added in Section 5.4 of our revised draft.